Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
PipeParallel always combines: required main stuff invariant (delete combined_output, add native.Composite) (#1014)
Louis Choquel1215516de3cd
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 155195698..6f47b107f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,20 +2,31 @@## [Unreleased]+### Changed++- **`PipeParallel` always combines (breaking):** A parallel now always combines its branch outputs into its declared `output` concept and stamps the combination as its main output — a pipe run always delivers a main stuff. The `combined_output` field is deleted from the MTHDS language: `output` is now the (single, truthful) combination concept. Migration: pipes declaring both fields just drop the `combined_output` line; `add_each_output`-only pipes replace their placeholder `output` with `Composite` or a structured concept matching the branch `result` names. `add_each_output` keeps its meaning (also expose each branch output by name) and now defaults to `false` with no one-of-two constraint.+- **Static validation of parallel outputs:** The declared `output` of a `PipeParallel` is validated at author time: it must be `Composite` or a structured concept whose fields are compatible with the branch `result` names (required fields ⊆ result names; result names ⊆ declared fields) **and types** — when a field is typed with a content class, the branch feeding it must produce that concept or one refining it (conceptual compatibility: a concept refining `Text` fits a `TextContent` field). Scalar native concepts (`Text`, `Image`, ...), `Dynamic`, `Anything`, multiplicity suffixes (`Foo[]`, `Foo[N]`), and type-incompatible branches are rejected by `/validate` and library loading instead of failing at runtime inside the combine step.+- **The main-stuff invariant is now enforced at every post-run boundary (breaking):** With every pipe stamping a main output, the defensive "maybe there is no main stuff" branches for completed runs are gone. Wire models make the field required (`main_stuff_name: str` on `PipelexPipeRunOutput` and on the `/execute` response extension), result delivery always writes the `main_stuff.*` artifact files (a completed run with no main stuff now fails delivery loudly instead of storing an empty envelope), and the graph tracer and OTel/Langfuse output capture read the main stuff directly. `PipeOutput.optional_main_stuff` is deleted — use `PipeOutput.main_stuff`; `WorkingMemory.get_optional_main_stuff()` remains for pre-run/empty memories.+- **Orchestrator SPI: the delivery axis is now two methods (breaking):** `OrchestratorProtocol.run` no longer takes a `delivery` parameter — it is the BLOCKING arm and returns the completed-run `PipelexPipeRunOutput` (which therefore requires `main_stuff_name` with no "not finished yet" escape hatch). The FIRE_AND_FORGET arm is the new `OrchestratorProtocol.start`, returning the new ids-only `PipelexPipeDispatchAck` (`pipeline_run_id` + `workflow_id`). `PipelexPipeRunOutput.is_completed` is deleted (it is always a completed output now); `DeliveryMode` remains the wire-input vocabulary that picks which method the endpoint calls. Orchestrators that cannot do genuine async (core's DIRECT) implement `start` by raising, unreachable behind the `supports_fire_and_forget` gate. Migration: orchestrator plugins split their `match delivery` bodies into the two methods; `/start`-serving runners call `orchestrator.start(...)` and read the ack's `workflow_id`.+- **`PipeParallel` honors `final_stuff_code`:** A requested final stuff code (e.g. pre-allocated by a `PipeBatch` for its branch outputs) is now stamped on the parallel's combined stuff instead of being discarded, so the combined output keeps its graph identity when a parallel runs as a batch branch.++- **Dry-run mock lists shrink from 3 to 2 items:** When a pipe input is declared as a list without a fixed size, the dry-run mock now fabricates 2 items instead of 3. Cuts dry-run work on list inputs without changing correctness.+- **Repo lint validates `.mthds` files against the locally generated MTHDS schema:** `plxt lint` in this repo now uses `derived/mthds_schema.json` (regenerated automatically by the lint make targets) instead of the released schema bundled with plxt, so language-surface additions like `size` are lintable here before they propagate to a plxt release.+### Added+- **`native.Composite` concept:** New native concept backed by `CompositeContent` — an untyped named composition holding its sub-contents as top-level fields (no wrapper key). It is the combination vehicle for parallels whose authors don't want to declare a bespoke concept, and supports the full content surface: `smart_dump`, kajson/transport round-trip, and markdown/HTML rendering.- **Portable image size for image generation (`size` on `PipeImgGen`):** A single `size` field carries a portable size intent across providers — either a size tier (`"1k"`, `"2k"`, `"4k"`: "this pixel class at my chosen `aspect_ratio`", mapped to each provider's own grid or computed dimensions) or an exact pixel size like `"2048x1152"` (deterministic; setting `aspect_ratio` alongside an exact size is a validation error since the size implies the ratio). Unsatisfiable requests are hard validation errors at blueprint-load time (or at runtime for late-resolved aliases/presets), never warn-and-ignore: Gemini 3-class models accept `1k`/`2k`/`4k`, `gpt-image-2` accepts `1k`/`2k` and arbitrary exact sizes within OpenAI's constraints, Gemini 2.5 / Flash Lite / legacy GPT Image / Flux / SDXL / Qwen accept `1k` only. On tier-grid (Gemini) models an exact size runs only on an exact grid-cell match, with near-misses rejected naming the nearest valid cells. When `size` is unset no size intent is sent (Gemini jobs omit `image_size`; OpenAI models keep their fixed 1K-class preset for the chosen ratio) and the provider default applies. The MTHDS JSON Schema exposes the field as tier-enum-or-`WxH`-pattern, and an optional `size` default is supported in `[cogt.img_gen_config.img_gen_param_defaults]`.- **Static validation for Google image models:** Google image models now carry `rules` blocks in the backend deck (new geometry taxonomies `gemini_2_5`, `gemini_3_pro`, `gemini_3_flash`, `gemini_3_flash_lite`), so unsupported (aspect ratio × size) combinations on Gemini models are caught at blueprint-load time instead of only failing at the provider call. The Google img-gen factory is keyed by taxonomy instead of hardcoded model names (breaking: the `GoogleImageGenModel` name enum is removed — model handles are deck config, not code constants).- **Gemini image-to-image (img2img) support:** The Google img-gen worker now forwards input images to the Gemini API as inline parts alongside the text prompt, enabling image editing and multi-image composition with the Nano Banana models. All Google-backend image models declare image inputs (`nano-banana`, `nano-banana-pro`, `nano-banana-2`, `nano-banana-2-lite` — matching the same models' declarations on the OpenRouter backend and the Pipelex gateway); `nano-banana-2` (`gemini-3.1-flash-image-preview`) is supported by the dimensions factory (same 1K/2K/4K resolution grid as Nano Banana Pro, per Google's published tables); added `nano-banana-2-lite` (`gemini-3.1-flash-lite-image`, 1K only) to the Google backend. Every img-gen worker now validates input images against the model's declared capability at job start: a model without image inputs (no `"images"` in its `inputs`) rejects img2img jobs with a clean `ImgGenParameterError` before any provider call — enforced once in the worker base class, so direct-request workers (Google Gemini native, chat-completions image models) and args-factory workers all fail early and explicitly instead of forwarding an unsupported request to the provider.- **Banner aspect ratios (`landscape_4_1`, `landscape_8_1`, `portrait_1_4`, `portrait_1_8`):** New `AspectRatio` values for the extreme banner/strip formats introduced by the Gemini 3.1 image models, supported by `nano-banana-2` and `nano-banana-2-lite` with Google's published pixel grids. All other image-gen backends reject them with a clean parameter error. The Google dimension tables were restructured by model generation (Gemini 2.5 grid vs Gemini 3 grid) with per-model capability gating, so unsupported ratio/size combinations raise `ImgGenParameterError` instead of crashing.-### Changed--- **Dry-run mock lists shrink from 3 to 2 items:** When a pipe input is declared as a list without a fixed size, the dry-run mock now fabricates 2 items instead of 3. Cuts dry-run work on list inputs without changing correctness.-- **Repo lint validates `.mthds` files against the locally generated MTHDS schema:** `plxt lint` in this repo now uses `derived/mthds_schema.json` (regenerated automatically by the lint make targets) instead of the released schema bundled with plxt, so language-surface additions like `size` are lintable here before they propagate to a plxt release.-### Fixed+- **`Composite` components keep their types across transport:** `WorkingMemory.dump_for_transport()` now stamps each composite component with the pipelex-private `__pipelex_class__`/`__pipelex_module__` markers (the same convention as list items), and hydration rebuilds typed `StuffContent` components from them — including `ListContent` components with typed items. Previously a delivered `native.Composite` hydrated with plain `dict` components, breaking typed access and rendering after an API/worker round-trip.+- **`add_each_output` is optional in the builder spec:** `PipeParallelSpec` now defaults `add_each_output` to `false` like the blueprint does, so a generated always-combine spec with just `branches` and `output` no longer fails validation before `to_blueprint()`.+- **`PipeCondition` `continue` with nothing to pass through fails loud:** The `continue` outcome delivers the current main stuff by passing it through. When there is no main stuff to pass through (e.g. the condition is the entry pipe over named inputs only), the run now raises a clear, actionable `PipeRunError` at the pipe level — in live and dry runs alike — instead of completing without a main output and crashing downstream surfaces (graph tracer, delivery, telemetry).+- **Stale main stuff after an `add_each_output`-only parallel:** A pipeline ending in an `add_each_output`-only `PipeParallel` used to silently report the *previous* step's output (or even an input) as its main result, and such a parallel as the top-level pipe on the API path completed with no main stuff at all. The parallel now always stamps its combined output, in live and dry runs alike.- **Img2img support check keyed on declared inputs:** `ImgGenParamSupport.check_job_params` now takes the full model spec and checks image-to-image support against the model's declared `inputs` (the universal capability field) instead of requiring an `input_images` rule. The rule is an args-factory formatting concern that Gemini-family models — whose request path never reads it — legitimately don't carry, so the old rules-keyed check wrongly reported the whole family as not supporting img2img.- **Google img-gen worker no longer hardcodes 1K:** The native Google worker and the gateway path now send the requested `image_size` to the Gemini API (omitted entirely when `size` is unset); previously the worker always sent `"1K"` regardless of intent, making 2K/4K generation unreachable.- **Unknown boot orchestrator now fails loud:** Requesting a boot orchestrator no installed plugin provides — via `--orchestrator <name>` or `Pipelex.make(boot_orchestrator=...)` — now raises `UnknownBootOrchestratorError` at boot instead of silently falling back to in-process execution. A typo or a missing orchestrator plugin (e.g. `--orchestrator temporal` without the Temporal plugin installed) is reported rather than quietly running the job on the wrong runtime. The requested name is matched against registered plugin names, the same namespace the boot gate uses.diff --git a/TODOS.md b/TODOS.mdnew file mode 100644index 000000000..d7b14d2a4--- /dev/null+++ b/TODOS.md@@ -0,0 +1,190 @@+# TODOS — Required main stuff (PipeParallel always combines)++**Branch:** `feature/Required-main-stuff` · **Design:** `wip/required-main-stuff.md` (full) · `wip/required-main-stuff-brief.html` (brief)++**Scope:** implement the design **excluding §9 (companion track — `/execute` vs `/start`→delivery result-shape unification)**, which is designed and tackled separately. Everything else is in scope: always-combine `PipeParallel`, delete `combined_output`, new `native.Composite` concept, static branch-name validation, invariant enforcement across in-repo surfaces, then the gated cross-repo sweep.++## The rule being implemented++> A pipe run always delivers a main stuff. Consequently, a pipeline always delivers a main stuff.++Enforced at the one non-stamping leaf: `PipeParallel` always combines its branch outputs into its declared `output` concept and stamps that as main stuff. `combined_output` is deleted (breaking, no back-compat per house policy). `native.Composite` is the untyped combination vehicle.++## Working conventions for this track++- **TDD:** write the red tests first for each behavioral change, then implement to green.+- **Gates:** `make agent-check` after code changes; targeted pytest during development (`.venv/bin/pytest -x -q <path>`); full `make agent-test` before each checkpoint.+- **Commits:** one commit per phase, landed at its checkpoint, so each review sub-agent gets a clean SHA range.+- **Checkpoint protocol (MANDATORY STOP):** at each checkpoint the agent must stop forward progress and, in order:+ 1. **Verify** — run the phase's full gates (`make agent-check` + `make agent-test`) and confirm the phase's acceptance criteria below.+ 2. **Update this file** — check off done items, record decisions taken, deviations from the design, and current code state in the "Checkpoint log" section, so a brand-new session can cold-start from this file alone. Update `wip/required-main-stuff.md` status line too.+ 3. **Fan out review** — spawn a **Sonnet-5 sub-agent with NO inherited context** to run the `/code-review` skill on the phase's changes. Hand it *only* a pointer to the changes (the phase's commit SHA range, e.g. `git diff <base>..HEAD`, or the working-tree diff) — never the plan, the design doc, the rationale, or your own conclusions. Target: clean solid software, not over-engineering.+ 4. **Triage findings** — fix real defects before proceeding; findings that are design tradeoffs (not silent bugs) go to a deferred-items doc under `wip/required-main-stuff/`, not reflexively applied.++## Phase 1 — Runtime + language surface (this repo)++### 1a. Red tests first++- [x] Regression test for the **stale-stamp bug**: a `PipeSequence` ending in an `add_each`-only `PipeParallel` must deliver the combined composite as main stuff, not the previous step's output (today it silently delivers step N-1's stuff).+- [x] **Terminal-parallel delivers main stuff**: an `add_each`-only parallel as top-level `main_pipe` with memory from `make_from_pipeline_inputs` (the API path) completes with a stamped main stuff (today `main_stuff_name` is `None` and `PipeOutput.main_stuff` raises).+- [x] **Static validation accept/reject cases**: `output = "Composite"` accepted; structured concept with fields matching branch `result` names accepted (required fields ⊆ result names, result names ⊆ declared fields); native non-composite (`Text`, `Image`, …), `Dynamic`, `Anything`, and multiplicity suffixes (`Foo[]`, `Foo[N]`) rejected at bundle/library validation time.+- [x] **`CompositeContent` round-trips**: `smart_dump`, kajson transport round-trip (`dump_for_transport` → rehydrate), `rendered_markdown` / `rendered_html` with named sub-contents surfaced as top-level fields (no wrapper key).+- [x] **Dry-run parity**: dry run of a parallel stamps the same combined main stuff shape as live run.++### 1b. `native.Composite` concept++- [x] Add `COMPOSITE = "Composite"` to `NativeConceptCode` in `pipelex/core/concepts/native/concept_native.py`.+- [x] New `CompositeContent(StuffContent)` in `pipelex/core/stuffs/composite_content.py` — pydantic `extra="allow"` so branch names are top-level serialized fields; full content surface (`smart_dump`, kajson round-trip, `rendered_markdown`/`rendered_html`).+- [x] Register in the class registry + concept factory wherever native concepts are wired.++### 1c. Always-combine runtime++- [x] `pipelex/pipe_controllers/parallel/pipe_parallel.py`: unconditional combine in `_live_run_controller_pipe` **and** `_dry_run_controller_pipe` via a shared helper (an existing TODO already asks for this); combination concept taken from `self.output`; `set_new_main_stuff` always called; parallel-combine graph edges always registered.+- [x] Investigate `final_stuff_code` while in this file (the parallel currently clears it with a copy-pasted `PipeBatch` log line): confirm what it drives (graph/trace identity) — final wiring decision is a Phase 2 item, but capture findings now.++### 1d. Delete `combined_output` across the language surface++- [x] `pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py`: delete field + the one-of-two (`add_each_output`/`combined_output`) validator.+- [x] `pipelex/pipe_controllers/parallel/pipe_parallel_factory.py`: stop resolving the combined concept separately.+- [x] `pipelex/builder/pipe/pipe_parallel_spec.py`: delete field + validators; simplify `to_blueprint()`; update pretty-render.+- [x] `pipelex/builder/operations/pipe_ops.py` + `pipelex/cli/agent_cli/commands/pipe_cmd.py`: remove `combined_output` display/handling code.+- [x] `pipelex/core/bundles/pipelex_bundle_blueprint.py`: drop the `combined_output` concept-ref collection; **add the new static structure-compatibility check** (design §3).++### 1e. Fixtures, schema, docs++- [x] Migrate test fixtures and data: `tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel_1.mthds`, `tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_*.mthds`, `tests/unit/pipelex/pipe_controllers/parallel/{data.py,test_pipe_parallel_blueprint.py}`, `tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.py`, `tests/unit/pipelex/builder/operations/test_pipe_spec_to_toml.py`, `tests/unit/pipelex/graph/test_mermaidflow.py`, `tests/integration/pipelex/pipeline/test_bundle_validator.py`, `tests/integration/pipelex/cli/test_agent_validate_pipe_in_bundle.py`, `tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py` (and any others `grep -rln combined_output` surfaces).+- [x] Regenerate the MTHDS JSON Schema: `.venv/bin/pipelex-dev generate-mthds-schema`.+- [x] Rewrite `docs/building-methods/pipes/pipe-controllers/PipeParallel.md` to the new semantics — current reality only, no historical narrative.+- [x] Changelog entry under `[Unreleased]`, marked breaking (delete `combined_output`; `output` is the combination concept; new `native.Composite`).+- [x] Gates: `make agent-check` green, targeted parallel/builder/graph tests green, full `make agent-test` green.++### ⛔ CHECKPOINT 1 — MANDATORY STOP++Acceptance: parallel always combines (live + dry), `combined_output` fully gone from source and fixtures, `native.Composite` round-trips, static validation catches bad `output` declarations at `/validate` time, all gates green.++- [x] Verify acceptance criteria + full gates (`make agent-check` + full `make agent-test` green).+- [x] Commit Phase 1 — `757146324` on top of `1b26b3fae`.+- [x] Update this file (checkpoint log below) + design doc status for cold start.+- [x] Fan out context-free Sonnet-5 `/code-review` sub-agent on the Phase 1 diff (pointer only).+- [x] Triage findings: review found NO correctness bugs; duplicated native-concept match extracted to `NativeConceptCode.is_composite` (fixed); double-parse finding deferred → `wip/required-main-stuff/deferred-phase-1-review.md`.++## Phase 2 — Enforce the invariant across in-repo surfaces++Once every pipe stamps, "a completed run has a main stuff" is guaranteed; tighten the defensive branches from optional to direct.++- [x] Graph tracer output-spec branch in `pipelex/core/pipes/pipe_abstract.py` — remove the defensive branch and its apologetic comment ("main_stuff may not exist for pipes like PipeParallel…").+- [x] Delivery executor `pipelex/pipe_run/delivery_executor.py`: always write the `main_stuff.*` artifact files for completed runs (kills the "empty envelope" failure mode).+- [x] CLI run cores: `pipelex/cli/commands/run/_run_core.py` + `pipelex/cli/agent_cli/commands/run/_run_core.py` — drop the "no main output produced" branches for live runs.+- [x] OTel telemetry attribute extraction: `pipelex/system/telemetry/otel_factory.py`.+- [x] Wire models go non-optional (`main_stuff_name: str`): `pipelex/runtime_bridge/payloads.py`, `pipelex/runtime_bridge/serialization.py` (`resolve_main_stuff_root_key`), `pipelex/pipeline/pipeline_response.py` (`PipelexRunResultExecute`), `pipelex/cli/agent_cli/commands/run/_run_core_api.py`. Temporal payload history is not a concern — never shipped to prod.+- [x] Audit every remaining `get_optional_main_stuff` call site: `WorkingMemory.get_optional_main_stuff` itself **stays** (pre-run/empty memory legitimately has none); each post-run boundary call site is either tightened or justified in the checkpoint log.+- [x] Resolve the **`final_stuff_code`** open question (design §7) using the Phase 1c findings: presumably honor it on the combined stuff like operators do.+- [x] Gates: `make agent-check` + full `make agent-test` green.++### ⛔ CHECKPOINT 2 — MANDATORY STOP++Acceptance: no defensive main-stuff branch left for completed live runs; wire `main_stuff_name` non-optional; call-site audit documented; all gates green. Cross-repo work starts fresh from here — this checkpoint's TODOS update is the cold-start context for that session.++- [x] Verify acceptance criteria + full gates.+- [x] Commit Phase 2 — `3c3cf4570` on top of `9acbd7a86`.+- [x] Update this file (checkpoint log: final call-site audit results, `final_stuff_code` decision) + design doc status.+- [x] Fan out context-free Sonnet-5 `/code-review` sub-agent on the Phase 2 diff (pointer only).+- [x] Triage findings: ONE confirmed bug (reproduced by the reviewer) — `PipeCondition`'s `continue` outcome doesn't stamp a main stuff, so the tightened tracer epilogue crashed with a raw `WorkingMemoryStuffNotFoundError` whenever a condition continued with no pre-existing main stuff under active tracing. FIXED at the semantic site (see Checkpoint 2 log "Review triage"). All other tightened surfaces judged internally consistent by the review; no simplification findings.++## Phase 3 — Cross-repo sweep (GATED)++**Gate: do not start before Phases 1–2 are merged to `main` and a pipelex version is cut.** Each repo below is its own git repo — commit/PR per its own conventions.++- [ ] `mthds/` — spec: `mthds-format.md`, `pipes-controllers.md`, `validation-rules.md`, `namespace-resolution.md`, `cross-package-references.md`, `docs/mthds_schema.json` — remove `combined_output`, define always-combine + `native.Composite`.+- [ ] `vscode-pipelex/` — pinned `mthds_schema.json` in taplo-common; `plxt` lint/format accepts the new shape and rejects `combined_output`.+- [ ] `conformance/` — `tests/pipelex/test_validate_subcommands.py` fixture updates; keep spec ↔ test links green (`make check-spec-links`).+- [ ] `mthds-plugins/` — skills references: `mthds-reference.md`, `build-phases.md`, `recursive-cheat-sheet.md`, `mthds-run/SKILL.md` across plugin variants.+- [ ] `pipelex-app/` — `src/types/core/pipes/pipe_controllers/pipe-parallel.ts`.+- [ ] `mthds-ui/` — `src/graph/types.ts`: `combined_output_concept` exec-data key (kept as-is in Phase 1 for compat; decide rename here per design §7) + graph-spec data fixtures.+- [ ] `cocode/` — migrate `ai_instruction_update.mthds`, `swe_docs.mthds`.+- [ ] `pipelex-website/` — `docs/mthds-doc.md`.+- [ ] Demo/workshop repos — `illustration_generator/bundle.mthds` copies.+- [ ] SDKs (`pipelex-sdk-js/`, `mthds-python/`) + `pipelex-starter-python/` — tighten `main_stuff` handling for completed runs; check `PipeOutputAbstract` (mthds-python) for encoded main-stuff optionality (design §7). Full `find_main_content` cleanup also needs the companion track — do only the invariant-side tightening here.+- [ ] `pipelex-temporal/` (private) — adapt to the split orchestrator SPI: move the `FIRE_AND_FORGET` arm out of `match delivery` into `start(...)` returning `PipelexPipeDispatchAck`; `run(...)` keeps only the blocking arm (no `delivery` param, no `is_completed`). Same check for `pipelex-mistralai-workflows/` if it registers an orchestrator.+- [ ] `pipelex-api/` — `/execute` calls `orchestrator.run(...)` without `delivery`; `/start` calls `orchestrator.start(...)` and reads the ack's `workflow_id` (today it reads `run_output.workflow_id` off a degenerate `PipelexPipeRunOutput`).++### ⛔ CHECKPOINT 3 — MANDATORY STOP++Acceptance: every consumer repo aligned with the released pipelex version; conformance + spec-links green; no repo still references `combined_output`.++- [ ] Verify per-repo test suites / checks.+- [ ] Update this file; close out or fold residual items into follow-up trackers.+- [ ] Fan out context-free Sonnet-5 `/code-review` sub-agents per changed repo (pointer only — the repo's diff/SHA range).+- [ ] Triage findings; then close this track (companion track remains open, separate design).++## Open questions (from design §7 — resolve during implementation)++- [x] `final_stuff_code`: RESOLVED in Phase 2 — the parallel honors it on the combined stuff (see Checkpoint 2 log).+- [ ] Graph `execution_data` key `combined_output_concept`: keep name in Phase 1 for `mthds-ui` compat; rename decision in Phase 3 (`mthds-ui` row).+- [ ] `add_each_output` naming: recommendation is **keep** (pure churn otherwise); revisit only if MTHDS spec editors want it (Phase 3, `mthds/` row).+- [ ] `PipeOutputAbstract` (mthds-python): check for encoded optionality (Phase 3, SDKs row).++## Checkpoint log++*(Filled in at each checkpoint — decisions taken, deviations from design, current code state, audit results. This section is the cold-start context for a fresh session.)*++### Checkpoint 1 — reached (Phase 1 complete)++**Code state:** Phase 1 fully implemented on `feature/Required-main-stuff`; one commit carries the whole phase (see git log). `make agent-check` green; full `make agent-test` green.++**What landed:**++- `native.Composite`: `NativeConceptCode.COMPOSITE` + `CompositeContent(StuffContent)` (`pipelex/core/stuffs/composite_content.py`, pydantic `extra="allow"`), wired into `ConceptFactory.make_native_concept`, `NativeConceptCode.structure_class`, and `CoreRegistryModels.STUFF`. Custom `rendered_markdown`/`rendered_html` iterate the named components; a `components` property exposes `model_extra`.+- Always-combine runtime: `PipeParallel._run_branches_and_combine` is the shared live/dry helper (resolves the old TODO); combine concept comes from `self.output`; `set_new_main_stuff` and PARALLEL_COMBINE graph edges are unconditional. `combined_output` field deleted from runtime model, blueprint, factory, spec, pipe_ops/pipe_cmd display, and bundle-blueprint concept-ref collection.+- Static validation, two layers: (1) parse-time in `PipeParallelBlueprint.validate_output` — rejects multiplicity suffixes and native non-`Composite`/`Dynamic`/`Anything` outputs; mirrored on the runtime model in `PipeParallel.validate_output_static` for direct constructions; (2) library-time in `PipeParallel.validate_output_with_library` — structure-compatibility check (required fields ⊆ result names; result names ⊆ declared fields; `CompositeContent` subclasses skip). Both feed `/validate` (bundle parse + `Library.validate...` → `pipe.validate_with_libraries()`).++**Decisions / deviations from the plan:**++- The TODOS 1d line said to put the structure-compatibility check in `pipelex_bundle_blueprint.py`; it lives on `PipeParallel.validate_output_with_library` instead, because the concept's structure class (registered Python class, generated class, refines chain) is only resolvable against the loaded library — the bundle blueprint cannot see class-backed structures. Design §3 says "bundle/library validation time", which this satisfies.+- kajson round-trip needed a `__json_encode__` hook on `CompositeContent`: pydantic stores extras outside `__dict__`, so kajson's default `__dict__` fallback silently dropped every component. The hook hands kajson the live components so nested contents keep their class metadata (`# noqa: PLW3201` — kajson protocol name).+- A description-only concept as a parallel output is rejected by the field-compat check naturally (its generated structure class is TextContent-derived, requiring `text`), with a message suggesting `Composite` — test pins this.+- Graph `execution_data` key `combined_output_concept` kept (now always set = declared output concept ref) for `mthds-ui` compat, per design §7; rename decision deferred to Phase 3.+- `tests/e2e/.../cv_job_match.mthds` had two parallels with now-invalid outputs (`Page[]`, `Text`) — migrated to `Composite` (was not on the plan's fixture list).+- Mermaidflow unit test's IOSpec name `combined_output` renamed `combined_result` (terminology hygiene only — it was never the field).++**`final_stuff_code` findings (1c investigation, wiring decision = Phase 2):**++- Producer: only `PipeBatch` sets it — a pre-allocated per-branch stuff code (`branch_output_item_code`) so the batch's aggregated `ListContent` item identity matches the branch's final output stuff in the graph.+- Consumers: `PipeLLM` and `PipeStructure` pass it as `code=` to `StuffFactory.make_stuff` for their output stuff. Other operators do not honor it (pre-existing gap worth noting in Phase 2).+- `PipeSequence` threads it to the last step only (clears for earlier steps); `PipeBatch` and `PipeParallel` clear it on entry.+- Implication: a `PipeParallel` as a `PipeBatch` branch drops the requested code, so the batch item's graph identity doesn't match the parallel's combined stuff. Now that the parallel always stamps, Phase 2 should honor `final_stuff_code` on the combined stuff (pass `code=` through `combine_stuffs` → `make_stuff`). A `TODO(Phase 2)` marks the spot in `_live_run_controller_pipe`.++**New/updated tests:** `tests/unit/pipelex/core/stuffs/test_composite_content.py` (round-trips incl. transport dump→hydrate), `tests/unit/pipelex/pipe_controllers/parallel/` (blueprint accept/reject + data.py), `tests/integration/.../pipe_parallel/test_pipe_parallel_always_combines.py` (+ fixture `parallel_always_combine.mthds`; stale-stamp regression + terminal-parallel main stuff; dry parity via `pipe_run_mode` parametrization), `tests/integration/.../pipe_parallel/test_pipe_parallel_output_validation.py` (library-level accept/reject via `acquire_library`). e2e graph expectations updated: the add_each-only graph now expects `parallel_combine: 2` edges.++### Checkpoint 2 — reached (Phase 2 complete)++**Code state:** Phase 2 fully implemented on `feature/Required-main-stuff`; commit `3c3cf4570` on top of `9acbd7a86`. `make agent-check` green; full `make agent-test` green (all tests passing). TDD followed: red tests written and confirmed failing before each behavioral change.++**What landed (invariant enforcement, per-surface):**++- **Graph tracer epilogue** (`pipe_abstract.py` `_run_pipe_traced` success path): `get_main_stuff()` direct read; `output_spec` built unconditionally; the `main_stuff is not None` gate on `output_concept_data` removed. The apologetic comment is gone.+- **Delivery executor** (`generate_result_files`): typed path reads `get_main_stuff()` (raises `WorkingMemoryStuffNotFoundError` when absent); raw path raises `PipeJobError` when the raw dump lacks a main stuff. The `try_local_hydrate_stuff` → raw-render fallback is unchanged (it's about local class availability, not missing main stuff). A missing main stuff surfaces as `StorageDeliveryError` at the `_store_results` boundary — the same handling path as any storage failure, so the caller's failure-webhook behavior is unchanged.+- **CLI run cores:** human core pretty-prints/saves/CSV-exports via `main_stuff` directly (the "--save-csv: no main stuff produced" exit branch is deleted); agent core builds `main_stuff_json` + `compact_result` unconditionally; agent API-runner core raises `PipeExecutionError` when a completed response has no main stuff under the announced key (the `main_stuff_name`-extension *name* fallback to `MAIN_STUFF_NAME` is kept — that's protocol-extension handling, not invariant softening).+- **OTel** (`make_output_json`): direct read; the `"{}"` fallback is gone (called on the success path only).+- **Wire models:** `PipelexPipeRunOutput.main_stuff_name: str` (required) and `PipelexRunResultExecute.main_stuff_name: str`. `resolve_main_stuff_root_key` returns `str` and raises `PipeJobError` on a memory with no main stuff; `PipelexRunResultExecute.from_pipe_output` now uses it (previously `aliases.get(MAIN_STUFF_NAME, MAIN_STUFF_NAME)` — which could announce a key that wasn't actually in `root`).+- **`PipeOutput.optional_main_stuff` DELETED** (breaking, in changelog). Workspace grep: only external consumer is `pipelex-demo-vibe` examples → Phase 3 sweep list.++**`final_stuff_code` resolution (design §7):** the parallel now honors it on the combined stuff. Implementation: `_run_branches_and_combine` captures `pipe_run_params.final_stuff_code` and clears it *before* branch fan-out (branches deep-copy the params, so they must not inherit the parent's requested code), then passes it as the new `code=` param on `StuffFactory.combine_stuffs` → `make_stuff`. Live/dry parity for free (shared helper). Pinned by `test_pipe_parallel_final_stuff_code.py` (combined stuff carries the code; branch outputs don't). **Noted, not addressed (pre-existing gaps, out of scope):** only `PipeLLM`/`PipeStructure` honor `final_stuff_code` among operators; `PipeBatch` still clears it on entry rather than stamping its aggregate list with it.++**Call-site audit (final state):** `grep get_optional_main_stuff` across `pipelex/` + `tests/` → the definition in `working_memory.py` (kept: pre-run/empty memory legitimately has none; public API for SDK-side pre-run checks) plus ONE justified in-source caller added by the review triage: `PipeCondition`'s pre-delivery checks (a mid-run "is there something to pass through / did anything get delivered" probe — legitimately optional at that point, converted into a loud `PipeRunError` when absent). Every post-run boundary call site was tightened.++**Review triage (Checkpoint 2):** The context-free review found and reproduced one real bug: the Phase 1 premise "every pipe stamps" was FALSE for `PipeCondition`'s `continue` outcome (live path returned the memory untouched) and for the dry run of an all-special-outcomes condition — so the tightened tracer epilogue (which fires on EVERY pipe node) crashed with a raw `WorkingMemoryStuffNotFoundError` on any continue-with-no-prior-main-stuff under tracing. Resolution (design-consistent, not a defensive revert): `continue` IS pass-through delivery — the delivered main stuff is the one already in memory; with nothing to pass through the pipe now raises a clear actionable `PipeRunError` at the semantic site ("map the outcome to a pipe, or place this condition after a producing step"), in both live and dry paths. Pinned by `tests/integration/.../pipe_condition/test_pipe_condition_continue_delivery.py` (pass-through identity + both fail-loud cases; run params constructed directly so keyless forced-DRY coercion can't swap the exercised path). Suite gap noted by the reviewer (nothing drove continue-with-no-main-stuff under tracing) is now covered. The reviewer confirmed the parallel `final_stuff_code` capture/clear mirrors `PipeBatch`'s existing pattern, and `combine_stuffs(code=)` has no empty-string/double-stamp edge. Related known-shaky area: `test_pipe_condition_continue_output_type.py` stays xfail (separate pre-existing bug, batch-over-continue aggregation semantics — untouched by this fix since batch branches always have a main stuff stamped).++**Test changes:** new `tests/integration/.../pipe_parallel/test_pipe_parallel_final_stuff_code.py`; `test_output_serialization.py` extended (aliased/direct root-key resolution + missing-main-stuff raises); `test_input_models.py` pins `main_stuff_name` required; `test_delivery_executor.py` mocks now carry a real main stuff + two new missing-main-stuff raise tests + `main_stuff.*` files asserted stored; `test_run_core_execution.py` obsolete `test_save_csv_no_main_stuff_exits` deleted; `test_run_pipe_tracer_metadata.py` + `test_run_cost_report.py` mock outputs made invariant-compliant; `test_mock_usage_direct.py` tightened.++**Cross-repo consequences flagged for Phase 3:** `main_stuff_name` is now required on the `/execute` response extension and `PipelexPipeRunOutput` — the closed `pipelex-transport` library and any consumer parsing these payloads must be checked; `pipelex-demo-vibe` uses the deleted `optional_main_stuff`.++### Between checkpoints — PR opened toward the Phase 3 gate++- Merged `origin/dev` into the branch (`fc96c1d96`; only conflict = both sides' `[Unreleased]` changelog sections, combined into unified Added/Changed/Fixed). Post-merge `make agent-check` green; full `make agent-test` re-run on the merged tree.+- PR **#1014** opened against `dev`. Phase 3 stays gated until the PR is merged and a pipelex version is cut.+- **Review round 1 (codex bot, CONFIRMED REAL):** requiring `main_stuff_name` on `PipelexPipeRunOutput` broke the fire-and-forget arm — `pipelex-temporal`'s `FIRE_AND_FORGET` branch constructs the payload with `main_stuff_name=None, is_completed=False` (its only not-completed constructor anywhere). Resolution chosen by the user (NOT the "make it optional again" convenient fix, which would re-spread null-guards): **split the delivery axis into the type system.** `OrchestratorProtocol` now has `run(...) -> PipelexPipeRunOutput` (blocking, completed, `main_stuff_name` stays required) and `start(...) -> PipelexPipeDispatchAck` (new ids-only ack model: `pipeline_run_id` + `workflow_id`); the `delivery` param is gone from the SPI (`DeliveryMode` stays as wire-input vocab picking the method); `PipelexPipeRunOutput.is_completed` DELETED — nothing anywhere read it. DIRECT implements `start` by raising `PipelexBridgeDispatchError` (unreachable behind the `supports_fire_and_forget` gate). Docs (`orchestrator-plugins.md`) + changelog updated; new Phase 3 rows added for `pipelex-temporal` and `pipelex-api`.+- **Review round 2 (`4ee873fe4`, pushed):** all remaining bot threads (codex ×2, greptile ×1, cubic ×5) confirmed real, fixed, replied and resolved in-thread — 0 unresolved threads on the PR. (1) `PipeParallelSpec.add_each_output` defaults to `false` like the blueprint (always-combine specs can omit it). (2) `validate_output_with_library` now also checks branch output **types** against structured field content classes — compatibility is conceptual (branch structure class IS the field class or a subclass; refinement passes since refining concepts generate inheriting classes — user's explicit call), matching exactly what pydantic accepts in `combine_stuffs`; conservative skips: non-content-class annotations (primitive fields from `.mthds` structures), list-producing branches. Gotcha pinned in-code: the `type: Literal[...]` field shadows the builtin in the class body, so the method signature needs `builtins.type[StuffContent]`. (3) **Greptile P1 (real):** `dump_for_transport()` lost Composite component types (plain dicts after hydration; the existing round-trip test only compared `smart_dump()`, hence the miss) — components are now marker-stamped (`__pipelex_class__`/`__pipelex_module__`, ListContent components as plain lists of stamped item dicts, mirroring the top-level convention) and `hydrate_content` rebuilds typed components; regression tests assert component *types* incl. a ListContent component. (4) HTML value rendering deduped into shared `pipelex/core/stuffs/html_rendering.render_value_html()` used by `StructuredContent` + `CompositeContent`. (5) wip §4 reworded implemented-not-proposal. Changelog + `PipeParallel.md` doc updated; `make agent-check` + full `make agent-test` green.+- **Review round 3 (`/review` sweep, own critical pass + testing/maintainability/adversarial sub-agents):** scope CLEAN, no bug bots missed on the shipped path. **Applied in-PR:** `DirectOrchestrator.start()` raise-path test (`tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py` — `supports_fire_and_forget is False` + `start()` raises `PipelexBridgeDispatchError`); gates green. **Deferred to `wip/required-main-stuff/deferred-review-round-3.md`** (per user, keep PR frozen — none reachable in prod): **F1** nested-`CompositeContent` transport is decode-asymmetric (inner composite components decay to raw dicts and leak `__pipelex_*` markers on the Temporal round-trip — real silent corruption but Temporal not shipped; fix before hosted-Temporal rollout: recurse in `_hydrate_list_item` for CompositeContent subclasses + nested test); **F2** `ConceptValueError` (bare `ValueError`, not `PipelexError`) from `get_structure_class()` in `validate_output_with_library` escapes the validate sweep's catches → whole-bundle abort vs per-pipe FAILURE (low reach; fix: re-raise as `PipeValidationError`); **M1** `rendered_html` table-wrapper + empty-literal still copy-pasted across Composite/Structured (extract `render_named_values_table`). Minor noted: `combined_output_concept` key rename stays a Phase-3 mthds-ui-coordinated item; unguarded `get_required_pipe` consistency nit; `from_pipe_output` non-COMPLETED latent; duplicate-result-name stays a runtime guard.++### Checkpoint 3 — not reacheddiff --git a/docs/building-methods/concepts/native-concepts.md b/docs/building-methods/concepts/native-concepts.mdindex f5e51e074..87ebb5209 100644--- a/docs/building-methods/concepts/native-concepts.md+++ b/docs/building-methods/concepts/native-concepts.md@@ -35,6 +35,7 @@ Here are all the native concepts you can use out of the box:| `JSON` | A JSON object | `JSONContent` || `Html` | HTML content with inner HTML and CSS class | `HtmlContent` || `SearchResult` | A web search result with answer and sources | `SearchResultContent` |+| `Composite` | A named composition of contents | `CompositeContent` || `Anything` | Any type of content | *No specific implementation* |## Native Concept Structures@@ -199,6 +200,19 @@ class SearchResultContent(StuffContent):**Use for:** Web search results, research findings, information retrieval with citations.+### CompositeContent++An untyped named composition of contents. Produced by `PipeParallel` when its output is `Composite`:++```python+class CompositeContent(StuffContent):+ model_config = ConfigDict(extra="allow")+```++Each named sub-content is a top-level field of the composite — there is no wrapper key, so the serialized shape matches what a bespoke structured concept with the same field names would produce.++**Use for:** Combining parallel branch results without declaring a bespoke concept.+### Anything!!! note "Special Concepts"diff --git a/docs/building-methods/pipes/pipe-controllers/PipeParallel.md b/docs/building-methods/pipes/pipe-controllers/PipeParallel.mdindex 9bd2a7dc6..3cbd45cb9 100644--- a/docs/building-methods/pipes/pipe-controllers/PipeParallel.md+++ b/docs/building-methods/pipes/pipe-controllers/PipeParallel.md@@ -1,10 +1,10 @@----description: "Execute multiple pipes concurrently with PipeParallel. Speed up independent tasks while keeping each branch isolated in its own memory."+description: "Execute multiple pipes concurrently with PipeParallel. Speed up independent tasks while keeping each branch isolated in its own memory, then deliver their combined results as one output."---# PipeParallel-The `PipeParallel` controller executes multiple pipes simultaneously. This is highly effective for running pipes concurrently in isolated branches.+The `PipeParallel` controller executes multiple pipes simultaneously, then combines their results into a single output. This is highly effective for running independent tasks concurrently in isolated branches.## How it works@@ -12,11 +12,17 @@ The `PipeParallel` controller executes multiple pipes simultaneously. This is hi1. **Isolation**: Before execution, `PipeParallel` creates a deep copy of the current `WorkingMemory` for each branch. This means every parallel pipe starts with the exact same state, but they run in complete isolation—a change in one branch will not affect another.2. **Concurrent Execution**: All specified pipes are executed at the same time using `asyncio.gather`.-3. **Output Handling**: After all parallel tasks have finished, their results are collected and added back to the main working memory. You can control how this happens with two parameters:- - `add_each_output`: If `true`, the individual result of each branch is added to the working memory under the name specified in its `result` key.- - `combined_output`: If you provide an output concept here, the results of all branches are bundled together into a single structured object. The field names of this object correspond to the `result` names of the branches.+3. **Combination**: After all branches have finished, their results are always combined into a single object of the pipe's declared `output` concept, keyed by each branch's `result` name. That combined object is the pipe's main output.+4. **Optional branch exposure**: If `add_each_output` is `true`, each branch's individual result is also added to the working memory under the name specified in its `result` key, so downstream pipes can consume branches directly.-You must use `add_each_output`, `combined_output`, or both.+## The output concept++The declared `output` of a `PipeParallel` is the concept the branch results are combined into. It must be one of:++- **`Composite`** — the native untyped composition: the combined object simply holds each branch result under its `result` name. Use this when you don't want to declare a bespoke concept.+- **A structured concept** whose fields correspond to the branch `result` names: every required field of the structure must match a branch `result` name, and every `result` name must match a declared field. Branch output *types* are checked too: when a field is typed with a content class, the branch feeding it must produce that concept or one refining it (e.g. a concept that refines `Text` fits a `TextContent` field).++Anything else is rejected at validation time: scalar native concepts (`Text`, `Image`, ...), `Dynamic`, `Anything`, and multiplicity suffixes (`Foo[]`, `Foo[3]`) are all invalid — a parallel combination is a named composite, never a scalar or a list (a list aggregation is [`PipeBatch`](PipeBatch.md)'s shape).## Configuration@@ -29,10 +35,9 @@ You must use `add_each_output`, `combined_output`, or both.| `type` | string | The type of the pipe: `PipeParallel` | Yes || `description` | string | A description of the parallel operation. | Yes || `inputs` | dictionary | The input concept(s) for the parallel operation, as a dictionary mapping input names to concept codes. | Yes |-| `output` | string | The output concept produced by the parallel operation. | Yes |+| `output` | string | The concept the branch results are combined into: `Composite` or a structured concept whose fields match the branch `result` names. | Yes || `branches` | array of tables| An array defining the pipes to run in parallel. Each table is a sub-pipe definition. | Yes |-| `add_each_output` | boolean | If `true`, adds the output of each parallel pipe to the working memory individually. Defaults to `true`. | No |-| `combined_output` | string | The name of a concept to use for a single, combined output object. The structure of this concept must have fields that match the `result` names from the `branches` array. | No |+| `add_each_output` | boolean | If `true`, also adds the output of each parallel pipe to the working memory individually under its `result` name. Defaults to `false`. | No |### Parallel Step Configuration@@ -49,7 +54,11 @@ Imagine you have a product description and you want to extract the product featu```toml[concept.ProductAnalysis]-structure = "ProductAnalysis" # A Pydantic model with 'features' and 'sentiment' fields+description = "Combined product analysis"++[concept.ProductAnalysis.structure]+features = { type = "text", description = "Product features", required = true }+sentiment = { type = "text", description = "Product sentiment", required = true }[pipe.extract_features]type = "PipeLLM"@@ -68,9 +77,8 @@ output = "ProductSentiment"type = "PipeParallel"description = "Extract features and sentiment at the same time"inputs = { description = "ProductDescription" }-output = "ProductAnalysis" # This name is for the combined output+output = "ProductAnalysis" # The concept the branch results are combined intoadd_each_output = true-combined_output = "ProductAnalysis"branches = [{ pipe = "extract_features", result = "features" },{ pipe = "analyze_sentiment", result = "sentiment" },@@ -78,13 +86,17 @@ branches = [```How this works:+1. The `analyze_product_in_parallel` pipe starts. It receives a `ProductDescription`.2. Two parallel branches are created, both with access to the `ProductDescription`.3. The `extract_features` pipe runs in one branch, and the `analyze_sentiment` pipe runs in the other, simultaneously.4. After both complete, `PipeParallel` collects the results. The output from `extract_features` is named `features`, and the output from `analyze_sentiment` is named `sentiment`.-5. Because `combined_output` is set to `ProductAnalysis`, a new structured object of type `ProductAnalysis` is created. This object is populated with the results, like `{"features": ..., "sentiment": ...}`.-6. This single `ProductAnalysis` object becomes the final output of the `analyze_product_in_parallel` pipe.+5. A new structured object of type `ProductAnalysis` is created and populated with the results, like `{"features": ..., "sentiment": ...}`. This object becomes the main output of the `analyze_product_in_parallel` pipe.+6. Because `add_each_output` is `true`, `features` and `sentiment` are also available individually in the working memory for downstream pipes.++If you don't want to declare a `ProductAnalysis` concept, set `output = "Composite"` instead: the combined object then holds the same `{"features": ..., "sentiment": ...}` shape as an untyped composition.## Related Documentation-- [Pipeline Orchestration](../../../features/pipeline-orchestration.md) - Overview of pipeline orchestration capabilities\ No newline at end of file+- [Pipeline Orchestration](../../../features/pipeline-orchestration.md) - Overview of pipeline orchestration capabilities+- [Native Concepts](../../concepts/native-concepts.md) - Includes the `Composite` conceptdiff --git a/docs/under-the-hood/orchestrator-plugins.md b/docs/under-the-hood/orchestrator-plugins.mdindex e7124418a..248c2acf5 100644--- a/docs/under-the-hood/orchestrator-plugins.md+++ b/docs/under-the-hood/orchestrator-plugins.md@@ -8,7 +8,7 @@ description: "How Pipelex dispatches a pipe run by orchestration mode through thA pipe a host runtime invokes through the runtime bridge runs along **two orthogonal axes**:- **`orchestration_mode`** — *which* orchestrator runs the pipe. An **open string token**, not a closed enum: core owns only `"direct"` (in-process); every other token is contributed by the plugin that owns its orchestrator — [`"temporal"`](https://pipelex.com/products#temporal) (durable, on a Temporal worker fleet) by `pipelex-temporal`, [`"mistral-workflows"`](https://pipelex.com/products#mistral-workflows) (decomposed into Mistral Workflows primitives) by `pipelex-mistralai-workflows`. Neither is built into the open-source `pipelex` core: both ship as external, closed-source host-runtime backends, distributed privately rather than on PyPI as part of Pipelex's [workflow-orchestration offer](https://pipelex.com/products#durable-execution).-- **`delivery`** — *whether the caller waits*. A **closed** core `DeliveryMode` enum (`BLOCKING` / `FIRE_AND_FORGET`), set by the endpoint and passed as a parameter to `run`, never received from a caller. An orchestrator honors it per its nature; `supports_fire_and_forget` advertises whether it can do genuine async.+- **`delivery`** — *whether the caller waits*. A **closed** core `DeliveryMode` enum (`BLOCKING` / `FIRE_AND_FORGET`) on the wire input, set by the endpoint, never received from a caller. On the SPI it is expressed as *which method* the endpoint calls — `run` (blocking) or `start` (fire-and-forget) — so each return type is truthful on its own; `supports_fire_and_forget` advertises whether an orchestrator can do genuine async.An **orchestrator** is what knows how to run a pipe under one token. Core names no orchestrator by import or by string. The bridge resolves the orchestrator for the requested token from a registry (keyed by the token `str`) and calls its `run` — `"direct"` is contributed by a core plugin, `"temporal"` by the Temporal plugin, `"mistral-workflows"` by the external `pipelex-mistralai-workflows` plugin. A lookup miss raises a generic `MissingOrchestratorError` that names no orchestrator. This page documents that seam, the **Orchestrator SPI** a host-runtime plugin compiles against, and how the Temporal plugin is wired.@@ -21,7 +21,7 @@ An **orchestrator** is what knows how to run a pipe under one token. Core names→ build the PipeJob (boundary decode + library scope + trace_context)→ orchestrator = get_orchestrator_registry().get_optional(mode=orchestration_mode)→ if orchestrator is None: raise MissingOrchestratorError(mode) # generic, names no orchestrator- → return await orchestrator.run(pipe_job=..., delivery_assignment=..., delivery=...)+ → return await orchestrator.run(pipe_job=..., delivery_assignment=...) # or orchestrator.start(...) for FIRE_AND_FORGET```The registry is built once at boot from whatever the discovered plugins contributed (`build_registrar` → `OrchestratorRegistry` on the hub). There is no `match orchestration_mode:` anywhere in the bridge — the token set is open, so validation is the registry lookup itself; adding a mode's behavior means registering an orchestrator for its token, nothing in core changes.@@ -39,10 +39,12 @@ An orchestrator satisfies `OrchestratorProtocol` (`pipelex/plugins/orchestrator_class OrchestratorProtocol(Protocol):supports_fire_and_forget: bool- async def run(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None, delivery: DeliveryMode) -> PipelexPipeRunOutput: ...+ async def run(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput: ...++ async def start(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeDispatchAck: ...```-`run` honors the endpoint-chosen `delivery` per the orchestrator's nature (in-process always blocks; a distributed orchestrator awaits completion for `BLOCKING` and returns a workflow id for `FIRE_AND_FORGET`). `supports_fire_and_forget` is the capability a runner reads *before* dispatch — `/start` rejects honestly (4xx) when the resolved mode cannot do genuine async, instead of silently running blocking and acking.+The delivery axis is split into two methods so each return type is truthful on its own. `run` is the BLOCKING arm: it awaits completion and returns the completed-run `PipelexPipeRunOutput` — which always carries a main stuff (`main_stuff_name` is required, no "not finished yet" escape hatch). `start` is the FIRE_AND_FORGET arm: it genuinely enqueues the job and returns a `PipelexPipeDispatchAck` (`pipeline_run_id` + `workflow_id`, nothing more — nothing has run yet). `supports_fire_and_forget` is the capability a runner reads *before* dispatch — `/start` rejects honestly (4xx) when the resolved mode cannot do genuine async, instead of silently running blocking and acking; an orchestrator that cannot (like core's DIRECT) implements `start` by raising, unreachable behind that gate.A plugin contributes one per token it serves by calling the registrar menu in its `register`, passing the token as a raw string (no enum, no cast):@@ -135,7 +137,7 @@ What an out-of-tree orchestrator imports *is* a contract. The SPI is a documente| Area | Modules / symbols ||---|---|-| Boundary serialization + boot | `pipelex.runtime_bridge.serialization` (`serialize_pipe_output`, `serialize_completed_output`, `PIPE_DISPATCH_ERRORS`), `pipelex.runtime_bridge.payloads` (`PipelexPipeRunInput`, `PipelexPipeRunOutput`), `pipelex.runtime_bridge.bootstrap` (`ensure_pipelex_booted`) |+| Boundary serialization + boot | `pipelex.runtime_bridge.serialization` (`serialize_pipe_output`, `serialize_completed_output`, `PIPE_DISPATCH_ERRORS`), `pipelex.runtime_bridge.payloads` (`PipelexPipeRunInput`, `PipelexPipeRunOutput`, `PipelexPipeDispatchAck`), `pipelex.runtime_bridge.bootstrap` (`ensure_pipelex_booted`) || Mode + delivery + errors | `pipelex.runtime_bridge.orchestration_mode` (`OrchestrationMode`, `DIRECT_ORCHESTRATION_MODE`), `pipelex.runtime_bridge.delivery_mode` (`DeliveryMode`), `pipelex.runtime_bridge.exceptions` (`MissingOrchestratorError`, `PipelexBridgeDispatchError`) || Working-memory hydration | `pipelex.runtime_bridge.primitives.hydration` (re-hydrate `working_memory_raw` → typed `WorkingMemory`; stayed open because it is host-agnostic — used by core delivery and the open `pipelex-api` runner, and re-used across the boundary by `pipelex-transport`) || Plugin contract | `pipelex.plugins.contract` (`PipelexPlugin`, `PLUGIN_API_VERSION`), `pipelex.plugins.registrar` (`PluginRegistrar` menu: `add_orchestrator`, `add_http_error_mapper`, `claim_*`, `add_teardown`; read accessor: `get_http_error_mappers`), `pipelex.plugins.orchestrator_registry` (`OrchestratorProtocol`) |@@ -153,10 +155,10 @@ What an out-of-tree orchestrator imports *is* a contract. The SPI is a documente`pipelex_temporal/temporal_plugin.py` (in the external `pipelex-temporal` distribution) is the reference orchestrator plugin. Its `register`:-- **always** (regardless of the boot gate): contributes a single `TemporalOrchestrator` registered once under the `"temporal"` token (import-light; `temporalio` is pulled lazily inside `run`). It advertises `supports_fire_and_forget = True`, and its `run` branches on the endpoint-chosen `delivery`: `BLOCKING` awaits completion and reports `make_workflow_id(...)`; `FIRE_AND_FORGET` calls `.start(...)` and returns an `is_completed=False` output carrying the workflow id;+- **always** (regardless of the boot gate): contributes a single `TemporalOrchestrator` registered once under the `"temporal"` token (import-light; `temporalio` is pulled lazily inside `run`/`start`). It advertises `supports_fire_and_forget = True`; its `run` awaits completion and reports `make_workflow_id(...)`, its `start` enqueues the workflow and returns a `PipelexPipeDispatchAck` carrying the workflow id;- **only when `plugins.boot_orchestrator == "temporal"`**: claims the content-generator / task-manager / pipe-router / pipe-run hub slots with thunks and registers the teardown callback — booting this process as a Temporal-default runtime.-The orchestrator itself (`pipelex_temporal/temporal_orchestrators.py`) carries both delivery bodies behind one exhaustive `match delivery`, keeping the `WorkflowExecutionError` catch and the `make_workflow_id` recompute in the blocking arm. It serializes its `PipeOutput` through `pipelex.runtime_bridge.serialization`, shared with the core DIRECT orchestrator so the boundary shape cannot drift.+The orchestrator itself (`pipelex_temporal/temporal_orchestrators.py`) keeps the `WorkflowExecutionError` catch and the `make_workflow_id` recompute in the blocking `run` arm. It serializes its `PipeOutput` through `pipelex.runtime_bridge.serialization`, shared with the core DIRECT orchestrator so the boundary shape cannot drift.The Temporal plugin is **external** — it ships as the `pipelex-temporal` distribution and is discovered through a `pipelex.plugins` entry point in that dist's `pyproject.toml`, not through `BUILTIN_PLUGINS`. Core's `BUILTIN_PLUGINS` (`pipelex/plugins/builtins.py`) holds only the always-shipped inference and `direct` plugins and explicitly excludes Temporal; installing `pipelex-temporal` is all it takes to make the `"temporal"` orchestrator available — zero config, no core import of `temporalio`. Its operational `worker` / `setup-namespace` commands ship as the standalone `pipelex-temporal` console script, so they travel with that dist.diff --git a/pipelex/builder/operations/pipe_ops.py b/pipelex/builder/operations/pipe_ops.pyindex f87801047..0f30b1f73 100644--- a/pipelex/builder/operations/pipe_ops.py+++ b/pipelex/builder/operations/pipe_ops.py@@ -245,8 +245,6 @@ def add_type_specific_fields(pipe_spec: PipeSpec, *, pipe_table: Table) -> None:elif isinstance(pipe_spec, PipeParallelSpec):pipe_table.add("add_each_output", pipe_spec.add_each_output)- if pipe_spec.combined_output:- pipe_table.add("combined_output", pipe_spec.combined_output)branches_array = tomlkit.array()for branch in pipe_spec.branches:branch_inline = tomlkit.inline_table()diff --git a/pipelex/builder/pipe/pipe_parallel_spec.py b/pipelex/builder/pipe/pipe_parallel_spec.pyindex 90084c46f..8cbbc2c79 100644--- a/pipelex/builder/pipe/pipe_parallel_spec.py+++ b/pipelex/builder/pipe/pipe_parallel_spec.py@@ -1,6 +1,6 @@from typing import Literal-from pydantic import Field, field_validator, model_validator+from pydantic import Fieldfrom rich.console import Groupfrom rich.table import Tablefrom rich.text import Text@@ -8,54 +8,32 @@ from typing_extensions import overridefrom pipelex.builder.pipe.pipe_spec import PipeSpecfrom pipelex.builder.pipe.sub_pipe_spec import SubPipeSpec-from pipelex.cogt.content_generation.dry_run_factory import MockFormat-from pipelex.core.concepts.validation import validate_concept_ref_or_codefrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprintfrom pipelex.tools.misc.pretty import PrettyPrintable-from pipelex.types import Selfclass PipeParallelSpec(PipeSpec):"""Spec for parallel pipe execution in the Pipelex framework.PipeParallel enables concurrent execution of multiple pipes, improving performance- for independent operations. All parallel pipes receive the same input context- and their outputs can be combined or kept separate.+ for independent operations. All parallel pipes receive the same input context.+ Their outputs are always combined into the pipe's declared output concept — either+ `Composite` (untyped named composition) or a structured concept whose fields+ correspond to the branch result names.Validation Rules:1. Branches list must not be empty.2. Each branch must be a valid SubPipeSpec.- 3. combined_output, when specified, must be a valid ConceptCode in PascalCase.- 4. Pipe codes in branches must reference existing pipes (snake_case).+ 3. Pipe codes in branches must reference existing pipes (snake_case).+ 4. The output must be `Composite` or a structured concept compatible with the+ branch result names (validated at the blueprint/library level)."""type: Literal["PipeParallel"] = "PipeParallel"pipe_category: Literal["PipeController"] = "PipeController"branches: list[SubPipeSpec] = Field(description="List of SubPipeSpec instances to execute concurrently.")- add_each_output: bool = Field(description="Whether to include individual pipe outputs in the combined result.")- combined_output: str | None = Field(- default=None,- description="Optional ConceptCode in PascalCase for the combined output structure.",- json_schema_extra={"mock_format": MockFormat.PASCAL_CASE},- )-- @field_validator("combined_output", mode="before")- @classmethod- def validate_combined_output(cls, combined_output: str) -> str:- if combined_output:- validate_concept_ref_or_code(concept_ref_or_code=combined_output)- return combined_output-- @model_validator(mode="after")- def validate_output_options(self) -> Self:- if not self.add_each_output and not self.combined_output:- msg = (- "PipeParallel requires either add_each_output to be True or combined_output to be set, "- "or both, otherwise the pipe won't output anything"- )- raise ValueError(msg)- return self+ add_each_output: bool = Field(default=False, description="Whether to also expose each branch output by its result name in memory.")@overridedef rendered_pretty(self, *, title: str | None = None, depth: int = 0) -> PrettyPrintable:@@ -69,8 +47,6 @@ class PipeParallelSpec(PipeSpec):# Add parallel configurationparallel_group.renderables.append(Text()) # Blank lineparallel_group.renderables.append(Text.from_markup(f"Add Each Output: [bold yellow]{self.add_each_output}[/bold yellow]"))- if self.combined_output:- parallel_group.renderables.append(Text.from_markup(f"Combined Output: [bold green]{self.combined_output}[/bold green]"))# Add parallel branches as a tableparallel_group.renderables.append(Text()) # Blank line@@ -107,5 +83,4 @@ class PipeParallelSpec(PipeSpec):pipe_category=self.pipe_category,branches=core_branches,add_each_output=self.add_each_output,- combined_output=self.combined_output,)diff --git a/pipelex/cli/agent_cli/commands/pipe_cmd.py b/pipelex/cli/agent_cli/commands/pipe_cmd.pyindex 4fe131965..6d8fa3d75 100644--- a/pipelex/cli/agent_cli/commands/pipe_cmd.py+++ b/pipelex/cli/agent_cli/commands/pipe_cmd.py@@ -128,8 +128,6 @@ def _add_type_specific_fields(pipe_spec: PipeSpec, *, pipe_table: tomlkit.TOMLDoelif isinstance(pipe_spec, PipeParallelSpec):pipe_table.add("add_each_output", pipe_spec.add_each_output)- if pipe_spec.combined_output:- pipe_table.add("combined_output", pipe_spec.combined_output)branches_array = tomlkit.array()for branch in pipe_spec.branches:branch_inline = tomlkit.inline_table()diff --git a/pipelex/cli/agent_cli/commands/run/_run_core.py b/pipelex/cli/agent_cli/commands/run/_run_core.pyindex 68963c095..df4ca9182 100644--- a/pipelex/cli/agent_cli/commands/run/_run_core.py+++ b/pipelex/cli/agent_cli/commands/run/_run_core.py@@ -73,18 +73,13 @@ async def run_pipeline_core()pipe_output = response.pipe_output- main_stuff = pipe_output.working_memory.get_optional_main_stuff()- main_stuff_json: dict[str, Any] = {}- if main_stuff:- main_stuff_json = {- "json": await main_stuff.content.rendered_json_async(),- "markdown": await main_stuff.content.rendered_markdown_async(),- "html": await main_stuff.content.rendered_html_async(),- }-- compact_result: dict[str, Any] | None = None- if main_stuff:- compact_result = json.loads(main_stuff_json["json"])+ main_stuff = pipe_output.working_memory.get_main_stuff()+ main_stuff_json: dict[str, Any] = {+ "json": await main_stuff.content.rendered_json_async(),+ "markdown": await main_stuff.content.rendered_markdown_async(),+ "html": await main_stuff.content.rendered_html_async(),+ }+ compact_result: dict[str, Any] = json.loads(main_stuff_json["json"])result = build_run_output(with_memory=with_memory,diff --git a/pipelex/cli/agent_cli/commands/run/_run_core_api.py b/pipelex/cli/agent_cli/commands/run/_run_core_api.pyindex dc607800a..24db2f790 100644--- a/pipelex/cli/agent_cli/commands/run/_run_core_api.py+++ b/pipelex/cli/agent_cli/commands/run/_run_core_api.py@@ -12,6 +12,7 @@ from mthds.runners.api.client import MthdsAPIClientfrom mthds.runners.api.models import MAIN_STUFF_NAMEfrom pipelex.cli.agent_cli.commands.run._output_helpers import build_run_output+from pipelex.pipeline.exceptions import PipeExecutionErrorasync def run_pipeline_core_api(@@ -50,22 +51,25 @@ async def run_pipeline_core_api(# extension-open RunResult — they ride model_extra, never named by the SDK.extensions: dict[str, Any] = response.model_extra or {}- # Extract main stuff content from the working memory- main_stuff_json: dict[str, Any] = {}+ # Extract the main stuff content from the working memory. A completed run always delivers+ # a main stuff — a response without one under the announced key is a runner contract violation.raw_main_stuff_name = extensions.get("main_stuff_name")main_stuff_name = raw_main_stuff_name if isinstance(raw_main_stuff_name, str) else MAIN_STUFF_NAMEmain_stuff = pipe_output.working_memory.root.get(main_stuff_name)- if main_stuff is not None:- main_stuff_json = {- "json": main_stuff.content,- "markdown": "",- "html": "",- }+ if main_stuff is None:+ msg = (+ f"Completed run '{response.pipeline_run_id}' response has no main stuff under key '{main_stuff_name}' — "+ "a completed run always delivers a main stuff."+ )+ raise PipeExecutionError(msg)- compact_result: dict[str, Any] | None = None- if main_stuff is not None:- content: Any = main_stuff.content- compact_result = cast("dict[str, Any]", content) if isinstance(content, dict) else {"result": content}+ main_stuff_json: dict[str, Any] = {+ "json": main_stuff.content,+ "markdown": "",+ "html": "",+ }+ content: Any = main_stuff.content+ compact_result: dict[str, Any] = cast("dict[str, Any]", content) if isinstance(content, dict) else {"result": content}return build_run_output(with_memory=with_memory,diff --git a/pipelex/cli/commands/run/_run_core.py b/pipelex/cli/commands/run/_run_core.pyindex a3c0a7b0c..0862a43da 100644--- a/pipelex/cli/commands/run/_run_core.py+++ b/pipelex/cli/commands/run/_run_core.py@@ -199,9 +199,9 @@ async def _execute_run(raise typer.Exit(1) from exc# Pretty print main_stuff unless disabled or in dry run mode- if not no_pretty_print and not dry_run and (main_stuff := pipe_output.optional_main_stuff):+ if not no_pretty_print and not dry_run:title = f"Final output of pipe [red]{pipe_code}[/red]"- main_stuff.pretty_print_stuff(title=title)+ pipe_output.main_stuff.pretty_print_stuff(title=title)# Determine if we need an output directoryoutput_path: Path | None = None@@ -235,31 +235,30 @@ async def _execute_run(# Save main_stuff files if enabledsaved_main_stuff_formats: list[str] = []if save_main_stuff and output_path:- main_stuff = pipe_output.working_memory.get_optional_main_stuff()- if main_stuff:- main_stuff_json = await main_stuff.content.rendered_json_async()- main_stuff_json_path = output_path / "main_stuff.json"- main_stuff_json_path.write_text(main_stuff_json, encoding="utf-8")- log.verbose(f"Main stuff JSON saved to: {main_stuff_json_path}")- saved_main_stuff_formats.append("json")-- main_stuff_md = await main_stuff.content.rendered_markdown_async()- main_stuff_md_path = output_path / "main_stuff.md"- main_stuff_md_path.write_text(main_stuff_md, encoding="utf-8")- log.verbose(f"Main stuff Markdown saved to: {main_stuff_md_path}")- saved_main_stuff_formats.append("md")-- main_stuff_html = await main_stuff.content.rendered_html_async()- main_stuff_html_path = output_path / "main_stuff.html"- main_stuff_html_path.write_text(main_stuff_html, encoding="utf-8")- log.verbose(f"Main stuff HTML saved to: {main_stuff_html_path}")- saved_main_stuff_formats.append("html")-- main_stuff_viewer = await render_stuff_viewer(main_stuff)- main_stuff_viewer_path = output_path / "main_stuff_viewer.html"- main_stuff_viewer_path.write_text(main_stuff_viewer, encoding="utf-8")- log.verbose(f"Main stuff HTML viewer saved to: {main_stuff_viewer_path}")- saved_main_stuff_formats.append("html_viewer")+ main_stuff = pipe_output.working_memory.get_main_stuff()+ main_stuff_json = await main_stuff.content.rendered_json_async()+ main_stuff_json_path = output_path / "main_stuff.json"+ main_stuff_json_path.write_text(main_stuff_json, encoding="utf-8")+ log.verbose(f"Main stuff JSON saved to: {main_stuff_json_path}")+ saved_main_stuff_formats.append("json")++ main_stuff_md = await main_stuff.content.rendered_markdown_async()+ main_stuff_md_path = output_path / "main_stuff.md"+ main_stuff_md_path.write_text(main_stuff_md, encoding="utf-8")+ log.verbose(f"Main stuff Markdown saved to: {main_stuff_md_path}")+ saved_main_stuff_formats.append("md")++ main_stuff_html = await main_stuff.content.rendered_html_async()+ main_stuff_html_path = output_path / "main_stuff.html"+ main_stuff_html_path.write_text(main_stuff_html, encoding="utf-8")+ log.verbose(f"Main stuff HTML saved to: {main_stuff_html_path}")+ saved_main_stuff_formats.append("html")++ main_stuff_viewer = await render_stuff_viewer(main_stuff)+ main_stuff_viewer_path = output_path / "main_stuff_viewer.html"+ main_stuff_viewer_path.write_text(main_stuff_viewer, encoding="utf-8")+ log.verbose(f"Main stuff HTML viewer saved to: {main_stuff_viewer_path}")+ saved_main_stuff_formats.append("html_viewer")# Save working memory to JSON if enabledworking_memory_output_path: str | None = None@@ -277,10 +276,7 @@ async def _execute_run(# the codec rejects a non-flat row concept with a clear CsvFlatnessError.if save_csv is not None:# The path string and its suffix were already validated up front (fail-fast, before the run).- csv_main_stuff = pipe_output.working_memory.get_optional_main_stuff()- if csv_main_stuff is None:- typer.secho("Failed to --save-csv: the pipeline produced no main stuff to write.", fg=typer.colors.RED, err=True)- raise typer.Exit(1)+ csv_main_stuff = pipe_output.working_memory.get_main_stuff()csv_content = csv_main_stuff.contentif not isinstance(csv_content, ListContent):typer.secho(diff --git a/pipelex/core/bundles/pipelex_bundle_blueprint.py b/pipelex/core/bundles/pipelex_bundle_blueprint.pyindex b6a425402..4756aa033 100644--- a/pipelex/core/bundles/pipelex_bundle_blueprint.py+++ b/pipelex/core/bundles/pipelex_bundle_blueprint.py@@ -234,9 +234,4 @@ class PipelexBundleBlueprint(BaseModel):local_ref = parse_concept_with_multiplicity(pipe_blueprint.output).concept_ref_or_codelocal_refs.append((local_ref, f"pipe.{pipe_code}.output"))- # Check combined_output for PipeParallel- if isinstance(pipe_blueprint, PipeParallelBlueprint) and pipe_blueprint.combined_output:- local_ref = parse_concept_with_multiplicity(pipe_blueprint.combined_output).concept_ref_or_code- local_refs.append((local_ref, f"pipe.{pipe_code}.combined_output"))-return local_refsdiff --git a/pipelex/core/concepts/concept_factory.py b/pipelex/core/concepts/concept_factory.pyindex 080696e3b..5c8a82a73 100644--- a/pipelex/core/concepts/concept_factory.py+++ b/pipelex/core/concepts/concept_factory.py@@ -175,6 +175,13 @@ class ConceptFactory:description="A search result with answer and sources",structure_class_name=structure_class_name,)+ case NativeConceptCode.COMPOSITE:+ return Concept(+ code=native_concept_code,+ domain_code=SpecialDomain.NATIVE,+ description="A named composition of contents",+ structure_class_name=structure_class_name,+ )@classmethoddef make_domain_and_concept_code_from_concept_ref_or_code(diff --git a/pipelex/core/concepts/native/concept_native.py b/pipelex/core/concepts/native/concept_native.pyindex 46e5d89c3..94ce0ed09 100644--- a/pipelex/core/concepts/native/concept_native.py+++ b/pipelex/core/concepts/native/concept_native.py@@ -2,6 +2,7 @@ from pipelex.core.concepts.native.exceptions import NativeConceptDefinitionErrorfrom pipelex.core.concepts.validation import is_concept_ref_or_code_validfrom pipelex.core.domains.domain import SpecialDomainfrom pipelex.core.qualified_ref import QualifiedRef+from pipelex.core.stuffs.composite_content import CompositeContentfrom pipelex.core.stuffs.document_content import DocumentContentfrom pipelex.core.stuffs.dynamic_content import DynamicContentfrom pipelex.core.stuffs.html_content import HtmlContent@@ -27,11 +28,33 @@ class NativeConceptCode(StrEnum):JSON = "JSON"SEARCH_RESULT = "SearchResult"ANYTHING = "Anything"+ COMPOSITE = "Composite"@propertydef as_output_multiple_indeterminate(self) -> str:return f"{self.value}[]"+ @property+ def is_composite(self) -> bool:+ """Whether this native concept can hold a named composition (a PipeParallel combination)."""+ match self:+ case NativeConceptCode.COMPOSITE:+ return True+ case (+ NativeConceptCode.DYNAMIC+ | NativeConceptCode.TEXT+ | NativeConceptCode.IMAGE+ | NativeConceptCode.DOCUMENT+ | NativeConceptCode.HTML+ | NativeConceptCode.TEXT_AND_IMAGES+ | NativeConceptCode.NUMBER+ | NativeConceptCode.PAGE+ | NativeConceptCode.JSON+ | NativeConceptCode.SEARCH_RESULT+ | NativeConceptCode.ANYTHING+ ):+ return False+@propertydef concept_ref(self) -> str:return f"{SpecialDomain.NATIVE}.{self.value}"@@ -68,6 +91,8 @@ class NativeConceptCode(StrEnum):return JSONContentcase NativeConceptCode.SEARCH_RESULT:return SearchResultContent+ case NativeConceptCode.COMPOSITE:+ return CompositeContentcase NativeConceptCode.ANYTHING:# This doesn't have a dedicated content classreturn None@@ -123,6 +148,7 @@ class NativeConceptCode(StrEnum):| NativeConceptCode.ANYTHING| NativeConceptCode.JSON| NativeConceptCode.SEARCH_RESULT+ | NativeConceptCode.COMPOSITE):return False@@ -145,6 +171,7 @@ class NativeConceptCode(StrEnum):| NativeConceptCode.ANYTHING| NativeConceptCode.JSON| NativeConceptCode.SEARCH_RESULT+ | NativeConceptCode.COMPOSITE):return Falsecase NativeConceptCode.DYNAMIC:diff --git a/pipelex/core/memory/working_memory.py b/pipelex/core/memory/working_memory.pyindex 72ac82496..eb4541418 100644--- a/pipelex/core/memory/working_memory.py+++ b/pipelex/core/memory/working_memory.py@@ -12,6 +12,7 @@ from pipelex.core.memory.exceptions import (WorkingMemoryStuffNotFoundError,WorkingMemoryTypeError,)+from pipelex.core.stuffs.composite_content import CompositeContentfrom pipelex.core.stuffs.document_content import DocumentContentfrom pipelex.core.stuffs.html_content import HtmlContentfrom pipelex.core.stuffs.image_content import ImageContent@@ -493,7 +494,6 @@ class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):list_content = cast("ListContent[StuffContent]", content)serialized_items: list[dict[str, Any]] = []for item in list_content.items:- item_dict = item.model_dump(serialize_as_any=True)# Preserve per-item type metadata for the hydration side.# These keys are deliberately in pipelex's private namespace# (NOT kajson's `__class__` / `__module__`) so that kajson's@@ -501,8 +501,43 @@ class WorkingMemory(WorkingMemoryAbstract[Stuff], ContextProviderAbstract):# Temporal data-converter boundary — class binding is# pipelex's job, not kajson's, since the dynamic class may# only exist in a per-workflow ClassRegistry.- item_dict["__pipelex_class__"] = type(item).__name__- item_dict["__pipelex_module__"] = type(item).__module__- serialized_items.append(item_dict)+ serialized_items.append(_encode_content_with_class_markers(item))raw_root[stuff_name]["content"] = serialized_items+ elif isinstance(content, CompositeContent) and stuff_name in raw_root:+ # Composite components are extra="allow" fields: a plain model_dump loses+ # their classes, so stamp each component with the same pipelex-private+ # markers as list items. The nesting convention mirrors the top level:+ # a list value is a ListContent, a dict value is a single StuffContent.+ raw_root[stuff_name]["content"] = _encode_composite_for_transport(content)return raw+++def _encode_content_with_class_markers(content: StuffContent) -> dict[str, Any]:+ """Serialize one StuffContent with pipelex-private class markers for hydration."""+ if isinstance(content, CompositeContent):+ content_dict = _encode_composite_for_transport(content)+ else:+ content_dict = content.model_dump(serialize_as_any=True)+ content_dict["__pipelex_class__"] = type(content).__name__+ content_dict["__pipelex_module__"] = type(content).__module__+ return content_dict+++def _encode_composite_for_transport(composite: CompositeContent) -> dict[str, Any]:+ """Serialize a CompositeContent's components, each carrying its class markers.++ A ListContent component is encoded as a plain list of marker-stamped item dicts+ (same shape rule as the top-level transport format: list ⇒ ListContent, dict ⇒+ single StuffContent), so the hydration side rebuilds typed contents without+ heuristics.+ """+ encoded: dict[str, Any] = {}+ for component_name, component_value in composite.components.items():+ if isinstance(component_value, ListContent):+ list_component = cast("ListContent[StuffContent]", component_value)+ encoded[component_name] = [_encode_content_with_class_markers(item) for item in list_component.items]+ elif isinstance(component_value, StuffContent):+ encoded[component_name] = _encode_content_with_class_markers(component_value)+ else:+ encoded[component_name] = component_value+ return encodeddiff --git a/pipelex/core/pipes/pipe_abstract.py b/pipelex/core/pipes/pipe_abstract.pyindex 06c3cf95c..923c73ebb 100644--- a/pipelex/core/pipes/pipe_abstract.py+++ b/pipelex/core/pipes/pipe_abstract.py@@ -603,30 +603,27 @@ class PipeAbstract(ABC, BaseModel):# Record graph tracing successif tracer_manager is not None and parent_trace_context is not None:- # Capture output spec for data flow tracking- # Note: main_stuff may not exist for pipes like PipeParallel with add_each_output=true- main_stuff = pipe_output.working_memory.get_optional_main_stuff()- output_spec: IOSpec | None = None- if main_stuff is not None:- # E1: same gating as the input block — skip the discarded payload dumps in costs-only mode.- include_graph_data = parent_trace_context.emit_graph_events- output_spec = IOSpec(- name=output_name or main_stuff.stuff_name or "main_stuff",- concept=main_stuff.concept.code,- content_type=main_stuff.content.content_type,- digest=main_stuff.stuff_code,- data=main_stuff.content.smart_dump() if (include_graph_data and parent_trace_context.data_inclusion.stuff_json_content) else None,- data_text=main_stuff.content.rendered_pretty_text()- if (include_graph_data and parent_trace_context.data_inclusion.stuff_text_content)- else None,- data_html=main_stuff.content.rendered_pretty_html()- if (include_graph_data and parent_trace_context.data_inclusion.stuff_html_content)- else None,- )+ # Capture output spec for data flow tracking — a completed pipe run always delivers a main stuff.+ main_stuff = pipe_output.working_memory.get_main_stuff()+ # E1: same gating as the input block — skip the discarded payload dumps in costs-only mode.+ include_graph_data = parent_trace_context.emit_graph_events+ output_spec = IOSpec(+ name=output_name or main_stuff.stuff_name or "main_stuff",+ concept=main_stuff.concept.code,+ content_type=main_stuff.content.content_type,+ digest=main_stuff.stuff_code,+ data=main_stuff.content.smart_dump() if (include_graph_data and parent_trace_context.data_inclusion.stuff_json_content) else None,+ data_text=main_stuff.content.rendered_pretty_text()+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_text_content)+ else None,+ data_html=main_stuff.content.rendered_pretty_html()+ if (include_graph_data and parent_trace_context.data_inclusion.stuff_html_content)+ else None,+ )# Serialize output concept for registry if enabled (E1: also gated on emit_graph_events).output_concept_data: dict[str, Any] | None = None- if parent_trace_context.emit_graph_events and parent_trace_context.data_inclusion.pipe_and_concept_registry and main_stuff is not None:+ if parent_trace_context.emit_graph_events and parent_trace_context.data_inclusion.pipe_and_concept_registry:output_concept_data = self._make_single_concept_data_for_registry(main_stuff.concept, library_crate=library_crate)tracer_manager.on_pipe_end_success(diff --git a/pipelex/core/pipes/pipe_output.py b/pipelex/core/pipes/pipe_output.pyindex cfd5cc231..3df55d2a0 100644--- a/pipelex/core/pipes/pipe_output.py+++ b/pipelex/core/pipes/pipe_output.py@@ -35,10 +35,6 @@ class PipeOutput(PipeOutputAbstract[WorkingMemory]):def main_stuff(self) -> Stuff:return self.working_memory.get_main_stuff()- @property- def optional_main_stuff(self) -> Stuff | None:- return self.working_memory.get_optional_main_stuff()-def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]:"""Get main stuff content as ListContent with items of type StuffContentType.If the items are of possibly various types, use item_type=StuffContent.diff --git a/pipelex/core/registry_models.py b/pipelex/core/registry_models.pyindex 5d3cb96a7..a1e34a8b8 100644--- a/pipelex/core/registry_models.py+++ b/pipelex/core/registry_models.py@@ -2,6 +2,7 @@ from typing import Any, ClassVarfrom pipelex.core.pipes.pipe_abstract import PipeAbstractTypefrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocol+from pipelex.core.stuffs.composite_content import CompositeContentfrom pipelex.core.stuffs.document_content import DocumentContentfrom pipelex.core.stuffs.dynamic_content import DynamicContentfrom pipelex.core.stuffs.html_content import HtmlContent@@ -102,6 +103,7 @@ class CoreRegistryModels(RegistryModels):PageContent,JSONContent,SearchResultContent,+ CompositeContent,]EXPERIMENTAL: ClassVar[list[ModelType]] = [diff --git a/pipelex/core/stuffs/composite_content.py b/pipelex/core/stuffs/composite_content.pynew file mode 100644index 000000000..eec191849--- /dev/null+++ b/pipelex/core/stuffs/composite_content.py@@ -0,0 +1,56 @@+import html as html_module+from typing import Any++from pydantic import ConfigDict+from typing_extensions import override++from pipelex.core.stuffs.html_rendering import render_value_html+from pipelex.core.stuffs.stuff_content import StuffContent+from pipelex.tools.misc.markdown_utils import convert_to_markdown+++class CompositeContent(StuffContent):+ """Untyped named composition of contents — the structure class of `native.Composite`.++ Holds its named sub-contents as top-level fields (pydantic `extra="allow"`), so the+ serialized shape matches a bespoke structured concept: `{name: content, ...}` with no+ wrapper key. This is the combination vehicle for a `PipeParallel` whose author does not+ want to declare a bespoke concept: each branch's `result` name becomes a field.+ """++ model_config = ConfigDict(extra="allow")++ @property+ def components(self) -> dict[str, Any]:+ """Named sub-contents of this composite, keyed by their given names."""+ return dict(self.model_extra or {})++ def __json_encode__(self) -> dict[str, Any]: # noqa: PLW3201 — kajson encoder hook, name fixed by kajson's protocol+ """Kajson encoding hook.++ Pydantic stores `extra="allow"` fields outside `__dict__`, so kajson's default+ `__dict__` fallback would silently drop every component. Handing kajson the live+ components lets it recursively encode each one with its own class metadata, so a+ kajson round-trip rebuilds typed sub-contents.+ """+ return self.components++ @property+ @override+ def short_desc(self) -> str:+ component_names = ", ".join(self.components) or "nothing"+ return f"composite of {component_names}"++ @override+ def rendered_markdown(self, *, level: int = 1, is_pretty: bool = False) -> str:+ return convert_to_markdown(data=self.smart_dump(), level=level, is_pretty=is_pretty)++ @override+ def rendered_html(self) -> str:+ rows: list[str] = []+ for component_name, component_value in self.components.items():+ rendered_value = render_value_html(component_value)+ rows.append(f"<tr><th>{html_module.escape(component_name)}</th><td>{rendered_value}</td></tr>")+ if not rows:+ return "<table><tr><td><em>empty</em></td></tr></table>"+ return f"<table>{''.join(rows)}</table>"diff --git a/pipelex/core/stuffs/html_rendering.py b/pipelex/core/stuffs/html_rendering.pynew file mode 100644index 000000000..0210f7ab5--- /dev/null+++ b/pipelex/core/stuffs/html_rendering.py@@ -0,0 +1,79 @@+import html as html_module+from datetime import date, datetime+from enum import Enum+from typing import Any, cast++from pipelex.core.stuffs.stuff_content import StuffContent+++def render_value_html(value: Any) -> str:+ """Render a value as HTML, recursively handling nested structures.++ Shared by every content class that renders named sub-values as HTML+ (StructuredContent fields, CompositeContent components), so escaping and+ formatting behavior cannot drift between them.++ Uses match/case to handle all common Pydantic field types:+ - None: renders as <em>None</em>+ - StuffContent subclasses: calls rendered_html() recursively+ - str: escapes HTML (or passes through if already HTML)+ - bool: renders as "True" or "False" (must be before int, as bool is subclass of int)+ - int: renders as string+ - float: renders as string+ - Enum: renders the enum value+ - datetime/date: renders in ISO format+ - list/tuple: renders as <ul> list+ - dict: renders as <dl> definition list+ - Other: escapes string representation+ """+ match value:+ case None:+ return "<em>None</em>"++ case StuffContent():+ return value.rendered_html()++ case str():+ # If it looks like HTML (starts with <), render as-is+ stripped = value.strip()+ if stripped.startswith("<") and (">" in stripped):+ return value+ return html_module.escape(value)++ case bool():+ # Must be before int because bool is a subclass of int+ return "True" if value else "False"++ case int():+ return str(value)++ case float():+ return str(value)++ case Enum():+ # Handles StrEnum and regular Enum+ return html_module.escape(str(value.value))++ case datetime():+ return html_module.escape(value.isoformat())++ case date():+ return html_module.escape(value.isoformat())++ case list() | tuple():+ list_value = cast("list[Any]", value)+ if len(list_value) == 0:+ return "<em>empty</em>"+ items = [f"<li>{render_value_html(item)}</li>" for item in list_value]+ return f"<ul>{''.join(items)}</ul>"++ case dict():+ dict_value = cast("dict[str, Any]", value)+ if len(dict_value) == 0:+ return "<em>empty</em>"+ items = [f"<dt>{html_module.escape(str(key))}</dt><dd>{render_value_html(val)}</dd>" for key, val in dict_value.items()]+ return f"<dl>{''.join(items)}</dl>"++ case _:+ # Fallback for any other types+ return html_module.escape(str(value))diff --git a/pipelex/core/stuffs/structured_content.py b/pipelex/core/stuffs/structured_content.pyindex 2087f867c..963f4ffe0 100644--- a/pipelex/core/stuffs/structured_content.py+++ b/pipelex/core/stuffs/structured_content.py@@ -1,6 +1,4 @@import html as html_module-from datetime import date, datetime-from enum import Enumfrom typing import Any, castfrom rich.pretty import Pretty@@ -8,6 +6,7 @@ from rich.table import Tablefrom typing_extensions import overridefrom pipelex.cogt.templating.text_format import TextFormat+from pipelex.core.stuffs.html_rendering import render_value_htmlfrom pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.tools.jinja2.image_registry import ImageRegistryfrom pipelex.tools.jinja2.image_renderable import ImageRenderable@@ -30,7 +29,7 @@ class StructuredContent(StuffContent):field_value = getattr(self, field_name)if field_value is None:continue- rendered_value = self._render_value_html(field_value)+ rendered_value = render_value_html(field_value)rows.append(f"<tr><th>{html_module.escape(field_name)}</th><td>{rendered_value}</td></tr>")if not rows:return "<table><tr><td><em>empty</em></td></tr></table>"@@ -41,74 +40,6 @@ class StructuredContent(StuffContent):"""Async version of rendered_html."""return self.rendered_html()- def _render_value_html(self, value: Any) -> str:- """Render a value as HTML, recursively handling nested structures.-- Uses match/case to handle all common Pydantic field types:- - None: renders as <em>None</em>- - StuffContent subclasses: calls rendered_html() recursively- - str: escapes HTML (or passes through if already HTML)- - bool: renders as "True" or "False" (must be before int, as bool is subclass of int)- - int: renders as string- - float: renders as string- - Enum: renders the enum value- - datetime/date: renders in ISO format- - list/tuple: renders as <ul> list- - dict: renders as <dl> definition list- - Other: escapes string representation- """- match value:- case None:- return "<em>None</em>"-- case StuffContent():- return value.rendered_html()-- case str():- # If it looks like HTML (starts with <), render as-is- stripped = value.strip()- if stripped.startswith("<") and (">" in stripped):- return value- return html_module.escape(value)-- case bool():- # Must be before int because bool is a subclass of int- return "True" if value else "False"-- case int():- return str(value)-- case float():- return str(value)-- case Enum():- # Handles StrEnum and regular Enum- return html_module.escape(str(value.value))-- case datetime():- return html_module.escape(value.isoformat())-- case date():- return html_module.escape(value.isoformat())-- case list() | tuple():- list_value = cast("list[Any]", value)- if len(list_value) == 0:- return "<em>empty</em>"- items = [f"<li>{self._render_value_html(item)}</li>" for item in list_value]- return f"<ul>{''.join(items)}</ul>"-- case dict():- dict_value = cast("dict[str, Any]", value)- if len(dict_value) == 0:- return "<em>empty</em>"- items = [f"<dt>{html_module.escape(str(key))}</dt><dd>{self._render_value_html(val)}</dd>" for key, val in dict_value.items()]- return f"<dl>{''.join(items)}</dl>"-- case _:- # Fallback for any other types- return html_module.escape(str(value))-@overridedef rendered_markdown(self, *, level: int = 1, is_pretty: bool = False) -> str:dict_dump = clean_model_to_dict(obj=self)diff --git a/pipelex/core/stuffs/stuff_factory.py b/pipelex/core/stuffs/stuff_factory.pyindex 923dbc7ba..3190d6933 100644--- a/pipelex/core/stuffs/stuff_factory.py+++ b/pipelex/core/stuffs/stuff_factory.py@@ -118,6 +118,7 @@ class StuffFactory:*,concept: Concept,name: str | None = None,+ code: str | None = None,) -> Stuff:"""Combine a dictionary of stuffs into a single stuff."""the_subclass = get_class_registry().get_required_subclass(name=concept.structure_class_name, base_class=StuffContent)@@ -130,6 +131,7 @@ class StuffFactory:concept=concept,content=the_stuff_content,name=name,+ code=code,)@classmethoddiff --git a/pipelex/pipe_controllers/condition/pipe_condition.py b/pipelex/pipe_controllers/condition/pipe_condition.pyindex 465078579..1fbd2f513 100644--- a/pipelex/pipe_controllers/condition/pipe_condition.py+++ b/pipelex/pipe_controllers/condition/pipe_condition.py@@ -241,11 +241,21 @@ class PipeCondition(PipeController):"selected_outcome": str(outcome),}- # Handle continue case+ # Handle continue case: `continue` delivers the current main stuff by passing it through.+ # With nothing to pass through, the run would deliver no main stuff — a contract violation+ # ("a pipe run always delivers a main stuff") failed loudly here, at the semantic site,+ # instead of crashing downstream (graph tracer, delivery, telemetry).if SpecialOutcome.is_continue(outcome):log.dev(f"PipeCondition '{self.code}' continued with outcome: {outcome}. Evaluated expression: {evaluated_expression}")self._register_execution_data(job_metadata, execution_data=execution_data_dict)- return PipeOutput(working_memory=working_memory)+ if working_memory.get_optional_main_stuff() is None:+ msg = (+ f"PipeCondition '{self.code}' resolved to the 'continue' outcome but the working memory has "+ "no main stuff to pass through — a pipe run must deliver a main stuff. Map the outcome to a "+ "pipe, or place this condition after a step that produces the output to pass through."+ )+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)if SpecialOutcome.is_fail(outcome):self._register_execution_data(job_metadata, execution_data=execution_data_dict)@@ -337,4 +347,13 @@ class PipeCondition(PipeController):"selected_outcome": "all_outcomes",}self._register_execution_data(job_metadata, execution_data=execution_data_dict)- return PipeOutput(working_memory=working_memory)+ # Dry-run parity with the live `continue` guard: the mapped pipes just dry-ran into this+ # memory and stamped their outputs; if none did (all-special-outcomes condition) and no+ # main stuff pre-existed, the run cannot deliver one — fail loudly at the pipe level.+ if working_memory.get_optional_main_stuff() is None:+ msg = (+ f"Dry run of PipeCondition '{self.code}' delivered no main stuff — with only special outcomes "+ "mapped and no pre-existing main stuff, the run cannot deliver an output."+ )+ raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)+ return PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel.py b/pipelex/pipe_controllers/parallel/pipe_parallel.pyindex e91f58a8c..af46e756a 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel.py@@ -1,17 +1,21 @@import asyncio+import builtinsfrom typing import TYPE_CHECKING, Any, Literal-from pydantic import field_validator, model_validator+from pydantic import field_validatorfrom typing_extensions import overridefrom pipelex import logfrom pipelex.core.concepts.concept import Concept+from pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.memory.working_memory import WorkingMemory-from pipelex.core.pipes.exceptions import PipeValidationError+from pipelex.core.pipes.exceptions import PipeValidationError, PipeValidationErrorTypefrom pipelex.core.pipes.inputs.exceptions import InputStuffSpecNotFoundErrorfrom pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecsfrom pipelex.core.pipes.inputs.input_stuff_specs_factory import InputStuffSpecsFactoryfrom pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.stuffs.composite_content import CompositeContent+from pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.graph.graph_tracer_manager import GraphTracerManagerfrom pipelex.graph.graphspec import IOSpec@@ -23,13 +27,11 @@ from pipelex.pipe_run.exceptions import PipeRunErrorfrom pipelex.pipe_run.pipe_run_params import PipeRunParamsfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.misc.string_utils import get_root_from_dotted_path-from pipelex.types import Selfif TYPE_CHECKING:from collections.abc import Coroutinefrom pipelex.core.stuffs.stuff import Stuff- from pipelex.core.stuffs.stuff_content import StuffContentfrom pipelex.libraries.library_crate import LibraryCrate@@ -38,7 +40,6 @@ class PipeParallel(PipeController):parallel_sub_pipes: list[SubPipe]add_each_output: bool- combined_output: Concept | None@field_validator("parallel_sub_pipes", mode="before")@classmethod@@ -102,15 +103,6 @@ class PipeParallel(PipeController):needed_inputs.add_stuff_spec(input_name, concept=stuff_spec.concept, multiplicity=stuff_spec.multiplicity)return needed_inputs- @model_validator(mode="after")- def validate_fields_add_each_output_and_combined_output(self) -> Self:- # Validate that either add_each_output or combined_output is set- if not self.add_each_output and not self.combined_output:- msg = f"PipeParallel'{self.code}'requires either add_each_output or combined_output to be set"- raise ValueError(msg)-- return self-@overridedef validate_inputs_static(self):pass@@ -121,29 +113,155 @@ class PipeParallel(PipeController):@overridedef validate_output_static(self):- pass+ if self.output.multiplicity is not None:+ msg = (+ f"PipeParallel '{self.code}' output must not declare a multiplicity: "+ "a parallel combination is a named composite, never a list (a list aggregation is PipeBatch's shape)."+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_MULTIPLICITY,+ domain_code=self.domain_code,+ pipe_code=self.code,+ )+ output_concept = self.output.concept+ if Concept.is_native_concept(concept=output_concept):+ native_code = NativeConceptCode(output_concept.code)+ if not native_code.is_composite:+ msg = (+ f"PipeParallel '{self.code}' output '{output_concept.concept_ref}' is invalid: "+ "the output of a parallel must be 'Composite' or a structured concept whose fields "+ "correspond to the branch result names."+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,+ domain_code=self.domain_code,+ pipe_code=self.code,+ provided_concept_code=output_concept.concept_ref,+ )@overridedef validate_output_with_library(self):- pass+ """Check that the declared output structure can hold the branch results.++ The declared output is either `Composite` (the untyped composition vehicle — accepts any+ branch names) or a structured concept whose fields are compatible with the branch `result`+ names: required fields ⊆ result names, result names ⊆ declared fields. This turns the+ runtime combine failure into an author-time error surfaced by `/validate`.+ """+ structure_class = self.output.concept.get_structure_class()+ if issubclass(structure_class, CompositeContent):+ return++ result_names = {sub_pipe.output_name for sub_pipe in self.parallel_sub_pipes if sub_pipe.output_name}+ declared_fields = set(structure_class.model_fields.keys())+ required_fields = {field_name for field_name, field_info in structure_class.model_fields.items() if field_info.is_required()}++ missing_required_fields = required_fields - result_names+ if missing_required_fields:+ msg = (+ f"PipeParallel '{self.code}' output '{self.output.concept.concept_ref}' requires field(s) "+ f"{sorted(missing_required_fields)} that no branch produces. Branch result names: {sorted(result_names)}. "+ "Each required field of the output structure must match a branch result name."+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,+ domain_code=self.domain_code,+ pipe_code=self.code,+ provided_concept_code=self.output.concept.concept_ref,+ variable_names=sorted(missing_required_fields),+ )++ extraneous_result_names = result_names - declared_fields+ if extraneous_result_names:+ msg = (+ f"PipeParallel '{self.code}' output '{self.output.concept.concept_ref}' has no field for branch "+ f"result(s) {sorted(extraneous_result_names)}. Declared fields: {sorted(declared_fields)}. "+ "Every branch result name must match a field of the output structure, or use 'Composite' "+ "as the output for an untyped composition."+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,+ domain_code=self.domain_code,+ pipe_code=self.code,+ provided_concept_code=self.output.concept.concept_ref,+ variable_names=sorted(extraneous_result_names),+ )++ self._validate_branch_output_types(structure_class=structure_class)++ # Note: builtins.type because the `type: Literal["PipeParallel"]` field shadows the+ # builtin in this class body, where signature annotations are evaluated.+ def _validate_branch_output_types(self, *, structure_class: builtins.type[StuffContent]) -> None:+ """Check each branch's output concept against its target field's content class.++ Compatibility is conceptual, not type equality: a branch fits a field when its+ output concept's structure class IS the field's content class or a subclass of+ it — refinement is honored structurally because a refining concept's generated+ class inherits from the refined concept's class (e.g. refines-Text ⇒ subclass+ of TextContent). This is exactly the relation pydantic accepts when+ `StuffFactory.combine_stuffs()` validates the combined content at run time, so+ the check turns that runtime failure into an author-time `/validate` error.++ Conservative by design: fields whose annotation is not a plain StuffContent+ subclass (primitives from .mthds-generated structures, Optionals, unions) and+ list-producing branches are skipped rather than guessed at.+ """+ for sub_pipe in self.parallel_sub_pipes:+ if not sub_pipe.output_name or sub_pipe.output_name not in structure_class.model_fields:+ continue+ field_annotation = structure_class.model_fields[sub_pipe.output_name].annotation+ if not isinstance(field_annotation, type) or not issubclass(field_annotation, StuffContent):+ continue+ branch_pipe = get_required_pipe(pipe_code=sub_pipe.pipe_code)+ if branch_pipe.output.multiplicity is not None:+ # A list-producing branch combines as a ListContent, not as the item class.+ continue+ branch_structure_class = branch_pipe.output.concept.get_structure_class()+ if not issubclass(branch_structure_class, field_annotation):+ msg = (+ f"PipeParallel '{self.code}' branch '{sub_pipe.pipe_code}' produces "+ f"'{branch_pipe.output.concept.concept_ref}' (structure '{branch_structure_class.__name__}') for result "+ f"'{sub_pipe.output_name}', but field '{sub_pipe.output_name}' of output "+ f"'{self.output.concept.concept_ref}' expects '{field_annotation.__name__}'. "+ "The branch output concept must match the field's content class or refine a concept that does."+ )+ raise PipeValidationError(+ message=msg,+ error_type=PipeValidationErrorType.INADEQUATE_OUTPUT_CONCEPT,+ domain_code=self.domain_code,+ pipe_code=self.code,+ provided_concept_code=branch_pipe.output.concept.concept_ref,+ variable_names=[sub_pipe.output_name],+ )@overridedef pipe_dependencies(self) -> set[str]:return {sub_pipe.pipe_code for sub_pipe in self.parallel_sub_pipes}- @override- async def _live_run_controller_pipe(+ async def _run_branches_and_combine(self,*,job_metadata: JobMetadata,working_memory: WorkingMemory,pipe_run_params: PipeRunParams,- output_name: str | None = None,- library_crate: "LibraryCrate | None" = None,+ output_name: str | None,+ library_crate: "LibraryCrate | None",) -> PipeOutput:- if pipe_run_params.final_stuff_code:- log.verbose(f"PipeBatch.run_pipe() final_stuff_code: {pipe_run_params.final_stuff_code}")- pipe_run_params.final_stuff_code = None+ """Run all branches on deep copies of the memory, then always combine their outputs.++ Shared by live and dry run: the sub-pipes' own run_pipe dispatch honors the run mode,+ so live and dry runs stamp the same combined main stuff shape.+ """+ # A requested final_stuff_code applies to this pipe's own final output — the COMBINED+ # stuff (e.g. PipeBatch pre-allocates it so its aggregated item identity matches the+ # branch's final output in the graph). Capture it and clear it before fanning out, so the+ # branch runs (which deep-copy the params) don't stamp it on their own outputs.+ final_stuff_code = pipe_run_params.final_stuff_code+ pipe_run_params.final_stuff_code = Nonetasks: list[Coroutine[Any, Any, PipeOutput]] = []@@ -160,51 +278,48 @@ class PipeParallel(PipeController):pipe_outputs = await asyncio.gather(*tasks)- output_stuff_content_items: list[StuffContent] = []output_stuffs: dict[str, Stuff] = {}output_stuff_contents: dict[str, StuffContent] = {}- # TODO: refactor this to use a specific function for this that can also be used in dry runfor output_index, pipe_output in enumerate(pipe_outputs):output_stuff = pipe_output.main_stuffsub_pipe_output_name = self.parallel_sub_pipes[output_index].output_nameif not sub_pipe_output_name:- msg = "PipeParallel requires a result specified for each parallel sub pipe"+ sub_pipe_code = self.parallel_sub_pipes[output_index].pipe_code+ msg = f"PipeParallel '{self.code}': sub-pipe '{sub_pipe_code}' output name not specified"raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)- if self.add_each_output:- working_memory.add_new_stuff(name=sub_pipe_output_name, stuff=output_stuff)- output_stuff_content_items.append(output_stuff.content)if sub_pipe_output_name in output_stuffs:# TODO: check that at the blueprint / factory level- msg = f"PipeParallel requires unique output names for each parallel sub pipe, but {sub_pipe_output_name} is already used"+ sub_pipe_code = self.parallel_sub_pipes[output_index].pipe_code+ msg = f"PipeParallel '{self.code}': sub-pipe '{sub_pipe_code}' duplicate output name '{sub_pipe_output_name}'"raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)+ if self.add_each_output:+ working_memory.add_new_stuff(name=sub_pipe_output_name, stuff=output_stuff)output_stuffs[sub_pipe_output_name] = output_stuff- if sub_pipe_output_name in output_stuff_contents:- # TODO: check that at the blueprint / factory level- msg = f"PipeParallel requires unique output names for each parallel sub pipe, but {sub_pipe_output_name} is already used"- raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)output_stuff_contents[sub_pipe_output_name] = output_stuff.contentlog.verbose(f"PipeParallel '{self.code}': output_stuff_contents[{sub_pipe_output_name}]: {output_stuff_contents[sub_pipe_output_name]}")- if self.combined_output:- combined_output_stuff = StuffFactory.combine_stuffs(- concept=self.combined_output,- stuff_contents=output_stuff_contents,- name=output_name,- )- working_memory.set_new_main_stuff(- stuff=combined_output_stuff,- name=output_name,- )+ # Always combine the branch outputs into the declared output concept and stamp it as main stuff:+ # a pipe run always delivers a main stuff.+ combined_output_stuff = StuffFactory.combine_stuffs(+ concept=self.output.concept,+ stuff_contents=output_stuff_contents,+ name=output_name,+ code=final_stuff_code,+ )+ working_memory.set_new_main_stuff(+ stuff=combined_output_stuff,+ name=output_name,+ )- # Register parallel combine edges BEFORE register_branch_outputs, because- # register_parallel_combine snapshots the original branch producers from- # _stuff_producer_map before register_controller_output overrides them- self._register_parallel_combine_with_graph_tracer(- job_metadata=job_metadata,- combined_stuff=combined_output_stuff,- branch_stuffs=output_stuffs,- )+ # Register parallel combine edges BEFORE register_branch_outputs, because+ # register_parallel_combine snapshots the original branch producers from+ # _stuff_producer_map before register_controller_output overrides them+ self._register_parallel_combine_with_graph_tracer(+ job_metadata=job_metadata,+ combined_stuff=combined_output_stuff,+ branch_stuffs=output_stuffs,+ )# Register branch outputs with graph tracer so DATA edges flow from PipeParallel to downstream consumersself._register_branch_outputs_with_graph_tracer(@@ -216,7 +331,7 @@ class PipeParallel(PipeController):execution_data_dict: dict[str, Any] = {"branch_count": len(self.parallel_sub_pipes),"add_each_output": self.add_each_output,- "combined_output_concept": self.combined_output.concept_ref if self.combined_output else None,+ "combined_output_concept": self.output.concept.concept_ref,}self._register_execution_data(job_metadata, execution_data=execution_data_dict)@@ -225,6 +340,24 @@ class PipeParallel(PipeController):pipeline_run_id=job_metadata.pipeline_run_id,)+ @override+ async def _live_run_controller_pipe(+ self,+ *,+ job_metadata: JobMetadata,+ working_memory: WorkingMemory,+ pipe_run_params: PipeRunParams,+ output_name: str | None = None,+ library_crate: "LibraryCrate | None" = None,+ ) -> PipeOutput:+ return await self._run_branches_and_combine(+ job_metadata=job_metadata,+ working_memory=working_memory,+ pipe_run_params=pipe_run_params,+ output_name=output_name,+ library_crate=library_crate,+ )+@overrideasync def _dry_run_controller_pipe(self,@@ -235,7 +368,7 @@ class PipeParallel(PipeController):output_name: str | None = None,library_crate: "LibraryCrate | None" = None,) -> PipeOutput:- # 1. Validate that all sub-pipes exist+ # Validate that all sub-pipes exist before dispatching the dry branchesfor sub_pipe in self.parallel_sub_pipes:try:get_required_pipe(pipe_code=sub_pipe.pipe_code)@@ -243,84 +376,12 @@ class PipeParallel(PipeController):msg = f"Dry run failed for pipe '{self.code}' (PipeParallel): sub-pipe '{sub_pipe.pipe_code}' not found"raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code) from exc- # 2. Run all sub-pipes in dry mode- tasks: list[Coroutine[Any, Any, PipeOutput]] = []-- for sub_pipe in self.parallel_sub_pipes:- tasks.append(- sub_pipe.run_pipe(- calling_pipe_code=self.code,- job_metadata=job_metadata,- working_memory=working_memory.make_deep_copy(),- sub_pipe_run_params=pipe_run_params.make_deep_copy(),- library_crate=library_crate,- ),- )-- pipe_outputs = await asyncio.gather(*tasks)-- # 3. Process outputs as in the regular run- output_stuffs: dict[str, Stuff] = {}- output_stuff_contents: dict[str, StuffContent] = {}-- for output_index, pipe_output in enumerate(pipe_outputs):- output_stuff = pipe_output.main_stuff- sub_pipe_output_name = self.parallel_sub_pipes[output_index].output_name- if not sub_pipe_output_name:- sub_pipe_code = self.parallel_sub_pipes[output_index].pipe_code- msg = f"Dry run failed for pipe '{self.code}' (PipeParallel): sub-pipe '{sub_pipe_code}' output name not specified"- raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)-- if self.add_each_output:- working_memory.add_new_stuff(name=sub_pipe_output_name, stuff=output_stuff)-- if sub_pipe_output_name in output_stuffs:- sub_pipe_code = self.parallel_sub_pipes[output_index].pipe_code- msg = (- f"Dry run failed for pipe '{self.code}' (PipeParallel): sub-pipe '{sub_pipe_code}' duplicate output name '{sub_pipe_output_name}'"- )- raise PipeRunError(message=msg, run_mode=pipe_run_params.run_mode, pipe_code=self.code)-- output_stuffs[sub_pipe_output_name] = output_stuff- output_stuff_contents[sub_pipe_output_name] = output_stuff.content-- # 4. Handle combined output if specified- if self.combined_output:- combined_output_stuff = StuffFactory.combine_stuffs(- concept=self.combined_output,- stuff_contents=output_stuff_contents,- name=output_name,- )- working_memory.set_new_main_stuff(- stuff=combined_output_stuff,- name=output_name,- )-- # Register parallel combine edges BEFORE register_branch_outputs, because- # register_parallel_combine snapshots the original branch producers from- # _stuff_producer_map before register_controller_output overrides them- self._register_parallel_combine_with_graph_tracer(- job_metadata=job_metadata,- combined_stuff=combined_output_stuff,- branch_stuffs=output_stuffs,- )-- # Register branch outputs with graph tracer so DATA edges flow from PipeParallel to downstream consumers- self._register_branch_outputs_with_graph_tracer(+ return await self._run_branches_and_combine(job_metadata=job_metadata,- output_stuffs=output_stuffs,- )-- execution_data_dict: dict[str, Any] = {- "branch_count": len(self.parallel_sub_pipes),- "add_each_output": self.add_each_output,- "combined_output_concept": self.combined_output.concept_ref if self.combined_output else None,- }- self._register_execution_data(job_metadata, execution_data=execution_data_dict)-- return PipeOutput(working_memory=working_memory,- pipeline_run_id=job_metadata.pipeline_run_id,+ pipe_run_params=pipe_run_params,+ output_name=output_name,+ library_crate=library_crate,)def _register_branch_outputs_with_graph_tracer(diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.pyindex a1c6f6886..98c18a7e7 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_blueprint.py@@ -1,12 +1,11 @@from typing import Literal-from pydantic import field_validator, model_validatorfrom typing_extensions import override-from pipelex.core.concepts.validation import is_concept_ref_or_code_valid+from pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.pipes.pipe_blueprint import PipeBlueprint+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicityfrom pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint-from pipelex.types import Selfclass PipeParallelBlueprint(PipeBlueprint):@@ -14,7 +13,6 @@ class PipeParallelBlueprint(PipeBlueprint):pipe_category: Literal["PipeController"] = "PipeController"branches: list[SubPipeBlueprint]add_each_output: bool = False- combined_output: str | None = None@property@override@@ -22,29 +20,29 @@ class PipeParallelBlueprint(PipeBlueprint):"""Return the set of pipe codes from the parallel branches."""return {branch.pipe for branch in self.branches}- @field_validator("combined_output", mode="before")- @classmethod- def validate_combined_output(cls, combined_output: str) -> str:- if combined_output:- if not is_concept_ref_or_code_valid(concept_ref_or_code=combined_output):- msg = f"Combined output '{combined_output}' is not a valid concept string or code"- raise ValueError(msg)- return combined_output-- @model_validator(mode="after")- def validate_output_options(self) -> Self:- if not self.add_each_output and not self.combined_output:+ @override+ def validate_output(self):+ """A parallel always combines its branch outputs into its declared output.++ The combination is a named composite, so the output must be `Composite` or a structured+ concept — never a scalar native concept, `Dynamic`, `Anything`, or a list (a list+ aggregation is PipeBatch's shape). The structured concept's field compatibility with the+ branch result names is checked later, at library validation time.+ """+ # generic_validate_output already checked the syntax, so this parse cannot fail+ output_parse_result = parse_concept_with_multiplicity(self.output)+ if output_parse_result.multiplicity is not None:msg = (- "PipeParallel requires either add_each_output to be True or combined_output to be set, "- "or both, otherwise the pipe won't output anything"+ f"PipeParallel output '{self.output}' must not declare a multiplicity: "+ "a parallel combination is a named composite, never a list (a list aggregation is PipeBatch's shape).")raise ValueError(msg)- return self-- @override- def validate_inputs(self):- pass-- @override- def validate_output(self):- pass+ concept_ref_or_code = output_parse_result.concept_ref_or_code+ if NativeConceptCode.is_native_concept_ref_or_code(concept_ref_or_code=concept_ref_or_code):+ native_code = NativeConceptCode(concept_ref_or_code.split(".")[-1])+ if not native_code.is_composite:+ msg = (+ f"PipeParallel output '{self.output}' is invalid: the output of a parallel must be 'Composite' "+ "or a structured concept whose fields correspond to the branch result names."+ )+ raise ValueError(msg)diff --git a/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py b/pipelex/pipe_controllers/parallel/pipe_parallel_factory.pyindex cbc5e00e9..afe5b2290 100644--- a/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py+++ b/pipelex/pipe_controllers/parallel/pipe_parallel_factory.py@@ -2,11 +2,9 @@ from typing import TYPE_CHECKING, Anyfrom typing_extensions import override-from pipelex.core.concepts.concept_factory import ConceptFactoryfrom pipelex.core.pipes.inputs.input_stuff_specs import InputStuffSpecsfrom pipelex.core.pipes.pipe_factory import PipeFactoryProtocolfrom pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpec-from pipelex.hub import get_required_conceptfrom pipelex.pipe_controllers.parallel.exceptions import PipeParallelFactoryErrorfrom pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallelfrom pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint@@ -38,29 +36,6 @@ class PipeParallelFactory(PipeFactoryProtocol[PipeParallelBlueprint, PipeParalleraise PipeParallelFactoryError(message=msg)sub_pipe = SubPipeFactory.make_from_blueprint(sub_pipe_blueprint)parallel_sub_pipes.append(sub_pipe)- if not blueprint.add_each_output and not blueprint.combined_output:- msg = (- f"Unexpected error in pipe '{pipe_code}': PipeParallel requires either add_each_output to be True or combined_output to be set, "- "or both, otherwise the pipe won't output anything"- )- raise PipeParallelFactoryError(- message=msg,- )-- # Handle combined_output if specified- if blueprint.combined_output:- combined_output_domain_and_code = ConceptFactory.make_domain_and_concept_code_from_concept_ref_or_code(- domain_code=domain_code,- concept_ref_or_code=blueprint.combined_output,- )- combined_output = get_required_concept(- concept_ref=ConceptFactory.make_concept_ref_with_domain(- domain_code=combined_output_domain_and_code.domain_code,- concept_code=combined_output_domain_and_code.concept_code,- ),- )- else:- combined_output = Nonereturn PipeParallel(domain_code=domain_code,@@ -70,5 +45,4 @@ class PipeParallelFactory(PipeFactoryProtocol[PipeParallelBlueprint, PipeParalleoutput=output,parallel_sub_pipes=parallel_sub_pipes,add_each_output=blueprint.add_each_output or False,- combined_output=combined_output,)diff --git a/pipelex/pipe_run/delivery_executor.py b/pipelex/pipe_run/delivery_executor.pyindex 2bac41f34..71af5b68a 100644--- a/pipelex/pipe_run/delivery_executor.py+++ b/pipelex/pipe_run/delivery_executor.py@@ -88,6 +88,11 @@ class DeliveryExecutor:registered classes; on failure it falls back to a generic dictrender so built-in content types still get typed rendering anddynamic concepts produce a readable JSON dump.++ A completed run always delivers a main stuff, so the main_stuff.*+ artifact files are always produced — a working memory without one is+ a contract violation that fails the delivery loudly, never an "empty+ envelope" silently missing its result files."""files: dict[str, ResultFile] = {}@@ -97,19 +102,21 @@ class DeliveryExecutor:content_type="application/json",)raw_main_stuff = self._get_raw_main_stuff_dict(pipe_output.working_memory_raw)- main_stuff = self.try_local_hydrate_stuff(raw_main_stuff) if raw_main_stuff is not None else None- if main_stuff is not None:- await self._generate_main_stuff_files(main_stuff, files=files)- elif raw_main_stuff is not None:+ if raw_main_stuff is None:+ msg = "Delivery of a completed run found no main stuff in the raw working memory — a completed run always delivers a main stuff."+ raise PipeJobError(msg)+ hydrated_main_stuff = self.try_local_hydrate_stuff(raw_main_stuff)+ if hydrated_main_stuff is not None:+ await self._generate_main_stuff_files(hydrated_main_stuff, files=files)+ else:self._generate_main_stuff_files_from_raw(raw_main_stuff, files=files)else:files["working_memory.json"] = ResultFile(data=clean_json_dumps(pipe_output.working_memory.smart_dump(), indent=2).encode("utf-8"),content_type="application/json",)- main_stuff = pipe_output.working_memory.get_optional_main_stuff()- if main_stuff is not None:- await self._generate_main_stuff_files(main_stuff, files=files)+ main_stuff = pipe_output.working_memory.get_main_stuff()+ await self._generate_main_stuff_files(main_stuff, files=files)graph_spec = pipe_output.graph_specif graph_spec:diff --git a/pipelex/pipeline/pipeline_response.py b/pipelex/pipeline/pipeline_response.pyindex 43b9f492c..71e5f5444 100644--- a/pipelex/pipeline/pipeline_response.py+++ b/pipelex/pipeline/pipeline_response.py@@ -2,8 +2,8 @@ from __future__ import annotationsfrom mthds.protocol.models import RunResultExecute, RunResultStart-from pipelex.core.memory.working_memory import MAIN_STUFF_NAMEfrom pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.runtime_bridge.serialization import resolve_main_stuff_root_keyfrom pipelex.types import StrEnum@@ -30,7 +30,8 @@ class PipelexRunResultExecute(RunResultExecute[PipeOutput]):created_at: strstate: RunStatefinished_at: str | None = None- main_stuff_name: str | None = None+ # A completed run always delivers a main stuff; this is the actual working-memory root key it lives under.+ main_stuff_name: str@classmethoddef from_pipe_output(@@ -48,7 +49,7 @@ class PipelexRunResultExecute(RunResultExecute[PipeOutput]):state=state,finished_at=finished_at,pipe_output=pipe_output,- main_stuff_name=pipe_output.working_memory.aliases.get(MAIN_STUFF_NAME, MAIN_STUFF_NAME),+ main_stuff_name=resolve_main_stuff_root_key(pipe_output=pipe_output),)diff --git a/pipelex/plugins/orchestrator_registry.py b/pipelex/plugins/orchestrator_registry.pyindex 25aa040de..466d16cca 100644--- a/pipelex/plugins/orchestrator_registry.py+++ b/pipelex/plugins/orchestrator_registry.py@@ -1,12 +1,11 @@from typing import TYPE_CHECKING, Protocol, runtime_checkable-from pipelex.runtime_bridge.delivery_mode import DeliveryModefrom pipelex.runtime_bridge.orchestration_mode import OrchestrationModeif TYPE_CHECKING:from pipelex.pipe_run.delivery_assignment import DeliveryAssignmentfrom pipelex.pipe_run.pipe_job import PipeJob- from pipelex.runtime_bridge.payloads import PipelexPipeRunOutput+ from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRunOutput@runtime_checkable@@ -19,14 +18,19 @@ class OrchestratorProtocol(Protocol):dispatches by token through the ``OrchestratorRegistry`` instead of branching on a``match``.- ``run`` takes the endpoint-chosen ``delivery`` (the wait-semantics axis) and honors- it per the orchestrator's nature: an in-process orchestrator always blocks; a- distributed one awaits completion for ``BLOCKING`` and returns a workflow id for- ``FIRE_AND_FORGET``.+ The wait-semantics axis (``DeliveryMode`` on the wire input) is expressed here as+ which method the endpoint calls, so each return type is truthful on its own:++ - ``run`` is the BLOCKING arm — it awaits completion and returns the completed-run+ ``PipelexPipeRunOutput`` (which therefore always carries a main stuff).+ - ``start`` is the FIRE_AND_FORGET arm — it genuinely enqueues the job and returns+ a ``PipelexPipeDispatchAck`` (ids only; nothing has run yet).``supports_fire_and_forget`` is the capability the runner reads *before* dispatch:``/start`` rejects (honestly, with a 4xx) when the resolved mode's orchestrator- cannot do genuine async, rather than silently running blocking and acking.+ cannot do genuine async, rather than silently running blocking and acking. An+ orchestrator with ``supports_fire_and_forget = False`` implements ``start`` by+ raising — the gate keeps it unreachable."""supports_fire_and_forget: bool@@ -36,9 +40,15 @@ class OrchestratorProtocol(Protocol):*,pipe_job: "PipeJob",delivery_assignment: "DeliveryAssignment | None",- delivery: DeliveryMode,) -> "PipelexPipeRunOutput": ...+ async def start(+ self,+ *,+ pipe_job: "PipeJob",+ delivery_assignment: "DeliveryAssignment | None",+ ) -> "PipelexPipeDispatchAck": ...+class OrchestratorRegistry:"""Read view over the orchestrators contributed by discovered plugins.diff --git a/pipelex/runtime_bridge/direct_orchestrator.py b/pipelex/runtime_bridge/direct_orchestrator.pyindex 787392929..7206769ed 100644--- a/pipelex/runtime_bridge/direct_orchestrator.py+++ b/pipelex/runtime_bridge/direct_orchestrator.py@@ -20,8 +20,7 @@ from pipelex.runtime_bridge.serialization import PIPE_DISPATCH_ERRORS, serializeif TYPE_CHECKING:from pipelex.pipe_run.delivery_assignment import DeliveryAssignmentfrom pipelex.pipe_run.pipe_job import PipeJob- from pipelex.runtime_bridge.delivery_mode import DeliveryMode- from pipelex.runtime_bridge.payloads import PipelexPipeRunOutput+ from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRunOutputclass DirectOrchestrator:@@ -32,7 +31,7 @@ class DirectOrchestrator:# rather than running blocking and falsely acking.supports_fire_and_forget = False- async def run(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None, delivery: DeliveryMode) -> PipelexPipeRunOutput: # noqa: ARG002 — delivery accepted for protocol uniformity; in-process always blocks+ async def run(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput:# DIRECT mode forces in-process execution even inside a Temporal-enabled# worker. Scope the in-process router as the active router for the WHOLE# run so nested controller sub-pipes — which dispatch through@@ -53,3 +52,10 @@ class DirectOrchestrator:pipe_output=pipe_output,workflow_id=None,)++ async def start(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeDispatchAck: # noqa: ARG002 — protocol signature; unreachable behind the supports_fire_and_forget gate+ msg = (+ f"DIRECT orchestrator cannot fire-and-forget pipe '{pipe_job.pipe.code}': in-process execution has no genuine async path. "+ "Callers must check `supports_fire_and_forget` before dispatching a fire-and-forget job."+ )+ raise PipelexBridgeDispatchError(msg)diff --git a/pipelex/runtime_bridge/payloads.py b/pipelex/runtime_bridge/payloads.pyindex 1553fdc59..fa2a4056d 100644--- a/pipelex/runtime_bridge/payloads.py+++ b/pipelex/runtime_bridge/payloads.py@@ -37,15 +37,21 @@ class PipelexPipeRunInput(BaseModel):class PipelexPipeRunOutput(BaseModel):- """JSON-safe output crossing the host-runtime / Temporal boundary."""+ """JSON-safe output of a COMPLETED run crossing the host-runtime / Temporal boundary.++ This is the return shape of a blocking dispatch (``OrchestratorProtocol.run``); a+ fire-and-forget dispatch returns a ``PipelexPipeDispatchAck`` instead — which is why+ a completed-run field like ``main_stuff_name`` can be required here with no+ "not finished yet" escape hatch.+ """model_config = ConfigDict(extra="forbid")output_dict: dict[str, Any]- main_stuff_name: str | None = None+ # A completed run always delivers a main stuff; this is the actual `root` key it lives under.+ main_stuff_name: strpipeline_run_id: strworkflow_id: str | None = None- is_completed: boolgraph_spec_dump: dict[str, Any] | None = None# graph_assembly_error / usage_assembly_error mirror the same fields on PipeOutput: a non-None# value means assembly of the graph / token usage failed, which a host must be able to tell@@ -55,3 +61,19 @@ class PipelexPipeRunOutput(BaseModel):graph_assembly_error: str | None = Nonetokens_usages_dump: list[dict[str, Any]] | None = Noneusage_assembly_error: str | None = None+++class PipelexPipeDispatchAck(BaseModel):+ """JSON-safe acknowledgment of a fire-and-forget dispatch.++ Returned by ``OrchestratorProtocol.start`` once an async-capable orchestrator has+ genuinely enqueued the job: nothing has run yet, so there is no output and no main+ stuff — only the ids a caller needs to track the run. Keeping this a distinct type+ (instead of an "incomplete" ``PipelexPipeRunOutput``) is what lets the completed-run+ payload require ``main_stuff_name`` outright.+ """++ model_config = ConfigDict(extra="forbid")++ pipeline_run_id: str+ workflow_id: strdiff --git a/pipelex/runtime_bridge/primitives/hydration.py b/pipelex/runtime_bridge/primitives/hydration.pyindex 2b06e8804..2e4590c3f 100644--- a/pipelex/runtime_bridge/primitives/hydration.py+++ b/pipelex/runtime_bridge/primitives/hydration.py@@ -5,6 +5,7 @@ from pydantic import ValidationErrorfrom pipelex.core.concepts.concept import Conceptfrom pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.stuffs.composite_content import CompositeContentfrom pipelex.core.stuffs.list_content import ListContentfrom pipelex.core.stuffs.stuff import Stufffrom pipelex.core.stuffs.stuff_content import StuffContent@@ -72,6 +73,27 @@ def _hydrate_list_item(raw_item: dict[str, Any] | str | StuffContent) -> StuffCoraise PipeJobError(msg)+def _hydrate_composite_component(raw_value: Any) -> Any:+ """Hydrate one component of a CompositeContent from its transport encoding.++ ``dump_for_transport`` encodes composite components with the same convention as+ the top level: a list value is a ListContent (marker-stamped item dicts), a dict+ value is a single StuffContent stamped with pipelex-private ``__pipelex_class__``+ / ``__pipelex_module__`` markers. Anything else (plain scalars, marker-less+ legacy payloads, instances already rebuilt by kajson) passes through unchanged —+ CompositeContent is untyped by design, so there is no concept to fall back on.+ """+ if isinstance(raw_value, list):+ raw_items = cast("list[dict[str, Any] | str | StuffContent]", raw_value)+ return ListContent(items=[_hydrate_list_item(raw_item) for raw_item in raw_items])+ if isinstance(raw_value, dict):+ raw_dict = cast("dict[str, Any]", raw_value)+ if "__pipelex_class__" in raw_dict:+ return _hydrate_list_item(raw_dict)+ return raw_dict+ return raw_value++def hydrate_content(raw_content: list[Any] | dict[str, Any] | str, *, concept: Concept) -> StuffContent:"""Hydrate a single StuffContent from a raw value.@@ -100,6 +122,15 @@ def hydrate_content(raw_content: list[Any] | dict[str, Any] | str, *, concept: Citems = [_hydrate_list_item(raw_item) for raw_item in raw_items]return ListContent(items=items)+ if isinstance(raw_content, dict):+ structure_class = get_class_registry().get_class(name=concept.structure_class_name)+ if structure_class is not None and issubclass(structure_class, CompositeContent):+ # Composite components carry their own class markers (extra="allow" fields+ # have no annotations to drive validation) — rebuild each one before+ # validating the composite, so typed access survives transport.+ components = {component_name: _hydrate_composite_component(raw_value) for component_name, raw_value in raw_content.items()}+ return structure_class.model_validate(components)+return StuffContentFactory.make_stuff_content_from_concept_required(concept=concept,value=raw_content,diff --git a/pipelex/runtime_bridge/serialization.py b/pipelex/runtime_bridge/serialization.pyindex fe180d19c..001e3e7f4 100644--- a/pipelex/runtime_bridge/serialization.py+++ b/pipelex/runtime_bridge/serialization.py@@ -69,7 +69,6 @@ def serialize_completed_output(main_stuff_name=main_stuff_name,pipeline_run_id=pipe_output.pipeline_run_id,workflow_id=workflow_id,- is_completed=True,graph_spec_dump=graph_spec_dump,graph_assembly_error=pipe_output.graph_assembly_error,tokens_usages_dump=tokens_usages_dump,@@ -77,13 +76,16 @@ def serialize_completed_output()-def resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str | None:+def resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str:"""Return the actual ``root`` dict key under which the main stuff lives.The main stuff can either sit directly at ``root[MAIN_STUFF_NAME]`` or bereferenced via ``aliases[MAIN_STUFF_NAME]`` pointing at its real name.Callers indexing the output_dict need the actual root key, not thestuff's display ``stuff_name``.++ A completed run always delivers a main stuff — a working memory without+ one at this boundary is a contract violation and raises ``PipeJobError``."""working_memory = pipe_output.working_memoryif MAIN_STUFF_NAME in working_memory.root:@@ -91,4 +93,7 @@ def resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str | None:aliased_target = working_memory.aliases.get(MAIN_STUFF_NAME)if aliased_target is not None and aliased_target in working_memory.root:return aliased_target- return None+ msg = (+ f"Completed run '{pipe_output.pipeline_run_id}' delivered no main stuff in its working memory — a completed run always delivers a main stuff."+ )+ raise PipeJobError(msg)diff --git a/pipelex/system/telemetry/otel_factory.py b/pipelex/system/telemetry/otel_factory.pyindex 1bef5d3c0..df5fe511b 100644--- a/pipelex/system/telemetry/otel_factory.py+++ b/pipelex/system/telemetry/otel_factory.py@@ -102,9 +102,8 @@ class OtelFactory:Returns:JSON string representing the output, potentially truncated."""- main_stuff = pipe_output.working_memory.get_optional_main_stuff()- if main_stuff is None:- return "{}"+ # Called on the success path only — a completed pipe run always delivers a main stuff.+ main_stuff = pipe_output.working_memory.get_main_stuff()output_dict: dict[str, Any] = {"concept": main_stuff.concept.simple_concept_ref,diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_3branch.mthds b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_3branch.mthdsindex aeea60d76..0454affc1 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_3branch.mthds+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_3branch.mthds@@ -34,7 +34,6 @@ description = "Analyze tone, length, and style in parallel with combined output"inputs = { input_text = "Text" }output = "Pg3CombinedResult"add_each_output = true-combined_output = "Pg3CombinedResult"branches = [{ pipe = "pg3_analyze_tone", result = "tone_result" },{ pipe = "pg3_analyze_length", result = "length_result" },diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_add_each.mthds b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_add_each.mthdsindex c4db2eebb..60c2fe80d 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_add_each.mthds+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_add_each.mthds@@ -24,7 +24,7 @@ steps = [type = "PipeParallel"description = "Generate short and detailed summaries in parallel"inputs = { input_text = "Text" }-output = "Text"+output = "Composite"add_each_output = truebranches = [{ pipe = "summarize_short", result = "short_summary" },diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_combined.mthds b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_combined.mthdsindex 10b227417..c0a07998e 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_combined.mthds+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/parallel_graph_combined.mthds@@ -1,5 +1,5 @@domain = "test_parallel_graph_combined"-description = "Test PipeParallel with combined_output wrapped in PipeSequence with follow-up consumer"+description = "Test PipeParallel with a structured combined output wrapped in PipeSequence with follow-up consumer"main_pipe = "pgc_analysis_then_summarize"[concept.PgcToneResult]@@ -29,7 +29,6 @@ description = "Analyze tone and length in parallel with combined output"inputs = { input_text = "Text" }output = "PgcCombinedResult"add_each_output = true-combined_output = "PgcCombinedResult"branches = [{ pipe = "pgc_analyze_tone", result = "tone_result" },{ pipe = "pgc_analyze_length", result = "length_result" },diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_data.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_data.pyindex cfd77ed2b..4a2e42664 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_data.py+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_data.py@@ -4,7 +4,7 @@ from typing import ClassVarclass ParallelCombinedGraphExpectationsBase:- """Base class for PipeParallel graph expectations with combined_output."""+ """Base class for PipeParallel graph expectations with a combined output."""PARALLEL_PIPE_CODE: ClassVar[str]EXPECTED_PIPE_CODES: ClassVar[set[str]]@@ -36,19 +36,20 @@ class ParallelAddEachGraphExpectations:# Expected number of edges by kindEXPECTED_EDGE_COUNTS: ClassVar[dict[str, int]] = {"contains": 4, # sequence->parallel, sequence->combine, parallel->short, parallel->detailed+ "parallel_combine": 2, # short_summary->combined, detailed_summary->combined (always-combine)"data": 2, # parallel->combine (short_summary), parallel->combine (detailed_summary)}class ParallelCombinedGraphExpectations(ParallelCombinedGraphExpectationsBase):- """Expected structure for the parallel_graph_combined graph (PipeSequence wrapping PipeParallel with combined_output)."""+ """Expected structure for the parallel_graph_combined graph (PipeSequence wrapping PipeParallel with a combined output)."""PARALLEL_PIPE_CODE: ClassVar[str] = "pgc_parallel_analysis"# Expected node pipe_codesEXPECTED_PIPE_CODES: ClassVar[set[str]] = {"pgc_analysis_then_summarize", # PipeSequence (outer controller)- "pgc_parallel_analysis", # PipeParallel (parallel controller with combined_output)+ "pgc_parallel_analysis", # PipeParallel (parallel controller with structured combined output)"pgc_analyze_tone", # PipeLLM (branch 1)"pgc_analyze_length", # PipeLLM (branch 2)"pgc_summarize_combined", # PipeLLM (downstream consumer of combined result)@@ -79,7 +80,7 @@ class Parallel3BranchGraphExpectations(ParallelCombinedGraphExpectationsBase):# Expected node pipe_codesEXPECTED_PIPE_CODES: ClassVar[set[str]] = {"pg3_sequence", # PipeSequence (outer controller)- "pg3_parallel", # PipeParallel (3-branch parallel with combined_output)+ "pg3_parallel", # PipeParallel (3-branch parallel with combined output)"pg3_analyze_tone", # PipeLLM (branch 1)"pg3_analyze_length", # PipeLLM (branch 2)"pg3_analyze_style", # PipeLLM (branch 3 - unused downstream)diff --git a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_graph.py b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_graph.pyindex 221ed96ad..1c0ca19eb 100644--- a/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_graph.py+++ b/tests/e2e/pipelex/pipes/pipe_controller/pipe_parallel/test_pipe_parallel_graph.py@@ -178,7 +178,7 @@ class TestPipeParallelGraph:pipe_code: str,expectations_class: type[ParallelCombinedGraphExpectationsBase],):- """Verify PipeParallel with combined_output generates correct graph structure.+ """Verify PipeParallel with a combined output generates correct graph structure.Parametrized with:- pgc_analysis_then_summarize: 2-branch PipeParallel wrapped in PipeSequence with follow-up consumer@@ -291,4 +291,4 @@ class TestPipeParallelGraph:title=f"Parallel Combined Graph Outputs ({pipe_code})",)- log.info(f"Structural validation passed: {pipe_code} combined_output graph is correct")+ log.info(f"Structural validation passed: {pipe_code} combined-output graph is correct")diff --git a/tests/e2e/pipelex/pipes/pipe_operators/pipe_compose/cv_job_match.mthds b/tests/e2e/pipelex/pipes/pipe_operators/pipe_compose/cv_job_match.mthdsindex 09828b1d7..158b1e19e 100644--- a/tests/e2e/pipelex/pipes/pipe_operators/pipe_compose/cv_job_match.mthds+++ b/tests/e2e/pipelex/pipes/pipe_operators/pipe_compose/cv_job_match.mthds@@ -28,7 +28,7 @@ steps = [type = "PipeParallel"description = "Extracts text content from both the CV and job offer PDFs concurrently"inputs = { cv_pdf = "Document", job_offer_pdf = "Document" }-output = "Page[]"+output = "Composite"branches = [{ pipe = "extract_cv", result = "cv_pages" },{ pipe = "extract_job_offer", result = "job_offer_pages" },@@ -53,7 +53,7 @@ model = "@default-text-from-pdf"type = "PipeParallel"description = "Analyzes both the CV and job offer documents concurrently to extract structured information"inputs = { cv_pages = "Page", job_offer_pages = "Page" }-output = "Text"+output = "Composite"branches = [{ pipe = "analyze_cv", result = "cv_analysis" },{ pipe = "analyze_job_offer", result = "job_requirements" },diff --git a/tests/integration/pipelex/cli/agent_cli/test_run_cost_report.py b/tests/integration/pipelex/cli/agent_cli/test_run_cost_report.pyindex 74d652158..734b0a1cd 100644--- a/tests/integration/pipelex/cli/agent_cli/test_run_cost_report.py+++ b/tests/integration/pipelex/cli/agent_cli/test_run_cost_report.py@@ -14,13 +14,32 @@ from pipelex.cli.agent_cli.commands.run._run_core import run_pipeline_core # nofrom pipelex.cogt.llm.llm_report import LLMTokensUsagefrom pipelex.cogt.usage.cost_category import CostCategoryfrom pipelex.cogt.usage.token_category import TokenCategory-from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.concepts.concept import Concept+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemoryfrom pipelex.core.pipes.pipe_output import PipeOutput+from pipelex.core.stuffs.stuff import Stuff+from pipelex.core.stuffs.text_content import TextContentfrom pipelex.pipeline.job_metadata import JobMetadata_RUN_CORE_MODULE = "pipelex.cli.agent_cli.commands.run._run_core"+def _memory_with_main_stuff() -> WorkingMemory:+ """A completed run always delivers a main stuff — the mocked runner output must honor that."""+ stuff = Stuff(+ stuff_code="main-code",+ stuff_name="result",+ concept=Concept(+ code="Text",+ domain_code="native",+ description="Plain text",+ structure_class_name="TextContent",+ ),+ content=TextContent(text="the result"),+ )+ return WorkingMemory(root={"result": stuff}, aliases={MAIN_STUFF_NAME: "result"})++def _non_zero_usage(job_metadata: JobMetadata) -> LLMTokensUsage:return LLMTokensUsage(job_metadata=job_metadata,@@ -63,7 +82,7 @@ class TestAgentRunCostReport:async def test_cost_report_in_with_memory_json(self, mocker: MockerFixture, job_metadata: JobMetadata, tmp_path: Path) -> None:pipe_output = PipeOutput(- working_memory=WorkingMemory(),+ working_memory=_memory_with_main_stuff(),pipeline_run_id="agent-run",tokens_usages=[_non_zero_usage(job_metadata)],)@@ -83,7 +102,7 @@ class TestAgentRunCostReport:async def test_cost_report_for_free_model_run(self, mocker: MockerFixture, job_metadata: JobMetadata, tmp_path: Path) -> None:"""A free/zero-price model with real tokens still surfaces a cost_report (total_cost 0)."""pipe_output = PipeOutput(- working_memory=WorkingMemory(),+ working_memory=_memory_with_main_stuff(),pipeline_run_id="agent-free",tokens_usages=[_free_model_usage(job_metadata)],)@@ -102,7 +121,7 @@ class TestAgentRunCostReport:async def test_no_cost_report_for_dry_run(self, mocker: MockerFixture, job_metadata: JobMetadata, tmp_path: Path) -> None:pipe_output = PipeOutput(- working_memory=WorkingMemory(),+ working_memory=_memory_with_main_stuff(),pipeline_run_id="agent-dry",tokens_usages=[_zero_cost_usage(job_metadata)],)diff --git a/tests/integration/pipelex/cli/test_agent_validate_pipe_in_bundle.py b/tests/integration/pipelex/cli/test_agent_validate_pipe_in_bundle.pyindex 3bc852529..dd3576f07 100644--- a/tests/integration/pipelex/cli/test_agent_validate_pipe_in_bundle.py+++ b/tests/integration/pipelex/cli/test_agent_validate_pipe_in_bundle.py@@ -59,9 +59,8 @@ prompt = "Summarize $doc"type = "PipeParallel"description = "Parallel referencing an unloaded cross-package branch."inputs = { doc = "XpkgDoc" }-output = "Text"+output = "Composite"add_each_output = true-combined_output = "Text"branches = [{ pipe = "ext->otherpkg.missing_pipe", result = "branch_result" },]diff --git a/tests/integration/pipelex/pipeline/test_bundle_validator.py b/tests/integration/pipelex/pipeline/test_bundle_validator.pyindex 5b6ac6f45..b7f75dff2 100644--- a/tests/integration/pipelex/pipeline/test_bundle_validator.py+++ b/tests/integration/pipelex/pipeline/test_bundle_validator.py@@ -100,9 +100,8 @@ prompt = "Summarize $doc"type = "PipeParallel"description = "Parallel referencing an unloaded cross-package branch"inputs = {{ doc = "Doc" }}-output = "Text"+output = "Composite"add_each_output = true-combined_output = "Text"branches = [{{ pipe = "ext->otherpkg.missing_pipe", result = "branch_result" }},]diff --git a/tests/integration/pipelex/pipeline/test_mock_usage_direct.py b/tests/integration/pipelex/pipeline/test_mock_usage_direct.pyindex de540fca2..edc2fc6a2 100644--- a/tests/integration/pipelex/pipeline/test_mock_usage_direct.py+++ b/tests/integration/pipelex/pipeline/test_mock_usage_direct.py@@ -111,7 +111,7 @@ class TestMockUsageDirect:aggregated = CostRegistry.aggregate_costs(pipe_output.tokens_usages)assert aggregated.has_reportable_usage# The structured output was produced by the mock object builder (a Topic with a name).- main_stuff = pipe_output.working_memory.get_optional_main_stuff()+ main_stuff = pipe_output.working_memory.get_main_stuff()assert main_stuff is not Noneasync def test_cost_report_renders_for_mock_usage(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:diff --git a/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.pynew file mode 100644index 000000000..77cf750e3--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_condition/test_pipe_condition_continue_delivery.py@@ -0,0 +1,132 @@+"""PipeCondition's `continue` outcome delivers the current main stuff by passing it through.++A pipe run always delivers a main stuff. For `continue`, the delivered main stuff is the one+already in the working memory (pass-through). When there is nothing to pass through (e.g. the+condition is the entry pipe and memory holds only named inputs), the run must fail loudly at the+pipe level with an actionable error — not crash downstream (graph tracer, delivery, telemetry)+with a bare WorkingMemoryStuffNotFoundError.+"""++from pathlib import Path+from typing import Callable++import pytest++from pipelex.config import get_config+from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.domains.domain import SpecialDomain+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.pipes.pipe_factory import PipeFactory+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.core.stuffs.text_content import TextContent+from pipelex.pipe_controllers.condition.pipe_condition import PipeCondition+from pipelex.pipe_controllers.condition.pipe_condition_blueprint import PipeConditionBlueprint+from pipelex.pipe_controllers.condition.special_outcome import SpecialOutcome+from pipelex.pipe_run.exceptions import PipeRunError+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipe_run.pipe_run_params import PipeRunParams+from pipelex.pipeline.job_metadata import JobMetadata+++def _make_continue_only_condition() -> PipeCondition:+ """A condition whose every outcome is `continue` (a pure pass-through gate)."""+ blueprint = PipeConditionBlueprint(+ description="Pass-through gate for continue-delivery tests",+ inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},+ output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",+ expression_template="{% if input_text.text %}skip{% else %}skip{% endif %}",+ outcomes={"skip": SpecialOutcome.CONTINUE},+ default_outcome=SpecialOutcome.CONTINUE,+ )+ return PipeFactory[PipeCondition].make_from_blueprint(+ domain_code="test_integration",+ pipe_code="continue_gate",+ blueprint=blueprint,+ )+++def _make_input_text_stuff():+ return StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),+ content=TextContent(text="hello"),+ name="input_text",+ )+++def _make_run_params(run_mode: PipeRunMode) -> PipeRunParams:+ # Constructed directly (not via the factory) so a keyless boot's forced-DRY coercion cannot+ # silently swap which controller code path (live vs dry) the test exercises.+ return PipeRunParams(run_mode=run_mode, pipe_stack_limit=get_config().pipelex.pipe_run_config.pipe_stack_limit)++[email protected](loop_scope="class")+class TestPipeConditionContinueDelivery:+ async def test_continue_with_main_stuff_passes_it_through(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):+ """With a main stuff in memory, `continue` delivers it unchanged."""+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])+ pipe_condition = _make_continue_only_condition()+ input_text_stuff = _make_input_text_stuff()+ working_memory = WorkingMemoryFactory.make_from_single_stuff(input_text_stuff)++ pipe_output = await pipe_condition.run_pipe(+ job_metadata=job_metadata,+ working_memory=working_memory,+ pipe_run_params=_make_run_params(PipeRunMode.LIVE),+ )++ assert pipe_output.main_stuff.stuff_code == input_text_stuff.stuff_code+ # Run identity must survive the pass-through: the continue path builds its own PipeOutput,+ # so it must stamp pipeline_run_id like every other controller — otherwise the blocking+ # bridge serializes SpecialPipelineId.UNTITLED and callers track the wrong run.+ assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_id++ async def test_continue_without_main_stuff_fails_loud(self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]):+ """With nothing to pass through, `continue` is a clear PipeRunError at the pipe level."""+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])+ pipe_condition = _make_continue_only_condition()+ working_memory = WorkingMemoryFactory.make_empty()+ working_memory.set_stuff(name="input_text", stuff=_make_input_text_stuff())++ with pytest.raises(PipeRunError) as exc_info:+ await pipe_condition.run_pipe(+ job_metadata=job_metadata,+ working_memory=working_memory,+ pipe_run_params=_make_run_params(PipeRunMode.LIVE),+ )+ assert "continue_gate" in str(exc_info.value)+ assert "main stuff" in str(exc_info.value)++ async def test_dry_run_of_special_outcomes_only_condition_without_main_stuff_fails_loud(+ self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]+ ):+ """Dry-run parity: an all-special-outcomes condition with no pre-existing main stuff cannot deliver."""+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])+ pipe_condition = _make_continue_only_condition()+ working_memory = WorkingMemoryFactory.make_empty()+ working_memory.set_stuff(name="input_text", stuff=_make_input_text_stuff())++ with pytest.raises(PipeRunError):+ await pipe_condition.run_pipe(+ job_metadata=job_metadata,+ working_memory=working_memory,+ pipe_run_params=_make_run_params(PipeRunMode.DRY),+ )++ async def test_dry_run_continue_with_main_stuff_preserves_run_id(+ self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]+ ):+ """Dry-run parity path (all-special-outcomes condition with a pre-existing main stuff) must also+ stamp pipeline_run_id, like the live continue path — the dry-run PipeOutput is built directly too.+ """+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_condition")])+ pipe_condition = _make_continue_only_condition()+ working_memory = WorkingMemoryFactory.make_from_single_stuff(_make_input_text_stuff())++ pipe_output = await pipe_condition.run_pipe(+ job_metadata=job_metadata,+ working_memory=working_memory,+ pipe_run_params=_make_run_params(PipeRunMode.DRY),+ )++ assert pipe_output.pipeline_run_id == job_metadata.pipeline_run_iddiff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/parallel_always_combine.mthds b/tests/integration/pipelex/pipes/controller/pipe_parallel/parallel_always_combine.mthdsnew file mode 100644index 000000000..f113f16b6--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/parallel_always_combine.mthds@@ -0,0 +1,87 @@+domain = "test_parallel_always_combine"+description = "Fixtures for PipeParallel always-combine semantics"+main_pipe = "pac_terminal_parallel"++[concept.PacDraft]+description = "A draft text"+refines = "Text"++[concept.PacToneResult]+description = "Result of tone analysis"+refines = "Text"++[concept.PacLengthResult]+description = "Result of length analysis"+refines = "Text"++[pipe.pac_terminal_parallel]+type = "PipeParallel"+description = "Add-each-only parallel combining its branch outputs into a Composite"+inputs = { input_text = "Text" }+output = "Composite"+add_each_output = true+branches = [+ { pipe = "pac_analyze_tone", result = "tone_result" },+ { pipe = "pac_analyze_length", result = "length_result" },+]++[pipe.pac_sequence_ending_in_parallel]+type = "PipeSequence"+description = "Sequence whose LAST step is an add-each-only parallel (stale-stamp regression)"+inputs = { input_text = "Text" }+output = "Composite"+steps = [+ { pipe = "pac_draft", result = "draft" },+ { pipe = "pac_parallel_on_draft", result = "combo" },+]++[pipe.pac_parallel_on_draft]+type = "PipeParallel"+description = "Parallel analyzing the draft, exposing branch outputs and combining them"+inputs = { draft = "PacDraft" }+output = "Composite"+add_each_output = true+branches = [+ { pipe = "pac_analyze_draft_tone", result = "tone_result" },+ { pipe = "pac_analyze_draft_length", result = "length_result" },+]++[pipe.pac_draft]+type = "PipeLLM"+description = "Write a short draft from the input text"+inputs = { input_text = "Text" }+output = "PacDraft"+model = "$testing-text"+prompt = "Write a one-sentence draft about: $input_text.text"++[pipe.pac_analyze_tone]+type = "PipeLLM"+description = "Analyze the tone of the text"+inputs = { input_text = "Text" }+output = "PacToneResult"+model = "$testing-text"+prompt = "Describe the tone of: $input_text.text"++[pipe.pac_analyze_length]+type = "PipeLLM"+description = "Analyze the length of the text"+inputs = { input_text = "Text" }+output = "PacLengthResult"+model = "$testing-text"+prompt = "Describe the length characteristics of: $input_text.text"++[pipe.pac_analyze_draft_tone]+type = "PipeLLM"+description = "Analyze the tone of the draft"+inputs = { draft = "PacDraft" }+output = "PacToneResult"+model = "$testing-text"+prompt = "Describe the tone of: $draft.text"++[pipe.pac_analyze_draft_length]+type = "PipeLLM"+description = "Analyze the length of the draft"+inputs = { draft = "PacDraft" }+output = "PacLengthResult"+model = "$testing-text"+prompt = "Describe the length characteristics of: $draft.text"diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel_1.mthds b/tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel_1.mthdsindex 8ae71ad97..e434be445 100644--- a/tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel_1.mthds+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/pipe_parallel_1.mthds@@ -14,7 +14,6 @@ description = "Analyze document length and content in parallel"inputs = { document = "DocumentInput" }output = "CombinedAnalysis"add_each_output = true-combined_output = "CombinedAnalysis"branches = [{ pipe = "analyze_length", result = "length_result" },{ pipe = "analyze_content", result = "content_result" },diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_always_combines.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_always_combines.pynew file mode 100644index 000000000..2f0c8addd--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_always_combines.py@@ -0,0 +1,79 @@+"""PipeParallel always combines its branch outputs into its declared output and stamps it as main stuff.++Covers the two failure modes of the old `combined_output`-gated behavior:+- stale stamp: a sequence ending in an add-each-only parallel used to silently deliver the previous+ step's stuff as main stuff;+- missing main stuff: an add-each-only parallel as top-level pipe (API path, memory built from+ pipeline inputs) used to complete with no main stuff at all.++Dry-run parity is covered by the pipe_run_mode parametrization: the same shape assertions run in+both live and dry modes.+"""++import pytest++from pipelex.core.stuffs.composite_content import CompositeContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipeline.runner import PipelexMTHDSProtocol++_LIBRARY_DIR = "tests/integration/pipelex/pipes/controller/pipe_parallel"++[email protected]_runnable[email protected](loop_scope="class")+class TestPipeParallelAlwaysCombines:+ async def test_terminal_parallel_delivers_main_stuff(self, pipe_run_mode: PipeRunMode):+ """An add-each-only parallel as top-level pipe must deliver the combined composite as main stuff."""+ runner = PipelexMTHDSProtocol(+ library_dirs=[_LIBRARY_DIR],+ pipe_run_mode=pipe_run_mode,+ )+ response = await runner.execute(+ pipe_code="pac_terminal_parallel",+ inputs={"input_text": TextContent(text="The quick brown fox jumps over the lazy dog.")},+ )+ pipe_output = response.pipe_output++ main_stuff = pipe_output.main_stuff+ assert main_stuff is not None+ assert main_stuff.concept.code == "Composite"+ assert isinstance(main_stuff.content, CompositeContent)+ combined_dump = main_stuff.content.smart_dump()+ assert set(combined_dump.keys()) == {"tone_result", "length_result"}++ # add_each_output still exposes the branch outputs by name+ working_memory = pipe_output.working_memory+ assert working_memory.get_stuff("tone_result") is not None+ assert working_memory.get_stuff("length_result") is not None++ async def test_sequence_ending_in_parallel_delivers_composite_not_stale_stuff(self, pipe_run_mode: PipeRunMode):+ """Stale-stamp regression: the sequence's main stuff must be the parallel's combined composite,+ not the previous step's output.+ """+ runner = PipelexMTHDSProtocol(+ library_dirs=[_LIBRARY_DIR],+ pipe_run_mode=pipe_run_mode,+ )+ response = await runner.execute(+ pipe_code="pac_sequence_ending_in_parallel",+ inputs={"input_text": TextContent(text="A tale of two cities, in one sentence.")},+ )+ pipe_output = response.pipe_output++ main_stuff = pipe_output.main_stuff+ assert main_stuff is not None+ assert main_stuff.concept.code == "Composite", (+ f"Main stuff must be the parallel's combined output, not the previous step's stuff (got concept '{main_stuff.concept.concept_ref}')"+ )+ assert isinstance(main_stuff.content, CompositeContent)+ combined_dump = main_stuff.content.smart_dump()+ assert set(combined_dump.keys()) == {"tone_result", "length_result"}++ # The combined stuff is stored under the step's result name+ working_memory = pipe_output.working_memory+ combo_stuff = working_memory.get_stuff("combo")+ assert combo_stuff is not None+ assert combo_stuff.stuff_code == main_stuff.stuff_codediff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_branch_type_validation.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_branch_type_validation.pynew file mode 100644index 000000000..1a01c0705--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_branch_type_validation.py@@ -0,0 +1,113 @@+from typing import Callable++import pytest+from pydantic import Field++from pipelex.core.concepts.concept_blueprint import ConceptBlueprint+from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.pipes.exceptions import PipeValidationError+from pipelex.core.pipes.pipe_factory import PipeFactory+from pipelex.core.stuffs.number_content import NumberContent+from pipelex.core.stuffs.structured_content import StructuredContent+from pipelex.core.stuffs.text_content import TextContent+from pipelex.hub import get_class_registry, get_concept_library, get_pipe_library+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint+from pipelex.pipe_controllers.sub_pipe_blueprint import SubPipeBlueprint+from pipelex.pipe_operators.llm.pipe_llm import PipeLLM+from pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprint+++class BtvCombinedResult(StructuredContent):+ """Structured combination target with content-typed fields."""++ summary: TextContent = Field(description="Summary of the analysis")+ score: NumberContent = Field(description="Numeric score of the analysis")+++DOMAIN_CODE = "test_btv"+++class TestPipeParallelBranchTypeValidation:+ """validate_output_with_library must check branch output TYPES against structured fields, not just names."""++ def _setup_library(self, *, score_branch_output: str) -> PipeParallel:+ """Build a library with a parallel whose 'score' branch outputs the given concept ref."""+ get_class_registry().register_class(BtvCombinedResult)++ concept_library = get_concept_library()+ combined_concept = ConceptFactory.make(+ concept_code="BtvCombinedResult",+ domain_code=DOMAIN_CODE,+ description="Combined analysis result",+ structure_class_name="BtvCombinedResult",+ )+ refined_text_concept = ConceptFactory.make_from_blueprint(+ domain_code=DOMAIN_CODE,+ concept_code="BtvSummary",+ blueprint_or_string_description=ConceptBlueprint(description="A summary", refines="Text"),+ )+ refined_number_concept = ConceptFactory.make_from_blueprint(+ domain_code=DOMAIN_CODE,+ concept_code="BtvScore",+ blueprint_or_string_description=ConceptBlueprint(description="A score", refines="Number"),+ )+ concept_library.add_concepts(concepts=[combined_concept, refined_text_concept, refined_number_concept])++ pipe_library = get_pipe_library()+ summary_pipe = PipeFactory[PipeLLM].make_from_blueprint(+ domain_code=DOMAIN_CODE,+ pipe_code="btv_summarize",+ blueprint=PipeLLMBlueprint(+ description="Summarize the input",+ inputs={"input_text": "Text"},+ output=f"{DOMAIN_CODE}.BtvSummary",+ prompt="Summarize: $input_text",+ ),+ concept_codes_from_the_same_domain=["BtvSummary", "BtvScore", "BtvCombinedResult"],+ )+ score_pipe = PipeFactory[PipeLLM].make_from_blueprint(+ domain_code=DOMAIN_CODE,+ pipe_code="btv_scoring",+ blueprint=PipeLLMBlueprint(+ description="Score the input",+ inputs={"input_text": "Text"},+ output=score_branch_output,+ prompt="Score: $input_text",+ ),+ concept_codes_from_the_same_domain=["BtvSummary", "BtvScore", "BtvCombinedResult"],+ )+ pipe_library.add_new_pipe(pipe=summary_pipe)+ pipe_library.add_new_pipe(pipe=score_pipe)++ pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(+ domain_code=DOMAIN_CODE,+ pipe_code="btv_parallel",+ blueprint=PipeParallelBlueprint(+ description="Parallel combining summary and score",+ inputs={"input_text": "Text"},+ output=f"{DOMAIN_CODE}.BtvCombinedResult",+ branches=[+ SubPipeBlueprint(pipe="btv_summarize", result="summary"),+ SubPipeBlueprint(pipe="btv_scoring", result="score"),+ ],+ ),+ concept_codes_from_the_same_domain=["BtvSummary", "BtvScore", "BtvCombinedResult"],+ )+ pipe_library.add_new_pipe(pipe=pipe_parallel)+ return pipe_parallel++ def test_compatible_branch_types_validate_clean(self, load_empty_library: Callable[[], str]):+ """Refinement is conceptual compatibility: refines-Text fits TextContent, refines-Number fits NumberContent."""+ load_empty_library()+ pipe_parallel = self._setup_library(score_branch_output=f"{DOMAIN_CODE}.BtvScore")++ pipe_parallel.validate_output_with_library()++ def test_incompatible_branch_type_raises(self, load_empty_library: Callable[[], str]):+ """A branch producing text into a NumberContent-typed field must fail /validate, not the runtime combine."""+ load_empty_library()+ pipe_parallel = self._setup_library(score_branch_output=f"{DOMAIN_CODE}.BtvSummary")++ with pytest.raises(PipeValidationError, match="score"):+ pipe_parallel.validate_output_with_library()diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_final_stuff_code.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_final_stuff_code.pynew file mode 100644index 000000000..43f912073--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_final_stuff_code.py@@ -0,0 +1,77 @@+"""PipeParallel honors a requested final_stuff_code on its combined output stuff.++When a PipeParallel runs as the branch pipe of a PipeBatch, the batch pre-allocates a stuff code+for each branch's final output so the aggregated list items keep their graph identity. The parallel+must stamp that code on its COMBINED stuff (like operators do on theirs), not discard it — and must+NOT leak it onto the branch outputs.+"""++from pathlib import Path+from typing import Callable++import pytest++from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.domains.domain import SpecialDomain+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.pipes.pipe_factory import PipeFactory+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.core.stuffs.text_content import TextContent+from pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel+from pipelex.pipe_controllers.parallel.pipe_parallel_blueprint import PipeParallelBlueprint+from pipelex.pipe_controllers.sub_pipe_factory import SubPipeBlueprint+from pipelex.pipe_run.pipe_run_mode import PipeRunMode+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory+from pipelex.pipeline.job_metadata import JobMetadata++_REQUESTED_CODE = "requested-final-stuff-code"++[email protected](loop_scope="class")+class TestPipeParallelFinalStuffCode:+ async def test_combined_stuff_carries_requested_final_stuff_code(+ self, job_metadata: JobMetadata, load_test_library: Callable[[list[Path]], None]+ ):+ """A dry run is enough: stamping the combined stuff is mode-independent (shared combine helper)."""+ load_test_library([Path("tests/integration/pipelex/pipes/controller/pipe_parallel")])+ pipe_parallel_blueprint = PipeParallelBlueprint(+ description="Parallel honoring final_stuff_code",+ inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},+ output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.COMPOSITE}",+ branches=[+ SubPipeBlueprint(pipe="analyze_sentiment", result="sentiment_result"),+ SubPipeBlueprint(pipe="count_words", result="word_count_result"),+ ],+ add_each_output=True,+ )+ pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(+ domain_code="test_integration",+ pipe_code="parallel_with_final_code",+ blueprint=pipe_parallel_blueprint,+ )++ input_text_stuff = StuffFactory.make_stuff(+ concept=ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.TEXT),+ content=TextContent(text="A short text to analyze."),+ name="input_text",+ )+ working_memory = WorkingMemoryFactory.make_from_single_stuff(input_text_stuff)++ pipe_run_params = PipeRunParamsFactory.make_run_params(pipe_run_mode=PipeRunMode.DRY)+ pipe_run_params.final_stuff_code = _REQUESTED_CODE++ pipe_output = await pipe_parallel.run_pipe(+ job_metadata=job_metadata,+ working_memory=working_memory,+ output_name="parallel_results",+ pipe_run_params=pipe_run_params,+ )++ main_stuff = pipe_output.main_stuff+ assert main_stuff.stuff_code == _REQUESTED_CODE, "the combined stuff must carry the requested final_stuff_code"++ # The requested code applies to the combined stuff only — branch outputs keep their own codes.+ final_working_memory = pipe_output.working_memory+ assert final_working_memory.get_stuff("sentiment_result").stuff_code != _REQUESTED_CODE+ assert final_working_memory.get_stuff("word_count_result").stuff_code != _REQUESTED_CODEdiff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_output_validation.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_output_validation.pynew file mode 100644index 000000000..7f2a9da2f--- /dev/null+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_output_validation.py@@ -0,0 +1,138 @@+"""Static validation of a PipeParallel's declared output against its branch result names.++The declared output must be `Composite` or a structured concept whose fields are compatible with+the branch result names (required fields ⊆ result names; result names ⊆ declared fields). This is+enforced at library validation time so `/validate` surfaces it as an author-time error instead of+a runtime combine failure.+"""++import pytest++from pipelex.core.pipes.exceptions import PipeValidationError+from pipelex.hub import clear_current_library, get_library_manager+from pipelex.pipeline.execution_seams import acquire_library++_BUNDLE_HEADER = """+domain = "pv_parallel_output"+description = "Static validation fixtures for PipeParallel output"++[concept.PvItem]+description = "An item to analyze"++[pipe.pv_branch_tone]+type = "PipeLLM"+description = "Tone branch"+inputs = { item = "PvItem" }+output = "Text"+prompt = "Describe the tone of $item"++[pipe.pv_branch_length]+type = "PipeLLM"+description = "Length branch"+inputs = { item = "PvItem" }+output = "Text"+prompt = "Describe the length of $item"+"""++_PARALLEL_TEMPLATE = """+[pipe.pv_parallel]+type = "PipeParallel"+description = "Parallel under test"+inputs = {{ item = "PvItem" }}+output = "{output}"+branches = [+ {{ pipe = "pv_branch_tone", result = "tone_result" }},+ {{ pipe = "pv_branch_length", result = "length_result" }},+]+"""++_CONCEPT_EXACT_MATCH = """+[concept.PvCombo]+description = "Combined results"++[concept.PvCombo.structure]+tone_result = { type = "text", description = "Tone", required = true }+length_result = { type = "text", description = "Length", required = true }+"""++_CONCEPT_WITH_OPTIONAL_EXTRA = """+[concept.PvComboLoose]+description = "Combined results with an optional extra field"++[concept.PvComboLoose.structure]+tone_result = { type = "text", description = "Tone", required = true }+length_result = { type = "text", description = "Length", required = true }+notes = { type = "text", description = "Optional notes", required = false }+"""++_CONCEPT_MISSING_REQUIRED = """+[concept.PvComboStrict]+description = "Combined results requiring a field no branch produces"++[concept.PvComboStrict.structure]+tone_result = { type = "text", description = "Tone", required = true }+length_result = { type = "text", description = "Length", required = true }+verdict = { type = "text", description = "Required field no branch produces", required = true }+"""++_CONCEPT_MISSING_DECLARED_FIELD = """+[concept.PvComboNarrow]+description = "Combined results lacking a field for one branch result"++[concept.PvComboNarrow.structure]+tone_result = { type = "text", description = "Tone", required = true }+"""++_CONCEPT_DESCRIPTION_ONLY = """+[concept.PvComboImplicit]+description = "A description-only concept (implicitly text-shaped)"+"""+++def _bundle(*, output: str, extra_concepts: str = "") -> str:+ return _BUNDLE_HEADER + extra_concepts + _PARALLEL_TEMPLATE.format(output=output)+++class TestPipeParallelOutputValidation:+ @pytest.mark.parametrize(+ ("test_id", "mthds_content"),+ [+ ("composite_output", _bundle(output="Composite")),+ ("structured_exact_match", _bundle(output="PvCombo", extra_concepts=_CONCEPT_EXACT_MATCH)),+ ("structured_with_optional_extra", _bundle(output="PvComboLoose", extra_concepts=_CONCEPT_WITH_OPTIONAL_EXTRA)),+ ],+ )+ def test_accepted_outputs(self, test_id: str, mthds_content: str):+ library_manager = get_library_manager()+ library_id = f"pv_accept_{test_id}"+ acquire_library(library_id=library_id, mthds_contents=[mthds_content])+ try:+ pass # loading without raising IS the assertion+ finally:+ library_manager.teardown(library_id=library_id)+ clear_current_library()++ @pytest.mark.parametrize(+ ("test_id", "mthds_content", "expected_fragment"),+ [+ (+ "missing_required_field",+ _bundle(output="PvComboStrict", extra_concepts=_CONCEPT_MISSING_REQUIRED),+ "verdict",+ ),+ (+ "result_name_not_declared",+ _bundle(output="PvComboNarrow", extra_concepts=_CONCEPT_MISSING_DECLARED_FIELD),+ "length_result",+ ),+ (+ "description_only_concept",+ _bundle(output="PvComboImplicit", extra_concepts=_CONCEPT_DESCRIPTION_ONLY),+ "text",+ ),+ ],+ )+ def test_rejected_outputs(self, test_id: str, mthds_content: str, expected_fragment: str):+ with pytest.raises(PipeValidationError) as exc_info:+ acquire_library(library_id=f"pv_reject_{test_id}", mthds_contents=[mthds_content])+ assert expected_fragment in str(exc_info.value)diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.pyindex f06da6405..b874939d8 100644--- a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_simple.py@@ -8,6 +8,7 @@ from pipelex.core.concepts.native.concept_native import NativeConceptCodefrom pipelex.core.domains.domain import SpecialDomainfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactory+from pipelex.core.stuffs.composite_content import CompositeContentfrom pipelex.core.stuffs.stuff_factory import StuffFactoryfrom pipelex.core.stuffs.text_content import TextContentfrom pipelex.pipe_controllers.parallel.pipe_parallel import PipeParallel@@ -31,14 +32,13 @@ class TestPipeParallelSimple:pipe_parallel_blueprint = PipeParallelBlueprint(description="Parallel text analysis pipeline",inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},- output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",+ output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.COMPOSITE}",branches=[SubPipeBlueprint(pipe="analyze_sentiment", result="sentiment_result"),SubPipeBlueprint(pipe="count_words", result="word_count_result"),SubPipeBlueprint(pipe="extract_keywords", result="keywords_result"),],add_each_output=True,- combined_output=None,)pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(@@ -61,7 +61,6 @@ class TestPipeParallelSimple:assert pipe_parallel.code == "parallel_text_analyzer"assert len(pipe_parallel.parallel_sub_pipes) == 3assert pipe_parallel.add_each_output is True- assert pipe_parallel.combined_output is None# Verify sub-pipes configurationassert pipe_parallel.parallel_sub_pipes[0].pipe_code == "analyze_sentiment"@@ -90,9 +89,9 @@ class TestPipeParallelSimple:assert pipe_output.working_memory is not Noneassert pipe_output.main_stuff is not None- # Verify working memory structure - should have original input + 3 parallel results+ # Verify working memory structure - original input + 3 parallel results + combined outputfinal_working_memory = pipe_output.working_memory- assert len(final_working_memory.root) == 4 # original input + 3 parallel outputs+ assert len(final_working_memory.root) == 5# Original input should still be thereoriginal_input = final_working_memory.get_stuff("input_text")@@ -136,10 +135,11 @@ class TestPipeParallelSimple:assert sentiment_result.content.text != keywords_result.content.textassert word_count_result.content.text != keywords_result.content.text- # Verify the main stuff points to the input (since no combined_output is set)+ # The parallel always combines: the main stuff is the composite of the branch outputsfinal_result = pipe_output.main_stuff- assert isinstance(final_result.content, TextContent)- assert final_result.content.text == "The weather is beautiful today. I love sunny days and outdoor activities."+ assert final_result.concept.code == "Composite"+ assert isinstance(final_result.content, CompositeContent)+ assert set(final_result.content.components.keys()) == {"sentiment_result", "word_count_result", "keywords_result"}async def test_parallel_short_text_analysis(self, job_metadata: JobMetadata, pipe_run_mode: PipeRunMode, load_test_library: Callable[[list[Path]], None]@@ -150,14 +150,13 @@ class TestPipeParallelSimple:pipe_parallel_blueprint = PipeParallelBlueprint(description="Parallel text analysis pipeline for short text",inputs={"input_text": f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}"},- output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.TEXT}",+ output=f"{SpecialDomain.NATIVE}.{NativeConceptCode.COMPOSITE}",branches=[SubPipeBlueprint(pipe="analyze_sentiment", result="sentiment_result"),SubPipeBlueprint(pipe="count_words", result="word_count_result"),SubPipeBlueprint(pipe="extract_keywords", result="keywords_result"),],add_each_output=True,- combined_output=None,)pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(@@ -186,9 +185,9 @@ class TestPipeParallelSimple:assert pipe_output is not Noneassert pipe_output.working_memory is not None- # Verify working memory structure+ # Verify working memory structure - original input + 3 parallel results + combined outputfinal_working_memory = pipe_output.working_memory- assert len(final_working_memory.root) == 4 # original input + 3 parallel outputs+ assert len(final_working_memory.root) == 5# Original input should still be thereoriginal_input = final_working_memory.get_stuff("input_text")diff --git a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.pyindex 1ddd7e105..d4d0fdf9a 100644--- a/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py+++ b/tests/integration/pipelex/pipes/controller/pipe_parallel/test_pipe_parallel_validation.py@@ -73,7 +73,6 @@ class TestPipeParallelValidation:output=ConceptFactory.make_concept_ref_with_domain(domain_code=domain_code, concept_code=concept_3.code),branches=[SubPipeBlueprint(pipe=real_pipe.code, result="analysis_result")],add_each_output=True,- combined_output=None,)pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(@@ -125,7 +124,6 @@ class TestPipeParallelValidation:output=ConceptFactory.make_concept_ref_with_domain(domain_code=domain_code, concept_code=concept_3.code),branches=[SubPipeBlueprint(pipe="test_pipe_1", result="result_1")],add_each_output=True,- combined_output=None,)pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(@@ -143,7 +141,6 @@ class TestPipeParallelValidation:assert pipe_parallel.output.concept.code == concept_3.codeassert pipe_parallel.output.concept.domain_code == domain_codeassert pipe_parallel.add_each_output is True- assert pipe_parallel.combined_output is Noneconcept_library.teardown()@@ -180,7 +177,6 @@ class TestPipeParallelValidation:output=ConceptFactory.make_concept_ref_with_domain(domain_code=domain_code, concept_code=concept_3.code),branches=[], # No sub-pipes to avoid dependency issuesadd_each_output=True,- combined_output=None,)pipe_parallel = PipeFactory[PipeParallel].make_from_blueprint(diff --git a/tests/unit/pipelex/builder/operations/test_pipe_spec_to_toml.py b/tests/unit/pipelex/builder/operations/test_pipe_spec_to_toml.pyindex c7a4a0e83..66fbc3d3b 100644--- a/tests/unit/pipelex/builder/operations/test_pipe_spec_to_toml.py+++ b/tests/unit/pipelex/builder/operations/test_pipe_spec_to_toml.py@@ -323,7 +323,7 @@ class TestPipeSpecToToml:"pipe_code": "my_par","description": "Parallel branches","inputs": {"doc": "Document"},- "output": "Text",+ "output": "Composite","add_each_output": True,"branches": [{"pipe_code": "branch_a", "result": "result_a"},@@ -337,24 +337,7 @@ class TestPipeSpecToToml:assert "add_each_output = true" in tomlassert "branch_a" in tomlassert "branch_b" in toml-- def test_parallel_combined_output_in_toml(self) -> None:- spec = parse_pipe_spec(- {- "pipe_code": "my_par",- "description": "Parallel with combined output",- "inputs": {"doc": "Document"},- "output": "Text",- "add_each_output": False,- "combined_output": "MergedResult",- "branches": [- {"pipe_code": "branch_a", "result": "result_a"},- ],- },- pipe_type="PipeParallel",- )- toml = pipe_spec_to_toml(spec)- assert 'combined_output = "MergedResult"' in toml+ assert "combined_output" not in toml# -- PipeCondition ----------------------------------------------------diff --git a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.py b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.pyindex 12a3c424b..b8aac68fb 100644--- a/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.py+++ b/tests/unit/pipelex/builder/pipe/pipe_controller/pipe_parallel/test_data.py@@ -31,16 +31,15 @@ class PipeParallelTestCases:SubPipeBlueprint(pipe="validate_data", result="validation"),],add_each_output=True,- combined_output=None,source="tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controllers/pipe_parallel/test_data.py PipeParallelTestCases.PARALLEL_WITH_EACH_OUTPUT", # noqa: E501),)- PARALLEL_WITH_COMBINED = (- "parallel_with_combined",+ PARALLEL_WITHOUT_EACH_OUTPUT = (+ "parallel_without_each_output",PipeParallelSpec(pipe_code="combined_parallel",- description="Parallel with combined output",+ description="Parallel combining into its declared output only",inputs={"input": "Input"},output="CombinedResult",branches=[@@ -48,10 +47,9 @@ class PipeParallelTestCases:SubPipeSpec(pipe_code="pipe2", result="result2"),],add_each_output=False,- combined_output="MergedData",),PipeParallelBlueprint(- description="Parallel with combined output",+ description="Parallel combining into its declared output only",inputs={"input": "Input"},output="CombinedResult",branches=[@@ -59,41 +57,37 @@ class PipeParallelTestCases:SubPipeBlueprint(pipe="pipe2", result="result2"),],add_each_output=False,- combined_output="MergedData",- source="tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controllers/pipe_parallel/test_data.py PipeParallelTestCases.PARALLEL_WITH_COMBINED", # noqa: E501+ source="tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controllers/pipe_parallel/test_data.py PipeParallelTestCases.PARALLEL_WITHOUT_EACH_OUTPUT", # noqa: E501),)- PARALLEL_WITH_BOTH_EACH_OUTPUT_AND_COMBINED = (- "parallel_with_both_each_output_and_combined",+ PARALLEL_OMITTING_ADD_EACH_OUTPUT = (+ "parallel_omitting_add_each_output",PipeParallelSpec(- pipe_code="combined_parallel",- description="Parallel with combined output",+ pipe_code="plain_combined_parallel",+ description="Always-combine parallel omitting add_each_output entirely",inputs={"input": "Input"},output="CombinedResult",branches=[SubPipeSpec(pipe_code="pipe1", result="result1"),SubPipeSpec(pipe_code="pipe2", result="result2"),],- add_each_output=True,- combined_output="MergedData",),PipeParallelBlueprint(- description="Parallel with combined output",+ description="Always-combine parallel omitting add_each_output entirely",inputs={"input": "Input"},output="CombinedResult",branches=[SubPipeBlueprint(pipe="pipe1", result="result1"),SubPipeBlueprint(pipe="pipe2", result="result2"),],- add_each_output=True,- combined_output="MergedData",- source="tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controllers/pipe_parallel/test_data.py PipeParallelTestCases.PARALLEL_WITH_BOTH_EACH_OUTPUT_AND_COMBINED", # noqa: E501+ add_each_output=False,+ source="tests/unit/pipelex/libraries/pipelines/builder/pipe/pipe_controllers/pipe_parallel/test_data.py PipeParallelTestCases.PARALLEL_OMITTING_ADD_EACH_OUTPUT", # noqa: E501),)TEST_CASES: ClassVar[list[tuple[str, PipeParallelSpec, PipeParallelBlueprint]]] = [PARALLEL_WITH_EACH_OUTPUT,- PARALLEL_WITH_COMBINED,- PARALLEL_WITH_BOTH_EACH_OUTPUT_AND_COMBINED,+ PARALLEL_WITHOUT_EACH_OUTPUT,+ PARALLEL_OMITTING_ADD_EACH_OUTPUT,]diff --git a/tests/unit/pipelex/cli/test_run_core_execution.py b/tests/unit/pipelex/cli/test_run_core_execution.pyindex 10098609c..336b97b81 100644--- a/tests/unit/pipelex/cli/test_run_core_execution.py+++ b/tests/unit/pipelex/cli/test_run_core_execution.py@@ -82,10 +82,10 @@ class TestRunCoreExecution:graph_spec: Any | None = None,) -> Any:working_memory = mocker.MagicMock()- working_memory.get_optional_main_stuff.return_value = main_stuff+ working_memory.get_main_stuff.return_value = main_stuffworking_memory.smart_dump.return_value = {"stuff": "dump"}return SimpleNamespace(- optional_main_stuff=main_stuff,+ main_stuff=main_stuff,graph_spec=graph_spec,working_memory=working_memory,)@@ -279,17 +279,6 @@ class TestRunCoreExecution:assert exc_info.value.exit_code == 1- @pytest.mark.usefixtures("config_mock", "console")- def test_save_csv_no_main_stuff_exits(self, mocker: MockerFixture, tmp_path: Path) -> None:- """--save-csv with no main stuff produced is an error after the run."""- pipe_output = self._make_pipe_output(mocker, main_stuff=None)- self._mock_runner(mocker, pipe_output)-- with pytest.raises(typer.Exit) as exc_info:- _run_async(_call_execute_run(save_csv=str(tmp_path / "out.csv")))-- assert exc_info.value.exit_code == 1-@pytest.mark.usefixtures("config_mock", "console")def test_save_csv_non_list_output_exits(self, mocker: MockerFixture, tmp_path: Path) -> None:"""--save-csv requires the main stuff to be a list."""diff --git a/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py b/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.pyindex 7d0bbd728..d46d4da0c 100644--- a/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py+++ b/tests/unit/pipelex/core/bundles/test_data_pipe_sorter.py@@ -52,7 +52,7 @@ class PipeSorterTestCases:"pipe_d": PipeParallelBlueprint(description="D depends on B and C",inputs={},- output="Text",+ output="Composite",branches=[SubPipeBlueprint(pipe="pipe_b", result="result_b"),SubPipeBlueprint(pipe="pipe_c", result="result_c"),diff --git a/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py b/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.pyindex 8d6d6bef7..897601bc2 100644--- a/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py+++ b/tests/unit/pipelex/core/pipes/test_run_pipe_tracer_metadata.py@@ -5,8 +5,11 @@ from typing import Callableimport pytestfrom pytest_mock import MockerFixture+from pipelex.core.concepts.concept import Conceptfrom pipelex.core.memory.working_memory_factory import WorkingMemoryFactoryfrom pipelex.core.pipes.pipe_factory import PipeFactory+from pipelex.core.stuffs.stuff import Stuff+from pipelex.core.stuffs.text_content import TextContentfrom pipelex.pipe_operators.llm.pipe_llm import PipeLLMfrom pipelex.pipe_operators.llm.pipe_llm_blueprint import PipeLLMBlueprintfrom pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory@@ -34,8 +37,19 @@ class TestRunPipeForwardsTracerMetadata:blueprint=blueprint,)+ main_stuff = Stuff(+ stuff_code="main-code",+ stuff_name="main_stuff",+ concept=Concept(+ code="Text",+ domain_code="native",+ description="Plain text",+ structure_class_name="TextContent",+ ),+ content=TextContent(text="hello"),+ )expected_output = mocker.MagicMock()- expected_output.working_memory.get_optional_main_stuff.return_value = None+ expected_output.working_memory.get_main_stuff.return_value = main_stuffmocker.patch.object(PipeLLM, "_live_run_operator_pipe", mocker.AsyncMock(return_value=expected_output))mocker.patch.object(PipeLLM, "validate_before_run", mocker.AsyncMock(return_value=None))mocker.patch.object(PipeLLM, "validate_after_run", mocker.AsyncMock(return_value=None))diff --git a/tests/unit/pipelex/core/stuffs/test_composite_content.py b/tests/unit/pipelex/core/stuffs/test_composite_content.pynew file mode 100644index 000000000..0ae0068fb--- /dev/null+++ b/tests/unit/pipelex/core/stuffs/test_composite_content.py@@ -0,0 +1,139 @@+from typing import cast++from kajson import kajson++from pipelex.core.concepts.concept_factory import ConceptFactory+from pipelex.core.concepts.native.concept_native import NativeConceptCode+from pipelex.core.memory.working_memory_factory import WorkingMemoryFactory+from pipelex.core.stuffs.composite_content import CompositeContent+from pipelex.core.stuffs.list_content import ListContent+from pipelex.core.stuffs.number_content import NumberContent+from pipelex.core.stuffs.stuff_factory import StuffFactory+from pipelex.core.stuffs.text_content import TextContent+from pipelex.runtime_bridge.primitives.hydration import hydrate_working_memory+++class TestCompositeContent:+ def test_named_sub_contents_are_top_level_fields(self):+ """Branch names must surface as top-level serialized fields, with no wrapper key."""+ composite = CompositeContent.model_validate(+ {+ "short_summary": TextContent(text="short"),+ "detailed_summary": TextContent(text="long and detailed"),+ }+ )+ dump = composite.smart_dump()+ assert set(dump.keys()) == {"short_summary", "detailed_summary"}+ assert dump["short_summary"]["text"] == "short"+ assert dump["detailed_summary"]["text"] == "long and detailed"++ def test_model_validate_preserves_unknown_fields(self):+ """The whole point of Composite: unknown fields must NOT be silently dropped."""+ composite = CompositeContent.model_validate({"alpha": {"text": "a"}, "beta": {"text": "b"}})+ dump = composite.smart_dump()+ assert set(dump.keys()) == {"alpha", "beta"}++ def test_kajson_round_trip(self):+ composite = CompositeContent.model_validate(+ {+ "tone_result": TextContent(text="cheerful"),+ "length_result": TextContent(text="short"),+ }+ )+ serialized = kajson.dumps(composite)+ rehydrated = kajson.loads(serialized)+ assert isinstance(rehydrated, CompositeContent)+ assert rehydrated.smart_dump() == composite.smart_dump()++ def test_transport_dump_and_hydrate_round_trip(self):+ """dump_for_transport → hydrate_working_memory must rebuild a CompositeContent main stuff."""+ composite = CompositeContent.model_validate(+ {+ "tone_result": TextContent(text="cheerful"),+ "length_result": TextContent(text="short"),+ }+ )+ concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.COMPOSITE)+ stuff = StuffFactory.make_stuff(concept=concept, content=composite, name="combo")+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff)++ raw = working_memory.dump_for_transport()+ hydrated_memory = hydrate_working_memory(raw)++ hydrated_stuff = hydrated_memory.get_stuff("combo")+ assert isinstance(hydrated_stuff.content, CompositeContent)+ assert hydrated_stuff.content.smart_dump() == composite.smart_dump()++ def test_transport_round_trip_preserves_component_types(self):+ """Hydrated composite components must be typed StuffContent instances, not plain dicts."""+ composite = CompositeContent.model_validate(+ {+ "tone_result": TextContent(text="cheerful"),+ "score": NumberContent(number=42),+ }+ )+ concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.COMPOSITE)+ stuff = StuffFactory.make_stuff(concept=concept, content=composite, name="combo")+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff)++ raw = working_memory.dump_for_transport()+ hydrated_memory = hydrate_working_memory(raw)++ hydrated_content = hydrated_memory.get_stuff("combo").content+ assert isinstance(hydrated_content, CompositeContent)+ components = hydrated_content.components+ tone_result = components["tone_result"]+ assert isinstance(tone_result, TextContent)+ assert tone_result.text == "cheerful"+ score = components["score"]+ assert isinstance(score, NumberContent)+ assert score.number == 42++ def test_transport_round_trip_preserves_list_component(self):+ """A ListContent component (list-producing branch) must survive transport with typed items."""+ composite = CompositeContent.model_validate(+ {+ "summary": TextContent(text="overall summary"),+ "highlights": ListContent[TextContent](items=[TextContent(text="first"), TextContent(text="second")]),+ }+ )+ concept = ConceptFactory.make_native_concept(native_concept_code=NativeConceptCode.COMPOSITE)+ stuff = StuffFactory.make_stuff(concept=concept, content=composite, name="combo")+ working_memory = WorkingMemoryFactory.make_from_single_stuff(stuff)++ raw = working_memory.dump_for_transport()+ hydrated_memory = hydrate_working_memory(raw)++ hydrated_content = hydrated_memory.get_stuff("combo").content+ assert isinstance(hydrated_content, CompositeContent)+ highlights = hydrated_content.components["highlights"]+ assert isinstance(highlights, ListContent)+ typed_highlights = cast("ListContent[TextContent]", highlights)+ assert [type(item) for item in typed_highlights.items] == [TextContent, TextContent]+ assert [item.text for item in typed_highlights.items] == ["first", "second"]++ def test_rendered_markdown_surfaces_named_sub_contents(self):+ composite = CompositeContent.model_validate(+ {+ "tone_result": TextContent(text="cheerful tone"),+ "length_result": TextContent(text="rather short"),+ }+ )+ markdown = composite.rendered_markdown()+ assert "tone_result" in markdown+ assert "cheerful tone" in markdown+ assert "length_result" in markdown+ assert "rather short" in markdown++ def test_rendered_html_surfaces_named_sub_contents(self):+ composite = CompositeContent.model_validate(+ {+ "tone_result": TextContent(text="cheerful tone"),+ "length_result": TextContent(text="rather short"),+ }+ )+ html = composite.rendered_html()+ assert "tone_result" in html+ assert "cheerful tone" in html+ assert "length_result" in html+ assert "rather short" in htmldiff --git a/tests/unit/pipelex/graph/test_mermaidflow.py b/tests/unit/pipelex/graph/test_mermaidflow.pyindex 274f8f623..2e2049985 100644--- a/tests/unit/pipelex/graph/test_mermaidflow.py+++ b/tests/unit/pipelex/graph/test_mermaidflow.py@@ -396,7 +396,7 @@ class TestMermaidflow:"status": NodeStatus.SUCCEEDED,"node_io": NodeIOSpec(inputs=[],- outputs=[IOSpec(name="combined_output", concept="MergedText", digest="combined_digest_001")],+ outputs=[IOSpec(name="combined_result", concept="MergedText", digest="combined_digest_001")],),}branch_a = {@@ -470,5 +470,5 @@ class TestMermaidflow:assert subgraph_end_idx is not None, "Controller subgraph end not found"subgraph_content = "\n".join(lines[subgraph_start_idx : subgraph_end_idx + 1])- assert "combined_output" in subgraph_content, "Combined output stuff should be inside the controller subgraph"+ assert "combined_result" in subgraph_content, "Combined output stuff should be inside the controller subgraph"assert ":::stuff" in subgraph_content, "Combined output stuff should have :::stuff class styling"diff --git a/tests/unit/pipelex/pipe_controllers/parallel/data.py b/tests/unit/pipelex/pipe_controllers/parallel/data.pyindex 9d65b0263..b6e132706 100644--- a/tests/unit/pipelex/pipe_controllers/parallel/data.py+++ b/tests/unit/pipelex/pipe_controllers/parallel/data.py@@ -13,7 +13,7 @@ class PipeParallelInputTestCases:PipeParallelBlueprint(description="Test case: valid_with_add_each_output",inputs={"data": "native.Text"},- output="native.Text",+ output="native.Composite",branches=[SubPipeBlueprint(pipe="process_a", result="result_a"),SubPipeBlueprint(pipe="process_b", result="result_b"),@@ -22,32 +22,16 @@ class PipeParallelInputTestCases:),)- VALID_WITH_COMBINED_OUTPUT: ClassVar[tuple[str, PipeParallelBlueprint]] = (- "valid_with_combined_output",+ VALID_WITHOUT_ADD_EACH_OUTPUT: ClassVar[tuple[str, PipeParallelBlueprint]] = (+ "valid_without_add_each_output",PipeParallelBlueprint(- description="Test case: valid_with_combined_output",+ description="Test case: valid_without_add_each_output",inputs={"data": "native.Text"},- output="native.Text",+ output="native.Composite",branches=[SubPipeBlueprint(pipe="analyze_1", result="analysis_1"),SubPipeBlueprint(pipe="analyze_2", result="analysis_2"),],- combined_output="native.Text",- ),- )-- VALID_WITH_BOTH_OUTPUT_OPTIONS: ClassVar[tuple[str, PipeParallelBlueprint]] = (- "valid_with_both_output_options",- PipeParallelBlueprint(- description="Test case: valid_with_both_output_options",- inputs={"data": "native.Text"},- output="native.Text",- branches=[- SubPipeBlueprint(pipe="compute_x", result="x"),- SubPipeBlueprint(pipe="compute_y", result="y"),- ],- add_each_output=True,- combined_output="native.Text",),)@@ -56,7 +40,7 @@ class PipeParallelInputTestCases:PipeParallelBlueprint(description="Test case: valid_three_parallels",inputs={"input_data": "native.Text"},- output="native.Text",+ output="native.Composite",branches=[SubPipeBlueprint(pipe="branch_1", result="result_1"),SubPipeBlueprint(pipe="branch_2", result="result_2"),@@ -71,41 +55,54 @@ class PipeParallelInputTestCases:PipeParallelBlueprint(description="Test case: valid_multiple_inputs",inputs={"text_data": "native.Text", "image_data": "native.Image"},- output="native.Text",+ output="native.Composite",branches=[SubPipeBlueprint(pipe="process_text", result="text_result"),SubPipeBlueprint(pipe="process_image", result="image_result"),],- combined_output="native.Text",),)VALID_CASES: ClassVar[list[tuple[str, PipeParallelBlueprint]]] = [VALID_WITH_ADD_EACH_OUTPUT,- VALID_WITH_COMBINED_OUTPUT,- VALID_WITH_BOTH_OUTPUT_OPTIONS,+ VALID_WITHOUT_ADD_EACH_OUTPUT,VALID_THREE_PARALLELS,VALID_MULTIPLE_INPUTS,]# Error test cases: (test_id, blueprint_dict, expected_error_message_fragment)# Using dicts instead of blueprints to avoid validation errors during import- ERROR_NO_OUTPUT_OPTIONS: ClassVar[tuple[str, dict[str, Any], str]] = (- "no_output_options",+ ERROR_NATIVE_NON_COMPOSITE_OUTPUT: ClassVar[tuple[str, dict[str, Any], str]] = (+ "native_non_composite_output",{- "description": "Test case: no_output_options",+ "description": "Test case: native_non_composite_output","inputs": {"data": "native.Text"},"output": "native.Text","branches": [{"pipe": "process_a", "result": "result_a"},{"pipe": "process_b", "result": "result_b"},],- "add_each_output": False,- "combined_output": None,+ "add_each_output": True,+ },+ "must be 'Composite' or a structured concept",+ )++ ERROR_MULTIPLICITY_OUTPUT: ClassVar[tuple[str, dict[str, Any], str]] = (+ "multiplicity_output",+ {+ "description": "Test case: multiplicity_output",+ "inputs": {"data": "native.Text"},+ "output": "native.Composite[]",+ "branches": [+ {"pipe": "process_a", "result": "result_a"},+ {"pipe": "process_b", "result": "result_b"},+ ],+ "add_each_output": True,},- "requires either add_each_output to be True or combined_output to be set",+ "must not declare a multiplicity",)ERROR_CASES: ClassVar[list[tuple[str, dict[str, Any], str]]] = [- ERROR_NO_OUTPUT_OPTIONS,+ ERROR_NATIVE_NON_COMPOSITE_OUTPUT,+ ERROR_MULTIPLICITY_OUTPUT,]diff --git a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_blueprint.py b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_blueprint.pyindex 24373dfcb..bd4a6fcb1 100644--- a/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_blueprint.py+++ b/tests/unit/pipelex/pipe_controllers/parallel/test_pipe_parallel_blueprint.py@@ -10,7 +10,7 @@ class TestPipeParallelBlueprint:blueprint = PipeParallelBlueprint(description="lorem ipsum",inputs={"data": "Text"},- output="Text",+ output="Composite",branches=[SubPipeBlueprint(pipe="process_a", result="result_a"),SubPipeBlueprint(pipe="process_b", result="result_b"),@@ -22,7 +22,7 @@ class TestPipeParallelBlueprint:blueprint = PipeParallelBlueprint(description="lorem ipsum",inputs={"data": "Text"},- output="Text",+ output="Composite",branches=[SubPipeBlueprint(pipe="step1", result="result1"),SubPipeBlueprint(pipe="step2", result="result2"),@@ -32,74 +32,57 @@ class TestPipeParallelBlueprint:)assert blueprint.pipe_dependencies == {"step1", "step2", "step3"}- def test_validate_combined_output_correct(self):+ @pytest.mark.parametrize("output", ["Composite", "native.Composite", "MergedData", "some_domain.MergedData"])+ def test_valid_outputs_accepted(self, output: str):blueprint = PipeParallelBlueprint(description="lorem ipsum",inputs={"data": "Text"},- output="Text",+ output=output,branches=[SubPipeBlueprint(pipe="process", result="result")],- combined_output="Text",)- assert blueprint.combined_output == "Text"+ assert blueprint.output == output- blueprint = PipeParallelBlueprint(- description="lorem ipsum",- inputs={"data": "Text"},- output="Text",- branches=[SubPipeBlueprint(pipe="process", result="result")],- combined_output="Number",- )- assert blueprint.combined_output == "Number"-- def test_validate_combined_output_incorrect(self):- with pytest.raises(ValidationError) as exc_info:+ @pytest.mark.parametrize("output", ["Text", "native.Text", "Image", "Number", "Dynamic", "Anything"])+ def test_native_non_composite_output_rejected(self, output: str):+ with pytest.raises(ValidationError, match="must be 'Composite' or a structured concept"):PipeParallelBlueprint(description="lorem ipsum",inputs={"data": "Text"},- output="Text",+ output=output,branches=[SubPipeBlueprint(pipe="process", result="result")],- combined_output="InvalidConcept!",)- assert "Combined output 'InvalidConcept!' is not a valid concept string or code" in str(exc_info.value)-- def test_validate_output_options_correct(self):- blueprint = PipeParallelBlueprint(- description="lorem ipsum",- inputs={"data": "Text"},- output="Text",- branches=[SubPipeBlueprint(pipe="process", result="result")],- add_each_output=True,- )- assert blueprint.add_each_output is True- blueprint = PipeParallelBlueprint(- description="lorem ipsum",- inputs={"data": "Text"},- output="Text",- branches=[SubPipeBlueprint(pipe="process", result="result")],- combined_output="Text",- )- assert blueprint.combined_output == "Text"+ @pytest.mark.parametrize("output", ["Composite[]", "MergedData[]", "MergedData[3]"])+ def test_multiplicity_output_rejected(self, output: str):+ with pytest.raises(ValidationError, match="must not declare a multiplicity"):+ PipeParallelBlueprint(+ description="lorem ipsum",+ inputs={"data": "Text"},+ output=output,+ branches=[SubPipeBlueprint(pipe="process", result="result")],+ )+ def test_add_each_output_defaults_to_false_and_is_valid_alone(self):+ """The one-of-two (add_each_output/combined_output) validator is gone: a parallel+ always combines into its declared output, so no extra flag is required.+ """blueprint = PipeParallelBlueprint(description="lorem ipsum",inputs={"data": "Text"},- output="Text",+ output="Composite",branches=[SubPipeBlueprint(pipe="process", result="result")],- add_each_output=True,- combined_output="Text",)- assert blueprint.add_each_output is True- assert blueprint.combined_output == "Text"+ assert blueprint.add_each_output is False- def test_validate_output_options_incorrect(self):- with pytest.raises(ValidationError) as exc_info:- PipeParallelBlueprint(- description="lorem ipsum",- inputs={"data": "Text"},- output="Text",- branches=[SubPipeBlueprint(pipe="process", result="result")],- add_each_output=False,- combined_output=None,+ def test_combined_output_field_is_gone(self):+ """combined_output was deleted from the language: passing it must be rejected."""+ with pytest.raises(ValidationError):+ PipeParallelBlueprint.model_validate(+ {+ "description": "lorem ipsum",+ "inputs": {"data": "Text"},+ "output": "Composite",+ "branches": [{"pipe": "process", "result": "result"}],+ "combined_output": "Composite",+ })- assert "PipeParallel requires either add_each_output to be True or combined_output to be set" in str(exc_info.value)diff --git a/tests/unit/pipelex/pipe_run/test_delivery_executor.py b/tests/unit/pipelex/pipe_run/test_delivery_executor.pyindex 5a4fa18bc..9ab0a6708 100644--- a/tests/unit/pipelex/pipe_run/test_delivery_executor.py+++ b/tests/unit/pipelex/pipe_run/test_delivery_executor.py@@ -6,6 +6,11 @@ from pytest_mock import MockerFixturefrom pipelex.base_exceptions import ErrorDomain, ErrorReportfrom pipelex.cogt.inference.error_classification import ProviderErrorMetadata, UserAction, UserActionKindfrom pipelex.cogt.inference.provider_name import ProviderName+from pipelex.core.concepts.concept import Concept+from pipelex.core.memory.exceptions import WorkingMemoryStuffNotFoundError+from pipelex.core.memory.working_memory import WorkingMemory+from pipelex.core.stuffs.stuff import Stuff+from pipelex.core.stuffs.text_content import TextContentfrom pipelex.pipe_run.delivery_assignment import (DeliveryAssignment,DeliveryStatus,@@ -13,10 +18,24 @@ from pipelex.pipe_run.delivery_assignment import (WebhookTarget,)from pipelex.pipe_run.delivery_executor import DeliveryExecutor-from pipelex.pipe_run.exceptions import StorageDeliveryError, WebhookDeliveryError+from pipelex.pipe_run.exceptions import PipeJobError, StorageDeliveryError, WebhookDeliveryErrorfrom pipelex.tools.network.exceptions import SsrfBlockedError+def _make_main_stuff() -> Stuff:+ return Stuff(+ stuff_code="main-code",+ stuff_name="main_stuff",+ concept=Concept(+ code="Text",+ domain_code="native",+ description="Plain text",+ structure_class_name="TextContent",+ ),+ content=TextContent(text="Hello delivery!"),+ )++@pytest.mark.asyncio(loop_scope="class")class TestDeliveryExecutor:async def test_execute_storage_only(self, mocker: MockerFixture) -> None:@@ -28,7 +47,7 @@ class TestDeliveryExecutor:mock_output = mocker.MagicMock()mock_output.working_memory_raw = Nonemock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}- mock_output.working_memory.get_optional_main_stuff.return_value = None+ mock_output.working_memory.get_main_stuff.return_value = _make_main_stuff()mock_output.graph_spec = Noneexecutor = DeliveryExecutor()@@ -45,6 +64,10 @@ class TestDeliveryExecutor:mock_storage.store.assert_called()stored_keys = [call.kwargs["key"] for call in mock_storage.store.call_args_list]assert any("test-user/plr-123/working_memory.json" in key for key in stored_keys)+ # A completed run always delivers a main stuff, so the main_stuff artifact files are always written.+ assert any("test-user/plr-123/main_stuff.json" in key for key in stored_keys)+ assert any("test-user/plr-123/main_stuff.md" in key for key in stored_keys)+ assert any("test-user/plr-123/main_stuff.html" in key for key in stored_keys)async def test_execute_webhook_only(self, mocker: MockerFixture) -> None:mock_client = mocker.AsyncMock()@@ -142,7 +165,7 @@ class TestDeliveryExecutor:mock_output = mocker.MagicMock()mock_output.working_memory_raw = Nonemock_output.working_memory.smart_dump.return_value = {}- mock_output.working_memory.get_optional_main_stuff.return_value = None+ mock_output.working_memory.get_main_stuff.return_value = _make_main_stuff()mock_output.graph_spec = Noneexecutor = DeliveryExecutor()@@ -157,6 +180,25 @@ class TestDeliveryExecutor:status=DeliveryStatus.COMPLETED,)+ async def test_generate_result_files_raises_without_main_stuff_typed(self, mocker: MockerFixture) -> None:+ """A completed run always delivers a main stuff — a typed working memory without one is a contract violation, not an empty envelope."""+ mock_output = mocker.MagicMock()+ mock_output.working_memory_raw = None+ mock_output.working_memory = WorkingMemory()+ mock_output.graph_spec = None++ with pytest.raises(WorkingMemoryStuffNotFoundError):+ await DeliveryExecutor().generate_result_files(mock_output)++ async def test_generate_result_files_raises_without_main_stuff_raw(self, mocker: MockerFixture) -> None:+ """Same contract on the raw (cross-process) path: a raw working memory without a main stuff fails loudly."""+ mock_output = mocker.MagicMock()+ mock_output.working_memory_raw = {"root": {}, "aliases": {}}+ mock_output.graph_spec = None++ with pytest.raises(PipeJobError):+ await DeliveryExecutor().generate_result_files(mock_output)+async def test_try_local_hydrate_stuff_returns_typed_for_builtin(self) -> None:from pipelex.core.stuffs.text_content import TextContent # noqa: PLC0415from pipelex.hub import get_class_registry # noqa: PLC0415@@ -298,7 +340,7 @@ class TestDeliveryExecutor:"content": [page],}},- "aliases": {},+ "aliases": {"main_stuff": "cv_pages"},}mock_output.graph_spec = None@@ -426,7 +468,7 @@ class TestDeliveryExecutor:mock_output = mocker.MagicMock()mock_output.working_memory_raw = Nonemock_output.working_memory.smart_dump.return_value = {"root": {}, "aliases": {}}- mock_output.working_memory.get_optional_main_stuff.return_value = None+ mock_output.working_memory.get_main_stuff.return_value = _make_main_stuff()mock_output.graph_spec = Noneexecutor = DeliveryExecutor()diff --git a/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py b/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.pynew file mode 100644index 000000000..75e84b9a5--- /dev/null+++ b/tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py@@ -0,0 +1,44 @@+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.pipe_run.pipe_job import PipeJob+from pipelex.pipe_run.pipe_run_params_factory import PipeRunParamsFactory+from pipelex.pipeline.job_metadata import JobMetadata+from pipelex.runtime_bridge.direct_orchestrator import DirectOrchestrator+from pipelex.runtime_bridge.exceptions import PipelexBridgeDispatchError++if TYPE_CHECKING:+ from pipelex.core.pipes.pipe_abstract import PipeAbstract+++class _StubPipe:+ """Minimal stand-in — DirectOrchestrator.start() only reads pipe_job.pipe.code for its error message."""++ code = "direct_test_pipe"+++def _make_pipe_job() -> PipeJob:+ """Build a PipeJob via model_construct to bypass pipe validation (start() never runs the pipe)."""+ return PipeJob.model_construct(+ pipe=cast("PipeAbstract", _StubPipe()),+ working_memory=None,+ working_memory_raw=None,+ pipe_run_params=PipeRunParamsFactory.make_run_params(),+ job_metadata=JobMetadata(user_id="test-user", pipeline_run_id="test-run"),+ output_name=None,+ library_crate=None,+ )+++class TestDirectOrchestratorStart:+ def test_does_not_support_fire_and_forget(self) -> None:+ """DIRECT is in-process only, so it must advertise no genuine async path."""+ assert DirectOrchestrator().supports_fire_and_forget is False++ @pytest.mark.asyncio+ async def test_start_raises_dispatch_error(self) -> None:+ """start() is unreachable behind the supports_fire_and_forget gate; called directly it must fail loudly, never ack."""+ orchestrator = DirectOrchestrator()+ with pytest.raises(PipelexBridgeDispatchError, match="supports_fire_and_forget"):+ await orchestrator.start(pipe_job=_make_pipe_job(), delivery_assignment=None)diff --git a/tests/unit/pipelex/runtime_bridge/test_input_models.py b/tests/unit/pipelex/runtime_bridge/test_input_models.pyindex 7cda7ff7e..1f706b553 100644--- a/tests/unit/pipelex/runtime_bridge/test_input_models.py+++ b/tests/unit/pipelex/runtime_bridge/test_input_models.py@@ -2,7 +2,7 @@ import pytestfrom pydantic import ValidationErrorfrom pipelex.runtime_bridge.delivery_mode import DeliveryMode-from pipelex.runtime_bridge.payloads import PipelexPipeRunInput, PipelexPipeRunOutput+from pipelex.runtime_bridge.payloads import PipelexPipeDispatchAck, PipelexPipeRunInput, PipelexPipeRunOutputclass TestInputOutputModels:@@ -45,27 +45,69 @@ class TestInputOutputModels:def test_output_required_fields(self):with pytest.raises(ValidationError):- PipelexPipeRunOutput.model_validate({"output_dict": {}}) # missing pipeline_run_id and is_completed+ PipelexPipeRunOutput.model_validate({"output_dict": {}}) # missing pipeline_run_id and main_stuff_name++ def test_output_requires_main_stuff_name(self):+ """A completed run always delivers a main stuff, so the boundary DTO requires its name."""+ with pytest.raises(ValidationError):+ PipelexPipeRunOutput.model_validate(+ {+ "output_dict": {},+ "pipeline_run_id": "run-1",+ }+ )def test_output_forbids_extra_fields(self):with pytest.raises(ValidationError):PipelexPipeRunOutput.model_validate({"output_dict": {},+ "main_stuff_name": "main","pipeline_run_id": "run-1",- "is_completed": True,"rogue_field": 42,})+ def test_output_rejects_is_completed(self):+ """The completed-run DTO has no is_completed flag: a fire-and-forget dispatch returns a PipelexPipeDispatchAck instead."""+ with pytest.raises(ValidationError):+ PipelexPipeRunOutput.model_validate(+ {+ "output_dict": {},+ "main_stuff_name": "main",+ "pipeline_run_id": "run-1",+ "is_completed": True,+ }+ )+def test_output_round_trip_via_json(self):original = PipelexPipeRunOutput(output_dict={"foo": "bar"},main_stuff_name="main",pipeline_run_id="run-1",workflow_id=None,- is_completed=True,graph_spec_dump=None,)round_tripped = PipelexPipeRunOutput.model_validate(original.model_dump(mode="json"))assert round_tripped == original++ def test_dispatch_ack_requires_both_ids(self):+ """A fire-and-forget ack is ids-only, and both ids are required — a genuine enqueue always has a workflow id."""+ ack = PipelexPipeDispatchAck(pipeline_run_id="run-1", workflow_id="wf-1")+ assert ack.pipeline_run_id == "run-1"+ assert ack.workflow_id == "wf-1"+ with pytest.raises(ValidationError):+ PipelexPipeDispatchAck.model_validate({"pipeline_run_id": "run-1"})+ with pytest.raises(ValidationError):+ PipelexPipeDispatchAck.model_validate({"workflow_id": "wf-1"})++ def test_dispatch_ack_forbids_output_fields(self):+ """The ack must not smuggle completed-run fields."""+ with pytest.raises(ValidationError):+ PipelexPipeDispatchAck.model_validate(+ {+ "pipeline_run_id": "run-1",+ "workflow_id": "wf-1",+ "output_dict": {},+ }+ )diff --git a/tests/unit/pipelex/runtime_bridge/test_output_serialization.py b/tests/unit/pipelex/runtime_bridge/test_output_serialization.pyindex 0f6b8cd3b..d0f35ff5a 100644--- a/tests/unit/pipelex/runtime_bridge/test_output_serialization.py+++ b/tests/unit/pipelex/runtime_bridge/test_output_serialization.py@@ -1,6 +1,30 @@-from pipelex.core.memory.working_memory import WorkingMemory+import pytest++from pipelex.core.concepts.concept import Concept+from pipelex.core.memory.working_memory import MAIN_STUFF_NAME, WorkingMemoryfrom pipelex.core.pipes.pipe_output import PipeOutput-from pipelex.runtime_bridge.serialization import serialize_completed_output+from pipelex.core.stuffs.stuff import Stuff+from pipelex.core.stuffs.text_content import TextContent+from pipelex.pipe_run.exceptions import PipeJobError+from pipelex.runtime_bridge.serialization import resolve_main_stuff_root_key, serialize_completed_output+++def _make_stuff(name: str) -> Stuff:+ return Stuff(+ stuff_code=f"code-{name}",+ stuff_name=name,+ concept=Concept(+ code="Text",+ domain_code="native",+ description="Plain text",+ structure_class_name="TextContent",+ ),+ content=TextContent(text="hello"),+ )+++def _memory_with_aliased_main(name: str) -> WorkingMemory:+ return WorkingMemory(root={name: _make_stuff(name)}, aliases={MAIN_STUFF_NAME: name})class TestSerializeCompletedOutput:@@ -12,7 +36,7 @@ class TestSerializeCompletedOutput:report or tell an assembly failure apart from assembly being off."""pipe_output = PipeOutput(- working_memory=WorkingMemory(),+ working_memory=_memory_with_aliased_main("result"),pipeline_run_id="run-with-usage",tokens_usages=[],usage_assembly_error="usage boom",@@ -28,12 +52,11 @@ class TestSerializeCompletedOutput:assert dto.usage_assembly_error == "usage boom"assert dto.graph_assembly_error == "graph boom"assert dto.pipeline_run_id == "run-with-usage"- assert dto.is_completed is Truedef test_absent_assembly_serializes_to_none(self) -> None:"""When the run produced no usage/graph, the boundary fields stay None (not [])."""pipe_output = PipeOutput(- working_memory=WorkingMemory(),+ working_memory=_memory_with_aliased_main("result"),pipeline_run_id="run-without-usage",)@@ -45,3 +68,31 @@ class TestSerializeCompletedOutput:assert dto.tokens_usages_dump is Noneassert dto.usage_assembly_error is Noneassert dto.graph_assembly_error is None++ def test_main_stuff_name_resolves_aliased_root_key(self) -> None:+ """The DTO's main_stuff_name is the actual root key the alias points at."""+ pipe_output = PipeOutput(+ working_memory=_memory_with_aliased_main("final_result"),+ pipeline_run_id="run-aliased",+ )++ dto = serialize_completed_output(pipe_output=pipe_output, workflow_id=None)++ assert dto.main_stuff_name == "final_result"++ def test_main_stuff_name_resolves_direct_root_key(self) -> None:+ """When the main stuff sits directly at root[MAIN_STUFF_NAME], that key is returned."""+ memory = WorkingMemory(root={MAIN_STUFF_NAME: _make_stuff(MAIN_STUFF_NAME)})+ pipe_output = PipeOutput(working_memory=memory, pipeline_run_id="run-direct")++ assert resolve_main_stuff_root_key(pipe_output=pipe_output) == MAIN_STUFF_NAME++ def test_missing_main_stuff_raises(self) -> None:+ """A completed run always delivers a main stuff — serializing one without it is a contract violation."""+ pipe_output = PipeOutput(+ working_memory=WorkingMemory(),+ pipeline_run_id="run-broken",+ )++ with pytest.raises(PipeJobError):+ serialize_completed_output(pipe_output=pipe_output, workflow_id=None)diff --git a/wip/required-main-stuff-brief.html b/wip/required-main-stuff-brief.htmlnew file mode 100644index 000000000..53a759c33--- /dev/null+++ b/wip/required-main-stuff-brief.html@@ -0,0 +1,468 @@+<title>Required main stuff — design brief</title>+<style>+ :root {+ --ground: #FBFCFC;+ --panel: #F1F5F6;+ --panel-deep: #E7EEF0;+ --ink: #1A2B32;+ --muted: #5B6E75;+ --line: #D8E0E3;+ --accent: #0E7490;+ --danger: #B3422F;+ --danger-soft: #F6E9E5;+ --good: #2F7D4F;+ --good-soft: #E6F1EA;+ --code-bg: #EEF3F4;+ }+ @media (prefers-color-scheme: dark) {+ :root {+ --ground: #10181C;+ --panel: #182228;+ --panel-deep: #1E2B33;+ --ink: #E8EDEF;+ --muted: #8FA3AB;+ --line: #2A3840;+ --accent: #3FB2D2;+ --danger: #E07856;+ --danger-soft: #33211C;+ --good: #5CB884;+ --good-soft: #1B2E24;+ --code-bg: #16212A;+ }+ }+ :root[data-theme="dark"] {+ --ground: #10181C;+ --panel: #182228;+ --panel-deep: #1E2B33;+ --ink: #E8EDEF;+ --muted: #8FA3AB;+ --line: #2A3840;+ --accent: #3FB2D2;+ --danger: #E07856;+ --danger-soft: #33211C;+ --good: #5CB884;+ --good-soft: #1B2E24;+ --code-bg: #16212A;+ }+ :root[data-theme="light"] {+ --ground: #FBFCFC;+ --panel: #F1F5F6;+ --panel-deep: #E7EEF0;+ --ink: #1A2B32;+ --muted: #5B6E75;+ --line: #D8E0E3;+ --accent: #0E7490;+ --danger: #B3422F;+ --danger-soft: #F6E9E5;+ --good: #2F7D4F;+ --good-soft: #E6F1EA;+ --code-bg: #EEF3F4;+ }++ * { box-sizing: border-box; }+ body {+ margin: 0;+ background: var(--ground);+ color: var(--ink);+ font: 400 16px/1.62 -apple-system, "SF Pro Text", "Segoe UI Variable", "Segoe UI", system-ui, sans-serif;+ -webkit-font-smoothing: antialiased;+ }+ .wrap { max-width: 46rem; margin: 0 auto; padding: 0 1.25rem; }+ .wrap-wide { max-width: 56rem; margin: 0 auto; padding: 0 1.25rem; }++ header.masthead { padding: 3rem 0 1.75rem; }+ .kicker {+ font-size: 0.6875rem;+ font-weight: 650;+ letter-spacing: 0.1em;+ text-transform: uppercase;+ color: var(--accent);+ margin: 0 0 0.75rem;+ }+ h1 {+ font-size: clamp(1.7rem, 4.5vw, 2.15rem);+ line-height: 1.18;+ font-weight: 750;+ letter-spacing: -0.015em;+ margin: 0 0 0.6rem;+ text-wrap: balance;+ }+ .standfirst { color: var(--muted); font-size: 1.05rem; margin: 0; max-width: 42rem; }+ .meta { display: flex; flex-wrap: wrap; gap: 0.4rem 1.4rem; margin-top: 1.1rem; font-size: 0.8125rem; color: var(--muted); }+ .meta code { font-size: 0.78rem; }++ .tldr {+ background: var(--panel);+ border-top: 1px solid var(--line);+ border-bottom: 1px solid var(--line);+ padding: 1.6rem 0 1.7rem;+ margin: 1.25rem 0 0;+ }+ .tldr .label {+ font-size: 0.6875rem;+ font-weight: 700;+ letter-spacing: 0.12em;+ text-transform: uppercase;+ color: var(--accent);+ margin: 0 0 0.9rem;+ }+ .tldr ul { margin: 0; padding: 0 0 0 1.1rem; display: grid; gap: 0.65rem; }+ .tldr li { padding-left: 0.2rem; }+ .tldr li::marker { color: var(--accent); }+ .tldr b { font-weight: 650; }++ main { padding: 0.5rem 0 3rem; }+ section { margin-top: 2.9rem; }+ h2 {+ font-size: 1.28rem;+ font-weight: 700;+ letter-spacing: -0.01em;+ line-height: 1.25;+ margin: 0.3rem 0 0.9rem;+ text-wrap: balance;+ }+ .sec-kicker {+ font-size: 0.6875rem;+ font-weight: 650;+ letter-spacing: 0.1em;+ text-transform: uppercase;+ color: var(--muted);+ margin: 0;+ }+ p { margin: 0 0 1rem; }+ .note { color: var(--muted); font-size: 0.875rem; }++ code, pre {+ font-family: ui-monospace, "SF Mono", "Cascadia Code", "JetBrains Mono", Consolas, monospace;+ font-size: 0.845rem;+ }+ code { background: var(--code-bg); border-radius: 4px; padding: 0.1em 0.35em; }+ pre {+ background: var(--code-bg);+ border: 1px solid var(--line);+ border-radius: 6px;+ padding: 0.9rem 1rem;+ overflow-x: auto;+ line-height: 1.55;+ margin: 0;+ }+ pre code { background: none; padding: 0; }+ .del { color: var(--danger); }+ .add { color: var(--good); }+ .dim { color: var(--muted); }++ figure { margin: 1.4rem 0; }+ .fig-frame {+ border: 1px solid var(--line);+ border-radius: 6px;+ background: var(--panel);+ padding: 0.75rem;+ overflow-x: auto;+ }+ .fig-frame svg { display: block; min-width: 720px; width: 100%; height: auto; }+ figcaption { font-size: 0.8125rem; color: var(--muted); margin-top: 0.55rem; line-height: 1.5; }++ table.modes { width: 100%; border-collapse: collapse; font-size: 0.9rem; margin: 1.2rem 0; }+ .modes-scroll { overflow-x: auto; }+ table.modes th, table.modes td { text-align: left; padding: 0.55rem 0.9rem 0.55rem 0; border-bottom: 1px solid var(--line); vertical-align: top; }+ table.modes th { font-size: 0.6875rem; font-weight: 650; letter-spacing: 0.08em; text-transform: uppercase; color: var(--muted); }+ table.modes td:first-child { font-weight: 650; white-space: nowrap; }+ .pill {+ display: inline-block; font-size: 0.72rem; font-weight: 650; letter-spacing: 0.02em;+ padding: 0.1rem 0.5rem; border-radius: 999px; white-space: nowrap;+ }+ .pill-danger { color: var(--danger); background: var(--danger-soft); }+ .pill-good { color: var(--good); background: var(--good-soft); }++ .duo { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin: 1.2rem 0; }+ .duo h3 { font-size: 0.6875rem; font-weight: 650; letter-spacing: 0.08em; text-transform: uppercase; color: var(--muted); margin: 0 0 0.5rem; }+ @media (max-width: 720px) { .duo { grid-template-columns: 1fr; } }++ ol.phases { margin: 1rem 0 0; padding-left: 1.2rem; display: grid; gap: 0.6rem; }+ ol.phases li::marker { color: var(--accent); font-weight: 650; }+ ol.phases b { font-weight: 650; }++ footer { border-top: 1px solid var(--line); padding: 1.4rem 0 3rem; font-size: 0.875rem; color: var(--muted); }+ a { color: var(--accent); text-underline-offset: 2px; }+ a:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; border-radius: 2px; }++ /* SVG diagram tokens */+ .sv-box { fill: var(--ground); stroke: var(--line); stroke-width: 1.2; rx: 6; }+ .sv-slot { fill: var(--ground); stroke: var(--line); stroke-width: 1.2; rx: 5; }+ .sv-slot-good { fill: var(--good-soft); stroke: var(--good); stroke-width: 1.3; rx: 5; }+ .sv-lane { fill: var(--panel-deep); stroke: none; rx: 8; }+ .sv-t { fill: var(--ink); font-size: 13px; font-weight: 650; }+ .sv-ts { fill: var(--muted); font-size: 11.5px; font-weight: 400; }+ .sv-mono { fill: var(--ink); font-size: 11.5px; font-family: ui-monospace, "SF Mono", Consolas, monospace; }+ .sv-mono-dim { fill: var(--muted); font-size: 11px; font-family: ui-monospace, "SF Mono", Consolas, monospace; }+ .sv-good-t { fill: var(--good); font-size: 11.5px; font-weight: 650; }+ .sv-danger-t { fill: var(--danger); font-size: 11.5px; font-weight: 650; }+ .sv-flow { stroke: var(--muted); stroke-width: 1.4; fill: none; }+ .sv-flow-danger { stroke: var(--danger); stroke-width: 1.6; fill: none; stroke-dasharray: 5 4; }+ .sv-flow-good { stroke: var(--good); stroke-width: 1.8; fill: none; }+</style>++<header class="masthead">+ <div class="wrap">+ <p class="kicker">Pipelex runtime · design note</p>+ <h1>A pipeline must always deliver its main output</h1>+ <p class="standfirst">Why reading a pipeline's result is uglier than it should be, the rule we adopted to fix it, and the API shape divergence we're tracking separately.</p>+ <div class="meta">+ <span>Branch <code>feature/Required-main-stuff</code></span>+ <span>Full design: <code>wip/required-main-stuff.md</code></span>+ </div>+ </div>+</header>++<div class="tldr">+ <div class="wrap">+ <p class="label">TL;DR</p>+ <ul>+ <li><b>Problem.</b> A pipeline doesn't always deliver a <code>main_stuff</code>. One construct — <code>PipeParallel</code> with only <code>add_each_output</code> — never stamps it, so callers get a <em>stale</em> value, <em>nothing at all</em>, or (hosted) an <em>empty result envelope</em>. Every consumer pays with defensive code.</li>+ <li><b>Decision.</b> Every pipe run now delivers a main stuff. <code>PipeParallel</code> <b>always combines</b> its branch outputs into its declared <code>output</code> concept. The redundant <code>combined_output</code> field is <b>deleted</b>, and a new <code>native.Composite</code> concept covers the "just give me the named results" case.</li>+ <li><b>Payoff.</b> <code>main_stuff</code> stops being optional across the runtime, the wire, and the SDKs — a stale-output bug disappears and a pile of defensive branches gets deleted.</li>+ <li><b>Side track.</b> The two execution surfaces return the same data in <b>different shapes</b>: blocking <code>POST /v1/execute</code> nests everything under <code>pipe_output</code> (working memory in its transport encoding, main stuff reached via the <code>main_stuff_name</code> pointer), while <code>POST /v1/start</code> → delivery relays flat artifacts via <code>GET /v1/runs/{id}/results</code> — <code>main_stuff</code> (content pre-extracted), <code>graph_spec</code>, and <code>working_memory</code> too, as a clean render. Same information, two encodings — every client normalizes. Unifying them is a companion track, designed separately.</li>+ </ul>+ </div>+</div>++<main>+ <div class="wrap">++ <section>+ <p class="sec-kicker">The symptom</p>+ <h2>Reading a result takes a page of guesswork</h2>+ <p>Our own starter project needs a normalizer just to get a pipeline's output. It checks whether <code>main_stuff</code> exists, falls back to walking the serialized working memory, and picks "the first entry whose content is a dict" — a heuristic, in code we hand to every new user:</p>+ <pre><code><span class="dim"># pipelex-starter-python/my_project/run_output.py (abridged)</span>+def find_main_content(results: RunResults) -> dict[str, Any] <span class="del">| None</span>:+ if isinstance(results.main_stuff, dict): <span class="dim"># /start → delivery shape…</span>+ return results.main_stuff+ working_memory = results.pipe_output.get("working_memory") <span class="dim"># …or /execute blocking shape</span>+ for entry in working_memory["root"].values(): <span class="dim"># heuristic: first dict wins</span>+ ...</code></pre>+ <p>Two distinct flaws feed this: <b><code>main_stuff</code> is optional</b>, and <b>the two execution paths return different shapes</b>. This note covers the fix for the first and the plan for the second.</p>+ </section>++ <section>+ <p class="sec-kicker">Root cause</p>+ <h2>One pipe doesn't stamp the main stuff</h2>+ <p>Every operator (<code>PipeLLM</code>, <code>PipeExtract</code>, <code>PipeCompose</code>, …) stamps its result as the working memory's <code>main_stuff</code>. So do <code>PipeBatch</code> and <code>PipeCondition</code>. The single exception: <b><code>PipeParallel</code> without <code>combined_output</code></b>. It drops each branch result into memory by name and leaves the <code>main_stuff</code> pointer untouched — and the language explicitly allows that as a pipeline's last step.</p>+ </section>+ </div>++ <div class="wrap-wide">+ <figure>+ <div class="fig-frame">+ <svg viewBox="0 0 860 315" role="img" aria-label="Before: a sequence ending in PipeParallel leaves the main_stuff pointer on the previous step's output">+ <defs>+ <marker id="arrA" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0,0 L8,4 L0,8 z" fill="var(--muted)"/></marker>+ <marker id="arrAd" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0,0 L8,4 L0,8 z" fill="var(--danger)"/></marker>+ </defs>+ <text x="20" y="26" class="sv-t">Before — PipeSequence ending in a PipeParallel (add_each_output only)</text>++ <rect x="20" y="44" width="215" height="62" class="sv-box" rx="6"/>+ <text x="36" y="70" class="sv-t">1 · PipeLLM extract_info</text>+ <text x="36" y="90" class="sv-good-t">stamps main_stuff ✓</text>++ <line x1="235" y1="75" x2="285" y2="75" class="sv-flow" marker-end="url(#arrA)"/>++ <rect x="288" y="44" width="252" height="62" class="sv-box" rx="6"/>+ <text x="304" y="70" class="sv-t">2 · PipeParallel (last step)</text>+ <text x="304" y="90" class="sv-danger-t">never stamps ✗</text>++ <path d="M 380 106 C 380 140, 380 150, 462 172" class="sv-flow" marker-end="url(#arrA)"/>+ <path d="M 470 106 C 470 140, 545 145, 630 172" class="sv-flow" marker-end="url(#arrA)"/>+ <text x="500" y="140" class="sv-ts">branch results added by name</text>++ <rect x="20" y="176" width="820" height="86" class="sv-lane" rx="8"/>+ <text x="36" y="200" class="sv-ts">working memory after the run</text>++ <rect x="36" y="210" width="130" height="38" class="sv-slot" rx="5"/>+ <text x="50" y="234" class="sv-mono">input_text</text>+ <rect x="182" y="210" width="130" height="38" class="sv-slot" rx="5"/>+ <text x="196" y="234" class="sv-mono">info</text>+ <rect x="328" y="210" width="200" height="38" class="sv-slot" rx="5"/>+ <text x="342" y="234" class="sv-mono">short_summary</text>+ <rect x="544" y="210" width="200" height="38" class="sv-slot" rx="5"/>+ <text x="558" y="234" class="sv-mono">detailed_summary</text>++ <path d="M 120 292 C 170 292, 210 280, 240 254" class="sv-flow-danger" marker-end="url(#arrAd)"/>+ <text x="20" y="298" class="sv-mono">main_stuff</text>+ <text x="260" y="298" class="sv-danger-t">stale — still points at step 1's output</text>+ </svg>+ </div>+ <figcaption>The caller asks for <code>main_stuff</code> and silently receives step 1's output. When the parallel is the whole pipeline (API input path), the pointer doesn't exist at all.</figcaption>+ </figure>+ </div>++ <div class="wrap">+ <div class="modes-scroll">+ <table class="modes">+ <thead><tr><th>Failure mode</th><th>When it happens</th><th>What the caller sees</th></tr></thead>+ <tbody>+ <tr>+ <td><span class="pill pill-danger">Stale</span></td>+ <td>The parallel ends a <code>PipeSequence</code> — the shared memory keeps the previous step's stamp</td>+ <td>The <em>wrong</em> output, silently</td>+ </tr>+ <tr>+ <td><span class="pill pill-danger">Missing</span></td>+ <td>The parallel <em>is</em> the main pipe, inputs came through the API path (no input alias)</td>+ <td><code>main_stuff_name = null</code>; typed accessors raise</td>+ </tr>+ <tr>+ <td><span class="pill pill-danger">Empty envelope</span></td>+ <td>The <code>/start</code> → delivery path — delivery always writes <code>working_memory.json</code>, but the <code>main_stuff.*</code> artifacts are only written when a stamp exists</td>+ <td>A "successful" run with <code>main_stuff: null</code> — clients dig through the raw memory dump</td>+ </tr>+ </tbody>+ </table>+ </div>++ <section>+ <p class="sec-kicker">Decision</p>+ <h2>PipeParallel always combines — and <code>combined_output</code> dies</h2>+ <p>The rule: <b>a pipe run always delivers a main stuff</b>. We enforce it at the one leaf that breaks it — <code>PipeParallel</code> now always combines its branch results into its declared <code>output</code> concept and stamps that as main stuff. Sequences, conditions, and nesting then deliver the invariant for free, and the stale pointer is always overwritten.</p>+ <p>This also kills a redundant field. Today a combining parallel declares the same concept twice (<code>output</code> <em>and</em> <code>combined_output</code>), while a non-combining one declares an <code>output</code> that is a lie — it never produces it. After the change, <code>output</code> is the single, truthful declaration:</p>+ </section>++ <div class="duo">+ <div>+ <h3>Before — output is a lie</h3>+ <pre><code>[pipe.parallel_summarize]+type = "PipeParallel"+inputs = { input_text = "Text" }+<span class="del">output = "Text" # never produced</span>+add_each_output = true+branches = [+ { pipe = "sum_short", result = "short_summary" },+ { pipe = "sum_long", result = "detailed_summary" },+]</code></pre>+ </div>+ <div>+ <h3>After — combines into what it declares</h3>+ <pre><code>[pipe.parallel_summarize]+type = "PipeParallel"+inputs = { input_text = "Text" }+<span class="add">output = "Composite" # named combination</span>+add_each_output = true+branches = [+ { pipe = "sum_short", result = "short_summary" },+ { pipe = "sum_long", result = "detailed_summary" },+]</code></pre>+ </div>+ </div>++ <p>Declare a bespoke structured concept and the combination is fully typed (today's <code>combined_output</code> behavior, minus the duplicate line). Don't want to declare one? The new <b><code>native.Composite</code></b> concept holds the branch results under their names — the "dict of results" experience — while staying a first-class stuff that renders, serializes, and travels like any other. And a new <b>static validation</b> checks branch names against the declared structure at authoring time, so today's runtime combine error becomes a <code>/validate</code> error.</p>+ </div>++ <div class="wrap-wide">+ <figure>+ <div class="fig-frame">+ <svg viewBox="0 0 860 300" role="img" aria-label="After: PipeParallel combines branch outputs into its declared concept and stamps it as main_stuff">+ <defs>+ <marker id="arrB" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0,0 L8,4 L0,8 z" fill="var(--muted)"/></marker>+ <marker id="arrBg" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0,0 L8,4 L0,8 z" fill="var(--good)"/></marker>+ </defs>+ <text x="20" y="26" class="sv-t">After — the same pipeline under the new rule</text>++ <rect x="288" y="44" width="252" height="62" class="sv-box" rx="6"/>+ <text x="304" y="70" class="sv-t">2 · PipeParallel (last step)</text>+ <text x="304" y="90" class="sv-good-t">always combines ✓</text>++ <path d="M 350 106 C 330 135, 300 145, 250 168" class="sv-flow" marker-end="url(#arrB)"/>+ <path d="M 420 106 C 420 135, 420 145, 420 168" class="sv-flow" marker-end="url(#arrB)"/>+ <path d="M 490 106 C 510 135, 570 140, 640 166" class="sv-flow-good" marker-end="url(#arrBg)"/>+ <text x="530" y="136" class="sv-good-t">combine into declared output</text>++ <rect x="20" y="172" width="820" height="86" class="sv-lane" rx="8"/>+ <text x="36" y="196" class="sv-ts">working memory after the run</text>++ <rect x="36" y="206" width="170" height="38" class="sv-slot" rx="5"/>+ <text x="50" y="230" class="sv-mono">short_summary</text>+ <rect x="222" y="206" width="190" height="38" class="sv-slot" rx="5"/>+ <text x="236" y="230" class="sv-mono">detailed_summary</text>+ <rect x="428" y="206" width="396" height="38" class="sv-slot-good" rx="5"/>+ <text x="442" y="230" class="sv-mono">Composite { short_summary, detailed_summary }</text>++ <path d="M 120 288 C 200 290, 340 280, 425 240" class="sv-flow-good" marker-end="url(#arrBg)"/>+ <text x="20" y="294" class="sv-mono">main_stuff</text>+ <text x="440" y="292" class="sv-good-t">always stamped, always the last step's result</text>+ </svg>+ </div>+ <figcaption>Symmetry worth remembering: <code>PipeBatch</code> aggregates homogeneous results into a <em>list</em>; <code>PipeParallel</code> now aggregates heterogeneous results into a <em>named composite</em>.</figcaption>+ </figure>+ </div>++ <div class="wrap">+ <section>+ <p class="sec-kicker">Side track</p>+ <h2><code>/execute</code> and <code>/start</code> → delivery speak different shapes</h2>+ <p>The second flaw behind the ugly starter code: the two execution surfaces carry the <em>same information</em> in structurally different results. <b><code>POST /v1/execute</code></b> blocks until the run completes and responds with the runner's native envelope: everything nested under <code>pipe_output</code>, the working memory in its cross-process <em>transport</em> encoding (type-metadata keys, lists flattened — built for rehydration, not reading), and the main stuff not extracted — you follow the <code>main_stuff_name</code> pointer into <code>root</code> and unwrap <code>.content</code> yourself.</p>+ <p><b><code>POST /v1/start</code></b> returns immediately; the run then goes through <b>delivery</b>, which writes a folder of result artifacts to storage — <code>working_memory.json</code>, <code>main_stuff.{json,md,html}</code>, <code>main_stuff_viewer.html</code>, <code>graphspec.json</code>, mermaid/reactflow views — and <code>GET /v1/runs/{id}/results</code> relays three of them as flat fields: <code>main_stuff</code> (the output content, pre-extracted), <code>graph_spec</code>, and <code>working_memory</code> (a clean render). So it's not that one path lacks the memory — <em>both carry it</em>. What differs is the envelope, the main-stuff addressing, and the memory encoding — and the SDK only names a subset (<code>working_memory</code> rides along as an unnamed extra field). Every client normalizes.</p>+ </section>+ </div>++ <div class="wrap-wide">+ <figure>+ <div class="fig-frame">+ <svg viewBox="0 0 860 250" role="img" aria-label="Blocking /execute and /start with delivery return different result shapes, normalized in client code">+ <defs>+ <marker id="arrC" markerWidth="8" markerHeight="8" refX="7" refY="4" orient="auto"><path d="M0,0 L8,4 L0,8 z" fill="var(--muted)"/></marker>+ </defs>++ <rect x="20" y="30" width="110" height="150" class="sv-box" rx="6"/>+ <text x="45" y="100" class="sv-t">Client</text>+ <text x="41" y="118" class="sv-ts">app / SDK</text>++ <line x1="130" y1="70" x2="185" y2="70" class="sv-flow" marker-end="url(#arrC)"/>+ <rect x="188" y="44" width="250" height="52" class="sv-box" rx="6"/>+ <text x="204" y="66" class="sv-t">Blocking execution</text>+ <text x="204" y="84" class="sv-mono-dim">POST /v1/execute</text>++ <line x1="438" y1="70" x2="490" y2="70" class="sv-flow" marker-end="url(#arrC)"/>+ <rect x="493" y="44" width="240" height="52" class="sv-slot" rx="5"/>+ <text x="507" y="66" class="sv-mono">{ pipe_output: {working_memory},</text>+ <text x="507" y="84" class="sv-mono"> main_stuff_name }</text>+ <text x="507" y="112" class="sv-ts">transport encoding · main stuff via pointer</text>++ <line x1="130" y1="145" x2="185" y2="145" class="sv-flow" marker-end="url(#arrC)"/>+ <rect x="188" y="119" width="250" height="52" class="sv-box" rx="6"/>+ <text x="204" y="141" class="sv-t">Start → delivery</text>+ <text x="204" y="159" class="sv-mono-dim">POST /v1/start → GET …/results</text>++ <line x1="438" y1="145" x2="490" y2="145" class="sv-flow" marker-end="url(#arrC)"/>+ <rect x="493" y="119" width="240" height="52" class="sv-slot" rx="5"/>+ <text x="507" y="141" class="sv-mono">{ main_stuff, graph_spec,</text>+ <text x="507" y="159" class="sv-mono"> working_memory }</text>+ <text x="507" y="187" class="sv-ts">flat artifacts · main stuff pre-extracted</text>++ <path d="M 733 70 C 780 75, 790 95, 795 105" class="sv-flow" marker-end="url(#arrC)"/>+ <path d="M 733 145 C 780 140, 790 122, 795 112" class="sv-flow" marker-end="url(#arrC)"/>+ <text x="620" y="215" class="sv-danger-t">same data, two shapes →</text>+ <text x="620" y="233" class="sv-danger-t">every client normalizes</text>+ </svg>+ </div>+ <figcaption>Companion track — not part of this refactor: converge on one result shape so SDK consumers read <code>main_stuff</code> the same way on both paths. Designed separately, after the invariant lands (it belongs to <code>pipelex-api</code> / <code>pipelex-platform</code> / SDK repos).</figcaption>+ </figure>+ </div>++ <div class="wrap">+ <section>+ <p class="sec-kicker">Rollout</p>+ <h2>Three phases, breaking by design</h2>+ <p>No backward compatibility, per house policy — <code>combined_output</code> disappears and existing <code>.mthds</code> files migrate (mechanically: delete the redundant line, or make the declared <code>output</code> truthful).</p>+ <ol class="phases">+ <li><b>Runtime + language surface</b> (pipelex repo): always-combine, delete the field, add <code>native.Composite</code> and the static branch-name check, migrate fixtures, regenerate the MTHDS schema.</li>+ <li><b>Enforce the invariant</b>: tighten the defensive <code>optional_main_stuff</code> branches (delivery, graph tracer, CLI, telemetry) and make <code>main_stuff_name</code> non-optional on the wire models.</li>+ <li><b>Cross-repo sweep</b> (gated on a pipelex release): MTHDS spec + JSON schema, <code>plxt</code> pinned schema, conformance, plugin skills, app/UI TypeScript types, cocode pipelines, docs sites, SDKs + starter.</li>+ </ol>+ </section>+ </div>+</main>++<footer>+ <div class="wrap">+ Full design with file-level inventory, rejected alternatives, and open questions: <code>wip/required-main-stuff.md</code> on <code>feature/Required-main-stuff</code>.+ </div>+</footer>diff --git a/wip/required-main-stuff.md b/wip/required-main-stuff.mdnew file mode 100644index 000000000..80ca92f06--- /dev/null+++ b/wip/required-main-stuff.md@@ -0,0 +1,206 @@+# Required main stuff — PipeParallel always combines++**Branch:** `feature/Required-main-stuff` · **Status:** Phases 1–2 COMPLETE (Checkpoint 2 cleared — invariant enforced at every in-repo post-run boundary: wire `main_stuff_name` required, delivery always writes `main_stuff.*`, defensive branches gone, `final_stuff_code` honored on the parallel's combined stuff; decisions & audit in `TODOS.md` → Checkpoint log). Next: Phase 3 (cross-repo sweep, GATED on merge to `main` + version cut).++## 1. Problem++A pipeline is the language's function abstraction, and every pipe already declares `output` in its blueprint — yet the runtime sometimes finishes a run without delivering a main stuff. The result is that `main_stuff` is optional everywhere, and every consumer pays for it with defensive code. The symptom that triggered this design: `pipelex-starter-python/my_project/run_output.py` needs a page of isinstance-walking heuristics just to read a pipeline's result.++### Verified mechanics (current state)++- Every operator (`PipeLLM`, `PipeCompose`, `PipeExtract`, `PipeImgGen`, `PipeSearch`, `PipeStructure`, `PipeFunc`, `PipeSignature`) unconditionally stamps its result via `WorkingMemory.set_new_main_stuff`.+- Among controllers: `PipeSequence` threads one evolving memory so the last step's stamp wins; `PipeBatch` stamps its aggregated `ListContent`; `PipeCondition` delegates to the chosen branch, which stamps.+- The **only** non-stamping path in the runtime is `PipeParallel` without `combined_output` (`pipe_parallel.py` — the `set_new_main_stuff` call sits inside `if self.combined_output`). The blueprint validator explicitly allows `add_each_output = true` alone, so this is a legal terminal step.+- `pipe_abstract.py` acknowledges the hole in a comment: *"main_stuff may not exist for pipes like PipeParallel with add_each_output=true"*.++### The three concrete failure modes++1. **Stale main stuff (silent wrong data).** Inside a `PipeSequence`, the parallel adds branch outputs by name into the shared memory but never touches the `main_stuff` alias. A pipeline ending in an `add_each`-only parallel silently reports the *previous* step's output (or even an input — `make_from_single_stuff` / `make_from_multiple_stuffs` alias an input as main stuff at construction) as its main result.+2. **Truly missing main stuff.** When such a parallel is the top-level `main_pipe` and memory came from `make_from_pipeline_inputs` (the API path — which sets no main-stuff alias), the run completes with no main stuff at all: `main_stuff_name` serializes as `None`, `PipeOutput.main_stuff` raises.+3. **Empty delivery envelope.** On the `/start` → delivery path, delivery always writes `working_memory.json`, but the `main_stuff.json/md/html/viewer` artifacts are only generated when a main stuff exists. An `add_each`-terminal method started via `POST /v1/start` completes "successfully" and `GET /v1/runs/{pipeline_run_id}/results` relays `main_stuff: null` — consumers are left digging through the raw `working_memory` render (which the SDKs don't even name — see below).++### Blast radius of the optionality++Defensive `get_optional_main_stuff` branches exist in the CLI pretty-printer, the CLI file/CSV writers, the agent CLI, OTel telemetry, the graph tracer, and the delivery executor. On the wire, `main_stuff_name: str | None` (bridge payloads, `PipelexRunResultExecute`), `RunResults.main_stuff: Any = None` in both SDKs, and downstream user projects re-implement the same heuristics.++### Related but out of scope++The starter's `run_output.py` ugliness has a second, independent cause: the **result-shape divergence** between the two execution surfaces. Both carry the same information, including the working memory — delivery writes a full artifact folder (`working_memory.json`, `main_stuff.{json,md,html}`, `main_stuff_viewer.html`, `graphspec.json`, mermaid/reactflow views) and `GET /v1/runs/{pipeline_run_id}/results` relays `main_stuff` + `graph_spec` + `working_memory` as flat fields — but the shapes differ on three axes:++- **Envelope**: `/execute` nests everything under `pipe_output`; `/results` returns flat top-level fields.+- **Main-stuff addressing**: `/results` pre-extracts the main stuff's *content* into `main_stuff`; on `/execute` the caller follows the `main_stuff_name` pointer into `working_memory.root` and unwraps `.content`.+- **Memory encoding**: `/execute` returns the `dump_for_transport()` shape (`__pipelex_class__` metadata, `ListContent` flattened — built for rehydration, not reading); delivery writes a clean `smart_dump()` render — except on the cross-process Temporal path, where it writes the raw transport dump it received (`working_memory_raw`), so even the delivered `working_memory.json` is not one stable shape today.++On top of that, the SDKs only *name* `main_stuff`, `graph_spec`, and `pipe_output` on their `RunResults`; the relayed `working_memory` rides along as an unnamed `extra="allow"` field. This refactor removes the optionality and the heuristic root-walk; unifying the shapes is a companion API/SDK change tracked separately (see §9, cross-repo follow-ups).++## 2. The decision++> **A pipe run always delivers a main stuff. Consequently, a pipeline always delivers a main stuff.**++This is enforced at the leaf cause, not the pipeline boundary: `PipeParallel` — the single non-stamping pipe — now **always combines its branch outputs into its declared `output` concept and stamps the combination as main stuff**. Composition (sequences, conditions, nesting) then gives the pipeline-level invariant for free, and the stale-stamp hazard disappears because the parallel always overwrites the alias.++`combined_output` **is deleted as a field**. It was redundant with `output`: in our own fixtures, pipes declare `output = "PgcCombinedResult"` and `combined_output = "PgcCombinedResult"` — the same value twice. Meanwhile `add_each`-only parallels declare an `output` that is a lie (e.g. `output = "Text"` for a pipe that produces named summaries). Killing the field makes `output` truthful and unique.++No backward compatibility, per house policy. Breaking changes are noted in the changelog; existing `.mthds` files migrate.++## 3. New PipeParallel semantics++### Field changes++| Field | Before | After |+|---|---|---|+| `output` | Required but meaningless when only `add_each_output` is set | Required and truthful: the concept the branch outputs are combined into |+| `combined_output` | Optional concept ref; triggers combination when set | **Deleted** |+| `add_each_output` | One of the two must be set (model validator) | Unchanged meaning ("also expose branch outputs by name in memory"), default `false`, no more one-of-two validator |+| `branches` | Each branch must declare a unique `result` name | Unchanged |++### Runtime behavior (live and dry run identically)++1. Run all branches on deep copies of the memory, as today.+2. If `add_each_output`, add each branch's main stuff into the parent memory under its `result` name, as today.+3. **Always** build the combination: `{result_name: branch_main_stuff.content}` → validate into the declared `output` concept's structure class via `StuffFactory.combine_stuffs` → `set_new_main_stuff(stuff=combined, name=output_name)`. This is today's `combined_output` code path made unconditional, with the concept taken from `output`.+4. Graph tracer registration (branch outputs, parallel-combine edges) as today, except the combine edges are now always registered.++### Typing rules for the declared `output`++- **Structured concept** (bespoke, with a structure class): the combination validates into it — today's typed `combined_output` path, unchanged. The structure's fields must correspond to the branch `result` names.+- **`native.Composite`** (new native concept, see §4): the untyped escape hatch — a named composition holding the branch contents as-is, the `dict[str, Any]`-shaped developer experience for authors who don't want to declare a bespoke concept.+- **Anything else is a static validation error**: native non-composite concepts (`Text`, `Image`, …), `Dynamic`, `Anything`, and any multiplicity suffix (`Foo[]`, `Foo[N]`) are rejected — a parallel combination is a named composite, never a scalar or a bare list (a list aggregation is `PipeBatch`'s shape).++### New static validation (bundle/library validation time)++- The declared `output` of a `PipeParallel` must be `Composite` or a structured concept whose declared fields are compatible with the branch `result` names (required fields ⊆ result names; result names ⊆ declared fields). This turns today's runtime `StuffFactoryError` from `combine_stuffs` into an author-time error, which the builder and `/validate` can surface.+- The one-of-two (`add_each_output` / `combined_output`) validators in both the blueprint and the spec are removed.++### Example migration++Before (`add_each`-only, lying output):++```toml+[pipe.parallel_summarize]+type = "PipeParallel"+description = "Generate short and detailed summaries in parallel"+inputs = { input_text = "Text" }+output = "Text" # never produced — a lie+add_each_output = true+branches = [+ { pipe = "summarize_short", result = "short_summary" },+ { pipe = "summarize_detailed", result = "detailed_summary" },+]+```++After (untyped composite — minimal migration):++```toml+[pipe.parallel_summarize]+type = "PipeParallel"+description = "Generate short and detailed summaries in parallel"+inputs = { input_text = "Text" }+output = "Composite" # truthful: named composition of the branch results+add_each_output = true # still exposes short_summary / detailed_summary by name+branches = [+ { pipe = "summarize_short", result = "short_summary" },+ { pipe = "summarize_detailed", result = "detailed_summary" },+]+```++Before (combined, redundant declaration):++```toml+output = "PgcCombinedResult"+add_each_output = true+combined_output = "PgcCombinedResult"+```++After — delete the `combined_output` line; nothing else changes.++## 4. The `Composite` native concept++The untyped fallback needs a real vehicle. Verified: `DynamicContent` cannot serve as-is — `StuffContent`/`CustomBaseModel` use default pydantic config, so `model_validate({"short_summary": ..., ...})` silently *drops* unknown fields. Silent data loss is disqualifying; overloading `Dynamic` (whose meaning is "concept resolved at runtime", used by `PipeFunc`) would also blur two distinct semantics.++Implemented design: `COMPOSITE = "Composite"` on `NativeConceptCode` with a `CompositeContent(StuffContent)` structure class that holds the named branch contents as top-level fields (pydantic `extra="allow"` so the branch names surface at the top level of the serialized content, exactly like the typed path — not nested under a wrapper key). It supports the full content surface: `smart_dump`, kajson round-trip (`__json_encode__` hands kajson the live components), transport round-trip (`dump_for_transport` stamps each component with `__pipelex_class__`/`__pipelex_module__` markers and hydration rebuilds typed contents), `rendered_markdown`/`rendered_html` (iterate named sub-contents), so `main_stuff.json/md/html/viewer` delivery works unchanged.++Brand note: this is MTHDS-language semantics, not a Pipelex runtime detail. The concept is `native.Composite` (no `pipelex_` anything), and the change belongs in the MTHDS spec (`mthds/` repo) in the same motion as killing `combined_output`.++Alternatives considered for the vehicle:++- `DynamicContent` with `extra="allow"` — rejected: `Dynamic` already means something else, and changing its config affects unrelated `PipeFunc` paths.+- `CompositeContent` with an explicit `composed: dict[str, StuffContent]` field — rejected: nests the payload under a wrapper key, so the wire shape of the untyped path would differ from the typed path.+- Runtime-synthesized anonymous pydantic model per parallel — rejected: the class wouldn't exist in the class registry on the other side of a process boundary, breaking rehydration.++## 5. What the invariant unlocks (follow-through tightening)++Once every pipe stamps, "a completed run has a main stuff" is guaranteed, and the following tighten from defensive to direct (in-repo):++- Graph tracer output-spec branch in `pipe_abstract.py` (and its apologetic comment).+- Delivery executor: always writes the `main_stuff.*` artifact files for completed runs.+- CLI run cores (bare + agent): drop the "no main output produced" branches for live runs.+- OTel telemetry attribute extraction.+- `resolve_main_stuff_root_key` / bridge payloads / `PipelexRunResultExecute.main_stuff_name`: producer-side always set; the field types can go non-optional (Temporal payload history is not a concern — never shipped to prod).++`WorkingMemory.get_optional_main_stuff` itself stays — a *pre-run* or empty memory legitimately has no main stuff; it's the post-run boundary accessors that tighten.++Cross-repo (flagged, not done here): SDKs can drop `main_stuff`-absent handling for completed runs, and the starter's `find_main_content` shrinks; full cleanup of the two-branch shape additionally needs the `/execute`-vs-`/start`→delivery result-shape unification (companion track).++## 6. Rejected alternatives (for the record)++- **Auto-stamp at the pipeline boundary** ("if no main stuff at the end, synthesize one"): papers over the stale-stamp bug inside sequences and needs an ill-defined generic aggregation rule. Fixing the one non-stamping pipe is strictly better.+- **Validation-error-only** ("a terminal pipe must deliver; force authors to add `combined_output` or a final step"): position-dependent validity (valid as sub-pipe, invalid as `main_pipe`) is confusing, and it makes casual fan-out methods annoying to author.+- **Raw `dict[str, Any]` main stuff with no concept**: escapes the concept system entirely — no rendering, no typed access, no static checks. The `Composite` concept gives the same developer experience while staying a first-class stuff.+- **Keep `combined_output` optional, defaulting to `output`**: keeps a redundant field alive with subtle precedence rules; deleting it is simpler and the migration is mechanical.+- **SDK-only ergonomics fix**: treats the symptom; every consumer stays defensive; the empty-delivery hole remains.++## 7. Open questions (resolve during implementation)++- **`final_stuff_code`**: the parallel currently clears it on entry (with a copy-pasted `PipeBatch` log line). Now that the parallel stamps a final stuff, it should presumably honor `final_stuff_code` on the combined stuff like operators do. Confirm what `final_stuff_code` drives (graph/trace identity) and wire it through.+- **Graph `execution_data` key `combined_output_concept`**: under the new semantics it is always set and equals the declared output concept. Keep the key name for `mthds-ui` compatibility in phase 1; consider renaming in the next coordinated `mthds-ui` bump.+- **`add_each_output` naming**: semantics unchanged, but with combination always on, a name like `expose_branches` might read better. Recommendation: keep the name — pure churn across skills/docs otherwise. Revisit only if the MTHDS spec editors want it.+- **`PipeOutputAbstract` (mthds-python protocol layer)**: check whether the protocol shape encodes main-stuff optionality that should tighten in the same motion (cross-repo).++## 8. Implementation plan++### Phase 1 — runtime + language surface (this repo)++- `pipe_parallel.py`: unconditional combine in `_live_run_controller_pipe` and `_dry_run_controller_pipe` (shared helper — the existing TODO already asks for it); concept from `self.output`; always register parallel-combine graph edges.+- `pipe_parallel_blueprint.py` / `pipe_parallel_factory.py`: delete `combined_output` + one-of-two validator; factory stops resolving the combined concept separately.+- `builder/pipe/pipe_parallel_spec.py`: delete `combined_output` + validators; `to_blueprint()` simplifies; update pretty-render; `builder/operations/pipe_ops.py` + `cli/agent_cli/commands/pipe_cmd.py` display code.+- `core/bundles/pipelex_bundle_blueprint.py`: drop the `combined_output` concept-ref collection; add the new static structure-compatibility check (§3).+- New native concept `Composite` + `CompositeContent` (§4): `concept_native.py`, new `composite_content.py`, class registry, concept factory.+- Tests (TDD: write the red tests first): regression test for the stale-stamp bug (sequence ending in `add_each`-only parallel must deliver the composite, not the previous step's stuff); terminal-parallel-delivers-main-stuff; static validation accept/reject cases; `CompositeContent` serialization + rendering round-trip; dry-run parity. Migrate existing fixtures (`pipe_parallel_1.mthds`, `parallel_graph_*.mthds`, unit/builder test data, mermaidflow test, bundle-validator + agent-validate tests).+- Regenerate the MTHDS JSON Schema (`pipelex-dev generate-mthds-schema`).+- Docs: rewrite `docs/building-methods/pipes/pipe-controllers/PipeParallel.md` to the new semantics (current reality only, no history); changelog entry under `[Unreleased]`, marked breaking.+- Gates: `make agent-check`, targeted pipes tests, then full `make agent-test`.++**CHECKPOINT 1** — parallel always combines; `combined_output` gone; all gates green. Update this doc: status, decisions taken, deviations. Natural handoff point before touching the delivery/telemetry surfaces.++### Phase 2 — enforce the invariant across in-repo surfaces++- Tighten the boundary consumers listed in §5 (graph tracer, delivery executor, CLI run cores, OTel, serialization payload types → `main_stuff_name: str`).+- Keep `WorkingMemory.get_optional_main_stuff` for pre-run memories; audit each remaining call site and either justify or tighten.+- Resolve the `final_stuff_code` open question.+- Gates: full `make agent-test`.++**CHECKPOINT 2** — invariant enforced in-repo; doc updated with the final call-site audit results. Cross-repo work starts fresh from here.++### Phase 3 — cross-repo sweep (gated, coordinate releases)++Do not start before phases 1–2 are merged and a pipelex version is cut.++| Repo | Change |+|---|---|+| `mthds/` | Spec: `mthds-format.md`, `pipes-controllers.md`, `validation-rules.md`, `namespace-resolution.md`, `cross-package-references.md`, `docs/mthds_schema.json` — remove `combined_output`, define always-combine + `native.Composite` |+| `vscode-pipelex/` | Pinned `mthds_schema.json` in taplo-common (plxt lint/format must accept the new shape and reject `combined_output`) |+| `conformance/` | `tests/pipelex/test_validate_subcommands.py` fixture updates; keep spec ↔ test links (`make check-spec-links`) |+| `mthds-plugins/` | Skills references: `mthds-reference.md`, `build-phases.md`, `recursive-cheat-sheet.md`, `mthds-run/SKILL.md` across plugin variants |+| `pipelex-app/` | `src/types/core/pipes/pipe_controllers/pipe-parallel.ts` |+| `mthds-ui/` | `src/graph/types.ts` (`combined_output_concept` exec-data key — see open question); graph-spec data fixtures |+| `cocode/` | `ai_instruction_update.mthds`, `swe_docs.mthds` migration |+| `pipelex-website/` | `docs/mthds-doc.md` |+| demo/workshop repos | `illustration_generator/bundle.mthds` copies |+| SDKs + starter | Tighten `main_stuff` handling for completed runs; pair with the result-shape unification companion track |++**CHECKPOINT 3** — cross-repo consumers aligned; close this doc or fold residual items into follow-up trackers.++## 9. Companion track (separate design)++Result-shape unification between `/execute` and `/start` → delivery. Both surfaces already carry the same information, working memory included; what differs is the envelope, the main-stuff addressing, and the memory encoding (see §1). Converge on one shape: e.g. `/execute` also pre-extracts `main_stuff` at top level, one canonical working-memory render on both surfaces (the transport dump stays an internal format), and the SDKs naming `working_memory` on the results surface instead of leaving it an unnamed extra. This is what deletes the *second* half of the starter's `find_main_content`. Belongs to `pipelex-api` / `pipelex-platform` / SDK repos; design separately once the invariant lands.diff --git a/wip/required-main-stuff/deferred-phase-1-review.md b/wip/required-main-stuff/deferred-phase-1-review.mdnew file mode 100644index 000000000..55f7919e7--- /dev/null+++ b/wip/required-main-stuff/deferred-phase-1-review.md@@ -0,0 +1,11 @@+# Deferred items — Phase 1 code review (Checkpoint 1)++Context-free review of commit `757146324` (Phase 1: always-combine PipeParallel). No correctness bugs found. Triage outcome of the two cleanup findings:++## Fixed at checkpoint++- **Duplicated native-concept rejection match** between `PipeParallel.validate_output_static` and `PipeParallelBlueprint.validate_output` — extracted into `NativeConceptCode.is_composite` (exhaustive match lives once, on the enum, per house style). Both validators now branch on the property.++## Deferred (design tradeoff, not a bug)++- **Double-parse of the pipe output string in blueprint validation.** `PipeBlueprint.generic_validate_output` parses `self.output` with `parse_concept_with_multiplicity`, then `PipeParallelBlueprint.validate_output` parses the same string again. Avoiding it means changing the `validate_output()` hook signature across every blueprint subclass to thread the `MultiplicityParseResult` through. This runs once per pipe definition at load time — not a hot path — so the signature churn across all pipe blueprints isn't worth it today. Revisit only if a blueprint-validation hook redesign happens for other reasons.diff --git a/wip/required-main-stuff/deferred-review-round-3.md b/wip/required-main-stuff/deferred-review-round-3.mdnew file mode 100644index 000000000..d3015626c--- /dev/null+++ b/wip/required-main-stuff/deferred-review-round-3.md@@ -0,0 +1,40 @@+# Deferred items — review round 3 (`/review` on PR #1014, pre-merge)++Independent context-free review (own critical pass + testing, maintainability, and adversarial sub-agents) on the full `dev..HEAD` diff. It converged on **one genuine latent defect** (F1), one verified robustness gap (F2), and cleanup items. The one test gap deemed worth closing in-PR (`DirectOrchestrator.start()` raise-path) was applied — see `tests/unit/pipelex/runtime_bridge/test_direct_orchestrator.py`. Everything below was **deliberately deferred** to keep the merge-ready PR frozen, since none is reachable in production today.++## F1 — Nested `CompositeContent` transport round-trip is asymmetric (latent silent corruption)++**Where:** encode `pipelex/core/memory/working_memory.py` (`_encode_composite_for_transport` / `_encode_content_with_class_markers`) vs decode `pipelex/runtime_bridge/primitives/hydration.py` (`_hydrate_composite_component` → `_hydrate_list_item`).++**What:** Encode recurses into a component that is itself a `CompositeContent` (stamping `__pipelex_class__="CompositeContent"`). Decode does **not** recurse: `_hydrate_list_item` resolves the `CompositeContent` class and calls `model_validate(clean_item)` without re-hydrating the inner components. Because `CompositeContent` is `extra="allow"`, the still-encoded inner components are stored verbatim — as raw `dict`s that **retain the pipelex-private `__pipelex_class__` / `__pipelex_module__` marker keys**.++**Failure scenario:** a nested `PipeParallel` (a branch whose own output is `Composite`), or a `CompositeContent` carried as a `ListContent` item, crosses the Temporal `dump_for_transport` → `hydrate_working_memory` boundary. The inner composite's components decay to `dict` (typed `.content_as(...)` / field access breaks) **and** `smart_dump()` / `rendered_markdown()` of that inner composite leaks the private marker keys into user/LLM-visible output. Silent — no exception at hydrate time. Confirmed by reproduction in the adversarial pass; the flat (depth-1) and `ListContent`-of-scalars cases are correct and tested — only the *nested composite* case is affected.++**Why deferred:** the Temporal/cross-process transport path has never shipped to production (see workspace memory `project_temporal_not_shipped`), and nested-parallel composition is an edge case. The corrupting code does ship in this PR, so this must be fixed **before the hosted-Temporal rollout**, not left indefinitely.++**Fix (small, localized):** in `_hydrate_list_item`, when the resolved class is a `CompositeContent` subclass, rebuild its components via `_hydrate_composite_component` before `model_validate` (mutual recursion; terminates on finite nesting). Add a nested round-trip test to `tests/unit/pipelex/core/stuffs/test_composite_content.py`: a composite whose component is a composite, asserting the inner component entries hydrate to typed `StuffContent` and carry no `__pipelex_*` keys.++## F2 — `ConceptValueError` (a bare `ValueError`) escapes the validate sweep++**Where:** `pipelex/pipe_controllers/parallel/pipe_parallel.py:153,223` call `concept.get_structure_class()`, which raises `ConceptValueError` (`pipelex/core/concepts/exceptions.py` — subclasses `ValueError`, **not** `PipelexError`; its sibling `ConceptError` does subclass `PipelexError`) when a structure class is not registered.++**What:** the validate sweep's step-1 wiring pass (`pipelex/pipeline/bundle_validator.py` `validate_with_libraries()`) catches only `PipeNotFoundError`; the per-pipe step-3 catch is `(PipelexError, ValidationError, FactoryException)`. A `ConceptValueError` matches neither, so it escapes `validate_pipes` and aborts validation of the **whole bundle** with an unhandled error instead of being reported as that one pipe's FAILURE. `PipeSequence`'s equivalent path avoids `get_structure_class()` (it uses `is_compatible`), so this exposure is net-new to `PipeParallel`.++**Why deferred:** low reachability — a loaded pipe's own output concept normally has its structure class registered. The failure mode (unhandled sweep abort vs clean per-pipe error) is poor DX but not a data-integrity issue.++**Fix (cheap):** catch `ConceptValueError` in `validate_output_with_library` and re-raise as `PipeValidationError` (or widen the sweep's step-1 catch).++## M1 — `rendered_html` table wrapper still duplicated++**Where:** `pipelex/core/stuffs/composite_content.py` (`rendered_html`) + `pipelex/core/stuffs/structured_content.py` (`rendered_html`).++**What:** the PR extracted per-value rendering into `render_value_html` (good), but the *table-of-named-values* wrapper and the verbatim empty-table literal (`<table><tr><td><em>empty</em></td></tr></table>`) are still copy-pasted in both classes — the same drift the extraction targeted, one level up.++**Fix:** add a shared `render_named_values_table(pairs)` helper in `html_rendering.py`; `StructuredContent` passes its non-`None` fields, `CompositeContent` passes all components. Pure refactor, no behavior change.++## Minor / noted (no action)++- **`combined_output_concept` graph execution-data key name** (`pipe_parallel.py`): now always `self.output.concept.concept_ref`. The misleading-name observation is correct, but the key is **deliberately kept** for `mthds-ui` compatibility — rename is already tracked as a Phase 3 cross-repo item (design open-question #2). Renaming before `mthds-ui`'s coordinated update would break it. Do not rename in-repo.+- **Unguarded `get_required_pipe` in `_validate_branch_output_types`** (`pipe_parallel.py`): asymmetric with `PipeSequence`'s cross-package-guarded form, but benign in the sweep (which anticipates unguarded sub-pipe resolution and catches `PipeNotFoundError`; `needed_inputs` would raise first anyway). Consistency nit for a future refactor.+- **`PipelexRunResultExecute.from_pipe_output`** (`pipeline/pipeline_response.py`): accepts non-`COMPLETED` states but unconditionally calls the now-raising `resolve_main_stuff_root_key`. Latent only — the sole caller passes `state=COMPLETED` on the success path. A future caller routing a failed `PipeOutput` here would have the real failure masked by a secondary `PipeJobError`. Tighten by asserting completion or dropping the `state` param if a caller ever needs the failed path.+- **Duplicate branch-result-name detection** stays a runtime guard (`validate_output_with_library` builds `result_names` as a set, silently collapsing duplicates; the runtime duplicate guard with its stale `# TODO` remains). Design tradeoff, not a bug — the runtime guard still catches it.