Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Improve graph rendering and validation pipeline handling (#1004)
Louis Choquel5fc5ee450f3d
diff --git a/.gitignore b/.gitignoreindex c9fe49d82..f767b4f8a 100644--- a/.gitignore+++ b/.gitignore@@ -43,6 +43,7 @@ reports/results/# personnal pipelex config files that overrides the default one+.worktreeincludepipelex_super.tomlpipelex_override.tomlpipelex_temporary_override.tomldiff --git a/TODOS.md b/TODOS.mdindex 4e6a3ae8a..16e2d8466 100644--- a/TODOS.md+++ b/TODOS.md@@ -1,179 +1,91 @@-# Remove the `ImgGenPrompt` native concept+# Active plan — GraphSpec source paths + explicit validate graph target-**Goal:** Delete `NativeConceptCode.IMG_GEN_PROMPT` (`"ImgGenPrompt"`) — the native *concept* `native.ImgGenPrompt`. It has no business being a native concept: it is structurally identical to `Text` (just a renamed alias), and nothing in the runtime requires it.+> **Status: ready to start (2026-06-23).** This is a two-part Pipelex runtime graph/validation plan. Work from the Pipelex repo root (`/Users/lchoquel/repos/Pipelex/pipelex`). Start by reading the two design notes: `wip/graph/graphspec-source-map-enrichment.md` and `wip/graph/validate-explicit-graph-target.md`. Implement them in that order: source-map enrichment first, explicit graph target second.-**Status:** Not started (plan approved, awaiting go-ahead to execute). Single-session task, low risk, bounded blast radius.+## Cold-Start Context-**Branch:** create a feature branch off `dev` (e.g. `refactor/drop-imggenprompt-native-concept`) — do not work directly on `dev`.+Today `GraphSpec` is an execution trace for one root pipe. It has `nodes[]`, `edges[]`, `pipe_registry`, and `concept_registry`. It does not currently include the `.mthds` bundle path where each pipe or concept was declared. The path information exists elsewhere: `LibraryCrate.source_map` maps `domain.pipe_code` and `domain.ConceptCode` to source paths, `LibraryManager.get_pipe_source()` exposes pipe origins for diagnostics, and `PipeJob.library_crate` is already attached by `prepare_pipe_job()` and crosses the Temporal boundary.-**Decisions confirmed (settled — do not re-litigate):**+Validation graph generation also has a separate root-selection constraint. The validation report graph arm currently uses `select_primary_blueprint(result.blueprints).main_pipe_ref`: first blueprint declaring `main_pipe`, else `None`. `best_effort_graph_spec(pipe_ref=None, ...)` returns `None`, so protocol/API validation succeeds without a graph when no `main_pipe` exists. CLI graph/view helpers are stricter: `dry_run_pipeline()` raises `PipelexInterpreterError("Bundle does not declare a main_pipe, cannot generate graph")` when graph output is explicitly requested and no `main_pipe` exists.-- Test fixture `multiplicity.mthds` outputs migrate to `Text` (not a locally-redefined `ImgGenPrompt` concept).-- `PipeImgGen.md` doc examples migrate their input concept from `ImgGenPrompt` → `Text` (full doc cleanup, not minimal).+The product direction is: keep `GraphSpec` as an execution-trace artifact, but make it more useful. First, enrich existing registries with declaration source paths. Second, let callers request a graph for a specific pipe, so bundles without `main_pipe` can still validate and graph a selected pipe.----+## Work Order-## Cold-start context (read this first)+1. **Source-map enrichment first.** This is additive and low-risk. It uses existing `LibraryCrate.source_map` and existing free-form registry payloads. It should not change root selection or validation behavior.+2. **Explicit graph target second.** This changes validation/graph routing across direct, Temporal/API, and CLI surfaces. It benefits from the source-enriched registries once graphs can target non-`main_pipe` pipes.-### What `native.ImgGenPrompt` actually is+## Phase 1 — Enrich GraphSpec Registries With Source Paths-It's a native concept that carries **zero structural distinction** from `Text`:+Goal: when `pipe_and_concept_registry` is enabled and a `LibraryCrate.source_map` entry exists, `graph_spec.pipe_registry[pipe_ref]["source"]` and `graph_spec.concept_registry[concept_ref]["source"]` carry the source bundle path. If source is unavailable, omit the field rather than emitting `source: null`.-- `NativeConceptCode.structure_class` returns `None` for it — no dedicated content class (`pipelex/core/concepts/native/concept_native.py:72-74`).-- `ConceptFactory.make_native_concept` maps it to **`TextContent`** with the description "A prompt for an image generator" (`pipelex/core/concepts/concept_factory.py:150-156`, the structure class name is literally `NativeConceptCode.TEXT.structure_class_name`).+Implementation shape:-So `native.ImgGenPrompt` is a semantic alias for `Text`. That's exactly why it doesn't belong as a built-in native concept — a user concept refining `Text` does the same job. **Migration for any consumer = replace `ImgGenPrompt` with `Text`.**+- In `pipelex/core/pipes/pipe_abstract.py`, add a source-aware pipe registry helper. It should start from `self.model_dump(mode="json")`, look up `library_crate.source_map.get(self.pipe_ref)`, and add `source` only when present.+- Update `_make_single_concept_data_for_registry()` and `_make_concept_data_for_registry()` to accept `library_crate: LibraryCrate | None`, look up `concept.concept_ref`, and add `source` only when present. Keep the existing `json_schema` behavior.+- In `_run_pipe_traced()`, replace `pipe_data = self.model_dump(mode="json")` and `concept_data = self._make_concept_data_for_registry()` with the source-aware helpers, passing the existing `library_crate` parameter.+- Do not add `source` to `NodeSpec`. A node is an invocation; the source path is declaration metadata, so it belongs in `pipe_registry` / `concept_registry`.+- Do not change `GraphSpec` model fields. The registries are already `dict[str, dict[str, Any]]`, and `PipeStartEvent.pipe_data` / `concept_data` already carry generic dictionaries.-### The name-collision trap (critical — do NOT over-scope)+Tests:-Three unrelated things share the name "ImgGenPrompt". **Only the first is being removed.** A naive grep audit conflates them and wildly overstates the blast radius ("requires refactoring the image-gen architecture" — this is FALSE).+- Add focused coverage proving registry entries include `source` when a fake or real `LibraryCrate.source_map` provides entries.+- Add a direct/in-process graph test using a real `.mthds` file loaded from disk: assert the pipe registry source is the bundle path and declared concept registry source is present.+- Add or extend Temporal/event-assembled coverage so `PipeStartEvent.pipe_data` / `concept_data` with `source` survive through `GraphSpecAssembler`.+- Add a sourceless path check: raw in-memory `mthds_contents` without sources still produces a graph and omits `source`.-| Thing | What it is | Action |-|---|---|---|-| `NativeConceptCode.IMG_GEN_PROMPT` | the native **concept** `native.ImgGenPrompt` | **REMOVE — the target** |-| `ImgGenPrompt` (BaseModel, `pipelex/cogt/img_gen/img_gen_prompt.py`) | runtime payload (positive/negative text + input images) that `PipeImgGen` builds and sends to the image generator | **LEAVE UNTOUCHED** |-| `TemplateCategory.IMG_GEN_PROMPT` (`pipelex/cogt/templating/template_category.py`) | Jinja2 template category for prompt rendering | **LEAVE UNTOUCHED — different enum** |+> **Checkpoint A — source paths in registries. DONE (2026-06-23).** Direct graph registry payloads now enrich pipes and concepts from `LibraryCrate.source_map`; event-assembled graph paths preserve `source` because `PipeStartEvent.pipe_data` / `concept_data` and `PipeEndSuccessEvent.output_concept_data` are copied unchanged by `GraphSpecAssembler`; sourceless in-memory bundle graphs still omit `source`. Edge-case decision: source enrichment uses only the executing job's active `LibraryCrate.source_map`; dependency or native entries with no source-map entry keep the previous behavior and omit `source` rather than emitting `source = null`. Verification: `CI=true .venv/bin/pytest tests/unit/pipelex/tracing/test_graphspec_assembler.py::TestGraphSpecAssembler::test_registry_source_payloads_survive_assembly tests/integration/pipelex/pipeline/test_dry_run_pipeline_graph.py::TestDryRunPipelineGraphTransport::test_graph_produced_with_tracing_disabled tests/integration/pipelex/pipeline/test_validate_graph_registry_sources.py -q` passed; `CI=true make agent-check` passed. `CI=true make agent-test` was attempted in the sandbox and reached unrelated environment failures (Temporal dev server `Operation not permitted`, restricted DNS/network HTTP tests, local HTTP server permission errors); unsandboxed rerun was rejected by approval policy because the broad suite contacts external services and starts local/networked servers.-Also leave untouched: `ImgGenPromptError` (`pipelex/cogt/exceptions.py`) and its docs — that's the exception class for the runtime model, unrelated to the native concept.+## Phase 2 — Add Explicit Graph Target Support-### How `PipeImgGen` actually consumes inputs (corrected mental model — the docs got this wrong)+Goal: callers can request graph generation for a specific pipe. The default remains unchanged: use the selected `main_pipe` when no explicit target is provided. If neither explicit target nor `main_pipe` exists, protocol/API validation should keep returning `graph_spec=None`, while CLI `--graph` / `--view` should keep failing clearly because the user explicitly requested a graph.-**`PipeImgGen` does NOT take an "`ImgGenPrompt` concept" as input.** The old docs implied you load a prompt into an `ImgGenPrompt`-typed stuff and feed it in. That framing is wrong and is a big reason this concept looked load-bearing when it isn't.+Implementation shape:-What actually happens:+- Add `graph_pipe_code: str | None = None` to `pipelex/pipeline/validate_in_process.py::validate_bundles_in_process()`. Compute `graph_target_ref = graph_pipe_code or select_primary_blueprint(result.blueprints).main_pipe_ref`, then pass it to `best_effort_graph_spec()`.+- Decide how `PipelexMTHDSProtocol.validate()` exposes this. If using protocol `extra`, validate and accept only a known key such as `graph_pipe_code`; otherwise keep protocol unchanged and expose the option only on concrete API/CLI surfaces.+- Temporal path already has `DryValidateArg.pipe_code`; ensure the hosted/API dispatch layer maps the explicit graph target into it. Rename only if the surrounding API request field also uses `graph_pipe_code`; avoid churn otherwise.+- Update `pipelex/pipeline/dry_run_pipeline.py` to accept `pipe_code: str | None = None`. If provided, execute that pipe; otherwise preserve the current `main_pipe` selection and missing-main error.+- Update `pipelex/graph/graph_rendering.py` helpers (`_dry_run_bundle()`, `generate_graph_for_bundle()`, `generate_view_for_bundle()`) to accept/pass the pipe override.+- Update `pipelex/cli/agent_cli/commands/validate/bundle_cmd.py` so `validate bundle --pipe X --graph` and `--view` graph `X` instead of ignoring `--pipe` and falling back to `main_pipe`.+- Decide and document bare vs qualified target behavior. Recommended: accept both; bare targets must be unique in the loaded library; existing ambiguity errors can surface on explicit CLI graph requests and degrade on best-effort report paths.-- `PipeImgGenBlueprint` (`pipelex/pipe_operators/img_gen/pipe_img_gen_blueprint.py`) has a **required `prompt: str`** field (a Jinja2/sigil **template**) and an optional `negative_prompt: str`. There is no prompt-concept input.-- Declared `inputs = { ... }` are the variables the template may reference, and they get **injected into the template** at run time (`PipeImgGenFactory.make` → `pipe_img_gen_factory.py:41-93`, then `ImgGenPromptBlueprint.make_img_gen_prompt` → `img_gen_prompt_blueprint.py:65-196`):- - **Text inputs** are interpolated into the prompt string via `$var` / Jinja.- - **Image inputs** are detected by `TemplateImageAnalyzer.analyze_template_for_images` (`pipelex/pipe_operators/shared/template_image_analyzer.py`), classified as `DIRECT` (one image), `DIRECT_LIST` (a list/tuple of images), or `NESTED` (a dotted path like `page.page_view`). At run time each referenced `ImageContent` is pulled from working memory, registered in an `ImageRegistry`, the reference in the text is replaced with an `[Image N]` placeholder token, and every image is collected into `ImgGenPrompt.input_images`.-- This is **exactly the vision pattern** used for image inputs to `PipeLLM`: images declared as inputs and referenced in the prompt are extracted and passed alongside the rendered text. It enables image-to-image, reference-image, and image-editing generation (e.g. Flux / GPT-Image with reference images), bounded by the model's `max_prompt_images`.-- The resulting runtime `ImgGenPrompt` model (`positive_text`, `negative_text`, `input_images`) is what's sent to the generator.+Tests:-So `PipeImgGen`'s only concept constraints are: declared image inputs must be `Image`-compatible (enforced at extraction time by the `ImageContent` type check), and its **output** must be `native.Image`-compatible (`pipe_img_gen.py:116-134`). It has **no** input-concept tie to `ImgGenPrompt`. The `IMG_GEN_PROMPT` symbols inside `pipe_img_gen_factory.py` / `pipe_img_gen_blueprint.py` are all `TemplateCategory.IMG_GEN_PROMPT` (the Jinja2 category), not the native concept. **Removing the native concept does not touch image generation behavior.**+- Protocol/direct validation: no `main_pipe` plus explicit graph target produces non-null `graph_spec`; explicit graph target overrides declared `main_pipe`; unknown explicit target degrades to `graph_spec=None` on best-effort validation reports.+- Agent CLI: `validate bundle no_main.mthds --pipe some_pipe --view` succeeds and includes `graphspec`; `validate bundle no_main.mthds --view` still errors; `validate bundle bundle_with_main.mthds --pipe other_pipe --view` graphs `other_pipe`.+- Temporal/API path: explicit graph target maps to `DryValidateArg.pipe_code`; direct and Temporal modes agree on the target behavior.-Reference example to base accurate docs on: `tests/integration/pipelex/pipes/img_gen_prompt_inputs/test_img_gen_prompt_image_extraction.py` (exercises image injection into the prompt).+> **Checkpoint B — explicit graph target parity. DONE (2026-06-23).** Direct/protocol validation now accepts `extra = {"graph_pipe_code": "..."}` as the only local runtime validate extension and passes it to `validate_bundles_in_process(graph_pipe_code=...)`; unknown, non-string, and blank validate extension args are rejected. The shared in-process graph arm targets `graph_pipe_code` first, else the selected `main_pipe`, else keeps `graph_spec = None`. CLI graph/view generation now passes `validate bundle --pipe X` into `generate_graph_for_bundle(..., pipe_code=X)` / `generate_view_for_bundle(..., pipe_code=X)`, and `dry_run_pipeline(..., pipe_code=X)` can graph bundles without `main_pipe`. Bare explicit dry-run targets now return the resolved domain-qualified graph target, so callers do not report the original ambiguous ref. Temporal keeps the existing wire field `DryValidateArg.pipe_code` as the explicit graph target to avoid churn; no field rename was made. Bare and qualified targets are both accepted via the existing pipe-library resolver; ambiguous or missing explicit targets degrade on best-effort validation report paths and surface as graph/view errors on explicit CLI graph requests. Tests were added for direct/protocol no-main, override, blank, and missing-target behavior; dry-run no-main explicit target; CLI pipe override passthrough; graph-rendering passthrough; and Temporal no-main/override behavior. Per user instruction, pytest was not run for this phase. Verification run: `CI=true make agent-check` passed.-### Why removal is allowed+## Non-Goals-Per workspace `CLAUDE.md`: "No backward compatibility … Breaking changes must be noted in changelogs but there is no deprecation transition period." Removing a native concept is a breaking change for any out-in-the-wild `.mthds` using `native.ImgGenPrompt` or `refines = "ImgGenPrompt"`, but that's acceptable with a CHANGELOG note. Migration is trivial (`→ Text`).+- Do not infer a root from "the only pipe in the bundle" unless a product decision explicitly asks for that. It creates a hidden second default alongside `main_pipe`.+- Do not create a static all-pipes `GraphSpec` under the same field. That would overload an execution-trace artifact with non-execution semantics.+- Do not require `main_pipe` to validate a bundle. Current validation correctly supports concept-only files, membership-only files, and top-down partial bundles.+- Do not move declaration source onto runtime `PipeAbstract` / `Concept` models unless registry enrichment proves insufficient. The source-map already exists on `LibraryCrate`.-### Scope facts already verified (so you don't re-check)+## Verification Guidance-- **Not** present in `derived/mthds_schema.json` → **no `pipelex-dev generate-mthds-schema` regen needed.**-- No shipped/library/kit/cookbook/methods/test-bed `.mthds` uses it as a concept. The **only** `.mthds` usage anywhere is one test fixture (see step 3).-- `tests/unit/pipelex/core/concepts/test_concept.py:101` iterates `NativeConceptCode.values_list()` dynamically → adapts automatically, no edit needed.-- No test hardcodes the native-concept count or the `IMG_GEN_PROMPT` member by name.+Follow Pipelex repo standards: use `make agent-test`, not `make test`. For narrow iterations, run the smallest relevant pytest path through the repo's supported agent-test entrypoint if available, then run broader graph/validation coverage before handing off. Likely target areas: `tests/unit/pipelex/graph/`, `tests/integration/pipelex/pipeline/`, `tests/integration/pipelex/temporal/`, and agent CLI validate tests. If implementation touches hosted API request mapping, also update and verify the matching `../pipelex-api` surface in a separate repo change.----+## Files Likely To Touch-## Implementation checklist+- `pipelex/core/pipes/pipe_abstract.py`+- `pipelex/pipeline/validate_in_process.py`+- `pipelex/pipeline/runner.py`+- `pipelex/pipeline/dry_run_pipeline.py`+- `pipelex/graph/graph_rendering.py`+- `pipelex/cli/agent_cli/commands/validate/bundle_cmd.py`+- `pipelex/temporal/tprl_pipe/act_dry_validate.py` only if field naming changes+- tests under `tests/unit/pipelex/graph/`, `tests/integration/pipelex/pipeline/`, `tests/integration/pipelex/temporal/`, and agent CLI validate coverage+- hosted API validate request/dispatch mapping in `../pipelex-api` if this is exposed over the public API-### 1. Code — `pipelex/core/concepts/native/concept_native.py`+## Definition Of Done-These three `match`/`case` blocks are **exhaustive with no `case _:` allowed** (project rule), so the member must be removed from the enum *and* from every arm in lockstep, or linting fails.--- [ ] Remove the enum member: line 26 `IMG_GEN_PROMPT = "ImgGenPrompt"`-- [ ] `structure_class` property: collapse `case NativeConceptCode.IMG_GEN_PROMPT | NativeConceptCode.ANYTHING:` (line ~72) → `case NativeConceptCode.ANYTHING:` (still returns `None`)-- [ ] `is_text_concept` classmethod: drop `| NativeConceptCode.IMG_GEN_PROMPT` from the `False` alternation (line ~123)-- [ ] `is_dynamic_concept` classmethod: drop `| NativeConceptCode.IMG_GEN_PROMPT` from the `False` alternation (line ~146)--### 2. Code — `pipelex/core/concepts/concept_factory.py`--- [ ] Delete the `case NativeConceptCode.IMG_GEN_PROMPT:` arm at lines ~150-156 (otherwise `make_native_concept`'s match is non-exhaustive and references a dead member)--### 3. Test fixture — the only real runtime break--`tests/integration/pipelex/pipes/pipelines/multiplicity.mthds` outputs `ImgGenPrompt` in three pipes; after removal the bundle fails to resolve the concept at test load time.--- [ ] Change `output = "ImgGenPrompt"` → `output = "Text"` at lines 44, 56, 66 (it was structurally `Text` anyway; the prompts produce a sentence of text)--### 4. Spec authoring metadata — `pipelex/builder/concept/concept_spec.py`--Not enforced and won't break linting, but leaving it advertises a concept that no longer exists to the spec/builder agent.--- [ ] Remove `ImgGenPrompt` from the `refines` field description string (line ~250) and from the `examples=[...]` list (line ~253)--### 5. Docs (this repo) — update `docs/`--This step has two parts: (a) the mechanical removal of the native concept from concept reference docs, and (b) a **substantive rewrite** of the PipeImgGen docs to teach the correct prompt-template + input-injection model (see "How `PipeImgGen` actually consumes inputs" above).--**(a) Concept-reference removal (mechanical):**--- [ ] `docs/building-methods/concepts/native-concepts.md` — remove the table row at line 37 (`| `ImgGenPrompt` | … |`) AND the `### ImgGenPrompt` section (lines 186-190)-- [ ] `docs/building-methods/concepts/define_your_concepts.md:150` — remove `ImgGenPrompt` from the inline list; also **drop the hardcoded "12"** count (project writing rule: no hardcoded counts) — rephrase to "Pipelex includes these built-in native concepts: …"-- [ ] `docs/features/concepts.md:25` — remove the `**ImgGenPrompt** — …` bullet--**(b) PipeImgGen rewrite (substantive — fix the misleading "feed it an ImgGenPrompt concept" framing):**--- [ ] `docs/building-methods/pipes/pipe-operators/PipeImgGen.md`:- - "How it works" (lines 9-13) — rewrite to state the truth: `PipeImgGen` takes a **`prompt` string template** (+ optional `negative_prompt`); declared `inputs` are **injected into the template** — text via `$var` interpolation, images via reference-and-inject (the vision pattern). It does not consume a dedicated prompt concept.- - Example inputs currently typed `"ImgGenPrompt"` (lines 28, 39, 101, 115, 129) → `"Text"`. These show *text* prompt inputs, so `Text` is the correct concept.- - **Add a new example: image inputs (image-to-image / reference image).** Show `inputs = { ref = "Image" }` (and a list variant, e.g. `refs = "Image[]"`), reference the image(s) in the `prompt` template, and explain that each referenced image is injected as an `[Image N]` token and passed to the generator — bounded by the model's `max_prompt_images`. Base it on `tests/integration/pipelex/pipes/img_gen_prompt_inputs/test_img_gen_prompt_image_extraction.py` so it's accurate.- - Line 138 prose ("you would first load a text prompt … into the input stuff (`ImgGenPrompt` concept)") — rewrite: the input is an ordinary `Text` stuff referenced by the `prompt` template; there is no `ImgGenPrompt` concept.-- [ ] `docs/features/image-generation.md:40` — reword "(or an ImgGenPrompt concept)". Replace with the correct model: PipeImgGen takes a prompt template and can inject both text and image inputs (reference images for image-to-image). Do not imply a prompt-concept input.-- [ ] **Do NOT touch** `docs/errors/img-gen-prompt-error.md` or `docs/errors/inference-and-providers.md:34` — those document `ImgGenPromptError` (the runtime exception), which stays.--### 6. CHANGELOG--- [ ] Add a `### Removed` bullet under `## [Unreleased]` in `CHANGELOG.md` (there is already a `### Removed` section there — append to it). Draft:-- > **Native concept `ImgGenPrompt` (pre-1.0 breaking):** Removed `native.ImgGenPrompt`. It was structurally identical to `Text` (no dedicated content class; the factory mapped it to `TextContent`), so it added a brand-new built-in concept with no semantic payload. `PipeImgGen` never depended on it — its inputs are ordinary `Text`-compatible variables and its only concept guard is an `Image`-compatible output. Migration: replace `ImgGenPrompt` / `refines = "ImgGenPrompt"` with `Text` in any `.mthds`. Unrelated and unchanged: the `ImgGenPrompt` runtime model, the `TemplateCategory.IMG_GEN_PROMPT` template category, and `ImgGenPromptError`.--### 7. Verify--- [ ] `make agent-check` (lint/format/pyright/mypy/plxt — catches any non-exhaustive match)-- [ ] `make tb` (boot/config sanity; cheap)-- [ ] Targeted tests for the touched areas (core + pipes + builder), e.g.:- `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/unit/pipelex/core/ tests/integration/pipelex/core/ tests/integration/pipelex/pipes/ tests/unit/pipelex/builder/ tests/integration/pipelex/builder/`-- [ ] If broad/uncertain, fall back to full `make agent-test`-- [ ] Final grep sanity: `rg -n '\bImgGenPrompt\b' pipelex/ tests/ docs/` should return ONLY runtime-model / `TemplateCategory` / `ImgGenPromptError` hits — no `native.ImgGenPrompt` concept refs and no `NativeConceptCode.IMG_GEN_PROMPT`.--### 8. Land--- [ ] Commit on the feature branch, push, open PR to `dev`-- [ ] Hand off the recap below to downstream repos (see next section)-------## Handoff recap (paste into ../mthds-plugins and other repos)--> **Heads-up: `ImgGenPrompt` is no longer a Pipelex native concept (breaking, pre-1.0).**->-> As of pipelex `<next version>`, `native.ImgGenPrompt` has been removed from `NativeConceptCode`. It was a structural duplicate of `Text` (no dedicated content class — the factory mapped it to `TextContent`), so it was a built-in concept with no semantic payload beyond its name.->-> **What this means for you:** any `.mthds` that uses `ImgGenPrompt` as an input/output concept, or `refines = "ImgGenPrompt"`, will fail validation against the new pipelex. **Fix = replace `ImgGenPrompt` with `Text`.**->-> **What did NOT change** (do not "fix" these — they are unrelated things that happen to share the name):-> - the `ImgGenPrompt` runtime model (`pipelex.cogt.img_gen.img_gen_prompt.ImgGenPrompt`) — still how `PipeImgGen` builds the generator payload;-> - the `TemplateCategory.IMG_GEN_PROMPT` Jinja2 template category;-> - the `ImgGenPromptError` exception class.->-> **Corrected mental model for `PipeImgGen` (the old docs were misleading — please fix yours too):** `PipeImgGen` does **not** consume a dedicated "prompt concept". It has a required **`prompt` string template** (+ optional `negative_prompt`), and its declared `inputs` are **injected into that template**:-> - **text inputs** (concept `Text`) are interpolated via `$var` / Jinja;-> - **image inputs** (concept `Image`, single or list) are referenced in the prompt and injected — each referenced image becomes an `[Image N]` token and is passed to the generator alongside the rendered text (this is the **same vision pattern** as image inputs to `PipeLLM`, and is what enables image-to-image / reference-image / editing, bounded by the model's `max_prompt_images`).->-> So the right way to teach/author `PipeImgGen` is: write a `prompt` template, declare `Text` and/or `Image` inputs, reference them in the template. There is no `ImgGenPrompt` concept anywhere in that flow. If any of your docs/examples say "load a prompt into an `ImgGenPrompt` concept and feed it to PipeImgGen", that was always wrong — replace it with the template-injection model above.->-> `PipeImgGen`'s only concept constraints: declared image inputs must be `Image`-compatible, and the output must be `Image`-compatible. The operator's runtime behavior is unchanged by this removal.->-> **Where to check in your repo** (grep for `ImgGenPrompt` and judge by context — concept ref vs. the model/category/error above):-> - `mthds-plugins/` — any skill, prompt, or doc that enumerates native concepts (e.g. build/check/explain skills), or example `.mthds` snippets using `ImgGenPrompt` as a concept.-> - `mthds/` (spec docs site) — native-concepts reference page / any built-in concept list.-> - `mthds-js/`, `mthds-python/` — hardcoded native-concept lists used for validation or codegen.-> - `vscode-pipelex/` (`plxt` CLI / LSP) — native-concept completion/validation tables.-> - `conformance/` — fixtures or expected-output corpora referencing `native.ImgGenPrompt`.-> - `pipelex-cookbook/`, `methods/`, `test-bed/`, `pipelex-starter-python/` — example `.mthds` files (pipelex's own bundled libraries were already verified clean).->-> Migration in all cases is the same: `ImgGenPrompt` (as a concept) → `Text`.-------## DO-NOT-TOUCH guardrail (the conflation traps, restated)--If you find yourself editing any of these, STOP — you've drifted into the name collision:--- `pipelex/cogt/img_gen/img_gen_prompt.py` (the `ImgGenPrompt` BaseModel) and everything that imports it (`img_gen_job.py`, `img_gen_job_factory.py`, `img_gen_prompt_blueprint.py`, `content_generator.py`, `pipe_img_gen.py`, `pipe_img_gen_factory.py`, and their tests).-- `pipelex/cogt/templating/template_category.py` + `pipelex/tools/jinja2/jinja2_environment.py` (`TemplateCategory.IMG_GEN_PROMPT`).-- `pipelex/cogt/exceptions.py` `ImgGenPromptError` + `docs/errors/img-gen-prompt-error.md` + `docs/errors/inference-and-providers.md`.--These all stay exactly as they are.+- `GraphSpec.pipe_registry` and `GraphSpec.concept_registry` include `source` when a source-map entry exists and omit it when not available.+- Source enrichment works through direct in-memory graphs and event-assembled Temporal graphs.+- Validation can graph an explicit pipe target without `main_pipe`.+- Default validation behavior stays unchanged when no explicit target is provided.+- CLI `validate bundle --pipe X --graph/--view` graphs `X`.+- Tests pin source enrichment, no-main explicit target, target-overrides-main, missing target degradation/error behavior, and direct/Temporal parity.diff --git a/pipelex/cli/agent_cli/commands/validate/bundle_cmd.py b/pipelex/cli/agent_cli/commands/validate/bundle_cmd.pyindex 7c3f04cfc..d29be12ca 100644--- a/pipelex/cli/agent_cli/commands/validate/bundle_cmd.py+++ b/pipelex/cli/agent_cli/commands/validate/bundle_cmd.py@@ -159,6 +159,7 @@ def validate_bundle_cmd(bundle_path=Path(bundle_path), # type: ignore[arg-type]graph_format=graph_format,library_dirs=library_dir_strings,+ pipe_code=pipe,direction=direction,))@@ -187,6 +188,7 @@ def validate_bundle_cmd(generate_view_for_bundle(bundle_path=Path(bundle_path), # type: ignore[arg-type]library_dirs=library_dir_strings,+ pipe_code=pipe,direction=direction,))diff --git a/pipelex/core/pipes/pipe_abstract.py b/pipelex/core/pipes/pipe_abstract.pyindex d36d4e687..06c3cf95c 100644--- a/pipelex/core/pipes/pipe_abstract.py+++ b/pipelex/core/pipes/pipe_abstract.py@@ -19,6 +19,7 @@ from pipelex.core.pipes.pipe_output import PipeOutputfrom pipelex.core.pipes.stuff_spec.stuff_spec import StuffSpecfrom pipelex.core.pipes.validation import is_variable_satisfied_by_inputsfrom pipelex.graph.graph_tracer_manager import GraphTracerManager, IOSpec, NodeKind+from pipelex.libraries.library_crate import LibraryCratefrom pipelex.pipe_run.pipe_run_mode import PipeRunModefrom pipelex.pipe_run.pipe_run_params import PipeRunParamsfrom pipelex.pipe_signature.exceptions import PipeSignatureNotExecutableError@@ -39,7 +40,6 @@ from pipelex.types import Selfif TYPE_CHECKING:from pipelex.graph.trace_context import TraceContext- from pipelex.libraries.library_crate import LibraryCratePipeAbstractType = type["PipeAbstract"]@@ -118,18 +118,31 @@ class PipeAbstract(ABC, BaseModel):execution_data=execution_data,)- def _make_single_concept_data_for_registry(self, concept: Concept) -> dict[str, Any]:+ def _make_pipe_data_for_registry(self, *, library_crate: LibraryCrate | None) -> dict[str, Any]:+ """Serialize this pipe for the graph registry, including declaration source when known."""+ pipe_data = self.model_dump(mode="json", serialize_as_any=True)+ if library_crate is not None:+ source = library_crate.source_map.get(self.pipe_ref)+ if source:+ pipe_data["source"] = source+ return pipe_data++ def _make_single_concept_data_for_registry(self, concept: Concept, *, library_crate: LibraryCrate | None) -> dict[str, Any]:"""Serialize a single concept for the graph registry, including its JSON Schema."""- concept_dict = concept.model_dump(mode="json")+ concept_dict = concept.model_dump(mode="json", serialize_as_any=True)+ if library_crate is not None:+ source = library_crate.source_map.get(concept.concept_ref)+ if source:+ concept_dict["source"] = sourcetry:concept_dict["json_schema"] = concept.get_structure_class().model_json_schema()except (TypeError, ValueError):concept_dict["json_schema"] = Nonereturn concept_dict- def _make_concept_data_for_registry(self) -> list[dict[str, Any]]:+ def _make_concept_data_for_registry(self, *, library_crate: LibraryCrate | None) -> list[dict[str, Any]]:"""Serialize all unique concepts from this pipe for the graph registry."""- return [self._make_single_concept_data_for_registry(concept) for concept in self.concept_dependencies]+ return [self._make_single_concept_data_for_registry(concept, library_crate=library_crate) for concept in self.concept_dependencies]@field_validator("code", mode="before")@classmethod@@ -445,7 +458,7 @@ class PipeAbstract(ABC, BaseModel):working_memory: WorkingMemory,pipe_run_params: PipeRunParams,output_name: str | None = None,- library_crate: "LibraryCrate | None" = None,+ library_crate: LibraryCrate | None = None,) -> PipeOutput:# Push the pipe's frame onto the stack, run it, and always pop it — even on failure —# so a failed pipe never leaves a stale frame behind on the shared pipe_stack, where it@@ -471,7 +484,7 @@ class PipeAbstract(ABC, BaseModel):working_memory: WorkingMemory,pipe_run_params: PipeRunParams,output_name: str | None = None,- library_crate: "LibraryCrate | None" = None,+ library_crate: LibraryCrate | None = None,) -> PipeOutput:"""Run the pipe with graph tracing — the inner body of `run_pipe()`.@@ -522,8 +535,8 @@ class PipeAbstract(ABC, BaseModel):pipe_data: dict[str, Any] | None = Noneconcept_data: list[dict[str, Any]] | None = Noneif parent_trace_context.emit_graph_events and parent_trace_context.data_inclusion.pipe_and_concept_registry:- pipe_data = self.model_dump(mode="json")- concept_data = self._make_concept_data_for_registry()+ pipe_data = self._make_pipe_data_for_registry(library_crate=library_crate)+ concept_data = self._make_concept_data_for_registry(library_crate=library_crate)graph_node_id, child_trace_context = tracer_manager.on_pipe_start(trace_context=parent_trace_context,@@ -614,7 +627,7 @@ class PipeAbstract(ABC, BaseModel):# Serialize output concept for registry if enabled (E1: also gated on emit_graph_events).output_concept_data: dict[str, Any] | None = Noneif parent_trace_context.emit_graph_events and parent_trace_context.data_inclusion.pipe_and_concept_registry and main_stuff is not None:- output_concept_data = self._make_single_concept_data_for_registry(main_stuff.concept)+ output_concept_data = self._make_single_concept_data_for_registry(main_stuff.concept, library_crate=library_crate)tracer_manager.on_pipe_end_success(lookup_key=parent_trace_context.lookup_key,@@ -634,7 +647,7 @@ class PipeAbstract(ABC, BaseModel):working_memory: WorkingMemory,pipe_run_params: PipeRunParams,output_name: str | None = None,- library_crate: "LibraryCrate | None" = None,+ library_crate: LibraryCrate | None = None,) -> PipeOutput:log.info(self._format_pipe_run_info(pipe_run_params=pipe_run_params))@@ -707,7 +720,7 @@ class PipeAbstract(ABC, BaseModel):working_memory: WorkingMemory,pipe_run_params: PipeRunParams,output_name: str | None = None,- library_crate: "LibraryCrate | None" = None,+ library_crate: LibraryCrate | None = None,) -> PipeOutput:log.verbose(f"Dry run of {self.type}: '{self.code}'")assert pipe_run_params.run_mode.is_dry, f"Dry run of {self.type} '{self.code}' called with run_mode = {pipe_run_params.run_mode}"@@ -727,7 +740,7 @@ class PipeAbstract(ABC, BaseModel):working_memory: WorkingMemory,pipe_run_params: PipeRunParams,output_name: str | None = None,- library_crate: "LibraryCrate | None" = None,+ library_crate: LibraryCrate | None = None,) -> PipeOutput:pass@@ -739,7 +752,7 @@ class PipeAbstract(ABC, BaseModel):working_memory: WorkingMemory,pipe_run_params: PipeRunParams,output_name: str | None = None,- library_crate: "LibraryCrate | None" = None,+ library_crate: LibraryCrate | None = None,) -> PipeOutput:passdiff --git a/pipelex/graph/graph_rendering.py b/pipelex/graph/graph_rendering.pyindex adb4dfb6c..fa1cbcd66 100644--- a/pipelex/graph/graph_rendering.py+++ b/pipelex/graph/graph_rendering.py@@ -104,6 +104,7 @@ async def _dry_run_bundle(bundle_path: Path,*,library_dirs: list[str] | None = None,+ pipe_code: str | None = None,) -> tuple[GraphSpec, str]:"""Dry-run a bundle file to produce a GraphSpec.@@ -113,6 +114,7 @@ async def _dry_run_bundle(Args:bundle_path: Path to the .mthds bundle file.library_dirs: Optional library directories for pipe resolution.+ pipe_code: Optional explicit pipe target for graph generation.Returns:Tuple of (GraphSpec, pipe_code).@@ -134,6 +136,7 @@ async def _dry_run_bundle(mthds_contents=[mthds_content],bundle_uris=[str(bundle_path)],library_dirs=effective_library_dirs,+ pipe_code=pipe_code,)@@ -142,6 +145,7 @@ async def generate_graph_for_bundle(*,graph_format: GraphFormat,library_dirs: list[str] | None = None,+ pipe_code: str | None = None,direction: FlowchartDirection | None = None,graph_name: str = "dry_run.html",) -> dict[str, Any]:@@ -154,6 +158,7 @@ async def generate_graph_for_bundle(bundle_path: Path to the .mthds bundle file.graph_format: Which graph format(s) to generate.library_dirs: Optional library directories for pipe resolution.+ pipe_code: Optional explicit pipe target for graph generation.direction: Flowchart layout direction (default: None, uses TB).graph_name: Filename for the generated HTML graph (default: "dry_run.html").@@ -165,7 +170,7 @@ async def generate_graph_for_bundle(PipelexError: If pipeline execution does not produce a graph spec.PipelineExecutionError: If dry-run execution fails."""- graph_spec, pipe_code = await _dry_run_bundle(bundle_path, library_dirs=library_dirs)+ graph_spec, resolved_pipe_code = await _dry_run_bundle(bundle_path, library_dirs=library_dirs, pipe_code=pipe_code)execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True,@@ -192,7 +197,7 @@ async def generate_graph_for_bundle(include_mermaidflow=include_mermaidflow,include_reactflow=include_reactflow,output_dir=output_dir,- pipe_code=pipe_code,+ pipe_code=resolved_pipe_code,direction=direction,)@@ -207,7 +212,7 @@ async def generate_graph_for_bundle(return {"graph_files": {key: str(path) for key, path in saved_files.items()},"graph_output_dir": str(output_dir),- "pipe_code": pipe_code,+ "pipe_code": resolved_pipe_code,"direction": str(direction) if direction else None,}@@ -216,6 +221,7 @@ async def generate_view_for_bundle(bundle_path: Path,*,library_dirs: list[str] | None = None,+ pipe_code: str | None = None,direction: FlowchartDirection | None = None,) -> dict[str, Any]:"""Generate a GraphSpec for a bundle via dry-run pipeline execution.@@ -226,6 +232,7 @@ async def generate_view_for_bundle(Args:bundle_path: Path to the .mthds bundle file.library_dirs: Optional library directories for pipe resolution.+ pipe_code: Optional explicit pipe target for graph generation.direction: Flowchart layout direction (default: None, uses config default).Returns:@@ -236,7 +243,7 @@ async def generate_view_for_bundle(PipelexError: If pipeline execution does not produce a graph spec.PipelineExecutionError: If dry-run execution fails."""- graph_spec, pipe_code = await _dry_run_bundle(bundle_path, library_dirs=library_dirs)+ graph_spec, resolved_pipe_code = await _dry_run_bundle(bundle_path, library_dirs=library_dirs, pipe_code=pipe_code)execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True,@@ -247,6 +254,6 @@ async def generate_view_for_bundle(return {"graphspec": graph_spec.model_dump(mode="json", by_alias=True),- "pipe_code": pipe_code,+ "pipe_code": resolved_pipe_code,"direction": str(effective_direction) if effective_direction else None,}diff --git a/pipelex/kit/agent_rules/pytest_standards.md b/pipelex/kit/agent_rules/pytest_standards.mdindex e7c6a1b41..963a879bb 100644--- a/pipelex/kit/agent_rules/pytest_standards.md+++ b/pipelex/kit/agent_rules/pytest_standards.md@@ -1,4 +1,4 @@-# Writing tests+# Writing Tests## Unit test generalities@@ -29,7 +29,7 @@ Apply the appropriate markers:- "inference: uses either an LLM or an image generation AI"- never add "@pytest.mark.dry_runnable" if you haven't set the "inference" marker-Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.+Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference` and `llm`.### Important rulesdiff --git a/pipelex/kit/agent_rules/python_standards.md b/pipelex/kit/agent_rules/python_standards.mdindex 8ec5031d8..4555fb4e2 100644--- a/pipelex/kit/agent_rules/python_standards.md+++ b/pipelex/kit/agent_rules/python_standards.md@@ -1,144 +1,193 @@# Coding Standards & Best Practices for Python Code-This document outlines the core coding standards, best practices, and quality control procedures for the codebase.-## Python Version Compatibility- - The project supports Python 3.10+ (`requires-python = ">=3.10,<3.15"`). Never use features introduced after Python 3.10 without a compatibility fallback.- - Common pitfalls:- - `datetime.UTC` was added in Python 3.11. Use `datetime.timezone.utc` instead.- - `StrEnum` was added in Python 3.11. Always import it from `pipelex.types` which handles retrocompatibility.- - `type` statement (PEP 695) was added in Python 3.12. Use `TypeAlias` from `typing` instead.- - `ExceptionGroup` / `except*` was added in Python 3.11. Avoid unless using the `exceptiongroup` backport.+- The project supports Python 3.10+ (`requires-python = ">=3.10,<3.15"`). Never use features introduced after Python 3.10 without a compatibility fallback.+- Common pitfalls:+ - `datetime.UTC` was added in Python 3.11. Use `datetime.timezone.utc` instead.+ - `StrEnum` was added in Python 3.11. Always import it from `pipelex.types` which handles retrocompatibility.+ - `type` statement (PEP 695) was added in Python 3.12. Use `TypeAlias` from `typing` instead.+ - `ExceptionGroup` / `except*` was added in Python 3.11. Avoid unless using the `exceptiongroup` backport.## Variables, loops and indexes- - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.- - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.- - Avoid inline for loops, unless it's ultra-simple and holds on oneline.- - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.+- Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.+- When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.+- Avoid inline for loops, unless it's ultra-simple and holds on oneline.+- If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.## Enums and tests- - When defining enums related to string values, always inherit from `StrEnum`- - When you need the enum value as a string, don't use `str(enum_var)` or `enum_var.value`, just use `enum_var` itself, that is the point of using StrEnum!- - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add @property methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases- - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.- - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):- ```python- from pipelex.types import StrEnum- ```+- When defining enums related to string values, always inherit from `StrEnum`+- When you need the enum value as a string, don't use `str(enum_var)` or `enum_var.value`, just use `enum_var` itself, that is the point of using StrEnum!+- Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add @property methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases+- As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.+- `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):+ ```python+ from pipelex.types import StrEnum+ ```## Optionals- Don't write things like `a = b if b else c`, write `a = b or c` instead.-## Imports+## Filesystem Paths++- **Use `pathlib.Path` for all filesystem paths.** Any function parameter, return type, or variable that holds a location of a file or directory on disk must be typed `Path`, not `str`. Convert to/from `str` only at genuine boundaries.+- **Boundaries where `str` is correct** (convert at the edge, then work with `Path` internally):+ - CLI argument parsing (typer/click options and arguments arrive as `str`).+ - Config / TOML (de)serialization — Pydantic config fields and `.toml` values are `str`.+ - Env-var parsing, e.g. splitting `PIPELEXPATH` on `os.pathsep`.+ - URIs / URLs — these are not filesystem paths.+ - `importlib.resources` package-resource traversables.+ - Third-party calls whose signature requires `str`.+- **Prefer `Path` idioms over `os.path`:** `base / "sub" / "file.txt"` (not `os.path.join`), `.parent` (not `os.path.dirname`), `.name` (not `os.path.basename`), `.suffix`, `.stem`, `.exists()`, `.is_dir()`, `.is_file()`, `.mkdir(parents=True, exist_ok=True)` (not `os.makedirs`), `.iterdir()` / `.rglob(...)` (not `os.listdir` / `os.walk`), `.read_text()` / `.read_bytes()` / `.write_text()` / `.write_bytes()` (not `open()`), `.unlink()`.+- **Watch the semantic shifts when porting `os.path` code — these are NOT drop-in equivalents:**+ - `os.path.relpath(p, base)` returns a relative path that may walk up with `..`; `Path.relative_to(base)` raises `ValueError` when `p` is not under `base`. Never swap one for the other blindly.+ - `os.path.abspath(p)` collapses `.`/`..` lexically but does **not** resolve symlinks. No single `Path` method matches it: `Path.resolve()` both normalizes `..` *and* follows symlinks, while `Path.absolute()` only prepends the cwd and normalizes neither (`Path("a/../b").absolute()` stays `<cwd>/a/../b`, whereas `os.path.abspath("a/../b")` is `<cwd>/b`). Reach for `Path.resolve()` in the common case; if you genuinely need abspath's no-symlink semantics, normalize explicitly (`Path(os.path.normpath(p.absolute()))`).+- **Logical (non-filesystem) "paths" stay `str`.** Dotted module/import paths, domain paths, and variable/key paths (`variable_path`, `dotted_path`, `var_path`, `key_path`, and similar) are identifiers, not locations on disk — keep them `str`. This rule is about files and directories, not everything named `*_path`.-### **Imports at the top of the file**+## Imports- - Avoid as much as possible import statements outside of a module's top-level scope.- - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting- - Do not bother with ordering the imports or removing unused imports, our Ruff linter will handle it for us.- - `if TYPE_CHECKING:` blocks must always be the **last** block in the imports section, placed after all regular imports.+### Imports at the top of the file-### **Removing unused imports**+- Avoid as much as possible import statements outside of a module's top-level scope.+- Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting+- Do not bother with ordering the imports or removing unused imports, our Ruff linter will handle it for us.+- `if TYPE_CHECKING:` blocks must always be the **last** block in the imports section, placed after all regular imports.- - To remove unused imports, run `make fix-unused-imports` or `make fui` (shorthand). This is faster and cheaper than rewriting with LLM.+### Removing unused imports-### **No re-exports in `__init__.py`**+- To remove unused imports, run `make fix-unused-imports` or `make fui` (shorthand). This is faster and cheaper than rewriting with LLM.- - Do NOT fill `__init__.py` files with re-exports.- - Always use direct full-path imports everywhere. For example:+### No re-exports in `__init__.py`-- **Logging and Pretty Printing**:+- Do NOT fill `__init__.py` files with re-exports.+- Always use direct full-path imports everywhere.- - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:- ```python- from pipelex import log, pretty_print+### Logging and Pretty Printing- log.info("Hello, world!")- ```- - Both have a title arg which is handy when logging/printing objects:+- Both `log()` and `pretty_print()` can be imported from `pipelex` directly:+ ```python+ from pipelex import log, pretty_print- ```python- log.verbose("Hello, world!", title="Your first Pipelex log")- pretty_print(output_object, title="Your first Pipelex output")- ```- - Both handle formatting json using Rich, pretty_print makes it prettier.+ log.info("Hello, world!")+ ```+- Both have a title arg which is handy when logging/printing objects:+ ```python+ log.verbose("Hello, world!", title="Your first Pipelex log")+ pretty_print(output_object, title="Your first Pipelex output")+ ```+- Both handle formatting json using Rich, pretty_print makes it prettier.-- **StrEnum and Self type**:+### StrEnum and Self type- - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):- ```python- from pipelex.types import Self, StrEnum- ```+- Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):+ ```python+ from pipelex.types import Self, StrEnum+ ```## Typing-### **Always Use Type Hints**-- - Every function parameter must be typed- - Every function return must be typed- - Use type hints for all variables where type is not obvious- - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]- - Use type hints for all fields- - Use Field(default_factory=...) for mutable defaults- - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.--### **BaseModel / Pydantic Standards**-- - Use `BaseModel` and respect Pydantic v2 standards- - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`- - Keep models focused and single-purpose- - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:- ```python- from pydantic import BaseModel, Field- from pipelex.tools.typing.pydantic_utils import empty_list_factory_of-- class MyModel(BaseModel):- names: list[str] = Field(default_factory=list) # OK for strings- numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")- items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")- ```+### Always Use Type Hints++- Every function parameter must be typed+- Every function return must be typed+- Use type hints for all variables where type is not obvious+- Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]+- Use type hints for all fields+- Use Field(default_factory=...) for mutable defaults+- Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.++### Data-holder shapes (NamedTuple, dataclass, BaseModel)++The deciding question is **does this value ever get serialized** — sent across a process or network boundary, persisted to disk or a database, or round-tripped through (de)serialization? If yes, it must be one of the two serialization-safe forms (a `BaseModel` or a pydantic dataclass): a `NamedTuple` or stdlib `@dataclass` loses its type on the way through and silently corrupts the data across the boundary. (A distributed executor like Temporal is the canonical case — its payloads cross the wire between worker processes — but the same applies to any cache, queue, API response, or on-disk record.) Stdlib `@dataclass` is banned outright; pick one of these four forms:++- **`NamedTuple`** (`from typing import NamedTuple`) — small fixed records replacing bare tuples, or any holder of a `@runtime_checkable` Protocol / arbitrary non-pydantic field (avoids `arbitrary_types_allowed`). Not serialization-safe, zero validation. Avoid when the value is returned in a union beside a real `tuple` — it would also satisfy `isinstance(x, tuple)` and blur the two arms; use a frozen `BaseModel` there instead.+- **pydantic dataclass** (`from pydantic.dataclasses import dataclass`) — internal records that are built and field-read but never serialized; the lowest-churn swap from a stdlib `@dataclass` (just change the decorator). Default to `frozen=True`; drop it only when the instance is genuinely mutated in place after construction.+- **frozen `BaseModel`** (`model_config = ConfigDict(frozen=True)`) — immutable value objects that need the full model API (`model_dump`, validators, discriminated unions), must sit in a uniformly-`BaseModel` layer, or must stay type-distinct from a `tuple` in a union.+- **mutable `BaseModel`** — objects genuinely mutated in place that also want the model API or layer consistency.++Never reach for `TypeAdapter` — needing it means the shape was chosen wrong. And don't convert an existing serialized `BaseModel` to a dataclass just because dataclasses are serialization-safe too: a model that relies on discriminated unions or validators must stay a `BaseModel`. When a serialized dataclass/model field references a type from an optional dependency, alias it to `Any` at runtime so it validates as `Any` without importing the extra (e.g. `if TYPE_CHECKING: from temporalio.common import RetryPolicy` / `else: RetryPolicy = Any`) — a bare forward ref instead raises `PydanticUserError` on construction.++Once you've landed on a `BaseModel` (the default for anything that serializes), respect Pydantic v2 standards:++- Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`+- Keep models focused and single-purpose+- For list fields with non-string items, use `empty_list_factory_of()` to avoid linter complaints:+ ```python+ from pydantic import BaseModel, Field+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of++ class MyModel(BaseModel):+ names: list[str] = Field(default_factory=list) # OK for strings+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")+ ```++## Keyword-only arguments++Across `pipelex/` source, non-subject function parameters must be **keyword-only**, so call sites are self-documenting: `do_thing(retries=3, timeout=30)` over the opaque `do_thing(3, 30)`. The compliant shape places a bare `*` after the subject:++- `def f(subject, *, opt1, opt2): ...` — the first non-`self`/`cls` parameter (the subject) may stay positional; everything after it must be keyword-only. Making the subject keyword-only too (`def f(*, a, b)`, `def m(self, *, a, b)`) is always allowed and is the form the auto-fixer produces.+- A lone subject (`def render(node)`) is compliant; a second bare positional (`def f(a, b)`, `def truncate(text, max_length=80)`) is a violation.++The rule is mechanically enforced by the `check-keyword-only` AST guard (`make cko`), which runs in `make agent-check`, in the `make check` aggregate, in CI, and via a Claude Code `PostToolUse` hook that checks each edited file at edit time. The tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs — dunders, pydantic validators/serializers, Typer/Temporal/pytest/Jinja2 framework entrypoints, `@override` impls, and a small curated symmetric-tuple allowlist — are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line).++**Auto-fix.** `make fix-keyword-only` (alias `fko`, or `pipelex-dev check-keyword-only --fix`) rewrites every mechanically-fixable violation by inserting a bare `*` as far left as possible — right after `self`/`cls` — so every non-`self`/`cls` parameter becomes keyword-only, and re-parses each rewrite before writing (a rewrite that would not parse is discarded and reported instead). It runs automatically early in `make agent-check`, before `ruff format`. It is **non-gating**: it mutates and reports but never exits non-zero — the read-only `check-keyword-only` (run last in `agent-check`, and the variant `make check` / CI use) owns the pass/fail gate. A few shapes a single `*` insert cannot fix are reported for a manual fix instead of rewritten: a `*args` is present, a keyword-only section already exists, or two-or-more positional-only params remain.++Auto-fix is not a substitute for review: the guard is blind to functions a framework or the interpreter invokes positionally (callbacks, `__import__` hooks, route handlers, Jinja2 filters) — pyright/mypy will pass a wrongly-keywordized callback, so `make agent-test` is the safety net after a bulk fix.## Factory Pattern- - Use factory pattern for object creation when dealing with multiple implementations- - Our factory methods are named `make_from_...` and such+- Use factory pattern for object creation when dealing with multiple implementations+- Our factory methods are named `make_from_...` and such## Protocols and Interfaces- - When a getter (e.g. `get_report_delegate()`, `get_library_manager()`) returns a `Protocol` type, callers must only rely on methods declared on that `Protocol`. If you need to call a method that lives on a concrete implementation, **extend the `Protocol`** — do not work around it with `isinstance(...)` checks, `cast(...)`, or inline imports of the concrete class.- - When extending the `Protocol`, also update every implementation (including no-op / null implementations) so structural typing is satisfied. For implementations where the method has nothing to do, give it a `pass` body — that is the whole point of having a `Protocol` rather than a concrete base class.- - Smell to watch for: an inline `from ... import ConcreteImpl # noqa: PLC0415` followed by `if isinstance(delegate, ConcreteImpl): delegate.some_method(...)`. This is almost always a missing method on the `Protocol`. Promote it.- - Use `@override` (from `typing_extensions`) on every implementation of a `Protocol` method, so that signature drift between the `Protocol` and its implementations is caught by the type checker.+- When a getter (e.g. `get_report_delegate()`, `get_library_manager()`) returns a `Protocol` type, callers must only rely on methods declared on that `Protocol`. If you need to call a method that lives on a concrete implementation, **extend the `Protocol`** — do not work around it with `isinstance(...)` checks, `cast(...)`, or inline imports of the concrete class.+- When extending the `Protocol`, also update every implementation (including no-op / null implementations) so structural typing is satisfied. For implementations where the method has nothing to do, give it a `pass` body — that is the whole point of having a `Protocol` rather than a concrete base class.+- Smell to watch for: an inline `from ... import ConcreteImpl # noqa: PLC0415` followed by `if isinstance(delegate, ConcreteImpl): delegate.some_method(...)`. This is almost always a missing method on the `Protocol`. Promote it.+- Use `@override` (from `typing_extensions`) on every implementation of a `Protocol` method, so that signature drift between the `Protocol` and its implementations is caught by the type checker.## Error Handling- - Always catch exceptions at the place where you can add useful context to it.- - Use try/except blocks with specific exceptions- - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError- - **NEVER write `except Exception:` (or bare `except:`).** Catching the generic `Exception` is FORBIDDEN except in two cases, each of which REQUIRES a one-line comment directly under the `except` naming which case applies: **(1)** the absolute root of a CLI command or an API endpoint handler; **(2)** the wrapped call invokes genuinely unbounded code whose exception surface cannot be enumerated — a user-registered callback or function, dynamic plugin dispatch, or a third-party SDK with no documented exception types (this is only for genuinely open-ended surfaces; "I didn't look the exceptions up" does NOT qualify). There are NO other exceptions to this rule. The following are NOT valid justifications and the agent must reject them when tempted:- - "I want to log and re-raise" → just let it propagate; logging happens at the root.- - "I want to swallow it just in case" → don't. Unknown failures must surface.- - "I want to convert it to our error type" → only if you know which specific exception(s) the call can raise; catch those explicitly. Catching a project base class (`PipelexError`) or a package base (`CogtError`, `ToolError`) is acceptable when the operation legitimately fails in many domain-specific ways — that is still specific. Catching `Exception` is not.- - "The third-party lib isn't well typed" → look up the actual exceptions it raises and catch those by name.- - "It's safer this way" → it isn't. It hides bugs. Crashing loudly on an unexpected exception is the desired behavior.- If you find yourself writing `except Exception` outside the two cases above, STOP and ask the user before proceeding.- - **Do NOT add try/except speculatively.** Only add a `try`/`except` when (a) a specific exception type is documented or known to be raised by the call, AND (b) you have a concrete recovery path or a meaningful context to attach. Otherwise, let the error bubble up — that is the correct behavior, not a gap to fill.- - **Use `try`/`finally` (without `except`) for required cleanup.** When a piece of code MUST run on the way out — releasing a resource, closing a tracer, tearing down a library, clearing per-context state on a manager — put it in a `finally` block (or a `with`/context manager). This is the correct way to guarantee cleanup on both the happy path and the error path: it does NOT suppress the exception, it just ensures cleanup happens before the exception propagates. Do not invent an `except Exception` just to attach cleanup; use `finally` instead. Conversely, do not move cleanup into the success path "to keep things simple" — if it must run on errors too, it belongs in `finally`.- - When reviewing existing code, treat any `except Exception` you encounter as a bug to fix (replace it with the specific exception(s) actually raised, or remove the try/except entirely), not as a precedent to follow.- - Always add `from exc` to the exception raise statements- - Always write the error message as a variable before raising it, for cleaner error traces-- ```python- try:- self.models_manager.setup()- except RoutingProfileLibraryNotFoundError as exc:- msg = "The routing library could not be found, please call `pipelex init config` to create it"- raise PipelexSetupError(msg) from exc- ```+- Always catch exceptions at the place where you can add useful context to it.+- Use try/except blocks with specific exceptions+- Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError+- **NEVER write `except Exception:` (or bare `except:`).** Catching the generic `Exception` is FORBIDDEN except in two cases, each of which REQUIRES a one-line comment directly under the `except` naming which case applies: **(1)** the absolute root of a CLI command or an API endpoint handler; **(2)** the wrapped call invokes genuinely unbounded code whose exception surface cannot be enumerated — a user-registered callback or function, dynamic plugin dispatch, or a third-party SDK with no documented exception types (this is only for genuinely open-ended surfaces; "I didn't look the exceptions up" does NOT qualify). There are NO other exceptions to this rule. The following are NOT valid justifications and the agent must reject them when tempted:+ - "I want to log and re-raise" → just let it propagate; logging happens at the root.+ - "I want to swallow it just in case" → don't. Unknown failures must surface.+ - "I want to convert it to our error type" → only if you know which specific exception(s) the call can raise; catch those explicitly. Catching a project base class (`PipelexError`) or a package base (`CogtError`, `ToolError`) is acceptable when the operation legitimately fails in many domain-specific ways — that is still specific. Catching `Exception` is not.+ - "The third-party lib isn't well typed" → look up the actual exceptions it raises and catch those by name.+ - "It's safer this way" → it isn't. It hides bugs. Crashing loudly on an unexpected exception is the desired behavior.+ If you find yourself writing `except Exception` outside the two cases above, STOP and ask the user before proceeding.+- **Do NOT add try/except speculatively.** Only add a `try`/`except` when (a) a specific exception type is documented or known to be raised by the call, AND (b) you have a concrete recovery path or a meaningful context to attach. Otherwise, let the error bubble up — that is the correct behavior, not a gap to fill.+- **Use `try`/`finally` (without `except`) for required cleanup.** When a piece of code MUST run on the way out — releasing a resource, closing a tracer, tearing down a library, clearing per-context state on a manager — put it in a `finally` block (or a `with`/context manager). This is the correct way to guarantee cleanup on both the happy path and the error path: it does NOT suppress the exception, it just ensures cleanup happens before the exception propagates. Do not invent an `except Exception` just to attach cleanup; use `finally` instead. Conversely, do not move cleanup into the success path "to keep things simple" — if it must run on errors too, it belongs in `finally`.+- When reviewing existing code, treat any `except Exception` you encounter as a bug to fix (replace it with the specific exception(s) actually raised, or remove the try/except entirely), not as a precedent to follow.+- **`from exc` on raise statements — default yes.** `raise NewError(msg) from exc` whenever you raise a new exception while handling another. The point is preserving the cause for debugging tracebacks.++ Carve-out — translation helpers. When the actual `raise` lives one frame down inside a translation helper (`raise_validation_error(message=str(exc))`, an API 4xx shaper, anything `(msg: str) -> NoReturn` whose job is shaping the outgoing response), Python's implicit `__context__` chaining already preserves the original — the helper's `raise` runs while your `except` frame is still active, structured logs walk the chain either way, and only the traceback intro line differs ("During handling of the above exception" vs "The above exception was the direct cause"). Don't bolt a `cause=exc` parameter onto every such helper just to swap implicit `__context__` for explicit `__cause__`. Revisit only if a helper grows an internal try/except that would break the implicit chain, or if a downstream tool (Sentry-style) is wired to distinguish the two.++- Always write the error message as a variable before raising it, for cleaner error traces++ ```python+ try:+ self.models_manager.setup()+ except RoutingProfileLibraryNotFoundError as exc:+ msg = "The routing library could not be found, please call `pipelex init config` to create it"+ raise PipelexSetupError(msg) from exc+ ```++## Error class location++Put every custom error class in a module named `exceptions.py` (default — one per package directory) or `<topic>_exceptions.py` (when a directory needs several separate-concern error modules); the base error class gets its own module. This is a discoverability contract, not style: tooling that enumerates errors (doc generation, URI/uniqueness checks) finds them by importing these well-known module names, so an error defined in a sibling non-error module (worker, factory, parser) may never be imported and silently escapes that tooling.++- **Default:** append to the package's existing `exceptions.py`.+- **Topical split:** when appending would cause a circular import, or a package hosts genuinely separate error families, use `<topic>_exceptions.py` importing only the base error class.+- **Forbidden:** `*_errors.py` (synonym, drift magnet), or defining an error alongside non-error code.++Use one casing per acronym (`LLMError` vs `LlmError` collide when kebab-cased into filenames/URIs). Enforce the convention with a test in CI.## Documentation@@ -146,11 +195,11 @@ This document outlines the core coding standards, best practices, and quality co```pythondef process_image(image_path: str, size: tuple[int, int]) -> bytes:"""Process and resize an image.-+Args:image_path: Path to the source imagesize: Tuple of (width, height) for resizing-+Returns:Processed image as bytes"""@@ -161,7 +210,7 @@ This document outlines the core coding standards, best practices, and quality co```pythonclass ImageProcessor:"""Handles image processing operations.-+Provides methods for resizing, converting, and optimizing images."""```diff --git a/pipelex/pipeline/dry_run_pipeline.py b/pipelex/pipeline/dry_run_pipeline.pyindex 655a03711..308ac381d 100644--- a/pipelex/pipeline/dry_run_pipeline.py+++ b/pipelex/pipeline/dry_run_pipeline.py@@ -28,11 +28,12 @@ async def dry_run_pipeline(*,bundle_uris: list[str] | None = None,library_dirs: list[str] | None = None,+ pipe_code: str | None = None,) -> tuple[GraphSpec, str]:"""Dry-run a full pipeline from MTHDS content, producing a GraphSpec.- Parses the content, identifies the main pipe, and executes the pipeline- in dry-run mode with graph tracing enabled and mock inputs.+ Parses the content, identifies the requested or main pipe, and executes the+ pipeline in dry-run mode with graph tracing enabled and mock inputs.All contents are parsed into blueprints and loaded together; the main_pipeis found from the first blueprint that declares one.@@ -41,6 +42,9 @@ async def dry_run_pipeline(mthds_contents: List of MTHDS bundle contents as strings.bundle_uris: Optional list of URIs for the bundles (used by runner for dedup).library_dirs: Optional library directories for pipe resolution.+ pipe_code: Optional explicit pipe target. When omitted, the first selected+ `main_pipe` remains the default. Bare and qualified refs are accepted+ according to the loaded pipe library's normal resolution rules.Returns:Tuple of (GraphSpec, domain-qualified main-pipe ref).@@ -54,19 +58,20 @@ async def dry_run_pipeline(msg = "mthds_contents must be provided"raise ValueError(msg)- # Pre-parse contents to extract the main pipe's domain-qualified ref via the one shared- # selection rule. Note: pipeline_run_setup will re-parse these contents. The double-parse- # is accepted because the runner interface requires the pipe ref upfront and does not- # expose the internally-resolved pipe ref in its response.+ # Pre-parse contents to extract the default main pipe's domain-qualified ref via the one+ # shared selection rule. Note: pipeline_run_setup will re-parse these contents. The+ # double-parse is accepted because the runner interface requires the pipe ref upfront+ # and does not expose the internally-resolved pipe ref in its response.blueprints = [PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=content) for content in mthds_contents]- main_pipe_ref = select_primary_blueprint(blueprints).main_pipe_ref+ if pipe_code is not None and not pipe_code.strip():+ msg = "pipe_code must not be blank when provided"+ raise PipelexInterpreterError(msg)+ pipe_ref = pipe_code if pipe_code is not None else select_primary_blueprint(blueprints).main_pipe_ref- if not main_pipe_ref:+ if not pipe_ref:msg = "Bundle does not declare a main_pipe, cannot generate graph"raise PipelexInterpreterError(msg)- pipe_ref: str = main_pipe_ref-execution_config = get_config().pipelex.pipeline_execution_config.with_execution_overrides(generate_graph=True,mock_inputs=True,@@ -94,4 +99,9 @@ async def dry_run_pipeline(msg = "Pipeline execution did not produce a graph spec"raise DryRunGraphNotProducedError(msg)- return pipe_output.graph_spec, pipe_ref+ resolved_pipe_ref = pipe_ref+ pipeline_ref = pipe_output.graph_spec.pipeline_ref+ if pipeline_ref.domain and pipeline_ref.main_pipe:+ resolved_pipe_ref = f"{pipeline_ref.domain}.{pipeline_ref.main_pipe}"++ return pipe_output.graph_spec, resolved_pipe_refdiff --git a/pipelex/pipeline/runner.py b/pipelex/pipeline/runner.pyindex c95fdf036..a38b1bd40 100644--- a/pipelex/pipeline/runner.py+++ b/pipelex/pipeline/runner.py@@ -76,6 +76,27 @@ class PipelexVersionInfo(VersionInfo):runtime_version: str | None = None+_VALIDATE_EXTRA_GRAPH_PIPE_CODE = "graph_pipe_code"+++def _extract_validate_graph_pipe_code(extra: dict[str, Any] | None) -> str | None:+ """Parse local Pipelex validate extension args."""+ if not extra:+ return None+ unknown_keys = sorted(key for key in extra if key != _VALIDATE_EXTRA_GRAPH_PIPE_CODE)+ if unknown_keys:+ msg = f"The local runtime defines no extension args for: {unknown_keys}."+ raise PipelineRequestError(msg)+ graph_pipe_code = extra.get(_VALIDATE_EXTRA_GRAPH_PIPE_CODE)+ if graph_pipe_code is not None and not isinstance(graph_pipe_code, str):+ msg = f"validate extra '{_VALIDATE_EXTRA_GRAPH_PIPE_CODE}' must be a string when provided."+ raise PipelineRequestError(msg)+ if graph_pipe_code is not None and not graph_pipe_code.strip():+ msg = f"validate extra '{_VALIDATE_EXTRA_GRAPH_PIPE_CODE}' must not be blank when provided."+ raise PipelineRequestError(msg)+ return graph_pipe_code++class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):"""Pipelex implementation of the mthds MTHDSProtocol.@@ -353,33 +374,34 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):failure?" decision is the consumer's (the CLI gates its exit code on`is_runnable`; this protocol method just reports it).- `graph_spec` is best-effort: when the batch declares a `main_pipe`- (primary-selection rule: first blueprint declaring one), it is dry-run- in-process via `dry_run_pipe_in_process` against the validation library- and the resulting graph rides on the report. A graph-arm domain failure+ `graph_spec` is best-effort: when `extra={"graph_pipe_code": "..."}`+ is supplied, that pipe is dry-run as the graph target; otherwise, when+ the batch declares a `main_pipe` (primary-selection rule: first+ blueprint declaring one), that pipe is dry-run in-process via+ `dry_run_pipe_in_process` against the validation library and the+ resulting graph rides on the report. A graph-arm domain failure(`PipelexError`, polyfactory `FactoryException`, or any `ValueError` —covering pydantic `ValidationError` and the ValueError-family domainerrors of the mock-input mint and schema resolution) degrades to`graph_spec=None` with validation still successful — the same contractas the Temporal validate activity, so every backend answers identically- for the same bundle. No declared `main_pipe` means no graph.+ for the same bundle. No explicit graph target and no declared+ `main_pipe` means no graph.Args:mthds_contents: MTHDS contents to load (always a list, even for one file).allow_signatures: Tolerate unimplemented pipe signatures (strict by default).- extra: Rejected — the local runtime defines no extension args. Extension- args are server-specific; pass them to the server that defines them.+ extra: Optional local runtime extension args. The only accepted key is+ `graph_pipe_code`, an explicit pipe target for graph generation.Returns:PipelexValidationReport with the structural artifacts of a valid bundle.Raises:- PipelineRequestError: When extension args are passed (the local runtime accepts none).+ PipelineRequestError: When unsupported or invalid extension args are passed.PipelexError: When the bundle is invalid (parse, static validation, or dry-run failure)."""- if extra:- msg = f"The local runtime defines no extension args; got {sorted(extra)}."- raise PipelineRequestError(msg)+ graph_pipe_code = _extract_validate_graph_pipe_code(extra)library_dirs = [Path(library_dir) for library_dir in self.library_dirs] if self.library_dirs else None# Delegate to the shared in-process orchestrator (library-window management,# graph arm, report assembly). The protocol `validate` interface carries only@@ -391,6 +413,7 @@ class PipelexMTHDSProtocol(MTHDSProtocol["PipeOutput"]):mthds_contents=mthds_contents,library_dirs=library_dirs,allow_signatures=allow_signatures,+ graph_pipe_code=graph_pipe_code,log_context="Protocol validate",)diff --git a/pipelex/pipeline/validate_in_process.py b/pipelex/pipeline/validate_in_process.pyindex d6c9a9657..27e5fa41c 100644--- a/pipelex/pipeline/validate_in_process.py+++ b/pipelex/pipeline/validate_in_process.py@@ -47,6 +47,7 @@ async def validate_bundles_in_process(mthds_sources: list[str] | None = None,library_dirs: Sequence[Path] | None = None,allow_signatures: bool = False,+ graph_pipe_code: str | None = None,log_context: str = "validate",) -> PipelexValidationReport:"""Parse, validate, and dry-run MTHDS bundles in-process; assemble the canonical report.@@ -67,6 +68,10 @@ async def validate_bundles_in_process((so `source` stays `None` on that path).library_dirs: Resolved library directories to load before the bundle.allow_signatures: Tolerate unimplemented pipe signatures (strict by default).+ graph_pipe_code: Optional explicit pipe target for the best-effort graph arm.+ When omitted, the graph arm keeps the existing default of targeting the+ selected bundle `main_pipe`. Bare and qualified refs are accepted according+ to the loaded pipe library's normal resolution rules.log_context: Label used in graph-arm degradation and teardown-failure logs.Returns:@@ -102,8 +107,9 @@ async def validate_bundles_in_process(# something inside the window later moves the contextvar.validation_library_id = get_current_library_id_or_none()pipe_io_contracts: dict[str, PipeIOContract] = build_pipe_io_contracts(result.pipes)+ graph_target_ref = graph_pipe_code if graph_pipe_code is not None else select_primary_blueprint(result.blueprints).main_pipe_refgraph_spec: GraphSpec | None = await best_effort_graph_spec(- pipe_ref=select_primary_blueprint(result.blueprints).main_pipe_ref,+ pipe_ref=graph_target_ref,library_id=validation_library_id,log_context=log_context,)diff --git a/tests/integration/pipelex/pipeline/test_dry_run_pipeline_graph.py b/tests/integration/pipelex/pipeline/test_dry_run_pipeline_graph.pyindex 08ec61e8e..bfae6df92 100644--- a/tests/integration/pipelex/pipeline/test_dry_run_pipeline_graph.py+++ b/tests/integration/pipelex/pipeline/test_dry_run_pipeline_graph.py@@ -15,6 +15,7 @@ import pytestfrom pytest_mock import MockerFixturefrom pipelex.config import get_config+from pipelex.core.interpreter.exceptions import PipelexInterpreterErrorfrom pipelex.pipeline.dry_run_pipeline import dry_run_pipelinefrom pipelex.system.configuration.configs import NdjsonTracingConfig, TracingBackend@@ -31,6 +32,17 @@ output = "Text"prompt = "Echo something""""+_DRY_RUN_GRAPH_NO_MAIN_MTHDS = f"""+domain = "{_DRY_RUN_GRAPH_DOMAIN}_no_main"+description = "No-main bundle for explicit dry_run_pipeline graph target tests"++[pipe.echo_subject]+type = "PipeLLM"+description = "Pipe used to exercise explicit graph target generation"+output = "Text"+prompt = "Echo something"+"""+@pytest.mark.asyncio(loop_scope="class")class TestDryRunPipelineGraphTransport:@@ -49,6 +61,9 @@ class TestDryRunPipelineGraphTransport:assert pipe_code == f"{_DRY_RUN_GRAPH_DOMAIN}.echo_subject"traced_pipe_codes = {node.pipe_code for node in graph_spec.nodes if node.pipe_code}assert "echo_subject" in traced_pipe_codes+ assert "source" not in graph_spec.pipe_registry[f"{_DRY_RUN_GRAPH_DOMAIN}.echo_subject"]+ for concept_data in graph_spec.concept_registry.values():+ assert "source" not in concept_dataasync def test_scoped_log_takes_priority_over_enabled_backend(self, tmp_path_factory: pytest.TempPathFactory, mocker: MockerFixture) -> None:"""With tracing enabled, the scoped in-memory log still owns the run — no trace files written."""@@ -63,3 +78,27 @@ class TestDryRunPipelineGraphTransport:assert graph_spec.nodesassert not list(traces_dir.iterdir()), "the configured ndjson backend must not be touched"++ async def test_explicit_pipe_code_without_main_pipe_produces_graph(self, mocker: MockerFixture) -> None:+ """An explicit graph target removes the main_pipe requirement for graph generation."""+ mocker.patch.object(get_config().pipelex.tracing_config, "is_enabled", False)+ self._forbid_event_log_factory(mocker)++ graph_spec, pipe_code = await dry_run_pipeline(+ mthds_contents=[_DRY_RUN_GRAPH_NO_MAIN_MTHDS],+ pipe_code="echo_subject",+ )++ assert pipe_code == f"{_DRY_RUN_GRAPH_DOMAIN}_no_main.echo_subject"+ traced_pipe_codes = {node.pipe_code for node in graph_spec.nodes if node.pipe_code}+ assert traced_pipe_codes == {"echo_subject"}++ async def test_blank_explicit_pipe_code_is_rejected(self) -> None:+ """A blank explicit target must not silently fall back to main_pipe."""+ with pytest.raises(PipelexInterpreterError) as exc_info:+ await dry_run_pipeline(+ mthds_contents=[_DRY_RUN_GRAPH_MTHDS],+ pipe_code=" ",+ )++ assert str(exc_info.value) == "pipe_code must not be blank when provided"diff --git a/tests/integration/pipelex/pipeline/test_protocol_validate.py b/tests/integration/pipelex/pipeline/test_protocol_validate.pyindex 3c603a8a1..16ae69258 100644--- a/tests/integration/pipelex/pipeline/test_protocol_validate.py+++ b/tests/integration/pipelex/pipeline/test_protocol_validate.py@@ -159,6 +159,51 @@ class TestProtocolValidate:finally:clear_current_library()+ async def test_explicit_graph_target_without_main_pipe_produces_graph(self, load_empty_library: Callable[[], str]) -> None:+ """Protocol validate can graph an explicit target even when the bundle has no main_pipe."""+ load_empty_library()+ try:+ runner = PipelexMTHDSProtocol()+ report = await runner.validate(mthds_contents=[_COMPLETE_MTHDS], extra={"graph_pipe_code": "summarize"})++ assert isinstance(report, PipelexValidationReport)+ assert report.graph_spec is not None+ traced_pipe_codes = {node.pipe_code for node in report.graph_spec.nodes if node.pipe_code}+ assert traced_pipe_codes == {"summarize"}+ finally:+ clear_current_library()++ async def test_explicit_graph_target_overrides_main_pipe(self, load_empty_library: Callable[[], str]) -> None:+ """Protocol validate graphs the explicit target rather than the declared main_pipe."""+ load_empty_library()+ try:+ runner = PipelexMTHDSProtocol()+ report = await runner.validate(mthds_contents=[_MAIN_PIPE_MTHDS], extra={"graph_pipe_code": "summarize"})++ assert isinstance(report, PipelexValidationReport)+ assert report.graph_spec is not None+ traced_pipe_codes = {node.pipe_code for node in report.graph_spec.nodes if node.pipe_code}+ assert traced_pipe_codes == {"summarize"}+ assert "outline_then_summarize" not in traced_pipe_codes+ finally:+ clear_current_library()++ async def test_unknown_explicit_graph_target_degrades_to_no_graph(self, load_empty_library: Callable[[], str]) -> None:+ """Unknown graph targets stay best-effort: validation succeeds with graph_spec=None."""+ load_empty_library()+ try:+ runner = PipelexMTHDSProtocol()+ report = await runner.validate(+ mthds_contents=[_MAIN_PIPE_MTHDS],+ extra={"graph_pipe_code": "protocol_validate_graph.does_not_exist"},+ )++ assert isinstance(report, PipelexValidationReport)+ assert report.graph_spec is None+ assert report.is_runnable is True+ finally:+ clear_current_library()+async def test_graph_failure_mid_window_degrades_and_lifecycle_holds(self,load_empty_library: Callable[[], str],diff --git a/tests/integration/pipelex/pipeline/test_validate_graph_registry_sources.py b/tests/integration/pipelex/pipeline/test_validate_graph_registry_sources.pynew file mode 100644index 000000000..a734bc3dd--- /dev/null+++ b/tests/integration/pipelex/pipeline/test_validate_graph_registry_sources.py@@ -0,0 +1,46 @@+from pathlib import Path++import pytest++from pipelex.pipeline.validate_in_process import validate_bundles_in_process++_SOURCE_GRAPH_DOMAIN = "validate_graph_registry_sources"+_SOURCE_GRAPH_MTHDS = f"""+domain = "{_SOURCE_GRAPH_DOMAIN}"+description = "Bundle for graph registry source path tests"+main_pipe = "echo_topic"++[concept.Topic]+description = "A generated topic"++[concept.Topic.structure]+name = {{ type = "text", description = "Topic name" }}++[pipe.echo_topic]+type = "PipeLLM"+description = "Pipe used to verify graph registry source paths"+inputs = {{ subject = "Text" }}+output = "Topic"+prompt = "Echo the $subject as a topic"+"""++[email protected](loop_scope="class")+class TestValidateGraphRegistrySources:+ async def test_registry_entries_include_source_from_mthds_sources(self, tmp_path: Path) -> None:+ """A validation graph carries declaration sources from the loaded bundle source map."""+ bundle_path = tmp_path / "source_graph.mthds"+ bundle_path.write_text(_SOURCE_GRAPH_MTHDS, encoding="utf-8")+ source = str(bundle_path)++ report = await validate_bundles_in_process(+ mthds_contents=[bundle_path.read_text(encoding="utf-8")],+ mthds_sources=[source],+ library_dirs=[],+ log_context="test_validate_graph_registry_sources",+ )++ assert report.graph_spec is not None+ assert report.graph_spec.pipe_registry[f"{_SOURCE_GRAPH_DOMAIN}.echo_topic"]["source"] == source+ assert report.graph_spec.concept_registry[f"{_SOURCE_GRAPH_DOMAIN}.Topic"]["source"] == source+ assert report.graph_spec.concept_registry[f"{_SOURCE_GRAPH_DOMAIN}.Topic"]["json_schema"]diff --git a/tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py b/tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.pyindex c39385df5..80d88d162 100644--- a/tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py+++ b/tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py@@ -330,6 +330,30 @@ class TestDryValidateActivityInMemory:assert result.dry_run_outputs["dry_validate_no_main.lone_pipe"].status == DryRunStatus.SUCCESSassert result.graph_spec is None+ async def test_no_main_pipe_with_explicit_pipe_code_produces_graph(self, temporal_client: TemporalClient) -> None:+ """An explicit pipe_code graphs a no-main bundle in the Temporal validate activity."""+ result = await self._execute(+ temporal_client,+ DryValidateArg(mthds_contents=[_NO_MAIN_MTHDS], pipe_code="dry_validate_no_main.lone_pipe"),+ )++ assert result.dry_run_outputs["dry_validate_no_main.lone_pipe"].status == DryRunStatus.SUCCESS+ assert result.graph_spec is not None+ traced_pipe_codes = {node.pipe_code for node in result.graph_spec.nodes if node.pipe_code}+ assert traced_pipe_codes == {"lone_pipe"}++ async def test_explicit_pipe_code_overrides_main_pipe(self, temporal_client: TemporalClient) -> None:+ """The Temporal graph arm targets the explicit pipe_code rather than the bundle main_pipe."""+ result = await self._execute(+ temporal_client,+ DryValidateArg(mthds_contents=[_ALPHA_MTHDS], pipe_code="dry_validate_alpha.alpha_second"),+ )++ assert result.dry_run_outputs["dry_validate_alpha.alpha_sequence"].status == DryRunStatus.SUCCESS+ assert result.graph_spec is not None+ traced_pipe_codes = {node.pipe_code for node in result.graph_spec.nodes if node.pipe_code}+ assert traced_pipe_codes == {"alpha_second"}+async def test_unknown_explicit_pipe_code_degrades_to_no_graph(self, temporal_client: TemporalClient) -> None:"""Graph-arm parity: an unknown explicit pipe_code is a graph-arm domain failure(PipeNotFoundError, a PipelexError) and degrades to graph_spec=None with validationdiff --git a/tests/unit/pipelex/cli/test_agent_validate_cmd.py b/tests/unit/pipelex/cli/test_agent_validate_cmd.pyindex fa48e920f..8eb36dd2d 100644--- a/tests/unit/pipelex/cli/test_agent_validate_cmd.py+++ b/tests/unit/pipelex/cli/test_agent_validate_cmd.py@@ -101,3 +101,43 @@ class TestValidateBundleCmd:assert error_obj["error"] is Trueassert error_obj["error_type"] == "PipelineExecutionError"assert "graph" in error_obj["message"].lower()++ def test_view_generation_uses_pipe_override(+ self,+ mocker: MockerFixture,+ capsys: pytest.CaptureFixture[str],+ tmp_path: Path,+ ) -> None:+ """Validate bundle --pipe X --view should graph X instead of falling back to main_pipe."""+ mthds_file = tmp_path / "bundle.mthds"+ mthds_file.write_text('[bundle]\nmain_pipe = "main_pipe"\n[domain]\ncode = "test"')++ mocker.patch(f"{VALIDATE_BUNDLE_CMD_MODULE}.make_pipelex_for_agent_cli")+ mocker.patch(f"{VALIDATE_BUNDLE_CMD_MODULE}.Pipelex.teardown_if_needed")++ validation_result: dict[str, Any] = {+ "success": True,+ "bundle_path": str(mthds_file),+ "validated_pipes": [{"pipe_ref": "other_pipe", "status": "SUCCESS"}],+ "total_pipes": 1,+ }+ mocker.patch(+ f"{VALIDATE_BUNDLE_CMD_MODULE}.validate_pipe_in_bundle_core",+ new=mocker.AsyncMock(return_value=validation_result),+ )+ view_mock = mocker.patch(+ f"{VALIDATE_BUNDLE_CMD_MODULE}.generate_view_for_bundle",+ new=mocker.AsyncMock(return_value={"graphspec": {}, "pipe_code": "other_pipe"}),+ )++ validate_bundle_cmd(+ path=str(mthds_file),+ pipe="other_pipe",+ view=True,+ output_format=CliOutputFormat.JSON,+ )++ view_mock.assert_awaited_once()+ assert view_mock.call_args.kwargs["pipe_code"] == "other_pipe"+ output = json.loads(capsys.readouterr().out)+ assert output["pipe_code"] == "other_pipe"diff --git a/tests/unit/pipelex/graph/test_graph_rendering_generate_graph.py b/tests/unit/pipelex/graph/test_graph_rendering_generate_graph.pyindex f612639a7..329d4af21 100644--- a/tests/unit/pipelex/graph/test_graph_rendering_generate_graph.py+++ b/tests/unit/pipelex/graph/test_graph_rendering_generate_graph.py@@ -163,3 +163,20 @@ class TestGenerateGraphForBundle:assert result["graph_output_dir"] == str(tmp_path)assert result["pipe_code"] == "pipe_code"assert result["direction"] == expected_direction++ async def test_pipe_code_override_passed_to_dry_run(+ self,+ mocker: MockerFixture,+ tmp_path: Path,+ ) -> None:+ """An explicit graph target should be passed to the pipeline dry-run helper."""+ mocks = self._setup_mocks(mocker, tmp_path, saved_files={})++ await generate_graph_for_bundle(+ bundle_path=mocks["bundle_path"],+ graph_format=GraphFormat.REACTFLOW,+ pipe_code="other_pipe",+ )++ mocks["mock_dry_run"].assert_awaited_once()+ assert mocks["mock_dry_run"].call_args.kwargs["pipe_code"] == "other_pipe"diff --git a/tests/unit/pipelex/graph/test_graph_rendering_generate_view.py b/tests/unit/pipelex/graph/test_graph_rendering_generate_view.pyindex 0bfcc5376..f3a80f264 100644--- a/tests/unit/pipelex/graph/test_graph_rendering_generate_view.py+++ b/tests/unit/pipelex/graph/test_graph_rendering_generate_view.py@@ -28,14 +28,14 @@ class TestGenerateViewForBundle:mocker: MockerFixture,tmp_path: Path,config_layout_direction: FlowchartDirection | None,- ) -> tuple[Path, Any]:+ ) -> tuple[Path, Any, Any]:"""Create the bundle file, patch collaborators, and return the bundle path and graph spec mock."""bundle_path = tmp_path / "bundle.mthds"bundle_path.write_text(BUNDLE_CONTENT, encoding="utf-8")graph_spec_mock = mocker.MagicMock()graph_spec_mock.model_dump.return_value = {"nodes": "node_payload", "edges": "edge_payload"}- mocker.patch(+ dry_run_mock = mocker.patch(f"{GRAPH_RENDERING_MODULE}.dry_run_pipeline",new_callable=mocker.AsyncMock,return_value=(graph_spec_mock, "pipe_code"),@@ -47,7 +47,7 @@ class TestGenerateViewForBundle:config_mock.pipelex.pipeline_execution_config.with_execution_overrides.return_value = execution_config_mockmocker.patch(f"{GRAPH_RENDERING_MODULE}.get_config", return_value=config_mock)- return bundle_path, graph_spec_mock+ return bundle_path, graph_spec_mock, dry_run_mockasync def test_returns_graphspec_json_dump(self,@@ -55,7 +55,7 @@ class TestGenerateViewForBundle:tmp_path: Path,) -> None:"""The graphspec key should carry the by-alias JSON model dump of the dry-run GraphSpec."""- bundle_path, graph_spec_mock = self._setup_mocks(mocker, tmp_path, config_layout_direction=None)+ bundle_path, graph_spec_mock, _dry_run_mock = self._setup_mocks(mocker, tmp_path, config_layout_direction=None)result = await generate_view_for_bundle(bundle_path=bundle_path)@@ -80,8 +80,21 @@ class TestGenerateViewForBundle:expected_direction: str | None,) -> None:"""An explicit direction wins, else the reactflow config layout direction, else None."""- bundle_path, _graph_spec_mock = self._setup_mocks(mocker, tmp_path, config_layout_direction=config_layout_direction)+ bundle_path, _graph_spec_mock, _dry_run_mock = self._setup_mocks(mocker, tmp_path, config_layout_direction=config_layout_direction)result = await generate_view_for_bundle(bundle_path=bundle_path, direction=direction_arg)assert result["direction"] == expected_direction++ async def test_pipe_code_override_passed_to_dry_run(+ self,+ mocker: MockerFixture,+ tmp_path: Path,+ ) -> None:+ """An explicit graph target should be passed to the pipeline dry-run helper."""+ bundle_path, _graph_spec_mock, dry_run_mock = self._setup_mocks(mocker, tmp_path, config_layout_direction=None)++ await generate_view_for_bundle(bundle_path=bundle_path, pipe_code="other_pipe")++ dry_run_mock.assert_awaited_once()+ assert dry_run_mock.call_args.kwargs["pipe_code"] == "other_pipe"diff --git a/tests/unit/pipelex/pipeline/test_runner_protocol_surfaces.py b/tests/unit/pipelex/pipeline/test_runner_protocol_surfaces.pyindex 2edf1f547..cf9148e55 100644--- a/tests/unit/pipelex/pipeline/test_runner_protocol_surfaces.py+++ b/tests/unit/pipelex/pipeline/test_runner_protocol_surfaces.py@@ -55,6 +55,18 @@ class TestRunnerProtocolSurfaces:assert str(exc_info.value) == "The local runtime defines no extension args; got ['abc', 'zed']."+ async def test_validate_rejects_blank_graph_pipe_code_extra(self) -> None:+ """A blank explicit graph target must not silently fall back to main_pipe."""+ runner = PipelexMTHDSProtocol()++ with pytest.raises(PipelineRequestError) as exc_info:+ await runner.validate(+ mthds_contents=['domain = "blank_graph_target"'],+ extra={"graph_pipe_code": " "},+ )++ assert str(exc_info.value) == "validate extra 'graph_pipe_code' must not be blank when provided."+async def test_start_raises_not_implemented_pointing_at_execute(self) -> None:"""The local runtime does not implement async start; the message redirects to execute."""runner = PipelexMTHDSProtocol()diff --git a/tests/unit/pipelex/tracing/test_graphspec_assembler.py b/tests/unit/pipelex/tracing/test_graphspec_assembler.pyindex cf48071a7..94fd57fb7 100644--- a/tests/unit/pipelex/tracing/test_graphspec_assembler.py+++ b/tests/unit/pipelex/tracing/test_graphspec_assembler.py@@ -69,6 +69,54 @@ class TestGraphSpecAssembler:assert result.nodes == []assert result.edges == []+ def test_registry_source_payloads_survive_assembly(self) -> None:+ """PipeStartEvent and PipeEndSuccessEvent registry payloads are copied unchanged."""+ pipe_source = "/fake/sourceful_pipe.mthds"+ input_concept_source = "/fake/input_concepts.mthds"+ output_concept_source = "/fake/output_concepts.mthds"+ node_id = _node_id(_WF_A, 0)++ result = GraphSpecAssembler.assemble(+ events=[+ PipeStartEvent(+ **_base(_WF_A, 0),+ node_id=node_id,+ pipe_code="sourceful_pipe",+ pipe_type="PipeLLM",+ node_kind=NodeKind.OPERATOR,+ domain_code="sourceful",+ pipe_data={+ "code": "sourceful_pipe",+ "domain_code": "sourceful",+ "type": "PipeLLM",+ "source": pipe_source,+ },+ concept_data=[+ {+ "code": "InputConcept",+ "domain_code": "sourceful",+ "source": input_concept_source,+ }+ ],+ ),+ PipeEndSuccessEvent(+ **_base(_WF_A, 1),+ node_id=node_id,+ ended_at=_time_at(1),+ output_concept_data={+ "code": "OutputConcept",+ "domain_code": "sourceful",+ "source": output_concept_source,+ },+ ),+ ],+ graph_id=_GRAPH_ID,+ )++ assert result.pipe_registry["sourceful.sourceful_pipe"]["source"] == pipe_source+ assert result.concept_registry["sourceful.InputConcept"]["source"] == input_concept_source+ assert result.concept_registry["sourceful.OutputConcept"]["source"] == output_concept_source+def test_simple_sequence(self) -> None:"""3 child pipes in sequence under a parent.diff --git a/wip/graph/graphspec-source-map-enrichment.md b/wip/graph/graphspec-source-map-enrichment.mdnew file mode 100644index 000000000..3394025fd--- /dev/null+++ b/wip/graph/graphspec-source-map-enrichment.md@@ -0,0 +1,185 @@+# GraphSpec Registry Source Paths++## Question++Can `GraphSpec` include the source bundle path for each pipe and concept by enriching `pipe_registry` and `concept_registry` entries from `LibraryCrate.source_map`?++Short answer: yes, this is feasible, and the lowest-churn path is to add a `source` field to registry payload dictionaries at graph event creation time.++## Current State++`GraphSpec` has:++- `nodes[]`: execution nodes with `pipe_code`, `pipe_type`, `description`, `domain_code`, IO, timing, status, metrics, and execution data.+- `pipe_registry`: `dict[str, dict[str, Any]]`, keyed by `domain.pipe_code`.+- `concept_registry`: `dict[str, dict[str, Any]]`, keyed by `domain.ConceptCode`.++The registry entries are optional. They are emitted only when `graph_config.data_inclusion.pipe_and_concept_registry` is true, which is true in the shipped config.++Today those registry entries do not include bundle source paths:++- pipe registry entries come from `PipeAbstract.model_dump(mode="json")`;+- concept registry entries come from `Concept.model_dump(mode="json")` plus `json_schema`;+- runtime `PipeAbstract` has no `source` field;+- runtime `Concept` does not receive the blueprint `source` when created.++The source data exists elsewhere:++- `PipeBlueprint.source` and `ConceptBlueprint.source` exist at blueprint level.+- `LibraryCrate.source_map` maps both `pipe_ref` and `concept_ref` to the source file path.+- `LibraryManager.get_pipe_source()` exposes pipe source for diagnostics, backed by the crate source map.+- `PipeJob.library_crate` is already attached by `prepare_pipe_job()` and crosses the Temporal boundary.++## Recommended Design++Enrich registry payloads before they are handed to the tracer:++- pipe registry entry: add `source` from `library_crate.source_map[self.pipe_ref]`;+- concept registry entry: add `source` from `library_crate.source_map[concept.concept_ref]`;+- if no crate or no source map entry exists, omit the field rather than emitting `source: null`.++This keeps source paths attached to the existing registry artifacts and avoids changing `NodeSpec`, `PipeAbstract`, or `Concept`.++Example resulting shape:++```json+{+ "pipe_registry": {+ "research.summarize": {+ "code": "summarize",+ "domain_code": "research",+ "type": "PipeLLM",+ "description": "Summarize notes",+ "inputs": {},+ "output": {},+ "source": "/workspace/research/summarize.mthds"+ }+ },+ "concept_registry": {+ "research.Summary": {+ "code": "Summary",+ "domain_code": "research",+ "description": "A summary",+ "structure_class_name": "...",+ "json_schema": {},+ "source": "/workspace/research/concepts.mthds"+ }+ }+}+```++## Why This Is Feasible++The graph event creation point has access to both sides:++- `PipeAbstract._run_pipe_traced(..., library_crate: LibraryCrate | None = None)` already receives the crate.+- The same function currently builds `pipe_data` and `concept_data` before calling `tracer_manager.on_pipe_start(...)`.+- `PipeAbstract._make_single_concept_data_for_registry()` already centralizes concept registry payload construction.+- `PipeStartEvent.pipe_data` and `PipeStartEvent.concept_data` are generic dictionaries, so serialized Temporal graph events can carry the additional `source` field without schema changes.+- `GraphTracer` and `GraphSpecAssembler` already copy those dictionaries into `GraphSpec.pipe_registry` / `concept_registry` unchanged.++No `GraphSpec` schema migration is required because the registries are already `dict[str, Any]` payloads.++## Implementation Plan++1. Thread source-map awareness into registry serialization.++ Suggested helpers in `PipeAbstract`:++ ```python+ def _make_pipe_data_for_registry(self, *, library_crate: LibraryCrate | None) -> dict[str, Any]:+ pipe_data = self.model_dump(mode="json")+ if library_crate is not None:+ source = library_crate.source_map.get(self.pipe_ref)+ if source:+ pipe_data["source"] = source+ return pipe_data+ ```++ Update concept helpers similarly:++ ```python+ def _make_single_concept_data_for_registry(+ self,+ concept: Concept,+ *,+ library_crate: LibraryCrate | None,+ ) -> dict[str, Any]:+ concept_dict = concept.model_dump(mode="json")+ if library_crate is not None:+ source = library_crate.source_map.get(concept.concept_ref)+ if source:+ concept_dict["source"] = source+ ...+ ```++2. Use those helpers from `_run_pipe_traced()`.++ Replace:++ ```python+ pipe_data = self.model_dump(mode="json")+ concept_data = self._make_concept_data_for_registry()+ ```++ with source-aware helper calls using the `library_crate` argument already present in `_run_pipe_traced()`.++3. Preserve current behavior when source is unavailable.++ This happens for:++ - in-memory protocol validation without `mthds_sources`;+ - tests or internal call sites that build `PipeJob` without `library_crate`;+ - native concepts or concepts loaded from code rather than a `.mthds` declaration;+ - dependency scenarios where the active crate does not include a source-map entry for a cross-package ref.++ In all of those cases, omit `source`.++4. Keep source on registry entries, not nodes.++ A node is an invocation. A registry entry is the declaration. Source path is declaration metadata, so the registry is the right layer. Renderers can join `node.domain_code + "." + node.pipe_code` to `pipe_registry` when they need to show source.++## Tests++Add coverage for both direct and assembled graph paths.++1. Unit-level helper test in the graph or pipe area:++ - construct a pipe with a fake `LibraryCrate(source_map={pipe_ref: "...", concept_ref: "..."})`;+ - run with `pipe_and_concept_registry=True`;+ - assert registry payloads include `source`.++2. Direct/in-memory graph integration:++ - use a real `.mthds` bundle loaded from disk;+ - run graph generation;+ - assert `graph_spec.pipe_registry["domain.pipe"]["source"]` is the bundle path;+ - assert declared concepts in `concept_registry` include their source path.++3. Temporal/event-assembled path:++ - ensure `PipeStartEvent.pipe_data` / `concept_data` carry `source`;+ - assert `GraphSpecAssembler` preserves the fields into `pipe_registry` / `concept_registry`.++4. Sourceless path:++ - validate/run from raw in-memory `mthds_contents` without `mthds_sources`;+ - assert graph generation still succeeds and registry entries simply omit `source`.++## Edge Cases++- Multi-file same-domain bundles: `LibraryCrate.source_map` already tracks the winning declaration source for each `pipe_ref` / `concept_ref`, including signature/concrete reconciliation, so use it as the source of truth.+- Duplicate declarations: no new logic needed; the crate factory already rejects or reconciles before execution.+- Cross-package dependencies: confirm whether the `PipeJob.library_crate` used by the executing pipe includes dependency source-map entries. If not, source enrichment will be incomplete for dependency pipes. That is acceptable for a first pass, but it should be documented and pinned with a test or follow-up.+- Privacy: source paths can be absolute filesystem paths. That is already true in validation diagnostics and `LibraryCrate.source_map`, but exposing them through API graph responses is a product/API decision. If public hosted responses should not leak worker filesystem paths, add a normalization/redaction policy before shipping this on hosted surfaces.++## Files To Touch++- `pipelex/core/pipes/pipe_abstract.py`+- tests under `tests/unit/pipelex/graph/` or `tests/unit/pipelex/core/pipes/`+- graph integration tests under `tests/integration/pipelex/pipeline/` or existing graph tracing tests+- Temporal tracing tests if the event-assembled path needs explicit coverage++## Recommendation++Implement this as a registry enrichment, not as a structural `GraphSpec` model change. It is additive, works for direct and Temporal graph assembly, and uses the already-canonical `LibraryCrate.source_map` instead of duplicating source-tracking state onto runtime pipe/concept models.diff --git a/wip/graph/validate-explicit-graph-target.md b/wip/graph/validate-explicit-graph-target.mdnew file mode 100644index 000000000..00d7e6169--- /dev/null+++ b/wip/graph/validate-explicit-graph-target.md@@ -0,0 +1,150 @@+# Validate Graphs With an Explicit Pipe Target++## Problem++Validation can only attach a `graph_spec` when the submitted batch declares a `main_pipe`. The current path is deliberate:++- `select_primary_blueprint()` picks the first blueprint declaring `main_pipe`; if none does, it returns the first blueprint with `main_pipe_ref=None`.+- `validate_bundles_in_process()` passes that `main_pipe_ref` to `best_effort_graph_spec()`.+- `best_effort_graph_spec()` returns `None` when `pipe_ref` is `None`.+- The CLI graph helpers use `dry_run_pipeline()`, which is stricter: no `main_pipe` raises `PipelexInterpreterError("Bundle does not declare a main_pipe, cannot generate graph")`.++This makes sense for the default report because `GraphSpec` is currently an execution trace for one root pipe, not a static diagram of every declared pipe in the bundle. But it blocks a useful case: a valid bundle with no declared `main_pipe`, or a partially built bundle where the caller wants to validate and graph one specific pipe.++## Proposed Lift++Support an explicit graph target pipe for validation graph generation.++The rule becomes:++- If the caller provides a graph target pipe, dry-run that pipe for `graph_spec`.+- Otherwise keep today's default: use the batch's selected `main_pipe`.+- If neither exists, keep returning `graph_spec=None` on protocol/API validation and keep the CLI graph command's explicit "cannot generate graph" error when a graph was requested.++This is intentionally not "graph the whole bundle." Whole-bundle static graphing is a different artifact with different semantics. This lift stays inside the existing execution-trace model by choosing one pipe to execute.++## Existing Machinery To Reuse++Most of the lower-level plumbing already supports this:++- `best_effort_graph_spec(pipe_ref=...)` can dry-run any resolved pipe ref against the already-open validation library.+- `act_dry_validate.DryValidateArg.pipe_code` already exists and is used as an override before falling back to `select_primary_blueprint(...).main_pipe_ref`.+- `pipeline_run_setup()` already supports `pipe_code` overriding `main_pipe` when executing MTHDS contents.+- `validate_bundle(..., dry_run_pipe_codes=[pipe_code])` already validates a single selected pipe within a bundle.++The missing work is making the direct/public validation surfaces carry the graph target consistently, and making CLI graph/view generation use it.++## Implementation Shape++1. Add an optional graph target parameter to the shared in-process validate orchestrator.++ Suggested signature:++ ```python+ async def validate_bundles_in_process(+ *,+ mthds_contents: list[str],+ mthds_sources: list[str] | None = None,+ library_dirs: Sequence[Path] | None = None,+ allow_signatures: bool = False,+ graph_pipe_code: str | None = None,+ log_context: str = "validate",+ ) -> PipelexValidationReport:+ ```++ The graph arm should call:++ ```python+ graph_target_ref = graph_pipe_code or select_primary_blueprint(result.blueprints).main_pipe_ref+ graph_spec = await best_effort_graph_spec(+ pipe_ref=graph_target_ref,+ library_id=validation_library_id,+ log_context=log_context,+ )+ ```++ This preserves today's `None` behavior when no target exists. Unknown `graph_pipe_code` should degrade to `graph_spec=None` on the best-effort report path, because `best_effort_graph_spec()` intentionally catches pipe-resolution domain failures.++2. Thread the same option through `PipelexMTHDSProtocol.validate()` if the protocol layer has an extension argument for it.++ Today local protocol `validate()` rejects every `extra` key. Two reasonable choices:++ - accept `extra={"graph_pipe_code": "..."}` for Pipelex-specific local runtime usage;+ - leave protocol `validate()` unchanged and expose the option only on API/CLI surfaces that already have concrete request fields.++ If the hosted MTHDS Protocol surface needs this, prefer a typed request field in the API schema and map it into the shared orchestrator rather than relying on untyped `extra`.++3. Wire the Temporal path using the field that already exists.++ `DryValidateArg.pipe_code` already chooses:++ ```python+ arg.pipe_code or select_primary_blueprint(validate_result.blueprints).main_pipe_ref+ ```++ Check the API dispatch layer and ensure the request's graph target field is passed into `DryValidateArg.pipe_code`. Rename to `graph_pipe_code` later if clarity is worth the wire break; no compatibility constraint blocks that, but avoid churn unless the surrounding API field also uses that name.++4. Update agent CLI `validate bundle --pipe X --graph/--view`.++ Today `--pipe` narrows validation, but `--graph` and `--view` still call `generate_graph_for_bundle()` / `generate_view_for_bundle()`, which eventually call `dry_run_pipeline()` with no pipe override and therefore require `main_pipe`.++ Add a pipe override to the graph rendering helpers:++ ```python+ async def _dry_run_bundle(+ bundle_path: Path,+ *,+ library_dirs: list[str] | None = None,+ pipe_code: str | None = None,+ ) -> tuple[GraphSpec, str]:+ ```++ Then either:++ - extend `dry_run_pipeline(..., pipe_code: str | None = None)` and let it pass the override into `runner.execute(pipe_code=pipe_code or main_pipe_ref, ...)`; or+ - bypass the pre-parse `main_pipe` requirement in `_dry_run_bundle()` when `pipe_code` is supplied.++ The first option is cleaner because it keeps one graph-producing bundle execution helper.++5. Decide bare vs qualified pipe target rules.++ Current pipe library lookup accepts either a unique bare code or a qualified `domain.code` ref. For multi-domain batches, qualified refs are safer. Recommended behavior:++ - accept both bare and qualified targets;+ - document that bare targets must be unique in the loaded library;+ - let existing `PipeLibrary.get_required_pipe()` ambiguity errors flow through the best-effort degradation path for report graph generation and through explicit CLI graph errors when the user requested graph output.++## Tests++Add tests at the existing pin points:++- `tests/integration/pipelex/pipeline/test_protocol_validate.py`+ - no `main_pipe` plus explicit graph target produces a non-null `graph_spec`;+ - explicit graph target overrides declared `main_pipe`;+ - unknown explicit graph target degrades to `graph_spec=None` on the validation report path.++- Agent CLI validate bundle tests+ - `validate bundle no_main.mthds --pipe some_pipe --view` succeeds and includes a `graphspec`;+ - `validate bundle no_main.mthds --view` still errors because graph was explicitly requested and no target exists;+ - `validate bundle bundle_with_main.mthds --pipe other_pipe --view` graphs `other_pipe`, not the declared `main_pipe`.++- Temporal dry-validate/API path+ - request with explicit graph target maps to `DryValidateArg.pipe_code`;+ - direct and Temporal modes produce the same `graph_spec` target behavior.++## Non-Goals++- Do not infer a root from "the only pipe in the bundle" unless product explicitly wants that policy. It sounds convenient but creates a hidden second default alongside `main_pipe`.+- Do not create a static all-pipes `GraphSpec` under the same field. That would overload an execution-trace artifact with non-execution semantics.+- Do not require `main_pipe` to validate a bundle. Current validation correctly supports concept-only files, membership-only files, and top-down partial bundles.++## Files To Touch++- `pipelex/pipeline/validate_in_process.py`+- `pipelex/pipeline/runner.py` if protocol `extra` or a typed protocol extension should expose it+- hosted API validate request/dispatch mapping in `pipelex-api`+- `pipelex/temporal/tprl_pipe/act_dry_validate.py` only if renaming or tightening `pipe_code`+- `pipelex/pipeline/dry_run_pipeline.py`+- `pipelex/graph/graph_rendering.py`+- `pipelex/cli/agent_cli/commands/validate/bundle_cmd.py`+- tests under `tests/integration/pipelex/pipeline/` and agent CLI validate coverage