Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
docs: fix code-doc drift across under-the-hood + advanced pages (Drift Hunt S2-3) (#1042)
Louis Choquel9814e2f4c12b
diff --git a/docs/advanced/index.md b/docs/advanced/index.mdindex 225473de9..b5a12f221 100644--- a/docs/advanced/index.md+++ b/docs/advanced/index.md@@ -15,7 +15,7 @@ There are two main ways to inject custom implementations:### 1. During Initialization```python-from pipelex import Pipelex+from pipelex.pipelex import Pipelexpipelex = Pipelex.make(reporting_delegate=MyReportingDelegate(),@@ -31,7 +31,7 @@ pipelex = Pipelex.make(from pipelex.hub import PipelexHubhub = PipelexHub()-hub.set_reporting_delegate(MyReportingDelegate())+hub.set_report_delegate(MyReportingDelegate())# ... and so on for other components```diff --git a/docs/advanced/observer-provider-injection.md b/docs/advanced/observer-provider-injection.mdindex f0b8c8597..0d9999153 100644--- a/docs/advanced/observer-provider-injection.md+++ b/docs/advanced/observer-provider-injection.md@@ -21,20 +21,20 @@ This data collection is designed for later analysis and is still a work in progrAll observers must implement the `ObserverProtocol`:```python-from typing import Any, Dict, Protocol+from typing import Any, Protocol-PayloadType = Dict[str, Any]+PayloadType = dict[str, Any]class ObserverProtocol(Protocol):- async def before_run(self, payload: PayloadType) -> None:+ async def observe_before_run(self, payload: PayloadType) -> None:"""Process and store the payload before the run"""...- async def successful_run(self, payload: PayloadType) -> None:+ async def observe_after_successful_run(self, payload: PayloadType) -> None:"""Process and store the payload after the run is successful"""...- async def failing_run(self, payload: PayloadType) -> None:+ async def observe_after_failing_run(self, payload: PayloadType) -> None:"""Process and store the payload after the run fails"""...```@@ -46,6 +46,7 @@ The payload contains different information depending on the execution phase:### Before Run```pythonpayload = {+ "pipeline_run_id": str, # Identifier of the pipeline run"pipe_job": PipeJob, # Contains pipe metadata and working memory}```@@ -53,6 +54,7 @@ payload = {### Successful Run```pythonpayload = {+ "pipeline_run_id": str, # Identifier of the pipeline run"pipe_job": PipeJob, # Initial job information"pipe_output": PipeOutput, # Results and final working memory}@@ -61,8 +63,9 @@ payload = {### Failing Run```pythonpayload = {+ "pipeline_run_id": str, # Identifier of the pipeline run"pipe_job": PipeJob, # Initial job information- # Note: error details are handled separately by the pipe router+ "error": Exception, # The exception that caused the failure}```@@ -80,24 +83,24 @@ class MyCustomObserver(ObserverProtocol):self.config_param = config_param# Initialize your storage/logging mechanism here- async def before_run(self, payload: PayloadType) -> None:+ async def observe_before_run(self, payload: PayloadType) -> None:pipe_job = payload["pipe_job"]# Process pipe_job data before execution# Example: Log pipe code, inputs, parameters- print(f"Starting pipe: {pipe_job.pipe_code}")+ print(f"Starting pipe: {pipe_job.pipe.code}")- async def successful_run(self, payload: PayloadType) -> None:+ async def observe_after_successful_run(self, payload: PayloadType) -> None:pipe_job = payload["pipe_job"]pipe_output = payload["pipe_output"]# Process successful execution data# Example: Log execution time, output size, tokens used- print(f"Completed pipe: {pipe_job.pipe_code}")+ print(f"Completed pipe: {pipe_job.pipe.code}")- async def failing_run(self, payload: PayloadType) -> None:+ async def observe_after_failing_run(self, payload: PayloadType) -> None:pipe_job = payload["pipe_job"]# Process failure data# Example: Log error context, partial results- print(f"Failed pipe: {pipe_job.pipe_code}")+ print(f"Failed pipe: {pipe_job.pipe.code}")```### Step 2: Register Your Observer with Dependency Injection@@ -119,9 +122,9 @@ hub.set_observer(my_observer)The observer is automatically called by the PipeRouterProtocol during pipe execution:-1. `before_run()` is called before any pipe execution-2. `successful_run()` is called after successful completion-3. `failing_run()` is called when an exception occurs+1. `observe_before_run()` is called before any pipe execution+2. `observe_after_successful_run()` is called after successful completion+3. `observe_after_failing_run()` is called when an exception occurs## Built-in LocalObserver Example@@ -143,8 +146,8 @@ get_pipelex_hub().set_observer(local_observer)The LocalObserver creates separate JSONL files for each event type:- `before_run.jsonl` - Pre-execution data-- `successful_run.jsonl` - Success events-- `failing_run.jsonl` - Failure events+- `after_successful_run.jsonl` - Success events+- `after_failing_run.jsonl` - Failure events## Advanced Use Cases@@ -159,15 +162,15 @@ class DatabaseObserver(ObserverProtocol):self.db_connection = db_connection_string# Initialize database connection- async def before_run(self, payload: PayloadType) -> None:+ async def observe_before_run(self, payload: PayloadType) -> None:# Store in databaseawait self._store_event("before_run", payload)- async def successful_run(self, payload: PayloadType) -> None:- await self._store_event("successful_run", payload)+ async def observe_after_successful_run(self, payload: PayloadType) -> None:+ await self._store_event("after_successful_run", payload)- async def failing_run(self, payload: PayloadType) -> None:- await self._store_event("failing_run", payload)+ async def observe_after_failing_run(self, payload: PayloadType) -> None:+ await self._store_event("after_failing_run", payload)async def _store_event(self, event_type: str, payload: PayloadType):# Your database storage logic here@@ -180,17 +183,17 @@ class MetricsObserver(ObserverProtocol):def __init__(self, metrics_client):self.metrics = metrics_client- async def before_run(self, payload: PayloadType) -> None:+ async def observe_before_run(self, payload: PayloadType) -> None:pipe_job = payload["pipe_job"]- self.metrics.increment(f"pipe.{pipe_job.pipe_code}.started")+ self.metrics.increment(f"pipe.{pipe_job.pipe.code}.started")- async def successful_run(self, payload: PayloadType) -> None:+ async def observe_after_successful_run(self, payload: PayloadType) -> None:pipe_job = payload["pipe_job"]- self.metrics.increment(f"pipe.{pipe_job.pipe_code}.success")+ self.metrics.increment(f"pipe.{pipe_job.pipe.code}.success")- async def failing_run(self, payload: PayloadType) -> None:+ async def observe_after_failing_run(self, payload: PayloadType) -> None:pipe_job = payload["pipe_job"]- self.metrics.increment(f"pipe.{pipe_job.pipe_code}.failed")+ self.metrics.increment(f"pipe.{pipe_job.pipe.code}.failed")```## Best Practices@@ -206,7 +209,9 @@ class MetricsObserver(ObserverProtocol):For automatic observer setup, integrate with your Pipelex initialization:```python-from pipelex import Pipelex+from pipelex.hub import get_pipelex_hub+from pipelex.pipelex import Pipelex+from my_project.observers import MyCustomObserverdef setup_pipelex():diff --git a/docs/under-the-hood/build-time-elaboration.md b/docs/under-the-hood/build-time-elaboration.mdindex 6a42fe8a4..72456424a 100644--- a/docs/under-the-hood/build-time-elaboration.md+++ b/docs/under-the-hood/build-time-elaboration.md@@ -78,8 +78,8 @@ Three layers guard against authoring mistakes around the output concept:The elaborator additionally:- Verifies that the synthetic codes (`<pipe_code>__draft_text`, `<pipe_code>__structure`) don't already exist in the bundle.-- Verifies that the synthetic codes pass `is_pipe_code_valid` (snake_case + length).-- After producing the new bundle, re-runs `PipelexBundleBlueprint.model_validate(elaborated.model_dump(...))` so bundle-level validators (concept refs, pipe refs, `main_pipe`) re-check the synthetic pipes. Any `ValidationError` is wrapped in `BundleElaboratorError` with context naming the originating pipe.+- Verifies that the synthetic codes pass `is_pipe_code_valid` (snake_case).+- After producing the new bundle, re-runs `PipelexBundleBlueprint.model_validate(elaborated.model_dump(...))` so bundle-level validators (concept refs, pipe refs, `main_pipe`) re-check the synthetic pipes. Any `ValidationError` here is wrapped in `PipelexUnexpectedError`, with a message naming the bundle's `domain` and the sorted list of synthetic pipe codes (`BundleElaboratorError` is used only for the earlier synthesis-time checks during the same `elaborate()` call).- Asserts that no synthesized blueprint itself carries `structuring_method = "preliminary_text"` — a recursive-elaboration guard. Today's synthesis explicitly sets `None`; the guard protects against future elaboration kinds copying fields wholesale.## Cost and runtime semanticsdiff --git a/docs/under-the-hood/codegen-projections.md b/docs/under-the-hood/codegen-projections.mdindex e49195894..19f84e72f 100644--- a/docs/under-the-hood/codegen-projections.md+++ b/docs/under-the-hood/codegen-projections.md@@ -2,7 +2,7 @@Codegen turns one **normalized library crate** — the flat, fully-qualified, self-contained, fingerprinted snapshot of a resolved library (see [`pipelex resolve`](../tools/cli/build/structures.md) and the [Library Crate Format](https://mthds.ai)) — into typed, documented artifacts for each consumer. The crate is the single authority: codegen never re-derives meaning from raw bundles and never calls a model.-This page describes the projection engine in `pipelex/codegen/`. The formal contract lives in `docs/specs/pipelex-codegen.md` (workspace root).+This page describes the projection engine in `pipelex/codegen/` and the contract it implements: deterministic projections off the crate, stamped generated files, a sibling `codegen.lock`, and an offline drift check whose verdict rides the exit code.## Two resolved layers, then emitters@@ -37,7 +37,7 @@ Native materialization itself is faithful-or-absent per native: a content-class## The CLI surface-Two command families drive the engine (formal contract: `docs/specs/pipelex-codegen.md`, workspace root):+Two command families drive the engine:- **`pipelex resolve [PATH]… [-f json|toml] [-L DIR]…`** — assembles the closure (working bundles + the local `.mthds/methods/` cache), requires it to be **valid**, and emits the normalized crate to stdout. The verdict rides the exit code, mirroring the bare `validate` group: `0` resolved, `1` the library is invalid (a negative verdict — no crate), `2` no verdict (empty closure / not found).- **`pipelex codegen <kind> …`** — the two-axis projection family (`kind` × `--target`):@@ -89,7 +89,7 @@ The verdict rides the exit code (mirroring `resolve` / `validate`): `0` current,## Serving the engine over HTTP-The same engine backs the `pipelex-api` routes (`POST /v1/resolve`, `POST /v1/codegen`, and the re-pointed `/v1/build/*` — see `docs/specs/pipelex-codegen.md` → "Route envelopes" at the workspace root). Two host-facing cores make that possible without any CLI plumbing:+The same engine backs the `pipelex-api` routes (`POST /v1/resolve`, `POST /v1/codegen`, and the re-pointed `/v1/build/*` — the route envelopes are documented in `pipelex-api`'s `docs/codegen.md`). Two host-facing cores make that possible without any CLI plumbing:- **`pipelex.pipeline.resolve_bundle.resolve_crate_from_contents`** resolves **in-memory** MTHDS contents (strings, with optional per-content sources) into the normalized crate. It mirrors `validate_bundle`'s in-memory arm — the same `translate_to_validate_bundle_error` cascade, so an invalid library raises the one shared `ValidateBundleError` and a resolve verdict cannot drift from a validate verdict — and the same **loaded-on-success contract**: the library is left loaded and current for the host to read live pipes from, and the host owns its teardown. Resolution is static (no dry-run sweep), matching `pipelex resolve`.- **`build_stamped_projection`** (above) gives the host the stamped artifact set plus the lock as pure content. A client that writes the served files and lock verbatim reproduces a local run byte-for-byte — the offline `codegen check` passes on the written tree exactly as it would locally. There is deliberately no server-side check route: the check is offline by design.diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.mdindex 01e2e9434..5e3710dfa 100644--- a/docs/under-the-hood/distributed-content-generation.md+++ b/docs/under-the-hood/distributed-content-generation.md@@ -51,11 +51,11 @@ The schema is a plain `dict` — fully serializable, inspectable on the wire, an### Schema-to-model reconstruction-On the worker, `model_class_from_json_schema()` (`pipelex/cogt/content_generation/schema_to_model_factory.py`) rebuilds a live Pydantic class from the embedded schema:+On the worker, `SchemaToModelFactory.make_from_json_schema(schema, *, class_name)` (`pipelex/cogt/content_generation/schema_to_model_factory.py`) rebuilds a live Pydantic class from the embedded schema:```python-source_code = _generate_source_from_schema(schema) # datamodel-code-generator-reconstructed_class = _exec_and_extract_class(source_code, class_name) # exec()+source_code = cls._generate_source_from_schema(schema) # datamodel-code-generator+reconstructed_class = cls._exec_and_extract_class(source_code, class_name=class_name) # exec()reconstructed_class.__kajson_class_source__ = source_code # attached for downstream use```@@ -63,7 +63,7 @@ The reconstruction pipeline:1. **Generate** — `datamodel-code-generator` converts the JSON schema into Python source code defining a `BaseModel` subclass.2. **Execute** — `exec()` compiles the source in an isolated namespace and extracts the named class.-3. **Cache** — A thread-safe SHA-256 hash cache (with double-check locking) avoids redundant generation for the same schema.+3. **Cache** — A thread-safe SHA-256 hash cache is guarded by a single lock held across the entire check-then-generate sequence, not double-check locking: the lock is acquired once, the cache is checked, and — on a miss — generation and exec happen before the lock is released. This kills same-schema thundering herds (concurrent first-misses each paying a full codegen+exec round) at the cost of serializing cache hits on other schemas behind any in-flight miss, an acceptable trade-off given real workloads use a small bounded set of distinct schemas cached for the process lifetime.4. **Tag** — The generated source code is attached as `__kajson_class_source__` on the class, enabling Kajson to carry it through the transport.!!! warning "Class name normalization"diff --git a/docs/under-the-hood/dry-run-mock-generation.md b/docs/under-the-hood/dry-run-mock-generation.mdindex 06e91ebc3..4134707b8 100644--- a/docs/under-the-hood/dry-run-mock-generation.md+++ b/docs/under-the-hood/dry-run-mock-generation.md@@ -33,7 +33,7 @@ Standard mock generators (like Polyfactory) produce random strings like `"uygNji```mermaidflowchart TD- A[pipelex validate --all] --> B[dry_run_pipes]+ A[pipelex validate --all] --> B[BundleValidator.validate_pipes]B --> C[WorkingMemoryFactory.make_mock_inputs]C --> D[DryRunFactory.make_dry_run_factory]D --> E[Polyfactory with custom providers]@@ -98,6 +98,7 @@ class DryRunFactory:def make_dry_run_factory(cls,object_class: type[BaseModelTypeVar],+ *,snake_case_field_names: set[str] | None = None,pascal_case_field_names: set[str] | None = None,) -> type[ModelFactory[BaseModelTypeVar]]:@@ -163,11 +164,12 @@ class ConceptSpec(StructuredContent):examples=["Text", "Image", "Document", "TextAndImages", "Number", "Page"],)-class PipeSpec(StructuredContent):- # Extract talent should be a valid ExtractTalent value- extract_talent: ExtractTalent | str = Field(- description="Select extraction model talent",- examples=list(ExtractTalent), # Polyfactory picks randomly from these+class PipeComposeSpec(PipeSpec):+ # Target format should be a valid TargetFormat value+ target_format: TargetFormat | str | None = Field(+ default=None,+ description="Target format for the output (template mode)",+ examples=list(TargetFormat), # Polyfactory picks randomly from these)```diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.mdindex d803ba21c..5aa2165d0 100644--- a/docs/under-the-hood/error-model.md+++ b/docs/under-the-hood/error-model.md@@ -179,11 +179,12 @@ Every inference worker's SDK-exception handler collapses to a three-step pipelinexcept (APIError, APIConnectionError, APITimeoutError) as exc:metadata = extract_openai_metadata(exc)classification = classify_inference_error(metadata)- raise render_llm_error(- family=InferenceErrorFamily.LLM_COMPLETION,+ raise render_inference_error(metadata=metadata,classification=classification,+ family=InferenceErrorFamily.LLM,model_desc=self.inference_model.desc,+ model_handle=self.inference_model.name,) from exc```@@ -193,7 +194,7 @@ The three steps live in three modules. Only the per-provider Extract functions s|--------|------|--------------|| `pipelex/cogt/inference/error_classification.py` | Extract | `ProviderErrorMetadata`, `SDKErrorEnvelope`, `UserAction`, `UserActionKind`, the 12 `extract_*_metadata` functions, plus pure discriminators (`is_quota_exhaustion`, `is_content_policy_violation`, `is_network_error`) exposed as `@property` on the metadata || `pipelex/cogt/inference/error_classify.py` | Classify | `classify_inference_error()` — provider-blind mapping from `ProviderErrorMetadata` → `ClassificationResult(category, user_action_kind, is_model_not_found)` |-| `pipelex/cogt/inference/error_render.py` | Render | `render_llm_error()` / `render_img_gen_error()` / `render_extract_error()` / `render_search_error()` — picks the `CogtError` subclass from `InferenceErrorFamily` plus `is_model_not_found` (e.g. `LLMModelNotFoundError` vs `LLMCompletionError`) |+| `pipelex/cogt/inference/error_render.py` | Render | `render_inference_error()` — picks the `CogtError` subclass from `InferenceErrorFamily` plus `is_model_not_found` (e.g. `LLMModelNotFoundError` vs `LLMCompletionError`) |Provider-specific nuance is normalized away in Extract (e.g. Google's `code` becomes `status_code`; AWS Bedrock error codes are mapped to HTTP statuses), so Classify has no provider branching. HTTP status drives classification; status-less errors dispatch on the SDK exception type name. The `tests/unit/pipelex/cogt/inference/test_provider_classification_parity.py` meta-test walks every `ProviderName` against the extract-fn registry so adding a new provider without wiring it fails fast.@@ -226,7 +227,7 @@ On structured-generation paths, `instructor` wraps the real SDK exception in anInference-failure leaf errors (`LLMCompletionError`, `ImgGenGenerationError`, …) are raised deep inside a plugin and do not know which model handle invoked them. Each worker family fills that in at its public-method chokepoint:```python-def fill_model_and_provider(self, model_handle: str | None, backend_name: str | None) -> None:+def fill_model_and_provider(self, model_handle: str | None, *, backend_name: str | None) -> None:"""Fill model_handle / backend_name from the worker, only when still unset."""```@@ -415,7 +416,7 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| `pipelex/cogt/exceptions.py` | `CogtError`, `InferenceErrorCategory` || `pipelex/cogt/inference/error_classification.py` | Extract — `ProviderErrorMetadata`, `SDKErrorEnvelope`, `UserAction`, `UserActionKind`, per-provider `extract_*_metadata` functions, pure discriminators || `pipelex/cogt/inference/error_classify.py` | Classify — `classify_inference_error()`, `ClassificationResult` |-| `pipelex/cogt/inference/error_render.py` | Render — `render_llm_error()` / `render_img_gen_error()` / `render_extract_error()` / `render_search_error()`, `InferenceErrorFamily` |+| `pipelex/cogt/inference/error_render.py` | Render — `render_inference_error()`, `InferenceErrorFamily` || `pipelex/cogt/inference/provider_name.py` | `ProviderName` enum keying the extract-fn registry || `pipelex/plugins/*/` | Per-provider inference workers — Layer 0 → 1 classification || `pipelex/pipeline/exceptions.py` | `PipelineExecutionError`, `PipeExecutionError` |diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.mdindex c7b430a03..a8716cda1 100644--- a/docs/under-the-hood/execution-graph-tracing.md+++ b/docs/under-the-hood/execution-graph-tracing.md@@ -32,11 +32,11 @@ Pipe Execution → GraphTracer → GraphSpec → Renderers → HTML/Mermaid|----------|-----|-----|--------|| Generate execution graph | `pipelex run pipe my_pipe --graph` | `PipelexMTHDSProtocol(execution_config=...).execute(...)` | GraphSpec JSON + HTML viewers || Force include full data | `--graph --graph-full-data` | `data_inclusion.stuff_json_content=True` | Data embedded in IOSpec |-| Force exclude data | `--graph --graph-no-data` | All `data_inclusion.*=False` | Previews only |+| Force exclude data | `--graph --graph-no-data` | The four stuff/error `data_inclusion` flags → `False` (`pipe_and_concept_registry` unaffected) | Previews only || Dry run with graph | `--dry-run --graph` | `PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, execution_config=...)` | Graph of mock execution |!!! info "Full Data Included by Default"- The default configuration includes full data in graphs (`stuff_json_content`, `stuff_text_content`, `stuff_html_content`, and `error_stack_traces` are all `true`). Use `--graph-full-data` or `--graph-no-data` only to override project-specific settings.+ The default configuration includes full data in graphs (`stuff_json_content`, `stuff_text_content`, `stuff_html_content`, `error_stack_traces`, and `pipe_and_concept_registry` are all `true`). Use `--graph-full-data` or `--graph-no-data` only to override project-specific settings — the flags toggle the first four; `pipe_and_concept_registry` is set only via config.---@@ -89,10 +89,10 @@ graph_spec = response.pipe_output.graph_spec| Output | File | Purpose ||--------|------|---------|-| `graphspec_json` | `_graphspec.json` | Canonical graph representation |-| `mermaidflow_mmd` | `_mermaid.mmd` | Mermaid flowchart code |-| `mermaidflow_html` | `_mermaid.html` | Standalone Mermaid viewer |-| `reactflow_html` | `_reactflow.html` | Interactive ReactFlow viewer |+| `graphspec_json` | `graphspec.json` | Canonical graph representation |+| `mermaidflow_mmd` | `mermaidflow.mmd` | Mermaid flowchart code |+| `mermaidflow_html` | `mermaidflow.html` | Standalone Mermaid viewer |+| `reactflow_html` | `reactflow.html` | Interactive ReactFlow viewer |---@@ -195,6 +195,9 @@ class GraphSpec(BaseModel):| `DATA` | Data flow (stuff passed between pipes) || `CONTAINS` | Parent-child containment (controller → children) || `SELECTED_OUTCOME` | Condition outcome selection |+| `BATCH_ITEM` | Batch fan-out: input list → item extracted for each batch iteration |+| `BATCH_AGGREGATE` | Batch fan-in: item outputs → aggregated output list |+| `PARALLEL_COMBINE` | Branch outputs → combined output in PipeParallel |Every `EdgeSpec` also carries an `optional` boolean (default `false`). On a `DATA` edge it marks that the producer's output is declared optional (`?` presence marker) — the value flowed on this run but may be absent on others. Renderers can use it to draw the edge distinctly (e.g. dashed).@@ -245,7 +248,7 @@ node_id, child_context = manager.on_pipe_start(# 4. On completion, report success with outputmanager.on_pipe_end_success(- graph_id=trace_context.graph_id,+ lookup_key=trace_context.lookup_key,node_id=node_id,ended_at=datetime.now(timezone.utc),output_spec=IOSpec(...),@@ -266,7 +269,7 @@ from pipelex.hub import scoped_event_logfrom pipelex.tracing.in_memory_event_log import InMemoryEventLogwith scoped_event_log(InMemoryEventLog()):- response = await runner.execute_pipeline(...) # graph assembles in memory+ response = await runner.execute(...) # graph assembles in memory```Semantics:@@ -288,7 +291,7 @@ class TraceContext(BaseModel):node_sequence: int # Counter for generating node IDsdata_inclusion: DataInclusionConfig # What data to capture- def copy_for_child(self, child_node_id: str, next_sequence: int) -> TraceContext:+ def copy_for_child(self, child_node_id: str, *, next_sequence: int) -> TraceContext:"""Create context for nested pipe execution."""return TraceContext(graph_id=self.graph_id,@@ -415,6 +418,7 @@ stuff_json_content = true # Include full serialized datastuff_text_content = true # Include ASCII text representationstuff_html_content = true # Include HTML representationerror_stack_traces = true # Include full stack traces+pipe_and_concept_registry = true # Include pipe and concept registries in the GraphSpec[pipelex.pipeline_execution_config.graph_config.graphs_inclusion]graphspec_json = true # Generate GraphSpec JSON@@ -427,22 +431,28 @@ reactflow_html = true # Generate ReactFlow HTML| Option | Description ||--------|-------------|-| `direction` | Flowchart direction (TB, LR, BT, RL) |+| `direction` | Flowchart direction (`top_down`, `left_to_right`) || `is_include_data_edges` | Show data flow edges || `is_include_contains_edges` | Show containment edges |+| `is_include_selected_outcome_edges` | Show condition-result (selected outcome) edges || `is_show_stuff_codes` | Show digest in stuff labels |-| `style.theme` | Mermaid theme (default, dark, forest, neutral) |+| `style.theme` | Mermaid theme (default, base, dark, forest, neutral) |### ReactFlowRenderingConfig| Option | Description ||--------|-------------|-| `layout_direction` | Dagre layout direction (TB, LR) |+| `is_use_cdn` | Load the graph viewer assets from a CDN (jsDelivr) instead of inlining them |+| `layout_direction` | Flowchart layout direction (`top_down`, `left_to_right`), converted internally to Dagre's TB/LR || `nodesep` | Node separation in pixels || `ranksep` | Rank separation in pixels || `edge_type` | Edge style (bezier, smoothstep, step, straight) || `initial_zoom` | Initial viewport zoom |+| `pan_to_top` | Pan the viewport to the top on load |+| `default_title` | Title of the generated HTML page |+| `show_batch_controller` | Render controller pipes as grouping containers || `style.theme` | UI theme (light, dark, system) |+| `style.palette` | Node color palette (`yellow_blue`, `dracula`) |---@@ -458,9 +468,10 @@ class IOSpec(BaseModel):preview: str | None # Truncated preview (max 200 chars)size: int | None # Content sizedigest: str | None # Unique identifier for data flow- data: Any | None # Full serialized content+ data: str | dict[str, Any] | list[str] | list[dict[str, Any]] | None # Full serialized contentdata_text: str | None # ASCII text representationdata_html: str | None # HTML representation+ extra: dict[str, Any] # Extra markers, e.g. `optional` set on a declared-optional (`?`) output```!!! warning "Preview Truncation"diff --git a/docs/under-the-hood/image-handling-in-llm-prompts.md b/docs/under-the-hood/image-handling-in-llm-prompts.mdindex 4e27c3a32..20597258d 100644--- a/docs/under-the-hood/image-handling-in-llm-prompts.md+++ b/docs/under-the-hood/image-handling-in-llm-prompts.md@@ -148,9 +148,8 @@ PageContent(The filter produces:```-text_and_images:- text: Welcome to the guide- images: [Image 1]+text_and_images: Welcome to the guide+[Image 1]page_view: [Image 2]```@@ -278,21 +277,22 @@ The `ImageRegistry` manages image numbering during prompt construction.### Key Properties-- **1-indexed** - Numbers start at 1 for readability+- **0-based indices, 1-based display** - `register_image` returns a 0-based index; the 1-based `[Image N]` display number is produced at render time- **Sequential** - Images numbered in order of registration-- **Deduplicated** - Same URL gets same number+- **Deduplicated** - Same URL gets same index```pythonclass ImageRegistry:- def register_image(self, image: ImageContent) -> int:- """Returns image number. Same URL = same number."""- if image.url in self._url_to_number:- return self._url_to_number[image.url]-- number = len(self._images) + 1+ def register_image(self, image: Any) -> int:+ """Register an image and return its 0-based index. Same URL = same index."""+ if image.url in self._image_urls:+ # Find existing index+ for index, existing in enumerate(self._images):+ if existing.url == image.url:+ return indexself._images.append(image)- self._url_to_number[image.url] = number- return number+ self._image_urls.add(image.url)+ return len(self._images) - 1```### Deduplication Example@@ -301,13 +301,13 @@ If the same image appears in multiple places:```python# First registration-registry.register_image(img_a) # Returns 1+registry.register_image(img_a) # Returns 0# Second registration of same URL-registry.register_image(img_a) # Returns 1 (not 2)+registry.register_image(img_a) # Returns 0 (not 1)# Different image-registry.register_image(img_b) # Returns 2+registry.register_image(img_b) # Returns 1```---@@ -332,11 +332,11 @@ The `with_images` filter uses the `ImageRenderable` protocol to handle StuffArte```python# StuffArtefact implements ImageRenderableif isinstance(value, ImageRenderable):- return value.render_with_images(registry, text_format=text_format)+ return value.render_with_images(registry=registry, text_format=text_format)# StuffArtefact.render_with_images() delegates to content-def render_with_images(self, registry, *, text_format) -> str:- return self._stuff.content.render_with_images(registry, text_format=text_format)+def render_with_images(self, *, registry, text_format) -> str:+ return self._stuff.content.render_with_images(registry=registry, text_format=text_format)```!!! note "ImageRenderable Protocol"@@ -354,15 +354,14 @@ The system validates image usage at both factory time and runtime:| Condition | Error ||-----------|-------|-| `\| with_images` on `Image` type | "Cannot use with_images on direct Image" |-| `\| with_images` on type with no nested images | "Type X has no nested image fields" |+| `\| with_images` on a type with no nested images (including a direct `Image`) | `WithImagesFilterError`: "Filter '\| with_images' used on variable 'X' but the type has no nested images. Remove the filter or use a type with nested images." |### Runtime| Condition | Error ||-----------|-------|| `with_images` on undefined value | "Cannot use with_images filter on undefined value" |-| `with_images` on non-ImageRenderable type (e.g., string) | "X does not implement the ImageRenderable protocol" |+| `with_images` on non-ImageRenderable type (e.g., string) | "The with_images filter received a X which does not implement the ImageRenderable protocol. This filter requires StuffArtefact, StuffContent subclasses, or lists containing such types. Did you accidentally apply another filter first that converted to string?" |The runtime check catches cases where filter chaining converts structured data to a string before `with_images` runs (e.g., `{{ pages | tag | with_images }}`).@@ -438,8 +437,8 @@ Pages:| File | Purpose ||------|---------|-| `pipelex/pipe_operators/llm/image_reference.py` | `ImageReference` and `ImageReferenceKind` models |-| `pipelex/pipe_operators/llm/template_image_analyzer.py` | Factory-time template analysis |+| `pipelex/pipe_operators/shared/image_reference.py` | `ImageReference` and `ImageReferenceKind` models |+| `pipelex/pipe_operators/shared/template_image_analyzer.py` | Factory-time template analysis || `pipelex/tools/jinja2/image_registry.py` | Runtime image tracking || `pipelex/tools/jinja2/jinja2_with_images_filter.py` | The `with_images` filter implementation |@@ -448,7 +447,7 @@ Pages:| File | Purpose ||------|---------|| `pipelex/tools/jinja2/jinja2_required_variables.py` | `VariableReference` for filter detection |-| `pipelex/pipe_operators/llm/llm_prompt_spec.py` | Prompt building with image collection |+| `pipelex/pipe_operators/llm/llm_prompt_blueprint.py` | Prompt building with image collection |---diff --git a/docs/under-the-hood/init-cli-flows.md b/docs/under-the-hood/init-cli-flows.mdindex 9f26f381a..0df8ebb54 100644--- a/docs/under-the-hood/init-cli-flows.md+++ b/docs/under-the-hood/init-cli-flows.md@@ -13,7 +13,7 @@ description: "Inside the pipelex init command — how focus-based dispatch setsThe `.pipelex/` directory contains two categories of files with different lifecycles:-1. **Config files** (`pipelex.toml`, `plxt.toml`, `mthds_schema.json`) — static templates, copied once, rarely touched by the user.+1. **Config files** (`pipelex.toml`, `plxt.toml`) — static templates copied verbatim from the kit (no interactive customization); re-running init overwrites them (full reset).2. **Inference files** (`inference/backends.toml`, `inference/routing_profiles.toml`, `inference/backends/*.toml`, `inference/deck/*.toml`) — interactive setup, customized per-project based on which AI backends the user selects.These two categories are managed by separate steps. `init_config()` copies only config files (skipping the `inference/` directory entirely). The inference step handles its own template copying and then runs interactive backend selection and routing customization. Each file is owned by exactly one step — `init_config()` explicitly skips the `inference/` directory via `INIT_SKIP_DIRS`, and skips `telemetry.toml` and `pipelex_service.toml` via `INIT_SKIP_FILES`. This separation ensures that re-running `pipelex init config` never overwrites a user's carefully tuned inference setup.@@ -48,7 +48,6 @@ All commands except `agreement` and `credentials` perform a **full reset** (over|----------|-------------|------------|------|| `pipelex.toml` | `init_config()` | Project or global | `.pipelex/pipelex.toml` || `plxt.toml` | `init_config()` | Project or global | `.pipelex/plxt.toml` |-| `mthds_schema.json` | `init_config()` | Project or global | `.pipelex/mthds_schema.json` || `backends.toml` | Inference step | Project or global | `.pipelex/inference/backends.toml` || `backends/*.toml` | Inference step | Project or global | `.pipelex/inference/backends/` || `deck/*.toml` | Inference step | Project or global | `.pipelex/inference/deck/` |@@ -259,7 +258,6 @@ Copies a telemetry template and prints instructions. No interactive prompts. Whi|-----------------|-----------|---------|| `pipelex.toml` | `init_config()` | Main Pipelex configuration || `plxt.toml` | `init_config()` | PLXT tooling configuration |-| `mthds_schema.json` | `init_config()` | MTHDS JSON Schema for IDE support || `inference/` | Inference step | All inference configuration (see below) || `inference/backends.toml` | Inference step | Backend registry (enabled/disabled flags) || `inference/backends/*.toml` | Inference step | Per-backend settings (API keys, endpoints) |@@ -308,7 +306,9 @@ The `ConfigLoader` (singleton: `config_manager`) resolves where configuration fi`find_project_root()` walks up from `cwd` looking for marker files:-- `.git`, `pyproject.toml`, `setup.py`, `setup.cfg`, `package.json`, `.hg`+- `.pipelex`, `.git`, `pyproject.toml`, `setup.py`, `setup.cfg`, `package.json`, `.hg`++A directory containing only a `.pipelex/` directory is therefore itself a valid project root.The home directory (`~`) is explicitly excluded — even if it contains a `package.json`, it is never treated as a project root.@@ -402,8 +402,7 @@ If no project root is found and `--global` is not set, the command fails with an|-------|------|---------|| `backends` | `list[str]` | Backend keys to enable (e.g. `"openai"`, `"anthropic"`, `"pipelex_gateway"`) || `primary_backend` | `str` | Required when 2+ backends selected and `pipelex_gateway` is not among them |-| `accept_gateway_terms` | `bool` | Sets gateway terms acceptance (defaults to `false` if omitted) |-| `telemetry_mode` | `str` | `"off"` (default), `"anonymous"`, or `"identified"` |+| `accept_gateway_terms` | `bool` | Sets gateway terms acceptance; when omitted, nothing is written and any existing acceptance state is left untouched |### Flow@@ -413,13 +412,15 @@ flowchart TDPARSE --> RESOLVE["Resolve target dir<br/>(project or --global)"]RESOLVE --> STEP1["Step 1: init_config()<br/>Copy config templates (skips inference/)"]STEP1 --> STEP15["Step 1.5: Copy inference templates<br/>(backends.toml, backends/*, deck/*, routing_profiles.toml)"]- STEP15 --> STEP2["Step 2: Configure backends<br/>Enable requested backends in backends.toml"]+ STEP15 --> STEP16["Step 1.6: Copy telemetry template<br/>(global: active defaults; project: commented-out)"]+ STEP16 --> STEP2["Step 2: Configure backends<br/>Enable requested backends in backends.toml"]STEP2 --> GW{pipelex_gateway<br/>in backends?}- GW -- Yes --> TERMS["Write gateway terms to ~/.pipelex/<br/>(always global)"]+ GW -- Yes --> TERMS["Write gateway terms to ~/.pipelex/<br/>(always global; skipped if accept_gateway_terms omitted)"]GW -- No --> STEP3TERMS --> STEP3["Step 3: Configure routing<br/>Auto-derive routing profile"]- STEP3 --> STEP4["Step 4: Configure telemetry<br/>Copy template + apply mode"]- STEP4 --> OUTPUT["Output structured JSON result"]+ STEP3 --> STEP4["Step 4: Mark inference setup completed<br/>(written to ~/.pipelex/)"]+ STEP4 --> STEP5["Step 5: Prime remote-config cache<br/>(no-op if gateway disabled or terms not accepted)"]+ STEP5 --> OUTPUT["Output result<br/>(Markdown, or JSON via --format json)"]```!!! note "No credentials step"@@ -431,7 +432,7 @@ flowchart TD|--------|---------------|---------------------|| Prompts | Interactive (Rich prompts) | None — all input from `--config` JSON || Gateway terms | Prompted interactively | From `accept_gateway_terms` field in config |-| Output | Rich console output | Structured JSON via `agent_success()` / `agent_error()` |+| Output | Rich console output | Markdown by default (`agent_success_formatted()` / `agent_error()`), structured JSON via `--format json` || Credentials | Prompted interactively | Not configured — use `pipelex-agent doctor` or `pipelex init credentials` || Focus dispatch | Supports individual focus (`config`, `inference`, etc.) | Runs config, inference, routing, telemetry (no credentials) || Target dir default | Global (`~/.pipelex/`), use `--local` for project | Project (`{project_root}/.pipelex/`), use `--global` for global |diff --git a/docs/under-the-hood/orchestrator-plugins.md b/docs/under-the-hood/orchestrator-plugins.mdindex ff65234f4..738f3b3b7 100644--- a/docs/under-the-hood/orchestrator-plugins.md+++ b/docs/under-the-hood/orchestrator-plugins.md@@ -8,9 +8,9 @@ description: "How Pipelex dispatches a pipe run by orchestration mode through thA pipe a host runtime invokes through the runtime bridge runs along **two orthogonal axes**:- **`orchestration_mode`** — *which* orchestrator runs the pipe. An **open string token**, not a closed enum: core owns only `"direct"` (in-process); every other token is contributed by the plugin that owns its orchestrator — [`"temporal"`](https://pipelex.com/products#temporal) (durable, on a Temporal worker fleet) by `pipelex-temporal`, [`"mistral-workflows"`](https://pipelex.com/products#mistral-workflows) (decomposed into Mistral Workflows primitives) by `pipelex-mistralai-workflows`. Neither is built into the open-source `pipelex` core: both ship as external, closed-source host-runtime backends, distributed privately rather than on PyPI as part of Pipelex's [workflow-orchestration offer](https://pipelex.com/products#durable-execution).-- **`delivery`** — *whether the caller waits*. A **closed** core `DeliveryMode` enum (`BLOCKING` / `FIRE_AND_FORGET`) on the wire input, set by the endpoint, never received from a caller. On the SPI it is expressed as *which method* the endpoint calls — `run` (blocking) or `start` (fire-and-forget) — so each return type is truthful on its own; `supports_fire_and_forget` advertises whether an orchestrator can do genuine async.+- **`delivery`** — *whether the caller waits*. A **closed** core `DeliveryMode` enum (`BLOCKING` / `FIRE_AND_FORGET`) on the wire input, set by the endpoint, never received from a caller. On the SPI it is expressed as *which method* the endpoint calls — `execute` (blocking) or `start` (fire-and-forget) — so each return type is truthful on its own; `supports_fire_and_forget` advertises whether an orchestrator can do genuine async.-An **orchestrator** is what knows how to run a pipe under one token. Core names no orchestrator by import or by string. The bridge resolves the orchestrator for the requested token from a registry (keyed by the token `str`) and calls its `run` — `"direct"` is contributed by a core plugin, `"temporal"` by the Temporal plugin, `"mistral-workflows"` by the external `pipelex-mistralai-workflows` plugin. A lookup miss raises a generic `MissingOrchestratorError` that names no orchestrator. This page documents that seam, the **Orchestrator SPI** a host-runtime plugin compiles against, and how the Temporal plugin is wired.+An **orchestrator** is what knows how to run a pipe under one token. Core names no orchestrator by import or by string. The bridge resolves the orchestrator for the requested token from a registry (keyed by the token `str`) and calls its `execute` — `"direct"` is contributed by a core plugin, `"temporal"` by the Temporal plugin, `"mistral-workflows"` by the external `pipelex-mistralai-workflows` plugin. A lookup miss raises a generic `MissingOrchestratorError` that names no orchestrator. This page documents that seam, the **Orchestrator SPI** a host-runtime plugin compiles against, and how the Temporal plugin is wired.---@@ -21,7 +21,7 @@ An **orchestrator** is what knows how to run a pipe under one token. Core names→ build the PipeJob (boundary decode + library scope + trace_context)→ orchestrator = get_orchestrator_registry().get_optional(mode=orchestration_mode)→ if orchestrator is None: raise MissingOrchestratorError(mode) # generic, names no orchestrator- → return await orchestrator.run(pipe_job=..., delivery_assignment=...) # or orchestrator.start(...) for FIRE_AND_FORGET+ → return await orchestrator.execute(pipe_job=..., delivery_assignment=...) # or orchestrator.start(...) for FIRE_AND_FORGET```The registry is built once at boot from whatever the discovered plugins contributed (`build_registrar` → `OrchestratorRegistry` on the hub). There is no `match orchestration_mode:` anywhere in the bridge — the token set is open, so validation is the registry lookup itself; adding a mode's behavior means registering an orchestrator for its token, nothing in core changes.@@ -39,12 +39,12 @@ An orchestrator satisfies `OrchestratorProtocol` (`pipelex/plugins/orchestrator_class OrchestratorProtocol(Protocol):supports_fire_and_forget: bool- async def run(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput: ...+ async def execute(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput: ...async def start(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeDispatchAck: ...```-The delivery axis is split into two methods so each return type is truthful on its own. `run` is the BLOCKING arm: it awaits completion and returns the completed-run `PipelexPipeRunOutput` — which always carries a main stuff (`main_stuff_name` is required, no "not finished yet" escape hatch). `start` is the FIRE_AND_FORGET arm: it genuinely enqueues the job and returns a `PipelexPipeDispatchAck` (`pipeline_run_id` + `workflow_id`, nothing more — nothing has run yet). `supports_fire_and_forget` is the capability a runner reads *before* dispatch — `/start` rejects honestly (4xx) when the resolved mode cannot do genuine async, instead of silently running blocking and acking; an orchestrator that cannot (like core's DIRECT) implements `start` by raising, unreachable behind that gate.+The delivery axis is split into two methods so each return type is truthful on its own. `execute` is the BLOCKING arm: it awaits completion and returns the completed-run `PipelexPipeRunOutput` — which always carries a main stuff (`main_stuff_name` is required, no "not finished yet" escape hatch). `start` is the FIRE_AND_FORGET arm: it genuinely enqueues the job and returns a `PipelexPipeDispatchAck` (`pipeline_run_id` + `workflow_id`, nothing more — nothing has run yet). `supports_fire_and_forget` is the capability a runner reads *before* dispatch — `/start` rejects honestly (4xx) when the resolved mode cannot do genuine async, instead of silently running blocking and acking; an orchestrator that cannot (like core's DIRECT) implements `start` by raising, unreachable behind that gate.A plugin contributes one per token it serves by calling the registrar menu in its `register`, passing the token as a raw string (no enum, no cast):@@ -52,7 +52,7 @@ A plugin contributes one per token it serves by calling the registrar menu in itregistrar.add_orchestrator(mode="temporal", orchestrator=TemporalOrchestrator())```-Constructing the orchestrator instance must be **import-light** — it must not import the host-runtime SDK (`temporalio`, …) at module scope or in its `__init__`. The heavy import happens lazily inside `run` (and a friendly `MissingOrchestratorError` is raised there if the mode's extra is absent), so discovering and registering the plugin never pulls the SDK. This is what keeps boot import-light even on a process that will never use the mode.+Constructing the orchestrator instance must be **import-light** — it must not import the host-runtime SDK (`temporalio`, …) at module scope or in its `__init__`. The heavy import happens lazily inside `execute` (and a friendly `MissingOrchestratorError` is raised there if the mode's extra is absent), so discovering and registering the plugin never pulls the SDK. This is what keeps boot import-light even on a process that will never use the mode.### A missing orchestrator is a generic, plugin-decoupled error@@ -62,7 +62,7 @@ A token with no registered orchestrator (its plugin is not installed) raises `Mi## Bundle validators: the `/validate` counterpart-`/validate` rides the same open-token seam as `run`. A **bundle validator** produces a validation verdict for one orchestration mode, the way an orchestrator runs a pipe for one mode: the core `direct` plugin contributes the in-process validator, and an external orchestrator plugin contributes a worker-dispatched validator under its own token — which core never names. A host runtime resolves the requested mode and dispatches through the `BundleValidatorRegistry` on the hub instead of branching on a backend.+`/validate` rides the same open-token seam as `execute`. A **bundle validator** produces a validation verdict for one orchestration mode, the way an orchestrator runs a pipe for one mode: the core `direct` plugin contributes the in-process validator, and an external orchestrator plugin contributes a worker-dispatched validator under its own token — which core never names. A host runtime resolves the requested mode and dispatches through the `BundleValidatorRegistry` on the hub instead of branching on a backend.A validator satisfies `BundleValidatorProtocol` (`pipelex/plugins/bundle_validator_registry.py`):@@ -80,9 +80,9 @@ class BundleValidatorProtocol(Protocol):The `library_dirs` annotation is a quoted forward reference because `Sequence` and `Path` are imported under `TYPE_CHECKING` in `bundle_validator_registry.py`; a plugin that imports them at module scope can use the unquoted form.-Two contract points distinguish it from `OrchestratorProtocol.run`:+Two contract points distinguish it from `OrchestratorProtocol.execute`:-- **No fire-and-forget arm.** Validation is inherently blocking, so — unlike orchestration's `run`/`start` split — there is no `start` counterpart and no `supports_fire_and_forget` capability flag.+- **No fire-and-forget arm.** Validation is inherently blocking, so — unlike orchestration's `execute`/`start` split — there is no `start` counterpart and no `supports_fire_and_forget` capability flag.- **Verdict-as-value, not raise.** `validate_bundles` *returns* the verdict — `BundleValidationVerdict` is the union of the valid arm (a `ValidationReport`) and the invalid arm (an `ErrorReport` carrying `validation_errors`) — and raises only for a no-verdict infra fault, which a host runtime maps to a 5xx. This is the same valid/invalid pair the API maps onto its 200-always `/validate` wire, so the verdict contract is backend-independent.The seam is deliberately typed at the MTHDS-protocol level (`ValidationReport` from `mthds.protocol`), not the concrete `PipelexValidationReport` envelope: the concrete report's module reaches the hub, so naming it from this hub-reachable seam would close an import cycle, and the seam is generic across orchestrators (language-standard altitude), so it speaks the protocol report — the Pipelex-runtime envelope is the concrete `ValidationReport` subtype the validators actually produce, and the API recovers that precise type at its edge. `library_dirs` is host context the in-process arm needs to load the method library; a worker-dispatched arm ignores it — its worker loads its own library.@@ -155,7 +155,7 @@ At each ordered hub slot, `Pipelex.setup` resolves in this precedence:2. a plugin slot-claim thunk;3. the core default.-A slot claim must never silently override an explicit injection. Two slots take no explicit `setup()` parameter and so skip step 1: the task manager (no core default either — unclaimed means no task manager wiring at all) and the isolated-execution probe (whose core default is the always-False predicate above). Teardown runs the plugin-registered teardown callbacks **LIFO**, before core teardown, so a worker's in-flight runtime resources release first.+A slot claim must never silently override an explicit injection. Only the content generator and pipe router take an explicit `setup()` parameter; the other slots skip step 1 — the task manager (no core default either — unclaimed means no task manager wiring at all), the isolated-execution probe (whose core default is the always-False predicate above), and the pipe run (plugin thunk or core default, nothing else). Teardown runs the plugin-registered teardown callbacks **LIFO**, before core teardown, so a worker's in-flight runtime resources release first.---@@ -198,10 +198,10 @@ What an out-of-tree orchestrator imports *is* a contract. The SPI is a documente`pipelex_temporal/temporal_plugin.py` (in the external `pipelex-temporal` distribution) is the reference orchestrator plugin. Its `register`:-- **always** (regardless of the boot gate): contributes a single `TemporalOrchestrator` registered once under the `"temporal"` token (import-light; `temporalio` is pulled lazily inside `run`/`start`), the matching `TemporalBundleValidator` under the same token (the worker-dispatched `/validate` arm), and an HTTP error mapper classifying Temporal transport faults. The orchestrator advertises `supports_fire_and_forget = True`; its `run` awaits completion and reports `make_workflow_id(...)`, its `start` enqueues the workflow and returns a `PipelexPipeDispatchAck` carrying the workflow id;+- **always** (regardless of the boot gate): contributes a single `TemporalOrchestrator` registered once under the `"temporal"` token (import-light; `temporalio` is pulled lazily inside `execute`/`start`), the matching `TemporalBundleValidator` under the same token (the worker-dispatched `/validate` arm), and an HTTP error mapper classifying Temporal transport faults. The orchestrator advertises `supports_fire_and_forget = True`; its `execute` awaits completion and reports `make_workflow_id(...)`, its `start` enqueues the workflow and returns a `PipelexPipeDispatchAck` carrying the workflow id;- **only when `plugins.boot_orchestrator == "temporal"`**: claims the content-generator / task-manager / pipe-router / pipe-run / isolated-execution-probe hub slots with thunks and registers the teardown callback — booting this process as a Temporal-default runtime.-The orchestrator itself (`pipelex_temporal/temporal_orchestrators.py`) keeps the `WorkflowExecutionError` catch and the `make_workflow_id` recompute in the blocking `run` arm. It serializes its `PipeOutput` through `pipelex.runtime_bridge.serialization`, shared with the core DIRECT orchestrator so the boundary shape cannot drift.+The orchestrator itself (`pipelex_temporal/temporal_orchestrators.py`) keeps the `WorkflowExecutionError` catch and the `make_workflow_id` recompute in the blocking `execute` arm. It serializes its `PipeOutput` through `pipelex.runtime_bridge.serialization`, shared with the core DIRECT orchestrator so the boundary shape cannot drift.The Temporal plugin is **external** — it ships as the `pipelex-temporal` distribution and is discovered through a `pipelex.plugins` entry point in that dist's `pyproject.toml`, not through `BUILTIN_PLUGINS`. Core's `BUILTIN_PLUGINS` (`pipelex/plugins/builtins.py`) holds only the always-shipped inference and `direct` plugins and explicitly excludes Temporal; installing `pipelex-temporal` is all it takes to make the `"temporal"` orchestrator available — zero config, no core import of `temporalio`. Its operational `worker` / `setup-namespace` commands ship as the standalone `pipelex-temporal` console script, so they travel with that dist.diff --git a/docs/under-the-hood/pipe-routing-and-execution.md b/docs/under-the-hood/pipe-routing-and-execution.mdindex 4e8ed0609..1dd04ac75 100644--- a/docs/under-the-hood/pipe-routing-and-execution.md+++ b/docs/under-the-hood/pipe-routing-and-execution.md@@ -77,7 +77,7 @@ When a `.mthds` file declares a concept like `RawText = "Raw input text..."`, thThese dynamically-generated classes become the `content` type of `Stuff` objects in `WorkingMemory`.!!! info "Why Dynamic Classes Matter"- When a PipeJob is serialized (e.g., for Temporal transport), Kajson embeds `__class__` and `__module__` metadata. The receiving process must have these classes registered in its class registry to deserialize the payload.+ When `WorkingMemory` crosses to a Temporal worker, `dump_for_transport()` stamps content with pipelex-private `__pipelex_class__` / `__pipelex_module__` markers instead of Kajson's `__class__` / `__module__` — deliberately keeping Kajson's universal decoder out of the loop at the data-converter boundary. Pipelex's hydrator resolves those markers against the per-workflow `ClassRegistry`, the only place these dynamic concept classes are registered.---@@ -125,12 +125,13 @@ effective_pipe_router = self._resolve_hub_slot(`PipeRouter` implements `PipeRouterProtocol` with a minimal `_run_pipe_job()`:```python-async def _run_pipe_job(self, pipe_job, wfid=None):+async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput:return await pipe_job.pipe.run_pipe(job_metadata=pipe_job.job_metadata,- working_memory=pipe_job.working_memory,+ working_memory=pipe_job.get_working_memory(),output_name=pipe_job.output_name,pipe_run_params=pipe_job.pipe_run_params,+ library_crate=pipe_job.library_crate,)```diff --git a/docs/under-the-hood/reasoning-controls.md b/docs/under-the-hood/reasoning-controls.mdindex fc867a64b..7d1023827 100644--- a/docs/under-the-hood/reasoning-controls.md+++ b/docs/under-the-hood/reasoning-controls.md@@ -244,7 +244,7 @@ Google models use either `thinking_mode = "manual"` (Gemini 2.5 series) or `thin```toml[cogt.llm_config.google_config.effort_to_level_map]none = "disabled"-minimal = "low"+minimal = "minimal"low = "low"medium = "medium"high = "high"@@ -260,7 +260,7 @@ If the level map returns `"disabled"` (e.g., for `NONE` effort), thinking is dis**ADAPTIVE mode** (Gemini 3) sends a `thinking_level` value (e.g., `ThinkingLevel.LOW`, `ThinkingLevel.MEDIUM`, `ThinkingLevel.HIGH`) mapped from the `effort_to_level_map`. The Google SDK dynamically adjusts reasoning depth based on this level. No `thinking_budget` is set in adaptive mode.!!! note- `MINIMAL` and `LOW` both map to `"low"` in the level map. In ADAPTIVE mode they both produce `ThinkingLevel.LOW`. In MANUAL mode they are differentiated by the budget map (512 vs 1024 tokens).+ `MINIMAL` and `LOW` map to distinct levels in the level map (`"minimal"` and `"low"`). In ADAPTIVE mode they produce different results — `ThinkingLevel.MINIMAL` vs `ThinkingLevel.LOW`. In MANUAL mode they are further differentiated by the budget map (512 vs 1024 tokens).**MANUAL mode** (Gemini 2.5) resolves effort to a `thinking_budget` (token count) via the `effort_to_budget_maps` config:diff --git a/docs/under-the-hood/stuffartefact-and-image-rendering.md b/docs/under-the-hood/stuffartefact-and-image-rendering.mdindex 5aafea607..7230fae86 100644--- a/docs/under-the-hood/stuffartefact-and-image-rendering.md+++ b/docs/under-the-hood/stuffartefact-and-image-rendering.md@@ -48,7 +48,7 @@ StuffArtefact wraps Stuff → delegates to StuffContent → Protocol enables tra| `ImageContent` | Self-registers, returns `[Image N]` || `ListContent` | Iterates items, recurses into nested || `TextAndImagesContent` | Renders text first, then images |-| `StuffContent` (base) | Iterates Pydantic model fields recursively |+| `StructuredContent` | Iterates model fields, recurses into nested lists/tuples/dicts || `StuffArtefact` | Delegates to underlying content |---@@ -64,7 +64,7 @@ flowchart TBsubgraph FILTER["with_images Filter"]direction TBCHECK["isinstance(value, ImageRenderable)?"]- CALL["value.render_with_images(registry, text_format=format)"]+ CALL["value.render_with_images(registry=registry, text_format=text_format)"]CHECK --> CALLend@@ -77,7 +77,7 @@ flowchart TBIMG["ImageContent: register + return token"]LIST["ListContent: iterate items"]TEXT["TextAndImagesContent: text then images"]- BASE["StuffContent: iterate model_fields"]+ STRUCT["StructuredContent: iterate model_fields"]endsubgraph REGISTRY["ImageRegistry"]@@ -88,7 +88,7 @@ flowchart TBVAR --> CHECKCALL --> DELEGATE- DELEGATE --> IMG & LIST & TEXT & BASE+ DELEGATE --> IMG & LIST & TEXT & STRUCTIMG --> STORESTORE --> DEDUP```@@ -155,8 +155,8 @@ class ImageRenderable(Protocol):def render_with_images(self,- registry: ImageRegistry,*,+ registry: ImageRegistry,text_format: TextFormat,) -> str:"""Render to string, registering images to the registry.@@ -182,7 +182,7 @@ class ImageRenderable(Protocol):Self-registers and returns a token:```python-def render_with_images(self, registry, *, text_format) -> str:+def render_with_images(self, *, registry, text_format) -> str:image_index = registry.register_image(self)return f"[Image {image_index + 1}]"```@@ -192,13 +192,13 @@ def render_with_images(self, registry, *, text_format) -> str:Iterates items, delegating to nested ImageRenderable objects:```python-def render_with_images(self, registry, *, text_format) -> str:+def render_with_images(self, *, registry, text_format) -> str:parts: list[str] = []for item in self.items:if isinstance(item, ImageRenderable):- rendered = item.render_with_images(registry, text_format=text_format)+ rendered = item.render_with_images(registry=registry, text_format=text_format)else:- rendered = str(item)+ rendered = item.rendered_markdown()if rendered:parts.append(rendered)return "\n".join(parts)@@ -209,10 +209,10 @@ def render_with_images(self, registry, *, text_format) -> str:Renders text first, then registers each image:```python-def render_with_images(self, registry, *, text_format) -> str:+def render_with_images(self, *, registry, text_format) -> str:parts: list[str] = []if self.text:- parts.append(self.text.rendered_plain())+ parts.append(self.text.rendered_for_prompt(text_format=text_format))if self.images:for image in self.images:image_index = registry.register_image(image)@@ -220,26 +220,25 @@ def render_with_images(self, registry, *, text_format) -> str:return "\n".join(parts)```-### StuffContent (Base)+### StructuredContent-Default implementation iterates Pydantic model fields:+Default implementation for structured models iterates Pydantic model fields:```python-def render_with_images(self, registry, *, text_format) -> str:+def render_with_images(self, *, registry, text_format) -> str:parts: list[str] = []for field_name in type(self).model_fields:field_value = getattr(self, field_name)if field_value is None:continue- if isinstance(field_value, ImageRenderable):- rendered = field_value.render_with_images(registry, text_format=text_format)- else:- rendered = str(field_value)+ rendered = self._render_value_with_images(field_value, registry=registry, text_format=text_format)if rendered:parts.append(f"{field_name}: {rendered}")return "\n".join(parts)```+The `_render_value_with_images` helper recurses: a nested `ImageRenderable` gets `render_with_images(...)`, plain lists/tuples/dicts are traversed item by item, a non-renderable `StuffContent` falls back to `rendered_for_prompt(text_format=text_format)`, and anything else to `str(value)`. A plain `StuffContent` subclass that is not a `StructuredContent` does not implement the protocol.+---## Image Numbering@@ -284,15 +283,15 @@ def with_images(context: Context, value: Any, _: Any = None) -> str:registry = ImageRegistry()# 3. Get text format- text_format = TextFormat(context.get(Jinja2ContextKey.TEXT_FORMAT))+ text_format = TextFormat(context.get(Jinja2ContextKey.TEXT_FORMAT, default=TextFormat.PLAIN))# 4. Protocol-based renderingif isinstance(value, ImageRenderable):- return value.render_with_images(registry, text_format=text_format)+ return value.render_with_images(registry=registry, text_format=text_format)# 5. Handle plain sequencesif isinstance(value, (list, tuple)):- return _render_sequence_with_images(value, registry, text_format)+ return _render_sequence_with_images(value, registry=registry, text_format=text_format)# 6. Reject unsupported typesraise Jinja2ContextError(f"{type(value).__name__} does not implement ImageRenderable")@@ -332,7 +331,7 @@ def with_images(context: Context, value: Any, _: Any = None) -> str:| `pipelex/tools/jinja2/jinja2_with_images_filter.py` | `with_images` filter || `pipelex/tools/jinja2/jinja2_filters.py` | `tag` and `format` filters || `pipelex/tools/jinja2/image_registry.py` | Image tracking with deduplication |-| `pipelex/core/stuffs/stuff_content.py` | Base `render_with_images()` |+| `pipelex/core/stuffs/structured_content.py` | Field-iterating `render_with_images()` for structured models || `pipelex/core/stuffs/image_content.py` | Self-registering image token || `pipelex/core/stuffs/list_content.py` | List iteration for images || `pipelex/core/stuffs/text_and_images_content.py` | Text + images rendering |diff --git a/docs/under-the-hood/test-profile-configuration.md b/docs/under-the-hood/test-profile-configuration.mdindex 766be1bbb..685f793a1 100644--- a/docs/under-the-hood/test-profile-configuration.md+++ b/docs/under-the-hood/test-profile-configuration.md@@ -181,7 +181,7 @@ extract_models = ["@from_pdf"]The preprocessing command discovers available models from:-- **Backend TOML files** in `pipelex/backends/*.toml`+- **Backend TOML files** in `inference/backends/*.toml`, resolved from the project `.pipelex/` directory if present there, otherwise from the global `~/.pipelex/` directory- **Pipelex Gateway** remote configuration### 2. Profile Resolution@@ -208,14 +208,17 @@ tests/integration/pipelex/fixtures/_generated_model_sets.pyThis file contains:```python-LLM_MODEL_BACKEND_PAIRS: list[tuple[str, str]] = [- ("claude-4.5-haiku", "pipelex_gateway"),- ("gpt-4o-mini", "pipelex_gateway"),- ...+from tests.integration.pipelex.fixtures.model_combo import ModelCombo++LLM_COMBOS: list[ModelCombo] = [+ ModelCombo('claude-4.5-haiku', 'pipelex_gateway'),+ ModelCombo('gemini-2.5-flash-lite', 'pipelex_gateway'),+ ModelCombo('gpt-4o-mini', 'pipelex_gateway'),]-IMG_GEN_MODEL_BACKEND_PAIRS: list[tuple[str, str]] = [...]-EXTRACT_MODEL_BACKEND_PAIRS: list[tuple[str, str]] = [...]+IMG_GEN_COMBOS: list[ModelCombo] = [...]+EXTRACT_COMBOS: list[ModelCombo] = [...]+SEARCH_COMBOS: list[ModelCombo] = [...]```---