Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

docs: fix code-doc drift across under-the-hood + advanced pages (Drift Hunt S2-3) (#1042)

Louis Choquel9814e2f4c12b
diff --git a/docs/advanced/index.md b/docs/advanced/index.md
index 225473de9..b5a12f221 100644
--- a/docs/advanced/index.md
+++ b/docs/advanced/index.md
@@ -15,7 +15,7 @@ There are two main ways to inject custom implementations:
### 1. During Initialization
```python
-from pipelex import Pipelex
+from pipelex.pipelex import Pipelex
pipelex = Pipelex.make(
reporting_delegate=MyReportingDelegate(),
@@ -31,7 +31,7 @@ pipelex = Pipelex.make(
from pipelex.hub import PipelexHub
hub = PipelexHub()
-hub.set_reporting_delegate(MyReportingDelegate())
+hub.set_report_delegate(MyReportingDelegate())
# ... and so on for other components
```
diff --git a/docs/advanced/observer-provider-injection.md b/docs/advanced/observer-provider-injection.md
index f0b8c8597..0d9999153 100644
--- a/docs/advanced/observer-provider-injection.md
+++ b/docs/advanced/observer-provider-injection.md
@@ -21,20 +21,20 @@ This data collection is designed for later analysis and is still a work in progr
All observers must implement the `ObserverProtocol`:
```python
-from typing import Any, Dict, Protocol
+from typing import Any, Protocol
-PayloadType = Dict[str, Any]
+PayloadType = dict[str, Any]
class ObserverProtocol(Protocol):
- async def before_run(self, payload: PayloadType) -> None:
+ async def observe_before_run(self, payload: PayloadType) -> None:
"""Process and store the payload before the run"""
...
- async def successful_run(self, payload: PayloadType) -> None:
+ async def observe_after_successful_run(self, payload: PayloadType) -> None:
"""Process and store the payload after the run is successful"""
...
- async def failing_run(self, payload: PayloadType) -> None:
+ async def observe_after_failing_run(self, payload: PayloadType) -> None:
"""Process and store the payload after the run fails"""
...
```
@@ -46,6 +46,7 @@ The payload contains different information depending on the execution phase:
### Before Run
```python
payload = {
+ "pipeline_run_id": str, # Identifier of the pipeline run
"pipe_job": PipeJob, # Contains pipe metadata and working memory
}
```
@@ -53,6 +54,7 @@ payload = {
### Successful Run
```python
payload = {
+ "pipeline_run_id": str, # Identifier of the pipeline run
"pipe_job": PipeJob, # Initial job information
"pipe_output": PipeOutput, # Results and final working memory
}
@@ -61,8 +63,9 @@ payload = {
### Failing Run
```python
payload = {
+ "pipeline_run_id": str, # Identifier of the pipeline run
"pipe_job": PipeJob, # Initial job information
- # Note: error details are handled separately by the pipe router
+ "error": Exception, # The exception that caused the failure
}
```
@@ -80,24 +83,24 @@ class MyCustomObserver(ObserverProtocol):
self.config_param = config_param
# Initialize your storage/logging mechanism here
- async def before_run(self, payload: PayloadType) -> None:
+ async def observe_before_run(self, payload: PayloadType) -> None:
pipe_job = payload["pipe_job"]
# Process pipe_job data before execution
# Example: Log pipe code, inputs, parameters
- print(f"Starting pipe: {pipe_job.pipe_code}")
+ print(f"Starting pipe: {pipe_job.pipe.code}")
- async def successful_run(self, payload: PayloadType) -> None:
+ async def observe_after_successful_run(self, payload: PayloadType) -> None:
pipe_job = payload["pipe_job"]
pipe_output = payload["pipe_output"]
# Process successful execution data
# Example: Log execution time, output size, tokens used
- print(f"Completed pipe: {pipe_job.pipe_code}")
+ print(f"Completed pipe: {pipe_job.pipe.code}")
- async def failing_run(self, payload: PayloadType) -> None:
+ async def observe_after_failing_run(self, payload: PayloadType) -> None:
pipe_job = payload["pipe_job"]
# Process failure data
# Example: Log error context, partial results
- print(f"Failed pipe: {pipe_job.pipe_code}")
+ print(f"Failed pipe: {pipe_job.pipe.code}")
```
### Step 2: Register Your Observer with Dependency Injection
@@ -119,9 +122,9 @@ hub.set_observer(my_observer)
The observer is automatically called by the PipeRouterProtocol during pipe execution:
-1. `before_run()` is called before any pipe execution
-2. `successful_run()` is called after successful completion
-3. `failing_run()` is called when an exception occurs
+1. `observe_before_run()` is called before any pipe execution
+2. `observe_after_successful_run()` is called after successful completion
+3. `observe_after_failing_run()` is called when an exception occurs
## Built-in LocalObserver Example
@@ -143,8 +146,8 @@ get_pipelex_hub().set_observer(local_observer)
The LocalObserver creates separate JSONL files for each event type:
- `before_run.jsonl` - Pre-execution data
-- `successful_run.jsonl` - Success events
-- `failing_run.jsonl` - Failure events
+- `after_successful_run.jsonl` - Success events
+- `after_failing_run.jsonl` - Failure events
## Advanced Use Cases
@@ -159,15 +162,15 @@ class DatabaseObserver(ObserverProtocol):
self.db_connection = db_connection_string
# Initialize database connection
- async def before_run(self, payload: PayloadType) -> None:
+ async def observe_before_run(self, payload: PayloadType) -> None:
# Store in database
await self._store_event("before_run", payload)
- async def successful_run(self, payload: PayloadType) -> None:
- await self._store_event("successful_run", payload)
+ async def observe_after_successful_run(self, payload: PayloadType) -> None:
+ await self._store_event("after_successful_run", payload)
- async def failing_run(self, payload: PayloadType) -> None:
- await self._store_event("failing_run", payload)
+ async def observe_after_failing_run(self, payload: PayloadType) -> None:
+ await self._store_event("after_failing_run", payload)
async def _store_event(self, event_type: str, payload: PayloadType):
# Your database storage logic here
@@ -180,17 +183,17 @@ class MetricsObserver(ObserverProtocol):
def __init__(self, metrics_client):
self.metrics = metrics_client
- async def before_run(self, payload: PayloadType) -> None:
+ async def observe_before_run(self, payload: PayloadType) -> None:
pipe_job = payload["pipe_job"]
- self.metrics.increment(f"pipe.{pipe_job.pipe_code}.started")
+ self.metrics.increment(f"pipe.{pipe_job.pipe.code}.started")
- async def successful_run(self, payload: PayloadType) -> None:
+ async def observe_after_successful_run(self, payload: PayloadType) -> None:
pipe_job = payload["pipe_job"]
- self.metrics.increment(f"pipe.{pipe_job.pipe_code}.success")
+ self.metrics.increment(f"pipe.{pipe_job.pipe.code}.success")
- async def failing_run(self, payload: PayloadType) -> None:
+ async def observe_after_failing_run(self, payload: PayloadType) -> None:
pipe_job = payload["pipe_job"]
- self.metrics.increment(f"pipe.{pipe_job.pipe_code}.failed")
+ self.metrics.increment(f"pipe.{pipe_job.pipe.code}.failed")
```
## Best Practices
@@ -206,7 +209,9 @@ class MetricsObserver(ObserverProtocol):
For automatic observer setup, integrate with your Pipelex initialization:
```python
-from pipelex import Pipelex
+from pipelex.hub import get_pipelex_hub
+from pipelex.pipelex import Pipelex
+
from my_project.observers import MyCustomObserver
def setup_pipelex():
diff --git a/docs/under-the-hood/build-time-elaboration.md b/docs/under-the-hood/build-time-elaboration.md
index 6a42fe8a4..72456424a 100644
--- a/docs/under-the-hood/build-time-elaboration.md
+++ b/docs/under-the-hood/build-time-elaboration.md
@@ -78,8 +78,8 @@ Three layers guard against authoring mistakes around the output concept:
The elaborator additionally:
- Verifies that the synthetic codes (`<pipe_code>__draft_text`, `<pipe_code>__structure`) don't already exist in the bundle.
-- Verifies that the synthetic codes pass `is_pipe_code_valid` (snake_case + length).
-- After producing the new bundle, re-runs `PipelexBundleBlueprint.model_validate(elaborated.model_dump(...))` so bundle-level validators (concept refs, pipe refs, `main_pipe`) re-check the synthetic pipes. Any `ValidationError` is wrapped in `BundleElaboratorError` with context naming the originating pipe.
+- Verifies that the synthetic codes pass `is_pipe_code_valid` (snake_case).
+- After producing the new bundle, re-runs `PipelexBundleBlueprint.model_validate(elaborated.model_dump(...))` so bundle-level validators (concept refs, pipe refs, `main_pipe`) re-check the synthetic pipes. Any `ValidationError` here is wrapped in `PipelexUnexpectedError`, with a message naming the bundle's `domain` and the sorted list of synthetic pipe codes (`BundleElaboratorError` is used only for the earlier synthesis-time checks during the same `elaborate()` call).
- Asserts that no synthesized blueprint itself carries `structuring_method = "preliminary_text"` — a recursive-elaboration guard. Today's synthesis explicitly sets `None`; the guard protects against future elaboration kinds copying fields wholesale.
## Cost and runtime semantics
diff --git a/docs/under-the-hood/codegen-projections.md b/docs/under-the-hood/codegen-projections.md
index e49195894..19f84e72f 100644
--- a/docs/under-the-hood/codegen-projections.md
+++ b/docs/under-the-hood/codegen-projections.md
@@ -2,7 +2,7 @@
Codegen turns one **normalized library crate** — the flat, fully-qualified, self-contained, fingerprinted snapshot of a resolved library (see [`pipelex resolve`](../tools/cli/build/structures.md) and the [Library Crate Format](https://mthds.ai)) — into typed, documented artifacts for each consumer. The crate is the single authority: codegen never re-derives meaning from raw bundles and never calls a model.
-This page describes the projection engine in `pipelex/codegen/`. The formal contract lives in `docs/specs/pipelex-codegen.md` (workspace root).
+This page describes the projection engine in `pipelex/codegen/` and the contract it implements: deterministic projections off the crate, stamped generated files, a sibling `codegen.lock`, and an offline drift check whose verdict rides the exit code.
## Two resolved layers, then emitters
@@ -37,7 +37,7 @@ Native materialization itself is faithful-or-absent per native: a content-class
## The CLI surface
-Two command families drive the engine (formal contract: `docs/specs/pipelex-codegen.md`, workspace root):
+Two command families drive the engine:
- **`pipelex resolve [PATH]… [-f json|toml] [-L DIR]…`** — assembles the closure (working bundles + the local `.mthds/methods/` cache), requires it to be **valid**, and emits the normalized crate to stdout. The verdict rides the exit code, mirroring the bare `validate` group: `0` resolved, `1` the library is invalid (a negative verdict — no crate), `2` no verdict (empty closure / not found).
- **`pipelex codegen <kind> …`** — the two-axis projection family (`kind` × `--target`):
@@ -89,7 +89,7 @@ The verdict rides the exit code (mirroring `resolve` / `validate`): `0` current,
## Serving the engine over HTTP
-The same engine backs the `pipelex-api` routes (`POST /v1/resolve`, `POST /v1/codegen`, and the re-pointed `/v1/build/*` — see `docs/specs/pipelex-codegen.md` → "Route envelopes" at the workspace root). Two host-facing cores make that possible without any CLI plumbing:
+The same engine backs the `pipelex-api` routes (`POST /v1/resolve`, `POST /v1/codegen`, and the re-pointed `/v1/build/*` — the route envelopes are documented in `pipelex-api`'s `docs/codegen.md`). Two host-facing cores make that possible without any CLI plumbing:
- **`pipelex.pipeline.resolve_bundle.resolve_crate_from_contents`** resolves **in-memory** MTHDS contents (strings, with optional per-content sources) into the normalized crate. It mirrors `validate_bundle`'s in-memory arm — the same `translate_to_validate_bundle_error` cascade, so an invalid library raises the one shared `ValidateBundleError` and a resolve verdict cannot drift from a validate verdict — and the same **loaded-on-success contract**: the library is left loaded and current for the host to read live pipes from, and the host owns its teardown. Resolution is static (no dry-run sweep), matching `pipelex resolve`.
- **`build_stamped_projection`** (above) gives the host the stamped artifact set plus the lock as pure content. A client that writes the served files and lock verbatim reproduces a local run byte-for-byte — the offline `codegen check` passes on the written tree exactly as it would locally. There is deliberately no server-side check route: the check is offline by design.
diff --git a/docs/under-the-hood/distributed-content-generation.md b/docs/under-the-hood/distributed-content-generation.md
index 01e2e9434..5e3710dfa 100644
--- a/docs/under-the-hood/distributed-content-generation.md
+++ b/docs/under-the-hood/distributed-content-generation.md
@@ -51,11 +51,11 @@ The schema is a plain `dict` — fully serializable, inspectable on the wire, an
### Schema-to-model reconstruction
-On the worker, `model_class_from_json_schema()` (`pipelex/cogt/content_generation/schema_to_model_factory.py`) rebuilds a live Pydantic class from the embedded schema:
+On the worker, `SchemaToModelFactory.make_from_json_schema(schema, *, class_name)` (`pipelex/cogt/content_generation/schema_to_model_factory.py`) rebuilds a live Pydantic class from the embedded schema:
```python
-source_code = _generate_source_from_schema(schema) # datamodel-code-generator
-reconstructed_class = _exec_and_extract_class(source_code, class_name) # exec()
+source_code = cls._generate_source_from_schema(schema) # datamodel-code-generator
+reconstructed_class = cls._exec_and_extract_class(source_code, class_name=class_name) # exec()
reconstructed_class.__kajson_class_source__ = source_code # attached for downstream use
```
@@ -63,7 +63,7 @@ The reconstruction pipeline:
1. **Generate** — `datamodel-code-generator` converts the JSON schema into Python source code defining a `BaseModel` subclass.
2. **Execute** — `exec()` compiles the source in an isolated namespace and extracts the named class.
-3. **Cache** — A thread-safe SHA-256 hash cache (with double-check locking) avoids redundant generation for the same schema.
+3. **Cache** — A thread-safe SHA-256 hash cache is guarded by a single lock held across the entire check-then-generate sequence, not double-check locking: the lock is acquired once, the cache is checked, and — on a miss — generation and exec happen before the lock is released. This kills same-schema thundering herds (concurrent first-misses each paying a full codegen+exec round) at the cost of serializing cache hits on other schemas behind any in-flight miss, an acceptable trade-off given real workloads use a small bounded set of distinct schemas cached for the process lifetime.
4. **Tag** — The generated source code is attached as `__kajson_class_source__` on the class, enabling Kajson to carry it through the transport.
!!! warning "Class name normalization"
diff --git a/docs/under-the-hood/dry-run-mock-generation.md b/docs/under-the-hood/dry-run-mock-generation.md
index 06e91ebc3..4134707b8 100644
--- a/docs/under-the-hood/dry-run-mock-generation.md
+++ b/docs/under-the-hood/dry-run-mock-generation.md
@@ -33,7 +33,7 @@ Standard mock generators (like Polyfactory) produce random strings like `"uygNji
```mermaid
flowchart TD
- A[pipelex validate --all] --> B[dry_run_pipes]
+ A[pipelex validate --all] --> B[BundleValidator.validate_pipes]
B --> C[WorkingMemoryFactory.make_mock_inputs]
C --> D[DryRunFactory.make_dry_run_factory]
D --> E[Polyfactory with custom providers]
@@ -98,6 +98,7 @@ class DryRunFactory:
def make_dry_run_factory(
cls,
object_class: type[BaseModelTypeVar],
+ *,
snake_case_field_names: set[str] | None = None,
pascal_case_field_names: set[str] | None = None,
) -> type[ModelFactory[BaseModelTypeVar]]:
@@ -163,11 +164,12 @@ class ConceptSpec(StructuredContent):
examples=["Text", "Image", "Document", "TextAndImages", "Number", "Page"],
)
-class PipeSpec(StructuredContent):
- # Extract talent should be a valid ExtractTalent value
- extract_talent: ExtractTalent | str = Field(
- description="Select extraction model talent",
- examples=list(ExtractTalent), # Polyfactory picks randomly from these
+class PipeComposeSpec(PipeSpec):
+ # Target format should be a valid TargetFormat value
+ target_format: TargetFormat | str | None = Field(
+ default=None,
+ description="Target format for the output (template mode)",
+ examples=list(TargetFormat), # Polyfactory picks randomly from these
)
```
diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.md
index d803ba21c..5aa2165d0 100644
--- a/docs/under-the-hood/error-model.md
+++ b/docs/under-the-hood/error-model.md
@@ -179,11 +179,12 @@ Every inference worker's SDK-exception handler collapses to a three-step pipelin
except (APIError, APIConnectionError, APITimeoutError) as exc:
metadata = extract_openai_metadata(exc)
classification = classify_inference_error(metadata)
- raise render_llm_error(
- family=InferenceErrorFamily.LLM_COMPLETION,
+ raise render_inference_error(
metadata=metadata,
classification=classification,
+ family=InferenceErrorFamily.LLM,
model_desc=self.inference_model.desc,
+ model_handle=self.inference_model.name,
) from exc
```
@@ -193,7 +194,7 @@ The three steps live in three modules. Only the per-provider Extract functions s
|--------|------|--------------|
| `pipelex/cogt/inference/error_classification.py` | Extract | `ProviderErrorMetadata`, `SDKErrorEnvelope`, `UserAction`, `UserActionKind`, the 12 `extract_*_metadata` functions, plus pure discriminators (`is_quota_exhaustion`, `is_content_policy_violation`, `is_network_error`) exposed as `@property` on the metadata |
| `pipelex/cogt/inference/error_classify.py` | Classify | `classify_inference_error()` — provider-blind mapping from `ProviderErrorMetadata` → `ClassificationResult(category, user_action_kind, is_model_not_found)` |
-| `pipelex/cogt/inference/error_render.py` | Render | `render_llm_error()` / `render_img_gen_error()` / `render_extract_error()` / `render_search_error()` — picks the `CogtError` subclass from `InferenceErrorFamily` plus `is_model_not_found` (e.g. `LLMModelNotFoundError` vs `LLMCompletionError`) |
+| `pipelex/cogt/inference/error_render.py` | Render | `render_inference_error()` — picks the `CogtError` subclass from `InferenceErrorFamily` plus `is_model_not_found` (e.g. `LLMModelNotFoundError` vs `LLMCompletionError`) |
Provider-specific nuance is normalized away in Extract (e.g. Google's `code` becomes `status_code`; AWS Bedrock error codes are mapped to HTTP statuses), so Classify has no provider branching. HTTP status drives classification; status-less errors dispatch on the SDK exception type name. The `tests/unit/pipelex/cogt/inference/test_provider_classification_parity.py` meta-test walks every `ProviderName` against the extract-fn registry so adding a new provider without wiring it fails fast.
@@ -226,7 +227,7 @@ On structured-generation paths, `instructor` wraps the real SDK exception in an
Inference-failure leaf errors (`LLMCompletionError`, `ImgGenGenerationError`, …) are raised deep inside a plugin and do not know which model handle invoked them. Each worker family fills that in at its public-method chokepoint:
```python
-def fill_model_and_provider(self, model_handle: str | None, backend_name: str | None) -> None:
+def fill_model_and_provider(self, model_handle: str | None, *, backend_name: str | None) -> None:
"""Fill model_handle / backend_name from the worker, only when still unset."""
```
@@ -415,7 +416,7 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT
| `pipelex/cogt/exceptions.py` | `CogtError`, `InferenceErrorCategory` |
| `pipelex/cogt/inference/error_classification.py` | Extract — `ProviderErrorMetadata`, `SDKErrorEnvelope`, `UserAction`, `UserActionKind`, per-provider `extract_*_metadata` functions, pure discriminators |
| `pipelex/cogt/inference/error_classify.py` | Classify — `classify_inference_error()`, `ClassificationResult` |
-| `pipelex/cogt/inference/error_render.py` | Render — `render_llm_error()` / `render_img_gen_error()` / `render_extract_error()` / `render_search_error()`, `InferenceErrorFamily` |
+| `pipelex/cogt/inference/error_render.py` | Render — `render_inference_error()`, `InferenceErrorFamily` |
| `pipelex/cogt/inference/provider_name.py` | `ProviderName` enum keying the extract-fn registry |
| `pipelex/plugins/*/` | Per-provider inference workers — Layer 0 → 1 classification |
| `pipelex/pipeline/exceptions.py` | `PipelineExecutionError`, `PipeExecutionError` |
diff --git a/docs/under-the-hood/execution-graph-tracing.md b/docs/under-the-hood/execution-graph-tracing.md
index c7b430a03..a8716cda1 100644
--- a/docs/under-the-hood/execution-graph-tracing.md
+++ b/docs/under-the-hood/execution-graph-tracing.md
@@ -32,11 +32,11 @@ Pipe Execution → GraphTracer → GraphSpec → Renderers → HTML/Mermaid
|----------|-----|-----|--------|
| Generate execution graph | `pipelex run pipe my_pipe --graph` | `PipelexMTHDSProtocol(execution_config=...).execute(...)` | GraphSpec JSON + HTML viewers |
| Force include full data | `--graph --graph-full-data` | `data_inclusion.stuff_json_content=True` | Data embedded in IOSpec |
-| Force exclude data | `--graph --graph-no-data` | All `data_inclusion.*=False` | Previews only |
+| Force exclude data | `--graph --graph-no-data` | The four stuff/error `data_inclusion` flags → `False` (`pipe_and_concept_registry` unaffected) | Previews only |
| Dry run with graph | `--dry-run --graph` | `PipelexMTHDSProtocol(pipe_run_mode=PipeRunMode.DRY, execution_config=...)` | Graph of mock execution |
!!! info "Full Data Included by Default"
- The default configuration includes full data in graphs (`stuff_json_content`, `stuff_text_content`, `stuff_html_content`, and `error_stack_traces` are all `true`). Use `--graph-full-data` or `--graph-no-data` only to override project-specific settings.
+ The default configuration includes full data in graphs (`stuff_json_content`, `stuff_text_content`, `stuff_html_content`, `error_stack_traces`, and `pipe_and_concept_registry` are all `true`). Use `--graph-full-data` or `--graph-no-data` only to override project-specific settings — the flags toggle the first four; `pipe_and_concept_registry` is set only via config.
---
@@ -89,10 +89,10 @@ graph_spec = response.pipe_output.graph_spec
| Output | File | Purpose |
|--------|------|---------|
-| `graphspec_json` | `_graphspec.json` | Canonical graph representation |
-| `mermaidflow_mmd` | `_mermaid.mmd` | Mermaid flowchart code |
-| `mermaidflow_html` | `_mermaid.html` | Standalone Mermaid viewer |
-| `reactflow_html` | `_reactflow.html` | Interactive ReactFlow viewer |
+| `graphspec_json` | `graphspec.json` | Canonical graph representation |
+| `mermaidflow_mmd` | `mermaidflow.mmd` | Mermaid flowchart code |
+| `mermaidflow_html` | `mermaidflow.html` | Standalone Mermaid viewer |
+| `reactflow_html` | `reactflow.html` | Interactive ReactFlow viewer |
---
@@ -195,6 +195,9 @@ class GraphSpec(BaseModel):
| `DATA` | Data flow (stuff passed between pipes) |
| `CONTAINS` | Parent-child containment (controller → children) |
| `SELECTED_OUTCOME` | Condition outcome selection |
+| `BATCH_ITEM` | Batch fan-out: input list → item extracted for each batch iteration |
+| `BATCH_AGGREGATE` | Batch fan-in: item outputs → aggregated output list |
+| `PARALLEL_COMBINE` | Branch outputs → combined output in PipeParallel |
Every `EdgeSpec` also carries an `optional` boolean (default `false`). On a `DATA` edge it marks that the producer's output is declared optional (`?` presence marker) — the value flowed on this run but may be absent on others. Renderers can use it to draw the edge distinctly (e.g. dashed).
@@ -245,7 +248,7 @@ node_id, child_context = manager.on_pipe_start(
# 4. On completion, report success with output
manager.on_pipe_end_success(
- graph_id=trace_context.graph_id,
+ lookup_key=trace_context.lookup_key,
node_id=node_id,
ended_at=datetime.now(timezone.utc),
output_spec=IOSpec(...),
@@ -266,7 +269,7 @@ from pipelex.hub import scoped_event_log
from pipelex.tracing.in_memory_event_log import InMemoryEventLog
with scoped_event_log(InMemoryEventLog()):
- response = await runner.execute_pipeline(...) # graph assembles in memory
+ response = await runner.execute(...) # graph assembles in memory
```
Semantics:
@@ -288,7 +291,7 @@ class TraceContext(BaseModel):
node_sequence: int # Counter for generating node IDs
data_inclusion: DataInclusionConfig # What data to capture
- def copy_for_child(self, child_node_id: str, next_sequence: int) -> TraceContext:
+ def copy_for_child(self, child_node_id: str, *, next_sequence: int) -> TraceContext:
"""Create context for nested pipe execution."""
return TraceContext(
graph_id=self.graph_id,
@@ -415,6 +418,7 @@ stuff_json_content = true # Include full serialized data
stuff_text_content = true # Include ASCII text representation
stuff_html_content = true # Include HTML representation
error_stack_traces = true # Include full stack traces
+pipe_and_concept_registry = true # Include pipe and concept registries in the GraphSpec
[pipelex.pipeline_execution_config.graph_config.graphs_inclusion]
graphspec_json = true # Generate GraphSpec JSON
@@ -427,22 +431,28 @@ reactflow_html = true # Generate ReactFlow HTML
| Option | Description |
|--------|-------------|
-| `direction` | Flowchart direction (TB, LR, BT, RL) |
+| `direction` | Flowchart direction (`top_down`, `left_to_right`) |
| `is_include_data_edges` | Show data flow edges |
| `is_include_contains_edges` | Show containment edges |
+| `is_include_selected_outcome_edges` | Show condition-result (selected outcome) edges |
| `is_show_stuff_codes` | Show digest in stuff labels |
-| `style.theme` | Mermaid theme (default, dark, forest, neutral) |
+| `style.theme` | Mermaid theme (default, base, dark, forest, neutral) |
### ReactFlowRenderingConfig
| Option | Description |
|--------|-------------|
-| `layout_direction` | Dagre layout direction (TB, LR) |
+| `is_use_cdn` | Load the graph viewer assets from a CDN (jsDelivr) instead of inlining them |
+| `layout_direction` | Flowchart layout direction (`top_down`, `left_to_right`), converted internally to Dagre's TB/LR |
| `nodesep` | Node separation in pixels |
| `ranksep` | Rank separation in pixels |
| `edge_type` | Edge style (bezier, smoothstep, step, straight) |
| `initial_zoom` | Initial viewport zoom |
+| `pan_to_top` | Pan the viewport to the top on load |
+| `default_title` | Title of the generated HTML page |
+| `show_batch_controller` | Render controller pipes as grouping containers |
| `style.theme` | UI theme (light, dark, system) |
+| `style.palette` | Node color palette (`yellow_blue`, `dracula`) |
---
@@ -458,9 +468,10 @@ class IOSpec(BaseModel):
preview: str | None # Truncated preview (max 200 chars)
size: int | None # Content size
digest: str | None # Unique identifier for data flow
- data: Any | None # Full serialized content
+ data: str | dict[str, Any] | list[str] | list[dict[str, Any]] | None # Full serialized content
data_text: str | None # ASCII text representation
data_html: str | None # HTML representation
+ extra: dict[str, Any] # Extra markers, e.g. `optional` set on a declared-optional (`?`) output
```
!!! warning "Preview Truncation"
diff --git a/docs/under-the-hood/image-handling-in-llm-prompts.md b/docs/under-the-hood/image-handling-in-llm-prompts.md
index 4e27c3a32..20597258d 100644
--- a/docs/under-the-hood/image-handling-in-llm-prompts.md
+++ b/docs/under-the-hood/image-handling-in-llm-prompts.md
@@ -148,9 +148,8 @@ PageContent(
The filter produces:
```
-text_and_images:
- text: Welcome to the guide
- images: [Image 1]
+text_and_images: Welcome to the guide
+[Image 1]
page_view: [Image 2]
```
@@ -278,21 +277,22 @@ The `ImageRegistry` manages image numbering during prompt construction.
### Key Properties
-- **1-indexed** - Numbers start at 1 for readability
+- **0-based indices, 1-based display** - `register_image` returns a 0-based index; the 1-based `[Image N]` display number is produced at render time
- **Sequential** - Images numbered in order of registration
-- **Deduplicated** - Same URL gets same number
+- **Deduplicated** - Same URL gets same index
```python
class ImageRegistry:
- def register_image(self, image: ImageContent) -> int:
- """Returns image number. Same URL = same number."""
- if image.url in self._url_to_number:
- return self._url_to_number[image.url]
-
- number = len(self._images) + 1
+ def register_image(self, image: Any) -> int:
+ """Register an image and return its 0-based index. Same URL = same index."""
+ if image.url in self._image_urls:
+ # Find existing index
+ for index, existing in enumerate(self._images):
+ if existing.url == image.url:
+ return index
self._images.append(image)
- self._url_to_number[image.url] = number
- return number
+ self._image_urls.add(image.url)
+ return len(self._images) - 1
```
### Deduplication Example
@@ -301,13 +301,13 @@ If the same image appears in multiple places:
```python
# First registration
-registry.register_image(img_a) # Returns 1
+registry.register_image(img_a) # Returns 0
# Second registration of same URL
-registry.register_image(img_a) # Returns 1 (not 2)
+registry.register_image(img_a) # Returns 0 (not 1)
# Different image
-registry.register_image(img_b) # Returns 2
+registry.register_image(img_b) # Returns 1
```
---
@@ -332,11 +332,11 @@ The `with_images` filter uses the `ImageRenderable` protocol to handle StuffArte
```python
# StuffArtefact implements ImageRenderable
if isinstance(value, ImageRenderable):
- return value.render_with_images(registry, text_format=text_format)
+ return value.render_with_images(registry=registry, text_format=text_format)
# StuffArtefact.render_with_images() delegates to content
-def render_with_images(self, registry, *, text_format) -> str:
- return self._stuff.content.render_with_images(registry, text_format=text_format)
+def render_with_images(self, *, registry, text_format) -> str:
+ return self._stuff.content.render_with_images(registry=registry, text_format=text_format)
```
!!! note "ImageRenderable Protocol"
@@ -354,15 +354,14 @@ The system validates image usage at both factory time and runtime:
| Condition | Error |
|-----------|-------|
-| `\| with_images` on `Image` type | "Cannot use with_images on direct Image" |
-| `\| with_images` on type with no nested images | "Type X has no nested image fields" |
+| `\| with_images` on a type with no nested images (including a direct `Image`) | `WithImagesFilterError`: "Filter '\| with_images' used on variable 'X' but the type has no nested images. Remove the filter or use a type with nested images." |
### Runtime
| Condition | Error |
|-----------|-------|
| `with_images` on undefined value | "Cannot use with_images filter on undefined value" |
-| `with_images` on non-ImageRenderable type (e.g., string) | "X does not implement the ImageRenderable protocol" |
+| `with_images` on non-ImageRenderable type (e.g., string) | "The with_images filter received a X which does not implement the ImageRenderable protocol. This filter requires StuffArtefact, StuffContent subclasses, or lists containing such types. Did you accidentally apply another filter first that converted to string?" |
The runtime check catches cases where filter chaining converts structured data to a string before `with_images` runs (e.g., `{{ pages | tag | with_images }}`).
@@ -438,8 +437,8 @@ Pages:
| File | Purpose |
|------|---------|
-| `pipelex/pipe_operators/llm/image_reference.py` | `ImageReference` and `ImageReferenceKind` models |
-| `pipelex/pipe_operators/llm/template_image_analyzer.py` | Factory-time template analysis |
+| `pipelex/pipe_operators/shared/image_reference.py` | `ImageReference` and `ImageReferenceKind` models |
+| `pipelex/pipe_operators/shared/template_image_analyzer.py` | Factory-time template analysis |
| `pipelex/tools/jinja2/image_registry.py` | Runtime image tracking |
| `pipelex/tools/jinja2/jinja2_with_images_filter.py` | The `with_images` filter implementation |
@@ -448,7 +447,7 @@ Pages:
| File | Purpose |
|------|---------|
| `pipelex/tools/jinja2/jinja2_required_variables.py` | `VariableReference` for filter detection |
-| `pipelex/pipe_operators/llm/llm_prompt_spec.py` | Prompt building with image collection |
+| `pipelex/pipe_operators/llm/llm_prompt_blueprint.py` | Prompt building with image collection |
---
diff --git a/docs/under-the-hood/init-cli-flows.md b/docs/under-the-hood/init-cli-flows.md
index 9f26f381a..0df8ebb54 100644
--- a/docs/under-the-hood/init-cli-flows.md
+++ b/docs/under-the-hood/init-cli-flows.md
@@ -13,7 +13,7 @@ description: "Inside the pipelex init command — how focus-based dispatch sets
The `.pipelex/` directory contains two categories of files with different lifecycles:
-1. **Config files** (`pipelex.toml`, `plxt.toml`, `mthds_schema.json`) — static templates, copied once, rarely touched by the user.
+1. **Config files** (`pipelex.toml`, `plxt.toml`) — static templates copied verbatim from the kit (no interactive customization); re-running init overwrites them (full reset).
2. **Inference files** (`inference/backends.toml`, `inference/routing_profiles.toml`, `inference/backends/*.toml`, `inference/deck/*.toml`) — interactive setup, customized per-project based on which AI backends the user selects.
These two categories are managed by separate steps. `init_config()` copies only config files (skipping the `inference/` directory entirely). The inference step handles its own template copying and then runs interactive backend selection and routing customization. Each file is owned by exactly one step — `init_config()` explicitly skips the `inference/` directory via `INIT_SKIP_DIRS`, and skips `telemetry.toml` and `pipelex_service.toml` via `INIT_SKIP_FILES`. This separation ensures that re-running `pipelex init config` never overwrites a user's carefully tuned inference setup.
@@ -48,7 +48,6 @@ All commands except `agreement` and `credentials` perform a **full reset** (over
|----------|-------------|------------|------|
| `pipelex.toml` | `init_config()` | Project or global | `.pipelex/pipelex.toml` |
| `plxt.toml` | `init_config()` | Project or global | `.pipelex/plxt.toml` |
-| `mthds_schema.json` | `init_config()` | Project or global | `.pipelex/mthds_schema.json` |
| `backends.toml` | Inference step | Project or global | `.pipelex/inference/backends.toml` |
| `backends/*.toml` | Inference step | Project or global | `.pipelex/inference/backends/` |
| `deck/*.toml` | Inference step | Project or global | `.pipelex/inference/deck/` |
@@ -259,7 +258,6 @@ Copies a telemetry template and prints instructions. No interactive prompts. Whi
|-----------------|-----------|---------|
| `pipelex.toml` | `init_config()` | Main Pipelex configuration |
| `plxt.toml` | `init_config()` | PLXT tooling configuration |
-| `mthds_schema.json` | `init_config()` | MTHDS JSON Schema for IDE support |
| `inference/` | Inference step | All inference configuration (see below) |
| `inference/backends.toml` | Inference step | Backend registry (enabled/disabled flags) |
| `inference/backends/*.toml` | Inference step | Per-backend settings (API keys, endpoints) |
@@ -308,7 +306,9 @@ The `ConfigLoader` (singleton: `config_manager`) resolves where configuration fi
`find_project_root()` walks up from `cwd` looking for marker files:
-- `.git`, `pyproject.toml`, `setup.py`, `setup.cfg`, `package.json`, `.hg`
+- `.pipelex`, `.git`, `pyproject.toml`, `setup.py`, `setup.cfg`, `package.json`, `.hg`
+
+A directory containing only a `.pipelex/` directory is therefore itself a valid project root.
The home directory (`~`) is explicitly excluded — even if it contains a `package.json`, it is never treated as a project root.
@@ -402,8 +402,7 @@ If no project root is found and `--global` is not set, the command fails with an
|-------|------|---------|
| `backends` | `list[str]` | Backend keys to enable (e.g. `"openai"`, `"anthropic"`, `"pipelex_gateway"`) |
| `primary_backend` | `str` | Required when 2+ backends selected and `pipelex_gateway` is not among them |
-| `accept_gateway_terms` | `bool` | Sets gateway terms acceptance (defaults to `false` if omitted) |
-| `telemetry_mode` | `str` | `"off"` (default), `"anonymous"`, or `"identified"` |
+| `accept_gateway_terms` | `bool` | Sets gateway terms acceptance; when omitted, nothing is written and any existing acceptance state is left untouched |
### Flow
@@ -413,13 +412,15 @@ flowchart TD
PARSE --> RESOLVE["Resolve target dir<br/>(project or --global)"]
RESOLVE --> STEP1["Step 1: init_config()<br/>Copy config templates (skips inference/)"]
STEP1 --> STEP15["Step 1.5: Copy inference templates<br/>(backends.toml, backends/*, deck/*, routing_profiles.toml)"]
- STEP15 --> STEP2["Step 2: Configure backends<br/>Enable requested backends in backends.toml"]
+ STEP15 --> STEP16["Step 1.6: Copy telemetry template<br/>(global: active defaults; project: commented-out)"]
+ STEP16 --> STEP2["Step 2: Configure backends<br/>Enable requested backends in backends.toml"]
STEP2 --> GW{pipelex_gateway<br/>in backends?}
- GW -- Yes --> TERMS["Write gateway terms to ~/.pipelex/<br/>(always global)"]
+ GW -- Yes --> TERMS["Write gateway terms to ~/.pipelex/<br/>(always global; skipped if accept_gateway_terms omitted)"]
GW -- No --> STEP3
TERMS --> STEP3["Step 3: Configure routing<br/>Auto-derive routing profile"]
- STEP3 --> STEP4["Step 4: Configure telemetry<br/>Copy template + apply mode"]
- STEP4 --> OUTPUT["Output structured JSON result"]
+ STEP3 --> STEP4["Step 4: Mark inference setup completed<br/>(written to ~/.pipelex/)"]
+ STEP4 --> STEP5["Step 5: Prime remote-config cache<br/>(no-op if gateway disabled or terms not accepted)"]
+ STEP5 --> OUTPUT["Output result<br/>(Markdown, or JSON via --format json)"]
```
!!! note "No credentials step"
@@ -431,7 +432,7 @@ flowchart TD
|--------|---------------|---------------------|
| Prompts | Interactive (Rich prompts) | None — all input from `--config` JSON |
| Gateway terms | Prompted interactively | From `accept_gateway_terms` field in config |
-| Output | Rich console output | Structured JSON via `agent_success()` / `agent_error()` |
+| Output | Rich console output | Markdown by default (`agent_success_formatted()` / `agent_error()`), structured JSON via `--format json` |
| Credentials | Prompted interactively | Not configured — use `pipelex-agent doctor` or `pipelex init credentials` |
| Focus dispatch | Supports individual focus (`config`, `inference`, etc.) | Runs config, inference, routing, telemetry (no credentials) |
| Target dir default | Global (`~/.pipelex/`), use `--local` for project | Project (`{project_root}/.pipelex/`), use `--global` for global |
diff --git a/docs/under-the-hood/orchestrator-plugins.md b/docs/under-the-hood/orchestrator-plugins.md
index ff65234f4..738f3b3b7 100644
--- a/docs/under-the-hood/orchestrator-plugins.md
+++ b/docs/under-the-hood/orchestrator-plugins.md
@@ -8,9 +8,9 @@ description: "How Pipelex dispatches a pipe run by orchestration mode through th
A pipe a host runtime invokes through the runtime bridge runs along **two orthogonal axes**:
- **`orchestration_mode`** — *which* orchestrator runs the pipe. An **open string token**, not a closed enum: core owns only `"direct"` (in-process); every other token is contributed by the plugin that owns its orchestrator — [`"temporal"`](https://pipelex.com/products#temporal) (durable, on a Temporal worker fleet) by `pipelex-temporal`, [`"mistral-workflows"`](https://pipelex.com/products#mistral-workflows) (decomposed into Mistral Workflows primitives) by `pipelex-mistralai-workflows`. Neither is built into the open-source `pipelex` core: both ship as external, closed-source host-runtime backends, distributed privately rather than on PyPI as part of Pipelex's [workflow-orchestration offer](https://pipelex.com/products#durable-execution).
-- **`delivery`** — *whether the caller waits*. A **closed** core `DeliveryMode` enum (`BLOCKING` / `FIRE_AND_FORGET`) on the wire input, set by the endpoint, never received from a caller. On the SPI it is expressed as *which method* the endpoint calls — `run` (blocking) or `start` (fire-and-forget) — so each return type is truthful on its own; `supports_fire_and_forget` advertises whether an orchestrator can do genuine async.
+- **`delivery`** — *whether the caller waits*. A **closed** core `DeliveryMode` enum (`BLOCKING` / `FIRE_AND_FORGET`) on the wire input, set by the endpoint, never received from a caller. On the SPI it is expressed as *which method* the endpoint calls — `execute` (blocking) or `start` (fire-and-forget) — so each return type is truthful on its own; `supports_fire_and_forget` advertises whether an orchestrator can do genuine async.
-An **orchestrator** is what knows how to run a pipe under one token. Core names no orchestrator by import or by string. The bridge resolves the orchestrator for the requested token from a registry (keyed by the token `str`) and calls its `run` — `"direct"` is contributed by a core plugin, `"temporal"` by the Temporal plugin, `"mistral-workflows"` by the external `pipelex-mistralai-workflows` plugin. A lookup miss raises a generic `MissingOrchestratorError` that names no orchestrator. This page documents that seam, the **Orchestrator SPI** a host-runtime plugin compiles against, and how the Temporal plugin is wired.
+An **orchestrator** is what knows how to run a pipe under one token. Core names no orchestrator by import or by string. The bridge resolves the orchestrator for the requested token from a registry (keyed by the token `str`) and calls its `execute` — `"direct"` is contributed by a core plugin, `"temporal"` by the Temporal plugin, `"mistral-workflows"` by the external `pipelex-mistralai-workflows` plugin. A lookup miss raises a generic `MissingOrchestratorError` that names no orchestrator. This page documents that seam, the **Orchestrator SPI** a host-runtime plugin compiles against, and how the Temporal plugin is wired.
---
@@ -21,7 +21,7 @@ An **orchestrator** is what knows how to run a pipe under one token. Core names
→ build the PipeJob (boundary decode + library scope + trace_context)
→ orchestrator = get_orchestrator_registry().get_optional(mode=orchestration_mode)
→ if orchestrator is None: raise MissingOrchestratorError(mode) # generic, names no orchestrator
- → return await orchestrator.run(pipe_job=..., delivery_assignment=...) # or orchestrator.start(...) for FIRE_AND_FORGET
+ → return await orchestrator.execute(pipe_job=..., delivery_assignment=...) # or orchestrator.start(...) for FIRE_AND_FORGET
```
The registry is built once at boot from whatever the discovered plugins contributed (`build_registrar` → `OrchestratorRegistry` on the hub). There is no `match orchestration_mode:` anywhere in the bridge — the token set is open, so validation is the registry lookup itself; adding a mode's behavior means registering an orchestrator for its token, nothing in core changes.
@@ -39,12 +39,12 @@ An orchestrator satisfies `OrchestratorProtocol` (`pipelex/plugins/orchestrator_
class OrchestratorProtocol(Protocol):
supports_fire_and_forget: bool
- async def run(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput: ...
+ async def execute(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput: ...
async def start(self, *, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeDispatchAck: ...
```
-The delivery axis is split into two methods so each return type is truthful on its own. `run` is the BLOCKING arm: it awaits completion and returns the completed-run `PipelexPipeRunOutput` — which always carries a main stuff (`main_stuff_name` is required, no "not finished yet" escape hatch). `start` is the FIRE_AND_FORGET arm: it genuinely enqueues the job and returns a `PipelexPipeDispatchAck` (`pipeline_run_id` + `workflow_id`, nothing more — nothing has run yet). `supports_fire_and_forget` is the capability a runner reads *before* dispatch — `/start` rejects honestly (4xx) when the resolved mode cannot do genuine async, instead of silently running blocking and acking; an orchestrator that cannot (like core's DIRECT) implements `start` by raising, unreachable behind that gate.
+The delivery axis is split into two methods so each return type is truthful on its own. `execute` is the BLOCKING arm: it awaits completion and returns the completed-run `PipelexPipeRunOutput` — which always carries a main stuff (`main_stuff_name` is required, no "not finished yet" escape hatch). `start` is the FIRE_AND_FORGET arm: it genuinely enqueues the job and returns a `PipelexPipeDispatchAck` (`pipeline_run_id` + `workflow_id`, nothing more — nothing has run yet). `supports_fire_and_forget` is the capability a runner reads *before* dispatch — `/start` rejects honestly (4xx) when the resolved mode cannot do genuine async, instead of silently running blocking and acking; an orchestrator that cannot (like core's DIRECT) implements `start` by raising, unreachable behind that gate.
A plugin contributes one per token it serves by calling the registrar menu in its `register`, passing the token as a raw string (no enum, no cast):
@@ -52,7 +52,7 @@ A plugin contributes one per token it serves by calling the registrar menu in it
registrar.add_orchestrator(mode="temporal", orchestrator=TemporalOrchestrator())
```
-Constructing the orchestrator instance must be **import-light** — it must not import the host-runtime SDK (`temporalio`, …) at module scope or in its `__init__`. The heavy import happens lazily inside `run` (and a friendly `MissingOrchestratorError` is raised there if the mode's extra is absent), so discovering and registering the plugin never pulls the SDK. This is what keeps boot import-light even on a process that will never use the mode.
+Constructing the orchestrator instance must be **import-light** — it must not import the host-runtime SDK (`temporalio`, …) at module scope or in its `__init__`. The heavy import happens lazily inside `execute` (and a friendly `MissingOrchestratorError` is raised there if the mode's extra is absent), so discovering and registering the plugin never pulls the SDK. This is what keeps boot import-light even on a process that will never use the mode.
### A missing orchestrator is a generic, plugin-decoupled error
@@ -62,7 +62,7 @@ A token with no registered orchestrator (its plugin is not installed) raises `Mi
## Bundle validators: the `/validate` counterpart
-`/validate` rides the same open-token seam as `run`. A **bundle validator** produces a validation verdict for one orchestration mode, the way an orchestrator runs a pipe for one mode: the core `direct` plugin contributes the in-process validator, and an external orchestrator plugin contributes a worker-dispatched validator under its own token — which core never names. A host runtime resolves the requested mode and dispatches through the `BundleValidatorRegistry` on the hub instead of branching on a backend.
+`/validate` rides the same open-token seam as `execute`. A **bundle validator** produces a validation verdict for one orchestration mode, the way an orchestrator runs a pipe for one mode: the core `direct` plugin contributes the in-process validator, and an external orchestrator plugin contributes a worker-dispatched validator under its own token — which core never names. A host runtime resolves the requested mode and dispatches through the `BundleValidatorRegistry` on the hub instead of branching on a backend.
A validator satisfies `BundleValidatorProtocol` (`pipelex/plugins/bundle_validator_registry.py`):
@@ -80,9 +80,9 @@ class BundleValidatorProtocol(Protocol):
The `library_dirs` annotation is a quoted forward reference because `Sequence` and `Path` are imported under `TYPE_CHECKING` in `bundle_validator_registry.py`; a plugin that imports them at module scope can use the unquoted form.
-Two contract points distinguish it from `OrchestratorProtocol.run`:
+Two contract points distinguish it from `OrchestratorProtocol.execute`:
-- **No fire-and-forget arm.** Validation is inherently blocking, so — unlike orchestration's `run`/`start` split — there is no `start` counterpart and no `supports_fire_and_forget` capability flag.
+- **No fire-and-forget arm.** Validation is inherently blocking, so — unlike orchestration's `execute`/`start` split — there is no `start` counterpart and no `supports_fire_and_forget` capability flag.
- **Verdict-as-value, not raise.** `validate_bundles` *returns* the verdict — `BundleValidationVerdict` is the union of the valid arm (a `ValidationReport`) and the invalid arm (an `ErrorReport` carrying `validation_errors`) — and raises only for a no-verdict infra fault, which a host runtime maps to a 5xx. This is the same valid/invalid pair the API maps onto its 200-always `/validate` wire, so the verdict contract is backend-independent.
The seam is deliberately typed at the MTHDS-protocol level (`ValidationReport` from `mthds.protocol`), not the concrete `PipelexValidationReport` envelope: the concrete report's module reaches the hub, so naming it from this hub-reachable seam would close an import cycle, and the seam is generic across orchestrators (language-standard altitude), so it speaks the protocol report — the Pipelex-runtime envelope is the concrete `ValidationReport` subtype the validators actually produce, and the API recovers that precise type at its edge. `library_dirs` is host context the in-process arm needs to load the method library; a worker-dispatched arm ignores it — its worker loads its own library.
@@ -155,7 +155,7 @@ At each ordered hub slot, `Pipelex.setup` resolves in this precedence:
2. a plugin slot-claim thunk;
3. the core default.
-A slot claim must never silently override an explicit injection. Two slots take no explicit `setup()` parameter and so skip step 1: the task manager (no core default either — unclaimed means no task manager wiring at all) and the isolated-execution probe (whose core default is the always-False predicate above). Teardown runs the plugin-registered teardown callbacks **LIFO**, before core teardown, so a worker's in-flight runtime resources release first.
+A slot claim must never silently override an explicit injection. Only the content generator and pipe router take an explicit `setup()` parameter; the other slots skip step 1 — the task manager (no core default either — unclaimed means no task manager wiring at all), the isolated-execution probe (whose core default is the always-False predicate above), and the pipe run (plugin thunk or core default, nothing else). Teardown runs the plugin-registered teardown callbacks **LIFO**, before core teardown, so a worker's in-flight runtime resources release first.
---
@@ -198,10 +198,10 @@ What an out-of-tree orchestrator imports *is* a contract. The SPI is a documente
`pipelex_temporal/temporal_plugin.py` (in the external `pipelex-temporal` distribution) is the reference orchestrator plugin. Its `register`:
-- **always** (regardless of the boot gate): contributes a single `TemporalOrchestrator` registered once under the `"temporal"` token (import-light; `temporalio` is pulled lazily inside `run`/`start`), the matching `TemporalBundleValidator` under the same token (the worker-dispatched `/validate` arm), and an HTTP error mapper classifying Temporal transport faults. The orchestrator advertises `supports_fire_and_forget = True`; its `run` awaits completion and reports `make_workflow_id(...)`, its `start` enqueues the workflow and returns a `PipelexPipeDispatchAck` carrying the workflow id;
+- **always** (regardless of the boot gate): contributes a single `TemporalOrchestrator` registered once under the `"temporal"` token (import-light; `temporalio` is pulled lazily inside `execute`/`start`), the matching `TemporalBundleValidator` under the same token (the worker-dispatched `/validate` arm), and an HTTP error mapper classifying Temporal transport faults. The orchestrator advertises `supports_fire_and_forget = True`; its `execute` awaits completion and reports `make_workflow_id(...)`, its `start` enqueues the workflow and returns a `PipelexPipeDispatchAck` carrying the workflow id;
- **only when `plugins.boot_orchestrator == "temporal"`**: claims the content-generator / task-manager / pipe-router / pipe-run / isolated-execution-probe hub slots with thunks and registers the teardown callback — booting this process as a Temporal-default runtime.
-The orchestrator itself (`pipelex_temporal/temporal_orchestrators.py`) keeps the `WorkflowExecutionError` catch and the `make_workflow_id` recompute in the blocking `run` arm. It serializes its `PipeOutput` through `pipelex.runtime_bridge.serialization`, shared with the core DIRECT orchestrator so the boundary shape cannot drift.
+The orchestrator itself (`pipelex_temporal/temporal_orchestrators.py`) keeps the `WorkflowExecutionError` catch and the `make_workflow_id` recompute in the blocking `execute` arm. It serializes its `PipeOutput` through `pipelex.runtime_bridge.serialization`, shared with the core DIRECT orchestrator so the boundary shape cannot drift.
The Temporal plugin is **external** — it ships as the `pipelex-temporal` distribution and is discovered through a `pipelex.plugins` entry point in that dist's `pyproject.toml`, not through `BUILTIN_PLUGINS`. Core's `BUILTIN_PLUGINS` (`pipelex/plugins/builtins.py`) holds only the always-shipped inference and `direct` plugins and explicitly excludes Temporal; installing `pipelex-temporal` is all it takes to make the `"temporal"` orchestrator available — zero config, no core import of `temporalio`. Its operational `worker` / `setup-namespace` commands ship as the standalone `pipelex-temporal` console script, so they travel with that dist.
diff --git a/docs/under-the-hood/pipe-routing-and-execution.md b/docs/under-the-hood/pipe-routing-and-execution.md
index 4e8ed0609..1dd04ac75 100644
--- a/docs/under-the-hood/pipe-routing-and-execution.md
+++ b/docs/under-the-hood/pipe-routing-and-execution.md
@@ -77,7 +77,7 @@ When a `.mthds` file declares a concept like `RawText = "Raw input text..."`, th
These dynamically-generated classes become the `content` type of `Stuff` objects in `WorkingMemory`.
!!! info "Why Dynamic Classes Matter"
- When a PipeJob is serialized (e.g., for Temporal transport), Kajson embeds `__class__` and `__module__` metadata. The receiving process must have these classes registered in its class registry to deserialize the payload.
+ When `WorkingMemory` crosses to a Temporal worker, `dump_for_transport()` stamps content with pipelex-private `__pipelex_class__` / `__pipelex_module__` markers instead of Kajson's `__class__` / `__module__` — deliberately keeping Kajson's universal decoder out of the loop at the data-converter boundary. Pipelex's hydrator resolves those markers against the per-workflow `ClassRegistry`, the only place these dynamic concept classes are registered.
---
@@ -125,12 +125,13 @@ effective_pipe_router = self._resolve_hub_slot(
`PipeRouter` implements `PipeRouterProtocol` with a minimal `_run_pipe_job()`:
```python
-async def _run_pipe_job(self, pipe_job, wfid=None):
+async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput:
return await pipe_job.pipe.run_pipe(
job_metadata=pipe_job.job_metadata,
- working_memory=pipe_job.working_memory,
+ working_memory=pipe_job.get_working_memory(),
output_name=pipe_job.output_name,
pipe_run_params=pipe_job.pipe_run_params,
+ library_crate=pipe_job.library_crate,
)
```
diff --git a/docs/under-the-hood/reasoning-controls.md b/docs/under-the-hood/reasoning-controls.md
index fc867a64b..7d1023827 100644
--- a/docs/under-the-hood/reasoning-controls.md
+++ b/docs/under-the-hood/reasoning-controls.md
@@ -244,7 +244,7 @@ Google models use either `thinking_mode = "manual"` (Gemini 2.5 series) or `thin
```toml
[cogt.llm_config.google_config.effort_to_level_map]
none = "disabled"
-minimal = "low"
+minimal = "minimal"
low = "low"
medium = "medium"
high = "high"
@@ -260,7 +260,7 @@ If the level map returns `"disabled"` (e.g., for `NONE` effort), thinking is dis
**ADAPTIVE mode** (Gemini 3) sends a `thinking_level` value (e.g., `ThinkingLevel.LOW`, `ThinkingLevel.MEDIUM`, `ThinkingLevel.HIGH`) mapped from the `effort_to_level_map`. The Google SDK dynamically adjusts reasoning depth based on this level. No `thinking_budget` is set in adaptive mode.
!!! note
- `MINIMAL` and `LOW` both map to `"low"` in the level map. In ADAPTIVE mode they both produce `ThinkingLevel.LOW`. In MANUAL mode they are differentiated by the budget map (512 vs 1024 tokens).
+ `MINIMAL` and `LOW` map to distinct levels in the level map (`"minimal"` and `"low"`). In ADAPTIVE mode they produce different results — `ThinkingLevel.MINIMAL` vs `ThinkingLevel.LOW`. In MANUAL mode they are further differentiated by the budget map (512 vs 1024 tokens).
**MANUAL mode** (Gemini 2.5) resolves effort to a `thinking_budget` (token count) via the `effort_to_budget_maps` config:
diff --git a/docs/under-the-hood/stuffartefact-and-image-rendering.md b/docs/under-the-hood/stuffartefact-and-image-rendering.md
index 5aafea607..7230fae86 100644
--- a/docs/under-the-hood/stuffartefact-and-image-rendering.md
+++ b/docs/under-the-hood/stuffartefact-and-image-rendering.md
@@ -48,7 +48,7 @@ StuffArtefact wraps Stuff → delegates to StuffContent → Protocol enables tra
| `ImageContent` | Self-registers, returns `[Image N]` |
| `ListContent` | Iterates items, recurses into nested |
| `TextAndImagesContent` | Renders text first, then images |
-| `StuffContent` (base) | Iterates Pydantic model fields recursively |
+| `StructuredContent` | Iterates model fields, recurses into nested lists/tuples/dicts |
| `StuffArtefact` | Delegates to underlying content |
---
@@ -64,7 +64,7 @@ flowchart TB
subgraph FILTER["with_images Filter"]
direction TB
CHECK["isinstance(value, ImageRenderable)?"]
- CALL["value.render_with_images(registry, text_format=format)"]
+ CALL["value.render_with_images(registry=registry, text_format=text_format)"]
CHECK --> CALL
end
@@ -77,7 +77,7 @@ flowchart TB
IMG["ImageContent: register + return token"]
LIST["ListContent: iterate items"]
TEXT["TextAndImagesContent: text then images"]
- BASE["StuffContent: iterate model_fields"]
+ STRUCT["StructuredContent: iterate model_fields"]
end
subgraph REGISTRY["ImageRegistry"]
@@ -88,7 +88,7 @@ flowchart TB
VAR --> CHECK
CALL --> DELEGATE
- DELEGATE --> IMG & LIST & TEXT & BASE
+ DELEGATE --> IMG & LIST & TEXT & STRUCT
IMG --> STORE
STORE --> DEDUP
```
@@ -155,8 +155,8 @@ class ImageRenderable(Protocol):
def render_with_images(
self,
- registry: ImageRegistry,
*,
+ registry: ImageRegistry,
text_format: TextFormat,
) -> str:
"""Render to string, registering images to the registry.
@@ -182,7 +182,7 @@ class ImageRenderable(Protocol):
Self-registers and returns a token:
```python
-def render_with_images(self, registry, *, text_format) -> str:
+def render_with_images(self, *, registry, text_format) -> str:
image_index = registry.register_image(self)
return f"[Image {image_index + 1}]"
```
@@ -192,13 +192,13 @@ def render_with_images(self, registry, *, text_format) -> str:
Iterates items, delegating to nested ImageRenderable objects:
```python
-def render_with_images(self, registry, *, text_format) -> str:
+def render_with_images(self, *, registry, text_format) -> str:
parts: list[str] = []
for item in self.items:
if isinstance(item, ImageRenderable):
- rendered = item.render_with_images(registry, text_format=text_format)
+ rendered = item.render_with_images(registry=registry, text_format=text_format)
else:
- rendered = str(item)
+ rendered = item.rendered_markdown()
if rendered:
parts.append(rendered)
return "\n".join(parts)
@@ -209,10 +209,10 @@ def render_with_images(self, registry, *, text_format) -> str:
Renders text first, then registers each image:
```python
-def render_with_images(self, registry, *, text_format) -> str:
+def render_with_images(self, *, registry, text_format) -> str:
parts: list[str] = []
if self.text:
- parts.append(self.text.rendered_plain())
+ parts.append(self.text.rendered_for_prompt(text_format=text_format))
if self.images:
for image in self.images:
image_index = registry.register_image(image)
@@ -220,26 +220,25 @@ def render_with_images(self, registry, *, text_format) -> str:
return "\n".join(parts)
```
-### StuffContent (Base)
+### StructuredContent
-Default implementation iterates Pydantic model fields:
+Default implementation for structured models iterates Pydantic model fields:
```python
-def render_with_images(self, registry, *, text_format) -> str:
+def render_with_images(self, *, registry, text_format) -> str:
parts: list[str] = []
for field_name in type(self).model_fields:
field_value = getattr(self, field_name)
if field_value is None:
continue
- if isinstance(field_value, ImageRenderable):
- rendered = field_value.render_with_images(registry, text_format=text_format)
- else:
- rendered = str(field_value)
+ rendered = self._render_value_with_images(field_value, registry=registry, text_format=text_format)
if rendered:
parts.append(f"{field_name}: {rendered}")
return "\n".join(parts)
```
+The `_render_value_with_images` helper recurses: a nested `ImageRenderable` gets `render_with_images(...)`, plain lists/tuples/dicts are traversed item by item, a non-renderable `StuffContent` falls back to `rendered_for_prompt(text_format=text_format)`, and anything else to `str(value)`. A plain `StuffContent` subclass that is not a `StructuredContent` does not implement the protocol.
+
---
## Image Numbering
@@ -284,15 +283,15 @@ def with_images(context: Context, value: Any, _: Any = None) -> str:
registry = ImageRegistry()
# 3. Get text format
- text_format = TextFormat(context.get(Jinja2ContextKey.TEXT_FORMAT))
+ text_format = TextFormat(context.get(Jinja2ContextKey.TEXT_FORMAT, default=TextFormat.PLAIN))
# 4. Protocol-based rendering
if isinstance(value, ImageRenderable):
- return value.render_with_images(registry, text_format=text_format)
+ return value.render_with_images(registry=registry, text_format=text_format)
# 5. Handle plain sequences
if isinstance(value, (list, tuple)):
- return _render_sequence_with_images(value, registry, text_format)
+ return _render_sequence_with_images(value, registry=registry, text_format=text_format)
# 6. Reject unsupported types
raise Jinja2ContextError(f"{type(value).__name__} does not implement ImageRenderable")
@@ -332,7 +331,7 @@ def with_images(context: Context, value: Any, _: Any = None) -> str:
| `pipelex/tools/jinja2/jinja2_with_images_filter.py` | `with_images` filter |
| `pipelex/tools/jinja2/jinja2_filters.py` | `tag` and `format` filters |
| `pipelex/tools/jinja2/image_registry.py` | Image tracking with deduplication |
-| `pipelex/core/stuffs/stuff_content.py` | Base `render_with_images()` |
+| `pipelex/core/stuffs/structured_content.py` | Field-iterating `render_with_images()` for structured models |
| `pipelex/core/stuffs/image_content.py` | Self-registering image token |
| `pipelex/core/stuffs/list_content.py` | List iteration for images |
| `pipelex/core/stuffs/text_and_images_content.py` | Text + images rendering |
diff --git a/docs/under-the-hood/test-profile-configuration.md b/docs/under-the-hood/test-profile-configuration.md
index 766be1bbb..685f793a1 100644
--- a/docs/under-the-hood/test-profile-configuration.md
+++ b/docs/under-the-hood/test-profile-configuration.md
@@ -181,7 +181,7 @@ extract_models = ["@from_pdf"]
The preprocessing command discovers available models from:
-- **Backend TOML files** in `pipelex/backends/*.toml`
+- **Backend TOML files** in `inference/backends/*.toml`, resolved from the project `.pipelex/` directory if present there, otherwise from the global `~/.pipelex/` directory
- **Pipelex Gateway** remote configuration
### 2. Profile Resolution
@@ -208,14 +208,17 @@ tests/integration/pipelex/fixtures/_generated_model_sets.py
This file contains:
```python
-LLM_MODEL_BACKEND_PAIRS: list[tuple[str, str]] = [
- ("claude-4.5-haiku", "pipelex_gateway"),
- ("gpt-4o-mini", "pipelex_gateway"),
- ...
+from tests.integration.pipelex.fixtures.model_combo import ModelCombo
+
+LLM_COMBOS: list[ModelCombo] = [
+ ModelCombo('claude-4.5-haiku', 'pipelex_gateway'),
+ ModelCombo('gemini-2.5-flash-lite', 'pipelex_gateway'),
+ ModelCombo('gpt-4o-mini', 'pipelex_gateway'),
]
-IMG_GEN_MODEL_BACKEND_PAIRS: list[tuple[str, str]] = [...]
-EXTRACT_MODEL_BACKEND_PAIRS: list[tuple[str, str]] = [...]
+IMG_GEN_COMBOS: list[ModelCombo] = [...]
+EXTRACT_COMBOS: list[ModelCombo] = [...]
+SEARCH_COMBOS: list[ModelCombo] = [...]
```
---