Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Cleanup: prune stale wip docs, dead config keys, and refresh error-model docs (#1009)
Louis Choquelc15894443ffe
diff --git a/.claude/settings.json b/.claude/settings.jsonindex 9eead5f6c..4dda9cc6f 100644--- a/.claude/settings.json+++ b/.claude/settings.json@@ -53,8 +53,5 @@]}]- },- "enabledPlugins": {- "temporal@temporal-marketplace": true}-}+}\ No newline at end of filediff --git a/.gitignore b/.gitignoreindex f767b4f8a..489f50b20 100644--- a/.gitignore+++ b/.gitignore@@ -62,7 +62,6 @@ settings.local.jsonstructures/.pipelex/storage.pipelex/traces-.pipelex/temporal-payload-store# Pipelex runtime trace artifacts (workflow ndjson)**/traces/diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.tomlindex 3126ec3dc..25d68f91d 100644--- a/.pipelex/pipelex.toml+++ b/.pipelex/pipelex.toml@@ -70,7 +70,7 @@ theme = "dark"palette = "dracula"####################################################################################################-# Tracing Config (event log for distributed tracing across Temporal workers)+# Tracing Config (event log for distributed tracing across workers)####################################################################################################[pipelex.tracing_config]@@ -84,10 +84,6 @@ traces_dir = ".pipelex/traces"table_name = ""region = ""-[pipelex.tracing_config.temporal_dynamodb]-table_name = ""-region = ""-##################################################################################################### Storage Config####################################################################################################diff --git a/AGENTS.md b/AGENTS.mdindex 0abde3faa..6b3cc8853 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -101,145 +101,194 @@## Coding Standards & Best Practices for Python Code-This document outlines the core coding standards, best practices, and quality control procedures for the codebase.-### Python Version Compatibility- - The project supports Python 3.10+ (`requires-python = ">=3.10,<3.15"`). Never use features introduced after Python 3.10 without a compatibility fallback.- - Common pitfalls:- - `datetime.UTC` was added in Python 3.11. Use `datetime.timezone.utc` instead.- - `StrEnum` was added in Python 3.11. Always import it from `pipelex.types` which handles retrocompatibility.- - `type` statement (PEP 695) was added in Python 3.12. Use `TypeAlias` from `typing` instead.- - `ExceptionGroup` / `except*` was added in Python 3.11. Avoid unless using the `exceptiongroup` backport.+- The project supports Python 3.10+ (`requires-python = ">=3.10,<3.15"`). Never use features introduced after Python 3.10 without a compatibility fallback.+- Common pitfalls:+ - `datetime.UTC` was added in Python 3.11. Use `datetime.timezone.utc` instead.+ - `StrEnum` was added in Python 3.11. Always import it from `pipelex.types` which handles retrocompatibility.+ - `type` statement (PEP 695) was added in Python 3.12. Use `TypeAlias` from `typing` instead.+ - `ExceptionGroup` / `except*` was added in Python 3.11. Avoid unless using the `exceptiongroup` backport.### Variables, loops and indexes- - Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.- - When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.- - Avoid inline for loops, unless it's ultra-simple and holds on oneline.- - If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.+- Variable names should have a minimum length of 3 characters. No exceptions: name your `for` loop indexes like `index_foobar`, your exceptions `exc` or more specific like `validation_error` when there are several layers of exceptions, and use `for key, value in ...` for key/value pairs.+- When looping on the keys of a dict, use `for key in the_dict` rather than `for key in the_dict.keys()` otherwise you won't pass linting.+- Avoid inline for loops, unless it's ultra-simple and holds on oneline.+- If you have a variable that will get its value differently through different code paths, declare it first with a type, e.g. `pipe_code: str` but DO NOT give it a default value like `pipe_code: str = ""` unless it's really justified. We want the variable to be unbound until all paths are covered, and the linters will help us avoid bugs this way.### Enums and tests- - When defining enums related to string values, always inherit from `StrEnum`- - When you need the enum value as a string, don't use `str(enum_var)` or `enum_var.value`, just use `enum_var` itself, that is the point of using StrEnum!- - Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add @property methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases- - As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.- - `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):- ```python- from pipelex.types import StrEnum- ```+- When defining enums related to string values, always inherit from `StrEnum`+- When you need the enum value as a string, don't use `str(enum_var)` or `enum_var.value`, just use `enum_var` itself, that is the point of using StrEnum!+- Never test equality to an enum value: use match/case, even to single out 1 case out of 10 cases. To avoid heavy match/case code in awkward places, add @property methods to the enum class such as `is_foobar()`. This is to avoid bugs: when new enum values are added we want the linter to complain. Use the `|` operator to group cases+- As our match/case constructs over enums are always exhaustive, NEVER add a default `case _: ...`. Otherwise, you won't pass linting.+- `StrEnum` must be imported from `pipelex.types` (handles python retrocompatibility):+ ```python+ from pipelex.types import StrEnum+ ```### Optionals- Don't write things like `a = b if b else c`, write `a = b or c` instead.+### Filesystem Paths++- **Use `pathlib.Path` for all filesystem paths.** Any function parameter, return type, or variable that holds a location of a file or directory on disk must be typed `Path`, not `str`. Convert to/from `str` only at genuine boundaries.+- **Boundaries where `str` is correct** (convert at the edge, then work with `Path` internally):+ - CLI argument parsing (typer/click options and arguments arrive as `str`).+ - Config / TOML (de)serialization — Pydantic config fields and `.toml` values are `str`.+ - Env-var parsing, e.g. splitting `PIPELEXPATH` on `os.pathsep`.+ - URIs / URLs — these are not filesystem paths.+ - `importlib.resources` package-resource traversables.+ - Third-party calls whose signature requires `str`.+- **Prefer `Path` idioms over `os.path`:** `base / "sub" / "file.txt"` (not `os.path.join`), `.parent` (not `os.path.dirname`), `.name` (not `os.path.basename`), `.suffix`, `.stem`, `.exists()`, `.is_dir()`, `.is_file()`, `.mkdir(parents=True, exist_ok=True)` (not `os.makedirs`), `.iterdir()` / `.rglob(...)` (not `os.listdir` / `os.walk`), `.read_text()` / `.read_bytes()` / `.write_text()` / `.write_bytes()` (not `open()`), `.unlink()`.+- **Watch the semantic shifts when porting `os.path` code — these are NOT drop-in equivalents:**+ - `os.path.relpath(p, base)` returns a relative path that may walk up with `..`; `Path.relative_to(base)` raises `ValueError` when `p` is not under `base`. Never swap one for the other blindly.+ - `os.path.abspath(p)` collapses `.`/`..` lexically but does **not** resolve symlinks. No single `Path` method matches it: `Path.resolve()` both normalizes `..` *and* follows symlinks, while `Path.absolute()` only prepends the cwd and normalizes neither (`Path("a/../b").absolute()` stays `<cwd>/a/../b`, whereas `os.path.abspath("a/../b")` is `<cwd>/b`). Reach for `Path.resolve()` in the common case; if you genuinely need abspath's no-symlink semantics, normalize explicitly (`Path(os.path.normpath(p.absolute()))`).+- **Logical (non-filesystem) "paths" stay `str`.** Dotted module/import paths, domain paths, and variable/key paths (`variable_path`, `dotted_path`, `var_path`, `key_path`, and similar) are identifiers, not locations on disk — keep them `str`. This rule is about files and directories, not everything named `*_path`.+### Imports-#### **Imports at the top of the file**+#### Imports at the top of the file- - Avoid as much as possible import statements outside of a module's top-level scope.- - Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting- - Do not bother with ordering the imports or removing unused imports, our Ruff linter will handle it for us.- - `if TYPE_CHECKING:` blocks must always be the **last** block in the imports section, placed after all regular imports.+- Avoid as much as possible import statements outside of a module's top-level scope.+- Do not import libraries in functions or classes unless in very specific cases, to be discussed with the user, as they would required a `# noqa: ...` comment to pass linting+- Do not bother with ordering the imports or removing unused imports, our Ruff linter will handle it for us.+- `if TYPE_CHECKING:` blocks must always be the **last** block in the imports section, placed after all regular imports.-#### **Removing unused imports**+#### Removing unused imports- - To remove unused imports, run `make fix-unused-imports` or `make fui` (shorthand). This is faster and cheaper than rewriting with LLM.+- To remove unused imports, run `make fix-unused-imports` or `make fui` (shorthand). This is faster and cheaper than rewriting with LLM.-#### **No re-exports in `__init__.py`**+#### No re-exports in `__init__.py`- - Do NOT fill `__init__.py` files with re-exports.- - Always use direct full-path imports everywhere. For example:+- Do NOT fill `__init__.py` files with re-exports.+- Always use direct full-path imports everywhere.-- **Logging and Pretty Printing**:+#### Logging and Pretty Printing- - Both `log()` and `pretty_print()` can be imported from `pipelex` directly:- ```python- from pipelex import log, pretty_print+- Both `log()` and `pretty_print()` can be imported from `pipelex` directly:+ ```python+ from pipelex import log, pretty_print- log.info("Hello, world!")- ```- - Both have a title arg which is handy when logging/printing objects:+ log.info("Hello, world!")+ ```+- Both have a title arg which is handy when logging/printing objects:+ ```python+ log.verbose("Hello, world!", title="Your first Pipelex log")+ pretty_print(output_object, title="Your first Pipelex output")+ ```+- Both handle formatting json using Rich, pretty_print makes it prettier.- ```python- log.verbose("Hello, world!", title="Your first Pipelex log")- pretty_print(output_object, title="Your first Pipelex output")- ```- - Both handle formatting json using Rich, pretty_print makes it prettier.+#### StrEnum and Self type-- **StrEnum and Self type**:-- - Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):- ```python- from pipelex.types import Self, StrEnum- ```+- Both `StrEnum` and `Self` must be imported from `pipelex.types` (handles python retrocompatibility):+ ```python+ from pipelex.types import Self, StrEnum+ ```### Typing-#### **Always Use Type Hints**-- - Every function parameter must be typed- - Every function return must be typed- - Use type hints for all variables where type is not obvious- - Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]- - Use type hints for all fields- - Use Field(default_factory=...) for mutable defaults- - Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.--#### **BaseModel / Pydantic Standards**-- - Use `BaseModel` and respect Pydantic v2 standards- - Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`- - Keep models focused and single-purpose- - For list fields with non-string items in BaseModels, use `empty_list_factory_of()` to avoid linter complaints:- ```python- from pydantic import BaseModel, Field- from pipelex.tools.typing.pydantic_utils import empty_list_factory_of-- class MyModel(BaseModel):- names: list[str] = Field(default_factory=list) # OK for strings- numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")- items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")- ```+#### Always Use Type Hints++- Every function parameter must be typed+- Every function return must be typed+- Use type hints for all variables where type is not obvious+- Use dict, list, tuple types with lowercase first letter: dict[], list[], tuple[]+- Use type hints for all fields+- Use Field(default_factory=...) for mutable defaults+- Use `# pyright: ignore[specificError]` or `# type: ignore` only as a last resort. In particular, if you are sure about the type, you often solve issues by using cast() or creating a new typed variable.++#### Data-holder shapes (NamedTuple, dataclass, BaseModel)++The deciding question is **does this value ever get serialized** — sent across a process or network boundary, persisted to disk or a database, or round-tripped through (de)serialization? If yes, it must be one of the two serialization-safe forms (a `BaseModel` or a pydantic dataclass): a `NamedTuple` or stdlib `@dataclass` loses its type on the way through and silently corrupts the data across the boundary. (A distributed executor like Temporal is the canonical case — its payloads cross the wire between worker processes — but the same applies to any cache, queue, API response, or on-disk record.) Stdlib `@dataclass` is banned outright; pick one of these four forms:++- **`NamedTuple`** (`from typing import NamedTuple`) — small fixed records replacing bare tuples, or any holder of a `@runtime_checkable` Protocol / arbitrary non-pydantic field (avoids `arbitrary_types_allowed`). Not serialization-safe, zero validation. Avoid when the value is returned in a union beside a real `tuple` — it would also satisfy `isinstance(x, tuple)` and blur the two arms; use a frozen `BaseModel` there instead.+- **pydantic dataclass** (`from pydantic.dataclasses import dataclass`) — internal records that are built and field-read but never serialized; the lowest-churn swap from a stdlib `@dataclass` (just change the decorator). Default to `frozen=True`; drop it only when the instance is genuinely mutated in place after construction.+- **frozen `BaseModel`** (`model_config = ConfigDict(frozen=True)`) — immutable value objects that need the full model API (`model_dump`, validators, discriminated unions), must sit in a uniformly-`BaseModel` layer, or must stay type-distinct from a `tuple` in a union.+- **mutable `BaseModel`** — objects genuinely mutated in place that also want the model API or layer consistency.++Never reach for `TypeAdapter` — needing it means the shape was chosen wrong. And don't convert an existing serialized `BaseModel` to a dataclass just because dataclasses are serialization-safe too: a model that relies on discriminated unions or validators must stay a `BaseModel`. When a serialized dataclass/model field references a type from an optional dependency, alias it to `Any` at runtime so it validates as `Any` without importing the extra (e.g. `if TYPE_CHECKING: from temporalio.common import RetryPolicy` / `else: RetryPolicy = Any`) — a bare forward ref instead raises `PydanticUserError` on construction.++Once you've landed on a `BaseModel` (the default for anything that serializes), respect Pydantic v2 standards:++- Use the modern `ConfigDict` when needed, e.g. `model_config = ConfigDict(extra="forbid", strict=True)`+- Keep models focused and single-purpose+- For list fields with non-string items, use `empty_list_factory_of()` to avoid linter complaints:+ ```python+ from pydantic import BaseModel, Field+ from pipelex.tools.typing.pydantic_utils import empty_list_factory_of++ class MyModel(BaseModel):+ names: list[str] = Field(default_factory=list) # OK for strings+ numbers: list[int] = Field(default_factory=empty_list_factory_of(int), description="A list of numbers")+ items: list[MyItem] = Field(default_factory=empty_list_factory_of(MyItem), description="A list of items")+ ```++### Keyword-only arguments++Across `pipelex/` source, non-subject function parameters must be **keyword-only**, so call sites are self-documenting: `do_thing(retries=3, timeout=30)` over the opaque `do_thing(3, 30)`. The compliant shape places a bare `*` after the subject:++- `def f(subject, *, opt1, opt2): ...` — the first non-`self`/`cls` parameter (the subject) may stay positional; everything after it must be keyword-only. Making the subject keyword-only too (`def f(*, a, b)`, `def m(self, *, a, b)`) is always allowed and is the form the auto-fixer produces.+- A lone subject (`def render(node)`) is compliant; a second bare positional (`def f(a, b)`, `def truncate(text, max_length=80)`) is a violation.++The rule is mechanically enforced by the `check-keyword-only` AST guard (`make cko`), which runs in `make agent-check`, in the `make check` aggregate, in CI, and via a Claude Code `PostToolUse` hook that checks each edited file at edit time. The tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs — dunders, pydantic validators/serializers, Typer/pytest/Jinja2 framework entrypoints, `@override` impls, and a small curated symmetric-tuple allowlist — are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line).++**Auto-fix.** `make fix-keyword-only` (alias `fko`, or `pipelex-dev check-keyword-only --fix`) rewrites every mechanically-fixable violation by inserting a bare `*` as far left as possible — right after `self`/`cls` — so every non-`self`/`cls` parameter becomes keyword-only, and re-parses each rewrite before writing (a rewrite that would not parse is discarded and reported instead). It runs automatically early in `make agent-check`, before `ruff format`. It is **non-gating**: it mutates and reports but never exits non-zero — the read-only `check-keyword-only` (run last in `agent-check`, and the variant `make check` / CI use) owns the pass/fail gate. A few shapes a single `*` insert cannot fix are reported for a manual fix instead of rewritten: a `*args` is present, a keyword-only section already exists, or two-or-more positional-only params remain.++Auto-fix is not a substitute for review: the guard is blind to functions a framework or the interpreter invokes positionally (callbacks, `__import__` hooks, route handlers, Jinja2 filters) — pyright/mypy will pass a wrongly-keywordized callback, so `make agent-test` is the safety net after a bulk fix.### Factory Pattern- - Use factory pattern for object creation when dealing with multiple implementations- - Our factory methods are named `make_from_...` and such+- Use factory pattern for object creation when dealing with multiple implementations+- Our factory methods are named `make_from_...` and such### Protocols and Interfaces- - When a getter (e.g. `get_report_delegate()`, `get_library_manager()`) returns a `Protocol` type, callers must only rely on methods declared on that `Protocol`. If you need to call a method that lives on a concrete implementation, **extend the `Protocol`** — do not work around it with `isinstance(...)` checks, `cast(...)`, or inline imports of the concrete class.- - When extending the `Protocol`, also update every implementation (including no-op / null implementations) so structural typing is satisfied. For implementations where the method has nothing to do, give it a `pass` body — that is the whole point of having a `Protocol` rather than a concrete base class.- - Smell to watch for: an inline `from ... import ConcreteImpl # noqa: PLC0415` followed by `if isinstance(delegate, ConcreteImpl): delegate.some_method(...)`. This is almost always a missing method on the `Protocol`. Promote it.- - Use `@override` (from `typing_extensions`) on every implementation of a `Protocol` method, so that signature drift between the `Protocol` and its implementations is caught by the type checker.+- When a getter (e.g. `get_report_delegate()`, `get_library_manager()`) returns a `Protocol` type, callers must only rely on methods declared on that `Protocol`. If you need to call a method that lives on a concrete implementation, **extend the `Protocol`** — do not work around it with `isinstance(...)` checks, `cast(...)`, or inline imports of the concrete class.+- When extending the `Protocol`, also update every implementation (including no-op / null implementations) so structural typing is satisfied. For implementations where the method has nothing to do, give it a `pass` body — that is the whole point of having a `Protocol` rather than a concrete base class.+- Smell to watch for: an inline `from ... import ConcreteImpl # noqa: PLC0415` followed by `if isinstance(delegate, ConcreteImpl): delegate.some_method(...)`. This is almost always a missing method on the `Protocol`. Promote it.+- Use `@override` (from `typing_extensions`) on every implementation of a `Protocol` method, so that signature drift between the `Protocol` and its implementations is caught by the type checker.### Error Handling- - Always catch exceptions at the place where you can add useful context to it.- - Use try/except blocks with specific exceptions- - Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError- - **NEVER write `except Exception:` (or bare `except:`).** Catching the generic `Exception` is FORBIDDEN except in two cases, each of which REQUIRES a one-line comment directly under the `except` naming which case applies: **(1)** the absolute root of a CLI command or an API endpoint handler; **(2)** the wrapped call invokes genuinely unbounded code whose exception surface cannot be enumerated — a user-registered callback or function, dynamic plugin dispatch, or a third-party SDK with no documented exception types (this is only for genuinely open-ended surfaces; "I didn't look the exceptions up" does NOT qualify). There are NO other exceptions to this rule. The following are NOT valid justifications and the agent must reject them when tempted:- - "I want to log and re-raise" → just let it propagate; logging happens at the root.- - "I want to swallow it just in case" → don't. Unknown failures must surface.- - "I want to convert it to our error type" → only if you know which specific exception(s) the call can raise; catch those explicitly. Catching a project base class (`PipelexError`) or a package base (`CogtError`, `ToolError`) is acceptable when the operation legitimately fails in many domain-specific ways — that is still specific. Catching `Exception` is not.- - "The third-party lib isn't well typed" → look up the actual exceptions it raises and catch those by name.- - "It's safer this way" → it isn't. It hides bugs. Crashing loudly on an unexpected exception is the desired behavior.- If you find yourself writing `except Exception` outside the two cases above, STOP and ask the user before proceeding.- - **Do NOT add try/except speculatively.** Only add a `try`/`except` when (a) a specific exception type is documented or known to be raised by the call, AND (b) you have a concrete recovery path or a meaningful context to attach. Otherwise, let the error bubble up — that is the correct behavior, not a gap to fill.- - **Use `try`/`finally` (without `except`) for required cleanup.** When a piece of code MUST run on the way out — releasing a resource, closing a tracer, tearing down a library, clearing per-context state on a manager — put it in a `finally` block (or a `with`/context manager). This is the correct way to guarantee cleanup on both the happy path and the error path: it does NOT suppress the exception, it just ensures cleanup happens before the exception propagates. Do not invent an `except Exception` just to attach cleanup; use `finally` instead. Conversely, do not move cleanup into the success path "to keep things simple" — if it must run on errors too, it belongs in `finally`.- - When reviewing existing code, treat any `except Exception` you encounter as a bug to fix (replace it with the specific exception(s) actually raised, or remove the try/except entirely), not as a precedent to follow.- - Always add `from exc` to the exception raise statements- - Always write the error message as a variable before raising it, for cleaner error traces-- ```python- try:- self.models_manager.setup()- except RoutingProfileLibraryNotFoundError as exc:- msg = "The routing library could not be found, please call `pipelex init config` to create it"- raise PipelexSetupError(msg) from exc- ```+- Always catch exceptions at the place where you can add useful context to it.+- Use try/except blocks with specific exceptions+- Convert third-party exceptions to our custom ones except in pydantic validators where you can raise a ValueError or a TypeError+- **NEVER write `except Exception:` (or bare `except:`).** Catching the generic `Exception` is FORBIDDEN except in two cases, each of which REQUIRES a one-line comment directly under the `except` naming which case applies: **(1)** the absolute root of a CLI command or an API endpoint handler; **(2)** the wrapped call invokes genuinely unbounded code whose exception surface cannot be enumerated — a user-registered callback or function, dynamic plugin dispatch, or a third-party SDK with no documented exception types (this is only for genuinely open-ended surfaces; "I didn't look the exceptions up" does NOT qualify). There are NO other exceptions to this rule. The following are NOT valid justifications and the agent must reject them when tempted:+ - "I want to log and re-raise" → just let it propagate; logging happens at the root.+ - "I want to swallow it just in case" → don't. Unknown failures must surface.+ - "I want to convert it to our error type" → only if you know which specific exception(s) the call can raise; catch those explicitly. Catching a project base class (`PipelexError`) or a package base (`CogtError`, `ToolError`) is acceptable when the operation legitimately fails in many domain-specific ways — that is still specific. Catching `Exception` is not.+ - "The third-party lib isn't well typed" → look up the actual exceptions it raises and catch those by name.+ - "It's safer this way" → it isn't. It hides bugs. Crashing loudly on an unexpected exception is the desired behavior.+ If you find yourself writing `except Exception` outside the two cases above, STOP and ask the user before proceeding.+- **Do NOT add try/except speculatively.** Only add a `try`/`except` when (a) a specific exception type is documented or known to be raised by the call, AND (b) you have a concrete recovery path or a meaningful context to attach. Otherwise, let the error bubble up — that is the correct behavior, not a gap to fill.+- **Use `try`/`finally` (without `except`) for required cleanup.** When a piece of code MUST run on the way out — releasing a resource, closing a tracer, tearing down a library, clearing per-context state on a manager — put it in a `finally` block (or a `with`/context manager). This is the correct way to guarantee cleanup on both the happy path and the error path: it does NOT suppress the exception, it just ensures cleanup happens before the exception propagates. Do not invent an `except Exception` just to attach cleanup; use `finally` instead. Conversely, do not move cleanup into the success path "to keep things simple" — if it must run on errors too, it belongs in `finally`.+- When reviewing existing code, treat any `except Exception` you encounter as a bug to fix (replace it with the specific exception(s) actually raised, or remove the try/except entirely), not as a precedent to follow.+- **`from exc` on raise statements — default yes.** `raise NewError(msg) from exc` whenever you raise a new exception while handling another. The point is preserving the cause for debugging tracebacks.++ Carve-out — translation helpers. When the actual `raise` lives one frame down inside a translation helper (`raise_validation_error(message=str(exc))`, an API 4xx shaper, anything `(msg: str) -> NoReturn` whose job is shaping the outgoing response), Python's implicit `__context__` chaining already preserves the original — the helper's `raise` runs while your `except` frame is still active, structured logs walk the chain either way, and only the traceback intro line differs ("During handling of the above exception" vs "The above exception was the direct cause"). Don't bolt a `cause=exc` parameter onto every such helper just to swap implicit `__context__` for explicit `__cause__`. Revisit only if a helper grows an internal try/except that would break the implicit chain, or if a downstream tool (Sentry-style) is wired to distinguish the two.++- Always write the error message as a variable before raising it, for cleaner error traces++ ```python+ try:+ self.models_manager.setup()+ except RoutingProfileLibraryNotFoundError as exc:+ msg = "The routing library could not be found, please call `pipelex init config` to create it"+ raise PipelexSetupError(msg) from exc+ ```++### Error class location++Put every custom error class in a module named `exceptions.py` (default — one per package directory) or `<topic>_exceptions.py` (when a directory needs several separate-concern error modules); the base error class gets its own module. This is a discoverability contract, not style: tooling that enumerates errors (doc generation, URI/uniqueness checks) finds them by importing these well-known module names, so an error defined in a sibling non-error module (worker, factory, parser) may never be imported and silently escapes that tooling.++- **Default:** append to the package's existing `exceptions.py`.+- **Topical split:** when appending would cause a circular import, or a package hosts genuinely separate error families, use `<topic>_exceptions.py` importing only the base error class.+- **Forbidden:** `*_errors.py` (synonym, drift magnet), or defining an error alongside non-error code.++Use one casing per acronym (`LLMError` vs `LlmError` collide when kebab-cased into filenames/URIs). Enforce the convention with a test in CI.### Documentation@@ -247,11 +296,11 @@ This document outlines the core coding standards, best practices, and quality co```pythondef process_image(image_path: str, size: tuple[int, int]) -> bytes:"""Process and resize an image.-+Args:image_path: Path to the source imagesize: Tuple of (width, height) for resizing-+Returns:Processed image as bytes"""@@ -262,7 +311,7 @@ This document outlines the core coding standards, best practices, and quality co```pythonclass ImageProcessor:"""Handles image processing operations.-+Provides methods for resizing, converting, and optimizing images."""```@@ -293,11 +342,11 @@ Non-subject function parameters across `pipelex/` source must be **keyword-only*- `def f(subject, *, opt1, opt2): ...` — the first non-`self`/`cls` parameter (the subject) may stay positional; everything after it must be keyword-only. Making the subject keyword-only too (`def f(*, a, b)`) is always allowed and often preferable.- A lone subject (`def render(node)`) is compliant; a second bare positional (`def f(a, b)`, `def truncate(text, max_length=80)`) is a violation.-The rule is mechanically enforced by the `check-keyword-only` AST guard, which runs in `make agent-check`, in the `make check` aggregate, and in CI; the tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs (dunders, pydantic validators/serializers, Typer/Temporal/pytest/Jinja2 framework entrypoints, `@override` impls) are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line). Watch for functions a framework or the interpreter invokes positionally (callbacks, `__import__` hooks, route handlers): the type checker is blind to those, so `make agent-test` is the safety net.+The rule is mechanically enforced by the `check-keyword-only` AST guard, which runs in `make agent-check`, in the `make check` aggregate, and in CI; the tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs (dunders, pydantic validators/serializers, Typer/pytest/Jinja2 framework entrypoints, `@override` impls) are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line). Watch for functions a framework or the interpreter invokes positionally (callbacks, `__import__` hooks, route handlers): the type checker is blind to those, so `make agent-test` is the safety net.The full specification — the two exceptions, the carve-out list, the symmetric-tuple allowlist, the escape hatch, and worked examples — is in [`docs/contribute/keyword-only-arguments.md`](docs/contribute/keyword-only-arguments.md).-## Writing tests+## Writing Tests### Unit test generalities@@ -328,7 +377,7 @@ Apply the appropriate markers:- "inference: uses either an LLM or an image generation AI"- never add "@pytest.mark.dry_runnable" if you haven't set the "inference" marker-Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference`and `llm`.+Several markers may be applied. For instance, if the test uses an LLM, then it uses inference, so you must mark with both `inference` and `llm`.#### Important rulesdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex e8e63cc7c..32e9a32b8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,7 +14,7 @@### Changed-- **Orchestration & Delivery split (pre-1.0 breaking, plugin SPI):** The flat execution-mode axis is split into an open `orchestration_mode` token (which orchestrator runs the job — `direct` in core, `temporal`/`mistralai-workflows` from plugins) and a closed `DeliveryMode` enum (the wait-semantics axis — `BLOCKING` / `FIRE_AND_FORGET`). The orchestrator plugin contract changes accordingly: `OrchestratorProtocol.run` now takes a required `delivery: DeliveryMode` keyword, an orchestrator must declare a `supports_fire_and_forget: bool` capability, and the registries are re-keyed by the open `orchestration_mode` token (the former `PipelexExecutionMode` enum is removed). A plugin built against the prior contract is incompatible and must be rebuilt. `PipelexPipeRunInput` now carries `orchestration_mode` + `delivery`.+- **Orchestration & Delivery split (pre-1.0 breaking, plugin SPI):** The flat execution-mode axis is split into an open `orchestration_mode` token (which orchestrator runs the job — `direct` in core, other modes supplied by plugins) and a closed `DeliveryMode` enum (the wait-semantics axis — `BLOCKING` / `FIRE_AND_FORGET`). The orchestrator plugin contract changes accordingly: `OrchestratorProtocol.run` now takes a required `delivery: DeliveryMode` keyword, an orchestrator must declare a `supports_fire_and_forget: bool` capability, and the registries are re-keyed by the open `orchestration_mode` token (the former `PipelexExecutionMode` enum is removed). A plugin built against the prior contract is incompatible and must be rebuilt. `PipelexPipeRunInput` now carries `orchestration_mode` + `delivery`.- **Honest fire-and-forget rejection:** `/start` now rejects a fire-and-forget request when the resolved orchestrator cannot do genuine async (e.g. the in-process `direct` mode), returning a 4xx instead of silently running to completion and falsely acking.- **`dry_run_pipeline` return shape:** `dry_run_pipeline` now returns the fully resolved, domain-qualified pipe reference alongside the `GraphSpec`.@@ -82,8 +82,6 @@- **`is_valid` on the canonical validation report**: `PipelexValidationReport` gains `is_valid: Literal[True] = True`, the always-true discriminant of the valid arm of the hosted `/validate` response union (mirrored by pipelex-api's `InvalidReport`'s `Literal[False]`). It sits beside `is_runnable`: a sound bundle may still be not-yet-runnable.- **Offline CLI unit tests**: Coverage for the pipelex-internal logic behind the CLI commands — `doctor` diagnostics, `run` execution and its sync wrapper, the `build` codegen cores, the readiness gate, `show`/`which`, and the gateway/telemetry/signature error handlers (the spec'd CLI interface stays owned by the conformance suite).- **Offline inference & runtime unit tests**: Coverage for layers that can break without a provider — structured-output↔instructor mode mapping, model-deck reference checks, the image-gen argument and worker-routing factories, gateway request-shaping and extract parsing, the Mistral factory, the local observer sink, the output renderer, builder spec validation / `to_blueprint()`, the pipeline runner's error paths, the TOML config-sync engine, and the storage config validators.-- **Offline Temporal entry-point unit tests**: Coverage for operational surfaces that previously only failed inside a live cluster — the worker CLI, the codec HTTP server (encode/decode, CORS, error→status mapping), and the server connection logic (API-key resolution, TLS, codec selection).-### Changed- **[BREAKING] Keyword-only public API**: Top-level public surfaces now require keyword arguments after the first parameter. A caller passing a second-or-later argument positionally must switch to keyword form — affected methods include `Pipelex.make(integration_mode, *, ...)`, `Pipelex.setup(integration_mode, *, ...)`, and `PipelexHub.setup_config(config_cls, *, ...)`. Downstream consumers to check: `pipelex-api`, `pipelex-worker`, `n8n-nodes-pipelex`, cookbook, starter.@@ -91,8 +89,7 @@- **Signature subject corrections**: Reordered parameters so the true semantic subject is the first positional argument, e.g. `parse_pipe_spec(spec_data, *, pipe_type)`, `hydrate_content(raw_content, *, concept)`, and `write_manifest(manifest, *, deck_dir)`. `parse_pipe_spec` is consumed by the hosted runner API (`pipelex-api`), updated in lockstep.- **[BREAKING] Canonical protocol validation report**: `PipelexValidationReport` is reworked into the canonical typed shape shared by every backend and moved to `pipelex/pipeline/validation_report.py` — a typed `bundle_blueprint` (replacing the untyped single-or-list `blueprint` dump), `pipe_io_contracts` keyed by namespaced `pipe_ref` (new `pipelex/pipeline/pipe_io_contracts.py`), `validated_pipes`, and the `pending_signatures` / `is_runnable` verdict. It is assembled in exactly one place (`build_validation_report`) with a single shared primary-blueprint rule (`select_primary_blueprint`).- **[BREAKING] `validated_pipes` key `pipe_code` → `pipe_ref`**: the entry has always carried the namespaced `pipe_ref` (`domain.code`) under a key named for the wrong identity; the key now says what the value is.-- **Protocol `validate` now produces a best-effort `graph_spec`**: when the batch declares a `main_pipe`, the local runtime dry-runs it in-process and ships the resulting `GraphSpec`, via one shared implementation (`best_effort_graph_spec`) the Temporal validate activity also calls; a graph-arm domain failure degrades to `graph_spec=None` with validation still successful. (`dry_run_pipe_in_process` moved to `pipelex/pipe_run/dry_run_in_process.py` to break an import cycle.)-- **[BREAKING] Temporal validate activity carries the full worker-side artifact set**: `DryValidateResult` gains required `pending_signatures` and `pipe_io_contracts` fields computed worker-side, so a version-skewed worker fails loudly at deserialization instead of defaulting to a silently wrong verdict. (Temporal validate has never shipped to prod.)+- **Protocol `validate` now produces a best-effort `graph_spec`**: when the batch declares a `main_pipe`, the local runtime dry-runs it in-process and ships the resulting `GraphSpec` via one shared implementation (`best_effort_graph_spec`); a graph-arm domain failure degrades to `graph_spec=None` with validation still successful. (`dry_run_pipe_in_process` moved to `pipelex/pipe_run/dry_run_in_process.py` to break an import cycle.)- **[BREAKING] `MTHDS_PROTOCOL_VERSION` deleted**: the hardcoded duplicate is gone from `runner.py`; the SDK's `PROTOCOL_VERSION` (`mthds.protocol.protocol`) is the single source of truth, imported directly by consumers including the hosted API.- **Additive multi-file library construction**: Same-domain `.mthds` libraries can now be built as separate, additive files (forward-declared `PipeSignature` headers plus concrete definitions in sibling files), enabling parallel top-down construction. A `PipeSignature` and a same-code concrete pipe reconcile (contracts compared by normalized concept identity) instead of colliding; concept and qualified pipe references resolve against the merged library across sibling and separately-loaded files; domain `description` / `system_prompt` merge order-independently; and a successful `validate` reports library-wide `pending_signatures` plus an `is_runnable` verdict in both JSON and markdown.- **[BREAKING] `PipeSignature` evicted from the executable pipe taxonomy**: a signature is a contract, not a way of running, so `PipeType.PIPE_SIGNATURE` and `PipeCategory.PIPE_SIGNATURE` are removed (any code matching on them must drop that arm). It stays a `PipeAbstract` subclass with `pipe_category = None`, and `is_signature` is now a class fact rather than an enum read.@@ -102,8 +99,8 @@### Fixed- **Protocol model deck no longer loses aliases to cross-category collisions**: `PipelexModelDeck.aliases` / `waterfalls` are now keyed by model category (`{category: {alias: model}}`) instead of being flattened with `update()`, which silently kept only the last category's entry for an alias name shared across categories. Breaking for the deck's extension shape.-- **JSON-Schema rendering failures on the validate surfaces are now structured errors**: a pydantic schema-generation failure while building `pipe_io_contracts` now raises the new non-retryable `PipeIOContractError` (naming the offending pipe/input/concept) instead of a raw exception that the Temporal boundary would retry forever.-- **Protocol `validate` teardown can no longer mask the real error**: a raising library teardown is suppressed while a body error is propagating, and the validation library id is captured once so the graph arm and the teardown always target the same library (mirrors the Temporal validate activity's guards).+- **JSON-Schema rendering failures on the validate surfaces are now structured errors**: a pydantic schema-generation failure while building `pipe_io_contracts` now raises the new `PipeIOContractError` (naming the offending pipe/input/concept) instead of a raw exception.+- **Protocol `validate` teardown can no longer mask the real error**: a raising library teardown is suppressed while a body error is propagating, and the validation library id is captured once so the graph arm and the teardown always target the same library.- **Empty `mthds_contents` is rejected with a structured error**: `validate_bundle` and `select_primary_blueprint` now raise `ValidateBundleError` instead of crashing primary-blueprint selection with a raw `IndexError`; the hand-rolled first-declaring-`main_pipe` loops in `execution_seams`, `dry_run_pipeline`, and `inputs_ops` are folded into the single `select_primary_blueprint`.- **Structured-info invariant is now total — parse-level failures carry a structured item**: a malformed `.mthds` file (a TOML-syntax error, an empty blueprint, or a bundle-elaborator failure) is raised with only a message and no categorized data, so an invalid verdict previously rode an **empty** `validation_errors[]` for the single most common failure mode. The shared builder (`build_validation_error_items`) gains a last-resort `fallback_message` residual: when no categorized error and no `dry_run` residual produced an item, it emits one `blueprint_validation` item carrying the message (no `source`, no `error_type` — the bundle could not become a blueprint at all). Both surfaces — `ValidateBundleError.to_error_report()` and the agent CLI's `extract_validation_errors()` — pass it, so every invalid verdict now carries a non-empty `validation_errors[]`, never a bare message.- **Qualified same-domain pipe references resolve across files**: a controller referencing a sibling-file pipe by qualified name (`research.find_key_findings`) is now deferred to the merged library like bare references; a reference no file declares is still rejected at load.@@ -114,7 +111,6 @@- **Mistral chat requests now send the system message before the user message**: `make_simple_messages` previously appended it after the user message, contradicting its own docstring and the OpenAI-typed sibling.- **String concept values in bundle specs are constructible again**: a bare string in the `ConceptSpec | str` union now validates (no longer crashing the `mode="before"` validator) and passes through to the blueprint as the concept's description instead of into `structure`, where the loader rejected it.- **LocalObserver JSONL records always carry the true lifecycle event name**: a payload's own `event_type` key can no longer overwrite the event name in the written record, which had broken event-type filtering for JSONL consumers.-- **Temporal worker startup no longer logs a misleading project name**: starting for the current project (no explicit argument) now omits the name instead of logging `'None'`.- **Tests under `tests/**/build/` are no longer silently skipped**: pytest's default `norecursedirs` includes `build`, so the whole `tests/unit/pipelex/cli/commands/build/` tree was never collected; the config now overrides `norecursedirs` and sets `testpaths = ["tests"]`.- **`Anything`-output option numbering is now deterministic**: the output renderer sorts a `PipeCondition`'s possible outputs by pipe code instead of iterating a set, so `output_option_N` / `schema_option_N` numbering is stable across runs.- **Documentation examples**: Updated code snippets in `docs/under-the-hood/` to match the new keyword-only signatures.@@ -134,12 +130,9 @@### Changed-- **`--dry-run` now honors the configured backend — the mock moved from the operators to the cogt leaf.** `run_mode` rides a new `CogtRunParams` carrier (derived from `PipeRunParams.run_mode`, stamped on every cogt assignment, serialized across the Temporal wire), so a DRY run on a Temporal backend dispatches the REAL `act_llm_gen_*` / extract / img-gen / search / render activities and the leaf mocks **inside** the activity — testing dispatch, scheduling, serialization, routing, and cross-worker propagation at zero AI cost, with no API keys and no storage IO (the img/extract DRY branches sit above the store step). `ContentGeneratorDry` is deleted: operators no longer swap generators (the base operator's dry path simply reuses the live path), and the in-process validation scopes pass an inline `ContentGenerator`. Object mocks are now schema-built on every backend — identical mock on direct and Temporal; classes with exotic format constraints should declare `examples`/`mock_format` (a re-validation failure surfaces as a typed `DryRunObjectFidelityError`, and a deterministic mock-build failure as the new non-retryable `DryRunMockBuildError`). The acceptance gate is Tier 17 of the `temporal-e2e-validate` skill (GREEN on all leaf families + keyless worker, RED-proven) plus a Mode-1 pytest.-- **Unified dry run — `--mock-inference` removed (breaking).** There is now exactly one non-live run mode: `--dry-run`. The mock-inference mode is retired; its capability — non-zero synthetic usage so the cost report renders, used to validate cross-worker cost-report assembly cheaply — survives as an **internal** `is_mock_usage` sub-flag of DRY on `CogtRunParams` (replacing `is_mock_inference`; setting it on a LIVE run is a validation error). It is exposed on the Python surface (`PipelexMTHDSProtocol` / `execute` / `prepare_pipe_job` via `is_mock_usage=...`) and as a hidden test-only CLI trigger, deliberately undocumented. Dry-run leaf coverage was already uniform across every operator, so `MockInferenceUnsupportedError` and its img-gen/extract/search guards are deleted; `MockInferenceObjectFidelityError` is renamed `DryRunObjectFidelityError` (mock-built objects are DRY-only now); the `mock_inference` usage sentinel is renamed `mock_usage`.-- **Keyless boot (`needs_inference=False`) now forces every run to DRY instead of installing a mock generator.** Generator selection is purely backend-keyed (a keyless Temporal submitter still dispatches and mocks inside activities); the forced-DRY flag is consumed at `PipeRunParamsFactory.make_run_params` — the single writer of `run_mode` — so every execution entry point is covered, including the runtime bridge.-- **Temporal runs without graph events skip the guaranteed-empty trace-flush activity.** Post-fix, the workflow's in-memory trace buffer is populated only by inline (workflow-thread) emissions — and with the unified dry run mocking inside the leaf activities, the only inline source left is graph events when graph tracing is on. Every usage emission (LIVE, DRY, or mock-usage) happens activity-side through the per-process event log, so scheduling `act_flush_trace_events` with an empty payload was a wasted activity round-trip per workflow in the fan-out (history events, task-queue dispatch, worker slot — including every child workflow spawned by controllers). The schedule is now gated payload-purely on `emit_graph_events`, so the decision replays identically and the command stream stays a pure function of the payload.-- **The "runner-side usage emission engaged" log is now INFO and states the actual contract.** Since activity-side usage emissions always go through the per-process activity event log (by design, co-located or split deployment alike), the old per-process WARNING — which claimed the fallback was specific to split deployments — fired on every standard deployment with a false explanation. Renamed to `log_once_runner_fallback_engaged`, logged once per process at INFO with the writer id.-+- **`--dry-run` now mocks at the cogt leaf instead of swapping out the operators.** `run_mode` rides a new `CogtRunParams` carrier (derived from `PipeRunParams.run_mode`, stamped on every cogt assignment), so a DRY run mocks **inside** the inference leaf at zero AI cost, with no API keys and no storage IO (the img/extract DRY branches sit above the store step). `ContentGeneratorDry` is deleted: operators no longer swap generators (the base operator's dry path simply reuses the live path), and the in-process validation scopes pass an inline `ContentGenerator`. Object mocks are now schema-built; classes with exotic format constraints should declare `examples`/`mock_format` (a re-validation failure surfaces as a typed `DryRunObjectFidelityError`, and a deterministic mock-build failure as the new `DryRunMockBuildError`).+- **Unified dry run — `--mock-inference` removed (breaking).** There is now exactly one non-live run mode: `--dry-run`. The mock-inference mode is retired; its capability — non-zero synthetic usage so the cost report renders — survives as an **internal** `is_mock_usage` sub-flag of DRY on `CogtRunParams` (replacing `is_mock_inference`; setting it on a LIVE run is a validation error). It is exposed on the Python surface (`PipelexMTHDSProtocol` / `execute` / `prepare_pipe_job` via `is_mock_usage=...`) and as a hidden test-only CLI trigger, deliberately undocumented. Dry-run leaf coverage was already uniform across every operator, so `MockInferenceUnsupportedError` and its img-gen/extract/search guards are deleted; `MockInferenceObjectFidelityError` is renamed `DryRunObjectFidelityError` (mock-built objects are DRY-only now); the `mock_inference` usage sentinel is renamed `mock_usage`.+- **Keyless boot (`needs_inference=False`) now forces every run to DRY instead of installing a mock generator.** Generator selection is purely backend-keyed; the forced-DRY flag is consumed at `PipeRunParamsFactory.make_run_params` — the single writer of `run_mode` — so every execution entry point is covered, including the runtime bridge.### Removed- **`dry_run_config.apply_to_jinja2_rendering`** config key: it was dead — PipeCompose renders templates directly and the jinja2 parse check survives in the templating leaf's DRY branch. Remove it from your `.pipelex/pipelex.toml` override if present (config is strict and will reject the unknown key).@@ -147,32 +140,14 @@### Fixed- **Resubmitting a `pipeline_run_id` after its run finished no longer fails permanently.** `PipelineManager` was the one per-run registry with no per-run removal: every run permanently stranded its key, so a second submission of the same `pipeline_run_id` against a long-lived server process raised `PipelineManagerAlreadyExistsError` — surfacing as an unrecoverable 500 from the hosted runner API until process restart. The registry entry is now freed on every exit path: `execute`'s `finally` removes it after each run (success or failure), and `pipeline_run_setup` removes its own registration when setup fails after registering (the caller never learns the id on that path, so setup must self-clean). Serial resubmission of a completed/failed run id now succeeds; only genuinely concurrent same-id runs still collide — deliberately, because the collision raise fires before `open_tracer` and shields the live direct-mode tracer (keyed by the caller-suppliable `pipeline_run_id`) from `open_tracer`'s stale-key pop-and-replace healing. The hosted API maps the remaining concurrent-duplicate case to 409 Conflict (pipelex-api change).-- **The temporalio extra is no longer imported on every pipelex boot.** `reporting_manager.py` (imported by `pipelex.pipelex` on every `Pipelex.make()` / CLI invocation) eagerly imported `temporalio.activity` at module level to detect in-activity usage emissions — putting the entire temporalio import (Rust bridge, protobuf, ~130ms measured) on every boot's critical path wherever `pipelex[temporal]` is installed, including processes that never touch Temporal, and silently breaking the runtime bridge's documented lazy-import contract. The gate now sniffs `sys.modules` instead: a process that never imported `temporalio.activity` cannot be inside a Temporal activity (the activity context is set by temporalio's own machinery, which requires the module). Worker processes are unaffected. A subprocess boot-cost guard pins the lazy-import contract.-- **Per-run Temporal worker-local state is keyed by run id, not workflow id.** The per-workflow library (`wf_{run_id}`), the graph tracer key, and the report-delegate event-log context key now use `workflow.info().run_id` — replay-stable but unique per run — instead of the workflow id. Workflow ids are reused across workflow-level `retry_policy` attempts, Temporal reset, and resubmission of the same `pipeline_run_id` (workflow ids derive deterministically from it), so workflow-id keying let a closed predecessor run's late eviction cleanup tear down a live successor run's library, tracer, and usage context on the same worker — failing the successor inline. With run-id keying such cross-run collisions are structurally impossible; `open_fresh_library` remains the same-run self-heal. `LibraryManager.teardown` also now forgets a library entry even if the library's own teardown raises (pop-first + `finally`), so a raising teardown can never turn into permanent worker-local poison, and the stale-entry healing path is tolerant of concurrent teardown.-- **An evicted Temporal workflow no longer poisons same-worker reruns through a leaked per-workflow library.** `WfPipeRouter`'s worker-local cleanup (per-workflow library + crate fingerprint, report-delegate event-log context) sat in the `finally` block AFTER the awaited `act_flush_trace_events` activity. An eviction-time interruption raised at that await (`_WorkflowBeingEvictedError` is a `BaseException`) escaped the `except Exception` around the flush and aborted the rest of the `finally`, leaking the library and its crate fingerprint in the worker-local `LibraryManager` under the deterministic `wf_{workflow_id}`. On a same-worker replay or re-run of that workflow id, `open_library` returned the stale library, a fresh `ClassRegistry` was attached, and `load_from_crate` was fingerprint-skipped — so the crate's dynamic classes never landed in the new registry and inline hydration failed where history recorded success: command-stream divergence, and with dynamic-class results on a single-worker queue, a persistent silent hang. Two complementary fixes: `WfPipeRouter` now opens its per-workflow library via the new `LibraryManager.open_fresh_library`, which force-tears-down any pre-existing library under the id (a leftover there can only be an interrupted predecessor), and the workflow `finally` now runs all worker-local cleanup BEFORE the awaited flush activity, which moved last — an eviction can only skip the flush itself, never the teardown. Guarded by an integration test that installs the leaked predecessor state and re-runs the workflow under the same id.-- **Costs-only Temporal runs no longer hang on a routine sticky-cache eviction.** In costs-only tracing mode (`emit_usage_events=True, emit_graph_events=False`), the workflow's in-memory trace buffer was populated exclusively by co-located activity completions writing cross-thread through the process-global `ReportingManager`, and the `act_flush_trace_events` schedule was gated on that buffer being non-empty. Activities don't re-execute on replay, so after a sticky-cache eviction the recreated buffer stayed empty, the recorded flush command was never re-emitted, and Temporal raised `[TMPRL1100]` — a silently hung workflow, on a single worker, with no config drift. Two changes make the command stream a pure function of payload + history: the flush-activity schedule is now gated purely on the payload field `emit_graph_events` (see the Changed entry above; runs without graph events skip the guaranteed-empty flush entirely) — and activity-side usage emissions always go through the per-process runner-fallback event log — the same path used when activities run on a remote worker — instead of the workflow's in-sandbox buffer, even when the activity is co-located with the workflow worker. Guarded by a replay regression test that records a costs-only history and replays it through `temporalio.worker.Replayer`.-- **Per-workflow tracing setup no longer depends on worker-local leak state.** `GraphTracerManager.open_tracer` raised `ValueError` when a stale tracer key was left behind by a prior interrupted execution on the same worker (e.g. a workflow evicted before its cleanup ran), and `WfPipeRouter` papered over that with a best-effort `except Exception` that silently disabled tracing — dropping the recorded flush command on replay and diverging the trace context carried by all downstream commands. `open_tracer` now pops and replaces a stale tracer (tearing it down, with a WARNING), and the catch-all around tracing setup is gone: the setup block is pure in-memory state, so any failure is a real bug that surfaces deterministically.-- **Temporal workflows no longer hang when worker tracing config differs across replays.** `WfPipeRouter` decided whether to set up per-workflow tracing — and therefore whether to schedule `act_flush_trace_events` — by reading `tracing_config.is_enabled` from worker-local config inside the workflow body. Replaying a workflow on a worker whose tracing config differed from the recorder's (worker restart, rolling deploy, scale-in) produced a divergent command stream, and Temporal's `[TMPRL1100]` nondeterminism error surfaced as a silently hung workflow stuck in workflow-task retries. The workflow now gates tracing setup purely on the presence of `trace_context` in the payload — the submitter's decision, recorded in history — and the worker-local `is_enabled` check happens only inside the flush activity, where reading local config is deterministic-safe (it already no-ops there when tracing is disabled). Side benefit: per-workflow graph assembly no longer requires the worker to have a tracing backend enabled. Guarded by a replay regression test that flips the worker's tracing config mid-workflow.- **`dry_run_pipeline` now owns its graph transport — graph generation no longer depends on the host's `tracing_config`.** The function requests a graph explicitly (`generate_graph=True`) but relied on the host's configured tracing backend as the emit/assemble channel, so a host with `tracing_config.is_enabled = false` (e.g. pipelex-api's `/validate` in direct mode, where the web-app's Dry Run button lands) always got "Pipeline execution did not produce a graph spec" and a response without `graph_spec`. The run now traces through a scoped `InMemoryEventLog` — the graph comes back under any tracing config, and validation dry-runs stop writing NDJSON/DynamoDB trace events as a side effect when tracing IS enabled. The no-graph contract violation is now a typed `DryRunGraphNotProducedError` (was a bare `PipelexError`, ambiguous in host logs that record only the exception type) — raised by both `dry_run_pipeline` and `dry_run_pipe_in_process`.-- **tz-aware payloads no longer hang Temporal callers on hosts without a timezone database — kajson bumped to 0.7.0.** kajson ≤ 0.6.0 decoded timezone-aware datetimes via `ZoneInfo(...)`, which needs the IANA tz database: system tz files or the `tzdata` package. On hosts with neither (slim containers, uv-managed standalone Pythons — including the CI test runner, where `tzdata` only arrived transitively via pandas on Python < 3.11), decoding any aware-datetime payload crossing the Temporal boundary (e.g. the `GraphSpec` on `act_dry_validate`'s result) raised `ZoneInfoNotFoundError` _inside the workflow task_ — which Temporal retries forever, so the submitter hung instead of failing. Fixed upstream in kajson 0.7.0: its self-sufficient timezone wire format carries the UTC offset (decoding degrades gracefully to a fixed offset instead of raising, and plain UTC decodes with zero tz-database dependence), and kajson now declares `tzdata` as its own runtime dependency, so a tz database is always available for named zones.+- **tz-aware datetime payloads no longer fail to decode on hosts without a timezone database — kajson bumped to 0.7.0.** kajson ≤ 0.6.0 decoded timezone-aware datetimes via `ZoneInfo(...)`, which needs the IANA tz database: system tz files or the `tzdata` package. On hosts with neither (slim containers, uv-managed standalone Pythons — including the CI test runner, where `tzdata` only arrived transitively via pandas on Python < 3.11), decoding any aware-datetime payload raised `ZoneInfoNotFoundError`. Fixed upstream in kajson 0.7.0: its self-sufficient timezone wire format carries the UTC offset (decoding degrades gracefully to a fixed offset instead of raising, and plain UTC decodes with zero tz-database dependence), and kajson now declares `tzdata` as its own runtime dependency, so a tz database is always available for named zones.### Added- **Protocol `validate` / `models` / `version` on `PipelexMTHDSProtocol`** — `validate` wraps `validate_bundle` (blueprints + per-pipe structures into the protocol `ValidationReport`); `models` wraps the builder's `list_models` into a `ModelDeck`; `version` reports `protocol_version` 0.6.0 with the installed pipelex version.-- **Temporal test hang diagnostics (CI).** Setting `PIPELEX_HANG_DUMP_DIR` arms a per-test watchdog in the temporal integration suite that dumps every thread's stack to a file before pytest-timeout's thread-method kill destroys the process, and mirrors temporalio's failure logs (which carry the workflow-task traceback) to the same directory. The tests workflow prints these dumps when a job fails — pytest-xdist swallows worker stderr on a crash, so this is the only post-mortem signal for a hung test on a remote runner.--- **`act_dry_validate` — the whole `/validate` job as ONE in-process Temporal activity.** New activity (+ one-step wrapper workflow `wf_dry_validate` and submitter helper `dispatch_dry_validate`) that runs the validation sweep **and** the graph-producing dry-run in one round-trip: the library is loaded once, the sweep borrows it, the graph dry-run traces into an in-memory event log against the same library, and the result carries `{per-pipe status map, GraphSpec | None}`. Zero nested Temporal dispatch, no NDJSON/DynamoDB tracing I/O, no usage/cost reporting. The graph is best-effort (an expected dry-run failure returns `graph_spec=None` with validation still successful); the sweep half is `validate_bundle` itself, so validation failures cross the activity boundary as structured `ErrorReport`s carrying the same categorized `ValidateBundleError` identity the direct path surfaces (a strict-mode signature refusal keeps its offending-pipe and signature refs in the caller-facing message). Registered in the `pipe` task pack; verified end-to-end in a real 3-process deployment (Tier 2d of the `temporal-e2e-validate` skill).--- **`dry_run_pipe_in_process` — graph-producing dry-run that never dispatches.** New entry in `pipelex/pipe_run/dry_run_pipeline.py`: dry-runs a pipe against an already-open library fully in-process and returns its `GraphSpec`, even under a Temporal-enabled hub. Three ContextVar scopes pin the whole run in-process: `scoped_event_log` (in-memory trace transport), `scoped_pipe_router` (nested controller sub-pipes resolve a local router instead of the `TemporalPipeRouter`), and the new `pipelex.hub.scoped_content_generator` (inference leaves resolve an inline dry generator instead of `ContentGeneratorInWorkflow` — holds regardless of whether the DRY mock lives at the pipe level or the leaf). This is the graph half of running `/validate` as one in-process Temporal activity.--- **`scoped_event_log` — a shared, scope-pinned trace-event transport.** New `pipelex.hub.scoped_event_log(event_log)` context manager (with `get_event_log_override()` accessor, mirroring `scoped_pipe_router`): both the write side (tracer emission in `pipeline_run_setup`) and the read side (`assemble_tracing`) now prefer the scoped instance over building their own backend via `make_event_log`, so a run can trace into ONE shared `InMemoryEventLog` — no NDJSON file, no DynamoDB round-trip — and still assemble its `GraphSpec` from it. A set override implies tracing-enabled (honored even when `tracing_config.is_enabled` is off); the scoped instance's `close()` must be idempotent or a no-op (the write-side tracer closes its transport at teardown — `InMemoryEventLog`'s is a no-op). ContextVar-scoped, so concurrent runs with separate scopes stay isolated. This is the foundation for running the `/validate` graph dry-run fully in-process and in-memory inside a Temporal activity.--### Coordination--- **pipelex-worker lockstep** — the Temporal payload models cross into `pipelex-worker` via the `pipelex` pin; the worker must ship the same `pipelex` version as `pipelex-api` when the MTHDS Protocol unification deploys.-## [v0.32.1] - 2026-06-09### Added@@ -189,19 +164,15 @@## [v0.32.0] - 2026-06-09-This cycle reworks **cost reporting** so it survives distributed execution and stops leaking. Cost now rides on the run result instead of a side buffer: a multi-worker Temporal run aggregates usage from every worker into a single end-of-run report, the success-path registry leak is gone by removal, and a new `--costs` switch decouples cost collection from `--graph`. It also extracts the **framework-agnostic runtime bridge** so any host runtime — not just Mistral Workflows — can embed Pipelex through one boundary.+This cycle reworks **cost reporting** so it stops leaking. Cost now rides on the run result instead of a side buffer: the success-path registry leak is gone by removal, and a new `--costs` switch decouples cost collection from `--graph`. It also extracts the **framework-agnostic runtime bridge** so any host runtime — not just Mistral Workflows — can embed Pipelex through one boundary.### Added-- **`--temporal` / `--no-temporal` on `pipelex validate`.** The `bundle`, `pipe`, and `method` validate subcommands (and `validate --all`) gain the same boot override `run` already had — it sets `temporal.is_enabled` for the process. Today the validation sweep always runs in-process regardless of backend (it scopes its own in-process router), so the flag does not change _what_ validation does; it is the lever for booting validation under a Temporal-enabled hub — e.g. to verify the sweep stays in-process and never dispatches nested controller sub-pipes to Temporal. It is also forward-looking: once validation can run as a standalone Temporal activity, this flag becomes the switch that dispatches the sweep through that activity.-- **`--costs` / `--no-costs` (default on).** `pipelex run pipe|method|bundle` gains a dedicated cost switch that emits usage tracing events and renders the end-of-run cost report. It rides the shared trace-event transport independently of `--graph`, so `--no-graph --costs` reports cost without building a graph and `--graph --no-costs` builds a graph with no cost report. It replaces the removed `--cost-report` (see Changed).-- **Distributed cost reporting.** In Temporal mode, usage emitted from inference activities running on separate worker processes is now aggregated into a single cost report at the submitter. Usage is assembled from the trace-event stream onto `PipeOutput` in both direct and Temporal execution, so a cross-worker run reports its true total — previously cross-worker usage was emitted but never rendered into a report.-- **`tokens_usages` on `PipeOutput`.** New `tokens_usages: list[AnyTokensUsage] | None` and `usage_assembly_error: str | None` fields, mirroring `graph_spec` / `graph_assembly_error`. Cost is now part of the run result and is exposed automatically wherever a `PipeOutput` is returned, including the Pipelex API response. Render it with `CostRegistry.generate_report(tokens_usages=...)`.-- **`--mock-inference`.** A LIVE run that fakes AI calls at the inference leaf with reportable synthetic usage: operators dispatch real activities (exercising the true distributed path) but no tokens are billed, so distributed cost reporting can be validated cheaply and deterministically. Mutually exclusive with `--dry-run`. It covers the LLM leaf; image-generation / extract / search under `--mock-inference` fail loud with `MockInferenceUnsupportedError` (pointing at `--dry-run`) rather than silently calling the real provider.+- **`--mock-inference`.** A LIVE run that fakes AI calls at the inference leaf with reportable synthetic usage, so cost reporting can be validated cheaply and deterministically without billing tokens. Mutually exclusive with `--dry-run`. It covers the LLM leaf; image-generation / extract / search under `--mock-inference` fail loud with `MockInferenceUnsupportedError` (pointing at `--dry-run`) rather than silently calling the real provider.- **Cost report in the agent CLI JSON.** `pipelex-agent run ... --with-memory` attaches a best-effort structured `cost_report` (`{total_cost, by_model}`, real USD) to its JSON envelope when the run did reportable work and summary aggregation succeeds — treat it as optional (absent for dry runs, `--no-costs`, the API-runner path, or an aggregation failure). The agent surface stays JSON-only — no Rich table on stderr; compact mode is unchanged.@@ -217,20 +188,12 @@ This cycle reworks **cost reporting** so it survives distributed execution and s- **Dry runs never emit a cost report; free-model runs still do.** Suppression keys on whether the run did reportable work (any tokens or any cost), not on total cost alone: a dry run (zero tokens, zero cost) is suppressed, while a real run on a free / zero-price model (e.g. Ollama) still reports its token usage with a zero total.-- **Temporal `act_assemble_graph` renamed to `act_assemble_tracing`.** It now assembles both the graph (`GraphSpec`) and usage (`tokens_usages`) from a single trace-event read, gated per concern by the run's `--graph` / `--costs` flags.-### Fixed- **Docs deploy crashed on a fresh runner with `InferenceSetupRequiredError`.** `pipelex-dev generate-error-pages` (a prerequisite of every `docs-*` make target) bootstrapped Pipelex with `needs_inference=True`, so on a CI runner with the gateway enabled but no on-disk service config it hit the first-run inference-setup gate and exited non-zero — breaking the `Deploy docs` workflow on `main`. The command only introspects `PipelexError` subclasses to write markdown and never calls inference, so it now bootstraps with `needs_inference=False`.- **`UsageRegistry` success-path leak.** The per-run cost registry was opened during run setup but closed only on the failure path, so every successful run leaked its registry — and in a long-lived process (e.g. the Pipelex API) reusing a `pipeline_run_id` could collide on the orphaned registry. The registry is removed entirely, so the leak is structurally impossible.-- **Temporal silent-hang fail-safe.** A pipelex domain error raised _inline_ in workflow code — never dispatched through an activity — was neither an `ActivityError` nor an `ApplicationError`, so Temporal treated it as a workflow-task failure and retried it indefinitely: a silent, resource-burning hang that only surfaced (as a generic timeout) after the workflow execution timeout. `WfPipeRouter` and `WfPipeRun` now end their boundary handling with a `PipelexError` catch-all that converts a genuine inline error to a terminal, classified failure (the parent also still fires the FAILED webhook), and the worker registers `PipelexError` in `workflow_failure_exception_types` as a backstop so any uncaught domain error fails the workflow terminally instead of hanging. Scoped to domain errors only — transient Temporal/infra errors keep their (correct) task-retry behavior.--- **`PipeSearch` on Temporal no longer hangs.** Web search was the only inference operator that ran its leaf inline instead of through the `ContentGenerator` seam, so it had no Temporal activity. Run through the Temporal path, a search step executed its real HTTP call inside the workflow event loop, and on failure raised a raw `SearchJobFailureError` (neither an `ActivityError` nor a `WorkflowExecutionError`) that Temporal treated as a workflow-task failure and retried forever — the submitter hung instead of getting a result. Even on success it was replay-unsafe (the search re-ran on every workflow replay). `PipeSearch` now dispatches its leaf through new `act_search_gen_sourced_answer` / `act_search_gen_structured` activities like every other operator: results are recorded in workflow history (replay-safe) and failures cross the workflow boundary as a terminal `WorkflowExecutionError` carrying the structured `ErrorReport` — identical classification to the local path. A `runner-search` worker scope and per-activity routing entries are available for split deployments.--- **Bundle validation no longer leaks nested controller sub-pipes to Temporal.** With Temporal enabled, the in-process dry-run sweep (`BundleValidator`) ran the top-level pipe in-process but dispatched every _nested_ controller sub-pipe through the hub-default router — the Temporal router — turning a no-cost dry run into real top-level workflow dispatches. A standalone `PipeBatch` / `PipeParallel` swept directly fanned out N concurrent same-id top-level dispatches and collided (`WorkflowAlreadyStartedError`), surfacing as **HTTP 422** on the runner's `/validate` route; other shapes silently round-tripped Temporal. The sweep now scopes its in-process router for the whole sweep (mirroring the DIRECT-mode runtime bridge), so nested controllers resolve the in-process router and validation stays fully in-process regardless of the configured backend.-## [v0.31.0] - 2026-06-04This release hardens Pipelex at its edges. The headliners: a full **error-handling overhaul** that gives every error a stable, RFC 7807-shaped identity and carries it all the way out to webhooks and external surfaces; **lenient validation with pipe signatures**, so you can sketch and dry-run a whole pipeline top-down before a single pipe is implemented; and a first, **experimental cut of CSV tabular support** for reading and writing typed lists straight from `.csv`.@@ -239,11 +202,11 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli- **Stable error identity (RFC 7807).** Every `PipelexError` now exposes `title()` and a stable `type_uri()` — a real, dereferenceable `https://docs.pipelex.com/latest/errors/<error>/` URL — and every `ErrorReport` carries both as populated fields, so consumers read `report.title` / `report.type_uri` directly instead of humanizing class names. `ErrorReport.to_problem_document()` renders an `application/problem+json` document with no web-framework dependency, and a new `DisclosureMode` (`VERBOSE` / `STRICT`) controls how much leaks onto external surfaces: `STRICT` drops provider/model attribution and redacts any message that wasn't authored to be caller-facing, keeping only the stable identifiers.-- **Structured error payloads on failure webhooks.** When a run fails, the delivery webhook now includes an `error` object — the full `ErrorReport` as a dict — so receivers can rehydrate it with `ErrorReport.from_dict(...)`, render an RFC 7807 response, or route on `error_domain` / `retryable`. Applies to both Temporal and direct execution. A `WebhookTarget.payload` that collides with a Pipelex-reserved key (`pipeline_run_id`, `status`, `result_url`, `error`) is now rejected at construction.+- **Structured error payloads on failure webhooks.** When a run fails, the delivery webhook now includes an `error` object — the full `ErrorReport` as a dict — so receivers can rehydrate it with `ErrorReport.from_dict(...)`, render an RFC 7807 response, or route on `error_domain` / `retryable`. A `WebhookTarget.payload` that collides with a Pipelex-reserved key (`pipeline_run_id`, `status`, `result_url`, `error`) is now rejected at construction.- **Per-class error reference pages.** Every `PipelexError` subclass now has a generated reference page under `docs/errors/`, surfaced in the docs site as "Error Reference" — so a `type_uri` dereferences straight to a populated page. Regenerate with the new `pipelex-dev generate-error-pages` command (`make generate-error-pages`, alias `make gep`); pages a maintainer claims with a `<!-- pipelex:authored -->` marker are preserved across runs.-- **`request_id` on `JobMetadata`.** An optional caller-supplied request id, threaded through every activity / workflow / submitter hop and into the Temporal log context. Set it at dispatch with `pipeline_run_setup(..., request_id="...")` and read it back off `job_metadata.request_id`.+- **`request_id` on `JobMetadata`.** An optional caller-supplied request id, threaded through the run and into the log context. Set it at dispatch with `pipeline_run_setup(..., request_id="...")` and read it back off `job_metadata.request_id`.- **`PipeSignature` — contract-only pipes for top-down design.** A new pipe type (`type = "PipeSignature"`) declares a pipe's `inputs`, `output`, and `description` with no implementation, so an author or agent can sketch a complete pipeline before committing to the operator that will eventually do the work. At dry-run time a signature mints a mock output matching its declared type and multiplicity; at runtime it raises `PipeSignatureNotExecutableError`. The optional `signature_for` field hints which pipe type the stub stands in for. See [Signature Pipes](https://docs.pipelex.com/latest/building-methods/pipes/signature-pipes/).@@ -257,7 +220,7 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli### Changed-- **`ErrorReport` is now a frozen Pydantic `BaseModel`** (was a frozen Pydantic dataclass) — still immutable and still round-trips through `to_dict()` / `from_dict()`, but an attempted mutation now raises `pydantic.ValidationError` instead of `dataclasses.FrozenInstanceError`. Relatedly, **`recover_error_report()` is now total**: it always returns an `ErrorReport`, synthesizing one from the new `UnrecoverableWorkflowFailureError` (surfacing the deepest worker-side cause) when a Temporal failure carries no embedded report — callers no longer branch on `None`.+- **`ErrorReport` is now a frozen Pydantic `BaseModel`** (was a frozen Pydantic dataclass) — still immutable and still round-trips through `to_dict()` / `from_dict()`, but an attempted mutation now raises `pydantic.ValidationError` instead of `dataclasses.FrozenInstanceError`.- **Dry-run and validation consolidated into `BundleValidator`.** The standalone `dry_run_pipe` / `dry_run_pipes` functions and the modules `pipelex/pipe_run/dry_run.py`, `dry_run_with_graph.py`, and `dry_pipe_router.py` are gone; their work now lives in `BundleValidator`, and `validate_bundle` / `BundleValidator` gained the `allow_signatures: bool = False` flag (strict by default). Validation ordering and outputs are unchanged — only the import surface moved. Callers importing from `pipelex.pipe_run.dry_run` must switch to `BundleValidator`.@@ -267,8 +230,6 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli- **Filesystem path helpers moved to `pathlib.Path`.** Helpers in `pipelex.tools.misc.file_utils` and `pipelex.tools.misc.json_utils` (`save_text_to_path`, `load_json_from_path`, `load_binary`, `copy_file`, `ensure_directory_exists`, `get_incremental_file_path`, and the rest) now take and return `Path` instead of `str`; path handling is `Path`-based throughout `pipelex/`, converting to/from `str` only at boundaries. Callers passing bare strings must wrap them with `Path(...)`.-- **Safer Temporal defaults.** `[temporal.search_attributes]` now ships `enabled = false` (opt in per env) so a local boot can't fail on the unregistered-attribute check, and the default `[temporal.queue_options.temporal_task_queue]` block is removed to avoid leaving an orphan overlay when a queue is renamed.-- **`teardown_current_library()` renamed to `clear_current_library()`** in `pipelex.hub`, pairing it cleanly with `set_current_library()`.- **Dev tooling: `pyright` bumped `1.1.408 → 1.1.410`** (drove two behavior-neutral internal adjustments).@@ -284,7 +245,7 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli- **urllib3 floor `>=2.7.0`** to patch two high-severity issues: CVE-2026-44431 (GHSA-qccp-gfcp-xxvc) forwards sensitive headers across origins on proxied low-level redirects, and CVE-2026-44432 (GHSA-mf9v-mfxr-j63j) bypasses the decompression-bomb safeguards in parts of the streaming API. Floor added to the runtime `dependencies`.- **pymdown-extensions floor `>=10.21.3`** to patch CVE-2026-46338 (GHSA-62q4-447f-wv8h, medium): a regression in `pymdownx.snippets` reintroduced the sibling-prefix path-traversal bypass despite `restrict_base_path`. Docs-only; floor added to the `docs` extra.- **idna floor `>=3.15`** (lockfile resolves to 3.18) to patch CVE-2026-45409 (GHSA-65pc-fj4g-8rjx, medium): specially crafted inputs to `idna.encode()` could bypass the CVE-2024-3651 fix. Floor added to the runtime `dependencies`.-- **Proactive floor bumps in the same hardening pass**, past known-vulnerable releases: pillow `>=12.1.1`, protobuf `>=6.33.5`, python-dotenv `>=1.2.2`, requests `>=2.33.0` (runtime); aiohttp `>=3.12.14` (`temporal` extra); Pygments `>=2.20.0` (`docs` extra).+- **Proactive floor bumps in the same hardening pass**, past known-vulnerable releases: pillow `>=12.1.1`, protobuf `>=6.33.5`, python-dotenv `>=1.2.2`, requests `>=2.33.0` (runtime); Pygments `>=2.20.0` (`docs` extra).## [v0.30.3] - 2026-05-28@@ -338,27 +299,17 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli- **New `PipeStructure` operator** that turns text into a structured concept via a single LLM call. Takes one `Text`-compatible input (or a domain concept that `refines = "Text"`), produces any structured output with the usual multiplicity options (`Foo`, `Foo[]`, `Foo[N]`). Useful whenever the source text comes from a PDF extraction, a search result, an upstream pipe, or any non-LLM origin. Documented at [building-methods/pipes/pipe-operators/PipeStructure.md](building-methods/pipes/pipe-operators/PipeStructure.md).-- **Custom search attributes on the Temporal namespace, with a strict prerequisite at worker boot.** Pipelex sets five custom Keyword search attributes on every workflow start — `PipeCode`, `PipelineRunId`, `SessionId`, `UserId`, `DomainCode` — so the Temporal dashboard can filter, group, and audit workflows by their actual semantic identity rather than by opaque workflow ids. A real Temporal cluster rejects every `StartWorkflowExecution` RPC that references an unregistered attribute, so the check at worker boot is a hard fail (raises `SearchAttributeRegistrationError`) on a reachable namespace — the previous "warn and continue" framing was dishonest because workflows would then fail at every dispatch with a much less actionable error. The exception message embeds both the `pipelex setup-temporal-namespace` invocation and the equivalent raw `temporal operator search-attribute create` command. `RPCError` (control-plane unreachable) stays a soft fail; the in-process test server is registered up front by the integration conftest.--- **`[temporal.search_attributes]` config block** — master `enabled` toggle and an `attributes` subset selector for the five built-ins. `enabled = false` skips both the worker-boot registration check and the per-workflow attribute attachment (use this when the namespace policy is owned by a third party who will not register attributes). The validator rejects unknown attribute names so typos like `"PipelineRunID"` surface at config load instead of silently producing no attribute. The Pydantic model lives in `pipelex/temporal/config_temporal.py` so the validator can reference `BUILTIN_SEARCH_ATTRIBUTES` without pulling `temporalio` into the config-load path.--- **`pipelex setup-temporal-namespace` CLI command** — wraps `OperatorService.AddSearchAttributes` against the configured server profile so operators don't need a separate `temporal` / `tcld` install for the common case. Reads the same `[temporal.search_attributes].attributes` list and `[temporal.temporal_config]` block the worker uses, so the names registered here can never drift from the names the worker will dispatch with. `--dry-run` prints the equivalent raw `temporal operator search-attribute create` command without executing. `--server <profile>` targets a non-default server profile. On Temporal Cloud namespaces where the worker API key lacks `OperatorService.AddSearchAttributes` permission, the command catches `RPCError(PERMISSION_DENIED)` and prints the fallback runbook (raw `temporal` CLI command, `tcld` for Cloud, Cloud UI link).--- **Per-workflow static summary and details in the Temporal dashboard.** Every top-level workflow start now sets `static_summary` (200-byte Markdown like `translate_doc — Translate a document from English to French`) and `static_details` (Markdown table of pipe code, domain, pipeline run id, user, session, optional library crate, optional inputs). Every `workflow.execute_activity(...)` call sets a per-call `summary=` carrying the call-specific meaning (`LLM text · pipe=translate_doc · model=gpt-4o`, `Img gen N× · pipe=cover · model=fal-flux-dev · n=3`, etc.). The formatting policy lives in the new `pipelex/temporal/tprl/observability.py` module.-- **Documented `render_js` and `include_raw_html` on `PipeExtract`.** Both fields were already shipping on `PipeExtractBlueprint`; only docs were missing. `render_js = true` asks the extraction backend to render JavaScript before fetching web-page content; `include_raw_html = true` populates each extracted `Page`'s `raw_html` field with the fetched HTML. Added to [building-methods/pipes/pipe-operators/PipeExtract.md](building-methods/pipes/pipe-operators/PipeExtract.md).- **Documented `XHIGH` value on `ReasoningEffort`.** The enum already shipped seven levels; the `under-the-hood/reasoning-controls.md` table listed only six. `XHIGH` sits between `HIGH` and `MAX` and maps to provider-specific xhigh values where supported.-- **Bounded fan-out concurrency for `PipeBatch`.** A `PipeBatch` over many items no longer spawns every branch — every coroutine, every deep-copied working memory, every inference call — at once. Branches now run in bounded chunks driven by the new `[pipelex.pipeline_execution_config]` setting `max_concurrency` (default `8`; set to the literal `"unbounded"` for unbounded fan-out, the previous behavior). This is the resilience-without-Temporal pillar: it keeps a large workload (one pipe over thousands of documents) from overwhelming asyncio, memory, and provider rate limits. Results still preserve input order, and a branch failure still propagates (first error by input index wins). When a batch fans out over a large number of items, an advisory log line points at the Temporal track as the durable, rate-limited path. `PipeParallel`, which fans over a fixed pipe-defined branch set, is unchanged.+- **Bounded fan-out concurrency for `PipeBatch`.** A `PipeBatch` over many items no longer spawns every branch — every coroutine, every deep-copied working memory, every inference call — at once. Branches now run in bounded chunks driven by the new `[pipelex.pipeline_execution_config]` setting `max_concurrency` (default `8`; set to the literal `"unbounded"` for unbounded fan-out, the previous behavior). It keeps a large workload (one pipe over thousands of documents) from overwhelming asyncio, memory, and provider rate limits. Results still preserve input order, and a branch failure still propagates (first error by input index wins). `PipeParallel`, which fans over a fixed pipe-defined branch set, is unchanged.- **Authoritative `error_domain` → HTTP-status mapping.** `pipelex/base_exceptions.py` now owns the mapping that downstream HTTP APIs (`pipelex-relay`, `pipelex-back-office`) need to render an `ErrorReport` as an HTTP response: `error_domain_to_http_status()` (the pure domain table — `INPUT` → 422, `CONFIG`/`RUNTIME`/unknown → 500) and the `ErrorReport.http_status` property, which adds a provider-429 passthrough so the API can emit a `Retry-After` header from `provider_metadata.retry_after_seconds`. The library stays HTTP-agnostic — no web-framework dependency, just the mapping table; downstream FastAPI handlers call the helper instead of reinventing the contract.-- **Category-aware error boundary on Temporal activities.** Every in-scope Temporal activity is now decorated with `@convert_pipelex_errors` (new module `pipelex/temporal/tprl/activity_error_boundary.py`), which converts a `PipelexError` raised inside the activity into a `TemporalError` at the activity boundary. This packs the structured `ErrorReport` into `ApplicationError.details` and derives `non_retryable` from the error's `InferenceErrorCategory` — so the workflow-side `TemporalError.from_app_error` keeps `error_category`, `user_action`, `model` and `provider` instead of landing in its `error_report is None` fallback, and a non-retryable `CogtError` (`CONFIGURATION`, `CONTENT`, `CAPACITY`) is no longer retried. Resilience for inference failures lives entirely on the Temporal track — direct (non-Temporal) execution makes a single pipeline-level attempt. `act_assemble_graph` is deliberately left unwired — it is best-effort observability that swallows every failure and degrades to `None`. As part of this, `TemporalError._log_critical` / `_log_error` now select `activity_log` vs `workflow_log` based on `activity.in_activity()`, since `workflow.logger` raises `_NotInWorkflowEventLoopError` outside a workflow event loop.--- **`AMBIGUOUS` inference-error category for outcome-uncertain failures.** New `InferenceErrorCategory.AMBIGUOUS` for failures where the error type is known but the outcome is not — the operation may or may not have committed (e.g. a connection dropped mid-request). It is non-retryable, like `CONFIGURATION` / `CONTENT` / `CAPACITY`, but semantically distinct from `UNKNOWN`, which means the error could not be classified at all. The `azure_rest` image-generation worker now raises `AMBIGUOUS` for mid-request transport failures — `ReadError` / `WriteError` / `RemoteProtocolError` and `ReadTimeout` / `WriteTimeout` — so Temporal does not auto-retry a non-idempotent, billable image submit whose outcome is unknown; pre-request failures (`ConnectError` / `ConnectTimeout` / `PoolTimeout`) stay `TRANSIENT` and retryable.+- **`AMBIGUOUS` inference-error category for outcome-uncertain failures.** New `InferenceErrorCategory.AMBIGUOUS` for failures where the error type is known but the outcome is not — the operation may or may not have committed (e.g. a connection dropped mid-request). It is non-retryable, like `CONFIGURATION` / `CONTENT` / `CAPACITY`, but semantically distinct from `UNKNOWN`, which means the error could not be classified at all. The `azure_rest` image-generation worker now raises `AMBIGUOUS` for mid-request transport failures — `ReadError` / `WriteError` / `RemoteProtocolError` and `ReadTimeout` / `WriteTimeout` — so an automatic-retry layer won't re-fire a non-idempotent, billable image submit whose outcome is unknown; pre-request failures (`ConnectError` / `ConnectTimeout` / `PoolTimeout`) stay `TRANSIENT` and retryable.-- **Explicit, uniform Tier 1 transport retry across every inference worker.** A new `[cogt]` setting `transport_max_retries` (default `2`) is now wired explicitly into every inference SDK client factory — Anthropic, OpenAI / Azure OpenAI, the Portkey-backed gateway clients, Mistral, and Google — instead of each factory silently inheriting whatever retry posture its SDK happens to default to. The two SDK families that default to _no_ transport retry are brought up to the same floor: the Mistral client gets a bounded-backoff `RetryConfig` (`retry_connection_errors=True`) and the Google GenAI client gets `HttpOptions(retry_options=...)`. The genuinely SDK-less path — the `azure_rest` image-generation worker, which talks to Azure over raw `httpx` — gets a `tenacity`-based transport-retry wrapper (new module `pipelex/cogt/inference/transport_retry.py`) that retries connection failures and transient HTTP statuses (408/409/429/5xx) and honors `Retry-After`. When no `Retry-After` header is present, the fallback backoff uses full jitter (`wait_random_exponential`) so a burst of failures retried together does not re-fire in lockstep. On a non-idempotent submit-style POST it narrows the retry to failures that prove the server did no billable work — a request that was never delivered, or a 408/429 rejection — withholding an ambiguous 5xx, a 409 conflict, and a post-delivery timeout such as a `ReadTimeout`. Transport retry is now a deliberate, configured, uniform policy rather than a per-provider accident. This is "Tier 1" of the retry model — direct (non-Temporal) execution still makes a single pipeline-level attempt on top of this transport floor; durable resilience remains the Temporal track. (The `portkey-ai` SDK does not expose a retry knob — it carries its own internal retry — so the gateway's `AsyncPortkey` client is left as-is; only its underlying OpenAI clients are wired.)+- **Explicit, uniform Tier 1 transport retry across every inference worker.** A new `[cogt]` setting `transport_max_retries` (default `2`) is now wired explicitly into every inference SDK client factory — Anthropic, OpenAI / Azure OpenAI, the Portkey-backed gateway clients, Mistral, and Google — instead of each factory silently inheriting whatever retry posture its SDK happens to default to. The two SDK families that default to _no_ transport retry are brought up to the same floor: the Mistral client gets a bounded-backoff `RetryConfig` (`retry_connection_errors=True`) and the Google GenAI client gets `HttpOptions(retry_options=...)`. The genuinely SDK-less path — the `azure_rest` image-generation worker, which talks to Azure over raw `httpx` — gets a `tenacity`-based transport-retry wrapper (new module `pipelex/cogt/inference/transport_retry.py`) that retries connection failures and transient HTTP statuses (408/409/429/5xx) and honors `Retry-After`. When no `Retry-After` header is present, the fallback backoff uses full jitter (`wait_random_exponential`) so a burst of failures retried together does not re-fire in lockstep. On a non-idempotent submit-style POST it narrows the retry to failures that prove the server did no billable work — a request that was never delivered, or a 408/429 rejection — withholding an ambiguous 5xx, a 409 conflict, and a post-delivery timeout such as a `ReadTimeout`. Transport retry is now a deliberate, configured, uniform policy rather than a per-provider accident. This is the transport-level floor of the retry model — pipeline execution makes a single attempt on top of it. (The `portkey-ai` SDK does not expose a retry knob — it carries its own internal retry — so the gateway's `AsyncPortkey` client is left as-is; only its underlying OpenAI clients are wired.)- **Offline mode for Pipelex Gateway setup and dry-run.** When the gateway is enabled but the remote config service is temporarily unreachable, Pipelex now falls back to a previously primed on-disk cache (`~/.pipelex/cache/remote_config.json`, schema-versioned) instead of failing setup outright. Dry-run, validation, and `pipelex-agent run bundle --dry-run` complete normally; only the actual inference call still needs the network at runtime. The cache is primed on every successful fetch and on `pipelex init` while online. When the gateway is disabled (BYOK), no remote fetch is attempted at all — setup is fully offline. A new `RemoteConfigStaleWarning` (`UserWarning`) is emitted whenever stale cache is in use; the agent CLI surfaces it on the JSON envelope as `warnings: [{"type": "RemoteConfigStale", ...}]`. Telemetry is suppressed (no-op) when running on a cached config so stale model identities don't pollute metrics. The doc/fixture generators (`pipelex-dev update-gateway-models`, `preprocess_test_models_cmd`) refuse the cache fallback via a new `require_fresh=True` flag, so committed reference docs and test fixtures never bake in stale data.@@ -380,42 +331,12 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli- **`pipelex-agent validate bundle` graph-format option renamed `--format` → `--graph-format` (breaking).** The `--format`/`-f` flag that selected the graph renderer (`mermaidflow` / `reactflow` / `both`) is now `--graph-format`/`-f`, freeing `--format` to be the uniform markdown/json output-format flag across every agent-CLI command.-- **Top-level Workflow ID shape change (breaking).** Workflow IDs go from `{env}{session5}-{rand5}-{ClassName}` (e.g. `EdgdJ-HR5fd-TemporalPipeRun-pipe-router`) to `{env_prefix}{pipeline_run_id}` (e.g. `ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c`). The session-id and random-id components and the calling class name are gone; identity flows entirely from Pipelex's existing `JobMetadata.pipeline_run_id`. Operational tooling that grepped for the old shape must update.--- **Pipeline run chain semantics shift (breaking, behavioral).** Because the Workflow ID is now derived from `pipeline_run_id`, callers that pass a stable `pipeline_run_id` to `PipelineFactory.make_pipeline(pipeline_run_id=...)` and re-execute now land on the same Temporal Workflow Execution Chain (with a fresh `run_id` per execution under the SDK-default `ALLOW_DUPLICATE` reuse policy). The previous behavior produced a fresh workflow_id per execution by accident, via the truncated session id and random shortuuid components — not by design. This is now documented behavior, not a bug.--- **Child Workflow ID separator (breaking).** Child workflow ids use `_` as the separator (never `/`), so an id can be reused verbatim as an S3 key or file name without creating spurious path segments. Examples: the fixed-role `wf_pipe_router` child of `wf_pipe_run` is `{parent}_pipe-router`; a dynamic sub-pipe spawned by a router is `{parent}_{pipe_code}-{8-hex-chars}` (the 8 hex chars come from `workflow.uuid4()` for replay-safety). Operational tooling parsing the nested-id format must update.--- **Activity ID change (breaking).** Pipelex no longer customizes `activity_id`. The Temporal Python SDK auto-assigns deterministic sequential integers (`"1"`, `"2"`, …) per workflow run, which guarantees per-`(workflow_id, run_id)` uniqueness by construction and is replay-safe (assigned by history position). Per-call meaning that previously lived in `activity_id` now lives in the per-activity `summary=`. Anything that filtered or grouped Event History by the old semantic strings (`"craft-text"`, `"craft-object-direct"`, `"jinja2-text"`, `"extract-pages"`, etc.) must read the per-activity `summary` or the Activity Type instead.--- **`wfid` parameter removed (breaking).** The `wfid: str | None = None` parameter is dropped from `PipeRunProtocol`, `PipeRouterProtocol`, `ContentGeneratorProtocol`, and every implementation (`PipeRun`, `PipeRouter`, `DryPipeRouter`, `TemporalPipeRun`, `TemporalPipeRouter`, `ContentGenerator`, `ContentGeneratorDry`, `ContentGeneratorInWorkflow`). The parameter was a legacy artifact — `wfid` originally seeded child workflow ids, was later co-opted as a default `activity_id`, and had no production callers. Identity now flows entirely via `JobMetadata.pipeline_run_id`; observability is auto-derived from `pipe_code` / `domain_code`. No deprecation cycle; per Pipelex's no-backward-compat policy, callers update at once.--- **`ContentGeneratorInWorkflow` LRU + replay short-circuit deleted.** The worker-singleton `_seen_activity_ids: OrderedDict[(workflow_id, run_id), set[str]]` cache, the `_MAX_SEEN_RUNS` bound, and `_record_activity_id` (with its `workflow.unsafe.is_replaying()` short-circuit) are gone. They existed solely to defend against the duplicate-activity-id failure mode that no longer exists now that Pipelex does not customize `activity_id`. Future contributors: do not reintroduce worker-singleton state on the activity-dispatch path — the determinism guarantee is the SDK assigning ids by history position.-- **`structuring_method = "preliminary_text"` on `PipeLLM` works again, via build-time elaboration.** When `PipelexInterpreter` parses a `.mthds` file with a `PipeLLM` carrying `structuring_method = "preliminary_text"`, the new `BundleElaborator` rewrites it before any pipe runs into a `PipeSequence` of two synthetic pipes: a `PipeLLM` producing `Text` (step 1, inheriting the original prompt + inputs + model) and a `PipeStructure` producing the original output (step 2, optionally using `model_to_structure`). The synthetic codes are tracked in a `PipelexBundleBlueprint.elaboration_metadata` side-table (excluded from serialization). Output multiplicity is preserved: step 1 always emits a single `Text`; step 2 emits `Foo`, `Foo[]`, or `Foo[N]` according to the original output. Two LLM calls are issued per invocation. The user-facing pipe code is unchanged, so callers, `main_pipe`, and the run API keep working as before. Mechanism documented at [under-the-hood/build-time-elaboration.md](under-the-hood/build-time-elaboration.md).- **`PipeLLM` runtime no longer knows about `structuring_method`.** The field is now a pure build-time directive. The runtime `PipeLLM` class no longer carries `structuring_method`, the validator that rejected mismatched output concepts has moved to `PipeLLMBlueprint` (where it fires at parse time), and the `NotImplementedError` previously raised at runtime when `preliminary_text` was selected is gone — the elaborator handles it. `PipeLLMSpec` exposes `structuring_method` so AI agents authoring via specs can still opt in.- **`StructuringMethod` enum import path moved.** The enum is now defined in `pipelex.pipe_operators.llm.pipe_llm_blueprint` (it lives next to the blueprint that consumes it) instead of `pipelex.pipe_operators.llm.pipe_llm`. Direct importers must update their `from pipelex.pipe_operators.llm.pipe_llm import StructuringMethod` line to `from pipelex.pipe_operators.llm.pipe_llm_blueprint import StructuringMethod`.-- **Collapsed the `tprl_content_generation/` workflow layer.** Each content-generation call (LLM text/object/object-list, image generation, Jinja2 rendering, document extraction, PDF page-view rendering) previously went through a dedicated child workflow (`WfMakeLLMText`, `WfMakeObject`, `WfMakeImages`, `WfMakeJinja2Text`, `WfMakeExtract`, `WfRenderPageViews`) that wrapped a single `act_*` activity. The wrappers added one workflow-scheduled / started / completed history-event triplet per call without changing durability guarantees — every retry still happens at the activity level. `ContentGeneratorInWorkflow` now calls `workflow.execute_activity(act_*, ...)` directly from inside `WfPipeRouter`, deleting all `WfMake*` / `WfRenderPageViews` workflows, both `ContentGeneratorTop` and `ContentGeneratorChild` generators along with their factories, and the `content_generator_models.py` assignment-types module. Per-step meaning in the Temporal Web UI now comes from per-activity `summary=` (and Activity Type), replacing the old semantic `activity_id` strings. Page-views augmentation (`should_include_page_views`) now works in Temporal mode: `make_extract_pages` dispatches `act_render_page_views` for `document_uri` inputs and builds an inline single-element `[ImageContent(url=...)]` list for `image_uri` inputs, then attaches each page-view to its `PageContent`. Previously the direct-mode generator handled both branches but the child-workflow path (`WfMakeExtract`) only ran the extract activity and silently skipped the page-views step, so `should_include_page_views=True` was a no-op under Temporal.--- **Per-activity, per-handle Temporal task-queue routing.** Replaced the provisional `WorkerConfig.inference_task_queue` (one-off LLM-text override) with a general `activity_queues: dict[str, ActivityRouteConfig]` table. Each entry declares a per-activity `default` queue and an optional `by_handle` map keyed by runtime handle (LLM model handle for text/object/object-list, `img_gen_handle` for image generation, `extract_handle` for document extraction). At dispatch time, `WorkerConfig.resolve_queue(activity_name, routing_key)` walks three layers: (1) `activity_queues[activity_name].by_handle[routing_key]`, (2) `activity_queues[activity_name].default`, (3) the worker-wide `default_task_queue` default. Every `ContentGeneratorInWorkflow` call site now passes `task_queue=worker_config.resolve_queue(...)` uniformly — the asymmetric "LLM-text-only kwarg" is gone, the `_inference_dispatch_kwargs` stopgap is deleted, and `WorkerConfig.inference_task_queue` is removed (no backward-compat shim). Deployments can now scale OpenAI vs Anthropic worker pools independently, isolate image generation onto dedicated runners, or route specific OCR backends to dedicated queues — all via TOML config, with no code change required. Default `pipelex.toml` ships with an empty `activity_queues` table, so existing single-queue deployments are unaffected.--- **Per-queue submitter options and named worker-runtime profiles.** Layers on top of the per-activity routing above. (1) `[temporal.queue_options.<queue>]` declares per-queue `start_to_close_timeout`, `schedule_to_close_timeout`, `schedule_to_start_timeout`, `heartbeat_timeout`, `retry_policy_config`, and cluster-wide `max_task_queue_activities_per_second`. (2) `activity_queues.<activity>.handle_options.<handle>` declares rare per-handle overrides for a single model/backend variant. (3) `[temporal.worker_runtime_profiles.profiles.<name>]` declares concurrency slots, pollers, heartbeat throttles, worker-local rate cap, and graceful shutdown timeout; one worker process selects one profile via `pipelex worker --profile <name>`. The new `WorkerConfig.resolve_dispatch(activity_name, routing_key)` composes baseline → `queue_options[resolved_queue]` → `handle_options[routing_key]` last-wins for scalars, and unions `non_retryable_error_types` additively across all three layers (baseline main list + queue `_extra` + handle `_extra`). The `RetryPolicyConfig` schema split into a baseline class (owns `non_retryable_error_types`) and an overlay class (owns `non_retryable_error_types_extra`); `extra="forbid"` rejects the wrong field on each layer so the additive composition rule is enforced at config load. Every `workflow.execute_activity(...)` site in `ContentGeneratorInWorkflow` now splats `worker_config.resolve_dispatch(...).to_execute_kwargs()` — fixing a long-standing bug where the workflow-level `workflow_execution_timeout` was used as every activity's `start_to_close_timeout` (a 1h budget on a jinja2 render, same 1h on a slow PDF extract).--- **Strict `--task-queue` validation at worker CLI startup.** `pipelex worker --task-queue X` now fast-fails with `WorkerTaskQueueUnknownError` (and a Levenshtein "Did you mean?" suggestion) when `X` isn't declared in `default_task_queue`, any `activity_queues` entry, or any `queue_options` key. Strict counterpart to the lenient config-load WARN on routing entries that name unknown queues.--- **Dispatch resolution tracing.** Set `temporal.temporal_config.temporal_log_config.is_dispatch_resolution_traced = true` to emit one INFO log line per `workflow.execute_activity` call, including the resolved queue, timeout, retry attempts, and the layer (`baseline` / `queue_options` / `handle_options`) each scalar came from. Off by default.--- **Specialized worker scopes.** New `runner-llm`, `runner-img-gen`, `runner-extract`, `runner-jinja2` scopes under `[temporal.worker_scopes.scopes]` for deployment manifests that want one worker pool per backend class. Each scope registers only its activities (e.g. `runner-llm` registers `act_llm_gen_text`, `act_llm_gen_object`, `act_llm_gen_object_list`). `act_render_page_views` is registered under both `runner-img-gen` and `runner-extract` (belt-and-suspenders for the two paths that need it).--- **`temporal.worker_config.task_queue` renamed to `default_task_queue`.** Migration map entry under `[migration.migration_maps.config]` maps the old key. New required field `default_activity_start_to_close_timeout` (baseline activity timeout, default `"0:10:00"`) replaces the previous accidental reuse of `workflow_execution_timeout` for activities.--- **Cluster-wide queue rate cap moved from Python constant to TOML overlay.** `TemporalTaskManager.make_worker` previously hardcoded `max_task_queue_activities_per_second=1000` on the Temporal `Worker(...)` constructor for every queue. It now reads this knob from `queue_options[task_queue].max_task_queue_activities_per_second`; the shipping default `[temporal.queue_options.temporal_task_queue]` sets it to `1000`, so out-of-the-box behavior on the default queue is unchanged. Deployments using non-default queue names should add their own `[temporal.queue_options.<queue>]` entries with the cap appropriate for that backend pool. The per-worker `max_activities_per_second = 1000` lives on the runtime profile and is unchanged.--- **Removed dead code: `pipelex/temporal/wrapper/`.** The `start_tprl_activity` wrapper had zero callers.-- **Retry removed from the gateway workers; `[cogt.tenacity_config]` removed (breaking).** The `tenacity`-based retry inside `GatewayExtractWorker` and `GatewaySearchWorker` is gone, along with the `[cogt.tenacity_config]` config block and its `TenacityConfig` model. Because config models forbid unknown keys, an existing `~/.pipelex/pipelex.toml` (or any layered override) that still carries `[cogt.tenacity_config]` will fail to load — remove that block from your config. Transient transport blips are left to the provider SDK clients' own retry; the `tenacity` library itself is still used elsewhere (FAL job polling, remote-config fetch) and remains a dependency.- **`PipeRouter.run()` now reports `CogtError` failures to the observer.** A `CogtError` raised out of pipe execution previously propagated past `run()` without an `observe_after_failing_run` notification (only `PipeRunError` was caught). It is now observed on the failing path, then re-raised as-is — the cause chain is preserved and it is not wrapped into `PipeRouterError` (only the `PipeRunError` path still wraps).@@ -432,14 +353,12 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli- **Inference-failure `ErrorReport`s now carry `model` and `provider` in production.** A real LLM / image-gen / extract / search failure used to surface an `ErrorReport` with `model = None` and `provider = None`: the leaf errors (`LLMCompletionError`, `ImgGenGenerationError`, `ExtractJobFailureError`, `SearchJobFailureError`, `ExtractOutputError`) carry no model or provider of their own, and `CogtError.to_error_report()` only duck-typed whatever attributes happened to be set on the exception. Each inference worker family now fills `model_handle` / `backend_name` from the worker — where both are unambiguously known — at its public-method chokepoint (`gen_text` / `gen_object`, `gen_image` / `gen_image_list`, `extract_pages`, `search_sourced_answer` / `search_structured`), via the new `CogtError.fill_model_and_provider()`. The fill never overwrites a value an inner error already set and skips the `"unknown"` placeholder external plugins report. `model_handle` / `backend_name` are now declared on `CogtError` (so `to_error_report()` reads them directly instead of `getattr`), making them uniformly `str | None` across the exception hierarchy.-- **Two worker error-classification miscategorizations corrected.** `LinkupNoResultError` (a search or fetch that returned nothing) had no explicit branch in either Linkup worker's `_classify_linkup_error` and fell through to the `TRANSIENT` catch-all — marking a query that cannot succeed by retrying as retryable. It is now classified `CONTENT` + `CHANGE_INPUT` in both the search and the extract worker. **Behavior change:** a no-result search is now non-retryable (`CONTENT.is_retryable` is `False`), so Temporal's activity retry policy no longer retries a query that returns nothing on every attempt. Separately, the `FileNotFoundError` branch in the Docling and pypdfium2 extract workers was classified `CONFIGURATION` — inconsistent with its sibling branches (`CONTENT` + `CHANGE_INPUT`) and with the rest of the codebase, where `CONFIGURATION` is reserved for setup problems. A missing input file is a content problem; it is now `CONTENT`. This second change does not alter retry behavior — `CONFIGURATION` and `CONTENT` are both non-retryable.+- **Two worker error-classification miscategorizations corrected.** `LinkupNoResultError` (a search or fetch that returned nothing) had no explicit branch in either Linkup worker's `_classify_linkup_error` and fell through to the `TRANSIENT` catch-all — marking a query that cannot succeed by retrying as retryable. It is now classified `CONTENT` + `CHANGE_INPUT` in both the search and the extract worker. **Behavior change:** a no-result search is now non-retryable (`CONTENT.is_retryable` is `False`), so an automatic-retry layer no longer retries a query that returns nothing on every attempt. Separately, the `FileNotFoundError` branch in the Docling and pypdfium2 extract workers was classified `CONFIGURATION` — inconsistent with its sibling branches (`CONTENT` + `CHANGE_INPUT`) and with the rest of the codebase, where `CONFIGURATION` is reserved for setup problems. A missing input file is a content problem; it is now `CONTENT`. This second change does not alter retry behavior — `CONFIGURATION` and `CONTENT` are both non-retryable.- **Wrapped exceptions now surface the underlying inference error's classification.** `PipelexError.to_error_report()` enriches its report from the `__cause__` chain, so a `PipelineExecutionError` — or any wrapper around a transient `CogtError` — now reports `error_category`, `retryable`, `model`, and `provider` instead of dropping them. Previously the agent-CLI JSON / markdown error output for a failed pipeline run lost the worker's category and retryability once the error had been wrapped by the pipe operator → router → runner layers, leaving an agent unable to tell a transient failure from a fatal one.- **MTHDS JSON schema no longer leaks the `elaboration_metadata` side-table.** `PipelexBundleBlueprint.elaboration_metadata` carried `Field(exclude=True)` so it stayed out of `model_dump()` / `model_validate()` round-trips, but Pydantic v2's `exclude=True` does not affect `model_json_schema()` — so `derived/mthds_schema.json` ended up declaring a top-level `elaboration_metadata` property plus `$defs/ElaborationMetadata` and `$defs/StepRole` entries, even though the side-table is process-local in the reference runtime. Wrapped the field type with `SkipJsonSchema[...]` so it is hidden from the schema. After regenerating, the three entries are gone from the public schema, matching the spec's silence on them.-- **Cross-process Temporal decode of `ListContent` payloads for dynamic concepts.** `WorkingMemory.dump_for_temporal()` was tagging each list item with `__class__` / `__module__` markers so the receiving worker could rebuild the right subclass for `Anything[]` outputs. Those keys are also kajson's universal-decoder protocol, so the Temporal data converter tried to eagerly bind dynamic-concept classes (e.g. `structured_output_test__Invoice`) at the payload boundary — before the child workflow's per-workflow `ClassRegistry` was loaded. In a true 3-process topology the class is not in the global registry (the child workflow tore down its scoped registry in `finally`), so `kajson.loads` raised `KajsonDecoderError` → `RuntimeError: Failed decoding arguments` and any `--temporal` run that produced a list of dynamic-structured-concept items hung. Renamed the markers to the pipelex-private namespace `__pipelex_class__` / `__pipelex_module__` in `WorkingMemory.dump_for_temporal()` and `_hydrate_list_item()`; kajson's decoder gates strictly on `__class__` (`kajson/json_decoder.py:132`) so pipelex's nested dicts now pass through untouched and class binding stays inside pipelex's hydrator where the per-workflow registry lives. Also extended `CLEAN_JSON_FIELDS_TO_SKIP` in `pipelex/tools/misc/json_utils.py` to strip both marker families from user-facing JSON output.-- **A directory containing a `.pipelex/` config dir is now recognized as a project root.** `.pipelex` was added to `PROJECT_ROOT_MARKERS`, so project-level config (e.g. `.pipelex/inference/backends.toml`) is honored even when the directory has no `.git`, `pyproject.toml`, or other source-project marker. Previously such a directory fell through to the global `~/.pipelex/` config, silently ignoring the project's own overrides — so a backend disabled in the project's `backends.toml` could still demand credentials because the global config (where it was enabled) was loaded instead. The home directory remains excluded from project-root detection, so the global `~/.pipelex/` is unaffected.- **`PipeLLM` outputting a concept that refines the native `JSON` concept no longer crashes with `NameError: name 'Any' is not defined`.** On a LIVE run, such a concept resolves to a structured-output model carrying a `dict[str, Any]` field inherited from `JSONContent`. `SchemaToModelFactory` generates that model as source with `from __future__ import annotations` (every annotation becomes a string) and then rebuilds each class to resolve the string annotations. The rebuild namespace was assembled from the exec'd user types plus a hand-listed `Literal`, but `typing.Any` is a special form, not a `type`, so it was filtered out — `model_rebuild` then raised `PydanticUndefinedAnnotation` evaluating `"dict[str, Any]"`. The rebuild namespace is now the exec namespace itself (minus `__builtins__`), so it carries exactly the names the generated source was written against and cannot drift as codegen emits other typing constructs. Fixed in `pipelex/cogt/content_generation/schema_to_model_factory.py`; covers both the sender path (`make_from_json_schema`) and the cross-process receiver path (`make_types_from_source`).@@ -474,7 +393,7 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli### Fixed-- **Temporal storage delivery no longer fails when `working_memory_raw` contains rehydrated Pydantic instances.** `WorkingMemory.dump_for_temporal()` embeds `__class__`/`__module__` markers on `ListContent` items so the receiving workflow can hydrate them. Kajson's Temporal data converter then eagerly rebuilds those dicts back into `BaseModel` instances (e.g. `PageContent` from `Page[]` outputs) on the activity worker that runs delivery — even though `working_memory_raw` is typed as `dict[str, Any]`. `clean_json_content()` previously walked only dicts/lists/scalars and let `BaseModel` instances reach `json.dumps`, which raised `TypeError: Object of type PageContent is not JSON serializable` and aborted `act_deliver`. `clean_json_content` now reduces any `BaseModel` it encounters via `model_dump(serialize_as_any=True)` (the canonical `smart_dump` path) before continuing the recursive walk.+- **`clean_json_content()` no longer crashes on nested Pydantic instances in `working_memory_raw`.** When `working_memory_raw` (typed `dict[str, Any]`) held rehydrated `BaseModel` instances (e.g. `PageContent` from `Page[]` outputs), `clean_json_content()` walked only dicts/lists/scalars and let `BaseModel` instances reach `json.dumps`, which raised `TypeError: Object of type PageContent is not JSON serializable`. `clean_json_content` now reduces any `BaseModel` it encounters via `model_dump(serialize_as_any=True)` (the canonical `smart_dump` path) before continuing the recursive walk.## [v0.26.3] - 2026-05-06@@ -486,7 +405,7 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli### Fixed-- **`choices` fields no longer fail validation with `'EnumName.MEMBER_NAME'` errors.** A concept declared with `choices = [...]` produces a `Literal[...]` field on the dynamic Pydantic class. That schema is round-tripped through `SchemaToModelFactory.make_from_json_schema` (used to rebuild dynamic models on Temporal workers and to feed structured-output schemas to LLM providers). Previously the round-trip silently re-emitted the field as a plain Python `Enum` class — e.g. `Literal["Strong Match", "Good Match", "Partial Match", "Poor Match"]` became `class Recommendation(Enum): Poor_Match = "Poor Match"; ...`. LLMs filling that schema then returned the enum's Python repr (`"Recommendation.Poor_Match"`) instead of the literal string (`"Poor Match"`), which failed Pydantic validation against the original choice set with errors like `Invalid choice errors: 'recommendation': got 'Recommendation.Poor_Match', expected one of 'Strong Match', 'Good Match', 'Partial Match' or 'Poor Match'`. `_generate_source_from_schema` now passes `enum_field_as_literal=LiteralType.All` to `datamodel-code-generator`, so `enum: [strings]` schema nodes round-trip as `Literal[...]` instead of being regenerated as `Enum` classes. `_exec_source_to_types` now also exposes `Literal` in the rebuild namespace so `model_rebuild` resolves the deferred annotations.+- **`choices` fields no longer fail validation with `'EnumName.MEMBER_NAME'` errors.** A concept declared with `choices = [...]` produces a `Literal[...]` field on the dynamic Pydantic class. That schema is round-tripped through `SchemaToModelFactory.make_from_json_schema` (used to feed structured-output schemas to LLM providers). Previously the round-trip silently re-emitted the field as a plain Python `Enum` class — e.g. `Literal["Strong Match", "Good Match", "Partial Match", "Poor Match"]` became `class Recommendation(Enum): Poor_Match = "Poor Match"; ...`. LLMs filling that schema then returned the enum's Python repr (`"Recommendation.Poor_Match"`) instead of the literal string (`"Poor Match"`), which failed Pydantic validation against the original choice set with errors like `Invalid choice errors: 'recommendation': got 'Recommendation.Poor_Match', expected one of 'Strong Match', 'Good Match', 'Partial Match' or 'Poor Match'`. `_generate_source_from_schema` now passes `enum_field_as_literal=LiteralType.All` to `datamodel-code-generator`, so `enum: [strings]` schema nodes round-trip as `Literal[...]` instead of being regenerated as `Enum` classes. `_exec_source_to_types` now also exposes `Literal` in the rebuild namespace so `model_rebuild` resolves the deferred annotations.## [v0.26.1] - 2026-05-05@@ -498,35 +417,25 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli### Highlights-- **Temporal distributed execution** — Pipelex pipelines can now run as Temporal workflows across worker processes. New `pipelex[temporal]` extra, `pipelex worker` CLI, `PipeRun` layer, and `LibraryCrate` IR for cross-process library propagation.-- **Distributed tracing** — pluggable event-log backends (NDJSON, DynamoDB, in-memory, buffering) emit cross-worker `TraceEvent`s that reassemble offline into a complete `GraphSpec`. Enabled by default; works in single-process mode too.+- **Distributed tracing** — pluggable event-log backends (NDJSON, DynamoDB, in-memory, buffering) emit `TraceEvent`s that reassemble into a complete `GraphSpec`. Enabled by default; works in single-process mode too.### Added-#### Temporal distributed execution+#### Pipeline execution & delivery-- **`pipelex[temporal]` extra and full Temporal integration package (`pipelex/temporal/`)**: includes `temporal_manager`, `temporal_connect`, `temporal_data_converter` (kajson-based JSON), `sandbox_manager`, `task_manager`, and a `pipelex worker` CLI subcommand that boots a worker against a configured queue. Toggle via `[temporal] is_enabled` in `pipelex.toml`. Bumps `temporalio` to `1.23.0`.-- **`PipeRun` layer (`pipelex/pipe_run/`)**: new `PipeRunProtocol` with `PipeRun` (in-process) and `TemporalPipeRun` (workflow-orchestrated) implementations. Wraps pipe execution + delivery in a single orchestration unit. In Temporal mode, `WfPipeRun` orchestrates `WfPipeRouter` as a child workflow followed by delivery activities. `TemporalPipeRun.start()` is non-blocking and returns `workflow_id` + `WorkflowHandle` immediately.-- **`TemporalPipeRouter`**: single router that auto-detects context (top-level vs child workflow) and dispatches accordingly.-- **Delivery framework**: `DeliveryAssignment` model with `WebhookDeliveryAssignment` (HTTP POST) and `StorageDeliveryAssignment` (S3 / local). Deliveries run as Temporal activities with automatic retry, or inline in direct mode.-- **`LibraryCrate` IR (`pipelex/libraries/library_crate.py`, `library_crate_factory.py`)**: serializable intermediate representation that lets a Temporal worker reconstruct the exact set of libraries (concepts, pipes, structures) required to execute a job — no shared filesystem assumed.-- **`StoragePayloadCodec`**: offloads Temporal payloads larger than a configurable threshold (default 1 MiB) to external storage (S3 or local) and replaces them with lightweight URI references. Configurable via `[temporal.payload_codec_config]`. New `pipelex codec serve` CLI launches the codec server consumed by the Temporal Web UI for payload inspection.-- **Schema-based dynamic model construction (`pipelex/cogt/content_generation/schema_to_model_factory.py`)**: rebuilds dynamic Pydantic models on a Temporal worker from the JSON schema embedded in `ObjectAssignment`, instead of relying on cross-process class identity. Cache bounded by an LRU (`_SCHEMA_CACHE_MAX_SIZE = 1024`); codegen serialized through a file lock to kill thundering-herd duplication on long-running workers receiving many distinct schemas. Adds `datamodel-code-generator>=0.55.0` as a runtime dependency.-- **Worker scopes**: `pipelex worker --scope ...` lets a worker subscribe to a subset of activities/workflows so deployments can specialize workers (e.g. inference workers vs orchestration workers).-- **`WorkerConfig.inference_task_queue`**: optional task-queue override for `act_llm_gen_text` so router and inference activities can be deployed on separate worker pools.+- **`PipeRun` layer (`pipelex/pipe_run/`)**: new `PipeRunProtocol` with the in-process `PipeRun` implementation. Wraps pipe execution + delivery in a single orchestration unit.+- **Delivery framework**: `DeliveryAssignment` model with webhook (HTTP POST) and storage (S3 / local) variants, run inline after a pipe completes.+- **`LibraryCrate` IR (`pipelex/libraries/library_crate.py`, `library_crate_factory.py`)**: serializable intermediate representation capturing the exact set of libraries (concepts, pipes, structures) required to execute a job — no shared filesystem assumed.+- **Schema-based dynamic model construction (`pipelex/cogt/content_generation/schema_to_model_factory.py`)**: rebuilds dynamic Pydantic models from the JSON schema embedded in `ObjectAssignment`, instead of relying on class identity. Cache bounded by an LRU (`_SCHEMA_CACHE_MAX_SIZE = 1024`); codegen serialized through a file lock to kill thundering-herd duplication when many distinct schemas arrive. Adds `datamodel-code-generator>=0.55.0` as a runtime dependency.- **Pre-generated `pipeline_run_id` support**: `pipeline_run_setup()`, `PipelineFactory`, and `PipelineManager` accept an optional `pipeline_run_id` parameter for external run-record creation.-- **`workflow_id` in start response**: `PipelexPipelineStartResponse` includes `workflow_id`.-- **Temporal `callbacks` passthrough**: `WorkflowExecutor.start_workflow()` accepts optional Temporal `Callback` objects.#### Distributed tracing (`pipelex/tracing/`)-- **`EventLogProtocol` with pluggable backends**: `NdjsonEventLog`, `DynamoDBEventLog`, `InMemoryEventLog`, `BufferingEventLog`. Emits `TraceEvent`s as pipes start/finish across processes. Configured via `[pipelex.tracing_config]` (`backend = "ndjson" | "dynamodb"`). Tracing is **enabled by default** and decoupled from Temporal — single-process runs also produce a trace.+- **`EventLogProtocol` with pluggable backends**: `NdjsonEventLog`, `DynamoDBEventLog`, `InMemoryEventLog`, `BufferingEventLog`. Emits `TraceEvent`s as pipes start/finish across processes. Configured via `[pipelex.tracing_config]` (`backend = "ndjson" | "dynamodb"`). Tracing is **enabled by default** — single-process runs also produce a trace.- **`pipelex[dynamodb]` extra**: `pip install "pipelex[dynamodb]"` for the DynamoDB event log backend.-- **`writer_id` on `TraceEvent`** (default `"primary"`): every event records which writer (worker / process) produced it, propagated through every backend. Standalone activity worker processes use a distinct `writer_id` so they can emit into the same backend partition as workflow workers without colliding on `(workflow_id, sequence)`. Stamped at construction time (no copy-on-emit). Enables cross-worker dedup and per-writer aggregation.+- **`writer_id` on `TraceEvent`** (default `"primary"`): every event records which writer (worker / process) produced it, propagated through every backend. Separate writer processes use a distinct `writer_id` so multiple writers can emit into the same backend partition without colliding on `(workflow_id, sequence)`. Stamped at construction time (no copy-on-emit). Enables cross-writer dedup and per-writer aggregation.- **NDJSON multi-writer key scheme**: dedup key is `(workflow_id, writer_id, type, sequence)`; sort key is `(workflow_id, sequence, writer_id)` — sequence is primary so the runner's `UsageReportEvent` does not sort before the router's `PipeStartEvent`. The file-handle cache key includes `writer_id`, so two writers for the same `(pipeline_run_id, workflow_id)` cannot share a stale handle.- **DynamoDB sort key format**: `EVENT#{workflow_id}#{writer_id}#{sequence:010d}`.-- **Cross-worker `UsageReportEvent` emission from runner processes**: runner processes that never execute `WfPipeRouter.run()` (so `_event_log_contexts` has no entry for the workflow's `lookup_key`) emit usage events through a process-local event log built from `tracing_config`. The fallback log is cached per process via a `threading.Lock`-guarded factory in `pipelex.tracing.activity_event_log` and keyed by a stable per-process `writer_id = f"act_{pid}_{uuid8}"`. A single one-shot warning per process records that the fallback engaged. Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught and logged at WARNING — never silently dropped, never re-raised into the inference path.-- **`ReportingManager` registry lookup split into `_get_registry_strict` and `_get_or_create_registry`**: strict raises on miss and is used by `_report_*_job`, so runner processes silently skip the registry add when `open_registry` was never called; or-create is used by `inject_tokens_usages`, the console cost-report path, and `generate_report` for never-opened runs. Activity workers do not accumulate orphan registries.- **`ContentGeneratorDry` emits `UsageReportEvent`s**: invokes `report_inference_job` with a synthetic `LLMTokensUsage` so dry-run mode produces real events observable through the same backend path as live runs.- **`GraphSpecAssembler`**: rebuilds a complete `GraphSpec` offline from a stream of `TraceEvent`s — equivalence with the live `GraphTracer` is pinned by tests.- **`UsageAggregator`**: aggregates per-pipeline LLM/extract/img-gen usage across workers from the event log.@@ -541,14 +450,14 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli#### Documentation-- New "Under the Hood" pages: `pipe-routing-and-execution.md`, `temporal-integration.md`, `distributed-content-generation.md`. Cover routing/dispatch, Temporal architecture, and the dynamic-model + storage-codec serialization story.+- New "Under the Hood" pages: `pipe-routing-and-execution.md` and `distributed-content-generation.md`. Cover routing/dispatch and the dynamic-model serialization story.- New `AGENTS.md` at repo root, generated from the same source as `CLAUDE.md` via `make rules`.#### Other-- **`PipeOutput.graph_assembly_error`**: optional `str` field set when the Temporal `act_assemble_graph` activity raises so consumers can distinguish "graph never produced" from "graph assembly failed". Previously the failure was logged as a warning and `graph_spec` silently stayed `None`.+- **`PipeOutput.graph_assembly_error`**: optional `str` field set when graph assembly raises, so consumers can distinguish "graph never produced" from "graph assembly failed". Previously the failure was logged as a warning and `graph_spec` silently stayed `None`.- **`pipe extract` web URL support** with full test coverage in `tests/integration/pipelex/pipes/pipe_operators/pipe_extract/`.-- **`needs_inference` flag on CLI commands**: non-inference subcommands (e.g. `worker`, `doctor`) skip inference setup, shaving cold-start time.+- **`needs_inference` flag on CLI commands**: non-inference subcommands (e.g. `doctor`) skip inference setup, shaving cold-start time.- **Environment-specific config**: `RunEnvironment` is now loaded from the `ENVIRONMENT` env var (was `ENV`). Accepted values: `local`, `dev`, `staging`, `prod`.### Changed@@ -557,7 +466,7 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli- Anthropic: bare `"credit"` token replaced with `"credit balance"`, `"out of credits"`, `"insufficient credit"`. No longer misclassifies `"your credit card was declined"` or `"we will credit your account"` as quota exhaustion.- Google: bare `"billing"` token replaced with `"billing limit"`, `"billing quota"`, `"billing exceeded"`, and `"billing account"`. Continues to match real 429-class messages (`"Billing account is not active"`, `"billing limit reached"`); stops matching unrelated billing-mentioning text.- Mistral / Gateway: bare `"billing"` likewise narrowed to `"billing limit"` / `"billing quota"`. Mistral additionally distinguishes `"out of credits"` vs `"insufficient credits"`.-- **Removed duplicate `ContentGenerationError` class**: the unused `pipelex.cogt.content_generation.exceptions.ContentGenerationError` was deleted (had zero external imports). Its subclasses re-parented: `NeitherUrlNorDataError` → `PipelexError`, `UnsafeSchemaError` → new `SecurityError`. The temporal `pipelex.temporal.exceptions.ContentGenerationError` is now the only class with that name.+- **Removed duplicate `ContentGenerationError` class**: the unused `pipelex.cogt.content_generation.exceptions.ContentGenerationError` was deleted (had zero external imports). Its subclasses re-parented: `NeitherUrlNorDataError` → `PipelexError`, `UnsafeSchemaError` → new `SecurityError`.- **Storage providers unified**: `StorageConfig` flattened to inheritance-based config so providers (`local`, `s3`) share a single shape across delivery and trace storage.- **`make rules` now generates both `CLAUDE.md` and `AGENTS.md` by default**: the `pipelex.kit_config.preferred_agent_target` setting has been renamed to `preferred_agent_targets` and is now a list. The default is `["claude", "agents"]`. Cursor remains exclusive (`["cursor"]`) and cannot be combined with single-file targets. Downstream projects overriding this setting must rename the key and wrap the value in a list.- **Cold start trimmed**: heavy SDK imports (`boto3`, `huggingface_hub`, etc.) deferred behind `TYPE_CHECKING` or function-local guards across multiple plugins.@@ -584,9 +493,9 @@ This release hardens Pipelex at its edges. The headliners: a full **error-handli### Security-- **`schema_to_model` rejects `x-python-*` codegen extensions and restricts `__import__` to an allowlist.** `datamodel-code-generator` honors the `x-python-import` JSON Schema extension by emitting arbitrary `from <module> import <name>` statements with no sanitization. Combined with the prior gap that `_make_restricted_builtins()` did not block `__import__`, an attacker able to plant a crafted `object_class_schema` on a Temporal payload (where `ObjectAssignment.object_class_schema: dict[str, Any]` round-trips untouched) could cause arbitrary modules to be imported during `exec()` of generated code. Two-layer defense added: (1) `_reject_unsafe_schema_extensions` raises `UnsafeSchemaError` if any `x-python-*` key is present anywhere in the schema; (2) `__import__` in the exec namespace is now wrapped to allow only `pydantic`, `typing`, `typing_extensions`, `enum`, `datetime`, `decimal`, `uuid`, `__future__`, `collections`, and `re`.+- **`schema_to_model` rejects `x-python-*` codegen extensions and restricts `__import__` to an allowlist.** `datamodel-code-generator` honors the `x-python-import` JSON Schema extension by emitting arbitrary `from <module> import <name>` statements with no sanitization. Combined with the prior gap that `_make_restricted_builtins()` did not block `__import__`, an attacker able to plant a crafted `object_class_schema` (where `ObjectAssignment.object_class_schema: dict[str, Any]` round-trips untouched) could cause arbitrary modules to be imported during `exec()` of generated code. Two-layer defense added: (1) `_reject_unsafe_schema_extensions` raises `UnsafeSchemaError` if any `x-python-*` key is present anywhere in the schema; (2) `__import__` in the exec namespace is now wrapped to allow only `pydantic`, `typing`, `typing_extensions`, `enum`, `datetime`, `decimal`, `uuid`, `__future__`, `collections`, and `re`.- **Restricted exec builtins, path sanitization, and atomic fingerprint caching** in the schema-to-model pipeline.-- **CORS hardened on codec server error paths**; no leaked exception strings; prefix guards on storage URIs.+- **No leaked exception strings** on error paths; prefix guards on storage URIs.## [v0.25.2] - 2026-04-30diff --git a/CLAUDE.md b/CLAUDE.mdindex f574e13d4..16d481bfd 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -162,7 +162,7 @@ Non-subject function parameters across `pipelex/` source must be **keyword-only*- `def f(subject, *, opt1, opt2): ...` — the first non-`self`/`cls` parameter (the subject) may stay positional; everything after it must be keyword-only. Making the subject keyword-only too (`def f(*, a, b)`) is always allowed and often preferable.- A lone subject (`def render(node)`) is compliant; a second bare positional (`def f(a, b)`, `def truncate(text, max_length=80)`) is a violation.-The rule is mechanically enforced by the `check-keyword-only` AST guard, which runs in `make agent-check`, in the `make check` aggregate, and in CI; the tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs (dunders, pydantic validators/serializers, Typer/Temporal/pytest/Jinja2 framework entrypoints, `@override` impls) are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line). Watch for functions a framework or the interpreter invokes positionally (callbacks, `__import__` hooks, route handlers): the type checker is blind to those, so `make agent-test` is the safety net.+The rule is mechanically enforced by the `check-keyword-only` AST guard, which runs in `make agent-check`, in the `make check` aggregate, and in CI; the tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs (dunders, pydantic validators/serializers, Typer/pytest/Jinja2 framework entrypoints, `@override` impls) are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line). Watch for functions a framework or the interpreter invokes positionally (callbacks, `__import__` hooks, route handlers): the type checker is blind to those, so `make agent-test` is the safety net.The full specification — the two exceptions, the carve-out list, the symmetric-tuple allowlist, the escape hatch, and worked examples — is in [`docs/contribute/keyword-only-arguments.md`](docs/contribute/keyword-only-arguments.md).diff --git a/Makefile b/Makefileindex 081b908c7..4835feac5 100644--- a/Makefile+++ b/Makefile@@ -703,10 +703,10 @@ agent-test: env# Debug variant of agent-test for when the suite hangs or fails opaquely.# Use this instead of agent-test when:# - a prior `make agent-test` hung (or got killed) without a verdict-# - failures are integration-temporal with worker crashes / "node down" noise+# - failures show xdist worker crashes / "node down" noise# - you need to see what's actually happening during the run# Differences vs agent-test:-# - pkill stale `pytest` + `temporal-sdk-python` processes first (zombies from+# - pkill stale `pytest` processes first (zombies from# prior hung runs compound contention and cause more hangs)# - outer wall-clock `timeout` so fixture-teardown hangs and xdist# worker-replace loops can't run forever (`pytest --timeout` is per-test only)@@ -724,9 +724,8 @@ agent-test-debug: envecho " The outer wall-clock cap is the whole point of this target — install before using."; \exit 1; \fi- @echo "• Cleaning stale pytest + temporal-sdk-python processes..."+ @echo "• Cleaning stale pytest processes..."@pkill -9 -f "pytest" 2>/dev/null || true- @pkill -9 -f "temporal-sdk-python" 2>/dev/null || true@sleep 1@echo "• Running with outer timeout=$(DEBUG_TIMEOUT)s, log: $(DEBUG_LOG)"@echo " Live progress: tail -f $(DEBUG_LOG)"@@ -749,7 +748,7 @@ agent-test-debug: envecho " Full log: $(DEBUG_LOG)"; \echo " Tip: grep failures by error class name, not formatted message —"; \echo " grep -B 2 -A 10 'YourErrorClass' $(DEBUG_LOG)"; \- echo " If failures are integration-temporal with worker crashes, re-run that"; \+ echo " If failures show xdist worker crashes ('node down'), re-run that"; \echo " slice serially: .venv/bin/pytest --timeout=60 -q tests/integration/.../"; \echo " Playbook: docs/agents/debugging-hanging-pytest-runs.md"; \else \diff --git a/TODOS.md b/TODOS.mddeleted file mode 100644index 4341002dd..000000000--- a/TODOS.md+++ /dev/null@@ -1,339 +0,0 @@-# Orchestrator-agnostic runner & deployment flavors — implementation TODOS--**Effort:** make `pipelex-api` an orchestrator-agnostic base, then build N thin deployment flavors on top of it (Temporal today, Mistral next). The *consumer-side* sequel to the plugin-system externalization (that tracker is archived at [`wip/archive/plugin-system-implementation-tracker.md`](wip/archive/plugin-system-implementation-tracker.md)).--**Status:** **Phases A + B + C DONE (committed & pushed) + two post-C follow-on extensions DONE (pushed).** Phase A/B core seams (`pipelex`: `4b5449f88` Phase A, `75196e2e3` seam refinement, `7378afdd8` Phase B docs; `pipelex-temporal` `6d9adff`). **Phase C** = `pipelex-api` made orchestrator-agnostic, commit **`a39841e`** on `refactor/orchestrator-agnostic-base` (cut from `dev`): F1 execution via the orchestrator-registry hub seam (byte-equivalent — see the F1 deviation note in the as-built), F2 validate always DIRECT, F3 exception-handler mappers consumed at app construction, `[api]` config = Option C `api.toml` via `load_plugin_config`, `temporal` extra dropped. Clean-context `/code-review`s + all gates green (308 tests, pyright 0, mypy 0, pylint 10, openapi-check). **MAJOR GATE 2 / Checkpoint C (THE gate) cleared.** **Follow-on extension plans then landed on top of Phase C** (their own plan docs + `feature/*` branches — see "Follow-on extensions" below): orchestrator-dispatched `/validate` (reverses/extends F2), per-request `execution_mode` on `/execute` (extends F1, **PR #27**), and `/start` deriving the fire-and-forget variant of `execution_mode` (`f5e3ef7` + review fixes `7e4bc13`). **NEXT = Phase D** (`pipelex-api-hosted` Temporal flavor — MAJOR GATE 3), unchanged by the follow-ons. The in-`pipelex` plugin-system foundation (seams inverted, Temporal externalized, `plugins.boot_orchestrator` gate) was done earlier on `refactor/Plugins-3`.--**Current branches:** the follow-on work + this tracker live on **`feature/Orchestrator-dispatched-validate`** — `pipelex` (`_plugins`) and `pipelex-temporal` (tip `459e04d`). `pipelex-api` is a stack: `a39841e` (Phase C, `refactor/orchestrator-agnostic-base`) → `72c0efc` (validate, `feature/Orchestrator-dispatched-validate`) → `feature/Execute-per-request-mode` (execute **PR #27** `191c900`, then `/start` f&f derivation `f5e3ef7` + review fixes `7e4bc13`, **tip `7e4bc13`**). The PR #27 stack through `191c900` is pushed/CI-green; `f5e3ef7` + `7e4bc13` and this tracker update are **not pushed**. Phase D onward each lands in its own repo/branch.--### Follow-on extensions (post-Phase-C — DONE; not part of the original A–F phases)--Extension plans written + executed **after** Phase C; each has its own plan doc and `feature/*` branch line. They do **not** change the D/E/F roadmap below.--- **Orchestrator-dispatched `/validate`** — plan [`wip/plugins/orchestrator-dispatched-validate.md`](wip/plugins/orchestrator-dispatched-validate.md). **Reverses/extends F2:** `/validate` is now `execution_mode`-aware via a NEW per-call `BundleValidatorRegistry` seam (mirrors `OrchestratorRegistry`) — `direct` validates in-process (F2 kept for the agnostic base), `temporal_*` dispatches the whole job to a worker. Verdict-as-value (`ApiRunner.validate` override removed → `validate_verdict`; route discriminates on `isinstance(verdict, PipelexValidationReport)`). Landed across all 3 repos on `feature/Orchestrator-dispatched-validate`: core PR **#998** (CI-green + Greptile-happy), `pipelex-api` tip **`72c0efc`**, `pipelex-temporal` tip **`459e04d`**. **DONE.**-- **Per-request `execution_mode` on `/execute`** — plan [`wip/plugins/execute-per-request-execution-mode.md`](wip/plugins/execute-per-request-execution-mode.md) (+ `execute-per-request-mode-deferred.md`). **Extends F1:** `/execute` now dispatches by resolved `execution_mode` through the `OrchestratorRegistry` (symmetric with `/start`), rejects fire-and-forget with a 400, and rehydrates the orchestrator's JSON-safe output back into the full `PipeOutput`. **`pipelex-api`-only** (reuses the existing seam — no core/temporal change). **PR #27** (`feature/Execute-per-request-mode` → `feature/Orchestrator-dispatched-validate`, tip **`191c900`**): CI green, Greptile 5/5, gstack `/review` merge-ready; **not merged (user's call).** Deferred items in `wip/plugins/execute-per-request-mode-deferred.md` (DIRECT round-trip cost; pre-existing `/start` resource leak; `/start` schema gap + dangling-`#/$defs/` OpenAPI ref pattern; failing-pipe error double-wrap via the shared seam).- - **CI gotcha (any `pipelex-api` PR on these branches):** the editable `pipelex = ../_plugins` pin breaks CI (no sibling worktree on the runner). Fix = pin core to a **git+https SHA** in `pyproject.toml` `[tool.uv.sources]` (core repo is PUBLIC → no creds) + relock `uv.lock`; flip back to the PyPI `==` pin at release. Done on PR #27 as `c60be1f` (pinned to core `51ff09417`).-- **`/start` derives the fire-and-forget variant of `execution_mode`** — commit **`f5e3ef7`** (+ review fixes **`7e4bc13`**) on `feature/Execute-per-request-mode`, stacked on PR #27. **Endpoint, not deployment:** fire-and-forget vs blocking is a property of the *endpoint*. A deployment configures ONE *synchronous* `execution_mode`; `/execute` + `/validate` dispatch it as-is, while `/start` derives its f&f sibling (`temporal_blocking → temporal_fire_and_forget`; `direct`/`mistral_native` have none → dispatched unchanged, blocking). Kills the `/execute` 400 / forced-direct footgun — one coherent mode across all three endpoints. Route-local `_async_start_mode` (kept off the core enum per smallest-surface). **`pipelex-api`-only** (no core/temporal change). **xhigh-workflow `/code-review` + fixes (`7e4bc13`):** (1) `ApiConfig` now rejects a *configured* f&f `execution_mode` (field validator on `is_fire_and_forget`) — the docs declared it unconfigurable but nothing enforced it (a baked f&f would 400 every `/execute` + misroute `/validate`); fail-fast at load, per-request f&f override on `/start` still allowed (policy-gated). (2) Corrected the `mistral_native` `workflow_id` claim in the `_async_start_mode` docstring + call-site comment + `configuration.md` — `mistral_native` runs per-call and answers with a *non-null* `workflow_id` (the run id); only `direct` returns `None`. (3) Added a pure-mapping test of `_async_start_mode` over every mode + a `mistral_native` end-to-end `/start` case (both previously unverified despite the test docstring claiming the coverage); `test_api_config`'s locked-Temporal helper switched to the synchronous `temporal_blocking` + a configured-f&f-rejection test added. Gates green (ruff/pyright 0/mypy 0/pylint 10, full unit suite pass, OpenAPI in sync). Review verdict: the F1-gate ordering (403 runs *before* the f&f derivation) and single-caller transform are correct; deferred low-priority items (route-local-vs-enum altitude; a custom blocking-only orchestrator plugin now needs a f&f arm for `/start`) noted as by-design. **DONE; not pushed.**-- **DONE (Phases 1–4) — re-cut `execution_mode` into `orchestration_mode` (open string token) + `DeliveryMode` (closed enum)** — plan [`wip/plugins/orchestration-mode-and-delivery-split.md`](wip/plugins/orchestration-mode-and-delivery-split.md). **Phase 1 (core, `_plugins` `bd4e52827`) + Phase 2 (`pipelex-temporal` `01d7dc3`) + Phase 3 (`pipelex-api` `dd84a9a`, THE gate) landed on `feature/orchestration-mode-delivery-split` in each repo — gates green + clean-context `/code-review` each (Phase 3: no must-fix). Phase 4 = the external `pipelex-mistralai-workflows` plugin migrated to the new core surface (deleted-enum import + registration under the deleted member + own `execution_mode` wire field + old `run` signature → `orchestration_mode`/`delivery`/new protocol), commit `c2013fb` on `feature/Mistral-native`, green against the merged core (agent-check + full integration suite). §8 `/start`-honesty risk resolved (honest 400 safe). DEVIATION (Phase 4): the Mistral plugin pins core editable at `../_workflows` (branch `feature/mistralai-2x-bump`), NOT the orchestration-split worktree — so Phase 4 first required merging `feature/orchestration-mode-delivery-split` into `feature/mistralai-2x-bump` (`_workflows` merge commit `a1448ad8f`, conflict-free; core gates green) to give the plugin a core carrying BOTH the split and the mistralai-2x bump. Spec/conformance untouched (no `docs/specs/` heading or `conformance/` test asserts mode values → `make check-spec-links` not triggered). See the plan's §10 Phase 4 as-built.** **Supersedes the flat-`PipelexExecutionMode` framing of the two bullets above:** it removes the conflation those plans worked around — `_async_start_mode`, the `/execute` fire-and-forget 400, the `ApiConfig` f&f field-validator, and the `/validate` double-registration. Two orthogonal axes: an **open** orchestration token (core owns only `"direct"`; plugins own `"temporal"` / `"mistralai-workflows"` — realigns with "the base names no orchestrator") + a **closed** `DeliveryMode {BLOCKING, FIRE_AND_FORGET}` that endpoints set, never receive. `/start` becomes **honest** (4xx when the resolved mode can't do async) instead of degrading. Design **locked 2026-06-22**; Direction A (core → `pipelex-temporal` → `pipelex-api`, one repo/branch/commit per phase). **Top risk to settle before the API phase:** does the MTHDS-Protocol/conformance suite require `/start` to return `202` on a non-async (`direct`) backend? See the plan's §8.-------This is the **execution tracker**. The *why* and *how* live in [`wip/plugins/orchestrator-agnostic-runner-and-flavors.md`](wip/plugins/orchestrator-agnostic-runner-and-flavors.md) (the reviewed, decision-locked plan). **Read that doc before starting any phase** — this file tracks progress against it; it does not replace it. **Resuming in a fresh session?** Start with [`wip/plugins/resume-and-verify-state.md`](wip/plugins/resume-and-verify-state.md) — paste-ready orientation + commands to verify the state against the repos before doing anything. Adjacent background: [`wip/plugins/temporal-config-out-of-core.md`](wip/plugins/temporal-config-out-of-core.md) (the boot-gate refactor that immediately precedes this work) and the plugin-system design/SPI docs in [`wip/plugins/`](wip/plugins/).--> **How to use this file.** Tick boxes as you land work. **Do not skip a `🛑 CHECKPOINT`** — each is a hard stop with three mandatory actions (verify · capture cold-start context · fan-out `/code-review`). Three checkpoints are flagged **MAJOR GATE** — they are the points the source plan calls out as the ones that must hold before the effort can continue.->-> **Fan-out convention for `/code-review`.** Spawn each review sub-agent with **no inherited context** — hand it only a pointer to the changes under review (the phase's commit SHA, `git diff <base>..HEAD`, or the unstaged/working-tree files), never the plan, the rationale, or your own conclusions. A clean-context reviewer reviews the code as written, not your story about it — that independence is the point of fanning it out.-------## Cold-start primer (read this first if you're new to the session)--**Goal of the whole effort:** today `pipelex-api` is wired directly to Temporal in production code — it is *not* an orchestrator-agnostic runner. Make the public `pipelex/pipelex-api` base name **no** orchestrator (ship only the in-process/direct path), then express each deployment as **base image + exactly one orchestrator plugin + that plugin's activation**. That is what lets a Temporal flavor (`pipelex-api-hosted`) and a future Mistral flavor (`pipelex-api-mistral`) both exist on one base.--**The model shift — internalize before touching any repo.** The plugin system split a previously-monolithic idea ("Temporal is on") into orthogonal concepts:--- **Presence** = the plugin distribution is installed, so its `pipelex.plugins` entry point is discovered at boot and `register()` runs, contributing its `execution_mode` arms to the orchestrator registry. Presence makes a mode *dispatchable*; it does **not** make it *used*.-- **Activation** = the runtime is told to use it. It decomposes into **two independent knobs that are not derivable from each other:**--| Knob | Owns | Set by | Temporal | Mistral | Direct |-|---|---|---|---|---|---|-| `execution_mode` | which mode a *top-level* run dispatches as, through the bridge | runner config default + optional per-request override | `temporal_blocking` / `temporal_fire_and_forget` | `mistral_native` | `direct` |-| `plugins.boot_orchestrator` | whether to claim the process-global hub slots (content generator / pipe router / pipe run / task manager) at boot | deployment config TOML | `"temporal"` | *unset* | *unset* |--**Mistral is the proof the two knobs are independent.** A Temporal deployment needs *both* (the boot gate stands up the Temporal client/task-manager the orchestrator enqueues against; `execution_mode` tells the bridge to enqueue). A Mistral deployment needs **only** `execution_mode = mistral_native` — its plugin claims **no** hub slots (self-contained per-call orchestrator), so there is no boot gate to set. A plain in-process deployment sets neither.--**Boot-global vs per-call orchestrators** — *why* the two knobs exist. A **boot-global** orchestrator (Temporal) replaces process-wide execution machinery → must claim hub slots at boot, gated by `boot_orchestrator`. A **per-call** orchestrator (Mistral) only adds a dispatch arm → presence is enough; selection is per-run via `execution_mode`. Classify any new orchestrator first; the classification decides whether `boot_orchestrator` is even meaningful for it.--**Install channel follows license, not convenience.**--- `pipelex-temporal` is **private** → pinned `git+ssh` ref (precedent: `pipelex-shared @ git+ssh://…/infra-python-tools.git@<ref>`). Only **private** images may pin it.-- `pipelex-mistralai-workflows` is **public** → plain PyPI (`pipelex-mistralai-workflows==X`). Currently pre-publish; gated on its own `mistralai-2.x` work — treat as PyPI once published.--**Verified anchors (2026-06-21, this worktree)** — the plan's foundations are present and current:--- `plugins.boot_orchestrator` gate exists: `configs.py:258`, applied in `pipelex.py:206-207` (only overridden when the param is non-`None`, so a TOML value is **preserved** — D1 safe).-- Plugin contract: `PLUGIN_API_VERSION = 1` (`pipelex/plugins/contract.py:10`); `PluginRegistrar` already hosts `add_inference_backend` / `add_model_lister` / `add_orchestrator` / `claim_*` / `add_teardown` (no `add_cli_command` — removed by Option A). F3 adds one method here.-- `ErrorReport` is a core type (`pipelex/base_exceptions.py:309`) → F3's mapper signature uses it (no FastAPI in core).-- Config loader env-layering pattern to mirror for D2: `ConfigManager._override_files_for_dir` / `load_config` (`pipelex/system/configuration/config_loader.py`), keyed on `runtime_manager.environment`.-- The §2 gap is real and current: `pipelex-api/api/` still names temporal in `exception_handlers.py`, `routes/pipelex/validate.py`, `routes/pipelex/pipeline.py`. All sibling repos exist (`pipelex-temporal`, `pipelex-api`, `pipelex-api-hosted`, `pipelex-worker`, `pipelex-mistralai-workflows`).--**Locked decisions (D1, D2, F1, F2, F3)** — do not re-litigate; if you believe one is wrong, raise it explicitly before deviating. Full text in §3 of the source doc.--- **D1** — `boot_orchestrator` is set from the flavor's deployment TOML (`[plugins] boot_orchestrator = "temporal"`). No `pipelex-api` change for the gate; verified preserved by `setup()`.-- **D2** — plugin config files become env-aware, mirroring the main config. Add a **reusable core helper** so every plugin self-loads `name.toml` → `name_{env}.toml` → `name_override.toml` with deep-merge. `pipelex-temporal` adopts it. One image bakes all envs; `PIPELEX_ENV` selects at runtime.-- **F1** — `execution_mode` = runner config default + optional per-request override, gated by a per-deployment policy (a locked-down Temporal runner can refuse a caller forcing `direct`). Runner builds `PipelexPipeRunInput(execution_mode=…)` and dispatches via `run_pipe_via_bridge`; the bridge output already carries `workflow_id`, so async-start survives.-- **F2** — validate is **always DIRECT in-process** (`validate_bundles_in_process`). Drops the Temporal validate branch. **Behavior change:** the API runner now loads the method library to validate (prior design pushed that onto the worker) — document the resource/isolation implication on the hosted runner. **→ REVERSED / EXTENDED** by orchestrator-dispatched `/validate` (new per-call `BundleValidatorRegistry` seam): `/validate` is now `execution_mode`-aware — `direct` in-process (F2 kept for the agnostic base) | `temporal_*` dispatched to a worker (restored). Same canonical verdict either way. See [`wip/plugins/orchestrator-dispatched-validate.md`](wip/plugins/orchestrator-dispatched-validate.md).-- **F3** — orchestrator-specific HTTP error handling via a **new framework-agnostic plugin seam**: `PluginRegistrar.add_http_error_mapper(*, exc_type, to_error_report: Callable[[Exception], ErrorReport])`. The orchestrator plugin registers its transport-fault mapping; `pipelex-api` iterates the mappers at app construction and wraps each into a FastAPI handler using its existing RFC 7807 + `DisclosureMode` rendering. **FastAPI/Starlette stays only in `pipelex-api`; core and the plugin import neither.** Grows the contract by one optional capability → bump `PLUGIN_API_VERSION`.--**Delivery model (cross-repo).** Unlike the archived plugin-system tracker (one repo, one-commit-per-checkpoint, single PR), this effort spans repos. **Each phase lands in its own repo on its own branch as its own commit(s) + PR — there is no single cross-repo PR.** Ordering is load-bearing: the `pipelex` (A) and `pipelex-temporal` (B) enabling changes must land before `pipelex-api` (C) can consume the F3 seam.--**Local cross-pin strategy (for dev before anything is published/released):** pin editable across the chain so the seam can be exercised end-to-end locally — `pipelex-temporal` → editable `pipelex` (`[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }`), and `pipelex-api` → editable `pipelex` (+ a synthetic in-test plugin to prove the F3 seam without importing `temporalio`). The published `==` / git+ssh pins are the release-time flip (Phases D–F).--**Commands** (from `_plugins/CLAUDE.md`, for the `pipelex` Phase-A work): `make agent-check` (lint/types — always after changes) · `make agent-test` (full suite, silent on success — at end of phase) · `make tb` (boot/config sanity — fast, after config or seam edits). Always use the venv: `.venv/bin/...`. Consumer repos use their own `make agent-check` / `make agent-test` (same family conventions).-------## Image architecture (target — handles public + private plugins uniformly)--A **two-layer** build per flavor isolates plugin-install complexity (private SSH, version-locking) into one place and keeps the env-config layer trivial:--```-pipelex/pipelex-api:<ver> (public base, orchestrator-agnostic, Docker Hub)- ├─ flavored base = base + ONE plugin (built once per flavor → private ECR)- │ pipelex-api-temporal:<ver> = base + pipelex-temporal (git+ssh)- │ pipelex-api-mistral:<ver> = base + pipelex-mistralai-workflows (PyPI)- └─ env-config child = flavored base + COPY env TOMLs (per env; config-only)- pipelex-api-{dev,staging,prod} (Temporal)- pipelex-api-mistral-{dev,staging,prod}-```--**One parameterized install recipe.** Flavored-base Dockerfile takes `PLUGIN_SPEC` and always exposes a BuildKit SSH mount. PyPI plugin → plain install (SSH mount unused). Private plugin → `--mount=type=ssh` with the deploy key in the CI builder (`docker build --ssh default`); never bake a key into a layer. Install mechanics that bite (encode in the Dockerfile): base Python is a uv project venv at `/app/.venv` (install into it; don't let the resolver swap the pinned `pipelex`); the base purges `git`/`build-essential` in a late layer (re-add `git` + `openssh-client` before installing the private plugin); the plugin ref and base `pipelex` version are a **coupled pair that bump together** — track it in the flavor's version file.-------## Phase A — Core enabling changes (`pipelex`, the `_plugins` worktree)--**Goal:** add the two seams `pipelex-api` will consume, in the public core, with no orchestrator named and no FastAPI imported. Lands on `refactor/Plugins-3`.--- [x] **D2 — env-aware plugin config loader.** Add a reusable helper on `ConfigManager` (e.g. `load_plugin_config(*, name, schema)`) resolving `name.toml` (the calling plugin's packaged default) → `name_{environment}.toml` → `name_override.toml` with deep-merge, mirroring `load_config`'s env layering. Anchor: `pipelex/system/configuration/config_loader.py` (`_override_files_for_dir` / `load_config`, keyed on `runtime_manager.environment`).-- [x] **F3 — HTTP-error-mapper seam.** Add `add_http_error_mapper(*, exc_type: type[Exception], to_error_report: Callable[[Exception], ErrorReport])` to `PluginRegistrar` (`pipelex/plugins/registrar.py`, alongside the existing menu methods) plus a read accessor for the collected mappers. Signature uses `ErrorReport` (`pipelex/base_exceptions.py`) — **no FastAPI import.**-- [x] **Bump `PLUGIN_API_VERSION`** (`pipelex/plugins/contract.py:10`, `1 → 2`) and document the new optional capability in the orchestrator SPI doc (`docs/under-the-hood/orchestrator-plugins.md`). Note the contract bump means every plugin's `targets_api` must be re-confirmed (built-ins + `pipelex-temporal` in Phase B).-- [x] **Tests:** env-layered plugin-config round-trip (packaged default → `_{env}` → `_override` deep-merge); registrar collects/returns mappers; import-light boot still green (the seam adds no SDK/HTTP import — extend the existing import-light subprocess guard if needed).--### 🛑 CHECKPOINT A — hard stop (core seams exist; consumed at B/C)--- [x] **Verify:** `make agent-check` clean · `make tb` green · `make agent-test` green. Core still names **no** orchestrator and imports **no** FastAPI (`grep -rn "fastapi\|starlette" pipelex/` → only telemetry labels + `WorkerMode.FASTAPI` enum, no imports; temporal refs in core unchanged). The seam's *end-to-end* exercise lands at Checkpoint B (when the plugin contributes through it) — don't expect integration proof here.-- [x] **Capture cold-start context:** append `### Phase A — as-built` below (final helper name/signature, the registrar method + accessor signature, the `PLUGIN_API_VERSION` value, which plugins' `targets_api` still need re-confirming, test locations).-- [x] **Fan-out `/code-review`:** sub-agent runs `/code-review` on the Phase A `pipelex` diff. Triage findings into the as-built (apply cheap ones; defer design-tradeoffs to a `wip/plugins/` follow-up per the deferral convention). _(done — verdict clean; nit #2 applied; tradeoff #1 deferred to `wip/plugins/phase-c-http-error-mapper-consumer-path.md`)_-- [x] **Commit** on `refactor/Plugins-4`. This is a natural session boundary before switching to the `pipelex-temporal` repo.-------## Phase B — `pipelex-temporal` adopts the new seams (additive, private repo)--**Goal:** the private Temporal plugin uses the D2 helper and contributes through the F3 seam. Additive — no behavior change to its existing orchestrator/slot-claim contributions.--- [x] **D2 adoption.** Switch `load_temporal_config()` to the new helper; ship `temporal_{env}.toml` resolution (the packaged default `temporal.toml` stays). One image bakes all envs; `PIPELEX_ENV` selects.-- [x] **F3 mapper.** In `TemporalPlugin.register()`, register `temporalio.TemporalError → ErrorReport` classified transient / `RUNTIME` (port the classification currently in `pipelex-api`'s `handle_temporal_error`). Keep `register` import-light — via a lazy `exc_type_provider` thunk + a lazy SDK import, `register` imports no `temporalio` (the Phase A seam was refined to take the provider; see Phase A as-built "Why lazy").-- [x] **Re-confirm `targets_api`** matches the new `PLUGIN_API_VERSION` from Phase A. (`targets_api = PLUGIN_API_VERSION`, auto-tracks to 2 via the editable pin — verified by the suite running against the Phase A core.)-- [x] Pin editable `pipelex` (`../_plugins`) for local testing against the Phase A core. (Already present in `pyproject.toml` `[tool.uv.sources]`.)--### 🛑 CHECKPOINT B — hard stop · **MAJOR GATE 1** (source-doc Checkpoint 1)--- [x] **Verify:** `pipelex-temporal` `make agent-check` + `make agent-test` green against the Phase A `pipelex`. The seam now has a real consumer: a `pipelex-temporal` install contributes its mapper through `add_http_error_mapper` and self-loads `temporal_{env}.toml`. Import-light `register` still proven (no `temporalio` at registration) — subprocess blocks `temporalio` and runs `register` to completion.-- [x] **Capture cold-start context:** `### Phase B — as-built` appended below.-- [x] **Fan-out `/code-review`:** two forked `/code-review`s (core seam refinement + pipelex-temporal diff) — both **clean, no bugs**. Triage recorded in the Phase B as-built (one NIT applied: slot-claim count in the import-light test; rest deferred).-- [x] **Commit** in `pipelex-temporal` (`refactor/own-temporal-config`, `6d9adff`). Core seam refinement + Phase B docs on `refactor/Plugins-4` (`75196e2e3`, `7378afdd8`).-------## Phase C — `pipelex-api` decoupling (the big one, public Docker Hub)--**Goal:** make the base orchestrator-agnostic. **No `pipelex.temporal.*`, no `temporalio`, anywhere in `api/`.** Each of F1/F2/F3 removes one of the §2 coupling sites.--- [x] **Execution (F1).** Rewrite `ApiRunner.start()` (`api/routes/pipelex/pipeline.py`). **DEVIATION (flagged in as-built):** dispatch via `get_orchestrator_registry().get_optional(mode=execution_mode).run(pipe_job, delivery_assignment)` (the identical final dispatch `run_pipe_via_bridge` performs) fed the **rich** `pipeline_run_setup` job — NOT `PipelexPipeRunInput`+`run_pipe_via_bridge`, which is lossy (drops `request_id`/`output_multiplicity`/`dynamic_output_concept_ref`/registration/telemetry) and would need a forbidden 2nd core change to fix. `workflow_id` off `run_output`. `execution_mode` from `api.toml` (Option C) + per-request override gated by `allow_request_execution_mode_override`. `make_temporal_pipe_run` import/usage gone. (`execute` was already agnostic via the `get_pipe_run()` hub slot — untouched.)-- [x] **Validate (F2).** Collapsed `ApiRunner.validate()` to the single `validate_bundles_in_process` path. Deleted the `temporal.is_enabled` branch + `dispatch_dry_validate`/`DryValidateArg`/`PipelexInterpreter`/`build_validation_report` imports + the route's `WorkflowExecutionError` catch. Behavior change documented in `pipelex-api/docs/pipe-validate.md`.-- [x] **Exception handlers (F3).** Removed the `temporalio` import + hardcoded `handle_temporal_error`. `register_exception_handlers(…, http_error_mappers=…)` registers one handler per `exc_type` via `_make_orchestrator_error_handler` → existing `_problem_response`. `api/main.py` resolves the mappers at import via `build_registrar(...).get_http_error_mappers()` (deferred-doc option 1 — no 2nd core change). `handle_pipelex_error` kept.-- [x] **Deps.** Dropped the `temporal` extra from `pyproject.toml`; removed the `[temporal]` block from the base `.pipelex/pipelex.toml`. Base depends on **no** orchestrator plugin.-- [x] **Tests.** `test_exception_handlers.py` rewritten to the synthetic-plugin shape (real `PluginRegistrar` + `add_http_error_mapper` + `get_http_error_mappers`, no `temporalio`); `test_validate_*` temporal arms removed; conformance webhook test mocks `get_orchestrator_registry`. New `test_api_config.py` covers the default + override policy (forbidden → 403, allowed honored, same-as-default ok) at the resolver AND end-to-end on `POST /start`.--### 🛑 CHECKPOINT C — hard stop · **MAJOR GATE 2 — THE gate** (source-doc Checkpoint 2)--Flavors are meaningless until the base is clean. This is the decisive checkpoint.--- [x] **Verify:** `pipelex-api` suite green (307 pass) with **no** `pipelex.temporal`/`temporalio` import in `api/` (`grep` → only incidental prose; `import api.main` → `temporalio` not in `sys.modules`). Direct-mode validate + execute work with no orchestrator plugin installed. Override policy refuses a forbidden per-request mode (403). All gates green (lint/pyright 0/mypy 0/pylint 10/openapi-check).-- [x] **Capture cold-start context:** `### Phase C — as-built` appended below (config Option C + root-key gotcha, the F1 dispatch deviation + why, override policy, synthetic-plugin test pattern, validate doc location).-- [x] **Fan-out `/code-review`:** two clean-context reviews on the working-tree diff (correctness/byte-equivalence + tests/config) — both **correct, no silent bugs, byte-equivalence verified**. Triage recorded in the Phase C as-built (5 applied incl. a re-added coverage test; 2 pre-existing deferred).-- [x] **Commit:** `pipelex-api` `a39841e` (`refactor/orchestrator-agnostic-base`) + this tracker update on `refactor/Plugins-4`. Nothing pushed. Strong session boundary before building flavors.-------## Phase D — `pipelex-api-hosted` (Temporal flavor, private ECR)--**Goal:** the first real flavor = agnostic base + `pipelex-temporal` + Temporal activation.--- [ ] **Flavored base** (`pipelex-api-temporal`): `FROM pipelex/pipelex-api:<ver>` + install `pipelex-temporal @ git+ssh://…@<ref>` per the image architecture (re-add `git` + `openssh-client`, SSH mount, version-locked to `<ver>`). Wire the deploy key into CI (`--ssh default`).-- [ ] **Config migration** in the baked `.pipelex/pipelex_{env}.toml`:- - [ ] Replace `[temporal] is_enabled = true` → `[plugins] boot_orchestrator = "temporal"`.- - [ ] Add `[api] execution_mode = "temporal_fire_and_forget"` (or `temporal_blocking` per route semantics) + the override policy.- - [ ] **Move** the connection tree (`temporal_config`, `worker_config`, `queue_options`, `search_attributes`) **out** of `pipelex_{env}.toml` into baked `.pipelex/temporal_{env}.toml` (the D2 plugin-config file). The main `pipelex_{env}.toml` carries no `[temporal.*]` keys — core would reject them.-- [ ] **Env-config child Dockerfile** stays a `COPY` — now copying `pipelex_{env}.toml` **and** `temporal_{env}.toml`.-- [ ] Record the (base `pipelex-api` version ⇄ `pipelex-temporal` ref) pair in the flavor's version file.--### 🛑 CHECKPOINT D — hard stop · **MAJOR GATE 3** (source-doc Checkpoint 3)--First flavor boots end-to-end — validates the whole chain before replicating for Mistral.--- [ ] **Verify:** dev image boots with Temporal active; a run enqueues to Temporal Cloud; `PIPELEX_ENV` selects the right `temporal_{env}.toml`. Validate runs DIRECT in-process (F2) and the runner is sized for the library load.-- [ ] **Capture cold-start context:** append `### Phase D — as-built` (the two-layer Dockerfile shape, the SSH/deploy-key CI wiring, the exact config keys moved to `temporal_{env}.toml`, the version-pair record location, any e2e boot evidence).-- [ ] **Fan-out `/code-review`:** sub-agent runs `/code-review` on the `pipelex-api-hosted` diff (Dockerfile + config migration). Triage as above.-- [ ] **Commit** in `pipelex-api-hosted` (its own branch). This is where outward-facing/deploy-breaking work begins — confirm go/no-go with the user before any actual deploy.-------## Phase E — `pipelex-api-mistral` (Mistral flavor, new repo) — GATED--**Goal:** prove the base supports a *second*, differently-shaped (per-call) orchestrator. **Gated on `pipelex-mistralai-workflows` being published to PyPI and its in-flight `mistralai-2.x` work settling — coordinate, don't disrupt.**--- [ ] **Scaffold** mirroring `pipelex-api-hosted` conventions.-- [ ] **Flavored base** (`pipelex-api-mistral`): `FROM pipelex/pipelex-api:<ver>` + `uv pip install pipelex-mistralai-workflows==X` (PyPI — no SSH, no `git` re-add).-- [ ] **Config** in baked `.pipelex/pipelex_{env}.toml`: `[api] execution_mode = "mistral_native"`. **No** `boot_orchestrator` (per-call orchestrator), **no** plugin config file (the Mistral plugin self-loads none today).--### 🛑 CHECKPOINT E — hard stop--- [ ] **Verify:** image boots with `mistral_native` dispatchable; a run routes through the Mistral orchestrator; no Temporal anything present (proves the base is genuinely agnostic, not Temporal-shaped).-- [ ] **Capture cold-start context:** append `### Phase E — as-built` (confirming the per-call vs boot-global distinction held in practice: no `boot_orchestrator`, no plugin config file needed).-- [ ] **Fan-out `/code-review`:** sub-agent runs `/code-review` on the new-repo scaffold + flavor config. Triage as above.-- [ ] **Commit** in `pipelex-api-mistral` (new repo).-------## Phase F — `pipelex-worker` pin flip (private; lands with the release)--**Goal:** flip the worker to the externalized Temporal plugin. Per the archived plugin-system cut-list — land in the **same commit** as the `pipelex` release pin bump.--- [ ] `pyproject.toml`: `pipelex[dynamodb,s3,temporal]==X` → `pipelex[dynamodb,s3]==X` + `pipelex-temporal @ git+ssh://…@<ref>`.-- [ ] `Dockerfile` CMD `["pipelex","worker",…]` → `["pipelex-temporal","worker",…]`; `Makefile` `pipelex worker` → `pipelex-temporal worker`.-- [ ] Set `boot_orchestrator = "temporal"` for the worker process.-- [ ] Run downstream consumers' suites against the new pins **before** publishing.--### 🛑 CHECKPOINT F — hard stop (final)--- [ ] **Verify:** worker runs on the externalized plugin; Temporal is fully external (no `temporal` extra anywhere in the public base); consumer suites green against the new pins.-- [ ] **Capture cold-start context:** append `### Phase F — as-built` (the published `pipelex-api` version, the `pipelex-temporal` ref, the pins flipped per repo, consumer-suite green evidence).-- [ ] **Fan-out `/code-review`:** sub-agent reviews the pin-flip diffs.-- [ ] **Ship:** coordinate the cross-repo release. Effort complete.-------## Per-repo handoff summary--| Phase | Repo | Change | Channel |-|---|---|---|---|-| A | `pipelex` (`_plugins`) | D2 env-aware plugin-config helper; F3 `add_http_error_mapper` seam + `PLUGIN_API_VERSION` bump; SPI doc | public, normal release |-| B | `pipelex-temporal` | use D2 helper (`temporal_{env}.toml`); register F3 `TemporalError → ErrorReport` mapper | private git+ssh |-| C | `pipelex-api` | drop `temporal` extra; execution via bridge (F1); validate DIRECT (F2); consume F3 mappers; remove all `pipelex.temporal`/`temporalio` imports | public Docker Hub |-| D | `pipelex-api-hosted` | flavored-base install of `pipelex-temporal` (git+ssh); `boot_orchestrator=temporal` + `[api] execution_mode`; connection tree → `temporal_{env}.toml` | private ECR |-| E | `pipelex-api-mistral` (new) | scaffold; flavored-base install of `pipelex-mistralai-workflows` (PyPI); `[api] execution_mode=mistral_native`, no boot gate | private ECR |-| F | `pipelex-worker` | pin `pipelex-temporal` (git+ssh); CLI → `pipelex-temporal worker`; `boot_orchestrator=temporal` | private |-------## Open items / risks to verify during execution--- **F3 — confirm transport faults are wrapped, or rely on the mapper.** Verify whether the Temporal orchestrator already wraps bare `temporalio` transport errors into `PipelexError`s (`TemporalServerError` etc.) before they reach HTTP. If it does, the F3 mapper is belt-and-suspenders; if not, it is the only thing keeping such errors off the catch-all 500. Either way the mapper makes the base correct without naming Temporal.-- **F1 — define the override policy precisely.** Decide the exact shape (`allow_request_execution_mode_override` bool vs a per-deployment allowlist of modes) and the default — recommend override **off** on hosted flavors. A caller must never force `direct` on a runner whose whole point is distributed execution.-- **F2 — quantify the library-load cost on the hosted runner.** Validate now loads the method library API-side. Confirm the hosted runner image/resources are sized for it; document in `pipelex-api` / `pipelex-api-hosted` docs.-- **Version-lock bookkeeping.** Record the (base `pipelex-api` version ⇄ plugin ref) pair in each flavor repo's version file so a bump can't drift them apart.-- **Mistral publish gating.** `pipelex-mistralai-workflows` must be published to PyPI and its `mistralai-2.x` work settled before Phase E.-- **Docs deploy.** The Temporal error pages left in core `docs/errors/` (type_uri dereference targets) still need a home decision when `pipelex-temporal` docs exist — tracked in the archived Phase-5 cut-over notes, not here. **Concrete instance from Phase B:** the F3 mapper emits `type_uri = .../errors/temporal-transport-error/`, but `TemporalTransportError` is a synthetic `error_type` string (not a `PipelexError` subclass), so `generate-error-pages` never emits a page — the RFC 7807 `type` link 404s. This is **pre-existing parity** (pipelex-api's old `handle_temporal_error` emitted the identical dangling URI), now plugin-owned. Resolve with the same "where do Temporal docs live" decision: author a `<!-- pipelex:authored -->` page once a home exists.-------## As-built log (append per phase at each checkpoint — keep this current for cold starts)--> Each checkpoint appends an `### Phase N — as-built` subsection here with: final names/signatures, divergences from plan, test evidence, and anything a cold resume needs.--### Phase A — as-built--**Branch:** `refactor/Plugins-4` (worktree `_plugins`), cut from `refactor/Plugins-3`. One commit for the whole phase.--**D2 — env-aware plugin config loader.** `pipelex/system/configuration/config_loader.py`:--- `ConfigLoader.load_plugin_config(*, name: str, package_dir: Path, schema: type[_PluginConfigT], extra_overrides: dict[str, Any] | None = None) -> _PluginConfigT` — loads, deep-merges and **validates** (returns the `schema` instance, not a raw dict — the `schema` param from the plan is honored as a validated return).-- **Signature divergence from the plan sketch `load_plugin_config(*, name, schema)`:** added `package_dir: Path`. The packaged default `{name}.toml` lives inside the *plugin's own* distribution; core cannot locate it, so the plugin passes its package dir (`Path(__file__).parent`). This is necessary, not optional.-- Layering (later wins per leaf, via `deep_update`): `{package_dir}/{name}.toml` → global `~/.pipelex/{name}_{env}.toml` → global `~/.pipelex/{name}_override.toml` → project `.pipelex/{name}_{env}.toml` → project `.pipelex/{name}_override.toml` (project only if distinct from global) → `extra_overrides`. `env` = `runtime_manager.environment`.-- Helper `ConfigLoader._plugin_override_files_for_dir(config_dir, *, name)` mirrors `_override_files_for_dir` but is **intentionally narrower**: only `{name}_{env}` + `{name}_override` — no `local` / `run_mode` / `temporary` tiers (a plugin config is env-selected + deployment-baked, not developer-scratch-layered). Missing files at any tier are skipped, so the packaged default alone is valid. Does **not** call `ensure_global_config_exists()` (no kit-template copy for plugin configs).-- New module-level `_PluginConfigT = TypeVar(..., bound=BaseModel)`; added `from pydantic import BaseModel`.--**F3 — HTTP-error-mapper seam.** `pipelex/plugins/registrar.py` (reflects the **lazy-provider refinement** landed during Phase B — see the next paragraph for why):--- Type aliases `HttpErrorMapperFn = Callable[[Exception], ErrorReport]` and `HttpErrorTypeProviderFn = Callable[[], type[Exception]]` (`ErrorReport` from `pipelex.base_exceptions` — a core type, no cycle, no FastAPI/SDK pull).-- Menu method `PluginRegistrar.add_http_error_mapper(*, exc_type_provider: HttpErrorTypeProviderFn, to_error_report: HttpErrorMapperFn) -> None` — the exc type is supplied as a **thunk**, not the bare class. `register` only appends a `_HttpErrorMapperContribution(exc_type_provider, to_error_report, source_plugin)` NamedTuple to `self.http_error_mappers: list[...]` + a generic `"http error mapper"` contribution line; it neither resolves the type nor imports the SDK.-- Read accessor `PluginRegistrar.get_http_error_mappers() -> dict[type[Exception], HttpErrorMapperFn]` **runs every provider** to build a freshly-keyed `{exc_type: mapper}` dict (so any SDK import is paid here, at the host's read time, not at registration), fail-loud `DuplicateHttpErrorMapperError` naming both plugins on a duplicate *resolved* type. Consumer can't mutate registrar state.-- New error class `DuplicateHttpErrorMapperError(PluginError)` in `pipelex/plugins/exceptions.py` (mirrors `DuplicateOrchestratorError`; fields `exc_type`/`first_plugin`/`second_plugin`).--**Why lazy (the Phase-B-forced refinement).** Phase A originally shipped `add_http_error_mapper(*, exc_type: type[Exception], …)` keyed on the concrete class. Phase B exposed the conflict: naming `temporalio.exceptions.TemporalError` imports the whole `temporalio` SDK (measured ~146 ms — pulls the `temporalio.api`/`temporalio.bridge` protobuf chain), so a bare `exc_type=` forces that import inside `TemporalPlugin.register` — violating Checkpoint B's hard acceptance ("import-light register, no temporalio at registration") and the plugin's own thunk discipline. The provider thunk defers the SDK import to `get_http_error_mappers` (a host runtime's app construction), where the plugin — and therefore its SDK — is by definition installed. Duplicate-by-type detection moves with it (to resolution time). `PLUGIN_API_VERSION` stays **2** (same capability, refined signature; nothing external consumed v2 yet). Landed as its own commit on `refactor/Plugins-4`.--**No hub wiring (deliberate).** Phase A is exactly the planned surface: registrar method + accessor. The mappers are **not** pushed onto the hub. **How `pipelex-api` reaches them in Phase C:** call `build_registrar(config=get_config())` at app construction (documented pure/repeatable fn; `cli/commands/plugins_cmd.py` already does this standalone) and iterate `registrar.get_http_error_mappers()`. No further core exposure needed → Phase C stays pipelex-api-only. If Phase C prefers reading the boot registrar instead of rebuilding, that needs a *new* public getter (the registrar is `Pipelex._plugin_registrar`, private today) — small follow-up, decide in Phase C.--**Version bump.** `pipelex/plugins/contract.py`: `PLUGIN_API_VERSION: int = 1 → 2` (+ comment noting v2 = the optional `add_http_error_mapper` capability). `PipelexPlugin` docstring corrected (dropped the stale "CLI commands" mention removed by Option A; added "model listers" and "HTTP-error mappers"). **`targets_api` re-confirmation:** all built-ins set `targets_api = PLUGIN_API_VERSION` (auto-tracks the bump — verified). **Still to re-confirm in Phase B:** the external `pipelex-temporal` plugin (and `pipelex-mistralai-workflows`, already an entry-point plugin) — both import `PLUGIN_API_VERSION` from core, so an editable-pinned core gives them `2` automatically, but their suites must run against the Phase A core to confirm.--**SPI doc.** `docs/under-the-hood/orchestrator-plugins.md`: new section "HTTP error mappers: rendering an orchestrator's transport faults" (plugin owns classification / core owns transport / host owns presentation), the registrar-menu table row now lists `add_http_error_mapper` + `get_http_error_mappers`, and the `PLUGIN_API_VERSION = 2` note. Error reference page `docs/errors/duplicate-http-error-mapper-error.md` generated via `pipelex-dev generate-error-pages` (+ index line in `inference-and-providers.md`).--**Tests (all green).**--- `tests/unit/pipelex/system/test_plugin_config_loader.py` — `TestPluginConfigLoader`: packaged-default-only, env-file deep-merge + sibling-env-ignored, override-beats-env, project-beats-global, extra_overrides-win-last, validation-after-merge. Hermetic via `PropertyMock` on `global_config_dir`/`project_config_dir`/`runtime_manager.environment`.-- `tests/unit/pipelex/plugins/test_http_error_mapper_seam.py` — `TestHttpErrorMapperSeam`: add+get (resolved type), **provider resolved lazily not at registration** (the import-light proof — an exploding provider survives `register`), contribution recorded without resolving, duplicate fails loud naming both at resolution time, accessor returns a fresh dict, empty-by-default.-- `tests/unit/pipelex/plugins/test_import_light_boot.py` — extended the BLOCKED set with `fastapi`/`starlette` so the subprocess guard now also proves registration imports no web framework (locks the F3 invariant at the discovery layer).--**Gates:** `make agent-check` clean (ruff/plxt/pyright 0/mypy 0/keyword-only ✓) · `make tb` green · `make agent-test` green. Core names no orchestrator; no `import fastapi`/`starlette` anywhere in core (the only `fastapi` strings are telemetry-integration labels + the `WorkerMode.FASTAPI` enum value, not imports).--**Code-review triage** (forked `/code-review`, high effort — verdict: clean, correct, faithful to D1/D2/F1/F2/F3; no bugs):--- **Applied:** test `begin_plugin(targets_api=…)` now uses `PLUGIN_API_VERSION` instead of a literal `2` (nit #2 — latent stale-on-bump).-- **Deferred (design tradeoff):** the Phase-C consumer path for the mappers (registrar has the accessor but it isn't on the hub and the boot registrar is private) → captured in [`wip/plugins/phase-c-http-error-mapper-consumer-path.md`](wip/plugins/phase-c-http-error-mapper-consumer-path.md). Recommended default: Phase C calls `build_registrar(config=get_config())` and iterates `get_http_error_mappers()` (keeps Phase C pipelex-api-only).-- **Info / no action:** `DuplicateHttpErrorMapperError` message uses `__qualname__` (only ever fires on a true same-class dup, so unambiguous); `load_plugin_config` has no `is_unit_testing` hermetic branch (correct — no run_mode tier; Phase B tests must mock the config dirs); D2 intentionally has no `.pipelex/{name}.toml` *base* tier (packaged default is the only base — worth a one-line plugin-authoring-doc callout when Phase B lands).--### Phase B — as-built--**Repos/branches:** core seam refinement on `refactor/Plugins-4` (`pipelex`, worktree `_plugins`) commit `75196e2e3`; Phase B proper on `refactor/own-temporal-config` (`pipelex-temporal`). pipelex-temporal pins core editable (`[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }`), so it builds against the Phase A core live.--**Seam refinement (core, `75196e2e3`) — prerequisite, see Phase A as-built "Why lazy".** `add_http_error_mapper(*, exc_type_provider: Callable[[], type[Exception]], to_error_report)`; `register` only stores a `_HttpErrorMapperContribution` NamedTuple; `get_http_error_mappers()` runs the providers at the host's read time and dedups on the resolved type. This is what lets `TemporalPlugin.register` name `temporalio.TemporalError` without importing `temporalio` (which costs ~146 ms — the whole api/bridge protobuf chain).--**D2 adoption.** `pipelex_temporal/temporal_config_loader.py`: `load_temporal_config()` now delegates to `config_manager.load_plugin_config(name="temporal", package_dir=Path(__file__).resolve().parent, schema=Temporal)`. Dropped the hand-rolled `load_toml_from_path` + single-`resolve_config_file` override; the helper layers packaged `temporal.toml` → global `~/.pipelex/temporal_{env}.toml` → `temporal_override.toml` → project `.pipelex/{...}` with deep-merge, `env = runtime_manager.environment`. **Behavior change:** the old loader read a plain `.pipelex/temporal.toml`; the new layering has **no plain-`.pipelex/temporal.toml` base tier** (packaged default is the only base — D2 design). No such override file existed in the repo, so nothing silently dropped. The env files themselves are baked per-deployment in Phase D; Phase B only switches the loader to support the layering. **Phase-D guardrail (deferred, from the Phase B `/code-review`):** a stray suffix-less `.pipelex/temporal.toml` left over after a botched migration would now vanish silently — a one-line `log.warning` when such a file is found would catch it. Deferred as an enhancement (not added now per smallest-correct-surface; revisit when Phase D migrates the connection tree).--**F3 mapper.** `pipelex_temporal/temporal_plugin.py`: two new module-level fns + one `register` line (unconditional, alongside the orchestrators — an API runner that only dispatches Temporal per-request still needs the mapper):--- `_temporal_transport_error_type() -> type[Exception]` — lazily `from temporalio.exceptions import TemporalError; return TemporalError`. Keyed on the **base** `TemporalError`: `RPCError` and `WorkflowFailureError` both subclass it, so the host's exception-MRO walk catches every bare transport fault. Invoked only by `get_http_error_mappers()`, never at register.-- `_temporal_transport_error_to_report(exc) -> ErrorReport` — ports pipelex-api's old `handle_temporal_error`: `error_type="TemporalTransportError"`, `message=str(exc)`, `title=pascal_case_to_sentence(...)` → "Temporal transport error", `type_uri=f"{URLs.error_docs_base}/{pascal_case_to_kebab(...)}/"` → `.../errors/temporal-transport-error/`, `error_category="transient"`, `error_domain=ErrorDomain.RUNTIME`, `retryable=True`. Imports no `temporalio` (only `str(exc)`); the identity pair uses the same core helpers `PipelexError.title`/`type_uri` use, so the docs URI matches the canonical scheme.-- Module-top imports added: `ErrorDomain`, `ErrorReport` (`pipelex.base_exceptions`), `pascal_case_to_kebab`/`pascal_case_to_sentence` (`pipelex.tools.misc.string_utils`), `URLs` (`pipelex.urls`) — all import-light core, no SDK.--**F3 is mostly belt-and-suspenders (open-item resolved).** `pipelex_temporal/tprl/workflow_caller.py` already wraps `RPCError` / `WorkflowFailureError` / `WorkflowAlreadyStartedError` into `WorkflowExecutionError` (a `PipelexError`) inside `execute_workflow`/`start_workflow`, so the common transport faults reach the API as `PipelexError` (handled generically by `handle_pipelex_error`). The F3 mapper is the safety net for any temporalio error that escapes that wrapping (e.g. a connection error raised outside execute/start) — and, more importantly, it is what makes the public `pipelex-api` base render such a fault correctly **without naming Temporal**. Implemented as planned regardless.--**`targets_api`.** Unchanged: `TemporalPlugin.targets_api = PLUGIN_API_VERSION` auto-tracks to 2 via the editable core pin. The whole suite running against the Phase A core is the confirmation.--**Tests (pipelex-temporal, all green).**--- `tests/unit/pipelex_temporal/test_temporal_config_env_layering.py` — `TestTemporalConfigEnvLayering`: packaged-default-only valid; `temporal_{env}.toml` deep-merges onto the real packaged default (overridden leaf wins, untouched leaves + sibling env ignored). Hermetic via `PropertyMock` on `ConfigLoader.global_config_dir`/`project_config_dir` + `runtime_manager.environment` (mirrors core's `test_plugin_config_loader`).-- `tests/unit/pipelex_temporal/test_temporal_plugin_http_error_mapper.py` — `TestTemporalPluginHttpErrorMapper`: register → `get_http_error_mappers()` keyed on `TemporalError` + full classification through the public seam; **import-light proof** = a fresh subprocess with a `temporalio` meta-path blocker imports the plugin and runs `register` (both `boot_orchestrator=None` and `"temporal"`) to completion, asserting `temporalio` never lands in `sys.modules`.-- Existing `test_temporal_config_loader.py` (packaged-default + hub cache) still green against the new loader.--**Gates:** pipelex-temporal `make agent-check` clean (ruff ✓ / pyright 0 / mypy 0) · `make agent-test` GREEN against the Phase A core (full suite incl. `temporal`-marked workflow integration tests, exit 0). Core (`refactor/Plugins-4`) `make agent-check` + plugins tests + `make tb` also green.--**Code-review triage** (two forked `/code-review`s, high effort — both verdicts: **clean, no bugs**):--- _Core seam refinement (`75196e2e3`):_ correct + internally consistent (no dangling `_http_error_mapper_sources`; `_add` still drives the 3 sibling adders; dedup attribution at resolution time correct; repeated `get_http_error_mappers()` side-effect-safe; NamedTuple standards-correct). **Deferred (recorded in `wip/plugins/phase-c-http-error-mapper-consumer-path.md`):** dedup raise now lands at the host's app-construction read time (theoretical for one-plugin-per-image); `plugins list` shows a generic `"http error mapper"` line (lost the per-type label — optional `display_label` param if it ever matters).-- _pipelex-temporal Phase B:_ F3 classification verified **byte-exact** vs pipelex-api `handle_temporal_error` (title/type_uri identical — same `URLs.error_docs_base` + helpers); `TemporalError`-base keying complete via MRO; import-light genuinely proven. **Applied:** import-light subprocess now also asserts `slot_claims` count per boot arm (0 off-gate, 4 on `"temporal"`). **Deferred:** dangling `temporal-transport-error` docs URI (pre-existing parity → folded into the "Temporal error pages need a home" open item); D2 base-tier-drop `log.warning` guardrail (Phase D).--### Phase C — as-built--**Repo/branch:** `pipelex-api` on `refactor/orchestrator-agnostic-base` (cut from `dev`), commit `a39841e`. The `pipelex` worktree only carries this tracker update (`refactor/Plugins-4`). Nothing pushed. pipelex-api pins core **editable** (`[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }`) so it builds against the Phase A core live; the published `==<ver>` pin (no `temporal` extra) is the release-time flip. Nothing pushed.--**Decision — `[api]` config home = Option C (the user chose it).** `execution_mode` + override policy live in a **pipelex-api-owned `api/api.toml`**, loaded via the Phase A D2 helper `config_manager.load_plugin_config(name="api", package_dir=Path(api/), schema=ApiConfig)` — NOT a core `[api]` section (core config is `extra="forbid"`, would reject it → would need a 2nd core change, which the plan forbids). **Mechanics gotcha:** `load_plugin_config` validates the **whole** merged TOML document against the schema (exactly like `temporal.toml`), so `api.toml` keys are **root-level — no `[api]` table wrapper** (the user's preview showed `[api]`; the real file has root keys; functionally identical). `ApiConfig(BaseModel, extra="forbid")` has NO field defaults (defaults live in `api.toml`, mirroring core's discipline): `execution_mode: PipelexExecutionMode`, `allow_request_execution_mode_override: bool`. Base ships `execution_mode = "direct"`, override off. `get_api_config()` is `@cache`'d (immutable per process), **warmed in `lifespan` after `Pipelex.make`** for fail-fast (matches `ERROR_DISCLOSURE`). A flavor bakes `.pipelex/api_{env}.toml` (Phase D).--**F1 — dispatch DEVIATION from the literal plan (flagged; the only option satisfying all three locked constraints).** The plan's F1 wording said "build `PipelexPipeRunInput` and dispatch via `run_pipe_via_bridge`." **That path is lossy:** `PipelexPipeRunInput` carries no `request_id` / `output_multiplicity` / `dynamic_output_concept_ref`, and `build_pipe_job_from_input` skips pipeline-manager registration (the 409-conflict path) + telemetry — so routing `start` through it would silently regress those features. The only non-lossy bridge variant is extending the **core** `PipelexPipeRunInput` = a 2nd core change the plan forbids ("Phase C = pipelex-api only"). **Chosen instead (Design B):** keep `pipeline_run_setup(...)` to build the rich `PipeJob` (preserves everything → byte-equivalent, which Checkpoint C explicitly demands), then dispatch via the agnostic hub seam `get_orchestrator_registry().get_optional(mode=execution_mode).run(pipe_job=…, delivery_assignment=…)` — the **identical final dispatch** `run_pipe_via_bridge` performs internally (bridge.py:92-95), just fed a live job instead of a serialized lossy input. Names no orchestrator, imports no SDK. `None` orchestrator → `MissingOrchestratorError(mode)` (a `PipelexError` → `handle_pipelex_error`, carries the install hint). `workflow_id` read off `run_output.workflow_id` (Temporal FF sets it, DIRECT returns `None`); returned `pipeline_run_id` stays the `pipeline_run_setup`-resolved id (byte-equivalent to old). Verified equivalent against `TemporalFireAndForgetOrchestrator.run` (pipelex-temporal) which calls the same `temporal_pipe_run.start(pipe_job)` the old code did. `execution_mode` resolved **first** in `start()` (before `pipeline_run_setup`) so a forbidden override 403s before any library load.--**F1 — override policy.** Per-request mode rides `PipelineApiExtras.execution_mode` (API-server extra; route `_validate_extras` parses it, `start` route passes `requested_execution_mode=`). `resolve_execution_mode(requested, *, config)`: `None`/equal-to-default → default; differs + override on → requested; differs + override off → `raise_forbidden(403, ErrorType.EXECUTION_MODE_OVERRIDE_FORBIDDEN)`. Documented in OpenAPI only via the `PipelineApiExtras` field description (NOT added to `PipelexApiStartRequest`, so the committed OpenAPI artifact's `/start` schema is unchanged — only the validate-route docstring regen'd).--**F2 — validate always DIRECT.** `ApiRunner.validate` collapsed to the single `validate_bundles_in_process(...)` path; dropped the `get_config().temporal.is_enabled` branch + `dispatch_dry_validate`/`DryValidateArg`/`PipelexInterpreter`/`build_validation_report` imports; dropped the route's `WorkflowExecutionError` catch in `validate.py` (unreachable). **Behavior change documented** in `pipelex-api/docs/pipe-validate.md` ("Where validation runs" + a deployment resource note: the runner now loads the method library API-side — size a flavor's runner for it).--> **F2 SUBSEQUENTLY REVERSED / EXTENDED** by the orchestrator-dispatched-`/validate` follow-on (its own plan: [`wip/plugins/orchestrator-dispatched-validate.md`](wip/plugins/orchestrator-dispatched-validate.md)). `/validate` is now `execution_mode`-aware through a new per-call `BundleValidatorRegistry` seam (V0 core, V1 `pipelex-temporal`, V2 `pipelex-api`): `ApiRunner.validate` (the override) was **removed** in favor of `ApiRunner.validate_verdict` (verdict-as-value); the route discriminates on `isinstance(verdict, PipelexValidationReport)` instead of catching `ValidateBundleError`. `direct` keeps the in-process path; `temporal_*` dispatches to a worker and assembles the same canonical report API-side. The library-load note is now flavor-conditional. Branches: all three repos on `feature/Orchestrator-dispatched-validate`.--**F3 — consume the mappers.** `api/exception_handlers.py`: dropped `from temporalio.exceptions import TemporalError` + hardcoded `handle_temporal_error`; `register_exception_handlers(app, *, disclosure_mode, http_error_mappers=None)` now registers one FastAPI handler per `(exc_type, mapper)` via `_make_orchestrator_error_handler` (runs the mapper → existing `_problem_response` RFC 7807 + DisclosureMode render). **Consumer path = deferred-doc option 1 (no 2nd core change):** `api/main.py` resolves the mappers at module import via `build_registrar(config=PipelexConfig.model_validate(config_manager.load_config())).get_http_error_mappers()` (the standalone pattern `plugins_cmd.py` uses — pure, pre-boot-safe; smoke-confirmed empty in the base). The module keeps its import-side-effect-free contract (mappers are a param, not a `build_registrar` call inside it), so tests register handlers on throwaway apps. **Base = zero mappers** ⇒ a bare orchestrator transport fault with no plugin installed falls to the sanitized catch-all 500 (the dedicated transient classification is now plugin-owned; intended for the agnostic base — proven by `test_unmapped_transport_error_falls_back_to_sanitized_500`).--**F3 test pattern (synthetic plugin, no temporalio).** `test_exception_handlers.py`: `_SyntheticTransportError(Exception)` (bare, non-PipelexError) + `_SyntheticOrchestratorError(PipelexError)` + `_synthetic_transport_to_report` mapper. `_synthetic_http_error_mappers()` builds a real `PluginRegistrar`, `begin_plugin(...)`, `add_http_error_mapper(exc_type_provider=lambda: _SyntheticTransportError, …)`, returns `get_http_error_mappers()` — exercises the **real producer→consumer seam** (same as `api/main.py`) without installing a plugin or importing an SDK. Conformance webhook test now mocks `get_orchestrator_registry` (fake orchestrator `run` does the real `DeliveryExecutor` delivery + returns a FF `PipelexPipeRunOutput`).--**Deps + base config.** `pyproject.toml`: `pipelex[mistralai,anthropic,google,google-genai,bedrock,fal]` (dropped `temporal`). Removed the whole `[temporal]` block from the base `.pipelex/pipelex.toml` (core would reject it). **Local-env cleanups (NOT committed, machine-only):** a stale `[temporal]` section in the gitignored `.pipelex/pipelex_override.toml` (had `worker_scopes`/search-attributes) broke loading against the temporal-free core — removed it; the global `~/.pipelex/pipelex.toml` only had the valid `tracing_config.temporal_dynamodb` backend (left alone). `tests/unit/conftest.py`: dropped the removed `temporal_enabled=False` kwarg from `Pipelex.make` (agnostic base defaults to in-process DIRECT, which is what the hermetic suite wants).--**Gates (all green):** `make lint` (ruff, incl. 2 TC002 moves — `PipelexValidationReport`+`PipelexExecutionMode` to TYPE_CHECKING in pipeline.py, both annotation-only under `from __future__ import annotations`; pydantic `models.py` keeps its runtime import) · `make pyright` 0 · `make mypy` 0 · `make pylint` 10.00/10 · **307 tests pass** (`make t`, xdist) · `make openapi-check` ✓ (only the validate docstring regen'd). `grep -rn temporal api/` → only incidental prose (`pipelex-temporal` plugin name, `temporalio.TemporalError` in a docstring, the execution-mode enum value strings). `import api.main` → `temporalio` NOT in `sys.modules` (agnostic base proven).--**Docs.** `docs/configuration.md` (new "Execution mode" section + de-Temporal'd the intro), `docs/pipe-validate.md` ("Where validation runs" + resource note), `docs/pipe-run.md` / `docs/index.md` / `docs/error-responses.md` (de-Temporal'd stale mentions), `CHANGELOG.md` `[Unreleased]`.--**Code-review triage** (two clean-context `/code-review`s on the working-tree diff — correctness/byte-equivalence + tests/config; both verdicts: **correct, byte-equivalence holds, no silent bugs, no 500 regression, clean coupling**):--- **Applied (5):** (1) _Real coverage loss_ — re-added the validate **no-verdict→5xx** invariant test (deleted with the Temporal arm; route still upholds it but nothing tested it) as `test_non_verdict_failure_is_not_a_200_verdict` (mocks `validate` to raise a non-`ValidateBundleError` `PipelexConfigError` → asserts ≥500 + problem+json + `is_valid` absent). (2) _Direct-mode `/start` blocks_ — the agnostic base runs `direct` in-process to completion before the 202, so the "asynchronous/non-blocking" promise is a **distributed-flavor** property; clarified in the `start` route docstring + `docs/pipe-run.md` (callback still fires either way). (3) Conformance webhook test now asserts `get_optional.assert_called_once_with(mode=DIRECT)` + `run.assert_awaited_once()` (closes the "wrong mode silently dispatched" gap). (4) conftest `get_api_config.cache_clear()` on setup+teardown (insurance vs `@cache` leak across tests). (5) Stale-doc fixes: `handle_unexpected_error` docstring + `ErrorType.INTERNAL_SERVER_ERROR` comment (dropped the `TemporalError` mention) + `test_validate_errors` module docstring.-- **Deferred (pre-existing / design, recorded here):** _`_validate_extras` mislabels error_type_ — a malformed `execution_mode`/`pipeline_run_id` in the start extras is a correct 422 but labeled `InvalidCallbackUrls` (one catch covers all three extras; the `message` still names the real field). Pre-existing pattern my new field slightly widened; proper fix = per-field error_type discrimination (more surface) → follow-up, untested today. _Double `build_registrar`_ — runs once at `api/main.py` import (mapper resolution) and once in `Pipelex.make` boot; harmless (pure fn), the deferred-doc option-1 tradeoff.-- **Verdict on the F1 deviation:** the correctness reviewer independently **verified byte-equivalence** of the orchestrator-registry dispatch vs the old `make_temporal_pipe_run().start()` (same `pipe_job`, same `delivery_assignment`, `pipeline_run_id` still from `pipeline_run_setup`, `workflow_id` off `run_output` — the F&F orchestrator calls the identical `temporal_pipe_run.start`). No silent change.--**Gates re-run after triage:** lint ✓ · pyright 0 · mypy 0 · pylint 10.00/10 · **308 tests pass** · openapi-check ✓ (the `start` + `validate` route docstrings regen'd the artifact).diff --git a/docs/agents/debugging-hanging-pytest-runs.md b/docs/agents/debugging-hanging-pytest-runs.mdindex 2fcfe6b38..1f49df4f5 100644--- a/docs/agents/debugging-hanging-pytest-runs.md+++ b/docs/agents/debugging-hanging-pytest-runs.md@@ -1,30 +1,33 @@# Playbook: debugging a hanging pytest run in one shot-**Audience.** Future-me (or any agent) the next time a `make agent-test` / `pytest -n auto` run hangs without finishing. Specific to this codebase, where the most common cause of a hang is Temporal integration-test infrastructure (in-process test servers, xdist worker crashes, fixture teardown), but the iteration loop generalizes to any hanging suite.+**Audience.** Future-me (or any agent) the next time a `make agent-test` / `pytest -n auto` run in this repo hangs without finishing. The common causes here are xdist worker crash-and-replace cycles and fixture-teardown hangs; the iteration loop below generalizes to any hanging suite.-**Why this exists.** In a recent session (Issue 2 fix to `with_conditional_worker`), I burned ~30 minutes across multiple test runs that all hung or got killed, requiring the user to flag the hang each time. Every individual decision was defensible; the *iteration loop* was broken. Following the playbook below, the whole thing — code fix, verify unit, verify integration, find the real failure, fix it, verify again — should finish in one autonomous pass.+!!! note "This playbook covers the core `pipelex` suite"+ Core `pipelex` has no `temporalio` dependency and spawns no Temporal processes — the Temporal distributed-execution backend is a separate, external product with its own test suite. If a hang traces into `temporalio` or a `temporal-sdk-python` process, it is coming from that separate suite, not from anything in this repo, and this playbook does not apply.++**Why this exists.** It is easy to burn ~30 minutes across multiple test runs that all hang or get killed, requiring the user to flag the hang each time. Every individual decision can be defensible while the *iteration loop* is broken. Following the playbook below, the whole thing — code fix, verify unit, find the real failure, fix it, verify again — should finish in one autonomous pass.----## What I did wrong (concrete, in order)+## The anti-patterns (concrete, in order)-1. **`make agent-test` is silent on success.** I backgrounded it and waited for harness notification. The output file stayed 0 bytes for 14 minutes. The user had to flag it. → No progress signal at all.+1. **`make agent-test` is silent on success.** Backgrounding it and waiting for a harness notification gives no progress signal — the output file can stay 0 bytes for many minutes. → No way to tell hung from slow.-2. **Re-ran with `-q` flag.** Even more aggressive buffering. Still 0 bytes for the whole run. → Same problem.+2. **`-q` buffers harder.** It holds output until session-end. Use `-v` for debug runs.-3. **Re-ran with `| tail -80`.** Pipe buffering means tail receives nothing until pytest exits. 0 bytes again. → Same problem.+3. **`| tail -80` buffers too.** Pipe buffering means tail receives nothing until pytest exits. Use a direct file redirect (`> file 2>&1`), not a pipe.-4. **Set `--timeout=120 --timeout-method=thread`.** Believed this bounded the run. Wrong: per-test timeout doesn't catch **fixture teardown hangs** (which is what `boot_temporal`'s `manager.teardown()` was doing), and it doesn't catch xdist's **worker-crash-and-replace cycle** which can loop indefinitely retrying failed integration tests.+4. **`--timeout=120 --timeout-method=thread` does not bound the run.** A per-test timeout doesn't catch **fixture-teardown hangs** and doesn't catch xdist's **worker-crash-and-replace cycle**, which can loop indefinitely retrying failed tests. You need an outer wall-clock cap too.-5. **Never killed zombie `temporal-sdk-python` processes between runs.** They accumulated (18+ at one point, PPID=1 after their parents died). Each holds a port and consumes resources. Compounding contention made each new run slower and more likely to crash, which spawned more zombies.+5. **Zombie processes between runs.** Leftover `pytest` sessions (PPID=1 after their parents died) accumulate, each holding resources. Compounding contention makes each new run slower and more crash-prone. Blanket-kill before every run.-6. **Two pytest sessions ran concurrently** (mine + a leftover from the user's earlier attempt). I assumed `pkill` of one PID killed the tree; it didn't. Contention multiplied. Should have done a blanket `pkill -9 -f pytest` first.+6. **Concurrent pytest sessions.** Yours plus a leftover from an earlier attempt. `pkill` of one PID does not kill the tree; do a blanket `pkill -9 -f pytest` first.-7. **Wrapped wait loops in `while kill -0 $PID; do sleep 10; done`.** Bounded by pytest's own runtime — i.e., not bounded at all when pytest hangs. No outer wall-clock cap.+7. **`while kill -0 $PID; do sleep 10; done`** is bounded by pytest's own runtime — i.e., not bounded at all when pytest hangs. No outer wall-clock cap.-8. **When pytest finally produced output, grep'd for "FAILED tests/" but missed the actual error.** The smoking gun (`failure-message="Asynchronous pipeline execution is not enabled..."`) was buried at line ~1490 of a 1497-line log, inside a Temporal server log line — not in a pytest FAILED line. Should have grep'd for the error class name directly from the start.+8. **Grepping for `FAILED tests/` misses the actual error.** The smoking gun is often a specific error message buried deep in a long log, sometimes inside a non-pytest log line — not on a `FAILED` line. Grep for the error **class name** directly from the start.-9. **Trusted xdist output as the source of truth.** Worker crashes ("node down: Not properly terminated") + worker replacement made it impossible to tell whether a failure was a real bug or xdist flakiness. The truth came from running serially.+9. **Trusting xdist output as the source of truth.** Worker crashes (`node down: Not properly terminated`) plus worker replacement make it impossible to tell a real bug from xdist flakiness. The truth comes from running serially.---@@ -34,12 +37,11 @@```bashpkill -9 -f "pytest" 2>/dev/null-pkill -9 -f "temporal-sdk-python" 2>/dev/nullsleep 1-ps -ef | grep -E "pytest|temporal-sdk" | grep -v grep | wc -l # must print 0+ps -ef | grep -E "pytest" | grep -v grep | wc -l # must print 0```-If that doesn't print 0, escalate (one of those processes is hung in uninterruptible I/O — investigate before continuing).+If that doesn't print 0, escalate (a process is hung in uninterruptible I/O — investigate before continuing).### For first-pass change verification@@ -49,48 +51,41 @@ Run *only the tests directly touched by the change*, serially, with `-x` (stop a.venv/bin/pytest -x --timeout=30 -v tests/path/to/specific_test.py```-Fast, fail-loud, no buffering questions. This is what verified the unit-test fix in 0.55s.+Fast, fail-loud, no buffering questions.### For broader verification-Run unit-only (no integration) with a per-test timeout and direct file redirect (no pipe):+Run unit-only with a per-test timeout and direct file redirect (no pipe), under an outer wall-clock cap:```bash-.venv/bin/pytest --timeout=120 --timeout-method=thread -q \+timeout 480 .venv/bin/pytest --timeout=120 --timeout-method=thread -q \-m "not (inference or llm or img_gen or extract or search) and not pipelex_api" \- tests/unit/ > /tmp/pytest_unit.log 2>&1 &-PID=$!-```--Then either `wait $PID` in a bash backgrounded with `run_in_background: true` so the harness notifies on exit, OR — if Temporal integration tests are in scope — wrap in an outer shell `timeout` so even a fixture-teardown hang gets killed:--```bash-timeout 480 .venv/bin/pytest ... > /tmp/pytest_full.log 2>&1-echo "exit=$?" # 124 = timeout fired+ tests/unit/ > /tmp/pytest_unit.log 2>&1+echo "exit=$?" # 0=pass, 1=fail, 124=outer timeout fired```-The 5555-test unit suite runs in ~52s serially. So 480s (8 min) is a generous outer bound that lets you see "did it finish" without ambiguity.+Or background it with `run_in_background: true` and `wait $PID` so the harness notifies on exit. The outer `timeout` is what catches a fixture-teardown hang or an xdist worker-replace loop that `--timeout` cannot.### When the run hangs anyway-- **Read the live log** (`tail -50 /tmp/pytest_full.log`) — direct-redirect means progress is visible as it happens. If `-v` was used, you see the last test name that started.-- **Check active processes** — `ps -ef | grep -E "pytest|temporal-sdk"` shows which xdist worker is alive and how long it's been running.-- **Don't poll.** If you've already wrapped in `timeout`, the outer cap will kill it; let the harness notification fire.+- **Read the live log** (`tail -50 /tmp/pytest_unit.log`) — direct-redirect means progress is visible as it happens. With `-v` you see the last test name that started.+- **Check active processes** — `ps -ef | grep pytest` shows which xdist worker is alive and how long it's been running.+- **Don't poll.** If you've wrapped in `timeout`, the outer cap will kill it; let the harness notification fire.### When the run completes with FAILEDGrep for the **error class name**, not the formatted message:```bash-grep -B 2 -A 10 "AsyncExecutionNotEnabledError\|YourSpecificErrorClass" /tmp/pytest_full.log+grep -B 2 -A 10 "YourSpecificErrorClass" /tmp/pytest_unit.log```-The formatted message can show up anywhere — including inside Temporal server log lines that aren't pytest FAILED markers. The error class name is greppable, stable, and points straight at the cause.+The formatted message can show up anywhere, including inside log lines that aren't pytest `FAILED` markers. The error class name is greppable, stable, and points straight at the cause.If the failures look infrastructure-related (worker crashes, "node down", "replacing crashed worker"), **re-run the failing slice serially** with no xdist:```bash-.venv/bin/pytest --timeout=60 --tb=short -q tests/integration/path/that/failed/+.venv/bin/pytest --timeout=60 --tb=short -q tests/path/that/failed/```If serial passes, the original failure was xdist contention — not your code. If serial fails, it's a real bug, and you now have a clean traceback to read (no xdist obscuring it).@@ -102,7 +97,6 @@ If serial passes, the original failure was xdist contention — not your code. I```bash# 0. Clean statepkill -9 -f "pytest" 2>/dev/null-pkill -9 -f "temporal-sdk-python" 2>/dev/nullsleep 1# 1. Fast verification of the change@@ -116,15 +110,9 @@ echo "exit=$?" # 0=pass, 1=fail, 124=outer timeout# 3. If any failures, grep by error class namegrep -B 2 -A 10 "YourErrorClass" /tmp/pytest_unit.log--# 4. The Temporal integration tests live in the sibling `pipelex-temporal` repo —-# if they're in scope, run them there, serially, with a shorter timeout-( cd ../pipelex-temporal && timeout 240 .venv/bin/pytest --timeout=60 --timeout-method=thread -q \- tests/integration/pipelex_temporal/ > /tmp/pytest_temporal.log 2>&1 )-echo "exit=$?"```-Total worst-case wall clock: ~13 minutes (30s + 8min + 4min). If anything exceeds that, the outer `timeout` kills it.+If anything exceeds the outer `timeout`, it gets killed rather than hanging the session.---@@ -135,12 +123,13 @@ Total worst-case wall clock: ~13 minutes (30s + 8min + 4min). If anything exceed| Direct file redirect (`> file 2>&1`) | No pipe buffering; progress is visible live. || `-q` only for final accepted runs, NOT debug runs | `-q` buffers everything until session-end. Use `-v` for debug. || `--timeout=N --timeout-method=thread` | Per-test cap. `thread` method works in async contexts where `signal` doesn't. |-| Outer shell `timeout N` | Catches **fixture teardown hangs** and **xdist worker-replace loops** that `--timeout` cannot. |+| Outer shell `timeout N` | Catches **fixture-teardown hangs** and **xdist worker-replace loops** that `--timeout` cannot. || `-x` for first-pass debug | Stops at first real failure; gives you a clean traceback fast. |-| Serial (no `-n auto`) for verifying flaky integration tests | xdist worker crashes mask real failures and add runtime via replacement cycles. Serial is honest. |-| `pkill -9 -f temporal-sdk-python` before every run | Zombie test servers compound contention across runs. |+| Serial (no `-n auto`) for verifying flaky tests | xdist worker crashes mask real failures and add runtime via replacement cycles. Serial is honest. || Grep for error **class name**, not message | Messages get embedded in non-pytest log lines; class names don't. |+`make agent-test-debug` (alias `make atd`) packages this loop: clean state, outer wall-clock cap, direct file redirect, and `-v` so each test name lands in the log as it runs.+---## When to stop iterating and surface to the user@@ -148,18 +137,7 @@ Total worst-case wall clock: ~13 minutes (30s + 8min + 4min). If anything exceedBail out and ask for help (don't burn more time) when:- After clean-state + outer-timeout run, the suite still doesn't terminate within 2× its expected duration.-- After serial re-run of the failing slice, you still can't tell whether the failure is your code or pre-existing flake.+- After serial re-run of the failing slice, you still can't tell whether the failure is your code or a pre-existing flake.- You've made a code change to fix a test failure and the same test now fails differently — pause and read the new failure carefully before changing more code.-- The error message points at sandbox / determinism / serialization issues you can't reason about from the surface (this is the Temporal sandbox class of bug — the one I hit).--In all four cases, write up what you know in 3–4 sentences and surface to the user. Better than spending another 20 minutes on dead ends.-------## Codebase-specific notes (Pipelex / Temporal)-- `make agent-test` uses `-n auto -q` — both buffer output and obscure progress. For debug runs, invoke pytest directly with `-v` and direct file redirect.-- `boot_temporal` fixture teardown (in the sibling `pipelex-temporal` repo's `tests/integration/pipelex_temporal/conftest.py`) is the known fixture-teardown hang point. If a hang traceback mentions `manager.teardown()` or `temporal_hub.reset()`, it's this.-- xdist worker crashes on integration temporal tests often show as `[gw N] node down: Not properly terminated` followed by `replacing crashed worker`. Treat as infrastructure flake until you've proven otherwise serially.-- `is_in_temporal_workflow()` is the right gate for code that might be invoked from inside a Temporal sandbox — `get_config()` is unsafe to read from inside, and many singletons return sandboxed-empty copies. This was the actual bug I hit; it would have been caught instantly by the playbook above if I'd run the integration tests serially on first iteration instead of trusting the xdist run.-- **A payload-conversion error inside a workflow task looks like a clean hang, not a failure.** When the data converter raises while decoding an activity result (or workflow argument), Temporal fails the *workflow task* and retries it indefinitely — the submitter's `execute_workflow` never returns, pytest-timeout's thread-method kill fires, and xdist reports `node down` with the real traceback destroyed along with the process. Historical trigger: kajson ≤ 0.6.0 decoded tz-aware datetimes via `ZoneInfo(...)`, which needs the IANA tz database — absent on slim containers and uv-managed standalone Pythons. Fixed upstream in kajson 0.7.0 (self-sufficient timezone wire format + `tzdata` declared as kajson's own dependency), but any other deterministic decode error (registry miss, schema drift) still presents as this same hang. To diagnose this class of hang on a remote runner, set `PIPELEX_HANG_DUMP_DIR=<dir>`: the temporal integration conftest arms a per-test watchdog that dumps all thread stacks to a file before the kill AND mirrors temporalio's failure logs (which carry the workflow-task traceback) into the same directory; the tests workflow prints the directory on job failure. To reproduce a missing-tz-database environment locally, run with `PYTHONTZPATH=/nonexistent` after uninstalling `tzdata`.+In all cases, write up what you know in 3–4 sentences and surface to the user. Better than spending another 20 minutes on dead ends.diff --git a/docs/contribute/keyword-only-arguments.md b/docs/contribute/keyword-only-arguments.mdindex 0c6d0dc96..9cfa49e78 100644--- a/docs/contribute/keyword-only-arguments.md+++ b/docs/contribute/keyword-only-arguments.md@@ -68,12 +68,11 @@ Only `field_validator`, `model_validator`, and `model_serializer` are actually u### Framework entrypoints-A def is skipped when it is registered with a framework whose calling convention is not ours to dictate. Matching is on the decorator's attribute suffix (the receiver name varies — `app`, `graph_app`, `show_app`, `kit_app`, `build_app`, ...), and on the Temporal and pytest decorators by their attribute/name:+A def is skipped when it is registered with a framework whose calling convention is not ours to dictate. Matching is on the decorator's attribute suffix (the receiver name varies — `app`, `graph_app`, `show_app`, `kit_app`, `build_app`, ...), and on the pytest decorators by their attribute/name:-- Typer/click commands and callbacks — any decorator that is an attribute access ending in `.command` or `.callback`.-- Temporal handlers — `activity.defn`, `workflow.run`, and defensively `workflow.signal`, `workflow.query`, `workflow.update` (none of the latter three are used yet, but they are part of the worker entrypoint surface). Note `workflow.defn` decorates classes, not functions, so it is irrelevant. The custom `@convert_pipelex_errors` decorator is stacked directly below `@activity.defn` and is NOT a framework decorator — the guard must scan the WHOLE decorator stack for a framework match, not just the innermost or outermost decorator.+- Typer/click commands and callbacks — any decorator that is an attribute access ending in `.command` or `.callback`. The guard scans the WHOLE decorator stack for a framework match (a framework decorator may sit above a custom one such as `@convert_pipelex_errors`), not just the innermost or outermost decorator.- pytest fixtures — `fixture`, matched on the bare decorator name so both `@pytest.fixture` and bare `@fixture` (`from pytest import fixture`) are covered (the only src-side fixtures live in the shipped `pipelex/test_extras/shared_pytest_plugins.py` plugin module).-- Jinja2 filters/tests/globals — `pass_context`, `pass_environment`, `pass_eval_context`, matched on the bare decorator name (covers both `@pass_context` from `from jinja2 import pass_context` and the attributed `@jinja2.pass_context`). The Jinja2 engine invokes a filter POSITIONALLY from template syntax — `{{ value | tag("name") }}` calls `tag(context, value, "name")` — so the wrapped callable's arguments cannot be made keyword-only. This is the same framework-entrypoint category as Typer/Temporal/pytest. A multi-argument filter without one of these decorators (rare) is covered by the `# kw-only: ignore` escape hatch; single-argument filters (`escape_script_tag(value)`) are compliant under the single-parameter rule and need nothing.+- Jinja2 filters/tests/globals — `pass_context`, `pass_environment`, `pass_eval_context`, matched on the bare decorator name (covers both `@pass_context` from `from jinja2 import pass_context` and the attributed `@jinja2.pass_context`). The Jinja2 engine invokes a filter POSITIONALLY from template syntax — `{{ value | tag("name") }}` calls `tag(context, value, "name")` — so the wrapped callable's arguments cannot be made keyword-only. This is the same framework-entrypoint category as Typer/pytest. A multi-argument filter without one of these decorators (rare) is covered by the `# kw-only: ignore` escape hatch; single-argument filters (`escape_script_tag(value)`) are compliant under the single-parameter rule and need nothing.FastAPI is intentionally absent: there are no route handlers in `pipelex/` source — the FastAPI server lives in the separate `pipelex-api/` repo.diff --git a/docs/errors/index.md b/docs/errors/index.mdindex 6ec44d69e..bd7ba0ab1 100644--- a/docs/errors/index.md+++ b/docs/errors/index.md@@ -20,5 +20,5 @@ the error came from:- [Inference & providers](inference-and-providers.md) — Inference (Cogt), Provider plugins.- [Platform & tooling](platform-and-tooling.md) — Base & root errors, Tools, Kit, System & configuration, CLI.-See [Error Model](../under-the-hood/error-model.md) for the underlying contract,-classification rules, and the cross-boundary Temporal bridge.+See [Error Model](../under-the-hood/error-model.md) for the underlying contract+and classification rules.diff --git a/docs/under-the-hood/error-model.md b/docs/under-the-hood/error-model.mdindex a84ce4a3f..d429417ba 100644--- a/docs/under-the-hood/error-model.md+++ b/docs/under-the-hood/error-model.md@@ -1,13 +1,13 @@---title: "Error Model"-description: "How Pipelex classifies, carries, and reports errors — the ErrorReport schema, inference error categories, error domains, the layer model, worker classification, and the Temporal error bridge."+description: "How Pipelex classifies, carries, and reports errors — the ErrorReport schema, inference error categories, error domains, the layer model, worker classification, and how classification survives a distributed worker boundary."---# Error Model-In Pipelex, an error is **data**, not a control-flow accident. Every failure is classified once — at the layer that knows the most about it — and that classification travels intact to every consumer: the human reading a Rich panel, the agent parsing JSON, the Temporal retry engine, and the HTTP adapter picking a status code.+In Pipelex, an error is **data**, not a control-flow accident. Every failure is classified once — at the layer that knows the most about it — and that classification travels intact to every consumer: the human reading a Rich panel, the agent parsing JSON, a distributed worker's retry engine, and the HTTP adapter picking a status code.-This page covers the contract that makes that possible: the `ErrorReport` schema, the classification enums, how inference workers classify SDK exceptions, how classification survives every wrapping layer, and how it crosses the Temporal boundary.+This page covers the contract that makes that possible: the `ErrorReport` schema, the classification enums, how inference workers classify SDK exceptions, how classification survives every wrapping layer, and how it survives serialization across a distributed worker boundary.---@@ -22,7 +22,7 @@ Three rules hold across the codebase, and everything else builds on them.**No broad catches in business logic.** `except Exception` is allowed only at CLI entry points and async task roots. Ruff rule `BLE001` enforces this — an unexpected exception crashes loudly instead of being silently swallowed.!!! info "Why classify, instead of just propagating the exception?"- A raw `openai.RateLimitError` tells a Python `except` clause what to catch, but it does not tell the Temporal retry engine whether to retry, the HTTP adapter which status to emit, or an agent whether the failure is the user's fault. Classification turns an exception into a decision input that every consumer can act on uniformly.+ A raw `openai.RateLimitError` tells a Python `except` clause what to catch, but it does not tell a distributed worker's retry engine whether to retry, the HTTP adapter which status to emit, or an agent whether the failure is the user's fault. Classification turns an exception into a decision input that every consumer can act on uniformly.---@@ -93,7 +93,7 @@ report.http_status # 422 / 429 / 500 — for HTTP adapters```!!! warning "`ErrorReport` is `extra="forbid"`"- `from_dict()` rejects unknown keys. `recover_error_report()` calls it directly: a report dict that is found but fails validation is an internal contract bug — the activity bridge and the submitter share the schema within one deploy. `recover_error_report()` catches the `ValidationError` and synthesizes an `UnrecoverableWorkflowFailureError` fallback (carrying the recovered message plus an `[error report failed schema validation]` marker) so failure-webhook delivery stays intact; the workflow still fails afterwards, keeping the contract bug visible. Any other caller of `from_dict()` should treat the validation failure as a bug to fix.+ `from_dict()` rejects unknown keys, so it is the strict inverse of `to_dict()`. A report dict that crosses a serialization boundary and fails validation on the way back is an internal contract bug — the writer and the reader share the schema within one deploy. A cross-boundary recovery helper that rebuilds a report (e.g. a distributed-worker bridge) is expected to catch that `ValidationError` and synthesize a fallback report so failure-webhook delivery stays intact while keeping the contract bug visible; any other caller of `from_dict()` should treat the validation failure as a bug to fix.---@@ -197,7 +197,7 @@ class ProviderErrorMetadata(BaseModel):```!!! warning "`body` is excluded from serialization"- The raw provider response `body` can carry account ids, billing details, or credential fragments. It is held in-process but `exclude`d from every serialized form — CLI JSON, agent output, Temporal details.+ The raw provider response `body` can carry account ids, billing details, or credential fragments. It is held in-process but `exclude`d from every serialized form — CLI JSON, agent output, and any serialized worker payload.`UserAction` pairs a discrete `UserActionKind` (`WAIT_AND_RETRY`, `CHECK_BILLING`, `CHECK_CREDENTIALS`, `CHANGE_INPUT`, `CHANGE_MODEL`, `CONTACT_SUPPORT`, `UNKNOWN`) with a free-form `detail` string — so the CLI can render consistent guidance while keeping provider-specific text.@@ -242,53 +242,21 @@ A wrapper keeps its own `error_type` and `message` but inherits every classifica----## The Temporal Error Bridge+## Crossing a Distributed Worker Boundary-When a pipe runs on a Temporal worker, the error must survive serialization across the **activity → workflow → submitter** boundary. Temporal's default failure converter would wrap a raw `PipelexError` without packing the `ErrorReport` or deriving the retry decision. The bridge closes that gap.+The error model is built to survive serialization. Because `ErrorReport` round-trips through `to_dict()` / `from_dict()`, a failure that happens on a remote worker can reach the submitting process with its full classification intact — not just a message string.-### Activity Side — `convert_pipelex_errors`+The runtime itself stays transport-agnostic: the machinery that carries an error across a worker boundary ships in the **host-runtime plugin** for each distributed backend, not in core. A backend plugin is responsible for three things.-A decorator applied beneath `@activity.defn` on every in-scope activity converts a `PipelexError` into a `TemporalError`.+**Packing.** Convert a `PipelexError` into the transport's failure type and stash `to_error_report().to_dict()` in its details payload, so worker and submitter code keep the full classification rather than a bare message. The same step derives the transport's retry decision from `InferenceErrorCategory.is_retryable`.-```python-@convert_pipelex_errors-async def act_llm_gen_text(llm_assignment: LLMAssignment) -> str:- return await llm_gen_text(llm_assignment=llm_assignment)-```--`TemporalError.from_message_exception()` does two things:--- **Derives `non_retryable`** from `InferenceErrorCategory.is_retryable` for category-carrying errors. (The configured `non_retryable_error_types` class-name list is the fallback for category-less exceptions.)-- **Packs the report** — `to_error_report().to_dict()` goes into `ApplicationError.details`, so workflow code keeps the full classification, not just a message string.--### Submitter Side — `recover_error_report`--Once the failure returns to the process that submitted the workflow, `recover_error_report()` walks the `__cause__` chain for the `ApplicationError`, pulls the details-packed dict, and rebuilds the `ErrorReport`. It is **total** — callers in the error-recovery path always get a structured report.--```python-def recover_error_report(exc: BaseException) -> ErrorReport:- report_dict = _find_error_report_dict(exc)- if report_dict is not None:- return ErrorReport.from_dict(report_dict)- return UnrecoverableWorkflowFailureError(_message_from_exc(exc)).to_error_report()-```--When no report dict is found in the chain — a non-Pipelex exception, a worker crash, a heartbeat timeout — the function synthesizes an `UnrecoverableWorkflowFailureError` report so the recovery path always has structured classification to surface. A report dict that fails `ErrorReport.from_dict` validation is treated as an internal contract bug (writer and reader share the schema within one deploy) and is also synthesized into the same `UnrecoverableWorkflowFailureError` fallback — carrying the recovered message plus an `[error report failed schema validation]` marker — so the failure webhook still fires; the workflow still fails afterwards, keeping the contract bug visible.--The recovered report is carried on `WorkflowExecutionError(error_report=...)`, whose `to_error_report()` override returns it. Since `WorkflowExecutionError` is a `PipelexError`, `PipelineExecutionError` inherits the classification natively.--### Workflow-Level Fail-Safe Floor--The activity bridge converts every error that crosses an *activity* boundary. But a `PipelexError` raised **inline in workflow code** — an operator that runs its leaf inline instead of dispatching it as an activity, or an inline setup error — is neither an `ActivityError` nor an `ApplicationError`. Temporal's default for an unhandled exception in workflow code that is *not* a recognized failure type is to treat it as a **workflow-task failure** and retry the task indefinitely (bounded only by the workflow execution timeout). That is the single worst outcome a durable-execution system can produce: invisible, resource-consuming, and surfaced only after ~the timeout as a generic timeout error rather than the real cause. The fail-safe floor closes that hole on two levels:--- **Workflow-level catch-all (rich path).** `WfPipeRouter.run()` and `WfPipeRun.run()` each end their boundary handling with an `except PipelexError` clause. `WfPipeRouter` converts a genuine inline error to a terminal `TemporalError` via `from_message_exception` (same conversion the activity bridge does — the structured report rides in `ApplicationError.details`). `WfPipeRun` routes it through its deferred-delivery path so the FAILED webhook still fires, then chains a details-carrying `TemporalError` so the classification also survives to the submitter. The catch-all only converts *genuine* inline errors: an escaping `PipelexError` that already carries a Temporal failure in its `__cause__` chain (e.g. a nested child-dispatch that already wrapped a sub-pipe failure as `WorkflowExecutionError`) is already terminal and recoverable, so it propagates untouched rather than being flattened to a generic report.+**Recovering.** On the submitter side, walk the returned failure, pull the packed dict, and rebuild the `ErrorReport`. Recovery is **total**: when no report dict is found — a non-Pipelex exception, a worker crash, a timeout — the plugin synthesizes a fallback report so the recovery path always has structured classification to surface.-- **Worker-level floor (belt-and-suspenders).** The worker registers `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]`. Any domain error that slips past the catch-alls still fails the workflow **terminally** instead of hanging — degrading to a synthesized `UnrecoverableWorkflowFailureError` report (message preserved, classification floored) rather than a silent retry loop. The scope is deliberately `PipelexError`, not `Exception`: genuinely transient Temporal/infra errors and deterministic-replay glitches are not domain errors, and workflow-task retry is the *correct* behavior for them — so they keep Temporal's default.+**A fail-safe floor.** Ensure a domain error that escapes the conversion path fails the unit of work *terminally* rather than hanging. In a durable-execution system the default for an unconverted exception might be to retry forever, so "convert all the errors we know about" is not enough — the floor must hold for the errors, and the code paths, that nobody enumerated.-The rule of thumb: in a durable-execution system the default for "unexpected exception in business logic inside a workflow" is the most dangerous behavior available (retry forever), so "we convert all the errors we know about" is not enough — the floor must hold for the errors, and the code paths, that nobody enumerated.+**Net effect:** a pipe failing on a remote worker reaches the CLI and HTTP adapters with the *same* `error_category` / `retryable` / `model` / `provider` / `user_action` as the identical failure run locally — and a failure that escapes conversion fails loud and bounded instead of hanging.-**Net effect:** a pipe failing on a Temporal worker reaches the CLI and HTTP adapters with the *same* `error_category` / `retryable` / `model` / `provider` / `user_action` as the identical failure run locally — and a failure that escapes inline fails loud and bounded instead of hanging.+See [Runtime Bridge & Transport](./runtime-bridge-and-transport.md) for the boundary these converters span; the per-backend converters themselves live in the host-runtime plugins.---@@ -331,7 +299,7 @@ A downstream FastAPI exception handler calls `ErrorReport.http_status` and is a### Inputs and Outputs-**Inputs.** `to_error_report()` takes a live `PipelexError`. `recover_error_report()` takes any `BaseException` and walks its `__cause__` chain. `ErrorReport.from_dict()` takes a `to_dict()` payload — strictly, raising `ValidationError` on drift.+**Inputs.** `to_error_report()` takes a live `PipelexError`. `ErrorReport.from_dict()` takes a `to_dict()` payload — strictly, raising `ValidationError` on drift. (A distributed-worker bridge adds a cross-boundary recovery helper that walks a returned failure's `__cause__` chain and rebuilds the report; it lives in the host-runtime plugin, not core.)**Outputs.** `to_error_report()` returns an `ErrorReport`; `to_dict()` returns a `None`-free `dict`. Side effects: telemetry events emitted on pipeline failure at Layer 3; the agent CLI writes to stderr and raises `typer.Exit(...)` — code 1 by default, or the validate surface's 0/1/2 policy (see [Validate exit-code policy](#validate-exit-code-policy-0-1-2)).@@ -354,8 +322,8 @@ flowchart TBREPORT --> AGENT["Agent CLI<br/>JSON / Markdown"]REPORT --> HTTP["HTTP adapters<br/>.http_status"]- W -.->|"@convert_pipelex_errors"| TEMP["Temporal bridge<br/>TemporalError → ApplicationError.details"]- TEMP -.->|"recover_error_report()"| REPORT+ W -.->|"pack on worker"| TEMP["Distributed worker bridge (plugin)<br/>report packed into transport details"]+ TEMP -.->|"recover on submitter"| REPORTclassDef src fill:#fff3e0,stroke:#e65100,color:#000classDef cls fill:#e8eaf6,stroke:#3949ab,color:#000@@ -417,7 +385,6 @@ report.error_category # "transient" / "capacity" / ...# Round-trip across a boundaryErrorReport.from_dict(payload) # strict inverse of to_dict()-recover_error_report(temporal_failure) # walk __cause__ for an ApplicationError# Retry decisionInferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT@@ -436,9 +403,6 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| `pipelex/cogt/inference/provider_name.py` | `ProviderName` enum keying the extract-fn registry || `pipelex/plugins/*/` | Per-provider inference workers — Layer 0 → 1 classification || `pipelex/pipeline/exceptions.py` | `PipelineExecutionError`, `PipeExecutionError` |-| `pipelex_temporal/tprl/temporal_error.py` *(in the closed `pipelex-temporal` plugin)* | `TemporalError`, `from_message_exception`, `recover_error_report` |-| `pipelex_temporal/tprl/activity_error_boundary.py` *(plugin)* | `convert_pipelex_errors` decorator |-| `pipelex_temporal/tprl/workflow_caller.py` *(plugin)* | `WorkflowExecutor`, `WorkflowExecutionError` recovery || `pipelex/cli/error_handlers.py` | Human CLI Rich panels — `display_error_panel()` || `pipelex/cli/agent_cli/commands/agent_output.py` | Agent CLI JSON / markdown delivery |@@ -454,8 +418,8 @@ InferenceErrorCategory.TRANSIENT.is_retryable # True — only TRANSIENT| LLM returns schema-mismatched JSON | `instructor` re-asks; if exhausted → `UNKNOWN` || Connection dropped mid-request | `AMBIGUOUS` → non-retryable (outcome unknown) || Wrapper exception (no own category) | Inherits cause's classification via enrichment |-| Failure on a Temporal worker | `ErrorReport` recovered from `ApplicationError.details` — same classification as local |-| Worker exception with no `ErrorReport` | Synthesized `UnrecoverableWorkflowFailureError` report — `error_domain = RUNTIME` |+| Failure on a distributed worker | `ErrorReport` recovered from the transport's serialized details — same classification as local |+| Worker exception with no `ErrorReport` | Synthesized fallback report — `error_domain = RUNTIME` |---diff --git a/pipelex/base_exceptions.py b/pipelex/base_exceptions.pyindex bce4bc912..8aea0dff7 100644--- a/pipelex/base_exceptions.py+++ b/pipelex/base_exceptions.py@@ -14,13 +14,12 @@ def iter_cause_chain(exc: BaseException) -> Iterator[BaseException]:The single cycle-guarded ``__cause__`` walk shared across the error-reportingpaths — classification recovery (``find_inference_error_category_in_chain``),- report extraction and message synthesis (``_find_error_report_dict`` /- ``_message_from_exc``), and the inline fail-safe guard- (``_carries_temporal_failure``). The ``id()`` set makes a cyclic ``__cause__``- chain terminate instead of spinning forever. That guard runs *on the error- path*, so getting it wrong would turn the failure being reported into the very- hang the reporting exists to surface — centralizing it here means it is- written, and audited, once.+ the is-self cycle check in ``_enrich_error_report_from_cause``, LLM-error+ formatting (``PipeLLM._format_llm_error``), and the agent-CLI source-location+ extraction. The ``id()`` set makes a cyclic ``__cause__`` chain terminate+ instead of spinning forever. That walk runs *on the error path*, so getting it+ wrong would turn the failure being reported into a hang — centralizing it here+ means it is written, and audited, once."""node: BaseException | None = excseen: set[int] = set()diff --git a/pipelex/cli/dev_cli/commands/keyword_only_guard.py b/pipelex/cli/dev_cli/commands/keyword_only_guard.pyindex d4dc21a8f..7030c7d66 100644--- a/pipelex/cli/dev_cli/commands/keyword_only_guard.py+++ b/pipelex/cli/dev_cli/commands/keyword_only_guard.py@@ -79,21 +79,10 @@ TYPER_DECORATOR_ATTRS = frozenset({"command", "callback"})#: pytest's ``fixture`` is written both as ``@pytest.fixture`` and as bare ``@fixture`` (``from pytest import fixture``).#: Jinja2's ``pass_context`` / ``pass_environment`` / ``pass_eval_context`` mark a filter/test/global whose#: wrapped callable is invoked POSITIONALLY by the Jinja2 engine (``{{ x|filter(arg) }}``), so its arguments-#: cannot be made keyword-only — same framework-entrypoint category as Typer/Temporal/pytest. Written bare+#: cannot be made keyword-only — same framework-entrypoint category as Typer/pytest. Written bare#: (``@pass_context``, ``from jinja2 import pass_context``) or attributed (``@jinja2.pass_context``).BARE_FRAMEWORK_DECORATOR_NAMES = frozenset({"fixture", "pass_context", "pass_environment", "pass_eval_context"})-#: Temporal handler decorators matched on their trailing two attribute segments.-TEMPORAL_DECORATOR_TAILS = frozenset(- {- ("activity", "defn"),- ("workflow", "run"),- ("workflow", "signal"),- ("workflow", "query"),- ("workflow", "update"),- }-)-#: Typer parameter-annotation metadata callables that mark a call-style CLI entrypoint.TYPER_ANNOTATION_ATTRS = frozenset({"Argument", "Option"})@@ -134,19 +123,6 @@ class Violation(NamedTuple):# ---------------------------------------------------------------------------------------def _attribute_tail(node: ast.expr) -> tuple[str, ...]:- """Return the trailing attribute segments of a dotted expression (e.g. ``a.b.c`` -> ``(a, b, c)``)."""- segments: list[str] = []- current: ast.expr = node- while isinstance(current, ast.Attribute):- segments.append(current.attr)- current = current.value- if isinstance(current, ast.Name):- segments.append(current.id)- segments.reverse()- return tuple(segments)--def _decorator_matches_carveout(decorator: ast.expr) -> bool:"""Whether a single decorator expression matches any framework / pydantic carve-out."""target: ast.expr = decorator.func if isinstance(decorator, ast.Call) else decorator@@ -169,14 +145,11 @@ def _decorator_matches_carveout(decorator: ast.expr) -> bool:if isinstance(target, ast.Attribute):if target.attr in TYPER_DECORATOR_ATTRS:return True- tail = _attribute_tail(target)- if len(tail) >= 2 and tail[-2:] in TEMPORAL_DECORATOR_TAILS:- return Truereturn Falsedef _has_carveout_decorator(node: ast.FunctionDef | ast.AsyncFunctionDef) -> bool:- """Scan the WHOLE decorator stack for a carve-out match (e.g. @activity.defn above @convert_pipelex_errors)."""+ """Scan the WHOLE decorator stack for a carve-out match (a framework decorator may sit above a custom one)."""return any(_decorator_matches_carveout(decorator) for decorator in node.decorator_list)@@ -353,7 +326,7 @@ class _Collector(ast.NodeVisitor):self._in_class_body: tuple[bool, ...] = ()@override- def visit_ClassDef(self, node: ast.ClassDef) -> None: # ast visitor naming+ def visit_ClassDef(self, node: ast.ClassDef) -> None: # pylint: disable=invalid-name # ast.NodeVisitor dispatch nameself._class_stack = (*self._class_stack, node.name)self._in_class_body = (*self._in_class_body, True)self.generic_visit(node)@@ -384,11 +357,11 @@ class _Collector(ast.NodeVisitor):self._in_class_body = self._in_class_body[:-1]@override- def visit_FunctionDef(self, node: ast.FunctionDef) -> None: # ast visitor naming+ def visit_FunctionDef(self, node: ast.FunctionDef) -> None: # pylint: disable=invalid-name # ast.NodeVisitor dispatch nameself._visit_function(node)@override- def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: # ast visitor naming+ def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None: # pylint: disable=invalid-name # ast.NodeVisitor dispatch nameself._visit_function(node)diff --git a/pipelex/cogt/inference/transport_retry.py b/pipelex/cogt/inference/transport_retry.pyindex 3e10edb97..67db9699b 100644--- a/pipelex/cogt/inference/transport_retry.py+++ b/pipelex/cogt/inference/transport_retry.py@@ -38,7 +38,7 @@ _PRE_REQUEST_TRANSPORT_ERRORS: tuple[type[httpx.TransportError], ...] = (httpx.PoolTimeout,)# The OpenAI / Anthropic SDKs honor Retry-After only up to ~60s, then fall back to backoff; match-# that boundary — a longer wait is the Temporal line, not something to chase in direct mode.+# that boundary — a longer wait is the durable-execution track's job, not something to chase in direct mode._MAX_RETRY_AFTER_SECONDS: float = 60.0# Full-jitter exponential backoff: each wait is drawn from uniform(0, exponential_bound) sodiff --git a/pipelex/errors/error_pages_generator.py b/pipelex/errors/error_pages_generator.pyindex db4a2b232..94ec0a47b 100644--- a/pipelex/errors/error_pages_generator.py+++ b/pipelex/errors/error_pages_generator.py@@ -404,7 +404,6 @@ _SUBSYSTEM_SECTIONS: tuple[tuple[str, str, str], ...] = (# Execution & runtime("pipe_run", "execution-and-runtime", "Pipe execution"),("pipeline", "execution-and-runtime", "Pipeline execution"),- ("temporal", "execution-and-runtime", "Temporal execution"),("runtime_bridge", "execution-and-runtime", "Runtime bridge"),("graph", "execution-and-runtime", "Graph"),("tracing", "execution-and-runtime", "Tracing"),@@ -539,8 +538,8 @@ def render_index_page(by_subsystem: dict[str, list[type[PipelexError]]]) -> str:lines.extend(["",- "See [Error Model](../under-the-hood/error-model.md) for the underlying contract,",- "classification rules, and the cross-boundary Temporal bridge.",+ "See [Error Model](../under-the-hood/error-model.md) for the underlying contract",+ "and classification rules.","",])diff --git a/pipelex/hub.py b/pipelex/hub.pyindex 4542cd18e..c43daa4dd 100644--- a/pipelex/hub.py+++ b/pipelex/hub.py@@ -1,5 +1,5 @@import sys-from collections.abc import Generator, Sequence+from collections.abc import Callable, Generator, Sequencefrom contextlib import contextmanagerfrom contextvars import ContextVarfrom pathlib import Path@@ -56,6 +56,15 @@ if TYPE_CHECKING:from pipelex.tracing.event_log_protocol import EventLogProtocol+def _never_in_isolated_execution() -> bool:+ """Core default isolated-execution probe: the in-process orchestrator has no replay/activity+ split, so an emission is never inside an isolated sub-execution. A boot-orchestrator plugin+ whose runtime has such a split (a Temporal worker) replaces this by claiming+ ``HubSlot.ISOLATED_EXECUTION_PROBE``.+ """+ return False++class PipelexHub:"""PipelexHub serves as a central dependency manager to break cyclic imports between components.It provides access to core providers and factories through a singleton instance,@@ -96,6 +105,11 @@ class PipelexHub:self._pipe_library: PipeLibraryAbstract | None = Noneself._pipe_router: PipeRouterProtocol | None = Noneself._pipe_run: PipeRunProtocol | None = None+ # Ambient probe claimed by a boot-orchestrator plugin (ISOLATED_EXECUTION_PROBE): True when+ # the current call runs inside an isolated sub-execution (a Temporal activity) whose emissions+ # must bypass the parent run's registered buffer. Core default never isolated (see+ # _never_in_isolated_execution); consumed by ReportingManager to route usage emissions.+ self._isolated_execution_probe: Callable[[], bool] = _never_in_isolated_execution# pipelineself._pipeline_manager: PipelineManagerAbstract | None = None@@ -230,6 +244,9 @@ class PipelexHub:def set_pipe_run(self, pipe_run: "PipeRunProtocol") -> None:self._pipe_run = pipe_run+ def set_isolated_execution_probe(self, probe: Callable[[], bool]) -> None:+ self._isolated_execution_probe = probe+def set_pipeline_manager(self, pipeline_manager: PipelineManagerAbstract):self._pipeline_manager = pipeline_manager@@ -386,6 +403,15 @@ class PipelexHub:raise RuntimeError(msg)return self._pipe_run+ def is_in_isolated_execution(self) -> bool:+ """True when the current call runs inside an isolated sub-execution whose side-effecting+ emissions must not be written into the parent run's registered (replay-deterministic) buffer.++ Delegates to the boot-orchestrator's claimed probe (a Temporal worker reports True while a+ call executes inside an activity); the core default is always False.+ """+ return self._isolated_execution_probe()+def get_required_pipeline_manager(self) -> PipelineManagerAbstract:if self._pipeline_manager is None:msg = "PipelineManager is not initialized"@@ -534,6 +560,11 @@ def get_report_delegate() -> ReportingProtocol:return get_pipelex_hub().get_report_delegate()+def is_in_isolated_execution() -> bool:+ """Module-level accessor — see :meth:`PipelexHub.is_in_isolated_execution`."""+ return get_pipelex_hub().is_in_isolated_execution()++_content_generator_override: ContextVar[ContentGeneratorProtocol | None] = ContextVar("content_generator_override", default=None)diff --git a/pipelex/kit/agent_rules/pipelex_standards.md b/pipelex/kit/agent_rules/pipelex_standards.mdindex 777cfaa50..cecabef06 100644--- a/pipelex/kit/agent_rules/pipelex_standards.md+++ b/pipelex/kit/agent_rules/pipelex_standards.md@@ -24,6 +24,6 @@ Non-subject function parameters across `pipelex/` source must be **keyword-only*- `def f(subject, *, opt1, opt2): ...` — the first non-`self`/`cls` parameter (the subject) may stay positional; everything after it must be keyword-only. Making the subject keyword-only too (`def f(*, a, b)`) is always allowed and often preferable.- A lone subject (`def render(node)`) is compliant; a second bare positional (`def f(a, b)`, `def truncate(text, max_length=80)`) is a violation.-The rule is mechanically enforced by the `check-keyword-only` AST guard, which runs in `make agent-check`, in the `make check` aggregate, and in CI; the tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs (dunders, pydantic validators/serializers, Typer/Temporal/pytest/Jinja2 framework entrypoints, `@override` impls) are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line). Watch for functions a framework or the interpreter invokes positionally (callbacks, `__import__` hooks, route handlers): the type checker is blind to those, so `make agent-test` is the safety net.+The rule is mechanically enforced by the `check-keyword-only` AST guard, which runs in `make agent-check`, in the `make check` aggregate, and in CI; the tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs (dunders, pydantic validators/serializers, Typer/pytest/Jinja2 framework entrypoints, `@override` impls) are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line). Watch for functions a framework or the interpreter invokes positionally (callbacks, `__import__` hooks, route handlers): the type checker is blind to those, so `make agent-test` is the safety net.The full specification — the two exceptions, the carve-out list, the symmetric-tuple allowlist, the escape hatch, and worked examples — is in [`docs/contribute/keyword-only-arguments.md`](docs/contribute/keyword-only-arguments.md).diff --git a/pipelex/kit/agent_rules/python_standards.md b/pipelex/kit/agent_rules/python_standards.mdindex 4555fb4e2..e7c595207 100644--- a/pipelex/kit/agent_rules/python_standards.md+++ b/pipelex/kit/agent_rules/python_standards.md@@ -132,7 +132,7 @@ Across `pipelex/` source, non-subject function parameters must be **keyword-only- `def f(subject, *, opt1, opt2): ...` — the first non-`self`/`cls` parameter (the subject) may stay positional; everything after it must be keyword-only. Making the subject keyword-only too (`def f(*, a, b)`, `def m(self, *, a, b)`) is always allowed and is the form the auto-fixer produces.- A lone subject (`def render(node)`) is compliant; a second bare positional (`def f(a, b)`, `def truncate(text, max_length=80)`) is a violation.-The rule is mechanically enforced by the `check-keyword-only` AST guard (`make cko`), which runs in `make agent-check`, in the `make check` aggregate, in CI, and via a Claude Code `PostToolUse` hook that checks each edited file at edit time. The tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs — dunders, pydantic validators/serializers, Typer/Temporal/pytest/Jinja2 framework entrypoints, `@override` impls, and a small curated symmetric-tuple allowlist — are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line).+The rule is mechanically enforced by the `check-keyword-only` AST guard (`make cko`), which runs in `make agent-check`, in the `make check` aggregate, in CI, and via a Claude Code `PostToolUse` hook that checks each edited file at edit time. The tree is fully compliant, so it hard-blocks on **any** violation. Carve-outs — dunders, pydantic validators/serializers, Typer/pytest/Jinja2 framework entrypoints, `@override` impls, and a small curated symmetric-tuple allowlist — are skipped automatically. A genuinely justified one-off uses an inline `# kw-only: ignore` comment on the `def` line (place it right after the open paren so `ruff format` keeps it on the header line).**Auto-fix.** `make fix-keyword-only` (alias `fko`, or `pipelex-dev check-keyword-only --fix`) rewrites every mechanically-fixable violation by inserting a bare `*` as far left as possible — right after `self`/`cls` — so every non-`self`/`cls` parameter becomes keyword-only, and re-parses each rewrite before writing (a rewrite that would not parse is discarded and reported instead). It runs automatically early in `make agent-check`, before `ruff format`. It is **non-gating**: it mutates and reports but never exits non-zero — the read-only `check-keyword-only` (run last in `agent-check`, and the variant `make check` / CI use) owns the pass/fail gate. A few shapes a single `*` insert cannot fix are reported for a manual fix instead of rewritten: a `*args` is present, a keyword-only section already exists, or two-or-more positional-only params remain.diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.tomlindex 3126ec3dc..25d68f91d 100644--- a/pipelex/kit/configs/pipelex.toml+++ b/pipelex/kit/configs/pipelex.toml@@ -70,7 +70,7 @@ theme = "dark"palette = "dracula"####################################################################################################-# Tracing Config (event log for distributed tracing across Temporal workers)+# Tracing Config (event log for distributed tracing across workers)####################################################################################################[pipelex.tracing_config]@@ -84,10 +84,6 @@ traces_dir = ".pipelex/traces"table_name = ""region = ""-[pipelex.tracing_config.temporal_dynamodb]-table_name = ""-region = ""-##################################################################################################### Storage Config####################################################################################################diff --git a/pipelex/kit/paths.py b/pipelex/kit/paths.pyindex 0848faa3a..5295565ce 100644--- a/pipelex/kit/paths.py+++ b/pipelex/kit/paths.py@@ -38,7 +38,6 @@ CONFIG_SYNC_EXCLUDED_FILES: frozenset[str] = GIT_IGNORED_CONFIG_FILES | {"telemeGIT_IGNORED_CONFIG_DIRS: frozenset[str] = frozenset({"storage",- "temporal-payload-store","traces",})diff --git a/pipelex/pipe_controllers/batch/pipe_batch.py b/pipelex/pipe_controllers/batch/pipe_batch.pyindex 8d34cc226..0e8a6be56 100644--- a/pipelex/pipe_controllers/batch/pipe_batch.py+++ b/pipelex/pipe_controllers/batch/pipe_batch.py@@ -19,6 +19,7 @@ from pipelex.pipe_run.pipe_job_factory import PipeJobFactoryfrom pipelex.pipe_run.pipe_run_params import BatchParams, PipeRunParamsfrom pipelex.pipeline.job_metadata import JobMetadatafrom pipelex.tools.misc.async_utils import gather_bounded+from pipelex.urls import URLsif TYPE_CHECKING:from collections.abc import Awaitable, Callable@@ -28,7 +29,7 @@ if TYPE_CHECKING:from pipelex.libraries.library_crate import LibraryCrate# When a single PipeBatch fans out over more than this many items, log a one-time advisory pointing at-# the Temporal track — bounded fan-out is a basic backpressure effort, not durable, rate-limited execution.+# durable execution — bounded fan-out is a basic backpressure effort, not durable, rate-limited execution.LARGE_BATCH_ADVISORY_THRESHOLD = 100@@ -146,7 +147,7 @@ class PipeBatch(PipeController):log.warning(f"PipeBatch '{self.code}' is fanning out over {item_count} items. Bounded fan-out "f"(max_concurrency={max_concurrency_setting}) is a basic backpressure effort, not durable execution — "- f"for a workload this size, consider running on Temporal for durable, rate-limited execution."+ f"for a workload this size, consider a durable execution backend for rate-limited, resumable runs: {URLs.durable_execution}")async def _run_branch(item_input_stuff: "Stuff", *, branch_output_item_code: str) -> PipeOutput:diff --git a/pipelex/pipe_run/exceptions.py b/pipelex/pipe_run/exceptions.pyindex 942453307..7d987aa8e 100644--- a/pipelex/pipe_run/exceptions.py+++ b/pipelex/pipe_run/exceptions.py@@ -6,11 +6,13 @@ class AsyncExecutionNotEnabledError(PipelexError):"""Raised when a route that depends on asynchronous execution is hit on adeployment that does not have an async execution backend enabled.- Backend-neutral on purpose: the same condition can be produced today by a- Temporal-backed runner finding ``[temporal] is_enabled = false`` and will be- produced by other async backends (e.g. Mistral Workflows) as support lands.- The class name, title, and detail therefore talk about *async execution* as- a capability of the deployment, not about any specific backend brand.+ Backend-neutral on purpose, and it lives in core precisely because it is the+ shared contract between the runner API — which maps it to an HTTP status (501)+ — and any async-execution backend plugin that raises it: the Temporal plugin+ today, other async backends (e.g. Mistral Workflows) as support lands. Core+ itself never raises it. The class name, title, and detail therefore talk about+ *async execution* as a capability of the deployment, not about any specific+ backend brand.``error_domain = CONFIG`` because the caller's request is well-formed; whatis missing is server-side configuration. The pipelex-api layer maps this@@ -31,9 +33,11 @@ class AsyncExecutionNotEnabledError(PipelexError):@classmethoddef with_default_message(cls) -> "AsyncExecutionNotEnabledError":- """Construct with the canonical backend-neutral message used by the- facade-level dispatch guard (``with_conditional_worker``) and the- lower-level ``WorkflowExecutor.temporal_client()`` boundary.+ """Construct with the canonical backend-neutral message.++ Raised by an async-execution backend plugin at its dispatch /+ client-acquisition boundary (e.g. the Temporal plugin) to refuse a request+ that needs async execution when no such backend is enabled."""return cls(cls.DEFAULT_MESSAGE)diff --git a/pipelex/pipelex.py b/pipelex/pipelex.pyindex 654ad2ab3..ee36097e5 100644--- a/pipelex/pipelex.py+++ b/pipelex/pipelex.py@@ -456,6 +456,17 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.if task_manager_factory is not None:task_manager_factory()+ # --- Isolated-execution probe ----------------------------------------------------------+ # Claimed only by a boot-orchestrator plugin whose runtime has a replay/activity split (e.g. a+ # Temporal worker): the thunk resolves the ambient predicate that reports whether the current+ # call runs inside an isolated sub-execution (an activity). ReportingManager consults it to+ # route an activity-side usage emission to the per-process log instead of the workflow's+ # registered buffer (audit H1). Unclaimed (any in-process boot), the fresh hub's default+ # reports "never isolated", so no wiring is needed here.+ isolated_execution_probe_factory = plugin_registrar.slot_claims.get(HubSlot.ISOLATED_EXECUTION_PROBE)+ if isolated_execution_probe_factory is not None:+ self.pipelex_hub.set_isolated_execution_probe(isolated_execution_probe_factory())+# --- Pipe Router -----------------------------------------------------------------------# Injection precedence (codex C8): explicit setup() param > plugin slot-claim thunk > core default.diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex 84398c551..54144c5c5 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -341,7 +341,7 @@ theme = "dark"palette = "dracula"####################################################################################################-# Tracing config (event log for distributed tracing across Temporal workers)+# Tracing config (event log for distributed tracing across workers)####################################################################################################[pipelex.tracing_config]@@ -355,10 +355,6 @@ traces_dir = ".pipelex/traces"table_name = ""region = ""-[pipelex.tracing_config.temporal_dynamodb]-table_name = ""-region = ""-##################################################################################################### Dry run config####################################################################################################diff --git a/pipelex/plugins/builtins.py b/pipelex/plugins/builtins.pyindex bbdbacaf9..690178120 100644--- a/pipelex/plugins/builtins.py+++ b/pipelex/plugins/builtins.py@@ -19,12 +19,6 @@ from pipelex.plugins.pypdfium2.pypdfium2_plugin import Pypdfium2Plugin# The plugins Pipelex ships with — discovered at boot ahead of any external# entry-point plugin. Each is import-light: importing this module imports no# backend SDK (the SDKs load lazily inside the make_worker closures).-#-# Temporal is NOT here: it ships as the external ``pipelex-temporal`` distribution,-# discovered via its ``pipelex.plugins`` entry point. Installing that dist makes the-# ``"temporal"`` orchestrator (and, when a process is booted under it via-# ``plugins.boot_orchestrator = "temporal"``, the worker runtime) available — zero-# config, no core import of ``temporalio``.BUILTIN_PLUGINS: list[PipelexPlugin] = [DirectOrchestratorPlugin(),OpenAIPlugin(),diff --git a/pipelex/plugins/bundle_validator_registry.py b/pipelex/plugins/bundle_validator_registry.pyindex 90701e1f1..f6fc0c942 100644--- a/pipelex/plugins/bundle_validator_registry.py+++ b/pipelex/plugins/bundle_validator_registry.py@@ -2,10 +2,9 @@A bundle validator produces a validation verdict for one orchestration mode, the way anorchestrator runs a pipe for one mode. The seam is generic across orchestrators: the-core ``direct`` plugin contributes the in-process validator (``"direct"``), the external-``pipelex-temporal`` plugin contributes the worker-dispatched validator (under-``"temporal"``), and a Mistral validator can slot in later — none of which core or a-host runtime names.+core ``direct`` plugin contributes the in-process validator (``"direct"``), and an+external orchestrator plugin can contribute a worker-dispatched validator under its own+mode token — which core never names.Verdict-as-value (not raise): ``validate_bundles`` *returns* the verdict — the validreport (a ``ValidationReport``) or the invalid ``ErrorReport`` (carrying@@ -49,13 +48,13 @@ class BundleValidatorProtocol(Protocol):"""How ``/validate`` produces a verdict under one orchestration mode.A validator plugin registers one of these per ``orchestration_mode`` token it serves- (``"direct"`` in core; ``"temporal"`` from ``pipelex-temporal``;- ``"mistralai-workflows"`` later). The API resolves the mode and dispatches through the- ``BundleValidatorRegistry`` instead of branching on a backend. Validation is inherently- blocking, so there is no ``delivery`` axis here (unlike ``OrchestratorProtocol.run``).+ (``"direct"`` in core; external orchestrator plugins register their own tokens). The+ API resolves the mode and dispatches through the ``BundleValidatorRegistry`` instead+ of branching on a backend. Validation is inherently blocking, so there is no+ ``delivery`` axis here (unlike ``OrchestratorProtocol.run``).``library_dirs`` is host context the in-process arm needs to load the method library;- a dispatched arm (Temporal) ignores it — its worker loads its own library.+ a worker-dispatched arm ignores it — its worker loads its own library."""async def validate_bundles(diff --git a/pipelex/plugins/registrar.py b/pipelex/plugins/registrar.pyindex 35af549cc..5345f4cd4 100644--- a/pipelex/plugins/registrar.py+++ b/pipelex/plugins/registrar.py@@ -62,6 +62,7 @@ class HubSlot(StrEnum):PIPE_ROUTER = "pipe_router"PIPE_RUN = "pipe_run"TASK_MANAGER = "task_manager"+ ISOLATED_EXECUTION_PROBE = "isolated_execution_probe"class PluginOrigin(StrEnum):@@ -222,6 +223,9 @@ class PluginRegistrar:def claim_task_manager(self, factory: Callable[[], Any]) -> None:self._claim(slot=HubSlot.TASK_MANAGER, factory=factory)+ def claim_isolated_execution_probe(self, factory: Callable[[], Any]) -> None:+ self._claim(slot=HubSlot.ISOLATED_EXECUTION_PROBE, factory=factory)+def add_teardown(self, callback: Callable[[], None]) -> None:self.teardown_callbacks.append(callback)self._active.contributions.append("teardown callback")diff --git a/pipelex/reporting/reporting_manager.py b/pipelex/reporting/reporting_manager.pyindex d6ccd0694..d230b12e3 100644--- a/pipelex/reporting/reporting_manager.py+++ b/pipelex/reporting/reporting_manager.py@@ -1,4 +1,3 @@-import sysfrom datetime import datetime, timezonefrom typing import NamedTuple, cast@@ -13,6 +12,7 @@ from pipelex.cogt.llm.llm_job import LLMJobfrom pipelex.cogt.search.search_job import SearchJobfrom pipelex.config import get_configfrom pipelex.graph.trace_context import TraceContext+from pipelex.hub import is_in_isolated_executionfrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.reporting.reporting_types import AnyTokensUsagefrom pipelex.system.exceptions import MissingDependencyError@@ -20,24 +20,6 @@ from pipelex.tracing.activity_event_log import ActivityEventLogCachefrom pipelex.tracing.event_log_protocol import EventLogProtocolfrom pipelex.tracing.trace_events import UsageReportEvent--def _is_in_temporal_activity() -> bool:- """True when the current call stack runs inside a Temporal activity.-- ``sys.modules`` sniff instead of importing temporalio: if ``temporalio.activity`` was never- imported in this process, no activity context can exist — the context is set by temporalio's- own machinery, which requires the module to be imported. Importing it here would put the- entire temporalio extra (Rust bridge, protobuf) on every boot's critical path wherever the- ``pipelex-temporal`` plugin is installed, even for processes that never touch Temporal.- Worker processes are unaffected: by the time any activity runs, ``temporalio.activity`` is- necessarily in ``sys.modules``.- """- activity_module = sys.modules.get("temporalio.activity")- if activity_module is None:- return False- return cast("bool", activity_module.in_activity())--# DynamoDB PutItem failures come in two sibling botocore base classes (neither subclasses the other):# ClientError (service-side throttle / auth) and BotoCoreError (transport / credential / timeout, e.g.# EndpointConnectionError, ReadTimeoutError, NoCredentialsError). Both are imported together (present or@@ -183,18 +165,19 @@ class ReportingManager(ReportingProtocol):lookup_key (workflow-thread or direct mode), emit through the cachedper-context event log.- Activity path: an emission coming from inside a Temporal activity NEVER- takes the fast path, even when the activity runs co-located with the- workflow worker. The registered context there is the workflow's- in-sandbox BufferingEventLog, and activities do not re-execute on- replay — a cross-thread write into that buffer makes the workflow's- buffer content depend on whether activities actually ran, breaking- replay determinism (audit finding H1). Activities must behave- identically whether co-located or remote: per-process fallback.+ Isolated-execution path: an emission from inside an isolated sub-execution+ (reported by the boot orchestrator's ``is_in_isolated_execution`` probe — a+ Temporal activity, even when co-located with the workflow worker) NEVER takes+ the fast path. The registered context there is the workflow's in-sandbox+ BufferingEventLog, and an isolated sub-execution does not re-execute on replay —+ a cross-thread write into that buffer makes the workflow's buffer content depend+ on whether the sub-execution actually ran, breaking replay determinism (audit+ finding H1). Such emissions must behave identically whether co-located or remote:+ per-process fallback.Fallback: when context lookup misses (runner process — set_event_log- was never called here) or the emission comes from an activity, emit- through the per-process activity event log so the event still lands in+ was never called here) or the emission comes from an isolated sub-execution,+ emit through the per-process activity event log so the event still lands inthe same backend partition as the rest of the run. See_emit_usage_event_runner_fallback for details."""@@ -213,7 +196,7 @@ class ReportingManager(ReportingProtocol):if not trace_context.emit_usage_events:return- if not _is_in_temporal_activity():+ if not is_in_isolated_execution():context = self._event_log_contexts.get(trace_context.lookup_key)if context is not None:self._emit_via_registered_context(context, trace_context=trace_context, tokens_usage=tokens_usage)diff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.pyindex 7e55f9c2e..a4cd63827 100644--- a/pipelex/system/configuration/configs.py+++ b/pipelex/system/configuration/configs.py@@ -97,7 +97,6 @@ class ReportingConfig(ConfigModel):class TracingBackend(StrEnum):NDJSON = "ndjson"DYNAMODB = "dynamodb"- TEMPORAL_DYNAMODB = "temporal_dynamodb"class NdjsonTracingConfig(ConfigModel):@@ -109,17 +108,11 @@ class DynamoDBTracingConfig(ConfigModel):region: str-class TemporalDynamoDBTracingConfig(ConfigModel):- table_name: str- region: str--class TracingConfig(ConfigModel):is_enabled: boolbackend: TracingBackend = Field(strict=False)ndjson: NdjsonTracingConfig | None = Nonedynamodb: DynamoDBTracingConfig | None = None- temporal_dynamodb: TemporalDynamoDBTracingConfig | None = Noneclass ObserverConfig(ConfigModel):@@ -151,7 +144,7 @@ class PipelineExecutionConfig(ConfigModel):is_generate_usage: boolgraph_config: GraphConfig- # Bounded fan-out concurrency for PipeBatch (the resilience-without-Temporal pillar).+ # Bounded fan-out concurrency for PipeBatch (the in-process backpressure pillar, short of a durable execution backend).# An integer caps the number of branches executed at once; the literal "unbounded" disables the bound.max_concurrency: Annotated[int, Field(ge=1)] | Literal["unbounded"]diff --git a/pipelex/tracing/event_log_factory.py b/pipelex/tracing/event_log_factory.pyindex e8976735c..a713555b5 100644--- a/pipelex/tracing/event_log_factory.py+++ b/pipelex/tracing/event_log_factory.py@@ -29,12 +29,3 @@ def make_event_log(tracing_config: TracingConfig, *, writer_id: str = "primary")region=tracing_config.dynamodb.region,writer_id=writer_id,)- case TracingBackend.TEMPORAL_DYNAMODB:- if tracing_config.temporal_dynamodb is None:- msg = "temporal_dynamodb config is required when backend is 'temporal_dynamodb'"- raise PipelexConfigError(msg)- return DynamoDBEventLog(- table_name=tracing_config.temporal_dynamodb.table_name,- region=tracing_config.temporal_dynamodb.region,- writer_id=writer_id,- )diff --git a/pipelex/urls.py b/pipelex/urls.pyindex 8e1bf0321..e6e6f65d6 100644--- a/pipelex/urls.py+++ b/pipelex/urls.py@@ -3,6 +3,7 @@ class URLs:logo_black_on_transparent = "https://d2cinlfp2qnig1.cloudfront.net/logo/Pipelex-logo-bot-1119x352.png"homepage = "https://pipelex.com"app = "https://app.pipelex.com/"+ durable_execution = "https://go.pipelex.com/durable-execution"repository = "https://github.com/Pipelex/pipelex"documentation = "https://docs.pipelex.com/"changelog = "https://docs.pipelex.com/latest/changelog/"diff --git a/tests/unit/pipelex/cli/dev/test_check_keyword_only_cmd.py b/tests/unit/pipelex/cli/dev/test_check_keyword_only_cmd.pyindex 4eb05e6c9..4558c9082 100644--- a/tests/unit/pipelex/cli/dev/test_check_keyword_only_cmd.py+++ b/tests/unit/pipelex/cli/dev/test_check_keyword_only_cmd.py@@ -205,30 +205,6 @@ class TestCheckKeywordOnly:)assert violations == []- def test_carveout_temporal_activity_defn_in_stack_skipped(self) -> None:- """@activity.defn anywhere in the decorator stack is skipped (scan whole stack)."""- violations = _violate(- """- @activity.defn- @convert_pipelex_errors- def act_render(job, context, options):- ...- """- )- assert violations == []-- def test_carveout_workflow_run_skipped(self) -> None:- """@workflow.run is skipped."""- violations = _violate(- """- class MyWorkflow:- @workflow.run- async def run(self, request, options):- ...- """- )- assert violations == []-def test_carveout_pytest_fixture_skipped(self) -> None:"""@pytest.fixture is skipped."""violations = _violate(diff --git a/tests/unit/pipelex/plugins/test_import_light_boot.py b/tests/unit/pipelex/plugins/test_import_light_boot.pyindex 15f311834..ce4c44fe1 100644--- a/tests/unit/pipelex/plugins/test_import_light_boot.py+++ b/tests/unit/pipelex/plugins/test_import_light_boot.py@@ -1,9 +1,8 @@"""Import-light guard (codex C12): discovering and registering the built-in plugins mustnot import any backend SDK (whether an optional extra like anthropic/linkup or a heavy core-dep like openai/portkey_ai/pypdfium2 — each plugin must defer its import into make_worker) nor-``temporalio`` (which ships only with the external ``pipelex-temporal`` plugin — core discovery-must never pull the Temporal SDK). Enforced in a subprocess whose meta-path finder raises on those-SDKs, which is deterministic where an in-process sys.modules check is not.+dep like openai/portkey_ai/pypdfium2) — each plugin must defer its import into make_worker.+Enforced in a subprocess whose meta-path finder raises on those SDKs, which is deterministic+where an in-process sys.modules check is not."""import subprocess # noqa: S404@@ -28,7 +27,6 @@ _GUARD_SCRIPT = textwrap.dedent("openai","portkey_ai","pypdfium2",- "temporalio",# No web framework either: the F3 HTTP-error-mapper seam is framework-agnostic# (it carries the core ``ErrorReport`` type). A host runtime owns FastAPI/Starlette;# discovery/registration in core must never pull one in.@@ -53,11 +51,9 @@ _GUARD_SCRIPT = textwrap.dedent(# whose load() could legitimately import an optional SDK.discovery._external_entry_points = lambda: []- # Temporal ships externally (pipelex-temporal); core's built-ins claim no hub slots. temporalio- # stays in the BLOCKED set above: core discovery must never pull the Temporal SDK, and- # temporal.is_enabled=True must not change that (no core built-in reads it). The slot-claim- # import-light guarantee for the Temporal thunks is pinned in the pipelex-temporal suite.- config = SimpleNamespace(plugins=SimpleNamespace(disabled=[]), temporal=SimpleNamespace(is_enabled=True))+ # Core's built-ins claim no hub slots; building the registrar must register them all+ # import-light, pulling none of the BLOCKED backend SDKs into sys.modules.+ config = SimpleNamespace(plugins=SimpleNamespace(disabled=[]))registrar = discovery.build_registrar(config=config)assert registrar.inference_backends, "expected the built-in LLM backends to be registered"assert registrar.model_listers, "expected the built-in model listers to be registered import-light"diff --git a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.pyindex d48ee6b66..992f97ad8 100644--- a/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py+++ b/tests/unit/pipelex/reporting/test_emit_usage_event_gating.py@@ -176,3 +176,39 @@ class TestEmitUsageEventGating:assert isinstance(emitted_event, UsageReportEvent)assert emitted_event.pipeline_run_id == "run_fastpath"assert emitted_event.writer_id == "fastpath_writer"++ def test_isolated_execution_routes_off_the_registered_fast_path(self, tmp_path: Path, mocker: MockerFixture) -> None:+ """Isolated sub-execution (probe True): a usage emission must NOT take the registered fast path.++ Even with a registered context and costs on, an emission from inside an isolated sub-execution+ (a Temporal activity) routes to the per-process fallback instead of the workflow's registered+ buffer — the buffer must stay a pure function of inline execution for replay determinism (audit+ H1). Asserts the registered context is skipped AND the event lands in the per-process log.+ """+ ActivityEventLogCache.reset_for_tests()+ _enable_ndjson_tracing(mocker, tmp_path)+ mocker.patch("pipelex.reporting.reporting_manager.is_in_isolated_execution", return_value=True)++ manager = ReportingManager()+ manager.setup()+ trace_context = _make_trace_context("run_isolated", emit_usage_events=True, emit_graph_events=True)++ event_log_spy = mocker.MagicMock()+ event_log_spy.writer_id = "registered_writer"+ event_log_spy.next_sequence.return_value = 0+ manager.set_event_log(+ context_key=trace_context.lookup_key,+ event_log=event_log_spy,+ workflow_id="wf_run_isolated",+ pipeline_run_id="run_isolated",+ )++ manager.report_inference_job(_make_llm_job("run_isolated", trace_context=trace_context))++ # Registered fast path skipped; the event lands in the per-process fallback (writer act_*).+ event_log_spy.emit.assert_not_called()+ reader = NdjsonEventLog(traces_dir=str(tmp_path))+ usage_events = [evt for evt in reader.read_events("run_isolated") if isinstance(evt, UsageReportEvent)]+ assert len(usage_events) == 1+ assert usage_events[0].writer_id.startswith("act_")+ ActivityEventLogCache.reset_for_tests()diff --git a/tests/unit/pipelex/reporting/test_isolated_execution_probe.py b/tests/unit/pipelex/reporting/test_isolated_execution_probe.pynew file mode 100644index 000000000..2a9943009--- /dev/null+++ b/tests/unit/pipelex/reporting/test_isolated_execution_probe.py@@ -0,0 +1,25 @@+"""Core seam for routing usage emissions out of an isolated sub-execution.++Detecting whether the current call runs inside an isolated sub-execution (an orchestrator's+worker activity) is an orchestrator concern, not core's. A boot-orchestrator plugin claims+``HubSlot.ISOLATED_EXECUTION_PROBE`` with an ambient predicate; core consults it through+``PipelexHub.is_in_isolated_execution`` and names no orchestrator.+"""++from pipelex.hub import PipelexHub+++class TestIsolatedExecutionProbe:+ def test_default_is_never_isolated(self) -> None:+ """A fresh hub (no boot-orchestrator claim) reports the in-process default: never isolated."""+ hub = PipelexHub()+ assert hub.is_in_isolated_execution() is False++ def test_claimed_probe_is_honored_and_consulted_each_call(self) -> None:+ """A claimed probe replaces the default and is ambient — consulted on every call, not cached."""+ hub = PipelexHub()+ flag = {"isolated": True}+ hub.set_isolated_execution_probe(lambda: flag["isolated"])+ assert hub.is_in_isolated_execution() is True+ flag["isolated"] = False+ assert hub.is_in_isolated_execution() is Falsediff --git a/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py b/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.pydeleted file mode 100644index 13ad69322..000000000--- a/tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py+++ /dev/null@@ -1,54 +0,0 @@-"""Lazy-import contract of the Temporal-activity gate in ``reporting_manager``.--``_is_in_temporal_activity`` must detect the activity context WITHOUT importing temporalio:-a module-level import would put the entire temporalio extra (Rust bridge, protobuf, ~130ms)-on every pipelex boot's critical path wherever ``pipelex-temporal`` is installed — including-processes that never touch Temporal (CLI validate runs, direct-mode servers, the hosted-runner image). The gate sniffs ``sys.modules`` instead: a process that never imported-``temporalio.activity`` cannot be inside a Temporal activity, because the activity context is-set by temporalio's own machinery, which requires the module to be imported.-"""--import subprocess # noqa: S404-import sys--import pytest-from pytest_mock import MockerFixture--from pipelex.reporting.reporting_manager import _is_in_temporal_activity # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]---class TestTemporalActivityGateLazyImport:- def test_false_when_temporalio_activity_not_imported(self, monkeypatch: pytest.MonkeyPatch) -> None:- """No ``temporalio.activity`` in ``sys.modules`` → no activity context can exist."""- monkeypatch.delitem(sys.modules, "temporalio.activity", raising=False)- assert _is_in_temporal_activity() is False-- def test_delegates_to_in_activity_when_module_present(self, monkeypatch: pytest.MonkeyPatch, mocker: MockerFixture) -> None:- """With ``temporalio.activity`` imported, the gate delegates to its ``in_activity()``."""- fake_activity_module = mocker.MagicMock()- fake_activity_module.in_activity.return_value = True- monkeypatch.setitem(sys.modules, "temporalio.activity", fake_activity_module)- assert _is_in_temporal_activity() is True- fake_activity_module.in_activity.assert_called_once_with()-- fake_activity_module.in_activity.return_value = False- assert _is_in_temporal_activity() is False-- def test_importing_reporting_manager_does_not_import_temporalio(self) -> None:- """Boot-cost guard: the module import must not pull temporalio into ``sys.modules``.-- Subprocess check so the assertion sees a pristine interpreter (the test process- itself may have temporalio loaded via the integration fixtures).- """- probe = "import sys; import pipelex.reporting.reporting_manager; raise SystemExit(2 if 'temporalio' in sys.modules else 0)"- result = subprocess.run( # noqa: S603- [sys.executable, "-c", probe],- capture_output=True,- text=True,- check=False,- timeout=120,- )- assert result.returncode == 0, (- f"importing pipelex.reporting.reporting_manager pulled temporalio into sys.modules (exit {result.returncode}); stderr: {result.stderr}"- )diff --git a/tests/unit/pipelex/test_iter_cause_chain.py b/tests/unit/pipelex/test_iter_cause_chain.pyindex f94eb5457..a4d558be9 100644--- a/tests/unit/pipelex/test_iter_cause_chain.py+++ b/tests/unit/pipelex/test_iter_cause_chain.py@@ -1,12 +1,13 @@"""Unit tests for ``iter_cause_chain`` — the centralized, cycle-guarded ``__cause__`` walk.-This primitive is load-bearing on the error-reporting path: ``_carries_temporal_failure``,-``find_inference_error_category_in_chain``, ``_find_error_report_dict``, ``_message_from_exc``-and the is-self cycle check in ``_enrich_error_report_from_cause`` all delegate to it. A walk-that started at ``exc.__cause__`` instead of ``exc`` would silently break the predicates that-inspect the seed node, while the indirect coverage (``test_message_from_exc``) stayed green.-These tests pin the two contract properties that nothing else exercises: the seed is yielded-first, and every node is visited exactly once even on a cyclic chain.+This primitive is load-bearing on the error-reporting path:+``find_inference_error_category_in_chain``, the is-self cycle check in+``_enrich_error_report_from_cause``, ``PipeLLM._format_llm_error`` and the agent-CLI+source-location extraction all delegate to it. A walk that started at ``exc.__cause__``+instead of ``exc`` would silently break the predicates that inspect the seed node, while+that indirect coverage stayed green. These tests pin the two contract properties that+nothing else exercises: the seed is yielded first, and every node is visited exactly once+even on a cyclic chain."""from pipelex.base_exceptions import iter_cause_chaindiff --git a/wip/README.md b/wip/README.mddeleted file mode 100644index bb8d42612..000000000--- a/wip/README.md+++ /dev/null@@ -1,43 +0,0 @@-# `wip/` - Work in Progress--This folder holds active planning, design, and current-state docs for Pipelex runtime work. Finished plans, point-in-time PR recaps, and old audit artifacts are collected under [`history/`](history/).--Start with [`executive.md`](executive.md) for priorities. Then open the track README for the area you are picking up.--## Active Tracks--- [`plugins/`](plugins/) - real plugin architecture for orchestrators and inference backends. This is the major architecture track: entry-point discovery, registries, orchestrator dispatch, Temporal extraction, version gating, and cutover follow-ups.-- [`distributed-execution/`](distributed-execution/) - Temporal/distributed execution productionization. P0/P1 tracing and cost reporting shipped; open work is P0.2 follow-ons, local cross-package crates, remote dependency fetch, nondeterminism decisions, and MISTRAL_NATIVE downstream reconciliation.-- [`dry-run-refactor/`](dry-run-refactor/) - unified dry-run and validation execution. Part A shipped, Part B/unified dry run is implemented, and Part C needs release/API merge and hardening decisions.-- [`error-handling/`](error-handling/) - current-state reference and open tail for structured errors: webhook signing, metadata long tail, delivery-path request ids, and Temporal fail-safe review backlog.-- [`runtime-bridge/`](runtime-bridge/) - runtime bridge review/deferred follow-ups. Most PR review items are resolved; active value is in remaining deferred hardening and testing gaps.-- [`concurrency/`](concurrency/) - direct-mode batching/backpressure design: fan-out scheduling, rate limiting, and batch partial failure.-- [`graph/`](graph/) - graph/validation graph improvements: source-map enrichment and explicit graph target selection.-- [`observer-and-telemetry/`](observer-and-telemetry/) - telemetry/identifier visibility notes, especially hosted privacy around structural ids.-- [`recursivity/`](recursivity/) - only low-priority additive multi-file polish remains active; completed recursivity records are in history.--## Active Standalone Docs--- [`webhook-signing.md`](webhook-signing.md) - HMAC body-signing rollout for completion webhooks.-- [`init-yes-mode-handoff.md`](init-yes-mode-handoff.md) - non-interactive `pipelex init --yes` for downstream CI.-- [`structured-logging.md`](structured-logging.md) - structured log fields and contextvars.-- [`runtime-code-fixes.md`](runtime-code-fixes.md) - verified runtime fixes to pick up.-- [`structured-validation-errors-deferred-findings.md`](structured-validation-errors-deferred-findings.md) - contained follow-ups on structured validation errors.-- [`validate-parse-level-source-attribution.md`](validate-parse-level-source-attribution.md) - source attribution for malformed TOML in multi-file validation.-- [`blueprint-elaboration-directives.md`](blueprint-elaboration-directives.md) - future elaboration directive system.-- [`stuffs-as-nodespec.md`](stuffs-as-nodespec.md) - future graph model rework for stuff nodes.-- [`doctor-output-ansi-under-force-color.md`](doctor-output-ansi-under-force-color.md) - optional plain diagnostic output decision.-- [`pathlib-refactor.workflow.js`](pathlib-refactor.workflow.js) - runnable path-refactor workflow.--## History--[`history/`](history/) now holds shipped or reference-only material moved out of the active surface:--- CSV support-- keyword-only arguments-- signature-based validation-- cost-reporting registry work-- completed test-grind notes-- completed recursivity handoffs-- old `archive/` records-- point-in-time HTML recapsdiff --git a/wip/blueprint-elaboration-directives.md b/wip/blueprint-elaboration-directives.mddeleted file mode 100644index 5a7584746..000000000--- a/wip/blueprint-elaboration-directives.md+++ /dev/null@@ -1,26 +0,0 @@-# Blueprint Elaboration Directives — going further--A **blueprint elaboration directive** is a build-time instruction in a `.mthds` bundle that `BundleElaborator` expands into concrete pipes when the library loads — so the authoring shape and the executed shape differ. The first (and so far only) directive is `preliminary_text`, which synthesizes a draft-text pipe plus a `PipeStructure` pipe in memory.--This doc collects the work to take that system further: make the pipes the elaborator synthesizes first-class across observability, generalize the single-purpose elaborator into a directive framework, and add the tooling and guards a growing elaboration layer needs. None of it is started — pick items up as the need arises.--## Synthetic-pipe handling--The elaborator synthesizes pipes (e.g. `__draft_text`, `__structure`) that today surface as ordinary pipes everywhere. These items make them recognizable and friendly across the observability surfaces.--- **Marker on graph nodes + `pipelex list` exclusion.** When emitting `NodeSpec`, look up `bundle.elaboration_metadata` and set `tags["synthetic"] = "true"` + `tags["parent_pipe_code"] = parent`. Requires either a runtime-only field on `PipeAbstract` or a side-registry keyed by pipe code. Also hide synthetic pipes from `pipelex list`. This is the keystone — the next two build on it.-- **Friendly rendering across logs/traces/run-reporting.** Once the marker lands, render `<parent_pipe_code> [<step_role_label>]` everywhere `self.code` appears for a synthetic pipe. Touches graph_tracer, run_reporting, journal, distributed-tracing.-- **`mthds-ui` graph viewer integration.** Once the marker lands, decide in the UI repo whether to nest synthetic pipes under their parent or hide them.--## Generalizing the framework--- **Directive plugin registry.** `BundleElaborator` handles exactly one directive today (`_elaborate_preliminary_text`). Promote it to a plugin registry when a SECOND elaboration directive appears — not before, to avoid premature abstraction.--## Tooling & guards--- **`pipelex-dev elaborate-bundle <path>`.** A debugging CLI that prints the elaborated bundle form without running it, so directive expansion can be inspected directly.-- **Bundle-load benchmark in CI.** Microbenchmark library load time and alert on >5% regression. Protects future elaboration passes from silently slowing startup.--## Serialization--- **Persist `elaboration_metadata` across serialization boundaries.** Today `Field(exclude=True)` drops it on every `model_dump`. When a cross-boundary consumer materializes (graph viewer over a serialized bundle, Temporal payload, persistent cache), flip `exclude=False`, regenerate the schema, ship a plxt bump. The regression test at `test_elaboration_metadata.py::test_bundle_round_trip_drops_elaboration_metadata` flips first.diff --git a/wip/concurrency/README.md b/wip/concurrency/README.mddeleted file mode 100644index 73fc01114..000000000--- a/wip/concurrency/README.md+++ /dev/null@@ -1,105 +0,0 @@-# Concurrency & batching — design doc--Status: **design scoping**. The one quick win this doc identified — retry jitter — has since shipped (see below). What remains is three larger pieces of design work, split into their own docs — [`fan-out-scheduling.md`](fan-out-scheduling.md), [`rate-limiting.md`](rate-limiting.md), and [`batch-partial-failure.md`](batch-partial-failure.md).--## Context & scope--The resilience strategy is settled: **Temporal is the resilience system.** We deliberately decided *not* to build a cheap resilience system in direct mode — resilience is the error-handling track's job (see [`../error-handling/`](../error-handling/)).--The open problem is narrower: the direct-mode batching/asyncio system breaks too easily. The driving question: *what is a best-practice basic backpressure / rate-limiting system at the asyncio layer, so the system does not crash when asked to process e.g. 1000 documents?*--"Basic backpressure so it doesn't fall over" — not durable retry, not resilience. That stays Temporal's job.--## Read this first: direct mode vs Temporal mode--Every change in this doc behaves differently depending on the execution mode, so the mode split is the frame for everything that follows — not an afterthought.--There are two modes:--- **Direct mode** — everything runs in one Python process, on one asyncio event loop.-- **Temporal mode** — the same pipes run as Temporal workflows, distributed across N worker processes.--**Temporal slices execution at the activity boundary.** From the temporal code (`temporal_pipe_router.py`, `wf_pipe_run.py`, `act_llm_generate.py`):--- **Controllers run as workflow code.** `TemporalPipeRouter` is a drop-in `PipeRouterProtocol`. `PipeBatch._live_run_controller_pipe` — including `gather_bounded` over the branch factories — runs *inside the workflow*. Each branch's `get_pipe_router().run(...)` becomes a **child workflow**.-- **Inference runs as an activity.** The actual LLM/img-gen/extract call is `act_llm_gen_text` & co. — a Temporal **activity**: normal Python, running in **N separate worker processes**.--The consequences that drive every decision below:--- Code **above** the activity boundary (controllers, `gather_bounded`) is **workflow code** → must be deterministic: no wall-clock, no `time.sleep`, no real-time `asyncio.sleep`. It is *not* disabled under Temporal — it still runs, as workflow code.-- Code **inside** the activity is **normal Python** → wall-clock, sleep, jitter all fine — but it runs **per-process**, blind to the other workers.--| Layer | Direct mode | Temporal mode |-| --- | --- | --- |-| `PipeBatch` fan-out (`gather_bounded`) | In-process coroutines + memory | Workflow code; bounds concurrent child-workflow starts |-| Inference call (LLM/img-gen/extract) | In-process async call | Activity, one of N worker processes |-| Retry | `transport_retry` (`tenacity`) | `transport_retry` *and* Temporal activity `RetryPolicy` |-| Rate / resource limits | Nothing today | Temporal task-queue + worker knobs (see `rate-limiting.md`) |--## Why the current system breaks too easily--Three distinct weaknesses, and they compound under load. Code inspected: `pipelex/tools/misc/async_utils.py` (`gather_bounded`), `pipelex/pipe_controllers/batch/pipe_batch.py` (`PipeBatch`), `pipelex/cogt/inference/transport_retry.py`.--### 1. `gather_bounded` chunks — head-of-line blocking--`gather_bounded` runs items in chunks of `max_concurrency` and waits for the *whole chunk* to drain before starting the next. If one item in a chunk of 8 takes 60s, the other 7 slots sit idle the whole time. Throughput collapses to "slowest item per chunk."--### 2. The bound is per-`PipeBatch`, not process-global--`max_concurrency = 8` (`pipelex.toml`) is applied independently by each `PipeBatch`. Nested batches, or sibling pipes running concurrently, multiply: two batches → 16 in flight, a nested batch → 64. Nothing sees the total.--### 3. Concurrency ≠ rate — and rate is what providers enforce--Providers rate-limit on **requests-per-minute (RPM)** and **tokens-per-minute (TPM)**, not on concurrency. 8 concurrent calls each finishing in ~200ms = ~40 RPS = ~2400 RPM, which blows a 500-RPM tier instantly. The result is a 429 storm; `transport_retry` then re-fires that storm. The backoff used to be jitter-free, so retries re-synchronized into a thundering herd — that part is now fixed (full jitter shipped, see below), but the underlying RPM/TPM overrun that triggers the storm remains.--The retry-jitter gap was the one true quick win and has since shipped (below). Weakness 1 looked like one but is not — fixing it correctly carries a design decision, so it has its own doc, [`fan-out-scheduling.md`](fan-out-scheduling.md). Weaknesses 2 + 3 are the hard part — a real rate limiter — covered in [`rate-limiting.md`](rate-limiting.md).--## Where `max_concurrency` lives today--`max_concurrency` is a field on `PipelineExecutionConfig` (`pipelex/system/configuration/configs.py`), read at `pipe_batch.py` via `get_config().pipelex.pipeline_execution_config.max_concurrency`. It is an `int >= 1` or the literal `"unbounded"`, set to `8` in `pipelex.toml`.--So today it is **one global config value**, not a per-`PipeBatch` setting — `BatchParams` carries only the input list/item names, no concurrency field. Every `PipeBatch` reads the same global number and applies it **independently** to its own fan-out.--Scope across processes:--- **Direct mode** — the submitter is the only process, so the global config value is global to everything.-- **Temporal mode** — `PipeBatch` runs as workflow code on a worker, and `get_config()` reads *that worker's* config. So the value comes from the worker, not the submitter — incidentally, because config is per-process. It is **not** the Temporal worker-runtime knob (`WorkerRuntimeProfile.max_concurrent_activities` & co. in `config_temporal.py` are separate); `max_concurrency` is a pure Pipelex setting the controller reads wherever it runs.--This is the mechanism behind weakness 2: a global *number* applied per-batch is not a global *bound*. Each `PipeBatch` independently permits `max_concurrency` branches, so sibling batches sum and nested batches multiply (`max_concurrency²`).--**Design option — make it per-`PipeBatch`.** An optional `max_concurrency` field could be added to `BatchParams` (the per-`PipeBatch` object parsed from `.mthds`), falling back to the global config. This is a *fan-out shape* knob — useful, but it does not fix weakness 2, since shape knobs still multiply under nesting. It carries one Temporal advantage worth noting: a value on `BatchParams` is part of the fixed pipe blueprint / workflow input, so it replays deterministically — strictly safer than `PipeBatch` reading global config inside workflow code, which `wf_pipe_run.py` already warns against for "config-derived" values that "change across deploys". Tracked as an open question — see [`rate-limiting.md`](rate-limiting.md).--## Shipped — retry jitter--`transport_retry.py` now uses full-jitter `wait_random_exponential` (`transport_retry.py:46`); it was previously jitter-free `wait_exponential`, so under a 429 storm every retry re-fired in lockstep. With full jitter each wait is drawn from `uniform(0, exponential_bound)`, breaking the lockstep.--**Direct vs Temporal:** `transport_retry` runs inside the activity in both modes — jitter is fine there (non-determinism allowed in activities). Pre-existing nuance, unaffected by this change: under Temporal there are already two retry layers — `tenacity` inside the activity and Temporal's activity `RetryPolicy` outside it. Whether the activity should fail faster and let `RetryPolicy` own retries is a separate, existing design question.--## Larger design work — separate docs--Three pieces of design work are too large to spec inline. Each has its own doc. They are related — all three touch `gather_bounded` or the inference path — but can be taken in separate sessions, with one coupling noted below.--### Fan-out scheduling — [`fan-out-scheduling.md`](fan-out-scheduling.md)--Replacing `gather_bounded`'s chunking with a sliding-window semaphore (weakness 1) is a real throughput win — idle slots disappear and a slow branch stops blocking its neighbors. But the obvious one-liner silently regresses fail-fast: it would run all branches even when branch 3 fails fatally, burning ~997 needless LLM calls in a 1000-document batch. Doing it right needs sibling cancellation, which **couples this doc to `batch-partial-failure.md`** (the cancellation behavior depends on the fail-fast vs collect-partial policy) and raises a Python-version question (`asyncio.TaskGroup` is 3.11+; the project still supports 3.10).--### Rate limiting — [`rate-limiting.md`](rate-limiting.md)--Weaknesses 2 + 3 — a per-`PipeBatch` (not global) bound, and the absence of any RPM/TPM limiter — are the real cause of the crash at scale. The fix is genuinely mode-dependent: direct mode needs an in-process limiter, Temporal mode needs server-side task-queue limits. It is **not one system that works in both modes**. The doc also records the finding that the Temporal-mode machinery already exists.--### Batch partial failure — [`batch-partial-failure.md`](batch-partial-failure.md)--`gather_bounded` raises on the first exception, so one malformed document discards every good result in the batch. Collecting partial results sounds like a small change, but it forces a decision on the *type* of a batch output — and the options range from a sparse list to an envelope concept, the latter being an MTHDS language-semantics change. The doc lays out the blast radius and the open questions.--## Layering summary--| Layer | Responsibility |-| --- | --- |-| `PipeBatch` | Fan-out *shape* — how work is split (scheduling redesign in `fan-out-scheduling.md`) |-| Inference rate/concurrency gate | The actual backpressure — see `rate-limiting.md` |-| `transport_retry` | Residual 429/5xx — full jitter (`wait_random_exponential`) shipped |-| Temporal | Durability — out of scope here |--## Suggested next step--The retry-jitter quick win has shipped. Take up the three design docs as design sessions: [`fan-out-scheduling.md`](fan-out-scheduling.md) and [`batch-partial-failure.md`](batch-partial-failure.md) are coupled through the failure-handling policy and are best taken together (or in that order); [`rate-limiting.md`](rate-limiting.md) is independent of both.diff --git a/wip/concurrency/batch-partial-failure.md b/wip/concurrency/batch-partial-failure.mddeleted file mode 100644index b52c92518..000000000--- a/wip/concurrency/batch-partial-failure.md+++ /dev/null@@ -1,75 +0,0 @@-# Batch partial failure — design notes--Status: **design scoping**, not yet a plan. One of the design tracks of the concurrency & batching work — see [`README.md`](README.md) for the overview. Read the README's "direct mode vs Temporal mode" section first — this doc assumes that frame.--## Why this is its own doc--`gather_bounded` raises on the first exception (first-error-aborts): one malformed document in a 1000-document batch discards every good result. Collecting partial results instead *sounds* like a quick win — and the `gather_bounded` change really is trivial (stop raising the first exception, return results-or-exceptions).--But that part is worthless on its own. The value is entirely in what `PipeBatch` does with a partial result, and that is a cross-cutting design problem: it touches the type system, the MTHDS language contract, the reporting layer, the graph tracer, and the Temporal boundary.--This doc is **coupled to [`fan-out-scheduling.md`](fan-out-scheduling.md)**: the failure *policy* decided here (fail-fast vs collect-partial) determines whether that doc's semaphore fan-out needs sibling cancellation. The two should be decided together, or this one first.--## The constraining fact: a failed branch leaves nothing behind--Today, a failed branch produces **no `PipeOutput` at all**. The exception propagates out of `get_pipe_router().run(...)` inside `PipeBatch._run_branch`, through the factory, and would be captured by `gather_bounded` as an exception object. The branch's deep-copied `branch_memory` is discarded — the parent only ever harvests `pipe_output.main_stuff`.--So "keep the failed branch" is not free: `_run_branch` would have to *catch* and *return* a failure-carrying object instead of letting the exception raise.--## The three questions that define the design--### 1. Do we keep the failed branch's working memory?--Recommendation: **no.** Each branch has its own deep-copied `WorkingMemory`. Keeping hundreds of failed branches' working memories alive for a large batch is a memory blowup that serves no one. Keep only the **`ErrorReport`** (the error-handling branch already defines that type). The branch's memory dies with the branch.--### 2. Do we need an envelope over output stuff?--This is the central fork, and it is a *language-semantics* decision, not an implementation detail.--`PipeBatch`'s output is a `ListContent` whose items are `StuffContent` of the branch's output **concept**. Every item must be a valid instance of that concept. A failed branch has no valid instance. Three options:--- **Envelope.** Output becomes `ListContent[BatchItemOutcome]`, where `outcome = success(stuff) | failure(error_report)`. Clean and explicit — but it **changes the output concept type**. Every downstream pipe consuming a batch output now consumes envelopes; every `.mthds` file using batch is affected; the concept system and `validate_output_with_library` must accommodate it. This is an MTHDS language change.-- **Sparse list.** Drop failed items; the output list is shorter than the input list. This breaks **positional alignment** between input and output — and the graph tracer relies on it: `register_batch_item_extraction` / `register_batch_aggregation` key by `item_index`. Downstream code that zips input↔output breaks silently.-- **Side channel.** The output `ListContent` stays the plain concept (successes only), and failures go into a separate structure on `PipeOutput` / `JobMetadata`. Keeps the type contract for the happy items, but reintroduces the alignment problem and adds a parallel reporting path.--The envelope is the "solid over quick" choice (per the project dev principles), but it is genuinely a language change. It would likely be **opt-in via a `BatchParams` flag**, so default batches keep the plain `ListContent` and fail-fast, and only batches that explicitly ask for partial-tolerance produce envelopes.--### 3. How do we report?--Tie into the error-handling branch's `ErrorReport`. A batch outcome is: counts (succeeded / failed) plus a per-failed-index `ErrorReport`. That touches:--- `agent_output.py` needs a "partial batch" renderer;-- the CLI needs an exit-code policy for partial success (is 847/1000 a success or a failure?);-- under Temporal each branch is a child workflow, so failures arrive as `ChildWorkflowError` — which already carries `ErrorReport` per `wip/error-handling/track-temporal-integration.md` — and the parent batch workflow has to aggregate them.--## Blast radius--- `gather_bounded` — trivial change (return results-or-exceptions).-- `PipeBatch._live_run_controller_pipe` harvest loop — currently assumes every entry is a `PipeOutput`.-- `PipeBatch._run_branch` — must catch and return a failure object instead of raising.-- The output concept type — envelope vs sparse vs side channel.-- Concept validation (`validate_output_with_library`).-- **Every downstream batch consumer, and every `.mthds` file using batch** — if the envelope option is taken.-- Graph tracer — `register_batch_aggregation` keys per `item_index`; a failed index has no `item_stuff_code`.-- Failed-branch working-memory disposal.-- `ErrorReport` aggregation across branches.-- `agent_output.py` rendering + CLI exit code.-- Temporal — `ChildWorkflowError` aggregation in the parent batch workflow.-- A new `BatchParams` opt-in flag.--## Direct vs Temporal--The decision logic is the same in both modes — the change runs as workflow code under Temporal, is deterministic, and is compatible. It is *more* valuable under Temporal: each batch branch is a durably-executed child workflow, so first-error-aborts there discards durably-completed work. The Temporal-mode shape of the change is aggregating per-branch `ChildWorkflowError`s instead of in-process exceptions.--## Open questions for the design session--- Envelope, sparse list, or side channel? (Recommendation leans envelope, opt-in via `BatchParams`.)-- Default behavior: fail-fast (status quo) or collect-partial? If opt-in, fail-fast stays the default.-- If envelope: what is `BatchItemOutcome` exactly, and how does the concept system express "list of outcomes of concept X"?-- Exit-code / success semantics for a partially-failed batch at the CLI and agent-output layer.-- Does a downstream pipe ever need to *consume* a partial batch (branch on per-item success), or is partial failure always terminal reporting?-- Interaction with [`fan-out-scheduling.md`](fan-out-scheduling.md): the failure policy chosen here drives whether the semaphore fan-out cancels siblings on first error (fail-fast) or runs every branch (collect-partial). This design also decides who interprets the results-or-exceptions list `gather_bounded` would return, and how.--## Suggested next step--Decide question 2 (envelope vs sparse vs side channel) first — every other question and the entire blast radius depend on it. Prototype the envelope option against one real downstream batch consumer to measure the true cost of the language change before committing.diff --git a/wip/concurrency/fan-out-scheduling.md b/wip/concurrency/fan-out-scheduling.mddeleted file mode 100644index cfa4b27e2..000000000--- a/wip/concurrency/fan-out-scheduling.md+++ /dev/null@@ -1,99 +0,0 @@-# Fan-out scheduling — design notes--Status: **design scoping**, not yet a plan. One of the design tracks of the concurrency & batching work — see [`README.md`](README.md) for the overview. Read the README's "direct mode vs Temporal mode" section first — this doc assumes that frame.--## Why this is its own doc--Replacing `gather_bounded`'s chunking with a sliding-window semaphore *looks* like a one-liner. The scheduling improvement is real and worth having, but the obvious implementation **silently regresses fail-fast behavior**, and fixing that properly:--- couples this change to the partial-failure policy in [`batch-partial-failure.md`](batch-partial-failure.md);-- raises a Python-version decision (`asyncio.TaskGroup` is 3.11+).--So it carries a design decision, not just an edit.--## The mechanism: chunk barrier vs sliding window--**Today — chunking.** `gather_bounded` splits the factories into chunks of `max_concurrency` and, for each chunk, does `await asyncio.gather(...)` before starting the next. The defining property: **a chunk does not advance until every item in it finishes.** With `max_concurrency=8` and 100 items, that is 13 sequential barriers.--**Proposed — semaphore.** All factories go into one `asyncio.gather`; each acquires a semaphore before running. At any instant ≤ `max_concurrency` run, but the moment *any one* finishes it releases the semaphore and the *next pending* factory starts immediately. No chunk boundaries — a continuous sliding window.--```python-sem = asyncio.Semaphore(max_concurrency)-async def _run(factory):- async with sem: # acquired before factory() → materialization still bounded- return await factory()-```--The semaphore is acquired *before* calling the factory, so the property the comment in `async_utils.py` protects still holds: at most `max_concurrency` deep-copied working memories materialized at once, per `PipeBatch`.--## What it wins--### Win 1 — throughput: idle slots disappear--Under chunking, a chunk's wall-clock cost is its **slowest** item, not its average. As fast items in a chunk finish, their slots sit idle waiting for the chunk's straggler. That idle time is pure waste.--Everyday case (not pathological): 100 items, durations spread uniformly 1–10s, `max_concurrency=8`.--- **Chunking:** each chunk of 8 takes `max` of 8 samples ≈ 9s. 13 chunks ≈ **~115s**.-- **Semaphore:** total work ÷ slots = `100 × 5.5s ÷ 8` ≈ **~70s**.--Roughly a **40% wall-clock reduction**, purely from never leaving a slot idle while work is pending. The wider the spread of item durations, the bigger the gap.--### Win 2 — a slow branch stops poisoning its neighbors--The qualitative win. Under chunking, one 60s branch does two kinds of damage: it pins the other 7 slots in its chunk idle for ~59s, *and* it delays the start of every later chunk. A single straggler taxes ~7 innocent branches plus everything queued behind it.--Under the semaphore, that 60s branch occupies exactly **one** slot for 60s. The other 7 keep cycling through the queue at full speed. The slow branch costs only itself.--## What it does not win--- The bound is unchanged: still `max_concurrency`, still per-`PipeBatch`. Memory footprint identical (≤ `max_concurrency` deep copies per batch).-- No global bound, no rate limiting — weaknesses 2 and 3 are untouched (that is [`rate-limiting.md`](rate-limiting.md)).--It is purely a *scheduling* improvement: same resources, no idle gaps.--## The catch: the naive change regresses fail-fast--Today's `gather_bounded` aborts early. From its docstring: it drains the current chunk, raises the first error by input index, and **starts no later chunk**. A fatal error in item 3 wastes at most one chunk's worth of items.--The naive semaphore version — `asyncio.gather(*(_run(f) for f in factories), return_exceptions=True)` — schedules *all* factories. With `return_exceptions=True`, gather does not stop on the first failure: it runs **every** branch, then raises. For a 1000-document batch where item 3 fails fatally, that is ~997 needless branch runs — and each branch run is real LLM/inference spend.--So the semaphore change done *correctly* needs **sibling cancellation**: on the first exception, cancel the not-yet-started factories. That cancellation is cheap and exactly right — a factory still waiting on the semaphore has not done its deep copy or fired its API call, so cancelling it costs nothing and saves everything.--## Coupling to `batch-partial-failure.md`--Whether cancellation is even wanted depends on the failure policy decided in [`batch-partial-failure.md`](batch-partial-failure.md):--- **Fail-fast (today's default)** → the semaphore version *must* add sibling cancellation, or it is a regression.-- **Collect-partial** → running every branch is the *intended* behavior; `return_exceptions=True` with no cancellation is correct, and the naive version is fine.--So the exact shape of this change is not independent. The two docs should be decided together, or this one should follow the partial-failure decision. A reasonable design keeps both: cancel-on-first-error when the policy is fail-fast, run-all when it is collect-partial — the semaphore wrapper takes the policy as a parameter.--## Decision point: `asyncio.TaskGroup` and Python 3.10--`asyncio.TaskGroup` does sibling cancellation natively — first exception cancels the rest — which is exactly the fail-fast behavior above. But `TaskGroup` is **Python 3.11+**, and it raises `ExceptionGroup`, also 3.11+. The project currently targets 3.10+ (`requires-python = ">=3.10,<3.15"`), and the coding standards explicitly say to avoid `ExceptionGroup` / `except*` unless using the `exceptiongroup` backport.--Three ways forward:--1. **Hand-roll cancellation.** Manual "on first exception, cancel pending tasks" logic works on 3.10 with no new dependency. More code than `TaskGroup`, but not much, and it keeps the door open.-2. **Drop Python 3.10.** Bump `requires-python` to `>=3.11`, use `TaskGroup` natively. Cleanest code, but a breaking support change with its own blast radius beyond this doc — needs a separate, deliberate call.-3. **Version-gated / backport.** Use `TaskGroup` on 3.11+, fall back on 3.10 (the `taskgroup` / `exceptiongroup` backports, or option 1's hand-rolled path). Most complexity for the least benefit.--This is a genuine project-level decision, not something to settle inside `gather_bounded`. Recommendation for scoping: option 1 unblocks this change today without forcing the 3.10 question; option 2 is worth raising on its own merits but should not be driven solely by this doc.--## Direct vs Temporal--`gather_bounded` runs as workflow code under Temporal — it is not disabled. `asyncio.Semaphore` involves no wall-clock; acquisition order follows task scheduling, which Temporal replays deterministically, so the semaphore is workflow-safe (the current chunking is too — worth a sanity check against the workflow sandbox). Under Temporal the bound's meaning shifts from "in-flight coroutines" to "concurrent child-workflow starts", still a useful bound.--Cancellation is also fine under Temporal — cancelling a not-yet-started child-workflow dispatch is deterministic. `TaskGroup`, if chosen, would need the same sandbox sanity check.--## Open questions for the design session--- Decide the failure policy in `batch-partial-failure.md` first, or jointly — the cancellation behavior here depends on it.-- `asyncio.TaskGroup` (drop 3.10) vs hand-rolled cancellation (keep 3.10) vs version-gated. See the decision point above.-- Should the semaphore wrapper take the failure policy as an explicit parameter (cancel-on-error vs run-all), so one primitive serves both modes?-- Does `gather_bounded` keep its current "raise first error by input index" ordering guarantee under the semaphore, where completion order is no longer chunk-aligned?--## Suggested next step--Settle the failure policy with `batch-partial-failure.md`, then implement the semaphore wrapper parameterized by that policy. Treat the Python-3.10 question as a separate, explicit decision rather than letting this change force it.diff --git a/wip/concurrency/rate-limiting.md b/wip/concurrency/rate-limiting.mddeleted file mode 100644index 937ba3edc..000000000--- a/wip/concurrency/rate-limiting.md+++ /dev/null@@ -1,72 +0,0 @@-# Rate limiting the inference layer — design notes--Status: **design scoping**, not yet a plan. One of the design tracks of the concurrency & batching work — see [`README.md`](README.md) for the overview. Read the README's "direct mode vs Temporal mode" section first — this doc assumes that frame.--## Why this is its own doc--The retry-jitter quick win in the README is small, isolated, and mode-agnostic. Rate limiting is none of those things:--- it is the change that actually prevents the crash at scale;-- it is genuinely **two different mechanisms** in the two execution modes;-- one of those mechanisms **already exists** and the other has to be built.--## The problem: concurrency ≠ rate--The current system has only one knob — `PipeBatch.max_concurrency` — and it bounds *concurrency*. Providers do not rate-limit on concurrency. They enforce **requests-per-minute (RPM)** and **tokens-per-minute (TPM)**.--Concurrency 8, with calls finishing in ~200ms, is ~40 RPS ≈ 2400 RPM — enough to blow a 500-RPM tier instantly. Concurrency caps cannot express "no more than 500 requests per minute"; only a rate limiter can.--Two further gaps:--- **The bound is per-`PipeBatch`, not global.** Nested or sibling batches multiply the real in-flight count. The protective bound must be a single shared gate, not a per-controller setting.-- **The scarce resource is the provider account.** The gate must be keyed by `(provider, model)` — or per account — because that is what the provider's limit applies to, shared across every pipe and batch.--## Direct vs Temporal: two mechanisms, not one--This is the crux. A rate limiter is wall-clock-based (token bucket, `asyncio.sleep`). That dictates where it can live.--| | Direct mode | Temporal mode |-| --- | --- | --- |-| Where the inference call runs | In-process async call | Inside an activity, one of N worker processes |-| Can a wall-clock token bucket live there? | Yes — normal async code | Yes inside the *activity* (not workflow code) — but only sees its own process |-| What a single in-process limiter achieves | A correct, global cap | Only a **per-worker** cap — N workers × limit = N× the intended rate |-| Correct mechanism | In-process token bucket + concurrency semaphore | Temporal **task-queue** rate limit (server-enforced, cluster-wide) + per-worker knobs |--The trap to avoid: building one in-memory limiter and assuming it protects both modes. Under Temporal it cannot — an in-memory bucket is blind to the other worker processes, so five workers each holding a 500-RPM bucket hammer the provider at 2500 RPM.--## Finding: the Temporal-mode machinery already exists--Inspected `pipelex/temporal/config_temporal.py`. The Temporal-side rate-limit knobs are already built and configurable — Temomral mode is a *configuration* story, not a build:--- **`QueueOptions.max_task_queue_activities_per_second`** (`config_temporal.py:281`) — the **cluster-wide** task-queue rate limit. Its own comment: *"Cluster-wide queue rate limit, conveyed to the Temporal server by every worker on this queue."* Server-enforced across all workers — exactly the RPM cap.-- **`WorkerRuntimeProfile.max_concurrent_activities`** (`config_temporal.py:340`) — per-worker concurrency ceiling.-- **`WorkerRuntimeProfile.max_activities_per_second`** (`config_temporal.py:344`) — per-worker activity rate limit.-- **`ActivityRouteConfig`** (`config_temporal.py:295`) with `by_handle` — routes `llm_handle` / `img_gen_handle` / `extract_handle` to dedicated task queues. So **per-provider keying already exists on the Temporal side**: give each provider its own queue, set that queue's `max_task_queue_activities_per_second`.--Consequence: the Temporal half needs only sane default values and documentation, not new code. **The actual build is purely the direct-mode gate.**--## Proposed: the direct-mode inference gate--A single process-global gate, sitting at the inference call site, that every provider request funnels through regardless of which `PipeBatch` spawned it. Keyed by `(provider, model)`. Two components:--- a **concurrency semaphore** — global ceiling on in-flight API calls;-- a **token-bucket rate limiter** on RPM (TPM too, if a pre-call token estimate is available; RPM is the 80%).--Implementation options: `aiolimiter.AsyncLimiter` is the standard asyncio token bucket (tiny, MIT), or hand-roll ~30 lines.--`PipeBatch.max_concurrency` can stay as a fan-out *shape* knob, but it stops being the protective bound — the gate is. Then "1 batch of 1000" and "10 batches of 100" hit the same gate and behave identically.--**Design principle: one mental model, two enforcement mechanisms.** The direct-mode gate's config should mirror the Temporal knobs conceptually — per-provider RPM + concurrency — so an operator configures one thing and it is enforced in-process in direct mode and by the Temporal server in Temporal mode. Ideally the config is sourced once (e.g. alongside the provider/backend definitions) and consumed by both paths.--## Open questions for the design session--- Where exactly is the single inference call site that every provider request funnels through? Is it the same code in direct mode and inside the Temporal activities? That is where the direct-mode gate must sit.-- How is the rate-limit config keyed and sourced — per `(provider, model)`, per account? Does it belong in the existing backend TOML configs alongside the provider definitions, so both the direct gate and the Temporal queue config can read one source?-- TPM limiting needs a pre-call token estimate. Is one already available, or is RPM-only acceptable for v1?-- Add `aiolimiter` as a dependency, or hand-roll the token bucket?-- Does `max_concurrency` survive as a fan-out shape knob once the global gate exists, or fold entirely into it? And if it survives, should it become per-`PipeBatch` — an optional field on `BatchParams`, falling back to the global config — rather than today's single global value? (Per-`PipeBatch` is also more Temporal-determinism-safe: a value on `BatchParams` is fixed workflow input, vs. `PipeBatch` reading global config inside workflow code. See the README's "Where `max_concurrency` lives today".)-- How does the gate interact with `transport_retry`? A 429 that slips through should feed back into the limiter (back off the bucket), not just retry blindly.--## Suggested next step--Locate the single inference call site that every provider request funnels through (the same one the Temporal activities wrap), then design the gate around it: where the shared limiter object lives, how it is keyed, and where its config is sourced so direct mode and Temporal mode read one model.diff --git a/wip/distributed-execution/README.md b/wip/distributed-execution/README.mddeleted file mode 100644index 2981879c2..000000000--- a/wip/distributed-execution/README.md+++ /dev/null@@ -1,125 +0,0 @@-# Distributed Execution — Plan & Track Map--The big direction: run **activities on standalone Worker pools** — separate processes from the workflow Worker pool — so workflow orchestration and inference activity execution scale independently.--This folder is the **distributed-execution track**: running MTHDS methods as Temporal workflows across separate worker processes. This README is the priority-ordered plan; the other docs are the per-topic designs and references it points to.--## Docs in this track--- [tracing-cost-reporting.md](tracing-cost-reporting.md) — as-built reference for tracing & cost reporting across workers, plus its deferred items (P0/P1 context).-- [local-cross-package-deps.md](local-cross-package-deps.md) — ship dependency blueprints inside the crate (P2).-- [remote-deps-from-github.md](remote-deps-from-github.md) — fetch dependencies from remote addresses (P3).-- [crate-first-architecture.md](crate-first-architecture.md) — the crate-first vision behind P2/P3, with deferred capabilities (stripping, fingerprint validation, cross-worker cache).-- [temporal-ids-and-naming.md](temporal-ids-and-naming.md) — implemented design for Temporal IDs, naming, and observability surfaces.-- [temporal-exception-model-revamp.md](temporal-exception-model-revamp.md) — deferred proposal to reparent workflow exceptions onto `ApplicationError`.-- [schema-reconstruction-hardening.md](schema-reconstruction-hardening.md) — deferred hardening of cross-process Pydantic-schema reconstruction.-- [unit-testing-worker-sandbox-validation.md](unit-testing-worker-sandbox-validation.md) — deferred bug: a sandboxed worker can't boot under `--is-unit-testing` because the registered test workflows fail temporalio sandbox validation.-- [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md) — **OPEN handoff:** how the `_bridge` dev-merge changed the runtime-bridge surface (param renames, deleted graph-assembly primitive, `graph_context→trace_context`, #967/#968) and what the downstream `pipelex-mistralai-workflows` + `_workflows` repos need reconciled. Diagnosis DONE — it is the authoritative breakage inventory; now also carries item 7, the PR #987 reporting-path behavioral break (activity-side usage emissions bypass the plugin's registered buffer).-- [workflow-nondeterminism-audit.md](workflow-nondeterminism-audit.md) — **diagnosis, HIGHs fixed:** verified audit of worker-local inputs leaking into the workflow command stream (post-PR-#984). Both HIGH findings (flush activity gated on an activity-fed buffer; library/fingerprint leak on eviction) and M1 are now FIXED with regression guards; the open remainder is config-derived dispatch options and inline randomness drifting command arguments. Carries the suggested fix order.-- [nondeterminism-fix-review-follow-ups.md](nondeterminism-fix-review-follow-ups.md) — **BURNED DOWN:** verified findings from the code reviews of the H1/M1 and H2 fixes, now all resolved (per-item statuses recorded inline). The priority item — keying per-run worker-local state by `run_id` instead of `workflow_id` — is FIXED with regression guards, along with the eviction-ordering test, the flush-skip in costs-only LIVE, the `finally` simplification, helper consolidation, and the smaller cleanups.-- [nondeterminism-pre-merge-fixes.md](nondeterminism-pre-merge-fixes.md) — **PRE-MERGE, ACTIONABLE:** the two findings from PR #987's xhigh review to fix before merge — `PipelineManager` 500s resubmission of the same `pipeline_run_id` process-permanently (and its collision raise is load-bearing for the direct-mode tracer keyspace — read the coupling warning before touching it), and the module-level temporalio import in `reporting_manager.py` that puts ~130ms on every pipelex boot. Cold-start ready: evidence, fix shapes, red-first tests, acceptance.-- [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md) — **DECISIONS NEEDED:** judgment calls deferred from the burn-down and from PR #987's pre-landing review army. Headliners: the graph-event path has no H1-style guard against in-activity tracer mutation (ties into T3), tracing-disabled workers silently drop submitter-requested events, a non-PipelexError in the unguarded tracing-setup block retries forever, and the run-id-keying residual leak/zombie-thread risks. Options + recommendations inside.-- [validate-sweep-temporal-leak-repro.md](validate-sweep-temporal-leak-repro.md) — **bug brief + e2e repro (DONE):** the `/validate` dry-run sweep leaked nested controller sub-pipes to Temporal (HTTP 422 on a standalone `PipeBatch`). Fixed (commit `3377babb`); all three guard layers now in place — in-process sentinel test, Mode-1 pytest (`test_validate_sweep_stays_in_process.py`), and the deployment-faithful Mode-2 scenario (Tier 2c in the `temporal-e2e-validate` skill, using the `pipelex validate bundle --temporal` flag).-- [temporal-e2e-validate-skill.md](temporal-e2e-validate-skill.md) — validation status of the `/temporal-e2e-validate` skill and the Step 9 (queue-options) skill bugs patched in its `queue-options-battery.md`. Full dry + live run all green; open follow-ups are the vestigial `act_jinja2_gen_text` removal and a stale Mode 1 "Known xfails" note.-- [mistral-workflows/](mistral-workflows/) — sub-track for landing the `pipelex-mistralai-workflows` plugin (the MISTRAL_NATIVE backend) onto current `dev`. Its [README](mistral-workflows/README.md) is the hub; verdict is split — land the bridge, hold the `mistralai` 1.x→2.x bump (blocked on instructor PR #2298). Carries the two ready-to-paste keep-alive prompts.-------## Priority order--| # | Item | Detail |-|---|---|---|-| **P0 ✓** | Tracing & cost reporting across separate-process workers | shipped — [tracing-cost-reporting.md](tracing-cost-reporting.md) |-| **P0.1 ✓** | Dry-run through activity dispatch (testing affordance) | shipped as `--mock-inference` — below |-| **P0.2** | Deferred follow-ons from P0 | below |-| **P1 ✓** | Cross-worker cost report assembly | shipped (Option B — usage rides on `PipeOutput`) — below |-| **P2** | Local cross-package dependencies in the crate | [local-cross-package-deps.md](local-cross-package-deps.md) |-| **P3** | Remote dependencies from GitHub | needs P2 — [remote-deps-from-github.md](remote-deps-from-github.md) |--P0/P0.1/P0.2/P1 formed one chain (P0 unlocked the rest); **P0, P0.1, and P1 are now shipped** (see [the registry tracker](../history/registry/README.md) for the as-built sequencing). P0.2 is still open (independent follow-ons). P2/P3 are an independent chain (P2 blocks P3). [tracing-cost-reporting.md](tracing-cost-reporting.md) carries the as-built design; T1 and T2 are fixed, T3 (request-scoped tracing state) remains open.--Sibling track (separate branch, not in this plan): the error-handling work — see [`error-handling/README.md`](../error-handling/README.md).-------## P0 — Tracing & cost reporting across separate-process workers — DONE--Shipped. The as-built design, what works today, and the open T2/T3 gaps are in [tracing-cost-reporting.md](tracing-cost-reporting.md). The remaining cross-worker read-back is P1; the explicitly-deferred follow-ons are under P0.2.-------## P0.1 — Dry-run through activity dispatch (testing affordance) — DONE--Shipped as **`--mock-inference`** (distinct from `--dry-run`): a LIVE run (`run_mode` stays LIVE so operators dispatch normally) whose AI call is faked at the cogt inference *leaf*, inside the dispatched activity. So the cross-process surfaces — `_event_log_contexts` lookup miss, runner-side fallback, `writer_id="act_*"` emission — all fire, with no LLM spend.--As built:--- `JobMetadata.is_mock_inference: bool` carries the signal into every activity (it crosses the Temporal boundary), written single-source from `prepare_pipe_job`. The leaf (`cogt/content_generation/llm_generate.py`) branches to a shared mock in `cogt/content_generation/dry_mock.py` when the flag is set.-- The synthetic usage is **reportable non-zero** (`MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY = {INPUT: 100, OUTPUT: 50}`, model `mock_inference`) — distinct from `--dry-run`'s zero-token, suppressed usage — so the cross-worker cost report actually renders. This is the durable reason `--mock-inference` ≠ `--dry-run` at the reporting layer.-- `temporal-e2e-validate` Tier 8b runs in this mode and surfaces `wf_*__w_act_*.ndjson` + a rendered cost report deterministically.--**Scope deviations (deliberate):** the leaf mock covers the **LLM leaf only**; image-gen / extract / search under `--mock-inference` fail loud with `MockInferenceUnsupportedError` (their output is stored *above* the leaf, so a leaf mock would push synthetic data through storage). `--mock-inference` is on the main `run` subcommands only (not the agent CLI). Full per-operator coverage and the eventual `is_mock_inference → run_mode=DRY` re-keying are tracked in [`../dry-run-refactor/followup-leaf-run-mode-mock.md`](../dry-run-refactor/followup-leaf-run-mode-mock.md) (B2). See [the registry feature](../history/registry/README.md) and its [`cost-reporting-overview.html`](../history/registry/cost-reporting-overview.html) for the full as-built.-------## P0.2 — Deferred follow-ons from P0--Each is independently scoped and not blocking. Pick up individually as signals dictate.--- **`tracing_config.strict_mode: bool`** — opt-in flag that raises (instead of WARNING + drop) when the runner-side emit path fails. For compliance/audit deployments where missing trace data is a hard error.-- **DynamoDB `BatchWriteItem` for runner emission** — the current path does one `PutItem` per usage event; `BatchWriteItem` batches up to 25. Defer until throughput shows it matters.-- **NDJSON shared-filesystem invariant enforcement** — the NDJSON backend assumes `traces_dir` is visible to all writer processes (router pool + runner pool + `act_flush_trace_events`); multi-host deployments need NFS/EFS. Add a startup check that warns if `traces_dir` looks local-only on a multi-worker deployment.-- **Per-thread `writer_id` for activity event log** — the surgical lock closes the duplicate-sequence race on `next_sequence()`; a per-thread writer namespace would eliminate the contention entirely and align with how Temporal's worker pool partitions activities (dedup drops to per-thread, no hot-path lock).-- **Project-wide migration of `JobMetadata.started_at` to timezone-aware datetimes** — the landed patch builds the synthetic `now` with the incoming `started_at`'s `tzinfo`, removing the immediate `TypeError` but not the underlying inconsistency: the default factory is naive (`datetime.now`) while several callers (e.g. `pipe_abstract.py:428`) construct aware datetimes. A clean migration standardizes on `datetime.now(timezone.utc)`, documents the tz contract on `JobMetadata`, and lets `duration` subtract without per-site tzinfo matching.-------## P1 — Cross-worker cost report assembly — DONE--Shipped via **Option B** (the locked design; see [the registry overview](../history/registry/cost-reporting-overview.html)): rather than wire `read events → inject_tokens_usages → generate_report` onto the process-singleton manager, usage rides on `PipeOutput` exactly like the graph spec, and the submitter renders the report from that field. This resolved P1 **and** the `UsageRegistry` success-path leak in one move — with nothing populating a submitter-side registry, the registry was removed outright.--As built:--- The single trace-event read in `assemble_tracing` / `act_assemble_tracing` (renamed from `graph_assembly` / `act_assemble_graph`) feeds `UsageAggregator.aggregate()` when `--costs` is on, setting `PipeOutput.tokens_usages` in both direct and Temporal modes (the same hook that builds the `GraphSpec`).-- The submitter renders via `CostRegistry.generate_report(tokens_usages=...)` fed from `pipe_output.tokens_usages` (main CLI: `pipelex/reporting/cost_report_renderer.py::render_run_cost_report`).-- A dedicated `--costs` / `is_generate_costs` switch (default on) gates usage events + `UsageAggregator` over the shared event-log transport, independent of `--graph`.-- Retired: `ReportingManager.inject_tokens_usages` / `generate_report` / `open_registry` / `close_registry`, the `UsageRegistry` model, and the local-only registry fallback (`_get_registry_strict` / `_try_add_to_registry`) are all gone.--Cross-worker case verified without spend: parent + child + activity usage aggregates into one end-of-run cost report — `tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py`, `test_mock_inference_temporal.py`, and the `temporal-e2e-validate` skill's Tier 8b. The full phased as-built is in [the registry tracker](../history/registry/README.md).-------## P2 — Local cross-package dependencies in the crate--Detailed plan: [local-cross-package-deps.md](local-cross-package-deps.md).--In one line: ship dependency blueprints inside the `LibraryCrate` so workers don't need the dependency packages on PIPELEXPATH. Prerequisite for P3.-------## P3 — Remote dependencies from GitHub--Detailed plan: [remote-deps-from-github.md](remote-deps-from-github.md).--In one line: extend the blueprint collector from P2 with a remote (GitHub) resolver so the crate can carry deps fetched from external addresses, making workers fully stateless.-------## Dependencies between items--```-P0 (Tracing across separate workers)- │- ├─► P0.1 (Dry-run through activity dispatch — testing affordance)- ├─► P0.2 (Deferred follow-ons — independent, pick up individually)- │- ▼-P1 (Cross-worker cost report assembly)--P2 (Local cross-package deps)- │- ▼-P3 (Remote deps from GitHub)-```--P0/P0.1/P0.2/P1 and P2/P3 are independent tracks; P0 unlocked P0.1, P0.2, and P1; P2 blocks P3. **P0, P0.1, and P1 are shipped**; P0.2 (independent follow-ons) and P2/P3 remain. P0.1 (`--mock-inference`) is a testing affordance that raised confidence in P0 / P1 and unlocks future cross-process work.diff --git a/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md b/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.mddeleted file mode 100644index 294fd1191..000000000--- a/wip/distributed-execution/bridge-changes-sibling-repo-reconciliation.md+++ /dev/null@@ -1,109 +0,0 @@-# Runtime-bridge changes → sibling-repo reconciliation--**Status: DIAGNOSED — 2026-06-07.** Research + diagnosis are done (this session grepped both sibling repos, checked the dependency pins, and read the reference implementations). No sibling code has been changed yet — the reconciliation is sequenced behind #966 landing (see "Sequenced reconciliation plan"). This doc is now the authoritative breakage inventory; act on it, don't re-investigate.--## TL;DR--`_bridge` (this worktree, branch `feature/Runtime-bridge-extraction`, PR **#966**) is the clean extraction of the framework-agnostic runtime bridge, now reconciled against dev's distributed cost-reporting work (#967 + #968). That reconciliation renamed/removed parts of the bridge's public surface. Two downstream consumers depend on that surface:--- **`pipelex-mistralai-workflows`** (`/Users/lchoquel/repos/Pipelex/pipelex-mistralai-workflows`) — the published package (PyPI `pipelex-mistralai-workflows`, import root `pipelex_mistralai_workflows`). The Mistral Workflows plugin that invokes Pipelex pipes from inside Mistral Workflows activities.-- **`_workflows`** (`/Users/lchoquel/repos/Pipelex/_workflows`) — a pipelex worktree on branch `feature/Mistral-workflows-merge-4` (the bigger Mistral-Workflows-integration line, PR #954 family).--**The break is deferred, not live.** Both consumers are pinned to / based on a *pre-rename* pipelex, so nothing is broken at this instant. The break materializes the moment either consumer is moved onto post-#966 pipelex. (Item 7 below — added 2026-06-11 — has a different trigger: it's a behavioral break from PR #987's reporting-path change, and it lands via the package's `pipelex>=0.27.0` floor on the next pipelex release containing #987, no rev bump required.) The fix is mechanical: the Mistral-native graph/usage path is a mirror of the Temporal path, and it needs the **exact same** `act_assemble_graph → act_assemble_tracing` rewiring the Temporal path already received in #967 — plus it gains cost reporting it never had.--## Dependency reality (why it's deferred)--- **`pipelex-mistralai-workflows/pyproject.toml`** pins `pipelex` via `[tool.uv.sources]` to a **git rev**: `pipelex = { git = "...pipelex.git", rev = "8f52d40130e3bf6043569b303b7984844272dd4d" }`. The editable `pipelex = { path = "../_workflows", editable = true }` override sits right above it, **commented out**. `[project].dependencies` separately declares `pipelex>=0.27.0`, but the `[tool.uv.sources]` git rev wins for resolution.- - Rev `8f52d40` = tip of `feature/Mistral-workflows-merge-3` (`backup/pre-dev-merge`), dated 2026-06-01. It is **pre-rename** (has `pipelex/graph/graph_context.py`, not `trace_context.py`) and is **not an ancestor of `origin/dev` nor of this `_bridge` branch**. So the package currently builds against old pipelex and is internally consistent — it does **not** break until that rev is bumped (or the editable override is uncommented against a reconciled `_workflows`).-- **`_workflows` (merge-4)** is its own pipelex copy and is also **pre-rename / pre-#967**: it still has `pipelex/graph/graph_context.py` (no `trace_context.py`), has **no** `pipelex/pipe_run/tracing_assembly.py`, and still ships the old `pipelex/runtime_bridge/primitives/graph_assembly.py`. `origin/dev` is **not** an ancestor of merge-4; merge-4 carries ~100+ commits not in `_bridge`. It has the bridge extraction (`runtime_bridge/` with bootstrap/bridge/execution_mode/primitives) but built on a base that predates dev's #967/#968.--Net: the editable override couples the two — when uncommented, the package's `pipelex` comes from `../_workflows`. So the package can only be **correctly fixed and validated** once `_workflows` itself carries the reconciled (post-#966/#967/#968) pipelex. Fixing the package in isolation against an un-merged `_workflows` would be wrong. This dictates the ordering below.--## Breakage inventory — `pipelex-mistralai-workflows` package--All paths under `pipelex_mistralai_workflows/`. Split into hard breaks (won't import/run against post-#966 pipelex) and a parity gap (compiles, but silently drops cost).--### Hard breaks (renamed/removed pipelex surface)--1. **`primitives/act_pipelex_assemble_graph.py`** — imports `from pipelex.runtime_bridge.primitives.graph_assembly import assemble_graph_for_pipeline_run`. **That module + function are deleted** (replaced by `pipelex/pipe_run/tracing_assembly.py::assemble_tracing`). This whole file must be rewritten as `act_pipelex_assemble_tracing.py` (see fix pattern). It defines `AssembleGraphArg` + `act_pipelex_assemble_graph`.-2. **`primitives/wf_pipe_run.py`** (lines ~32-35, 112-123) — imports `AssembleGraphArg` / `act_pipelex_assemble_graph`, constructs `AssembleGraphArg(...)`, calls the activity, and copies **only** `graph_spec` + `graph_assembly_error` onto `pipe_output`. Must call the new tracing activity, gate on the run's emit flags, and copy all four fields.-3. **`primitives/wf_pipe_router.py`** (lines ~134, 156-173) — reads `pipe_job.job_metadata.graph_context` and `model_copy(update={"graph_context": ...})`. **`JobMetadata.graph_context` → `trace_context`** (confirmed: `pipelex/pipeline/job_metadata.py:74 trace_context: TraceContext | None`). The per-step tracer-open attributes it reads (`graph_id`, `data_inclusion`, `parent_node_id`) **survive** on `TraceContext` unchanged, so only the container attribute/var name changes here.-4. **`streaming.py`** (lines ~144, 167) — calls `run_pipe_via_bridge(input_payload, graph_context=graph_context)`. **The kwarg is renamed `graph_context` → `trace_context`** (confirmed: `bridge.py:88 run_pipe_via_bridge(input_payload, trace_context: TraceContext | None = None)`). The local var fed from `tracer_manager.open_tracer(...)` can be renamed for clarity but only the kwarg is load-bearing.-5. **`primitives/__init__.py`** + **`primitives/registration.py`** — both re-export `act_pipelex_assemble_graph` (and `__init__` the `AssembleGraphArg` symbol via the module). Update to the new activity name when (1) is rewritten.--### Parity gap (no compile break, but wrong post-#967)--6. The Mistral-native path **never assembled usage/cost** — `wf_pipe_run.py` only ever set `graph_spec`. Post-#967 cost rides on `PipeOutput.tokens_usages` (+ `usage_assembly_error`). When you rewrite (1)+(2) to use `assemble_tracing`, you get usage assembly **for free** from the same single event read; copy `tokens_usages` + `usage_assembly_error` onto `pipe_output` too. This is the "Mistral-native gains cost reporting" win, not just a break-fix.--### Behavioral break added after this diagnosis (PR #987, 2026-06-11): activity-side usage emissions bypass the plugin's registered buffer--7. **`primitives/wf_pipe_router.py` (usage-buffer contract) + deployment config.** PR #987 (`fix/Config`, the workflow-determinism fixes) added an H1 routing gate to `pipelex/reporting/reporting_manager.py::_emit_usage_event` (~line 210): any emission made **inside a temporalio activity** unconditionally skips the registered-context fast path and goes to the per-process activity event log (`act_{pid}_{uuid8}` writer). Mistral Workflows **is temporalio underneath** — its `@activity` decorator wraps `temporalio.activity.defn` (`mistralai/workflows/core/activity.py:181/:187`) — so every usage report from a plugin leaf activity (`act_pipelex_leaf.py`) has `in_activity() == True`. Consequence for the plugin's pre-H1 co-located contract (`wf_pipe_router.py:175-180` registers a `BufferingEventLog` via `set_event_log`; `:208-212` flushes gated on `if buffered_events:`): the buffer **silently stops receiving usage events**. Not a compile break, and not dead code — graph events are tracer-emitted from the workflow body and still land in the buffer — but the flush stops carrying usage. In default configs this is a **diversion, not a loss**: the fallback stamps `workflow_id = tracer_key or graph_id` (same partition) and writes through the backend built from `tracing_config`, which the multi-writer assembly tolerates. It degrades to **actual loss** when the activity-side process's `tracing_config` is disabled or unbuildable (the fallback bare-returns). The diversion incidentally gives the plugin the same H1 replay-determinism protection pipelex's own `WfPipeRouter` adopted — but the plugin never opted in, and its writer_id/sequence expectations change. Trigger condition: any pipelex containing #987 — the package's `pipelex>=0.27.0` floor means a fresh non-editable install picks this up the moment such a release ships, independent of the #966 rename breaks above. Fix during step 3: mirror post-#987 pipelex `wf_pipe_router.py` — drop the usage half of the buffer contract (usage rides the per-process activity log; make the flush schedule payload-pure) or consciously keep the buffer for graph events only and document the usage diversion; either way, ensure the deployment enables `tracing_config` where the activities run. Cross-ref: `nondeterminism-follow-ups-decisions-needed.md` §6 (the generic mechanism — live here, not latent).--### Confirmed NOT broken (don't touch — already verified to resolve against post-#966 pipelex)--- `streaming_event_forwarder.py`, `streaming.py`, `wf_pipe_router.py`, `pipe_run.py`, `pipe_router.py`, `act_pipelex_leaf.py`, `act_pipelex_deliver.py`, `act_pipelex_flush_trace_events.py`, `scoped_library.py`, `_kajson_codec.py`, `dependency.py`, `activities.py` import these, all of which **survive**: `pipelex.graph.graph_config.DataInclusionConfig`, `pipelex.tracing.event_log_protocol.EventLogProtocol`, `pipelex.graph.graph_tracer_manager.GraphTracerManager`, `pipelex.runtime_bridge.bootstrap.ensure_pipelex_booted`, and the bridge primitives `trace_flush.flush_trace_events_to_backend` / `hydration.hydrate_working_memory` / `pipe_classification.is_controller_pipe` / `submitter_hydration.rehydrate_pipe_output_with_crate` / `delivery.execute_delivery`.-- **`tests/integration/test_bridge_temporal_*.py`** import `from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router` — that path **still exists** in `_bridge` (only `pipe_run_arg` moved out of `temporal.tprl_pipe` into `runtime_bridge.primitives`). Not a break.-- The package uses **none** of `is_generate_costs`, `UsageRegistry`, `open_registry`, `close_registry`, `generate_report`, `inject_tokens_usages`, `--cost-report` — so #968's cost-API removals don't hit it directly. Its cost story is purely the additive parity gain in (6).--## The canonical fix pattern (mirror the Temporal path)--The Temporal path already did this exact migration in #967. Use these `_bridge` files as the reference implementation:--- **`pipelex/pipe_run/tracing_assembly.py`** — `assemble_tracing(pipeline_run_id, *, assemble_graph: bool, assemble_usage: bool, domain_code=None, main_pipe_code=None) -> TracingAssembly`. One event read → both artifacts. `TracingAssembly(extra="forbid")` carries `graph_spec` / `graph_assembly_error` / `tokens_usages` / `usage_assembly_error`. Best-effort read failures (incl. `EventLogReadError`) are caught inside and returned on the `*_error` fields; only programming bugs propagate.-- **`pipelex/temporal/tprl_pipe/act_assemble_tracing.py`** — the activity wrapper. `AssembleTracingArg` adds `assemble_graph: bool` / `assemble_usage: bool` (mirror of the run's emit flags) on top of `pipeline_run_id` / `domain_code` / `main_pipe_code`, and returns `TracingAssembly`. Deliberately **not** wrapped in error-classification — observability, not a pipe step.-- **`pipelex/temporal/tprl_pipe/wf_pipe_run.py`** (the dispatch, ~lines 87-122) — gates on `trace_context is not None and (trace_context.emit_graph_events or trace_context.emit_usage_events)`, passes `assemble_graph=trace_context.emit_graph_events` / `assemble_usage=trace_context.emit_usage_events`, then copies **all four** populated fields onto `pipe_output`. This is the precise shape `wf_pipe_run.py` in the Mistral package should take.--Mistral-package edits, concretely:--- Rename `act_pipelex_assemble_graph.py` → `act_pipelex_assemble_tracing.py`: `AssembleTracingArg(pipeline_run_id, domain_code, main_pipe_code, assemble_graph, assemble_usage)`; body `return assemble_tracing(...)` returning `TracingAssembly`; keep the Mistral `@activity(...)` decorator.-- In `wf_pipe_run.py`: gate on the run's emit flags (the Mistral parent has the `PipeJob`, hence `job_metadata.trace_context`; if `trace_context` is None, default both to True to preserve today's always-assemble-graph behavior — or read the config gate the way DIRECT does, decide during impl), call the new activity, copy `graph_spec` / `graph_assembly_error` / `tokens_usages` / `usage_assembly_error`.-- In `wf_pipe_router.py` + `streaming.py`: the `graph_context` → `trace_context` renames in the breakage inventory above.-- In `wf_pipe_router.py`: resolve item 7 (post-#987 usage routing) — mirror pipelex's post-#987 `pipelex/temporal/tprl_pipe/wf_pipe_router.py` (payload-pure flush gate; usage events ride the per-process activity log instead of the registered buffer), or keep the buffer for graph events only and document the usage diversion.-- Update `primitives/__init__.py` + `registration.py` exports.--## `_workflows` (merge-4) reconciliation--merge-4 is a divergent pipelex worktree that has the bridge extraction but predates dev's #967/#968. To carry the reconciled bridge it must absorb dev — the **same** reconciliation `_bridge` just performed, but it also has its own in-tree copies to migrate:--- its `pipelex/runtime_bridge/primitives/graph_assembly.py` must be deleted in favor of `pipe_run/tracing_assembly.py` (the merge with dev surfaces this);-- `pipelex/graph/graph_context.py` → `trace_context.py` + the `graph_context` → `trace_context` rename across `JobMetadata` and all callers;-- `is_generate_costs` → `is_generate_usage`; `UsageRegistry` removal; `--cost-report` → `--costs` — all per #967/#968.--Two viable paths; recommend (b):--- **(a)** Merge `origin/dev` into `_workflows` directly and reconcile by hand (re-doing the tracing reconciliation a second time, in a much bigger tree).-- **(b) Preferred:** land #966 onto dev first, *then* merge the post-#966 dev into `_workflows`. #966 is the already-reconciled bridge extraction, so this folds the bridge reconciliation in once and leaves only the Mistral-Workflows-specific integration code to fix up in merge-4. Confirmed prerequisite: merge-4 is downstream of the bridge work conceptually but **not** an ancestor/descendant of either dev or this branch today — it genuinely needs a merge, not a fast-forward.--## Sequenced reconciliation plan (ordering matters)--1. **Land #966** (this `_bridge` branch) onto `dev`. It carries the reconciled bridge surface (`trace_context`, unified `assemble_tracing`, `EventLogReadError` layering, deleted `graph_assembly` primitive).-2. **Merge post-#966 `dev` into `_workflows` (merge-4)**, reconciling merge-4's in-tree pipelex copies (graph_assembly deletion, `graph_context→trace_context`, `is_generate_costs→is_generate_usage`, UsageRegistry removal, CLI flag). Path (b) above.-3. **Fix the `pipelex-mistralai-workflows` package** against the reconciled API — the breakage inventory + canonical fix pattern above. Validate by **uncommenting** the editable `pipelex = { path = "../_workflows", editable = true }` override (now that `_workflows` carries the reconciled pipelex) and running the package's own pyright/mypy/tests + its `tests/integration/test_bridge_temporal_*` and Mistral-native suites.-4. **Re-pin** the package: replace the editable override with a git rev (or a released `pipelex` version) that includes #966+#967+#968, then re-comment the editable override per the file's own instructions ("Strip this override before publishing... Add it back on the next dev cycle when the next breaking change to `pipelex.runtime_bridge` lands").--Until step 1 happens, steps 2-4 are blocked and **no sibling code should change** — editing either consumer now would only desync it from its current (pre-rename) pipelex.--**Ready-to-run per-repo handoffs** (self-contained, keyed to "#966 has hit dev"; this doc is the index, those are the executables):--- `_workflows` → `/Users/lchoquel/repos/Pipelex/_workflows/wip/post-966-dev-reconciliation-handoff.md`. Key insight there: #966 *is* merge-4's bridge extraction, so once it's on dev merge-4's bridge work is redundant and its only remaining unique value is the mistralai-2.x HOLD group (blocked on `instructor` PR #2298). Recommends retiring merge-4 in favor of a minimal HOLD-group branch off the new dev, rather than a conflict-heavy merge.-- `pipelex-mistralai-workflows` → `/Users/lchoquel/repos/Pipelex/pipelex-mistralai-workflows/wip/post-966-bridge-reconciliation-handoff.md`. The breakage inventory + canonical fix pattern below, written as concrete edits with code, plus the repin/validate/settle steps. Can pin straight to a post-#966 dev rev — does not require the `_workflows` consolidation.--## The MISTRAL_NATIVE question — ANSWERED--Original open question: did `_run_mistral_native` or the sibling package consume the now-deleted `assemble_graph_for_pipeline_run`?--- **`_run_mistral_native`** (in `_bridge` `pipelex/runtime_bridge/bridge.py:394`) is **thin** — it imports `make_mistral_workflows_pipe_run`, calls `pipe_run.run(...)`, and serializes via `_serialize_completed_output`. It does **not** touch graph/usage assembly. No change needed there.-- **The sibling package DOES** consume it — `primitives/act_pipelex_assemble_graph.py` imports `assemble_graph_for_pipeline_run` directly and `primitives/wf_pipe_run.py` orchestrates it. So the Mistral-native graph/usage handling needs the identical `act_assemble_graph → act_assemble_tracing` / unified-`assemble_tracing` rewiring the Temporal path got, and additionally **gains** usage/cost assembly it never performed (parity gap (6)).--## Current state of `_bridge` (this worktree)--- Branch `feature/Runtime-bridge-extraction`, worktree `/Users/lchoquel/repos/Pipelex/_bridge`, PR **#966** (#959 is closed).-- 0 behind `dev`; unpushed commits ahead carry dev's #967 + the merge (tracing reconciliation: `EventLogReadError` layering, `graph_assembly` primitive deletion, `graph_context → trace_context`) + the TODOS retarget.-- All green at last check: `make agent-check` (pyright clean, mypy clean), unit suite, DIRECT-tracing + `runtime_bridge` integration tests; temporal tracing tests collect clean.--## Pointers / prior art--- `wip/runtime-bridge/` — review-triage of #959/#966/#969 + open deferred follow-ups. README is the cold-start entry. (Superseded the retired `TODOS.md` PR reviewer's guide.)-- `wip/distributed-execution/tracing-cost-reporting.md` — #967's as-built tracing/cost-reporting design.-- Memory (auto-loaded): `project_runtime_bridge_pr959_review` (this PR's state), `project_registry_leak_fix` (#967), `project_temporal_not_shipped` (Temporal not in prod).-- Reference impls in `_bridge` for the Mistral-package fix: `pipelex/pipe_run/tracing_assembly.py`, `pipelex/temporal/tprl_pipe/act_assemble_tracing.py`, `pipelex/temporal/tprl_pipe/wf_pipe_run.py`.diff --git a/wip/distributed-execution/crate-first-architecture.md b/wip/distributed-execution/crate-first-architecture.mddeleted file mode 100644index b5508e77a..000000000--- a/wip/distributed-execution/crate-first-architecture.md+++ /dev/null@@ -1,258 +0,0 @@-# Crate-First Architecture Vision--> **Status**: Vision / future direction-------## Summary--The LibraryCrate should become the central concept in library management. Today, library loading drives everything and crates are a byproduct. The future direction inverts this: **collecting blueprints and building the crate is the primary operation; loading into a live Library is just one consumer of the crate**.--This document captures the architectural direction, the Phase 2 design decision that aligns with it, and the two future capabilities it enables: crate stripping and remote dependency resolution.-------## Phase 2 Design Decision: Blueprint Accumulation (Option A)--### The question--Phase 2 ships the `LibraryCrate` inside `PipeJob` so Temporal workers can load the full library context. Library loading happens in multiple steps (PIPELEXPATH dirs, extra library_dirs, mthds_contents), each going through `load_from_blueprints()` which internally builds a crate. The question was how to produce the final crate to ship.--### Options considered--**Option A — Accumulate blueprints, build one crate at the end.** Track all blueprints that go through `load_from_blueprints()` in a `_blueprints: dict[str, list[PipelexBundleBlueprint]]` on `LibraryManager`. After all loading is done, `get_crate(library_id)` calls `LibraryCrateFactory.make_from_blueprints()` once with the full list.--**Option B — Accumulate and merge crates.** Each `load_from_blueprints()` call produces a crate. Accumulate them in `_crates: dict[str, LibraryCrate]` on `LibraryManager`. Add a `merge()` classmethod to `LibraryCrate` that combines two crates (raise on key collision, recompute fingerprint). `get_crate()` returns the merged result.--### Decision: Option A--Option A is simpler:-- No `merge()` method needed on `LibraryCrate`-- No crate accumulation dict or fingerprint recomputation-- Fewer concepts for a developer to understand-- Blueprints are the natural unit of accumulation — they're what loading already produces--Option A is also directionally correct toward the crate-first architecture (see below). Blueprints are the raw input; the crate is built from them. Accumulating the input (blueprints) rather than intermediate outputs (crates) keeps the data flow clear.--### Why loading still happens incrementally--Loading is order-dependent: dependencies must be loaded before dependents (pipe validation needs concepts from deps to succeed). So we can't defer all loading to the end. The library needs content loaded step by step.--But **crate building for shipping purposes** is independent of the loading order. We let loading proceed as today (each step goes through its own internal crate), and build the final shipping crate once at the end from all accumulated blueprints.-------## The Crate-First Architecture--### Current state: loading-driven--```-CURRENT ARCHITECTURE-════════════════════-- load_libraries(dirs)- → parse .mthds files → blueprints- → resolve local deps → more blueprints- → load_from_blueprints(blueprints)- → LibraryCrateFactory → crate (internal, discarded)- → load_from_crate(crate) → live Library-- load_from_blueprints(mthds_contents_blueprints)- → same path-- [Phase 2] get_crate() → build final crate from accumulated blueprints-```--Library loading drives everything. Crates are an internal intermediate. Dependency resolution is interleaved with loading (`_load_single_dependency` parses, creates child Library, loads concepts, builds pipes — all in one method).--### Future state: crate-driven--```-CRATE-FIRST ARCHITECTURE-═════════════════════════-- Phase 1: COLLECT (blueprints only, no Library)- ──────────────────────────────────────────────- Parse local .mthds files → blueprints- Scan blueprints for deps (manifest + address-based)- Resolve deps: local lookup + remote fetch- Parse dep blueprints- Recurse for transitive deps- → all_blueprints[]-- Phase 2: BUILD CRATE (pure data transform)- ──────────────────────────────────────────────- LibraryCrateFactory.make_from_blueprints(all_blueprints)- Strip to transitive closure of target pipe (optional)- → one complete, self-contained LibraryCrate-- Phase 3: LOAD (same code path on submitter and worker)- ──────────────────────────────────────────────────────- load_from_crate(crate)- → validates, creates live objects, ready to execute-```--The key shift: **dependency resolution moves out of library loading and into blueprint collection.** The crate becomes the single source of truth. Both submitter and worker use the same `load_from_crate()` path.-------## Future Capability 1: Crate Stripping (Transitive Closure)--### What--From the entry pipe, walk the transitive closure of pipe and concept dependencies to determine the minimal subset needed. Strip the crate to only those concepts, pipes, and domains.--### Why--- Reduces Temporal payload size (relevant before StoragePayloadCodec, and even after for efficiency)-- Clarifies execution context — a worker receives exactly what it needs, nothing more-- Enables better error messages — "pipe X not in crate" vs. "pipe X not found" when deps are missing--### How it fits--Crate stripping is a pure data transform on blueprints. Controller blueprints contain pipe ref strings (PipeSequence has `steps`, PipeBatch has `pipe`, etc.), so the dependency graph can be traced from blueprints alone without loading into live Pipe objects.--In the crate-first architecture, stripping slots in between Phase 1 (collect) and Phase 2 (build):--```- all_blueprints = collect(...)- required_refs = trace_transitive_closure(target_pipe, all_blueprints)- stripped_blueprints = filter(all_blueprints, required_refs)- crate = LibraryCrateFactory.make_from_blueprints(stripped_blueprints)-```--### Current blocker--Today, dependency content goes into **child libraries** via `_load_single_dependency`, which has its own loading path that doesn't go through `load_from_crate()` or produce blueprints that accumulate. To strip, we need all blueprints (including dependency blueprints) in one flat list. This requires decoupling resolution from loading (see "What needs to change" below).-------## Future Capability 2: Remote Package Dependencies--### What--Resolve dependencies from remote package addresses (e.g., `github.com/org/repo/package`). Fetch the package (temporary clone or download), parse its bundles, and include the needed content in the crate.--### Why--- Enables running pipelines whose dependencies aren't pre-installed on the worker-- The crate becomes truly self-contained — a worker needs nothing beyond PIPELEXPATH base-- Enables cloud-native execution where workers are stateless--### How it fits--Remote fetching is pure I/O — it doesn't need a live Library. It produces blueprints, same as local resolution. In the crate-first architecture, it slots into Phase 1 (collect):--```- Phase 1: COLLECT- Parse local .mthds files → blueprints- Scan for deps (manifest + address-based)- For each dep:- if local: find installed package → parse blueprints- if remote: fetch package → parse blueprints ← NEW- Recurse for transitive deps- → all_blueprints[]-```--### Current blocker--Same as crate stripping: `_load_single_dependency` interleaves resolution with loading. Remote fetch would need to happen before loading, which means splitting the "resolve + collect blueprints" step from the "load into Library" step.-------## What Needs to Change (Incremental Path)--The path from current state to crate-first is incremental. Each step is independently valuable.--### Step 1: Phase 2 (now) — Blueprint accumulation--**Already decided.** Track blueprints in `LibraryManager._blueprints`, build one crate via `get_crate()`. No merge logic. This establishes the pattern of accumulating blueprints as the unit of crate construction.--Dependency content (child libraries) is NOT included in the crate. Known limitation, accepted for Phase 2.--### Step 2: Extract blueprint collector from dependency loading--Split `_load_single_dependency` into two parts:-1. **Collect**: resolve the dependency, parse its .mthds files into blueprints, determine exports-2. **Load**: create child Library, load domains/concepts/pipes, register aliases--The collector produces blueprints. The loader consumes them. Today they're one method.--After this split, the collector's blueprints can be accumulated into `_blueprints[library_id]` alongside the main package's blueprints. The crate becomes complete (includes dependency content).--### Step 3: Add remote fetch to the collector--With the collector extracted, remote fetch is a new resolution strategy alongside local lookup:--```python-def _collect_dependency_blueprints(self, dep) -> list[PipelexBundleBlueprint]:- match dep.resolution:- case ResolutionStrategy.LOCAL:- return self._collect_from_local(dep)- case ResolutionStrategy.REMOTE:- return self._collect_from_remote(dep) # NEW-```--### Step 4: Add crate stripping--With all blueprints (main + deps) accumulated, add a stripping step before crate construction:--```python-def get_crate(self, library_id, target_pipe_ref=None) -> LibraryCrate | None:- all_blueprints = self._blueprints.get(library_id)- if target_pipe_ref:- all_blueprints = strip_to_transitive_closure(target_pipe_ref, all_blueprints)- return LibraryCrateFactory.make_from_blueprints(all_blueprints)-```--### Step 5 (optional): Fully separate Collect → Build → Load--At this point, the three phases are logically separate but still wired through `LibraryManager`. The final step is to make them explicitly separate, so that:-- A CLI tool can collect + build a crate without loading (e.g., `pipelex crate build`)-- A worker can load from a crate without collecting (already works via `load_from_crate`)-- Tests can construct crates directly from blueprints without the full loading machinery-------## Cross-Package Refs in a Flattened Crate--One structural challenge for including dependency content in the crate: cross-package pipe refs use alias-based syntax (`alias::domain.pipe`). Today, these resolve through child library lookups. In a flattened crate, there are no child libraries.--Options:-1. **Resolve aliases during crate building** — replace `alias::domain.pipe` with the actual `domain.pipe` ref in all controller blueprints before they enter the crate. The crate contains only flat domain-qualified refs.-2. **Carry alias mappings in the crate** — add an `aliases: dict[str, str]` field that maps alias-prefixed refs to domain-qualified refs. `load_from_crate()` uses this to register aliased lookups.-3. **Defer** — cross-package deps on workers remain a known limitation until this is solved.--Option 1 is cleanest (the crate is fully resolved, no runtime lookup needed) but requires modifying blueprint data during crate construction. Option 2 is simpler to implement but adds runtime complexity. To be decided when we reach Step 2.-------## Relationship to Other Phases--| Phase | Relationship to crate-first |-|-------|-----------------------------|-| Phase 2 (Temporal crate) | Establishes blueprint accumulation pattern (Step 1) |-| Phase 3 (WM hydration) | Independent — concerns deserialization, not crate content |-| Phase 4 (StoragePayloadCodec) | Complementary — handles large crates; stripping reduces need |-| Crate stripping | Step 4 of this vision |-| Remote deps | Step 3 of this vision |-| Library fingerprint validation | Independent — compares base library fingerprints |-| Cross-worker cache | Complementary — caches the crate, benefits from stripping |-------## Deferred / future capabilities--Capabilities the crate-first direction enables but that are not planned for implementation now. Crate stripping and remote deps are detailed above (Future Capabilities 1 and 2); these round out the picture.--### Library fingerprint validation--**Status**: Future — no immediate need.--Verify that the worker's base library (PIPELEXPATH) matches the API's expectation. Detect version drift between API and worker deployments.--### Cross-worker cache via shared storage--**Status**: Future — complementary to crate stripping.--Cache loaded crates in shared storage (Redis, S3) so multiple workers don't redundantly load the same crate. Keyed by fingerprint. Smaller crates from stripping make this more effective.diff --git a/wip/distributed-execution/local-cross-package-deps.md b/wip/distributed-execution/local-cross-package-deps.mddeleted file mode 100644index b6df86738..000000000--- a/wip/distributed-execution/local-cross-package-deps.md+++ /dev/null@@ -1,27 +0,0 @@-# Local Cross-Package Dependencies in the Crate--> Part of the [distributed-execution plan](README.md) (P2); a step toward [crate-first-architecture.md](crate-first-architecture.md).--## Why--Today, cross-package dependencies (`alias->domain.ConceptCode`, `alias::domain.pipe_code`) resolve through child library lookups at loading time. The `LibraryCrate` shipped to Temporal workers does NOT include dependency content — only the main package's blueprints. This means workers must have all dependency packages pre-installed on PIPELEXPATH. The goal is to remove this requirement: the crate becomes truly self-contained, so workers can execute cross-package pipelines without the dependency packages installed.--## Key changes--1. **Extract blueprint collector from `_load_single_dependency`**: Split the current method into two parts:- - **Collect**: resolve the dependency, parse its `.mthds` files into blueprints, determine exports.- - **Load**: create child Library, load domains/concepts/pipes, register aliases.-- The collector produces blueprints that accumulate into `_blueprints[library_id]` alongside the main package's blueprints, so `get_crate()` produces a crate that includes dependency content.--2. **Resolve cross-package aliases in the flattened crate**: Cross-package refs use alias-based syntax (`alias->domain.ConceptCode`). In a flattened crate, there are no child libraries. Options:- - Resolve aliases during crate building (replace `alias->domain.ConceptCode` with `domain.ConceptCode` in all blueprints).- - Carry alias mappings in the crate (`aliases: dict[str, str]`) for runtime resolution.-- Decision TBD when starting implementation.--3. **Update `load_from_crate()`**: Handle dependency content in the flat crate — register domains, concepts, and pipes from deps.--## Acceptance--A PipeSequence referencing a cross-package concept/pipe executes on a Temporal worker that does **not** have the dependency package on PIPELEXPATH.diff --git a/wip/distributed-execution/mistral-workflows/README.md b/wip/distributed-execution/mistral-workflows/README.mddeleted file mode 100644index 0d28e4010..000000000--- a/wip/distributed-execution/mistral-workflows/README.md+++ /dev/null@@ -1,14 +0,0 @@-# Mistral Workflows merge — readiness & plan--Sub-track of [distributed-execution](../README.md): bringing the `pipelex-mistralai-workflows` plugin (Mistral's native Workflows as a Pipelex execution backend) onto current `dev`. This is the MISTRAL_NATIVE execution mode's home outside the core runtime.--## Docs--- [mistral-workflows-merge-readiness.md](mistral-workflows-merge-readiness.md) — **the hub.** Cold-start assessment of whether `feature/Mistral-workflows-merge-4` is safe to PR into `dev`. Verdict: the `runtime_bridge` extraction is clean/additive/green and safe to land; the blocker is a `mistralai` 1.x→2.x bump that drags in a temporary `instructor` git-fork pin on a core dep (blocked on instructor PR #2298). Recommends splitting — land the bridge, hold the SDK bump.-- [mistral-workflows-plugin-extract.md](mistral-workflows-plugin-extract.md) — design: extracting the plugin surface.-- [mistral-workflows-sub-module.md](mistral-workflows-sub-module.md) — design: the sub-module layout option.-- [mistral-workflows-merge-4-refresh-now-prompt.md](mistral-workflows-merge-4-refresh-now-prompt.md) — ready-to-paste prompt to keep `merge-4` alive while held: merge `dev` in now (only `CHANGELOG.md` conflicts; the mistral bump must survive).-- [mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md](mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md) — ready-to-paste prompt for after the bridge squash-merges to `dev`: rebuild the branch off `dev` and re-apply only the mistral delta, to dodge the squash-merge tangle.-- [docs-held-back-from-live-site.md](docs-held-back-from-live-site.md) — **re-publish checklist.** The Mistral Workflows user docs ship in the repo but are excluded from the published docs site (so it doesn't advertise the unreleased `pipelex-mistralai-workflows` package). What's held back and how to restore it when the package launches — claim the PyPI name first.--Pairs with [bridge-changes-sibling-repo-reconciliation.md](../bridge-changes-sibling-repo-reconciliation.md) in the parent track, which tracks what the downstream `pipelex-mistralai-workflows` repo needs reconciled after the `_bridge` surface changes.diff --git a/wip/distributed-execution/mistral-workflows/docs-held-back-from-live-site.md b/wip/distributed-execution/mistral-workflows/docs-held-back-from-live-site.mddeleted file mode 100644index 35ccb3048..000000000--- a/wip/distributed-execution/mistral-workflows/docs-held-back-from-live-site.md+++ /dev/null@@ -1,42 +0,0 @@-# Mistral Workflows user docs — held back from the live site (re-publish on launch)--**Status: HELD BACK on the bridge branch.** The Mistral Workflows user docs are written and live in the repo, but the bridge PR keeps them **out of the published docs site** (docs.pipelex.com) so the site doesn't advertise the `pipelex-mistralai-workflows` package before it exists on PyPI. Re-publish them when the package launches.--## Why--The PyPI distribution name `pipelex-mistralai-workflows` is not registered yet. Printing `pip install pipelex-mistralai-workflows` on a public page invites someone to squat the name. The package also isn't published, so any install instruction would just fail. So the bridge PR holds the user-facing docs back rather than shipping docs for an unreleased package.--The **code** (the bridge entry point, primitives, execution modes) ships in this PR regardless — only the user-facing Mistral Workflows *docs* are held back.--## What is held back--The six pages under `docs/distributed-execution/mistral-workflows/` (`index`, `installation`, `your-first-pipelex-workflow`, `execution-modes`, `streaming`, `choosing-a-backend`). They still exist in the repo; they are excluded from the MkDocs build via `exclude_docs:` in `mkdocs.yml`, so they are neither served nor in the nav. The parent pages were softened to "coming soon" and de-linked from them (otherwise the `mkdocs build --strict` in `doc-check.yml` fails on links to excluded pages).--## Prerequisite before re-publishing--1. **Claim the name on PyPI first** — register `pipelex-mistralai-workflows` (a `0.0.0` placeholder release is enough) so re-publishing the docs no longer advertises an unclaimed name. This is the real squatting fix; the doc hold is only defense-in-depth (the name already appears across the public repo).-2. The package must actually publish, so `pip install pipelex-mistralai-workflows` resolves.--## Re-publish checklist (reverse of the hold)--In `pipelex` (wherever the bridge has landed — `dev`, then merged into `_workflows`):--- **`mkdocs.yml`** — delete the `exclude_docs:` block for `distributed-execution/mistral-workflows/`, and restore the `Mistral Workflows:` nav subsection (Overview / Installation & Preview Status / Your First Pipelex Workflow / Execution Modes / Streaming Progress / Choosing a Backend) under `Distributed Execution`.-- **`docs/distributed-execution/index.md`** — restore the two-backend overview (Temporal + Mistral Workflows, with the links and the "what's shared" framing) instead of the Temporal-only + "coming soon" version.-- **`docs/features/distributed-execution.md`** — restore the second backend bullet (link + name) and the "Choosing a Backend" link; restore the Mistral mention in the "Get started" line.-- **`docs/distributed-execution/temporal/index.md`** — restore the "one of two backends … see Pipelex on Mistral Workflows / Choosing a Backend" cross-links.-- **`docs/under-the-hood/pipe-routing-and-execution.md`** — restore the two-host-runtime framing and the `See [Execution Modes]` link.-- **`docs/reliability/durable-execution.md`** — restore the two-backend wording and the Mistral / Choosing-a-Backend links.-- **`README.md`** — restore the real install instruction (`pip install pipelex-mistralai-workflows`) in the "Mistral Workflows orchestration" note, replacing the "coming soon / install details follow" placeholder.-- **`CHANGELOG.md`** — restore the install + import-path guidance (`pip install pipelex-mistralai-workflows`, import from `pipelex_mistralai_workflows.*`) in the "Mistral Workflows integration extracted" entry.--After restoring, run `mkdocs build --strict` (matches `doc-check.yml`) and confirm the Mistral pages build and the nav renders.--## Where the hold was done (to diff/revert against)--On the bridge branch (`feature/Runtime-bridge-extraction`, PR #969):--- de-advertising README/CHANGELOG (removed the `pip install` lines): commit `e8dfa0e2`.-- unpublishing the docs (mkdocs `exclude_docs` + nav removal + parent-page softening): the commit that adds this file.--`git show <sha>` on those two commits gives the exact reverse diff for the checklist above.diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.mddeleted file mode 100644index 8361abb17..000000000--- a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md+++ /dev/null@@ -1,27 +0,0 @@-# Prompt — rebuild `feature/Mistral-workflows-merge-4` after the runtime-bridge PR squash-merges to `dev` (use later)--Paste the block below into a fresh Claude Code session in `/Users/lchoquel/repos/Pipelex/_workflows`. **Only use this once `feature/Runtime-bridge-extraction` (PR #959) has landed on `dev`.** Until then, keep the branch fresh with `wip/mistral-workflows-merge-4-refresh-now-prompt.md` instead.-------Background: `feature/Mistral-workflows-merge-4` is a long-lived branch = **`dev` + a held-back `mistralai` 1.x→2.x SDK bump** (held on `instructor` PR #2298; see `wip/mistral-workflows-merge-readiness.md`). It carries its **own, original, unsquashed** copy of the runtime-bridge commits (`71de6fb1`, `1a0f5dfa`, the `runtime_bridge` extraction, etc.).--The runtime-bridge work has now squash-merged into `dev` as a **single** commit — so `dev` and `merge-4` both contain the bridge, but as **different SHAs with diverged content** (dev has the review-followed-up version; merge-4 has the older one). A plain `git merge dev` would now conflict hard across every `pipelex/runtime_bridge/**` file — the classic "long-lived branch outlives the squash-merge of its own content" tangle. **Do not try to merge through that.**--Instead, **rebuild the branch fresh off `dev` and re-apply only the mistral delta.** The branch's *unique* content is tiny — the mistral provider files plus the dependency bump; everything else (including the now-canonical, review-followed-up `runtime_bridge`) comes for free from `dev`.--Please do this:--1. `git fetch origin`. Confirm the bridge actually landed on `dev`: `git ls-tree origin/dev pipelex/runtime_bridge` should be non-empty. If it's empty, **stop** — the bridge hasn't landed; use the refresh prompt instead.-2. Preserve the old branch: rename `feature/Mistral-workflows-merge-4` → `archive/Mistral-workflows-merge-4-presquash` (or just leave it and branch anew). Don't delete it until the rebuild is verified green.-3. Create the new branch off `origin/dev` (call it `feature/Mistral-workflows-merge-5`, or reset `-4` to `origin/dev` if you prefer to keep the name).-4. Re-apply **only the mistral delta**. The authoritative file list is the "Hold on the branch" section of `wip/mistral-workflows-merge-readiness.md` — read it, don't trust the list below blindly. As of writing it is:- - **source:** `pipelex/plugins/mistral/{mistral_config,mistral_extract_worker,mistral_factory,mistral_llm_worker,mistral_llms}.py` (includes the `mistralai.client.*` 2.x import reorg and the `message is None` retry guard in `mistral_llm_worker.py`).- - **tests:** `tests/unit/pipelex/plugins/{test_plugin_pipelex_storage_images,test_transport_retry_wiring}.py` and `tests/unit/pipelex/plugins/mistral/{test_mistral_worker_error_handling,test_extract_mistral_metadata,test_mistral_llm_worker_object_error_handling,test_mistral_extract_worker_semantic,test_mistral_reasoning}.py`.- - Bring these over with `git checkout <old-merge-4-branch> -- <files…>`. Reconcile by hand if `dev` changed any of them in the meantime.-5. Re-apply the **dependency bump** to `pyproject.toml`: `mistralai>=2.4.4` and the `[tool.uv.sources] instructor = { git = "https://github.com/Ian321/instructor.git", rev = "…" }` fork pin (keep the `# Temporary: …` comment referencing instructor #2298). Then **regenerate the lockfile fresh** with `uv lock` — do **not** copy the old `uv.lock`; let it resolve against current `dev`.-6. Run `make agent-check` and `make agent-test`; both must be green.-7. Confirm the held-back state is correct: `uv.lock` resolves `mistralai` 2.x + `instructor` from the git rev, and the new branch's diff vs `dev` is *just* the mistral provider files + the two dependency files (no stale bridge churn). That small, crisp diff is exactly what becomes the PR once `instructor` ships mistralai-2.x support on PyPI.-8. Do not push or open a PR unless I ask — but do report the final diff-vs-`dev` summary so I can sanity-check the delta.--After this, update `wip/mistral-workflows-merge-readiness.md` and `wip/README.md` to reflect the new branch name and that the bridge is now upstream on `dev`.diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-refresh-now-prompt.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-refresh-now-prompt.mddeleted file mode 100644index ff7ed8259..000000000--- a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-4-refresh-now-prompt.md+++ /dev/null@@ -1,27 +0,0 @@-# Prompt — refresh `feature/Mistral-workflows-merge-4` from `dev` (use now)--Paste the block below into a fresh Claude Code session in `/Users/lchoquel/repos/Pipelex/_workflows`.-------We keep `feature/Mistral-workflows-merge-4` alive as a long-lived branch = **`dev` + a held-back `mistralai` 1.x→2.x SDK bump** (held because `instructor` has no PyPI release that supports mistralai 2.x — blocked on instructor PR #2298). The full rationale is in `wip/mistral-workflows-merge-readiness.md`; read its TL;DR before touching dependencies.--I want to bring `feature/Mistral-workflows-merge-4` up to date with the latest `dev`. Please:--1. `git fetch origin`, then check out `feature/Mistral-workflows-merge-4` (confirm the working tree is clean first).-2. Merge `origin/dev` into it.--**What to expect — heads up:**--- The merge conflicts in **`CHANGELOG.md` only**. It's the routine "both sides added a `[Unreleased]` entry" conflict — resolve by keeping both sets of bullets (union), no content is lost. Nothing else should conflict.-- **Do NOT merge `feature/Runtime-bridge-extraction` into this branch.** That branch's tip pins `mistralai 1.12.0` in the lockfile, which would conflict with — and risk silently reverting — this branch's whole reason to exist (the mistralai 2.4.4 bump). `dev` is the only correct merge source. (The runtime-bridge work isn't on `dev` yet anyway, and this branch already carries its own copy.)-- **The mistral SDK bump must survive untouched.** After resolving, verify the held-back dependency state is still intact:- - `pyproject.toml` still has `mistralai>=2.4.4` and the `[tool.uv.sources] instructor = { git = "https://github.com/Ian321/instructor.git", rev = "…" }` fork pin.- - `uv.lock` still resolves `mistralai` 2.x (not 1.12.x) and `instructor` from the git rev.- - `pipelex/plugins/mistral/*.py` keep the `mistralai.client.*` 2.x import paths.- - `dev` brings no mistral dependency changes, so none of the above should have moved — but check, because this is the one thing we must not lose.--3. Run `make agent-check` and `make agent-test`; both must be green before you finish.-4. Do **not** push or open a PR — this is just keeping the branch fresh. Leave it on the merged commit and summarize what came in from `dev` (e.g. v0.31.0 release, #960, #962, #963).--If anything other than `CHANGELOG.md` conflicts, stop and show me — that would mean the topology changed (e.g. the bridge already landed on `dev`), and the right move is then the rebuild path in `wip/mistral-workflows-merge-4-rebuild-after-bridge-lands-prompt.md`, not a plain merge.diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-readiness.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-readiness.mddeleted file mode 100644index 27212e8a6..000000000--- a/wip/distributed-execution/mistral-workflows/mistral-workflows-merge-readiness.md+++ /dev/null@@ -1,149 +0,0 @@-# Mistral Workflows — merge-to-`dev` readiness assessment--**Branch:** `feature/Mistral-workflows-merge-4` · **Assessed:** 2026-06-03 · **HEAD:** `ff15bd58` · **dev tip / merge-base:** `fa15d15c`--Cold-start reference for the decision: *is this branch safe to PR into `dev`?* Verified against the actual code. **This doc is the merge basis.** Companion design docs: [`mistral-workflows-plugin-extract.md`](mistral-workflows-plugin-extract.md), [`mistral-workflows-sub-module.md`](mistral-workflows-sub-module.md). Runtime-bridge review-triage + open follow-ups: [`../../runtime-bridge/README.md`](../../runtime-bridge/README.md).--## TL;DR verdict--The **`runtime_bridge` extraction is genuinely solid** — clean, additive, behavior-neutral, fully green — and safe to merge to `dev` on its own. The branch's problem is that it **couples that clean work to a Mistral SDK major bump (`mistralai` 1.x → 2.x) that drags in a temporary `instructor` git-fork pin on a *core* dependency**, blocked on an upstream PR that is still open with no release path. That dependency state is **not release-grade** and is the only reason not to merge the whole branch as-is.--**Recommendation: split.** Land the bridge now; hold the `mistralai` 2.x bump + `instructor` fork pin until `instructor` ships PyPI support for mistralai 2.x. **The safe branch `feature/Runtime-bridge-extraction` was built off `dev` and is green on 2026-06-03** (`agent-check` + `agent-test` pass; mistralai held at 1.x, temporalio 1.24 included, the `message is None` guard left with the HOLD group). Details in [Recommendation](#recommendation).--## Scope reality (read this first)--The branch *looks* huge — `dev..HEAD` lists ~50 commits and `git diff --stat dev...HEAD` is 138 files / ~19k insertions — but most of that is **already in `dev`** or is docs:--- The big **Temporal-primitives + text-then-object / PipeStructure** work already landed in `dev` as the squash-merge **#891** ("Temporal merge 3: distributed execution + PipeStructure + content generation collapse"). This branch carries the *original unsquashed commits* (different SHAs), so they show in `dev..HEAD` but **net out** in the diff.-- Because the merge-base equals `dev`'s tip (`fa15d15c`), `git diff dev...HEAD` and `git diff dev HEAD` are identical — both show the true net add-on-top-of-dev.-- **Net source diff vs `dev` is small: 26 files / +717 −95 in `pipelex/`.** The rest is additive docs (`docs/`, error pages), the vendored `.claude/skills/workflows/` reference set, and `wip/` planning docs.--So this is NOT "merging the whole Temporal + TTO + Mistral epic." It is "merging the `runtime_bridge` extraction + a Mistral SDK bump + additive docs."--## What's verified green ✅--Run on HEAD `ff15bd58`, working tree clean before and after:--- **`make agent-check`** → ruff + plxt (TOML/MTHDS) + **pyright 0 errors** + **mypy clean (2007 source files)**.-- **`make agent-test`** → full non-inference suite, `-n auto`, markers `(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api` → **all passed**. This *includes* unit + every key-free integration test: the `runtime_bridge` DIRECT integration test (`tests/integration/pipelex/runtime_bridge/test_bridge_direct.py`) and the in-process Temporal tests.-- `runtime_bridge` has **no `TODO`/`FIXME`/`NotImplementedError` stubs**; ~1,124 lines of new bridge tests (unit + integration).-- **Not covered by `agent-test`** (key-gated, can't run locally): a **live Mistral inference + OCR round-trip**. The mistralai 2.x bump is verified at type/import level and by the non-inference suite, but not end-to-end against the real API. Run this with a real key before relying on it in production.--## Quality of the net change--Clean. Highlights:--- **`pipelex/runtime_bridge/`** — framework-agnostic core so any host runtime (not just Mistral Workflows) can embed Pipelex. `bridge.py` (boundary types + `run_pipe_via_bridge` dispatch), `bootstrap.py` (`ensure_pipelex_booted`), `execution_mode.py` (`PipelexExecutionMode`), `exceptions.py`, and a `primitives/` subpackage (`delivery`, `graph_assembly`, `hydration`, `pipe_classification`, `submitter_hydration`, `trace_flush`, `pipe_run_arg`). Exhaustive `match/case`, enum `@property` predicates (respects the "never `==` an enum" rule), strong docstrings.-- **Temporal activities became thin wrappers** delegating to `runtime_bridge.primitives` — e.g. `act_assemble_graph` is now a ~4-line `@activity.defn` over `assemble_graph_for_pipeline_run(...)`; the body moved verbatim into the primitive. **Behavior-neutral lift** (the CHANGELOG entry asserts the same). Touched: `temporal/tprl_pipe/{act_assemble_graph,act_deliver,act_flush_trace_events,temporal_pipe_router,temporal_pipe_run,wf_pipe_router,wf_pipe_run}.py` + `pipe_run/delivery_executor.py`.-- **`MISTRAL_NATIVE` is the safe preview shape.** Inside `pipelex` it is an additive enum value whose dispatch branch (`bridge.py::_run_mistral_native`) does a *deferred import* of `pipelex_mistralai_workflows.primitives.pipe_run` and immediately raises **`MissingMistralWorkflowsPluginError`** (with a `pip install pipelex-mistralai-workflows` hint) if the external package is absent. No Mistral-specific logic lives in `pipelex`. **Merging it does not activate any preview behavior for existing users**; the `DIRECT` / `TEMPORAL_*` paths are untouched. The actual Mistral-native decomposition lives in the separate `pipelex-mistralai-workflows` repo.--The CHANGELOG `[Unreleased]` "Changed" bullet frames this correctly as a pure extraction + promotion of `runtime_bridge` with **no behavior change**.--## The blocker: dependency coupling ⚠️--This is the "something bad" risk for `dev`. All in `pyproject.toml` / `uv.lock`:--- `mistralai>=1.12.0` → **`mistralai>=2.4.4`** — a **major bump of the live Mistral LLM + OCR provider**. Forces the import reorg across `pipelex/plugins/mistral/*` (`mistralai.models` → `mistralai.client.models`, `mistralai.client.errors`, etc.). A genuine robustness fix rides along (a `message is None` guard in `mistral_llm_worker.py` that raises a transient/retryable error).-- **`[tool.uv.sources] instructor = { git = "https://github.com/Ian321/instructor.git", rev = "4ea22d2396ca35514929f27faed866115e8b3583" }`** — marked *"Temporary."* `instructor` is a **core base dependency** (`pyproject.toml` line ~28, `instructor>=1.13`), so the fork pins the **entire lockfile**, not just the mistralai extra. `uv.lock` resolves `instructor 1.15.1` from that git rev.-- `temporalio==1.23.0` → `temporalio==1.24.0` — minor pinned bump, low-risk, independent.--Why the fork pin matters (checked 2026-06-03):--- `instructor` PR **#2298 "support mistralai v2" is OPEN, not merged** (ready-for-review since 2026-05-06). There is **no PyPI `instructor` release that works with mistralai 2.x** — it's a known-broken combo (`instructor` issue **#2137**: mistralai 2.0.0 broke `from mistralai import Mistral`, which also breaks instructor for *other* providers when mistral is installed).-- So the fork pin is **load-bearing with no near-term removal path** — it depends on an external maintainer merging and releasing.-- **Published-package knock-on:** `[tool.uv.sources]` is a uv-workspace concept and is **not** baked into the published wheel's dependency metadata. It only saves *this repo's* lockfile/CI. A PyPI consumer running `pip install pipelex[mistralai]` would get `mistralai>=2.4.4` + PyPI-`instructor` (no 2.x support) → **broken structured-output combo**. So bumping the published `mistralai` floor couples pipelex's release health to instructor's, and should not reach a release until instructor publishes 2.x support.--## Recommendation--`runtime_bridge` has **zero `import mistralai`** — it is fully decoupled from the SDK bump. Two viable paths:--### Option A — split (preferred)--**Land on `dev` now** (additive, behavior-neutral, fully green):--- `pipelex/runtime_bridge/**` (incl. `primitives/`)-- Temporal rewiring: `pipelex/temporal/tprl_pipe/{act_assemble_graph,act_deliver,act_flush_trace_events,temporal_pipe_router,temporal_pipe_run,wf_pipe_router,wf_pipe_run}.py`, `pipelex/pipe_run/delivery_executor.py`-- The `runtime_bridge` tests, the additive `docs/` (error pages + `docs/distributed-execution/mistral-workflows/`), the `.claude/skills/workflows/` reference set-- The `MISTRAL_NATIVE` enum + deferred-dispatch + `MissingMistralWorkflowsPluginError` (harmless without the SDK bump — it just fails fast)-- The CHANGELOG "Mistral Workflows extracted" bullet-- The `temporalio==1.24.0` bump (verified independent of the SDK work; carried via `uv lock --upgrade-package temporalio` off dev's lock so mistralai stays pinned at 1.12.4)--**Hold on the branch** until `instructor` ships PyPI mistralai-2.x support (track #2298):--- `pyproject.toml` lines: `mistralai>=2.4.4` and the `[tool.uv.sources] instructor` fork pin. (`temporalio==1.24.0` is separable — it landed in the safe PR.)-- the coupled `uv.lock` changes-- **Mistral provider — source AND tests** (all carry the `mistralai.client.*` import reorg, which is required by 2.x):- - source: `pipelex/plugins/mistral/{mistral_config,mistral_extract_worker,mistral_factory,mistral_llm_worker,mistral_llms}.py`- - tests: `tests/unit/pipelex/plugins/{test_plugin_pipelex_storage_images,test_transport_retry_wiring}.py` and `tests/unit/pipelex/plugins/mistral/{test_mistral_worker_error_handling,test_extract_mistral_metadata,test_mistral_llm_worker_object_error_handling,test_mistral_extract_worker_semantic,test_mistral_reasoning}.py`-- The **`message is None` guard** in `mistral_llm_worker.py` — see the correction below; it is 2.x-coupled and must travel with the bump.--**Separability verified, and the safe branch built ✅ (2026-06-03).** Two passes. First, a throwaway check: reverted the complete HOLD group to dev (mistralai → 1.12.4, temporalio → 1.23.0, instructor → PyPI, provider source + seven tests), re-synced, `agent-check` + `agent-test` green. Then the real deliverable: branch **`feature/Runtime-bridge-extraction`** off `dev` (soft-reset so the net diff is one staged changeset), reverted the HOLD group, kept `temporalio==1.24.0`, regenerated the lock off dev's (mistralai held at 1.12.4), and re-ran both gates → **`agent-check` 0 errors / 2007 files, `agent-test` all passed.** The safe part — `runtime_bridge` (zero `import mistralai`) + Temporal rewiring + `MISTRAL_NATIVE` deferred dispatch + additive docs/tests — is provably independent of the SDK bump and the fork pin.--**Two findings the build earned:**--1. The HOLD group is wider than the provider *source* — the seven provider *test* files above were migrated to the 2.x `mistralai.client.*` paths and fail `pyright` (`reportMissingImports`) under mistralai 1.x, so they must move with the bump too.-2. **The `message is None` guard is NOT SDK-independent** (my earlier claim was wrong). Under mistralai 1.x, `response.choices[0].message` is typed non-optional (`AssistantMessage`), so `pyright` rejects the guard as `reportUnnecessaryComparison`. It only becomes valid under 2.x, where `message` is `AssistantMessage | None`. So the guard **stays with the HOLD group** — do not cherry-pick it into the safe PR.--### Option B — merge whole, with a gate--Acceptable **only if `dev` is treated as a pure integration branch**: merge as-is, but add an explicit gate that **this must not reach `main` / a release while the `instructor` fork pin exists**, plus a tracking issue tied to instructor #2298. Given the pin is blocked on someone else's open PR with no ETA, prefer Option A.--## Housekeeping--- **`../../../TODOS.md` — refreshed 2026-06-03 (done).** Its "End state delivered" had predated the `runtime_bridge/primitives/` subpackage and the `MISTRAL_NATIVE` mode, and implied no mistral-missing exception existed (the old `MistralWorkflowsNotInstalledError` was deleted, but the new `MissingMistralWorkflowsPluginError` replaced it). The runtime_bridge end-state bullet + the Temporal-rewiring bullet were corrected, and a banner now points here as the merge basis. If `runtime_bridge`'s surface changes again, update both files.-- **`mistralai-workflows` companion version** lives in the separate `pipelex-mistralai-workflows` repo (TODOS notes `mistralai-workflows==3.4.0` there) — out of scope for this pipelex-side merge decision.--### Tracking issue for the `instructor` fork pin (to file)--The fork pin must not silently rot. In-repo anchors today: this doc + the `# Temporary: …` comment above `[tool.uv.sources] instructor` in `pyproject.toml` (which already references #2298). The durable external tracker is a GitHub issue — none exists yet (`gh issue list --search instructor` was empty on 2026-06-03). Draft to file against `Pipelex/pipelex`:--```-Title: chore(deps): drop temporary `instructor` git-fork pin once mistralai 2.x lands on PyPI--The `mistralai` 1.x→2.x bump (feature/Mistral-workflows-merge-4) needs `instructor` to-support mistralai v2, which is NOT yet released on PyPI. Stopgap in pyproject.toml:-- [tool.uv.sources]- instructor = { git = "https://github.com/Ian321/instructor.git",- rev = "4ea22d2396ca35514929f27faed866115e8b3583" }--`instructor` is a CORE dependency, so this pins the entire lockfile (uv.lock resolves-instructor 1.15.1 from that git rev). Published-package knock-on: `[tool.uv.sources]` is-not in the wheel metadata, so `pip install pipelex[mistralai]` from PyPI gets mistralai 2.x-+ PyPI-instructor (no 2.x support) → broken structured-output combo. Must not reach a-release while the pin exists.--Blocked on:-- instructor PR #2298 "support mistralai v2" — https://github.com/567-labs/instructor/pull/2298 (open as of 2026-06-03)-- instructor issue #2137 — https://github.com/567-labs/instructor/issues/2137--Removal steps (once a PyPI instructor release supports mistralai 2.x):-1. Delete the `[tool.uv.sources] instructor = {...}` block in pyproject.toml.-2. Raise the `instructor>=…` floor to the first release with mistralai-2.x support.-3. `uv lock`; confirm instructor resolves from PyPI (not the git rev) in uv.lock.-4. `make agent-check && make agent-test`; run a live Mistral inference + OCR round-trip.--Context: wip/mistral-workflows-merge-readiness.md-```--File it by saving the body above to a file and running `gh issue create --repo Pipelex/pipelex --title '…' --body-file <path>`. Once filed, add the issue number to the `# Temporary: …` comment in `pyproject.toml` so the pin and the tracker cross-reference.--## Cold-start: re-verify in a fresh session--```bash-cd /Users/lchoquel/repos/Pipelex/_workflows-git rev-parse --short HEAD # expect ff15bd58 (or later)-git merge-base HEAD dev | cut -c1-8 # dev tip; == merge-base means dev fully merged in-git diff --stat dev...HEAD -- 'pipelex/**/*.py' # the real net source diff (~26 files)-grep -n "instructor\|mistralai\|temporalio" pyproject.toml # the dep coupling-make agent-check # ruff/plxt/pyright/mypy-make agent-test # full non-inference suite-```--To confirm `MISTRAL_NATIVE` is still a no-op-without-the-package boundary: `pipelex/runtime_bridge/bridge.py::_run_mistral_native` should deferred-import `pipelex_mistralai_workflows` and raise `MissingMistralWorkflowsPluginError`.--## Sources--- instructor PR #2298 — <https://github.com/567-labs/instructor/pull/2298> (open, not merged as of 2026-06-03)-- instructor issue #2137 — <https://github.com/567-labs/instructor/issues/2137>-- mistralai on PyPI — <https://pypi.org/project/mistralai/>diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-plugin-extract.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-plugin-extract.mddeleted file mode 100644index fed612968..000000000--- a/wip/distributed-execution/mistral-workflows/mistral-workflows-plugin-extract.md+++ /dev/null@@ -1,215 +0,0 @@-# Mistral Workflows ↔ Pipelex — Extract as a Real Mistral Plugin--Self-contained plan. Sibling document `mistral-workflows-sub-module.md` (the-former `TODOS.md`) records what was built in-tree under-`pipelex/plugins/mistralai_workflows/` during Phases 1.x–2.1. That work is the-*input* to this project, not background to redo.--## 1. Why--What we shipped is the **functional equivalent** of a Mistral Workflows plugin-but packaged as an optional extra of `pipelex` (`pipelex[mistralai-workflows]`,-imported from `pipelex.plugins.mistralai_workflows.*`). Mistral defines a-plugin as a **standalone, pip-installable package** that depends on-`mistralai-workflows>=2.0.0` and exposes activities / workflows / dependencies-under its own top-level package (the `mistralai.workflows.plugins.*`-namespace is reserved for Mistral-supported packages).--To match that contract we need to:--1. Stop shipping Mistral-specific code inside the `pipelex` distribution.-2. Ship a separate distribution (PyPI: `pipelex-mistralai-workflows`,- Python package: `pipelex_mistralai_workflows`) that `pip install`s- `pipelex` and `mistralai-workflows>=3.3.0` and re-exports the same- activities, with a Mistral-style "component" / dependency wrapper.-3. Keep the framework-agnostic embedding core (`bridge.py`,- `execution_mode.py`, `bootstrap.py`, `exceptions.py`) reachable from- *any* host, not just Mistral — so other durable runtimes (raw Temporal,- future plugins) can reuse it.--## 2. Goals & non-goals--**Goals**--- Pure-Python `pipelex` distribution: no `mistralai-workflows` extra, no- `pipelex/plugins/mistralai_workflows/` directory.-- New repo `pipelex-mistralai-workflows` scaffolded from- `pipelex-starter-python` and adapted for a library (not an app), with- identical lint/type/test toolchain to the rest of the workspace.-- Public surface preserved: a user who today writes- `from pipelex.plugins.mistralai_workflows.activities import pipelex_run_pipe`- has a clear, mechanical migration to the new import path.-- Mistral-idiomatic ergonomics: a `pipelex_dependency` (or equivalent- component) so workers wire Pipelex the same way they wire- `mistralai_chat_complete` today.-- CI on the new repo runs the integration test layers that need the- optional dep (currently layer-2 / layer-3 in `mistral-workflows-sub-module.md` §3).-- Docs split cleanly: embedding-core docs stay in pipelex; Mistral-specific- recipes move to the new repo's docs.--**Non-goals**--- Re-doing Phase 1.x / Phase 2.x design. The behavior, boundary types,- execution modes, streaming semantics, and gotchas are all locked in —- see `mistral-workflows-sub-module.md` §2 and §4. Treat them as spec.-- Goal 1 from the original doc (porting Pipelex orchestration to run *on*- Mistral Workflows). Still out of scope.-- Backwards-compatibility shims in `pipelex` for the old import path. Per- project rule (CLAUDE.md "No backward compatibility"), we just change it- and note the migration in the changelog.--## 3. Decision points to resolve before coding--A follow-up agent should not start until these are answered. Each has a-proposed default; flag any that need a human call.--1. **Where does the framework-agnostic core live?**- - Option A *(recommended default)*: keep `bridge.py`,- `execution_mode.py`, `bootstrap.py`, `exceptions.py` inside `pipelex`,- promoted out of `plugins/mistralai_workflows/` into something like- `pipelex/embedding/` (name TBD). The new plugin pkg imports from there.- Pros: any host (raw Temporal, future plugins) can reuse it; smaller- surface to maintain in the new repo.- - Option B: move *everything* into `pipelex-mistralai-workflows`. Pros:- `pipelex` stays leaner. Cons: any future host has to either depend on- the Mistral plugin pkg (wrong) or re-implement.- - This decision drives the rest of the file moves.--2. **Package + distribution names.**- - PyPI: `pipelex-mistralai-workflows` (proposed). Confirm naming aligns- with other Pipelex packages on PyPI.- - Top-level Python package: `pipelex_mistralai_workflows`. The- `mistralai.workflows.plugins.*` namespace is reserved, so we cannot- squat there.- - GitHub repo: `pipelex-mistralai-workflows` under the existing org,- side-by-side with the other repos listed in the workspace `CLAUDE.md`.--3. **Versioning & release cadence.**- - Pin a minimum `pipelex` version per release. Decide whether to track- `pipelex` major versions 1:1 or use independent SemVer.- - Decide whether the new repo follows the same release skill / version- conventions as `pipelex` (CHANGELOG.md format, `release/vX.Y.Z`- branches, etc.). Default: yes, identical.--4. **Mistral component wrapper shape.**- - Mistral's existing plugins ship a "dependency" (e.g.- `mistralai_chat_complete`) that workers register. We should ship at- least one — likely a wrapper around `ensure_pipelex_booted()` plus a- `LibraryCrate` snapshot. Final shape needs a quick read of how- `mistralai.workflows.plugins.mistralai` exposes its dependency before- committing.--5. **Cookbook example.**- - The deferred Phase 1.3 cookbook entry (`pipelex-cookbook/examples/c_advanced/mistral-workflows/`)- should land *after* the new package is on PyPI, importing from the new- path. Coordinate timing.--## 4. Workstreams--Three streams, parallelizable once §3 is resolved. Each has its own-follow-up agent / PR.--### Stream A — Refactor inside `pipelex`--In this repo (`_mistral/`, eventually merged back). High-level only; the-follow-up agent figures out the file moves once §3.1 is decided.--- Lift the framework-agnostic core to its new home (Option A) or remove it- entirely (Option B).-- Delete the Mistral-specific modules (`activities.py`, `streaming.py`,- `streaming_event_forwarder.py`) and their unit + integration tests.-- Drop the `[mistralai-workflows]` extra from `pyproject.toml`. Drop the- `[[tool.mypy.overrides]]` block that exists only because Mistral's- source uses PEP 695 syntax (re-add it in the new repo).-- Move docs: `docs/under-the-hood/mistralai-workflows-{plugin,recipes}.md`- go to the new repo's docs site. Leave a stub in pipelex docs that links- out.-- CHANGELOG entry under `[Unreleased]` describing the move + migration.-- Verify: `make agent-check` and `make agent-test` green; `git grep- mistralai_workflows` returns nothing in `pipelex/` after the move.--### Stream B — Scaffold `pipelex-mistralai-workflows` from `pipelex-starter-python`--New repo, side-by-side with the other workspace repos.--- Copy `pipelex-starter-python/` as the starting point. Rename the package- dir, rewrite `pyproject.toml` (`name`, `description`, dependencies,- package list, classifiers).-- Convert from "app starter" to "library":- - Drop the `my_project/hello_world.{mthds,py}` example.- - Add a real `LICENSE` (MIT, matching pipelex).- - Replace the README with one that explains: install, register the- activity on a worker, call it from a workflow, link to recipes.- - Wire `py.typed` (already present in starter — keep).-- `pyproject.toml` deltas vs starter:- - `name = "pipelex-mistralai-workflows"`.- - `dependencies = ["pipelex>=X.Y", "mistralai-workflows>=3.3.0"]` (no- `[mistralai,anthropic,...]` extras — this is a library, not an app).- - Re-add the `[[tool.mypy.overrides]]` for `mistralai.workflows.*`- (PEP 695 source) — copy from pipelex `pyproject.toml`.- - Pytest markers: copy the relevant subset; drop `inference`/`llm`/etc.- if the test suite only does layer-2 / layer-3 worker tests.-- Add a Makefile mirroring the `agent-check` / `agent-test` / `cleanderived`- targets used elsewhere in the workspace, so CLAUDE.md instructions in- this repo and the new repo overlap.-- Add `CLAUDE.md` for the new repo. Short — point at workspace `CLAUDE.md`- and call out: must not depend on internal `pipelex` paths, only the- promoted public embedding core.-- GitHub Actions: copy the matrix from `pipelex` if there is one, or wire- a minimal `uv sync && make agent-check && make agent-test` workflow.--### Stream C — Move the plugin code into the new repo--After A and B land (or in a coordinated PR pair).--- Move `activities.py`, `streaming.py`, `streaming_event_forwarder.py`- into `pipelex_mistralai_workflows/`. Rewrite imports to point at the- promoted embedding core in `pipelex`.-- Move the integration tests (`test_activities_direct.py`,- `test_activities_offloaded.py`, `test_activities_streaming.py`,- `test_bridge_temporal_blocking.py`, `test_bridge_temporal_fire_and_forget.py`)- and their fixtures (`conftest.py`, `test_data/bridge_test.mthds`,- `test_data/bridge_funcs.py`).-- Layer-1 framework-agnostic tests (`test_bridge_direct.py`, the unit tests- under `tests/unit/pipelex/plugins/mistralai_workflows/`) follow the- embedding core — they stay in `pipelex` if we picked Option A, move if- Option B.-- Add the Mistral component / dependency wrapper from §3.4.-- First release to PyPI as `0.1.0`. Tag, write release notes, link from- the pipelex CHANGELOG migration entry.--## 5. Migration story for users--Single-paragraph note in pipelex CHANGELOG and in the new repo's README:--> Mistral Workflows integration moved from-> `pipelex[mistralai-workflows]` / `pipelex.plugins.mistralai_workflows.*`-> into a dedicated package: `pip install pipelex-mistralai-workflows` and-> import from `pipelex_mistralai_workflows.*`. No API changes; the-> activities, boundary types, and execution modes are identical.--Per project rule, no compat shim. The pipelex release that drops the-extra and the first `pipelex-mistralai-workflows` release ship together.--## 6. Open risks (track but don't block)--- **Version coupling.** The plugin pkg depends on internal-but-public- embedding APIs of `pipelex`. Decide on a stable surface (probably what- `bridge.py` already exposes) and document it as such, or breakage will- cascade on every pipelex release.-- **Test parity in CI.** Today layer-2 / layer-3 tests run on every PR in- pipelex CI because `uv sync --all-extras` pulls the optional dep. Once- extracted, those tests only run in the new repo. Make sure both repos'- CI matrices are healthy before flipping the switch.-- **OffloadableField import path drift.** Already noted in- `mistral-workflows-sub-module.md` §6. Carries over verbatim.--## 7. Resuming a session--1. Read `mistral-workflows-sub-module.md` §2, §4, §5 — these are the- binding design decisions and gotchas you must respect.-2. Read this file end-to-end.-3. Resolve §3 with the user before writing code. Especially §3.1 (where- the framework-agnostic core lives) — every other choice depends on it.-4. Pick a stream from §4. Streams A and B can run in parallel; Stream C- waits on both.diff --git a/wip/distributed-execution/mistral-workflows/mistral-workflows-sub-module.md b/wip/distributed-execution/mistral-workflows/mistral-workflows-sub-module.mddeleted file mode 100644index 3168bc18d..000000000--- a/wip/distributed-execution/mistral-workflows/mistral-workflows-sub-module.md+++ /dev/null@@ -1,301 +0,0 @@-# Mistral Workflows ↔ Pipelex Plugin — Plan & Progress--Self-contained planning document. A fresh session can resume from this file-alone; no need to read prior conversation history.--For anything implemented (Phases 1.0–1.5), the code is authoritative. Read-`pipelex/plugins/mistralai_workflows/` and the matching tests under-`tests/{unit,integration}/pipelex/plugins/mistralai_workflows/`. The-user-facing reference lives at-`docs/under-the-hood/mistralai-workflows-{plugin,recipes}.md`.-------## Status board--| Phase | Scope | Status |-| ----- | ------------------------------------------------------ | ------ |-| 1.0 | Framework-agnostic core (`bridge.py`, modes, ...) | ✅ done |-| 1.1 | Tier-1 `pipelex_run_pipe` activity wrapper | ✅ done |-| 1.2 | `TEMPORAL_BLOCKING` + `TEMPORAL_FIRE_AND_FORGET` modes | ✅ done |-| 1.3 | Docs + CHANGELOG (mkdocs nav wired) | ✅ done¹ |-| 1.5 | Large-payload `pipelex_run_pipe_offloaded` variant | ✅ done |-| 2.0 | Streaming v1 — one Mistral `task()` per activity | ✅ done |-| 2.1 | Streaming v2 — per-step `task.update()` (DIRECT only) | ✅ done |--¹ Cookbook example deferred to a follow-up PR in the sibling-`pipelex-cookbook/` repo (suggested entry:-`examples/c_advanced/mistral-workflows/` with a Tier-1 DIRECT-mode worker-plus a Tier-2 typed activity that uses `library_crate_dump`).--### Outstanding from Phase 1 (not blocking Phase 2)--- [ ] `make agent-check` passes with `mistralai-workflows` NOT installed —- needs a fresh venv without the extra. The guard placement- (`activities.py` only; `bridge.py` / `execution_mode.py` /- `bootstrap.py` / `exceptions.py` are import-clean) should make this- pass; not yet verified end-to-end.-- [ ] CI matrix for the optional dep — deferred. CI runs- `uv sync --all-extras`, so layer-2 and layer-3 tests already run on- every PR. Reconsider when we add an extra whose tests must NOT run- on the default lane.-------## 1. Context--Integrate Pipelex with Mistral Workflows (`mistralai-workflows>=3.3.0`),-which wraps Temporal with a thicker DX layer. Two goals were considered:--- **Goal 1** — Port Pipelex orchestration to run *on* Mistral Workflows as- an alternative durable runtime. Blocked on Mistral exposing extension- hooks for payload converter, codec, sandbox passthrough, and a- bare-Temporal `run_worker` mode. **Out of scope.** File issues with the- Mistral team if/when we want to revisit.-- **Goal 2** — Let users invoke Pipelex pipes from inside their own- Mistral Workflows activities. **In scope.** Phase 1.x complete; Phase 2- pending.--The `mistralai-workflows` dependency must remain strictly optional.-------## 2. Locked-in design decisions (still binding for Phase 2)--- **Library crate transport** is dump-based:- `PipelexPipeRunInput.library_crate_dump: dict[str, Any] | None`.-- **Execution mode** is an exhaustive `StrEnum`, set per-call (not- per-worker): `DIRECT`, `TEMPORAL_BLOCKING`, `TEMPORAL_FIRE_AND_FORGET`.-- **Streaming** is phased. Phase 2.0 ships a single Mistral `task()` per- activity (started / completed / failed). Phase 2.1 ships per-step- `task.update()` events driven by Pipelex's `report_delegate`,- conditional on demand.-- **Plugin location**: `pipelex/plugins/mistralai_workflows/`.-- **Optional-dep guard** lives in `activities.py` (and will live in- `streaming.py`). `__init__.py` is empty per Pipelex's "no re-exports"- rule. `bridge.py`, `execution_mode.py`, `bootstrap.py`, `exceptions.py`- are framework-agnostic and importable without `mistralai-workflows`.-- **Boundary types are JSON-only.** No Pipelex internal types (`PipeJob`,- `PipeOutput`, `WorkingMemory`) cross the activity boundary.-------## 3. Module layout--```-pipelex/plugins/mistralai_workflows/-├── __init__.py # empty (no re-exports)-├── exceptions.py # 4 exception classes-├── execution_mode.py # PipelexExecutionMode StrEnum-├── bridge.py # framework-agnostic core (NO mistralai imports)-├── bootstrap.py # ensure_pipelex_booted() + DI helper-├── activities.py # Tier-1 wrappers + offloaded variant-└── streaming.py # Phase 2.0 — pipelex_run_pipe_streaming sibling activity--tests/unit/pipelex/plugins/mistralai_workflows/-├── test_input_models.py # boundary BaseModels-├── test_execution_mode.py # StrEnum properties-├── test_validation.py # _validate_input + decode helpers-└── test_dispatch.py # mode dispatch with mocked PipeRun--tests/integration/pipelex/plugins/mistralai_workflows/-├── conftest.py # bridge_test_library fixture-├── test_data/ # bridge_test.mthds + bridge_funcs.py-├── test_bridge_direct.py # layer 1 (no optional dep)-├── test_activities_direct.py # layer 2 (Mistral test worker)-├── test_activities_offloaded.py # layer 2 — offloaded variant-├── test_activities_streaming.py # layer 2 — Phase 2.0 streaming variant-├── test_bridge_temporal_blocking.py # layer 3 (+ temporal extra)-└── test_bridge_temporal_fire_and_forget.py # layer 3 (+ temporal extra)-```--Users import from full paths (no re-exports per Pipelex rule):--```python-from pipelex.plugins.mistralai_workflows.activities import (- pipelex_run_pipe,- pipelex_run_pipe_offloaded,- PipelexPipeRunInputOffloaded,- PipelexPipeRunOutputOffloaded,-)-from pipelex.plugins.mistralai_workflows.bridge import (- PipelexPipeRunInput, PipelexPipeRunOutput, run_pipe_via_bridge,-)-from pipelex.plugins.mistralai_workflows.execution_mode import PipelexExecutionMode-from pipelex.plugins.mistralai_workflows.bootstrap import ensure_pipelex_booted-from pipelex.plugins.mistralai_workflows.streaming import (- pipelex_run_pipe_streaming,- PipelexPipeRunStreamingState,- PIPELEX_PIPE_RUN_TASK_TYPE,-)-```-------## 4. Gotchas from Phase 1 (read before Phase 2)--These bit during Phase 1.1–1.2 and will likely bite again when adding-`streaming.py` plus a new integration test module. Save the cycles.--1. **Workflow sandbox rejects Pipelex imports** seen during workflow- class validation. Wrap them in- `mistralai_workflows.workflow.unsafe.imports_passed_through()` AND- pass `enforce_determinism=False` to `@workflow.define` for test- workflows. See `test_activities_direct.py:24-30` for the pattern.- Activities are unrestricted Python — this only matters inside- workflow class bodies.--2. **`OtelTraceId` search attribute** must be pre-registered on the- test namespace. Mistral's `@workflow.define` upserts it on every- workflow run, and the dev server rejects activations against an- unknown attribute. Pass- `search_attributes=[SearchAttributeKey.for_keyword("OtelTraceId")]`- to `WorkflowEnvironment.start_local`.--3. **Task queue mismatch.** Mistral's `@activity` wrapper schedules- activities on the *global* `mistralai_config.temporal.task_queue`- (default `"default"`), NOT the workflow's task queue. Spin a worker- on a custom test queue without overriding the global → activities- land on `"default"` and the workflow hangs forever. Fix: an autouse- module-scoped fixture pinning- `mistralai_config.temporal.task_queue = TEST_TASK_QUEUE` for the- duration of the test module. See `test_activities_direct.py:48-64`.--4. **Result shape.** Mistral's `convert_result_to_temporal_format`- wraps non-BaseModel return values in `{"result": ...}`. Return a- BaseModel directly from the workflow entrypoint to skip the- wrapping (we return `PipelexPipeRunOutput`).--5. **mypy + Mistral's PEP 695 type syntax.** `mistralai-workflows`'s- own source uses PEP 695 type parameters that mypy rejects under- `python_version=3.11`. The `[[tool.mypy.overrides]]` block for- `mistralai.workflows.*` (`follow_imports = "skip"`,- `ignore_errors = true`) in `pyproject.toml` is already in place.- Mypy will still flag `OffloadableField.get_value()` as returning- `Any` — assign through a typed intermediate variable.--6. **`PipeFunc` is NOT Temporal-compatible.** `asyncio.to_thread`- inside `PipeFunc` raises `NotImplementedError` in the deterministic- workflow event loop. The bundle ships `bridge_func_pipe` (PipeFunc,- DIRECT only) for the no-Temporal layers and `bridge_compose_pipe`- (PipeCompose, Temporal-compatible) for the Temporal layers; the- `bridge_envelope_pipe` (PipeCompose with inline-structured concept)- exists to exercise dynamic-concept round-trip via- `library_crate_dump`. Pick the right one per execution mode.--7. **`Pipelex.make()` is NOT idempotent** — it raises if a singleton- already exists. `bootstrap.ensure_pipelex_booted()` only calls it- when `Pipelex.get_optional_instance() is None`, so an externally-- booted singleton is adopted. Don't replace this guard with a- module-level boolean.-------## 5. Phase 2 — Streaming--Goal: surface live progress events from Pipelex pipes through Mistral's-`task()` event API so users can subscribe via-`create_capturing_mock_events_client` and friends.--### Phase 2.0 — One task per activity (started / completed / failed) ✅--A single Mistral task wraps the whole activity body — no per-step-granularity. Cheapest path to "the user sees something happen."--- [x] Created `pipelex/plugins/mistralai_workflows/streaming.py`. Reuses- the same optional-dep guard pattern as `activities.py` (top-level- `try`/`except ImportError` re-raising `MistralWorkflowsNotInstalledError`).-- [x] **Decision: sibling activity** `pipelex_run_pipe_streaming`. Reasons:- - Keeps the silent path silent — Tier-1 users without observability- needs don't pay event publishing overhead.- - Mirrors the `pipelex_run_pipe_offloaded` sibling pattern.- - Workers register only the variant they actually use.-- [x] Inside the activity: `async with Task[PipelexPipeRunStreamingState](...)`- then call `run_pipe_via_bridge`, then `update_state` to `phase="completed"`.- `Task.__aexit__` automatically emits `CustomTaskFailed` on exception- and the original exception propagates — no extra `try`/`except`- needed (per Pipelex "don't catch Exception speculatively" rule).-- [x] Layer-2 integration test- `tests/integration/pipelex/plugins/mistralai_workflows/test_activities_streaming.py`- uses `create_test_worker_with_events` + `EventContext` + a- `create_capturing_mock_events_client` to assert that exactly one- `CustomTaskStarted`, ≥1 `CustomTaskInProgress`, and one- `CustomTaskCompleted` are emitted with the right `custom_task_type`- (`pipelex.pipe_run`) and payload shape.--### Phase 2.1 — Per-step granularity (DIRECT mode only) ✅--- [x] Subscribe to Pipelex's trace event channel from inside the- activity. Implemented as a queue-backed `EventLogProtocol`- (`QueueEventLog` in `streaming_event_forwarder.py`) injected into a- per-call `GraphTracerManager` tracer. Note: `ReportingProtocol`- itself has no observer methods — the right abstraction is the- event log, not the reporting delegate.-- [x] Map Pipelex events to Mistral `task.update_state(...)` calls:- - `PipeStartEvent` → `in_progress` with `current_step_pipe_code`,- `current_step_node_id`, `last_event_kind="pipe_start"`,- `started_steps`.- - `PipeEndSuccessEvent` → `in_progress` with- `last_event_kind="pipe_end_success"`, `completed_steps`,- `last_output_stuff_name`.- - `PipeEndErrorEvent` → `in_progress` with- `last_event_kind="pipe_end_error"`. The activity's- `Task.__aexit__` then emits `CustomTaskFailed` with the- propagated exception.- - Other `TraceEventKind`s (edges, batch fan-out, controller- outputs, execution data, usage reports) intentionally suppressed- — too noisy for state updates, already captured by Pipelex's- own reporting / graph infrastructure.-- [x] Forwarder side-task drains the event log and terminates cleanly- on both success and failure. Uses `try` / `finally` (no- `except Exception`); the forwarder is fully drained *before*- writing the final `phase="completed"` state so the snapshot- reflects the right phase.-- [x] Multi-step test fixture: `bridge_sequence_pipe` (PipeSequence)- chaining `bridge_seq_step_one` and `bridge_seq_step_two`- (PipeCompose). Test asserts ≥3 pipe_start + ≥3 pipe_end_success- `CustomTaskInProgress` events with `started_steps` monotonic- `[1, 2, 3]` and per-step pipe codes in declaration order.--Scope: DIRECT mode only. TEMPORAL_BLOCKING / TEMPORAL_FIRE_AND_FORGET-keep Phase 2.0 single-pair semantics — per-step streaming across the-Temporal worker boundary would need cross-process tee logic on top of-the existing `pipeline_run_setup` event log infra, deferred until demand-surfaces.-------## 6. Open risks (track but don't block)--- [ ] Concurrent activities sharing process-global Pipelex state — already- mitigated by per-call library scoping in- `bridge.py::_scoped_library_for_crate`. Verify no leakage under- concurrent load when we get there.-- [ ] If Mistral upgrades change the `OffloadableField` import path- (currently `mistralai.workflows.core.encoding.fields_offloader`),- revisit the imports in `activities.py`.-------## 7. Resuming a session--1. Read this file end-to-end — it's short by design.-2. **For anything in Phase 1.x: the code is authoritative.** Read- `pipelex/plugins/mistralai_workflows/*.py` and its tests; do not- re-derive from this doc.-3. For Phase 2 work: start at §5, pick the first unchecked box, re-read- §4 (gotchas) before scaffolding the new test module.-4. After each phase, run:- ```- make agent-check- .venv/bin/pytest tests/unit/pipelex/plugins/mistralai_workflows/ \- tests/integration/pipelex/plugins/mistralai_workflows/- ```- Before declaring Phase 2 done, run the broader sweep:- ```- .venv/bin/pytest -n auto \- -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" \- tests/unit/pipelex/plugins/ tests/integration/pipelex/plugins/ \- tests/unit/pipelex/builder/ tests/integration/pipelex/builder/- ```-5. Update the status board and the §5 boxes as you go.diff --git a/wip/distributed-execution/nondeterminism-fix-review-follow-ups.md b/wip/distributed-execution/nondeterminism-fix-review-follow-ups.mddeleted file mode 100644index f7b3296d0..000000000--- a/wip/distributed-execution/nondeterminism-fix-review-follow-ups.md+++ /dev/null@@ -1,116 +0,0 @@-# Nondeterminism fixes (H1/M1/H2) — review follow-ups--> **BURN-DOWN COMPLETE.** Every item below reached a terminal state — see the **Status** line under each heading. Summary: 0/0bis/1/2/3/4/5/6/8/9/10/11 FIXED (with one sub-point of 3 dismissed); 7 SURFACED to [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md).--Source: code reviews of branch `fix/Config`. Round 1 reviewed the committed H1/M1 fixes (flush activity and tracing setup made pure functions of payload + history); its H2-related findings were handled by the H2 fix session. Round 2 (below, at the end) reviewed the H2 fix itself — its top finding is the priority item in this doc.--Companion diagnosis: [workflow-nondeterminism-audit.md](workflow-nondeterminism-audit.md).--## PRIORITY — from the H2-fix review (round 2)--### 0. Key per-run worker-local state by `run_id`, not `workflow_id` — `wf_pipe_router.py`--**This is a correctness hazard, not a cleanup.** The per-workflow library (`wf_{workflow_id}`), the tracer key, and the report-delegate event-log context are all keyed by `workflow.info().workflow_id`. The H2 fix guarantees an evicted predecessor instance's `finally` now runs its synchronous cleanup (verified against temporalio source: eviction resumes the coroutine, the `finally` executes, and the new flush command raises `_WorkflowBeingEvictedError` at the await). That reliable cleanup is destructive cross-run: workflow id X run A is cached on worker W; A is closed server-side (workflow-level `retry_policy` in `workflow_caller.py` reuses the id — as do Temporal reset and resubmission of the same `pipeline_run_id` via the deterministic `make_workflow_id`); successor run B starts on W and `open_fresh_library` opens fresh `wf_X`; the worker later processes A's eviction, and A's `finally` tears down B's LIVE library, pops B's tracer, and clears B's event-log context. B then fails inline (registry fallback / `LibraryError` from its own teardown → workflow-task retry loop). Pre-fix only the tracer pop was reachable at eviction; the reorder widened the blast radius to the library and usage-event context. The root defect (workflow_id-keyed worker-local state) predates the H2 fix.--Fix: key the per-run worker-local state by `workflow.info().run_id` (replay-stable, unique per run, new on retry/reset/continue-as-new) — e.g. `wf_{run_id}` for the library id and run-scoped tracer/event-log context keys. `open_fresh_library` stays as the same-run self-heal (eviction leak of run A heals when run A itself replays), and cross-run collisions become structurally impossible. Also make `open_fresh_library`'s check-then-teardown tolerant of concurrent teardown (the membership check and the teardown are not atomic across workflow threads). The existing eviction-leak test covers the leak-then-rerun direction only; add the late-eviction-destroys-successor direction if feasible.--**Status: FIXED.** All per-run worker-local state in `wf_pipe_router.py` is keyed by `wf_run_id = workflow.info().run_id` (library `wf_{run_id}`, tracer key, event-log context key), and the run-scoped identity is also what gets stamped into events/node ids (the runner fallback already stamped `tracer_key`, so stamps and keys stay one identity). `open_fresh_library` is now pop-based (`LibraryManager._pop_and_teardown_library`): the membership check IS the atomic pop, so concurrent teardown resolves to a clean miss. Tests: new `test_wf_pipe_router_run_scoped_state_keying.py` (parks the workflow mid-activity and asserts library/tracer/context exist under run-id keys and NOT under workflow-id keys — RED against the old keying); the eviction-leak test converted to start-then-install so the leaked state sits under the actual `wf_{run_id}`. The destroys-successor direction is pinned structurally by the keying test (no workflow-id-keyed state exists to destroy).--### 0bis. Pin the finally-ordering invariant with a cancellation test--The ordering half of the H2 fix ("no await before worker-local cleanup in the `finally`") is enforced only by comments; the regression test pins the `open_fresh_library` self-heal, not the ordering. Deterministic guard shape (verified constructible with existing infra): `substitute_activities` with a flush stub blocked on an `asyncio.Event` (pattern already in `test_wf_pipe_router_tracing_config_nondeterminism.py`), start the workflow, `handle.cancel()` while it sits at the flush await (`CancelledError` is the same BaseException-at-suspension-point shape as eviction), then assert the `LibraryManager` holds no per-run library and the report-delegate context is clear. Land this before or with item 5 below, which plans to edit the flush schedule.--**Status: FIXED (test landed), with one premise corrected empirically.** Cancellation is NOT the same shape as eviction: workflow cancellation delivered at an activity await is surfaced by the SDK as an `ActivityError` (an `Exception`), which the best-effort `except` around the flush swallows — the workflow then completes normally, so cancellation cannot abort the `finally` (good news: it cannot leak either). The guard test instead forces a real eviction: park the workflow at the blocked flush, shut the worker down (shutdown evicts every cached run, resuming the coroutine with the eviction `BaseException` at the await), then assert library/tracer/context are all clean. Verified RED against a deliberately reordered `finally` (cleanup moved after the await). Test: `test_wf_pipe_router_eviction_cleanup_ordering.py`. The router's `finally` comment was updated to drop the incorrect cancellation claim.--## Do anytime (independent of H2)--### 1. Fix the stale runner-fallback warning — `pipelex/tracing/activity_event_log.py`--`warn_once_runner_fallback_engaged` (~line 93) still says the fallback engages "because no `_event_log_contexts` entry was registered in this process" and "is expected when activities run on a separate worker pool from the workflow router". After the H1 fix, the runner fallback is the **universal** path for all activity-side usage emissions (`reporting_manager.py` routes every in-activity emission there, co-located or not), so this WARNING now fires once per worker process on every standard co-located deployment with a false explanation, and it no longer signals anything anomalous about split deployments.--Fix: rewrite the message to describe the actual post-H1 contract (activity emissions always use the per-process runner log, by design, for replay determinism), and consider downgrading to DEBUG/INFO since it is now the normal path — or keep a WARNING only for the genuinely anomalous case if one still exists.--**Status: FIXED.** Renamed to `log_once_runner_fallback_engaged`, downgraded to INFO, message now states the post-H1 contract (activities always emit through the per-process log, by design, co-located or split alike). No anomalous case remains distinguishable at that call site, so no WARNING arm. Module docstring of `activity_event_log.py` rewritten to match; the two unit tests updated to assert the INFO once-flag.--### 2. Remove the dead `_event_log_contexts.clear()` dance — `tests/integration/pipelex/temporal/tracing/helpers.py`--`_runner_isolated_act_llm_gen_text` (~line 220) clears `ReportingManager._event_log_contexts` (with `SLF001`/`reportPrivateUsage` suppressions) to force the fallback path, and its docstring explains that mechanism. Post-fix, `_emit_usage_event` checks `_is_in_temporal_activity()` **before** the contexts lookup, so the clear no longer changes which path is taken — it is dead code with a stale docstring, and the private-state mutation can interfere with other contexts registered in the same process.--Fix: delete the clear + the isinstance scaffolding, rewrite the docstring to say the fallback is now taken unconditionally in activities. This also collapses the helper toward the duplicated stub in item 3.--**Status: FIXED.** Clear + isinstance scaffolding deleted, docstring rewritten (helper renamed `_synthetic_usage_act_llm_gen_text`). The same dead clear existed in `test_split_worker_real_inference_cost.py::_real_runner_act_llm_gen_text` — removed there too, with its docstring and the module docstring corrected.--### 3. Deduplicate the new test scaffolding into shared helpers--The two new regression tests (`test_wf_pipe_router_costs_only_flush_nondeterminism.py`, `test_wf_pipe_router_tracing_config_nondeterminism.py`) plus the eviction test carry copy-pasted blocks, all verified line-for-line:--- the synthetic `LLMTokensUsage`/`LLMJob`/`report_inference_job` construction duplicates `_runner_isolated_act_llm_gen_text` in `tracing/helpers.py` (only the constant strings differ, and after item 2 the bodies become identical modulo constants) → extract a `make_synthetic_usage_llm_job(...)` builder;-- an identical `_act_flush_noop` `@activity.defn(name="act_flush_trace_events")` stub in both new files → move one copy into `tracing/helpers.py`;-- an identical scheduled-activity-names proto scan over fetched history in both new files (a near-variant also pre-exists in `tracing/test_split_worker_extract_pages.py`) → a `scheduled_activity_names(history)` helper;-- the eviction test's Greeting-stuff/`prepare_for_temporal` setup duplicates `library_crate/test_wf_deferred_hydration.py` down to the same payload → a shared builder in `library_crate` helpers.--Also: the new files re-implement the enable/restore-tracing fixture from `tracing/conftest.py` because they sit one directory up — hoisting a parametrizable fixture (or moving the tests under `tracing/`) removes that copy too.--**Status: FIXED**, except two sub-points dismissed on re-verification:--- Extracted `make_synthetic_usage_llm_job(...)`, a shared `act_flush_noop` stub, and `scheduled_activity_names(history)` into `tracing/helpers.py`; all router-level guard tests now consume them. Extracted `make_prepared_greeting_job(...)` into `library_crate/helpers.py`, consumed by both the eviction-leak test and `test_wf_deferred_hydration.py`.-- DISMISSED (fixture hoist): only ONE copy of the enable/restore-tracing fixture exists (`test_wf_pipe_router_tracing_config_nondeterminism.py::enable_tracing_restored`) — the costs-only test never enables tracing (its flush is stubbed and post-H1 the schedule doesn't read config). One user = nothing to dedup; moving the tests under `tracing/` would also subject them to that directory's `gha_disabled` CI skip, which would be a regression.-- DISMISSED (extract-pages scan): the "near-variant" in `tracing/test_split_worker_extract_pages.py` scans `(activity_type.name, activity_id)` pairs and `start_to_close_timeout`s, not bare names — different shape, not servable by `scheduled_activity_names` without contortion.--### 4. `open_tracer` stale-eviction re-implements `close_tracer` — `pipelex/graph/graph_tracer_manager.py`--The stale-tracer path (~line 131) inlines `pop` + `teardown()`, which is exactly `close_tracer`'s body. Delegate to keep one eviction path. **Caveat for the refactor:** gate the "evicting stale tracer" warning on `key in self._tracers` *before* delegating — `close_tracer`'s return value is ambiguous (`None` both when no tracer existed and when a costs-only tracer legitimately tears down without a `GraphSpec`).--**Status: FIXED** exactly as specified (membership-gated warning, then `self.close_tracer(key)`).--## Do with / after the H2 session (same code, avoid conflicts)--> The H2 fix has landed (eviction-safe `finally` ordering + `open_fresh_library`), so these are now unblocked — the `finally` block has its final shape.--These touch the exact `finally` block the H2 fix restructured; they were deferred to avoid churn while that fix was pending.--### 5. Skip the guaranteed-empty flush activity in costs-only LIVE runs — `wf_pipe_router.py`--Post-H1, the workflow's `BufferingEventLog` is deterministically empty in costs-only LIVE mode (the tracer gets `event_log=None`; all activity emissions go to the per-process fallback), yet every `WfPipeRouter` — including every child workflow spawned per sub-pipe by the controllers — schedules `act_flush_trace_events` with an empty payload: a wasted activity round-trip (history events, task-queue dispatch, worker slot) per workflow in the fan-out. The H1 rationale only forbids gating on **buffer content** (activity-fed, hence nondeterministic); a payload-pure gate is fine. Gate the schedule on `trace_context.emit_graph_events or run-mode-is-DRY` — both ride in the payload, so the decision replays identically.--**Status: FIXED.** Determinism argument re-derived independently before trusting it: (1) gate inputs (`emit_graph_events`, `pipe_run_params.run_mode`) are pure payload, identical on replay; (2) in costs-only LIVE the buffer's only possible feeders — inline tracer graph events and inline dry-run usage reporting — are both absent, and every LIVE usage emission is activity-side (always routed to the per-process fallback post-H1), so the skipped flush provably loses nothing; (3) Temporal is unshipped, so no in-flight histories exist to diverge against. Implemented as a `schedule_flush` bool computed in the tracing setup block (where `trace_context` is narrowed) and consumed in the `finally`. The costs-only regression test's vacuity guard inverted accordingly: the costs-only arm now asserts the flush is ABSENT from history and still replays cleanly; the graph arm keeps the original presence guard.--### 6. Trim the redundant null conjuncts in the `finally` block — `wf_pipe_router.py`--Verified redundant given the tracing setup block's atomicity: `and wf_tracer_key is not None` (~line 200), `and trace_context is not None` (~line 203), and the nested `if wf_tracer_key is not None` (~line 233) can never differ from their enclosing conditions. **Do not** go further and collapse the sentinels onto `trace_context is not None`: the sentinels are load-bearing — library-crate setup earlier in the `try` can raise before the tracing block runs, leaving them `None` while `trace_context` is set, and a `trace_context`-gated `finally` would then crash on `event_log.drain()` and unbound `wf_workflow_id`, masking the original error.--**Status: FIXED** (folded into the item 5/9 rewrite of the block). The tracer-close gate is now the single sentinel `wf_tracer_key is not None` (the `wf_graph_tracer_manager` variable is gone — the singleton getter is called in the finally, pure in-memory); the spec-assignment gate is `graph_spec is not None and pipe_output is not None` (costs-only returns `None` by `close_tracer`'s contract, noted in the comment); the nested `wf_tracer_key` re-check is gone (`clear_event_log` takes the always-bound `wf_run_id`, now assigned before the `try`). Sentinels (`wf_library_id`, `event_log`, `wf_tracer_key`) kept, as required.--## Smaller items from the H2-fix review (round 2)--### 8. Document `set_event_log`'s overwrite as load-bearing — `pipelex/reporting/reporting_manager.py`--The leaked-predecessor self-heal now exists as three uncoordinated idioms: `open_fresh_library` (explicit, WARNING), `open_tracer` pop-and-replace (explicit, WARNING), and `set_event_log`'s bare dict assignment (incidental, silent, undocumented). The silent overwrite is genuinely load-bearing: a leaked `_event_log_contexts` entry under the deterministic workflow id remains possible post-H2 (deadlock-detector thread abandonment skips `finally` entirely; worker kill between set and clear), and the overwrite is what heals it. A future hardening pass that adds "context already exists → raise" collision detection — exactly what pre-M1 `open_tracer` did — would reintroduce the eviction-poison class on the cost path with no test catching it. Fix: a comment on `set_event_log` stating the overwrite-on-existing-key is intentional and load-bearing for leaked-predecessor healing; ideally also a shared contract note ("worker-local state keyed by a deterministic workflow id must be self-healing on open/set") tying the three idioms together.--**Status: FIXED.** Comment added on the assignment in `set_event_log`, stating the shared contract and naming the two sibling idioms. (With item 0's run-id keying the leak window is same-run only, but the healing contract stands unchanged.)--### 9. Simplify the `finally` to one event-log block — `wf_pipe_router.py`--Verified equivalent and strictly simpler: move the library-teardown block BEFORE the event-log block, then keep `drain → close → clear_event_log → await flush` together under one `if event_log is not None:` with the await last inside it. Library teardown and event-log cleanup are mutually independent (teardown touches only `LibraryManager` dicts; `clear_event_log` is a bare `dict.pop`; nothing between drain and flush suspends), and the eviction-safety invariant is identical — all synchronous cleanup still precedes the only await. Deletes the `buffered_events` sentinel, the second gate, the TYPE_CHECKING-only `TraceEvent` import, and resolves the comment tension where "the flush activity is scheduled unconditionally" sits directly above a conditional. Natural to fold into item 5/6 since they edit the same block.--**Status: FIXED** (one rewrite together with 5 and 6). `buffered_events` sentinel, second gate, and the TYPE_CHECKING `TraceEvent` import all gone; the eviction-ordering guard test (item 0bis) pins the resulting invariant and was verified RED against a reordered block.--### 10. Extract a shared scoped-library helper for the two uuid-keyed copies — `runtime_bridge`--The seed-registry → open → `set_class_registry` → set-current → `load_from_crate` → teardown-in-finally ceremony exists in three hand-rolled copies. Verified: the two `runtime_bridge` copies (`bridge._scoped_library_for_crate`, `submitter_hydration.rehydrate_pipe_output_with_crate`) use per-call uuid4 ids, so they structurally cannot hit the leaked-predecessor collision and correctly do NOT need `open_fresh_library` — this is consolidation-for-maintenance only. Extract one shared async context manager for those two; leave `wf_pipe_router` bespoke — its teardown sequencing relative to the flush await IS the H2 fix, and folding it into a `with`-helper would fight that ordering.--**Status: FIXED.** One shared SYNC context manager `scoped_library_for_crate(library_crate, library_id_prefix)` in `pipelex/runtime_bridge/primitives/scoped_library.py` (the bridge's was only nominally async — `# noqa: RUF029` — so going sync removed the noqa too). Both copies replaced; id prefixes preserved (`runtime_bridge_*`, `rehydrate_*`); `wf_pipe_router` left bespoke as required, with the why documented on the helper.--### 11. One-line hardening: make `LibraryManager.teardown` forget the entry even if `library.teardown()` raises--Today all sub-teardowns are plain dict reassignments that provably cannot raise, so this is unreachable — but "teardown never raises" is an accident, not a contract. If a future teardown does real work (closing resources, a model gaining `validate_assignment`), a raising teardown would leave the entry in `_libraries` and turn `open_fresh_library` into a deterministic re-raise on every rerun of that id on that worker (permanent worker-local poison). Pop the entry first, or `del` in a `finally`.--**Status: FIXED** (folded into item 0's `_pop_and_teardown_library`): the entry is popped before `library.teardown()` runs and the associated maps are cleared in a `finally`. Unit tests in `tests/unit/pipelex/libraries/test_library_manager_teardown_forgets_entry.py` (RED first): a raising teardown still forgets the entry, and `open_fresh_library` recovers afterwards.--## Deferred design note (no action yet)--### 7. Graph events have no equivalent of the H1 usage-path guard--H1's fix enforces "only workflow-thread emissions land in the workflow buffer" for **usage** events only (via `ReportingManager`'s in-activity check). `GraphTracerManager` is imported in `wf_pipe_router.py` under `workflow.unsafe.imports_passed_through()`, so the singleton **is** shared across the sandbox boundary on co-located workers, and the workflow's tracer key rides into activity args via `job_metadata.trace_context`. No current activity-side code calls tracer hooks, so this is latent — but the first activity that does (bridge instrumentation, future activity-side tracing) would mutate the workflow's tracer from outside the workflow thread, corrupting replay-rebuilt graph data (the H1 mechanism on the graph path; post-fix it corrupts data payloads rather than the command stream, so it is data corruption, not TMPRL1100). The structural fix is the same request-scoped tracing-state direction already tracked as T3 in [tracing-cost-reporting.md](tracing-cost-reporting.md); until then, a guard or an explicit "activities must never call tracer hooks" contract note on the tracer manager would prevent the regression.--**Status: SURFACED** → [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md) (options + recommendation; ride along with T3).diff --git a/wip/distributed-execution/nondeterminism-follow-ups-decisions-needed.md b/wip/distributed-execution/nondeterminism-follow-ups-decisions-needed.mddeleted file mode 100644index 7bfd23c3b..000000000--- a/wip/distributed-execution/nondeterminism-follow-ups-decisions-needed.md+++ /dev/null@@ -1,59 +0,0 @@-# Nondeterminism follow-ups — decisions needed--Items from [nondeterminism-fix-review-follow-ups.md](nondeterminism-fix-review-follow-ups.md) that are real but a judgment call: design tradeoffs with several defensible shapes, surfaced here for discussion instead of being fixed unilaterally. Everything fixable in that doc has been fixed (statuses recorded there).--## 1. Graph events have no equivalent of the H1 usage-path guard (follow-ups item 7)--H1's fix enforces "only workflow-thread emissions land in the workflow buffer" for **usage** events only: `ReportingManager._emit_usage_event` checks `_is_in_temporal_activity()` before any context lookup, so an in-activity emission can never write into the workflow's replay-rebuilt `BufferingEventLog`.--The **graph** path has no such guard. `GraphTracerManager` is imported in `wf_pipe_router.py` under `workflow.unsafe.imports_passed_through()`, so the singleton IS shared across the sandbox boundary on co-located workers, and the workflow's tracer key rides into activity args via `job_metadata.trace_context`. No current activity-side code calls tracer hooks, so the hazard is latent — but the first activity that does (bridge instrumentation, future activity-side tracing) would mutate the workflow's tracer from outside the workflow thread, corrupting replay-rebuilt graph data. Post-H1/H2 this corrupts **data payloads** (the assembled GraphSpec), not the command stream, so it is silent data corruption rather than a [TMPRL1100] failure — harder to notice, not easier.--### Options--- **A. Structural fix (T3 direction):** request-scoped tracing state instead of the process-singleton `GraphTracerManager` — already tracked as Issue T3 in [tracing-cost-reporting.md](tracing-cost-reporting.md). Solves the class, not just this instance. Largest change; touches every tracer lookup site.-- **B. H1-style guard on the tracer hooks:** make the tracer-hook entry points check `_is_in_temporal_activity()` and refuse (or no-op + WARNING) in-activity calls against a workflow-keyed tracer. Small, targeted, mirrors the proven usage-path mechanism — but adds temporal-awareness to `pipelex/graph/`, which is otherwise temporal-free.-- **C. Contract note only:** an explicit "activities must never call tracer hooks" note on `GraphTracerManager`. Zero code risk, but nothing enforces it — exactly the kind of cross-file invariant that rots.--### Recommendation--B as the cheap interim guard if any activity-side tracing work is planned before T3 lands; otherwise C now and fold the real fix into T3. A is the right end state either way — this item should ride along with T3 rather than spawn a separate effort.--> Status: no action taken in the fix/Config burn-down (carried here from the follow-ups doc, which marked it "no action yet"). Decide when T3 is scheduled.-------The items below were surfaced by the pre-landing review army on PR #987 (gstack `/review`: specialists + adversarial passes). Same rule as above: real, but judgment calls — deferred rather than fixed unilaterally. The review's clear wins (full-teardown forget-even-on-raise hardening with every sibling teardown still attempted, stale-key healing unit tests, best-effort containment of the stale tracer's teardown inside the healing branch, conftest cache-reset hoist, doc/changelog corrections, finally-block key symmetry + discarded-buffer tripwire) were applied directly on the branch and are NOT listed here.--## 2. Tracing-disabled workers silently drop submitter-requested trace/cost events--The worker-local `tracing_config.is_enabled` check deliberately lives activity-side (deterministic-safe), but both drop sites are silent: `flush_trace_events_to_backend` bare-returns on a NON-empty event list when tracing is disabled, and `_emit_usage_event_runner_fallback` bare-returns before building the event log — post-H1 the path for every LIVE usage emission. A submitter that requested tracing (payload carries `trace_context` with emit flags) against a tracing-disabled worker fleet gets a green workflow and an empty event stream: costs undercounted, no operator signal. This is intentional for deliberately-disabled fleets, which is exactly why a warning is a judgment call — it would fire once per flush/process on every such deployment.--**Options:** (a) one-shot-per-process WARNING at each drop site when the payload requested emission; (b) a config-drift metric/health signal instead of logs; (c) keep silent (current), treat disabled-worker fleets as fully intentional. Leans (a) — cheap and aligned with "a run's terminal reporting must not silently vanish" — but decide alongside the broader config-drift observability story.--## 3. Non-`PipelexError` raised in the now-unguarded tracing-setup block retries forever--The catch-all around tracing setup was removed on purpose (M1: worker-local state must not decide setup success; failures must surface). But the workflow's inline fail-safe floor catches `PipelexError` only — a non-PipelexError from the setup block (e.g. a pydantic `ValidationError` from the `model_copy` calls after a refactor) escapes both except clauses, fails the workflow task, and Temporal retries it forever: the silent-hang failure mode this whole series eliminates, reintroduced for one error class. Being inline-deterministic it re-fires identically on every retry. **Option:** widen the inline fail-safe floor (or wrap just the setup block) to convert any inline exception deterministically into a terminal `TemporalError`. This is one instance of the already-open "should deterministic converter/workflow-task errors be fail-fast instead of retry-forever" question (see the dry-validate hardening follow-up) — decide them together.--## 4. Residual worker-local-state risks under run-id keying (follow-ups item 0 residuals)--Two accepted-but-undocumented residuals of the run-id rekeying, plus one perf note, recorded so they're findable when symptoms appear:--- **Permanent leak on finally-skipping paths:** run ids are unique, so state leaked by a path that skips the `finally` entirely (e.g. deadlock-detector thread abandonment) is never reclaimed by the overwrite/self-heal idioms that workflow-id reuse used to trigger. Unbounded growth of `LibraryManager._libraries` / tracers / event-log contexts over a long-lived worker's life, on rare paths. Mitigation candidates: TTL sweep, size-bounded eviction, worker-eviction callbacks.-- **Zombie abandoned thread vs the same run's replay:** if an abandoned (deadlock-detector) thread later unblocks, its `finally` tears down `wf_{run_id}` state while a retried replay of the SAME run (same run id) is live on the same worker — the replay then fails inline with a `LibraryError`, which the fail-safe floor converts to a terminal failure. Low probability (abandonment + later resume + timing window); structural fix would be epoch/ownership tokens on the per-run entries.-- **Blocking backend write per usage emission on the activity event loop** (`_emit_best_effort` → e.g. boto3 `put_item`): pre-existing on the split path, now universal post-H1. Dwarfed by inference latency today; offload to a thread or batch if emission frequency grows.--## 5. `LibraryManager._pipe_source_map` is a shared keyspace across concurrent libraries (pre-existing)--`load_from_crate` writes `_pipe_source_map[pipe_ref] = source` keyed by bare `pipe_ref`, and teardown pops the dying library's refs — semantics unchanged by this branch (verified against the pre-branch code) but newly load-bearing now that per-run libraries are the norm: two concurrent runs loading the same crate share keys, so run A's teardown deletes source attribution run B still needs, and loads are last-writer-wins. Blast radius is error-traceability diagnostics (`get_pipe_source`), not execution. **Fix shape:** scope the map per `library_id` (nested dict or move it onto `Library`) — touches `get_pipe_source` callers, so it didn't belong in this PR.--## 6. `_is_in_temporal_activity()` is a process-global signal, not a pipelex-run boundary (contract note)--Two latent edges of the H1 routing check: (a) a host app running a DIRECT pipelex pipeline inside its **own** temporalio activity (legit runtime-bridge embedding) gets its registered direct-mode context bypassed — emissions divert to the per-process fallback with a restamped `workflow_id`, ignoring a custom event-log backend; (b) an emission from a thread spawned inside an activity without contextvar propagation (the `run_in_executor(None, ...)` pattern that exists in `bedrock_client_boto3.py`) would report `in_activity()=False` and resurrect the H1 cross-thread buffer write. All current `report_inference_job` call sites run in the activity's own context, so both are latent **within this repo** — same class as item 1; fold into the same T3-adjacent decision. **Update 2026-06-11:** edge (a) is NOT latent for `pipelex-mistralai-workflows` — Mistral Workflows wraps temporalio activities, so the plugin's registered `BufferingEventLog` usage path is bypassed by this gate on any pipelex release containing PR #987. Recorded as breakage item 7 in [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md); fix it there, not here.--## 7. `PipelineManager` keying/lifecycle and the `/pipeline/start` submitter-state gap--Deferred from the pre-merge resubmission fix ([nondeterminism-pre-merge-fixes.md](nondeterminism-pre-merge-fixes.md), fix 1). What landed: a tolerant `remove_pipeline` called from `execute_pipeline`'s `finally` and from `pipeline_run_setup`'s own failure path, plus a 409 mapping in `pipelex-api`. What was deliberately NOT decided:--- **Should `PipelineManager` adopt run-unique keying / self-heal like the three patched siblings (library, tracer, event-log context)?** Today its collision raise is load-bearing: it fires at the top of `pipeline_run_setup`, before `open_tracer`, and is the only thing preventing two concurrent same-id direct-mode runs from cross-popping each other's LIVE tracer via `open_tracer`'s stale-key pop-and-replace healing (the tracer key in direct mode is the caller-suppliable `pipeline_run_id`, not a run-unique Temporal run id). Any move to self-heal/overwrite must simultaneously gate that healing to run-scoped keys or make direct-mode tracer keys run-unique. Note the registry is otherwise write-only — `get_pipeline`/`get_optional_pipeline` have no production callers — so the long-term question is really "what is this registry for".-- **The fire-and-forget `/pipeline/start` path has no submitter-side terminal hook.** `ApiRunner.start_pipeline` (pipelex-api) calls `pipeline_run_setup` and hands off to Temporal; nothing in the submitter process runs when the workflow later completes, so the manager entry (and any submitter-side state keyed by the run id) stays registered for the process lifetime. Consequence after the pre-merge fix: serial resubmission of a completed run's id works on the `execute` path, but on the `start` path it still collides — now a clean 409 instead of a 500, but still process-permanent for that id. Deciding where the start path's terminal cleanup belongs (post-handoff removal? completion webhook? TTL sweep alongside item 4's leak mitigation?) interacts with the tracer-shield coupling above and with what submitter-side state the start path actually needs once the workflow owns delivery.-diff --git a/wip/distributed-execution/nondeterminism-pre-merge-fixes.md b/wip/distributed-execution/nondeterminism-pre-merge-fixes.mddeleted file mode 100644index 8bfa5ab3e..000000000--- a/wip/distributed-execution/nondeterminism-pre-merge-fixes.md+++ /dev/null@@ -1,101 +0,0 @@-# PR #987 pre-merge fixes — PipelineManager resubmission 500 + eager temporalio import--> **STATUS: BOTH FIXES IMPLEMENTED (2026-06-11).** As built:->-> - **Fix 1:** `remove_pipeline` (tolerant pop) added to `PipelineManagerAbstract` + `PipelineManager`; called from `execute_pipeline`'s `finally` (after `close_tracer` — tracer-shield ordering) AND from `pipeline_run_setup`'s own failure path. The setup-side removal goes beyond the original fix shape: the runner can't clean setup failures (it never learns the id when setup raises), and the window between `add_new_pipeline` and the old `try` (i.e. `acquire_library`) also stranded the entry — the setup `try` now starts before `acquire_library` with a `library_acquired` guard preserving the old teardown ownership split. Concurrent-duplicate raise kept (load-bearing tracer shield), pinned by unit test. Tests: `tests/unit/pipelex/pipeline/test_pipeline_manager.py`, `tests/integration/pipelex/pipeline/test_pipeline_run_id_resubmission.py` (red-first).-> - **Fix 1 caveat:** the fire-and-forget `/pipeline/start` path (pipelex-api `ApiRunner.start_pipeline`) has no submitter-side terminal hook, so a started run's id stays registered for the submitter process lifetime — resubmission there now gets a clean 409 instead of a 500, but is still process-permanent. Recorded with the deeper keying/lifecycle question as item 7 in [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md).-> - **409 mapping:** landed in `pipelex-api` on branch `fix/pipeline-run-id-conflict-409` (override entry + route-level tests + changelog + `docs/error-responses.md` status-code section, which also gained the previously-undocumented 501 override). Uncommitted in that repo's working tree, agent-check + full suite green.-> - **Fix 2:** module-level temporalio import replaced by a `sys.modules` sniff in `_is_in_temporal_activity`; `bridge.py`'s lazy-import docstring claim is true again (no edit needed there). Tests: `tests/unit/pipelex/reporting/test_temporal_activity_gate_lazy_import.py` incl. subprocess boot-cost guard (red-first); temporal integration True-branch suites green.->-> Original brief below, kept for context.--> **Cold-start brief.** Two findings from the pre-merge xhigh review of PR #987 (`fix/Config`, "fix(temporal): workflow determinism") that should be fixed before merge. Everything else the review surfaced is either already captured in [nondeterminism-follow-ups-decisions-needed.md](nondeterminism-follow-ups-decisions-needed.md), recorded as breakage item 7 in [bridge-changes-sibling-repo-reconciliation.md](bridge-changes-sibling-repo-reconciliation.md) (the `pipelex-mistralai-workflows` reporting-path change), or is low-priority cleanup. Both fixes below were adversarially verified against the working tree on 2026-06-11 — line numbers were exact then; re-verify before editing, they drift.->-> Repo norms apply: red-first TDD for behavior changes, `make agent-check` clean, CHANGELOG `[Unreleased]` entries, no new bare `except Exception`.--## Fix 1 — `PipelineManager` makes resubmission of the same `pipeline_run_id` a process-permanent 500--### The bug--`PipelineManager` is the one per-run registry PR #987 did NOT make self-healing, and unlike the three it patched (library / tracer / event-log context), it has **no per-run removal at all**:--- `pipelex/pipeline/pipeline_manager.py:40-46` — `add_new_pipeline` raises `PipelineManagerAlreadyExistsError` when `pipeline_run_id` is already in `self.root`. This is the exact pre-M1 raise-on-collision shape the PR removed from `GraphTracerManager.open_tracer`.-- `pipelex/pipeline/pipeline_manager.py:20-21` — the only cleanup is whole-process `teardown()` (`self.root.clear()`). Grep confirms no `remove`/`pop`/per-run forget anywhere; the abstract has only setup/teardown/get/add.-- `pipelex/pipeline/runner.py:206` — the runner's `finally` meticulously closes the tracer, clears the event-log context, and tears down the per-run library, but **never touches the PipelineManager entry**. Every run permanently strands its key.--### Trigger (real today, not latent)--`pipelex/pipeline/pipeline_run_setup.py:141` calls `add_new_pipeline(pipe_code=pipe_code, pipeline_run_id=pipeline_run_id)` **before anything else** (before `acquire_library` at ~153 and `open_tracer` at ~205). The hosted runner API threads a **client-supplied** id into it: `pipelex-api/api/routes/pipelex/pipeline.py:83-99` passes `pipeline_run_id` from the request (`extras.pipeline_run_id`, populated externally, e.g. by an API gateway). So: two POSTs to the start endpoint with the same `pipeline_run_id` against the same long-lived server process → the second raises `PipelineManagerAlreadyExistsError` → `pipelex-api/api/exception_handlers.py` has **no** AlreadyExists/409 mapping → **unhandled 500** — and since the key is never removed, every retry of that id 500s until process restart.--This directly contradicts PR #987's own changelog, which names "resubmission of the same `pipeline_run_id`" as a real, supported scenario (it's one of the motivations for the run-id rekeying). Worker-side retry/reset/replay does NOT re-enter `add_new_pipeline` (it's submitter-side), which is why the PR's worker-side fixes don't cover it.--Aggravating detail: the registry is effectively write-only — `get_pipeline`/`get_optional_pipeline` have zero production callers in `pipelex/` or `pipelex-api/`. Today the dict exists to mint ids and raise on collision.--### ⚠️ Coupling — do NOT just make it self-healing--The raise at `pipeline_manager.py:41` is currently the **only** thing shielding direct mode from a second PR-#987 finding: `GraphTracerManager.open_tracer`'s new pop-and-replace healing (`pipelex/graph/graph_tracer_manager.py:139-149`) assumes "an existing key can only be a stale leftover", which is proven only for run-unique Temporal `run_id` keys. In direct mode the tracer key is the caller-suppliable `pipeline_run_id` (`pipeline_run_setup.py` ~205 opens with `tracer_key=None`, so `key = pipeline_run_id`). Two **concurrent** runs with the same id would have run 2 silently pop run 1's LIVE tracer (cross-wired graphs, then run 1's `finally` `close_tracer` pops run 2's tracer) — today that can't happen only because `add_new_pipeline` raises first, at line 141, before `open_tracer` is reached. A bare overwrite-on-collision in `add_new_pipeline` would unshield that path.--### Recommended fix shape--Keep the collision raise for **concurrent** duplicates (it's load-bearing, see above); fix the **process-permanence** and the API surface:--1. **Per-run removal in the runner's `finally`** (`pipelex/pipeline/runner.py:206` block): add a `remove_pipeline(pipeline_run_id)` (tolerant pop) to `PipelineManagerAbstract` + `PipelineManager` and call it alongside the existing tracer/event-log/library cleanup. Serial resubmission of a completed/failed run id then works; only genuinely concurrent same-id runs still collide — which is the correct, loud behavior given the tracer coupling.-2. **Map the collision to a 4xx in `pipelex-api`**: add `PipelineManagerAlreadyExistsError` → 409 in `pipelex-api/api/exception_handlers.py` so a true concurrent duplicate is a client-visible conflict, not a 500. (This is a `pipelex-api` change — per workspace rules it lands in that repo, with its own changelog entry.)-3. Leave the deeper question — whether `PipelineManager` should adopt run-unique keying / self-heal like the three patched siblings, and whether `open_tracer`'s healing should be gated to run-scoped keys — to the decisions doc; don't solve it in this PR. If you DO decide to self-heal here instead, you must simultaneously harden `open_tracer` (heal only run-scoped keys, or make direct-mode keys run-unique at registration).--### Tests (red-first)--- Unit (`tests/unit/pipelex/pipeline/`): resubmitting a `pipeline_run_id` after the previous run's cleanup ran → `add_new_pipeline` succeeds (RED before fix 1). A second `add_new_pipeline` while the first is still registered → still raises (pins the concurrent-duplicate guard and the tracer shield).-- Integration: a full `execute_pipeline` run followed by a second run with the same explicit `pipeline_run_id` succeeds end-to-end; the runner `finally` removes the entry even when the pipe fails (assert on the manager after a failing run).-- Note: removal must be tolerant (pop, not raise) — the entry may legitimately be absent if setup failed before line 141 committed it.--## Fix 2 — module-level `temporalio` import puts ~130ms + the Rust bridge on every pipelex boot--### The bug--`pipelex/reporting/reporting_manager.py:25-28`:--```python-try:- from temporalio import activity as _temporal_activity-except ImportError:- _temporal_activity = None # type: ignore[assignment]-```--`reporting_manager.py` is imported by `pipelex/pipelex.py` (module import chain), i.e. on **every** `Pipelex.make()` / CLI invocation. Wherever the `pipelex[temporal]` extra is installed — including processes that never touch Temporal (CLI `validate` runs, direct-mode servers, the `pipelex-api` runner image) — this now pays the full temporalio import at boot: **measured ~131ms** in this venv (`.venv/bin/python -c 'import time;t=time.perf_counter();from temporalio import activity;print(time.perf_counter()-t)'`), plus the Rust-bridge/protobuf memory footprint. It also silently breaks the documented lazy-import contract: `pipelex/runtime_bridge/bridge.py`'s module docstring still says "The Temporal extra is lazy-imported only inside the temporal-mode branches" — now false.--### Recommended fix shape--Replace the eager import with a `sys.modules` sniff — strictly better than a lazy cached import, because a process that never imported temporalio **cannot** be inside a Temporal activity (the activity context is set by temporalio's own machinery, which requires the module to be imported):--```python-import sys--def _is_in_temporal_activity() -> bool:- """True when the current call stack runs inside a Temporal activity.-- sys.modules sniff instead of importing temporalio: if temporalio.activity was never- imported in this process, no activity context can exist, and importing it here would- put the entire temporalio extra (Rust bridge, protobuf) on every boot's critical path.- """- activity_module = sys.modules.get("temporalio.activity")- if activity_module is None:- return False- return activity_module.in_activity()-```--Delete the module-level try/except entirely (no top-level `temporalio` name should remain). Worker processes are unaffected: by the time any activity runs, `temporalio.activity` is necessarily in `sys.modules`. Keep the gate's behavior contract identical — the existing unit suite (`tests/unit/pipelex/reporting/test_emit_runner_fallback.py`) pins it.--### Tests (red-first)--- Unit: `_is_in_temporal_activity()` returns False when `temporalio.activity` is absent from `sys.modules` (monkeypatch `sys.modules` with the key removed) and delegates to `in_activity()` when present — RED while the module-level import exists (the monkeypatch test can't even simulate "not imported" today since module import already bound the name).-- Boot-cost guard (the actual regression): a subprocess check asserting `import pipelex.reporting.reporting_manager` does **not** pull `temporalio` into `sys.modules`. Cheap and pins the lazy-import contract the docstring promises. Suggested home: `tests/unit/pipelex/reporting/`.-- Existing temporal integration suite must stay green — the H1 routing tests (`test_wf_pipe_router_costs_only_flush_nondeterminism.py`, split-worker tracing tests) exercise the True branch for real.--## Acceptance--- [x] Both fixes implemented with their red-first tests; the concurrent-duplicate raise and the tracer-shield coupling explicitly pinned by a test.-- [x] `pipelex-api` 409 mapping landed in that repo (branch `fix/pipeline-run-id-conflict-409`, uncommitted working tree, suite green).-- [x] `pipelex/runtime_bridge/bridge.py` module docstring's lazy-import claim is true again.-- [x] CHANGELOG `[Unreleased]` entries for both (pipelex), plus pipelex-api's changelog for the 409 mapping.-- [x] `make agent-check` clean; targeted suites (unit reporting + pipeline, temporal integration) green along the way; full `make agent-test` green at the end.diff --git a/wip/distributed-execution/remote-deps-from-github.md b/wip/distributed-execution/remote-deps-from-github.mddeleted file mode 100644index b971d5b0f..000000000--- a/wip/distributed-execution/remote-deps-from-github.md+++ /dev/null@@ -1,19 +0,0 @@-# Remote Dependencies from GitHub--> Part of the [distributed-execution plan](README.md) (P3); a step toward [crate-first-architecture.md](crate-first-architecture.md). Builds on the blueprint collector from [local-cross-package-deps.md](local-cross-package-deps.md).--The goal: dependencies can be fetched from remote addresses (e.g. `github.com/org/repo/package`), so the crate becomes fully self-contained for cloud-native execution where workers are stateless.--## Key changes--1. **Remote resolution strategy**: Add a `REMOTE` resolution strategy to the blueprint collector. Remote fetch clones/downloads the package from GitHub, parses its `.mthds` files into blueprints, and includes them in the collection.--2. **Dependency address format**: Define the address format for remote deps (e.g., `github.com/org/repo@version/path/to/package`). The address goes in the package manifest.--3. **Caching**: Cache fetched packages locally (content-addressed by address + version) to avoid redundant clones.--4. **Transitive deps**: Remote packages may themselves have dependencies (local or remote). The collector recurses.--## Acceptance--A pipeline with a remote GitHub dependency executes on a Temporal worker that does **not** have the dependency pre-installed.diff --git a/wip/distributed-execution/schema-reconstruction-hardening.md b/wip/distributed-execution/schema-reconstruction-hardening.mddeleted file mode 100644index 63fc7f3e8..000000000--- a/wip/distributed-execution/schema-reconstruction-hardening.md+++ /dev/null@@ -1,23 +0,0 @@-# Schema Reconstruction Hardening--> **Status**: Deferred — no immediate threat-model trigger.--Items related to `SchemaToModelFactory` (`pipelex/cogt/content_generation/schema_to_model_factory.py`), which reconstructs Pydantic BaseModel classes from JSON schemas across process boundaries (Temporal payloads). The current implementation `exec()`'s code generated by `datamodel-code-generator` and ships that source verbatim as `__kajson_class_source__` so receivers can re-exec it. See the module docstring for the full threat model.--## `exec()` → `pydantic.create_model()` migration--**Status**: Deferred — non-trivial refactor, no immediate threat-model trigger.--Replace `exec()` in `_exec_source_to_types` with programmatic class construction via `pydantic.create_model()`. Eliminates the exec primitive on both the sender path (`make_from_json_schema`) and the receiver path (`make_types_from_source`), and lets us delete `_make_restricted_builtins`, `_restricted_import`, and `_reject_unsafe_schema_extensions`.--Non-trivial because nested models, enums, and forward references currently rely on `datamodel-code-generator`'s output structure. A faithful port has to walk the JSON schema directly, build the class graph in dependency order, and apply `model_rebuild()` after the fact to resolve forward refs.--This is the right structural fix; the receiver-side gap below is a stopgap until it lands.--## Receiver-side schema validation gap--**Status**: Known gap — documented in `make_types_from_source`'s docstring; only worth closing if the migration above is materially delayed.--`make_from_json_schema` (sender) calls `_reject_unsafe_schema_extensions` before generating source — Layer 1. `make_types_from_source` (receiver) takes pre-generated source and execs it directly — only Layer 2 (`_make_restricted_builtins`) applies. An attacker who can inject a crafted `__kajson_class_source__` string into a Temporal payload bypasses Layer 1 entirely.--Interim mitigation: AST-scan the source before exec, rejecting any top-level node that isn't an allowlisted import, a class def, or a docstring (and any class body that isn't fields / `model_config` / docstring). Mirrors Layer 1's "reject loudly" failure mode at the structural level. Trade-off: tied to `datamodel-code-generator`'s emitted output shape, which is not a stable API. Skip if the `create_model()` migration is near-term; the migration closes the gap permanently.diff --git a/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md b/wip/distributed-execution/temporal-e2e-validate-prompt-menu.mddeleted file mode 100644index 5cb668ece..000000000--- a/wip/distributed-execution/temporal-e2e-validate-prompt-menu.md+++ /dev/null@@ -1,149 +0,0 @@-# `/temporal-e2e-validate` — prompt menu--Operator cheat-sheet of prompts for the `/temporal-e2e-validate` skill. Sibling of [`temporal-e2e-validate-skill.md`](temporal-e2e-validate-skill.md) (what the skill validates + current-state record). The cost-reporting slice has its own deeper cheat-sheet: [`../runtime-bridge/cost-reporting-validation-prompts.md`](../runtime-bridge/cost-reporting-validation-prompts.md).--Convention below: `$` marks prompts that spend real money (live inference / live image-gen / live extract). Everything else is free — dry-run or `--mock-inference`. A few of the live tiers (4, 5, 12) also *run* in dry mode for structural coverage but only catch real payload / cost bugs when run live; those are marked `$ (dry-capable)`.--## Run it ALL in one go--### Complete sweep, real spend authorized ($)--```text-/temporal-e2e-validate run the COMPLETE sweep with real spend authorized — do NOT skip the opt-in batteries or the live arms. Mode 1 pytest dry then live. Mode 2 every tier: 1/2/2b/2c/2d/3 core, 4 & 5 live (image payload), Step 4 graph tracing, Step 5 concurrent isolation, Tier 8 usage emission, Tier 8b cost reporting FULL (arms A–G), Tier 9 object-gen, Tier 11 extract, Tier 12 CV batch live, Tiers 13–16 error propagation, Tiers 6/7 codec. Then Step 8 routing battery (10a–10c) and Step 9 queue-options + worker-runtime-profile battery (Scenarios A–F). Finish with the master results table — PASS/FAIL per tier/scenario — and stop-and-report on any failure.-```--### Complete sweep, no spend (full structural pass)--```text-/temporal-e2e-validate run the full sweep DRY/mock only — no real spend. Mode 1 dry, Mode 2 all tiers in dry/mock (incl. 2c and 2d — both free; Tier 8b cost reporting via --mock-inference, arms A–B), Step 5 isolation, Tiers 6/7 codec, plus Scenario C (pytest) and Scenario F (CLI typo). Skip the live-only arms: image payload (4/5), CV batch live (12), routing battery (10a–10c), queue-options live scenarios (A/B/D/E), and the live cost arms (E–G). Master results table at the end.-```--Recommended order: run the **no-spend** sweep first to confirm the plumbing cheaply, then the **real-spend** sweep once it's green. The all-in-one is long and runs tiers sequentially (reporting after each); if a live backend isn't configured, that arm stops-and-reports rather than silently passing — see Prerequisites.--## By area--### Broad (skill's own recognized triggers)--```text-/temporal-e2e-validate # full regression; offers the two batteries as opt-in extras-/temporal-e2e-validate full temporal test # same, explicit-```--### Mode 1 — fast pytest (real server, in-process worker)--```text-/temporal-e2e-validate run the Mode 1 pytest suite, dry # library_crate: crate, hydration, isolation, controllers-/temporal-e2e-validate Mode 1 pytest, live LLM # $ same suite with real inference-```--### Core distributed execution (Mode 2, free)--```text-/temporal-e2e-validate validate core distributed execution — LibraryCrate, deferred hydration, parallel child workflows-/temporal-e2e-validate validate concurrent isolation (concept / pipe / multi-concept) # Step 5-/temporal-e2e-validate validate cross-process registry # Tier 2b-```--### Validation sweep stays in-process — Temporal-leak guard (free)--Guards the `/validate` dry-run leak (nested controller sub-pipes dispatching to Temporal → HTTP 422 on a standalone `PipeBatch`). Two layers, both free; the contract is "the validation sweep never dispatches to Temporal, even under a Temporal-enabled boot".--```text-/temporal-e2e-validate validate the dry-run sweep stays in-process under a Temporal-enabled boot # Mode 2 Tier 2c-```--- **Mode-1 pytest** (real `TemporalPipeRouter` as hub default, spies `WorkflowExecutor.execute_workflow`, asserts never called). It lives *outside* `library_crate/`, so the generic "Mode 1 pytest suite" prompt above does **not** pick it up — run it by path:-- ```text- timeout 180 .venv/bin/pytest tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py -m temporal --temporal-server local --timeout=60- ```-- (Self-contained — GREEN never dispatches, so no live server is actually required; `--temporal-server none` works too.)--- **Mode-2 Tier 2c** (deployment-faithful: `validate bundle --temporal` over a standalone `PipeBatch` → exit 0 **and** worker idle / no `WfPipeRouter` dispatch). Reached via the prompt above; full GREEN/RED procedure in `references/mode-2-tiers.md` Step 3, Tier 2c.--### Dry-run + validation as ONE in-memory activity — Tier 2d (free)--Sibling of Tier 2c: where 2c proves the *direct* `/validate` sweep never dispatches, 2d proves the *Temporal-dispatched* `/validate` runs the whole sweep **+** graph dry-run inside a single in-process activity (`wf_dry_validate` → `act_dry_validate`) — zero nested dispatch on the workflow history, in-memory tracing (no NDJSON/DDB), `{status map, GraphSpec}` in one round-trip, best-effort graph (`GRAPH: None` on a graph-arm failure, validation still OK). Needs split workers up (workflow lands on the router worker, the activity on the runner).--```text-/temporal-e2e-validate validate the Temporal-dispatched /validate runs as one in-process in-memory activity # Mode 2 Tier 2d-```--- **Submitter:** `.claude/skills/temporal-e2e-validate/scripts/submit_dry_validate.py` (dispatches the wrapper workflow, prints the status map, writes the GraphSpec JSON). Full GREEN/RED procedure + the API arm (real `../pipelex-api` route via TestClient) in `references/mode-2-tiers.md` Step 3, Tier 2d. RED note: the event-log-drop arm signals `GRAPH: None` where GREEN produced a real graph (it no longer exits 1 — the activity's parity best-effort catch degrades it); the content-generator arm's deterministic RED is the Mode-1 leaf-mock test below.-- **Mode-1 pytest companions** (also outside `library_crate/`, so run them by path — the generic "Mode 1 pytest suite" prompt does not pick them up):-- ```text- timeout 180 .venv/bin/pytest tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py tests/integration/pipelex/temporal/test_dry_run_graph_in_process.py -m temporal --timeout=120- ```-- (Self-contained against the in-process server — `--temporal-server none` default works; covers zero-nested-dispatch on the Temporal history, in-memory tracing, best-effort + bug-propagates arms, no-retry pin, ErrorReport crossing, concurrency.)--### Graph tracing (free)--```text-/temporal-e2e-validate validate cross-worker graph tracing / GraphSpec assembly # Step 4-```--### Images ($ — dry-capable, but run live to catch payload bugs)--```text-/temporal-e2e-validate image temporal # Tiers 4–5: image gen + image-to-LLM flow, payload storage-```--### Object generation / extract / batch--```text-/temporal-e2e-validate validate object generation cross-process # Tier 9, free-/temporal-e2e-validate validate make_extract_pages two-activity cross-process # Tier 11, free-/temporal-e2e-validate validate the CV batch deeply-nested controller stack # Tier 12, $ (dry-capable)-```--### Error propagation--```text-/temporal-e2e-validate validate error handling # Mode 1 Step 2b + Tiers 13–16-/temporal-e2e-validate error report propagation across the activity → workflow → submitter boundary-```--Covers LLM, extract, image-gen, and fanned-out child-workflow (PipeBatch) failures carrying a structured `ErrorReport`. Free — failures are mocked or forced via a bad-credential worker (401), no successful inference billed.--### Payload codec (free)--```text-/temporal-e2e-validate validate the StoragePayloadCodec — transparency + large-payload stress # Tiers 6–7-```--### Routing — v1 `activity_queues` ($)--```text-/temporal-e2e-validate routing # Tiers 10a–10c: multi-activity isolation, per-handle routing, two-activities-one-route-```--### Queue options + worker-runtime profiles — v2 (mixed)--```text-/temporal-e2e-validate queue options # Scenarios A–F-/temporal-e2e-validate runtime profile # named worker-runtime profiles via --profile-```--Scenarios: A multi-class routing, B per-queue timeout, D queue rate-limit, E missing-worker negative (all live `$`); C per-handle override (pytest, free); F the `--task-queue` CLI typo check with "did you mean?" (CLI startup, free).--### Cost reporting--```text-/temporal-e2e-validate cost reporting # free: mock arms A–B-/temporal-e2e-validate cost reporting full # $ every arm A–G-```--Full detail (arm-by-arm assertions, aliases, ask-first behavior) in [`../runtime-bridge/cost-reporting-validation-prompts.md`](../runtime-bridge/cost-reporting-validation-prompts.md).--## Recognized triggers vs descriptive prompts--Some prompts hit **explicit routes** in the skill (`full temporal test`, `validate error handling`, `cost report`, `queue options` / `runtime profile` / `routing`, `image temporal`). The per-tier ones (Tier 9, 11, 12, codec, graph tracing…) are not separate routes — the agent reads `references/mode-2-tiers.md` and runs the named tier, so a descriptive prompt naming the concern is enough.--## Prerequisites--- `tmux` and the `temporal` CLI installed; the Temporal dev server reachable (the skill starts one if needed).-- Mode 2 stands up split router+runner worker processes (see `references/mode-2-setup.md`).-- Live arms need credentials: LLM inference (or the Pipelex Gateway), real image-gen, and extract = Azure Doc Intel via the gateway. Missing a backend makes that arm fail at the **run** step, not the assertion — the skill reports which arm and why.diff --git a/wip/distributed-execution/temporal-e2e-validate-skill.md b/wip/distributed-execution/temporal-e2e-validate-skill.mddeleted file mode 100644index eae0fedc0..000000000--- a/wip/distributed-execution/temporal-e2e-validate-skill.md+++ /dev/null@@ -1,49 +0,0 @@-# temporal-e2e-validate skill — validation status & Step 9 fixes--Current-state record for the `/temporal-e2e-validate` skill (`.claude/skills/temporal-e2e-validate/`), captured 2026-06-02 on branch `feature/Validate-with-signatures-4-fix-dry-run`. It exists to cold-start a follow-up session: what the skill validates, what a full run found, the Step 9 skill bugs that were patched, and the open follow-ups.--For ready-to-paste invocation prompts (per-area + run-it-all-in-one-go), see the sibling [`temporal-e2e-validate-prompt-menu.md`](temporal-e2e-validate-prompt-menu.md).--## What the skill is--`/temporal-e2e-validate` validates that Pipelex pipelines execute correctly across separate Temporal worker processes. Two modes:--- **Mode 1** — pytest against a real Temporal dev server with an in-process worker (`SKILL.md`). Fast regression path; folds in the error-handling suite (activity error boundary, workflow error-report full chain, local parity arm).-- **Mode 2** — true 3-process setup (server + separate worker process(es) + submitter). Reference files under `references/`: `mode-2-setup.md` (server + workers), `mode-2-tiers.md` (Tiers 1–16 plus the later-added 2c — direct `/validate` sweep stays in-process, PR #976 — and 2d — Temporal-dispatched `/validate` as one in-memory activity, added with `act_dry_validate`), `routing-battery.md` (Step 8, v1 `activity_queues` routing), `queue-options-battery.md` (Step 9, v2 queue options + worker-runtime profiles).--## Full-run result (2026-06-02)--Everything green. The branch's dry-run changes did not break the skill's dry path.--- **Mode 1 dry** — `library_crate/` all pass; error-handling Temporal-boundary + local-parity arms pass. The 4 tests the skill calls "Known xfails" (StuffArtefact serialization / xdist flakiness) now **xpass** — stale note, not a regression.-- **Mode 2 dry** — Tiers 1, 2, 2b (cross-process registry via `act_deliver`), 3, 4, 5, 9, 12, isolation A/B/C, codec 6/7: all pass.-- **Tiers 13–16 (live error propagation)** — bad-credential worker; LLM / extract / image-gen / batch-child failures each crossed activity→workflow→submitter carrying the real classified `ErrorReport` (401), not a generic wrapper.-- **Step 8 (v1 routing)** — Tiers 10a/10b/10c all pass.-- **Step 9 (v2 queue options)** — Scenarios A–F all pass, but only **after fixing the skill bugs below**.--## Step 9 skill bugs found & patched--All pre-existing (orphan-queue validator landed in `304c3552`, "Temporal merge 3", on `main`); none caused by the dry-run branch. All fixes are in `references/queue-options-battery.md`.--1. **Orphan `q_capped` blocks router boot.** Step 9.0's base override declared `[temporal.queue_options.q_capped]` (for Scenario D's rate cap) with no `activity_queues` route referencing it. The config validator rejects an unrouted queue_options entry ("the overlay will never apply") and the router refuses to boot — taking down all Step 9 workers. **Fix:** removed `q_capped` from the base override; Scenario D now declares it in its own override block (which also routes `act_llm_gen_text -> q_capped`).-2. **Missing general runner → `--graph` hangs.** Step 9.0 spawned only specialized runners (`runner-llm`/`runner-img-gen`/`runner-extract`), each polling its own named queue. Un-routed activities — tracing (`act_flush_trace_events`, `act_assemble_graph`) and `act_deliver` — fall through to `temporal_task_queue`, which nothing polled, so any `--graph` run (Scenarios A–B) hangs on the first trace flush (`PENDING_SCHEDULED` forever). Step 8 only worked because it keeps the base `runner` from `mode-2-setup.md`. **Fix:** Step 9.0 now also spawns a general `runner` on the default queue.-3. **Scenario C unit-test node ID.** Referenced `test_resolve_dispatch.py::test_handle_options_override_queue` without the class. **Fix:** corrected to `...::TestResolveDispatch::test_handle_options_override_queue` and added a runnable command.-4. **Scenario E jinja2 path inapplicable.** Scenario E routed `act_jinja2_gen_text` to an orphan queue, but `PipeJinja2` was renamed to `PipeCompose`, which renders templates **inline** via `render_template` → `render_jinja2_async`. No pipe dispatches `act_jinja2_gen_text` anymore; it can't be triggered from a bundle. **Fix:** Scenario E now routes `act_llm_gen_text -> q_orphan` and runs the stock `native_text_sequence` bundle (objective — bounded `schedule_to_start` timeout vs hang — is activity-agnostic).-5. **Scenario E timing expectation wrong.** The old expectation ("~35s, FAIL if > 60s") ignored that each pipe step runs as a child workflow that **retries (~3 attempts)** on failure, so total ≈ `maxAttempts × schedule_to_start`. The bound itself works (verified: scheduled event carries `scheduleToStartTimeout`, fails `TIMEOUT_TYPE_SCHEDULE_TO_START`). **Fix:** Scenario E now uses a small bound (10s) and the PASS criteria account for the retry multiplier.-6. **Scenario F sanity-check targeted an unknown queue and over-claimed boot.** The "a known queue passes" check ran `--task-queue q_llm --is-unit-testing`, but `q_llm` is **not** known at that point — Scenario E's `cat >` override (full replace) declares only `q_orphan`, so the known set is `{q_orphan, temporal_task_queue}` and `q_llm` is (correctly) rejected, deterministically failing the very assertion it was meant to prove. Separately, the check claimed the worker "should start up without raising," but the task-queue validator runs early (`worker_cli.py:75`, before `Worker(...)` construction) and `--is-unit-testing` then trips a sandbox-validation error during Worker construction — so a known queue does *not* boot cleanly under that flag. **Fix:** target `temporal_task_queue` (the always-known `default_task_queue`) and assert PASS = absence of `WorkerTaskQueueUnknownError` (proceeds past the validator), with a note that the subsequent `wf_test_structured_output_cross_process` sandbox error is expected and orthogonal.--Also fixed in the same file: Scenario D's measurement (it read a single workflow, but the `act_llm_gen_text` activities live in **child** workflows, and `jq`'s `fromdateiso8601` can't parse fractional-second timestamps) — replaced with a cross-child aggregation + Python timing math; and the "all scenarios A-D are live-only" intro (C is pytest, F is CLI).--## Open follow-ups (NOT done — left for a decision)--- **Vestigial `act_jinja2_gen_text`.** Only reachable via the test fixture `pipelex/temporal/test_extras/wf_test_content_generator_child.py` (and the dry stub); still registered in `pipelex/temporal/tasks.py` and the `runner-jinja2` scope (`pipelex/pipelex.toml:665`). No pipe dispatches it (PipeCompose renders inline). Candidate for removal: the activity + its `act_jinja2_generate.py` + the `runner-jinja2` scope + the fixture path. User chose to defer; not removed.-- **Stale "Known xfails" note in `SKILL.md`** (Mode 1). The StuffArtefact-serialization xfails now xpass — the note should be refreshed or the xfail markers dropped. Out of scope for the Step 9 fixes; not patched.-- **Worker can't boot under `--is-unit-testing` (product bug, not a skill issue).** Surfaced by the Scenario F sanity-check: a sandboxed worker started with `--is-unit-testing` fails temporalio's workflow-sandbox validation on the registered test workflows, so it never boots. Queue-name-independent and orthogonal to the queue-options surface — the task-queue validator (`worker_cli.py:75`) already passed before it fires. Tracked with full evidence and likely fix locus in [unit-testing-worker-sandbox-validation.md](unit-testing-worker-sandbox-validation.md). Not investigated or fixed here.--## Cold-start: re-verify the patched skill--Prereqs: `tmux`, `temporal` CLI, `temporalio` in `.venv`. Start the dev server if down: `tmux new-session -d -s temporal-server 'temporal server start-dev'`. Note: `[temporal.search_attributes].enabled = false` on this branch, so `pipelex setup-temporal-namespace` registers nothing and the worker boot audit is skipped.--To re-verify Step 9 end-to-end, follow `references/mode-2-setup.md` then `references/queue-options-battery.md` Step 9.0 → Scenarios A–F → Step 9.t. Step 9 is **live** for A/B/D/E (real LLM + image-gen + extract calls — costs money); C is pytest, F is CLI-only. Watch for the two failure modes the fixes address: router refusing to boot (orphan queue) and `--graph` runs hanging (missing general runner). Every command must run under a hard `timeout` per the skill's Timeouts policy.--The skill itself is not in `pytest` — verification is running it. A clean teardown leaves only `temporal-server` running, `.env` untampered (Tiers 13–16 tamper it transiently), and no `.pipelex/pipelex_temporary_override.toml`.diff --git a/wip/distributed-execution/temporal-exception-model-revamp.md b/wip/distributed-execution/temporal-exception-model-revamp.mddeleted file mode 100644index 45182c152..000000000--- a/wip/distributed-execution/temporal-exception-model-revamp.md+++ /dev/null@@ -1,62 +0,0 @@-# Temporal exception model — revamp proposal (deferred)--## Problem statement--Pipelex exposes workflow execution failures as `pipelex.temporal.exceptions.WorkflowExecutionError`, which extends `PipelexError(Exception)`. The `WorkflowExecutor` wrapper in `pipelex/temporal/tprl/workflow_caller.py` raises this from inside workflows when a child workflow fails (catches `ChildWorkflowError` → re-raises `WorkflowExecutionError(msg) from exc.cause`).--But Temporal's Python SDK has a strict rule: only exceptions that subclass `temporalio.exceptions.FailureError` are treated as **workflow execution failures** (terminal, propagated to client as `WorkflowFailureError`). Anything else is treated as a **workflow task failure** (programmer bug — workflow task is retried indefinitely, the workflow never completes).--`WorkflowExecutionError` doesn't subclass `FailureError`. So any workflow that catches it and re-raises it (the pattern in `wf_pipe_run.py`) would hang forever — except we worked around it by registering `WorkflowExecutionError` in the Worker's `workflow_failure_exception_types` list.--## What this revamp would do--Make `WorkflowExecutionError` (and likely other workflow-layer exceptions in `pipelex/temporal/exceptions.py`) inherit from `temporalio.exceptions.ApplicationError` (which is a `FailureError`). Then:--- `raise WorkflowExecutionError(msg)` inside a workflow → Temporal recognizes it natively → workflow ends terminally → client sees `WorkflowFailureError`. No Worker-side registration needed.-- The `workflow_failure_exception_types=[WorkflowExecutionError]` block in `pipelex/temporal/temporal_task_manager.py:make_worker` becomes redundant and can be removed.-- The "invisible discipline" of having to register every new domain exception type with the Worker goes away — the type system enforces the contract.--## Why we didn't do it during the pre-Phase-6 cleanup--- It expanded scope beyond the planned cleanup.-- `ApplicationError` has a different `__init__` signature than `PipelexError`. `ApplicationError(message, *details, type=None, non_retryable=False, ...)` vs. `PipelexError(message)`. Compatibility requires either:- - Overriding `__init__` on `WorkflowExecutionError` to call both parent constructors compatibly, or- - Accepting that callers must pass `(message,)` only and pinning that contract.-- It mixes Pipelex's exception hierarchy with temporalio's. The other `TemporalFlowError` subclasses (`WorkflowInputError`, `TemporalConfigError`, `TemporalServerError`, etc.) probably should NOT inherit from `ApplicationError` — only the ones that might be raised from inside a workflow. Drawing that line cleanly is the actual design work.-- The Worker-side registration was good enough to ship; this is the cleaner-but-not-strictly-necessary final form.--## Scope of the revamp--Likely changes:--- `pipelex/temporal/exceptions.py` — rework the hierarchy. At minimum `WorkflowExecutionError` inherits from `ApplicationError`. Possibly `ContentGenerationError` too if it's ever raised from inside a workflow.-- `pipelex/temporal/temporal_task_manager.py:make_worker` — drop the `workflow_failure_exception_types=[WorkflowExecutionError]` kwarg (no longer needed).-- `tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py` — drop the matching kwarg on the test Worker. The test should still pass.-- Audit any other Worker constructions outside `make_worker` (none today, but possible in future tests / scripts) — they no longer need the registration.-- `pipelex/temporal/tprl/workflow_caller.py` — verify the `raise WorkflowExecutionError(msg) from exc` and `raise WorkflowExecutionError(msg) from exc.cause` patterns still work with the new `__init__` signature.--Likely **not** changed:--- The non-workflow-layer exceptions in `pipelex/base_exceptions.py` — those have nothing to do with Temporal.-- The catch sites — they still catch `WorkflowExecutionError`, and that name doesn't change.--## Open questions for the revamp--- Should `TemporalFlowError` itself inherit from `ApplicationError`, or only `WorkflowExecutionError`? The former is bolder (any "Temporal flow error" gets Temporal-native handling) but pulls in any subclass we add later. The latter is surgical.-- `ApplicationError` has a `non_retryable` field. Should `WorkflowExecutionError` set it on construction (e.g. always non-retryable since the parent workflow's retry policy is what controls retry semantics)? Or leave it to callers? Pre-Phase-5 behavior was effectively retryable-by-default (since `ChildWorkflowError.cause` was usually retryable `ApplicationError`).-- Should `ApplicationError.type` be set to a fixed string (e.g. `"WorkflowExecutionError"`) for filtering on the Temporal dashboard? Currently the Rust core logs already show `r#type: "WorkflowExecutionError"` — confirm whether that's auto-derived from the class name or needs explicit setting.-- What happens to error attribution if `WorkflowExecutionError` inherits from `ApplicationError` and gets re-raised across multiple workflow layers? `ApplicationError` chains via `cause` (a Temporal-specific cause chain), not Python's `__cause__`. Need to verify the chain stays intelligible.--## Trigger to do this--- A second-time encounter of the "I forgot to add my new exception to `workflow_failure_exception_types`" footgun (any test that hangs forever with the same root cause).-- A new domain exception type that needs to behave the same way as `WorkflowExecutionError`.-- A Phase 7+ effort to clean up the Temporal layer's error model holistically.-- About 1-2 hours of focused work, plus reviewer time.--## Related work--- The `workflow_failure_exception_types` registration is the current workaround this revamp would remove.-- `pipelex/temporal/tprl/workflow_caller.py` — the four `WorkflowExecutor` entry points and their named-exception catches.-- `pipelex/temporal/exceptions.py` — the current `TemporalFlowError` hierarchy.-- `tests/integration/pipelex/temporal/test_wf_pipe_run_failure_path.py` — the test that surfaced this issue.diff --git a/wip/distributed-execution/temporal-ids-and-naming.md b/wip/distributed-execution/temporal-ids-and-naming.mddeleted file mode 100644index eaa742575..000000000--- a/wip/distributed-execution/temporal-ids-and-naming.md+++ /dev/null@@ -1,421 +0,0 @@-# Temporal IDs and Naming — Design--## Status--**Implemented.** This doc is the authoritative reference for the architectural decisions.-Decisions taken in this session, recorded so the next session does not reopen them:--- **Workflow ID.** Top-level Workflow ID is `{env_prefix}{pipeline_run_id}` — the existing UUID from `JobMetadata`, with the existing run-mode prefix (`ut-`, `ci-`, `cc-`, `cct-`, or empty). No session truncation, no random suffix, no class name.-- **Activity ID.** No customization. The Temporal Python SDK auto-assigns sequential integers; Pipelex never passes `activity_id=`. The worker-singleton LRU and `is_replaying()` short-circuit in `content_generator_in_workflow.py` are deleted.-- **Search attributes.** Five custom Keyword search attributes (`PipeCode`, `PipelineRunId`, `SessionId`, `UserId`, `DomainCode`) are registered once per namespace and set on every workflow start.-- **`wfid` parameter.** Dropped from `PipeRunProtocol`, `PipeRouterProtocol`, `ContentGeneratorProtocol`, all implementations, and every call site. Identity flows entirely via `JobMetadata.pipeline_run_id` and `pipe_job.pipe.code`; observability is auto-derived.--## Guiding principle--Pipelex already mints the right identifiers: `pipeline_run_id`, `pipe_code`, `domain_code`, `user_id`, `session_id`. The Temporal layer does not need to invent its own — it threads Pipelex's identity model through Temporal's primitives one-for-one. Anything that diverges from this principle (truncated session prefixes, random suffixes, class names embedded in the Workflow ID, an `activity_id` co-opted from a workflow-id parameter) is a defect to be removed, not a convention to be preserved.--The redesign maps Pipelex identity to Temporal identity on a single sheet:--| Pipelex concept | Temporal primitive |-|---|---|-| `pipeline_run_id` (UUID from `PipelineFactory`) | Workflow ID (top-level) |-| Implicit "child role" (a router under a run, a sub-pipe under a router) | Suffix appended to parent's Workflow ID |-| Pipe family | Workflow Type (registered: `wf_pipe_run`, `wf_pipe_router`) |-| Generation method (`make_llm_text`, …) | Activity Type (registered: `act_llm_gen_text`, …) |-| One call to a generator method inside a workflow | Activity ID (SDK-default integer) |-| `pipe_code` | Search attribute `PipeCode` + leading segment of `static_summary` |-| `domain_code` | Search attribute `DomainCode` |-| `user_id` | Search attribute `UserId` |-| `session_id` | Search attribute `SessionId` |-| Pipe description | Tail of `static_summary` |-| Per-call observability context (model handle, class name, etc.) | Per-activity `summary` |--No information is lost on the way down; no information is invented along the way.--## The four layers--### Layer 1 — Identity (the IDs)--#### Top-level Workflow ID--```-{env_prefix}{pipeline_run_id}-```--- `env_prefix` is selected from `runtime_manager.run_mode` (unchanged from today's `make_top_workflow_id`): `ut-`, `ci-`, `cc-`, `cct-`, or empty for `NORMAL`.-- `pipeline_run_id` is the existing UUID from `JobMetadata.pipeline_run_id`, set upstream by `PipelineFactory.make_pipeline_run_id`. The design treats it as an opaque string; callers who want a stable id pass one to `PipelineFactory.make_pipeline`.--Examples:--- `ut-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c`-- `ci-3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c`-- `3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c`--`WorkflowIDReusePolicy` stays at the SDK default (`ALLOW_DUPLICATE`). Pipelex semantics are "a new run is a new run": if a caller reuses a `pipeline_run_id`, Temporal accepts the new workflow execution under the same chain with a fresh `run_id`. Pipelex does not rely on Temporal-side ID uniqueness for anti-replay; that protection belongs upstream.--#### Child Workflow ID--```-{parent_workflow_id}/{suffix}-```--The separator is `/`, not `-`. UUIDs already contain `-`, and a different separator keeps the path structure unambiguous to humans and trivial to parse with `split("/")`.--Two suffix conventions, applied at the two existing construction sites:--- **Fixed-role child (1:1, known role).** Used by `wf_pipe_run` when it spawns `wf_pipe_router` as its sole child. Suffix is the role name: `pipe-router`. Example: `ut-3f9c…b3c/pipe-router`.-- **Dynamic child (router-spawned sub-pipe).** Used by `TemporalPipeRouter._run_pipe_job` inside a workflow when it dispatches another workflow for a sub-pipe. Suffix is `{pipe_code}-{disambiguator}`. Example: `ut-3f9c…b3c/pipe-router/translate_doc-7c1e2f8a`.--The disambiguator is `str(workflow.uuid4())[:8]` — replay-safe (Temporal's `workflow.uuid4()` is deterministic) and 32 bits is comfortably enough entropy for siblings of a single parent. The `pipe_code` prefix gives the path its semantic readability; the uuid suffix guarantees siblings cannot collide even when the same pipe is spawned multiple times by the same parent.--Nested paths look like `ut-3f9c…b3c/pipe-router/translate_doc-7c1e2f8a/extract_text-4d9b1c83/…`. At ~25 chars per level, the design comfortably stays under Temporal's 1000-character Workflow ID ceiling for deep nesting.--#### Activity ID--The `activity_id=` argument is **never passed** by Pipelex code. Temporal's Python SDK auto-assigns sequential integers (`"1"`, `"2"`, …) per workflow run. These ids are:--- Unique per `(workflow_id, run_id)` by construction.-- Deterministic across replay (assigned by history position).-- Visible in Event History next to the Activity Type and per-activity summary.--This removes the entire reason the worker-singleton LRU exists in `content_generator_in_workflow.py`. The TDD gate test in `test_default_activity_id_collision_bug.py` turns green because the two `make_llm_text` calls now receive activity_ids `"1"` and `"2"`, both unique.--#### Run ID--Server-assigned. Today's code uses `(workflow_id, run_id)` as a key on the LRU; after the LRU is deleted, Pipelex does not touch `run_id` except to read it for diagnostics.--### Layer 2 — Type / Name (registered)--#### Workflow Type--Unchanged. Two registered types:--- `wf_pipe_run` — top-level orchestrator that runs the router then delivery.-- `wf_pipe_router` — runs one pipe (controller or operator).--No dynamic registration of per-pipe workflow types. Filtering by pipe in the dashboard goes through the `PipeCode` search attribute, which is the right Temporal primitive for that need. The cost of registering one workflow type per pipe family (worker fleet consistency, registration logistics) outweighs the benefit when search attributes already give us a filterable column.--#### Activity Type--Unchanged. The registered activity functions (`act_llm_gen_text`, `act_llm_gen_object`, `act_llm_gen_object_list`, `act_img_gen_images`, `act_jinja2_gen_text`, `act_render_page_views`, `act_extract_gen_extract_pages`, `act_assemble_graph`, `act_flush_trace_events`, `act_deliver`) already carry the right semantic granularity. The bug was at the id layer, not the type layer.--### Layer 3 — Observability (display only)--These fields do not affect identity or routing. They are the operator-facing surface. Pipelex uses zero of them today.--#### Static Summary--Set once at top-level workflow start, on the submitter side, via the `static_summary=` argument to `client.start_workflow` / `client.execute_workflow`. 200-byte limit, single-line Markdown.--Format:--```-{pipe_code} — {description}-```--- `pipe_code` comes from `pipe_job.pipe.code`.-- `description` comes from `pipe_job.pipe.description` if present; otherwise the dash and tail are omitted.-- The combined string is truncated to fit 200 bytes (UTF-8), with a trailing `…` if truncated.--Example: `translate_doc — Translate a document from English to French`.--#### Static Details--Set once at top-level workflow start, alongside Static Summary. 20 KB limit, multi-line Markdown.--Contents (markdown table):--```markdown-| Field | Value |-|---|---|-| Pipe | `translate_doc` |-| Domain | `documents` |-| Pipeline run | `3f9c8b2a-1e4d-4f5b-9c7a-2d8e1f0a6b3c` |-| User | `acme-corp` |-| Session | `EdgdJ7Yk4Q3HF2pXyZv9w8` |-| Library crate | `[email protected]` |-| Input | `working_memory keys: source_text, target_language` |-```--Library crate version and the input-shape line are best-effort — emitted when available, omitted otherwise. No exception is raised if the optional fields cannot be derived; the section simply shrinks.--#### Current Details--Not used in v1. The architecture supports adding this later from inside `WfPipeRouter` (e.g. `workflow.set_current_details(f"Executing {controller}…")`) to surface in-flight progress in the dashboard. Tracked in **Out of scope / future work**.--#### Per-Activity Summary--Set on every `workflow.execute_activity(...)` call via `summary=`. 200-byte limit. This is the field that carries the per-call meaning that previously (incorrectly) lived in `activity_id`.--Format by method, where `{pipe_code}` is read from `job_metadata.pipe_code`:--| Method | Summary format |-|---|---|-| `make_llm_text` | `LLM text · pipe={pipe_code} · model={llm_handle}` |-| `make_object` | `LLM object · pipe={pipe_code} · class={class_name}` |-| `make_object_list` | `LLM object list · pipe={pipe_code} · class={class_name}` |-| `make_single_image` | `Img gen 1× · pipe={pipe_code} · model={img_gen_handle}` |-| `make_image_list` | `Img gen N× · pipe={pipe_code} · model={img_gen_handle} · n={nb_images}` |-| `make_templated_text` | `Templated text · pipe={pipe_code}` |-| `make_render_page_views` | `Render page views · pipe={pipe_code}` |-| `make_extract_pages` (extract step) | `Extract pages · pipe={pipe_code} · handle={extract_handle}` |-| `make_extract_pages` (render step) | `Render page views (extract) · pipe={pipe_code}` |--The platform-level activities (`act_assemble_graph`, `act_flush_trace_events`, `act_deliver`) also get summaries: `Assemble graph · pipeline_run_id=…`, `Flush trace events · n={count}`, `Deliver · pipeline_run_id=… · status={status}`.--If `pipe_code` is unavailable (test fixtures with no pipe), the format degrades gracefully to the method label alone.--### Layer 4 — Search & Filter (custom search attributes)--#### Required namespace registration--Custom Keyword-typed search attributes, registered once per Temporal namespace:--| Attribute | Type | Value source |-|---|---|---|-| `PipeCode` | Keyword | `pipe_job.pipe.code` |-| `PipelineRunId` | Keyword | `pipe_job.job_metadata.pipeline_run_id` |-| `SessionId` | Keyword | `TemporalManager.get_instance().session_id` |-| `UserId` | Keyword | `pipe_job.job_metadata.user_id` |-| `DomainCode` | Keyword | `pipe_job.pipe.domain_code` |--These are required for the Pipelex Temporal integration. The Pipelex side does not register them automatically — the cluster admin runs the equivalent of:--```bash-temporal operator search-attribute create \- --namespace default \- --name PipeCode --type Keyword \- --name PipelineRunId --type Keyword \- --name SessionId --type Keyword \- --name UserId --type Keyword \- --name DomainCode --type Keyword-```--(or the corresponding `tctl` invocation, or the Temporal Cloud UI action.)--A bootstrap check at worker start performs `DescribeNamespace` and logs a clear warning naming any missing attributes, including the exact registration command. The check is a soft fail: in-process test servers (the CI default) skip search attributes entirely, and a dev environment without registration still runs — only the dashboard experience is degraded.--#### Where the values are set--- **Top-level workflow start (submitter side).** `WorkflowExecutor.execute_workflow` and `start_workflow` build the search-attribute dict from `pipe_job` and pass it via `search_attributes=` to `client.execute_workflow` / `client.start_workflow`.-- **Child workflow start (workflow-side).** `TemporalPipeRouter._run_pipe_job` (child branch) and `WfPipeRun` (the fixed `pipe-router` child) pass `search_attributes=` to `workflow.execute_child_workflow` / `workflow.start_child_workflow`. `PipeCode` and `DomainCode` reflect the child's pipe; the other three are inherited unchanged from the workflow context (`workflow_arg.job_metadata` + `TemporalManager`).--Built-in attributes (`WorkflowType`, `WorkflowId`, `StartTime`, `ExecutionStatus`, `TaskQueue`) come for free and need no registration.--#### Memo (v1: minimal)--Memo is left mostly empty in v1. The single planned entry carries non-filterable context for future expansion:--```python-memo = {"pipelex": {"library_crate": "<id>@<version>"}}-```--This is an extension point. Anything that should round-trip with the execution but does not need filtering (OTel trace IDs, dispatch-resolution trace pointers, larger input fingerprints) lives here in later versions.--## Determinism (the why behind each choice)--| Choice | Why it is replay-safe |-|---|---|-| Workflow ID set at submitter side | Set outside workflow code; no replay concern |-| SDK-default `activity_id` | Server-assigned deterministically by history position |-| Child workflow ID uses `workflow.uuid4()` | Temporal's `workflow.uuid4()` is deterministic by design |-| Child workflow ID uses `pipe_job.pipe.code` | Reads from workflow input, not from outside state |-| Search attribute values | Derived from `pipe_job` (workflow input) and `JobMetadata` — deterministic |-| Static summary / details | Set at submit time, outside workflow code |-| Per-activity summary | Derived from `job_metadata` + activity input — no I/O, no time, no randomness |--The invariant to defend in review: nothing on the activity-dispatch path may read worker-singleton state. Activity-id determinism comes entirely from the Temporal SDK assigning IDs by history position — that is the load-bearing fact for the whole design.--## Side-by-side: before / after--### Workflow list view (Temporal dashboard)--Before (today):--| Workflow ID | Type | Summary | PipeCode | Status |-|---|---|---|---|---|-| `EdgdJ-HR5fd-TemporalPipeRun-pipe-router` | `wf_pipe_run` | *(empty)* | *(unset)* | Running |-| `EdgdJ-HR5fd-TemporalPipeRun-pipe-router` (child) | `wf_pipe_router` | *(empty)* | *(unset)* | Running |-| `EdgdJ-HR5fd-TemporalPipeRun-pipe-router` (child) | `wf_pipe_router` | *(empty)* | *(unset)* | Running |--After:--| Workflow ID | Type | Summary | PipeCode | Status |-|---|---|---|---|---|-| `ut-3f9c…b3c` | `wf_pipe_run` | `translate_doc — Translate EN→FR` | `translate_doc` | Running |-| `ut-3f9c…b3c/pipe-router` | `wf_pipe_router` | `translate_doc — Translate EN→FR` | `translate_doc` | Running |-| `ut-3f9c…b3c/pipe-router/extract_text-4d9b1c83` | `wf_pipe_router` | `extract_text — Extract paragraphs` | `extract_text` | Running |--### Event history (Temporal dashboard, one workflow)--Before:--```-ActivityTaskScheduled: id=craft-text, type=act_llm_gen_text-ActivityTaskScheduled: id=craft-text, type=act_llm_gen_text ← collision, workflow crashes-```--After:--```-ActivityTaskScheduled: id=1, type=act_llm_gen_text- Summary: LLM text · pipe=translate_doc · model=gpt-4o-ActivityTaskScheduled: id=2, type=act_llm_gen_text- Summary: LLM text · pipe=translate_doc · model=gpt-4o-ActivityTaskScheduled: id=3, type=act_llm_gen_object- Summary: LLM object · pipe=translate_doc · class=Section-```--### Pipe author API--Before — `wfid` thread through everything, never actually used in production:--```python-async def make_llm_text(- self,- job_metadata: JobMetadata,- llm_setting_main: LLMSetting,- llm_prompt_for_text: LLMPrompt,- wfid: str | None = None, # what is this? who passes it? answer: nobody, only tests-) -> str: ...-```--After — single concern per parameter:--```python-async def make_llm_text(- self,- job_metadata: JobMetadata,- llm_setting_main: LLMSetting,- llm_prompt_for_text: LLMPrompt,-) -> str: ...-```--Identity (`pipeline_run_id`) and observability (`pipe_code`, `domain_code`, …) flow through `job_metadata`, which is already a required argument. Pipe authors do not think about Temporal ids.--## Detailed change set (the design's footprint in code)--This section specifies what the design implies for the codebase. Sequencing, effort, and risk belong in a separate implementation plan.--### Files modified--**`pipelex/temporal/temporal_manager.py`**--- Replace `make_top_workflow_id(base_id: str)` with `make_top_workflow_id_for_pipeline_run(pipeline_run_id: str)` returning `f"{prefix}{pipeline_run_id}"`. The session-id and random-id pieces are removed from the Workflow ID. `session_id` remains accessible on the manager — it is consumed by the `SessionId` search-attribute builder.--**`pipelex/temporal/tprl/workflow_caller.py`**--- `WorkflowExecutor.make_workflow_id(...)` simplifies to accept a `pipeline_run_id: str` and delegate to the new manager method.-- `execute_workflow` and `start_workflow` accept additional `search_attributes`, `static_summary`, `static_details`, `memo` parameters and pass them through to the SDK.--**`pipelex/temporal/tprl_pipe/temporal_pipe_run.py`**--- Drop the `wfid` parameter from `run(...)` and `start(...)`.-- Compute `pipeline_run_id = pipe_job.job_metadata.pipeline_run_id` and pass to `make_workflow_id`.-- Build and pass the search-attribute dict, static_summary, and static_details from the helpers below.--**`pipelex/temporal/tprl_pipe/temporal_pipe_router.py`**--- Drop the `wfid` parameter from `_run_pipe_job(...)`.-- Top-level branch: same as `temporal_pipe_run.py` — derive workflow id, search attributes, and summary from `pipe_job`.-- Child branch: `child_workflow_id = f"{parent_workflow_id}/{pipe_job.pipe.code}-{str(workflow.uuid4())[:8]}"`. Pass updated search attributes (`PipeCode` + `DomainCode` reflect the child's pipe; others inherit from the workflow argument's `job_metadata` and `TemporalManager`).--**`pipelex/temporal/tprl_pipe/wf_pipe_run.py`**--- Line 46: change `f"{workflow.info().workflow_id}-pipe-router"` to `f"{workflow.info().workflow_id}/pipe-router"` (separator switch).-- Pass `search_attributes=` to the `execute_child_workflow` call. `PipeCode` does not change between `wf_pipe_run` and its `wf_pipe_router` child (it is the same pipe — the router is just executing it), so the search attributes from the parent are reused unchanged.--**`pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py`**--- Delete `_seen_activity_ids`, `_MAX_SEEN_RUNS`, and `_record_activity_id`. Delete every call site of `_record_activity_id`.-- Drop the `wfid` parameter from every `make_*` method signature.-- Drop the `activity_id = wfid or "…"` line from every method.-- Drop the `activity_id=activity_id` argument on every `workflow.execute_activity(...)` call (SDK auto-generates).-- Add `summary=` per the per-activity-summary format table.--**`pipelex/cogt/content_generation/content_generator_protocol.py`**--- Drop `wfid` from every `make_*` method signature.--**`pipelex/cogt/content_generation/content_generator.py`** (direct mode), **`content_generator_dry.py`** (dry mode)--- Drop `wfid` parameters. Direct mode and dry mode never used the value for anything anyway.--**`pipelex/pipe_run/pipe_run.py`, `pipe_run_protocol.py`**--- Drop `wfid` from `run(...)`.--**`pipelex/pipe_run/pipe_router.py`, `pipe_router_protocol.py`, `dry_pipe_router.py`**--- Drop `wfid` from `run(...)` and `_run_pipe_job(...)`.--### Helpers to be added--A new module `pipelex/temporal/tprl/observability.py` consolidates the formatting policy in one place:--- `build_search_attributes(pipe_job: PipeJob) -> Mapping[str, list[str]]` — returns the dict for top-level workflow start.-- `build_search_attributes_for_child(child_pipe_job: PipeJob, parent_search_attrs: Mapping[str, list[str]]) -> Mapping[str, list[str]]` — updates `PipeCode` + `DomainCode` for a child pipe; inherits the rest.-- `build_static_summary(pipe: PipeBase) -> str` — returns the 200-byte-truncated summary string.-- `build_static_details(pipe_job: PipeJob, library_crate_id: str | None) -> str` — returns the markdown details block.-- `build_activity_summary(method_label: str, job_metadata: JobMetadata, **extras: str) -> str` — returns the 200-byte-truncated per-activity summary string.--A single home for the formatting policy keeps the call sites (in `content_generator_in_workflow.py`, `temporal_pipe_run.py`, `temporal_pipe_router.py`, `wf_pipe_run.py`) thin and trivially unit-testable.--### Tests--**To delete or rewrite:**--- `tests/unit/pipelex/temporal/test_content_generator_in_workflow.py::test_make_llm_text_threads_explicit_wfid` — concept no longer exists.-- `…::test_duplicate_wfid_raises_content_generation_error` — concept no longer exists (no duplicate to raise; LRU is gone).-- `…::test_default_wfids_for_image_methods_are_distinct` — concept no longer exists; replaced by an assertion that `activity_id` is unset (SDK default) and per-activity summaries are correctly populated.--**To turn green naturally (the TDD gate):**--- `tests/unit/pipelex/temporal/test_default_activity_id_collision_bug.py::test_two_make_llm_text_calls_without_wfid_should_succeed` — passes because `activity_id` is no longer customized; SDK assigns `"1"` then `"2"`.-- `…::test_two_make_llm_text_calls_with_same_explicit_wfid_should_succeed` — the test's call site cannot pass `wfid` after the parameter is dropped; the test collapses into the first one, so one of the two becomes the canonical regression gate.--**To add:**--- `test_workflow_id_construction.py` — top-level `{env_prefix}{pipeline_run_id}` shape and child slash-separated nesting (fixed-role and dynamic).-- `test_search_attribute_dict_construction.py` — the five keys and their value sources for representative `pipe_job`s.-- `test_observability_helpers.py` — `build_static_summary`, `build_static_details`, `build_activity_summary`, including the 200-byte truncation behavior.--## Answers to the open questions in the reference doc--Mapping every question from `temporal-id-primitives.md` §7 to a decision:--1. **Workflow ID shape.** `{env_prefix}{pipeline_run_id}`. Pipelex's existing UUID with the existing run-mode prefix. No info loss, no invented fields.-2. **Child workflow id derivation.** `{parent_workflow_id}/{role-or-pipe_code-and-uuid8}`. Slash separator for clarity; role-name suffix for fixed 1:1 children; `{pipe_code}-{workflow.uuid4()[:8]}` for dynamic ones.-3. **Activity id policy.** SDK-default integers. No customization. The disambiguator concern disappears with the LRU.-4. **Workflow Type granularity.** Stay with the small registered set (`wf_pipe_run`, `wf_pipe_router`). Filter by `PipeCode` search attribute instead of registering one type per pipe.-5. **Summary / details strategy.** `static_summary` = `{pipe_code} — {description}`; `static_details` = markdown table of identity + crate + input shape; `current_details` deferred to a later iteration.-6. **Search attribute schema.** `PipeCode`, `PipelineRunId`, `SessionId`, `UserId`, `DomainCode` — all Keyword.-7. **Memo contents.** Minimal in v1 (optional `library_crate` fingerprint). Reserved as extension point for non-filterable context (OTel pointers, dispatch trace pointers, larger fingerprints) in later versions.-8. **`wfid` parameter rename.** Dropped, not renamed. Identity flows entirely via `JobMetadata.pipeline_run_id`; observability is auto-derived from `pipe_code` / `domain_code`. No caller hint is needed for either purpose.--## Breaking changes (per `CLAUDE.md`, no transition period)--- Workflow ID format changes from `{env}{session5}-{rand5}-{ClassName}` to `{env_prefix}{pipeline_run_id}`. Any operational tooling that grepped for the old shape is updated.-- The `wfid` parameter is removed from public protocols (`PipeRunProtocol`, `PipeRouterProtocol`, `ContentGeneratorProtocol`). No production code passed it; tests are the only blast radius.-- Activity IDs change from semantic labels (`craft-text`, `craft-object-direct`, …) to SDK-default integers (`"1"`, `"2"`, …). Anything that filtered or grouped Event History by these strings now reads the per-activity `summary` or the Activity Type.-- Child Workflow ID separator changes from `-` to `/`. Operational tooling parsing the existing nested ID format adapts.--These changes are noted in `CHANGELOG.md` with the implementation.--## Search attribute registration runbook--For the implementer of the runbook step:--- Document the required attributes in `docs/temporal-deployment.md` (new section) with the exact `temporal operator search-attribute create` invocation.-- On worker startup, log a clearly-worded warning if a `DescribeNamespace` call reveals any of the required attributes are missing — the log line includes the exact registration command so the operator does not have to look it up.-- Pipelex Cloud / Temporal Cloud bootstrap scripts add the registration step. Self-hosted Temporal users follow the docs.--The bootstrap check is a soft fail: in-process test servers (CI default) skip search attributes entirely, and a dev environment without registration produces warnings but still runs (with a degraded dashboard experience). A hard fail would block legitimate dev setups for a dashboard-only concern.--## Out of scope / future work--- **Current Details for in-flight progress.** Wire `workflow.set_current_details(...)` inside `WfPipeRouter` to expose controller-by-controller progress in the Temporal dashboard. Useful for long-running pipes; not required to land the redesign.-- **Memo population.** Move the OTel trace pointer, dispatch resolution trace pointer, and any other large or non-filterable context into Memo. Touches the OTel and tracing modules.-- **Per-pipe Workflow Types.** Could register a Workflow Type per pipe family for dashboard-column filterability without search attributes. Rejected for v1 (registration logistics, worker fleet consistency cost) but the door is open if `PipeCode`-search-attribute filtering proves insufficient.-- **Search attribute schema versioning.** A small migration helper to add new attributes on existing namespaces will likely be needed once the v1 schema is in production.-- **Pipe-author DX surface.** With `wfid` gone, the protocol surfaces are clean. A future iteration could add an *optional* `display_label` parameter at the `PipeRun` entry point that prepends to `static_summary` — useful for CLI / test runs that want to tag a run findable in the dashboard. Not required; trivial to add later without breaking anything.diff --git a/wip/distributed-execution/tracing-cost-reporting.md b/wip/distributed-execution/tracing-cost-reporting.mddeleted file mode 100644index 08c52f859..000000000--- a/wip/distributed-execution/tracing-cost-reporting.md+++ /dev/null@@ -1,118 +0,0 @@-# Tracing & Cost Reporting — As Built--> **Status**: Shipped — the current tracing & cost-reporting implementation (supersedes an earlier distributed-tracing design, now retired). T1 **and** T2 are now fixed; cross-worker cost reporting works end to end.-> **Last updated**: 2026-06-07-> **Related**: [the distributed-execution plan](README.md), [the registry overview](../history/registry/cost-reporting-overview.html), and [the registry feature](../history/registry/README.md).--This file documents what actually shipped, plus the known gaps that the plan addresses.-------## What shipped--The original Step 6 plan called for a `TracingActivityInboundInterceptor` to set up an NDJSON event log per standalone activity. The team picked a different design:--- **Pluggable backends behind `EventLogProtocol`** — `pipelex/tracing/event_log_protocol.py`. Implementations:- - `NdjsonEventLog` — local file backend (`pipelex/tracing/ndjson_event_log.py`).- - `DynamoDBEventLog` — cloud backend (`pipelex/tracing/dynamodb_event_log.py`), schema-compatible with `pipelex-api-infra`'s `TraceEventDynamoDBAdapter`. Available behind `pip install "pipelex[dynamodb]"`.- - `BufferingEventLog` — in-memory buffer used **inside** Temporal workflow code (`pipelex/tracing/buffering_event_log.py`). Synchronous I/O is forbidden in workflow context, so workflows buffer here and flush via an activity.- - `InMemoryEventLog` — for tests.- - Backend chosen at runtime via `make_event_log(tracing_config)` in `pipelex/tracing/event_log_factory.py` (`TracingBackend.NDJSON | DYNAMODB | TEMPORAL_DYNAMODB`).-- **Workflow → activity flush, not interceptor.** `WfPipeRouter.run()` (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`) wires a `BufferingEventLog` into the per-workflow `GraphTracerManager` and into the `ReportingManager` via `set_event_log(context_key=run_id, ...)` — all per-run worker-local state is keyed by `workflow.info().run_id` (replay-stable, unique per run; workflow ids are reused across retry/reset/resubmission). After pipe execution, buffered events are drained and persisted by `act_flush_trace_events` (`pipelex/temporal/tprl_pipe/act_flush_trace_events.py`), which runs the synchronous boto3 / file writes off the workflow thread. Replay determinism (audit finding H1): the flush schedule is a pure function of the payload — gated on `trace_context` presence AND (`emit_graph_events` or DRY run mode), so costs-only LIVE runs (whose buffer is deterministically empty) skip the round-trip while every potentially-non-empty case schedules it (the activity no-ops on an empty list) — and only workflow-thread emissions (inline tracer graph events, dry-run inline usage) land in the buffer — `ReportingManager` routes activity-side usage emissions to the per-process fallback even when the activity is co-located with the workflow worker, so the buffer content and the command stream are pure functions of payload + history.-- **Per-context keying** — `ReportingManager` holds a dict of `_event_log_contexts` keyed by `graph_context.lookup_key`, so concurrent workflows don't trample each other.-- **Direct mode unchanged** — `pipeline_run_setup.py:282` calls the same `set_event_log` path, so the same `EventLogProtocol` machinery runs in-process for non-Temporal execution.-- **Tracing assembly across workers (graph + cost)** — `pipelex/pipe_run/tracing_assembly.py` (`assemble_tracing` / `assemble_tracing_on_output`) and `pipelex/temporal/tprl_pipe/act_assemble_tracing.py` read all events for a `pipeline_run_id` from the configured backend **once** and feed them into both `GraphSpecAssembler` (when `--graph`) and `UsageAggregator` (when `--costs`), setting `PipeOutput.graph_spec` and `PipeOutput.tokens_usages` respectively. (Renamed from `graph_assembly.py` / `act_assemble_graph` when T2 landed — see below.)--## What works today--| Scenario | Captured? | How |-|---|---|---|-| Direct mode (single process) | ✅ | `pipeline_run_setup` configures event log on `ReportingManager`; events emitted in-process. |-| Single-worker Temporal (workflows + activities on same Worker bundle) | ✅ | Activity-side usage events emit through the per-process fallback — same path as split workers, never the workflow's in-sandbox buffer (replay determinism, audit finding H1). Workflow-thread emissions (dry-run inline usage, graph events) use the buffer registered by `WfPipeRouter`. |-| Parent + child workflows on different worker processes | ✅ | Each workflow's process buffers its own events and flushes its own slice. All slices land in the same NDJSON dir / DynamoDB partition keyed by `pipeline_run_id`. |-| Cross-worker **graph** assembly | ✅ | `assemble_tracing_on_output` / `act_assemble_tracing` reads all events for the `pipeline_run_id` and runs `GraphSpecAssembler`. |-| Cross-worker **cost** assembly + report | ✅ | Same single event read feeds `UsageAggregator` → `PipeOutput.tokens_usages`; the submitter renders one cost report from it. (T2 — fixed.) |--## What is broken or missing--### Issue T1 — Activities on standalone (separate-process) workers lose usage events — **FIXED (P0)**--**Severity (was)**: Blocked the whole point of distributed activity workers.--**Status**: Resolved (P0 in [the distributed-execution plan](README.md)).--The fix has two independent parts that ship together:--- **Runner-side fallback in `_emit_usage_event`.** When the activity's `ReportingManager` has no entry in `_event_log_contexts` for the workflow's `lookup_key`, it now falls back to a process-local event log built from `tracing_config`. The event log is cached per process via `pipelex.tracing.activity_event_log.get_or_create_activity_event_log` (guarded by a module-level `threading.Lock` so concurrent first-emitters from multiple activity threads agree on a single writer_id and instance). The fallback uses `graph_context.tracer_key or graph_context.graph_id` as `workflow_id` (no new `JobMetadata` field needed — `tracer_key` is already populated by `WfPipeRouter.run`). Specific exceptions (`OSError`, `MissingDependencyError`, `PipelexConfigError`, `botocore.ClientError`) are caught at WARNING — never silently dropped, never `except Exception`. A one-shot INFO log per process records that activity-side emission engaged (it is the normal path for every in-activity emission, co-located or split).-- **Writer-id disambiguation in `TraceEvent`.** A new `writer_id: str = "primary"` field makes `(workflow_id, sequence)` collisions impossible across writers. NDJSON file naming becomes `wf_{workflow_id}__w_{writer_id}.ndjson` for non-primary writers; the `(pipeline_run_id, workflow_id, writer_id)` cache key prevents two writers from sharing a stale file handle. Read-side dedup key is `(workflow_id, writer_id, type, sequence)`; sort key is `(workflow_id, sequence, writer_id)` — sequence primary so a runner-side `UsageReportEvent` does not sort before earlier router events. DynamoDB SK becomes `EVENT#{workflow_id}#{writer_id}#{sequence:010d}`.--The `_get_registry` orphan-accumulation TODO at `reporting_manager.py` is moot: T2 removed the submitter-side `UsageRegistry` entirely (see below), so there is no longer any per-run registry to accumulate or leak. The four `_report_*_job` methods now only emit a `UsageReportEvent` onto the event log; nothing buffers usage in the manager.--The known retry-related over-counting case (R2) is documented and pinned by `test_retried_activity_emits_duplicate_usage_event_documenting_r2`. A `tracing_config.strict_mode` flag (raise instead of WARNING + drop) is deferred — see [the plan, §P0.2](README.md).--### Issue T2 — Cross-worker cost report assembly — **FIXED (P1)**--**Status**: Resolved. Usage now rides on `PipeOutput` exactly like the graph spec, and the submitter renders the cost report from that field — so a cross-worker run produces a single, complete report.--> **Design note (as resolved):** T2/P1 was the same lifecycle as the `UsageRegistry` success-path leak — the leak was the missing *close*, T2 was the missing *replay-populate*, of one per-run cost-aggregation buffer. **Option B (locked) resolved both by removal:** usage is assembled from the trace-event stream at the end of the run and rides back on `PipeOutput.tokens_usages`; with nothing populating a submitter-side registry, the registry was deleted and the leak became structurally impossible. The unified model and the locked decision are explained in [the registry overview](../history/registry/cost-reporting-overview.html); the feature index is [`../history/registry/README.md`](../history/registry/README.md).--**As built:**--- The single event read in `assemble_tracing` / `act_assemble_tracing` (see "What shipped" above) now feeds `UsageAggregator.aggregate(events) → list[AnyTokensUsage]` (`pipelex/tracing/usage_aggregator.py`) when `--costs` is on, setting `PipeOutput.tokens_usages` in both direct and Temporal modes.-- The submitter renders it via `CostRegistry.generate_report(tokens_usages=...)` — fed from `pipe_output.tokens_usages`, never from a process-local registry. The main CLI path goes through `pipelex/reporting/cost_report_renderer.py::render_run_cost_report`.-- The decoupling that made this safe: a dedicated `--costs` / `is_generate_costs` switch (default on) gates usage events + `UsageAggregator` over the shared event-log transport, independent of `--graph`. Previously usage events only emitted when `--graph` was on.-- The retired surface: `ReportingManager.inject_tokens_usages` / `generate_report` / `open_registry` / `close_registry` and the `UsageRegistry` model are **gone** — there is no submitter-side registry left to populate.--Net effect: usage events emitted by runner/worker processes are now aggregated into one cross-worker cost report. Validated without spend by `tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py` (read-back aggregation), `test_mock_inference_temporal.py` (`--mock-inference` cross-process), and the direct-mode `test_direct_tracing_assembly.py` / `test_cost_report_rendering.py`.--### Issue T3 — `set_event_log` is on a process-singleton `ReportingManager`--**Status**: Partially mitigated by P0. The runner-side fallback now bypasses `_event_log_contexts` entirely when the lookup misses, so the singleton-+-context-dict pattern no longer blocks standalone activity workers. The deeper "request-scoped tracing data instead of process-scoped state" refactor remains open and is the natural follow-up alongside P1.--**Severity**: Design smell that magnifies T1.--`ReportingManager` is a process-singleton accessed via `hub.get_report_delegate()`. The per-context `_event_log_contexts` dict + `lookup_key` scheme now serves only workflow-thread and direct-mode emissions: activity-side emissions always take the per-process fallback, whether the activity is co-located or remote (the H1 determinism fix made co-location irrelevant to the emit path). The deeper shape issue stands — request-scoped tracing data (e.g., `JobMetadata` / `TracingContext`) would be a better home than a process-local dict.-------## How this connects to the plan--- T1 was the **top priority** in the [distributed-execution plan](README.md) — resolved.-- T2 is **resolved** — usage rides on `PipeOutput` (assembled from the same event read as the graph spec) and the submitter renders one cross-worker report. The wiring chosen was *not* `read events → inject_tokens_usages → generate_report` but the leaner Option B: `read events → UsageAggregator → PipeOutput.tokens_usages → CostRegistry.generate_report(tokens_usages=...)`, which let the submitter-side registry be deleted outright.-- T3 is the architectural shape; the T1 fix mitigated the immediate impact and T2's removal of the submitter-side registry shrank the singleton's surface further. The deeper "request-scoped tracing state instead of process-singleton" refactor remains the open follow-up (see Deferred / [the registry tracker](../history/registry/README.md)).-------## Deferred items--Items related to the event log and graph tracing system that are explicitly out of scope for now.--### Event log backends--| Item | Status | Context |-|---|---|---|-| DynamoDB backend | **Shipped** | `pipelex/tracing/dynamodb_event_log.py` (+ `TEMPORAL_DYNAMODB` variant). Schema-compatible with `pipelex-api-infra`'s `TraceEventDynamoDBAdapter`. Selected via `[pipelex.tracing] backend = "dynamodb"`. |-| SQLite backend | **Not planned** — NDJSON + DynamoDB cover the matrix | Not built. Re-evaluate only if a use case appears that NDJSON can't serve and DynamoDB is overkill. |--### Event log protocol extensions--| Item | Status | Context |-|---|---|---|-| `subscribe()` on EventLogProtocol | **Deferred** — no consumer exists yet | Polling-based reads are sufficient for now; add when a real-time consumer exists. |-| Real-time progress consumer | **Deferred** — no consumer exists yet | The protocol supports it; build when needed. |-| TraceSinkProtocol abstraction | **Deferred** — EventLogProtocol is sufficient | The higher-level sink abstraction can wait for the cloud backend. |-| Large data externalization | **Deferred** — inline data is sufficient for now | Add StorageProvider offloading when traces grow too large. |--### Tracing architecture--| Item | Status | Context |-|---|---|---|-| Unified code path (event log everywhere) | **Effectively unified** | `pipeline_run_setup.py:282` wires `set_event_log()` on the report delegate for direct mode too (using `workflow_id="direct"`). The same `EventLogProtocol` machinery runs in both modes; only the in-process vs. Temporal lifecycle differs. |-| Auto-cleanup of traces | **Deferred** — manual cleanup is safer for now | NDJSON traces persist on disk until manually deleted. Safer for debugging and allows usage reporting to run after graph generation. DynamoDB cleanup follows TTL on the table. |--### Known issues--| Item | Status | Context |-|---|---|---|-| Causal event ordering in assembler | **Known limitation** — works for current topologies | `read_events()` sorts by `(workflow_id, sequence)` which groups by lexicographic workflow ID, not execution order. In parent/child workflow topologies, this can cause `_stuff_producer_map` overwrites in the wrong order during `GraphSpecAssembler.pass_one()`, producing incorrect DATA edge sources. Consider sorting by timestamp or processing events in a topology-aware order. |-| PipeCondition SELECTED_OUTCOME wiring | **Deferred** — no production use yet | PipeCondition has no tracer calls in production code currently. If SELECTED_OUTCOME edges are needed, that's a separate change. |diff --git a/wip/distributed-execution/unit-testing-worker-sandbox-validation.md b/wip/distributed-execution/unit-testing-worker-sandbox-validation.mddeleted file mode 100644index e1d08312a..000000000--- a/wip/distributed-execution/unit-testing-worker-sandbox-validation.md+++ /dev/null@@ -1,55 +0,0 @@-# Follow-up: worker can't boot under `--is-unit-testing` (test workflows fail sandbox validation)--Deferred bug, found 2026-06-02 on `feature/Validate-with-signatures-4-fix-dry-run` while re-running the `/temporal-e2e-validate` Step 9 battery. Surfaced by the Scenario F sanity-check (`references/queue-options-battery.md`); orthogonal to the queue-options surface that battery validates. Not investigated beyond what's recorded here.--## Symptom--Starting a standalone Temporal worker in unit-test mode **with the sandbox enabled** fails to boot:--```bash-.venv/bin/python -m pipelex.temporal.worker_cli --task-queue temporal_task_queue --is-unit-testing-# ... passes config load + the task-queue validator ...-# RuntimeError: Failed validating workflow <a test-only workflow>-```--The `--task-queue` validator (`worker_cli.py:75`) runs **earlier** and is unaffected — this failure happens later, during `Worker(...)` construction. So it does **not** block the queue-options/routing work; it blocks a sandboxed unit-testing worker from coming up at all.--## What fires--`--is-unit-testing` (`worker_cli.py:50`) sets `RunMode.UNIT_TEST` and registers the test workflows from `pipelex/temporal/test_extras/temporal_test_tasks.py` (`temporal_task_manager.py:175-184`): `TEMPORAL_TEST_WORKFLOWS` = `WfTestContentGeneratorChild`, `WfTestContentGeneratorPdfPageViews`, `WfTestStructuredOutputCrossProcess`. temporalio then validates each registered workflow against the default `SandboxedWorkflowRunner`, and at least one fails.--The failing workflow is **non-deterministic across the set** (validation order). Observed on two separate runs:--- `Failed validating workflow wf_test_content_generator_child`-- `Failed validating workflow wf_test_structured_output_cross_process`--Concrete sandbox-restriction causes captured:--- `temporalio.worker.workflow_sandbox._restrictions.RestrictedWorkflowAccessError: Cannot access random.getrandbits.__class__ from inside a workflow. If this is code from a module not used in a workflow or known to only be used deterministically from a workflow, mark the import as pass through.`-- `ImportError: cannot import name 'Style' from partially initialized module 'rich.style' (most likely due to a circular import)` — `rich` pulled in at validation time inside the sandbox.--## Why it matters / scope--The failure requires the **sandboxed** runner (the default). The battery's real workers are all started with `--is-not-sandboxed`, which swaps in the unsandboxed runner and skips this validation — so the bug is latent for those. The Scenario F sanity-check is the path that exercised `--is-unit-testing` *without* `--is-not-sandboxed`, exposing it.--Two things worth resolving:--1. **Can a unit-testing worker run sandboxed at all?** If `--is-unit-testing` is only ever meant to pair with `--is-not-sandboxed`, that pairing should be enforced/documented. If not, the test workflows must be sandbox-safe.-2. **The test workflows are not sandbox-clean.** Non-deterministic access (`random.getrandbits`) and non-passthrough imports (`rich`) inside a workflow body are exactly what the sandbox is meant to catch — so even setting aside the boot failure, these test workflows would mask real non-determinism if anything depended on them running sandboxed.--## Likely fix locus--- The sandbox passthrough-modules list in `pipelex/temporal/temporal_task_manager.py` (`SandboxRestrictions.default.with_passthrough_modules(...)`, ~lines 103-119) is missing the test-extras modules and their transitive imports (`rich`, the in-memory storage provider used by `WfTestStructuredOutputCrossProcess`). Adding them is the narrow fix.-- Alternatively, wrap the offending imports/calls in the test workflow modules with `workflow.unsafe.imports_passed_through()` so they're not sandbox-validated.-- Or, if unit-testing workers are never meant to be sandboxed, have `--is-unit-testing` imply the unsandboxed runner (and say so).--## Repro--```bash-# sandbox ENABLED (default) → fails on a test workflow-.venv/bin/python -m pipelex.temporal.worker_cli --task-queue temporal_task_queue --is-unit-testing 2>&1 | grep -E "RestrictedWorkflowAccessError|Failed validating workflow"--# sandbox DISABLED → confirmed: gets past validation, boots, and polls (kill via timeout)-timeout 25 .venv/bin/python -m pipelex.temporal.worker_cli --task-queue temporal_task_queue --is-unit-testing --is-not-sandboxed 2>&1 | grep -E "Temporal Worker started|Failed validating workflow"-# observed: no "Failed validating workflow"; exits rc=124 from the timeout (worker was polling)-```diff --git a/wip/distributed-execution/validate-sweep-temporal-leak-repro.md b/wip/distributed-execution/validate-sweep-temporal-leak-repro.mddeleted file mode 100644index cd199d245..000000000--- a/wip/distributed-execution/validate-sweep-temporal-leak-repro.md+++ /dev/null@@ -1,150 +0,0 @@-# Bug brief + e2e repro — the `/validate` dry-run sweep leaked nested controller sub-pipes to Temporal--> **Status:** bug FIXED (commit `3377babb` on `fix/Temporal-dry-run`). **All three guard layers are now implemented:** the in-process sentinel regression test, the **Mode-1 pytest** (`tests/integration/pipelex/temporal/test_validate_sweep_stays_in_process.py`), and the **Mode-2 scenario** (Tier 2c in the `temporal-e2e-validate` skill). This doc is retained as the cold-start brief + reproduction record. See [`/TODOS.md`](../../TODOS.md) for the PR-review guide.->-> **Track:** part of the [distributed-execution](./README.md) track — the class of bug that only the true 3-process topology surfaces.--## 1. What the bug was--Running the **Pipelex API** (`../pipelex-api`) Temporal-enabled (`.pipelex/pipelex_override.toml` → `[temporal] is_enabled = true`) with a separate **Pipelex worker** (`../pipelex-worker`) up, calling the runner's `/validate` endpoint on a bundle containing a controller pipe returned **HTTP 422**:--```-ValidateBundleError (error_domain=input), 422-detail: "Dry run failed with 1 unexpected pipe failure(s):- '<domain>.<batch_pipe>': ... status=DryRunStatus.FAILURE- error_message=\"Dry run failed for pipe '<domain>.<batch_pipe>':- Failed to execute workflow WfPipeRouter\""-```--The string **"Failed to execute workflow WfPipeRouter"** is produced only at `pipelex/temporal/tprl/workflow_caller.py` (the `(WorkflowAlreadyStartedError, RPCError)` branch of `WorkflowExecutor.execute_workflow`) — a **top-level** Temporal workflow dispatch from the submitter side. **A dry run should never reach that code.**--## 2. Root cause (one sentence)--`BundleValidator` builds a local in-process `PipeRouter` for the sweep but **never installs it as the active `get_pipe_router()` override**, so every nested controller sub-pipe (`pipe_controllers/sub_pipe.py`, `pipe_controllers/batch/pipe_batch.py` — all dispatch via `get_pipe_router()`) falls through to the **hub default**, which under a Temporal-enabled hub is the `TemporalPipeRouter` (wired in `pipelex/pipelex.py` when `get_config().temporal.is_enabled`). The top-level pipe ran in-process; its nested pipes leaked to Temporal.--**Why it 422s for a *standalone* `PipeBatch`/`PipeParallel` specifically:** the sweep dry-runs every pipe in the bundle individually. When it dry-runs a standalone batch pipe *directly*, the batch fans out over a mock list (`mock_inputs` mints ~3 items) and fires **N concurrent top-level Temporal dispatches with the same workflow id** (the id derives from the one shared `dry_run_pipeline_id`) → `WorkflowAlreadyStartedError` → `WorkflowExecutionError("Failed to execute workflow WfPipeRouter")` → the pipe is classified `FAILURE` → `DryRunError` → 422. A batch reached only *as a sub-pipe of a sequence* is a single serial dispatch — no collision — so it **silently round-trips Temporal and passes** (a false pass: still wrong, just not fatal).--## 3. The fix (already committed: `3377babb`)--Mirror the DIRECT-mode precedent in `pipelex/runtime_bridge/bridge.py::_run_direct`. In `pipelex/pipeline/bundle_validator.py`:--- keep the in-process router as `self._pipe_router` (don't discard it inside `PipeRun(...)`);-- wrap the per-pipe sweep loop in `validate_pipes` with `with scoped_pipe_router(self._pipe_router):`.--Now nested controllers resolve the in-process router; the whole sweep stays in-process regardless of backend.--**In-process regression guard (deterministic, no Temporal needed):**-`tests/integration/pipelex/pipeline/test_bundle_validator.py::TestBundleValidatorIntegration::test_standalone_batch_sweep_scopes_in_process_router` — installs a hub-default router that raises if reached, sweeps a standalone `PipeBatch`, asserts SUCCESS with the hub default never touched. (Verified RED without the scope.)--> ⚠️ A *separate* "secondary fix" to `dry_run_pipeline.py` (the graph step) was tried and **reverted** — it was misdiagnosed. See the correction block in [`/TODOS.md`](../../TODOS.md). Not relevant to this repro.--## 4. Why our PyTest suite didn't catch it (answer to "can in-process pytest reproduce it?")--**Partial confirm, with nuance — don't over-claim "impossible in-process":**--- The **leak itself** is reproducible in a single process with zero Temporal infrastructure — that's exactly what the committed regression test does (sentinel hub-default router). The **collision symptom** is *also* reproducible in-process in principle: the in-process Temporal test server (`--temporal-server none`/`time-skipping`) enforces the same workflow-id-reuse semantics, so concurrent same-id dispatches would still raise `WorkflowAlreadyStartedError`.-- So it escaped **not** because the topology can't show it, but because **no test crossed the seam "Temporal-enabled hub × the bundle-validation sweep":** the Temporal suite (`tests/integration/pipelex/temporal/`) exercises pipe *runs* via crate, never the validation *sweep*; the `BundleValidator` tests run under the default in-process hub, never the Temporal hub.-- Two reasons the **distributed (Mode 2) setup is still the right home** for a scenario:- 1. The silent **false-pass** shapes are invisible to a "does validate pass?" assertion — only an explicit "no Temporal dispatch happened" check (the sentinel test) or the standalone-batch collision shape exposes them.- 2. The **production symptom** (collision → 422 across a real API↔worker process boundary) matches the deployment topology exactly here, where a human actually hit it.--Net: the unit regression test guards the leak; the e2e scenario below guards the deployment-faithful symptom and documents the reproduction for the record.--## 5. The new scenario to add to `temporal-e2e-validate`--Lives under the skill at `.claude/skills/temporal-e2e-validate/`. Add this as a new scenario in `references/mode-2-tiers.md` (or a short new reference file).--**Which home?** This bug is **submitter-side** — the colliding top-level dispatches fire at *dispatch time*, regardless of whether the worker runs in the same process. So the true 3-process topology is the *deployment-faithful* home, **not the only one**: the same failure also reproduces under the classic in-process pytest harness. Treat Mode 2 here as the production-faithful demonstration (and the only thing that also exercises the genuinely cross-process surfaces — LibraryCrate propagation, serialization), and add the cheaper automated catch as a pytest companion (next subsection).--**Enabler that makes this clean:** `pipelex validate bundle` now has a `--temporal/--no-temporal` flag (added alongside the fix; parity with `pipelex run`). So the repro needs **no `pipelex_temporary_override.toml` juggling** — just pass `--temporal`. (The flag flips the boot's hub default to the Temporal router; with the fix, the sweep still stays in-process — which is the whole point being asserted.)--> **Forward-looking note (why the flag exists despite being a no-op for the sweep today):** once validation runs as a standalone Temporal activity ([`../dry-run-refactor/followup-temporal-validation-activity.md`](../dry-run-refactor/followup-temporal-validation-activity.md)), `--temporal` on `validate` stops being a no-op and becomes the switch that dispatches the sweep through that activity. Until then it only controls the boot, which is exactly what this scenario needs.--**Repro bundle — reuse an existing one, no new fixture needed:**-`tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds` already declares a **standalone `type = "PipeBatch"` pipe** (`batch_temporal_describe_topics`). When the sweep dry-runs that pipe directly, it fans out → the exact shape that 422'd in production.--### Prereqs-Temporal server + worker up, per `references/mode-2-setup.md` (read it first). The worker config is irrelevant to the leak — the leak is entirely submitter-side — but keep a worker up to mirror production (and so the first dispatch actually *starts*, yielding the true `WorkflowAlreadyStartedError` rather than an `RPCError`; both land in the same `except` branch, so both are RED, but the former matches production).--### GREEN — the fix in place (current `HEAD`)--```bash-# Server up (Mode 2). The boot connects to it; with the fix the sweep stays in-process,-# so the worker should receive NO workflow for this validate run.-timeout 120 .venv/bin/pipelex validate bundle \- tests/integration/pipelex/temporal/library_crate/temporal_batch.mthds \- --temporal 2>&1 | tail -20-echo "EXIT=$?"-```--Expected: `EXIT=0`, `Successfully validated bundle ...`. Strong check: the worker session shows **no `WfPipeRouter` / `WfPipeRun` execution** for this run (it stayed idle) — `tmux capture-pane -t temporal-worker-router -p -S -200 | grep -i WfPipeRouter` returns nothing new.--### RED — temporarily revert the fix to prove the scenario bites--The fix is committed, so you must undo it in the working tree. **Surgical (preferred)** — neutralize just the scope in `pipelex/pipeline/bundle_validator.py`, inside `validate_pipes`:--```python-# change THIS:-with scoped_pipe_router(self._pipe_router):- for pipe in sweepable_pipes:- results[pipe.pipe_ref] = await self._classify_pipe(...)--# to THIS (drop the scope — the leak returns):-for pipe in sweepable_pipes:- results[pipe.pipe_ref] = await self._classify_pipe(...)-```--(Alternative: `git revert --no-commit 3377babb` — but that also reverts the regression test + changelog; the surgical edit is cleaner.)--Re-run the GREEN command. Expected RED:--```-EXIT=1-Dry run failed with 1 unexpected pipe failure(s):- 'temporal_batch_test.batch_temporal_describe_topics': ... status=DryRunStatus.FAILURE- error_message="... Failed to execute workflow WfPipeRouter"-```--The worker session will now show `WfPipeRouter` activity (the leak — validation dispatched to Temporal). **Restore the fix immediately:**--```bash-git checkout -- pipelex/pipeline/bundle_validator.py # or: git revert --abort, if you used revert-```--### PASS criteria for the scenario-- GREEN run exits 0 **and** the worker received no workflow dispatch (validation stayed in-process under a Temporal-enabled boot).-- The documented RED procedure reproduces the `Failed to execute workflow WfPipeRouter` failure (sanity that the scenario actually exercises the leak path).--### Caveats to put in the scenario text-- **Shape matters.** Only a *standalone* `PipeBatch`/`PipeParallel` swept directly turns the leak fatal (concurrent same-id collision). A batch reached only as a sequence sub-pipe round-trips Temporal but passes — a false pass. `temporal_batch.mthds` has the standalone batch, so it's a valid RED trigger.-- **Worker up vs down both RED** (collision → `WorkflowAlreadyStartedError` vs no-worker → `RPCError`; same `except` branch). Keep the worker up to match production.-- **GREEN means "no dispatch happened."** Because the flag makes validation backend-agnostic, the strongest GREEN assertion is the worker-idle check, not just exit 0.--### Automated companion — a Mode-1 pytest (recommended alongside the Mode 2 scenario)--The Mode 2 scenario is agent-run, not CI-automated. Because this bug is submitter-side, the same catch can live as an ordinary pytest in the classic in-process Temporal harness (`tests/integration/pipelex/temporal/`, `--temporal-server none`/`local`, in-process worker) — fast, deterministic, runs in CI. That is the cheaper automated guard; Mode 2 remains the deployment-faithful demonstration. The three layers are complementary, not redundant:--| Layer | Where it runs | CI-automated | Cross-process surfaces (crate / serialization) | Status |-|---|---|---|---|---|-| Sentinel regression test (no Temporal) | plain pytest (`tests/integration/pipelex/pipeline/test_bundle_validator.py`) | ✅ | ✗ (deterministic stand-in for the hub default) | **exists** (`test_standalone_batch_sweep_scopes_in_process_router`) |-| **Mode-1 pytest (real Temporal, in-process server+worker)** | classic pytest temporal env | ✅ | ✗ (worker shares the process) | **proposed here** |-| Mode 2 scenario (3 separate processes) | the skill, agent-run | ✗ | ✓ | proposed in §5 |--Sketch of the Mode-1 pytest (author it in `tests/integration/pipelex/temporal/`, marked `temporal`, honoring `--temporal-server`):--- Boot/scope a **Temporal-enabled hub** so `get_pipe_router()`'s default is the real `TemporalPipeRouter` (mirror the existing temporal-suite fixtures that stand up the in-process server + worker).-- Run the real sweep — `BundleValidator().validate_pipes([...standalone PipeBatch...], ...)` — over `temporal_batch.mthds`'s `batch_temporal_describe_topics` (or an equivalent standalone `type = "PipeBatch"`).-- **Assert GREEN, strongly:** the sweep succeeds **and** no top-level workflow was started — e.g. spy on `pipelex.temporal.tprl.workflow_caller.WorkflowExecutor.execute_workflow` and assert it was never called (the deterministic analogue of the Mode 2 "worker-idle" check). This is what makes the fix's contract ("the sweep never reaches the Temporal router") an automated invariant, not just "validate didn't 422."-- Without the fix this test goes RED (the spy fires / a collision surfaces), exactly like the Mode 2 scenario.--This differs from the existing sentinel test in one meaningful way: it resolves the **real** `TemporalPipeRouter` as the hub default instead of a raising stand-in, so it also proves the *wiring* (config → hub default router) that the sentinel test deliberately bypasses.--## 6. Pointers--- Fix commit: `3377babb` — `fix(bundle-validator): prevent nested controller sub-pipes from leaking to Temporal during dry runs`.-- Fix site: `pipelex/pipeline/bundle_validator.py` (`__init__` keeps `self._pipe_router`; `validate_pipes` wraps the sweep in `scoped_pipe_router`). Precedent: `pipelex/runtime_bridge/bridge.py::_run_direct`.-- Hub wiring that makes the hub default Temporal: `pipelex/pipelex.py` (`set_pipe_router(make_temporal_pipe_router())` when `temporal.is_enabled`); resolution in `pipelex/hub.py::get_pipe_router` (contextvar override → else hub default).-- Leak sites: `pipelex/pipe_controllers/sub_pipe.py`, `pipelex/pipe_controllers/batch/pipe_batch.py` (`get_pipe_router().run(...)`).-- New CLI flag: `--temporal/--no-temporal` on `pipelex validate {bundle,pipe,method}` + `validate --all` (`pipelex/cli/commands/validate/`), guarded by `tests/unit/pipelex/cli/test_validate_temporal_flag.py`.-- In-process regression test: `tests/integration/pipelex/pipeline/test_bundle_validator.py::TestBundleValidatorIntegration::test_standalone_batch_sweep_scopes_in_process_router`.-- Skill: `.claude/skills/temporal-e2e-validate/SKILL.md` + `references/mode-2-setup.md` + `references/mode-2-tiers.md`.diff --git a/wip/distributed-execution/workflow-nondeterminism-audit.md b/wip/distributed-execution/workflow-nondeterminism-audit.mddeleted file mode 100644index eed830e82..000000000--- a/wip/distributed-execution/workflow-nondeterminism-audit.md+++ /dev/null@@ -1,124 +0,0 @@-# Workflow Nondeterminism Audit — worker-local inputs leaking into the command stream--**Status: H1, H2, and M1 fixed; remaining findings are diagnosis only.** Multi-agent audit of all code reachable inline from `@workflow.defn` bodies, run on top of the PR #984 fix (commit `db669ea62`, which gated `act_flush_trace_events` on the payload's `trace_context` instead of worker-local `tracing_config.is_enabled`). Every finding below was adversarially verified against the actual code; refuted candidates are listed at the end so they don't get re-chased.--**The breach definition used throughout:** any input read in code executing inline on the workflow thread (the `@workflow.run` body and everything it calls outside `@activity.defn` functions) that is not derived from the workflow payload or workflow APIs (`workflow.now()`, `workflow.uuid4()`, history), and that influences the command stream — whether/which/how many activities or child workflows get scheduled, their order, arguments, timeouts, retry policies, or task queues. Config reads inside activity bodies or at submission time are fine.--**Key calibration from verification (matters for prioritization):** Temporal's replay checker compares command *sequence, types, and IDs* — **not input payloads**. So worker-local inputs that flip *whether a command is emitted* cause the TMPRL1100 silent-hang class; inputs that only change *command arguments* (activity args, timeouts, retry policies, task queues of not-yet-scheduled commands) cause silent skew, not hangs. Both are breaches; only the first class reproduces the PR #984 symptom.--## Inline/activity boundary (map facts)--Everything from `WfPipeRouter.run` → `pipe.run_pipe` → controller/operator `_live_run_*` paths (Jinja prompt rendering, model-deck resolution, class-registry lookups, PipeFunc user functions) executes inline on the workflow thread. The activity boundary is exactly the `workflow.execute_activity(...)` calls inside `ContentGeneratorInWorkflow.make_*`. Controller pipes (`PipeSequence`/`PipeParallel`/`PipeCondition`/`PipeBatch`) dispatch every sub-pipe as a child `WfPipeRouter` via `hub.get_pipe_router()` → `TemporalPipeRouter`; operator pipes are *not* turned into activities wholesale — each runs inline inside its own child workflow and only its inference/templating/extract/img/search leaves cross to activities. `TemporalPipeRun` and `WorkflowExecutorFactory` are submitter-only (no in-workflow branch). The dry-run path (`run_mode=DRY`, `ContentGeneratorDry`) is reachable inline — `run_mode` comes from the payload and nothing on the Temporal dispatch path gates DRY out.-------## HIGH — can reproduce the TMPRL1100 silent hang (command-count divergence)--These two finish what PR #984 started: scheduling around the flush activity and library teardown in `wf_pipe_router.py` is still not a pure function of payload + history.--### H1. Flush activity gated on a buffer that activities populate cross-thread — empty on replay — **FIXED**--> **Fixed:** `WfPipeRouter` now schedules `act_flush_trace_events` unconditionally whenever the payload carries a `trace_context` (the activity no-ops on an empty list), and `ReportingManager._emit_usage_event` routes any emission coming from inside a Temporal activity (`activity.in_activity()`) to the per-process runner fallback — co-located activities never write into the workflow's in-sandbox buffer. Guarded by `tests/integration/pipelex/temporal/test_wf_pipe_router_costs_only_flush_nondeterminism.py`.--`pipelex/reporting/reporting_manager.py:176` + `pipelex/temporal/tprl_pipe/wf_pipe_router.py:221`--`WfPipeRouter` registers its `BufferingEventLog` in the process-global `ReportingManager._event_log_contexts` keyed by `wf_{workflow_id}`. The single Worker registers workflows and activities in one process, so co-located **activity** completions (`report_inference_job`) hit the fast path and mutate the workflow's buffer from the activity thread. In costs-only mode (`emit_usage_events=True, emit_graph_events=False` — a first-class payload option, see `pipeline_run_setup.py`) the tracer gets `event_log=None`, so those activity-side emissions are the *only* buffer content. Live execution drains a non-empty buffer and schedules `act_flush_trace_events`; on replay, activity results come from history without executing, the recreated buffer stays empty, and the `if buffered_events:` gate emits no flush command. A `ScheduleActivityTask` present in history but absent on replay → TMPRL1100 **from a routine sticky-cache eviction on the same worker** — no config skew or fleet heterogeneity needed. Symmetrically, routing activities to a different queue/worker, or `is_reporting_enabled=false` on the replaying worker (a `ReportingNoOp` never populates the buffer), flips the same command. Graph+usage mode is safe for the boolean (inline tracer emissions re-fire deterministically on replay) — only costs-only mode flips command emission.--**Fix:** gate the flush dispatch purely on the payload (`trace_context` presence / emit flags) and let the activity no-op on an empty event list; stop routing activity-side usage events through the workflow's in-sandbox buffer (co-located activities should use the same runner-fallback per-process event log they use when remote).--### H2. Eviction skips library teardown → leaked fingerprint poisons same-worker replay — **FIXED**--> **Fixed:** two complementary moves. (1) Self-healing setup: `WfPipeRouter` opens its per-workflow library via the new `LibraryManager.open_fresh_library`, which force-tears-down any pre-existing library under the deterministic `wf_{workflow_id}` — a leftover there can only be an interrupted predecessor, and a clean teardown removes the whole stale-state class (fingerprint, pipes, concepts, registry), which is strictly stronger than scoping the fingerprint dedup (re-loading a crate into a half-stale library would hit duplicate-add errors). (2) Eviction-safe cleanup ordering: the workflow `finally` now runs ALL worker-local cleanup (event-log drain/close, report-delegate context clear, library teardown) BEFORE the awaited flush activity, which moved last — a `BaseException` raised at that await can only skip the flush itself, never the teardown. Guarded by `tests/integration/pipelex/temporal/test_wf_pipe_router_eviction_library_leak.py`.--`pipelex/temporal/tprl_pipe/wf_pipe_router.py:96` (setup) and `:241` (teardown)--Worker-local cleanup (`get_library_manager().teardown(wf_library_id)`) sits in the `finally` block *after* the awaited flush activity. Eviction-time interruptions (`_WorkflowBeingEvictedError` — a `BaseException` subclass raised by `_assert_not_read_only` when the workflow is being deleted; also `CancelledError` paths) bypass the `except Exception` at line 229 and abort the `finally` before teardown runs, leaking `_libraries` and `_loaded_fingerprints` in the worker-local `LibraryManager` singleton, keyed by the *deterministic* `wf_{workflow_id}`. On same-worker sticky replay: `open_library` returns the stale library, `set_class_registry` installs a **fresh** registry seeded only from the global one, and `load_from_crate` is fingerprint-skipped (`library_manager.py:388-393`) — so the crate's dynamic classes are never registered into the new registry. `hydrate_working_memory` then raises `PipeJobError` inline where history recorded successful pipe execution → the workflow takes the error path instead of re-emitting recorded commands → command-stream divergence. Triggers are routine: cache eviction under load + tracing-on payload + same-worker replay. The common variant self-heals on the next task retry (that replay's synchronous `finally` tears the leak down), but the persistent-hang variant exists with dynamic-class activity results on a single-worker queue.--**Fix:** scope the fingerprint dedup to the registry instance (clear `_loaded_fingerprints` when `set_class_registry` swaps the registry), or have `WfPipeRouter` force-teardown any pre-existing library under its deterministic id before opening; additionally move worker-local cleanup before the awaited flush activity or into an eviction-safe guard.-------## MEDIUM — command-stream breaches (which/how many commands, or known-residual divergence)--### M1. The known residual: best-effort tracing-setup `except Exception` — **FIXED**--> **Fixed:** `GraphTracerManager.open_tracer` is now collision-proof (pop-and-replace of a stale tracer key with a WARNING, instead of raising), and the best-effort `except Exception` around tracing setup in `WfPipeRouter` was removed entirely — the setup block is pure in-memory state, so any failure is a real bug that surfaces deterministically (and re-fires identically on replay).--`pipelex/temporal/tprl_pipe/wf_pipe_router.py:160`--The catch-all around tracing setup nulls `event_log` on any worker-local failure, so the flush command at line 223 is never emitted while history contains it. Reachable with identical config on both workers: `GraphTracerManager.open_tracer` raises `ValueError` on a stale tracer key leaked by a prior interrupted execution (`graph_tracer_manager.py:133-135`, process-lifetime singleton keyed by `workflow_id`). Verification narrowed severity: fresh/restarted workers have empty singletons so routine churn does *not* trigger it — a leak requires a stuck eviction or `disable_safe_eviction` plus a same-`workflow_id` rerun on the poisoned worker. Aggravator: on *success* the block rewrites `workflow_arg.job_metadata` with `wf_trace_context`, which rides inside every child-workflow `PipeJob` and every content-generation activity argument — so setup success/failure also diverges the arguments of all subsequent commands. **Fix:** payload-pure flush gating (H1's fix) plus make `open_tracer` collision-proof (pop-and-replace instead of raise).--### M2. `PipeBatch` fan-out chunking from worker config--`pipelex/pipe_controllers/batch/pipe_batch.py:141` — `get_config().pipelex.pipeline_execution_config.max_concurrency` bounds `gather_bounded` over branches that each `workflow.execute_child_workflow`. The chunk size controls how many `StartChildWorkflowExecution` commands are emitted per workflow task and their grouping; a config edit rolled out while a batch workflow is in flight diverges replay (TMPRL1100) on any batch spanning multiple workflow tasks. **Fix:** carry `max_concurrency` in `pipe_run_params` (payload), stamped from config at the submission boundary.--### M3. Dry-run mock fan-out length from worker config--`pipelex/cogt/content_generation/content_generator_dry.py:94` (also `nb_extract_pages` at lines 233/262, `nb_list_items` again at 305, image params) — `dry_run_config.nb_list_items` sets the *length* of mock lists built inline; a downstream `PipeBatch` fans out one child workflow per item, so the recorded command count equals a worker-local config value. DRY runs reach Temporal: `dry_run_pipeline()` dispatches via `TemporalPipeRun` under a Temporal-enabled hub with no DRY gate on the path (PR #976 only localized the BundleValidator sweep). **Fix:** move dry-run synthesis into the activity (aligned with the dry-run-as-temporal-activity direction) or carry the dry-run params in the payload.--### M4. `PipeCondition` pipe resolution on the no-crate path--`pipelex/pipe_controllers/condition/pipe_condition.py:254` — the expression evaluation is deterministic from `WorkingMemory`, but the resolved pipe object selects *which* child workflow gets scheduled (and is embedded in its `PipeJob` argument). On the crate-backed path resolution is payload-pure (per-workflow library rebuilt from the crate); the **no-crate** path is real and supported (`PipeJob.library_crate` is `Optional`; `worker_cli.py` boot-loads a `worker_base` library), and there library drift between workers changes the child command or replaces it with an inline PipeNotFound error path. **Fix:** require crate-derived resolution for Temporal dispatch, or carry the resolved sub-pipe selection in the payload.--### M5. `PipeFunc` runs arbitrary user code inline; sync branch is broken outright--`pipelex/pipe_operators/func/pipe_func.py:183,189` — `PipeFunc` has no temporal-aware dispatch: the worker-local module-level `func_registry` read and the user function both execute inline on the workflow thread. Registry presence (function registered on one worker but not another — `LibraryCrate` carries blueprints, not functions) flips fail-vs-complete command paths. The coroutine branch runs unconstrained user async code inline, where nondeterministic success-vs-failure flips the command stream and user `asyncio` calls emit nondeterministically-parameterized timer commands. **Bonus functional bug found during verification:** the sync branch's `asyncio.to_thread` raises `NotImplementedError` under Temporal's workflow event loop (no `run_in_executor`) — sync PipeFunc functions cannot work inside workflows at all today; it fails deterministically, but it fails. **Fix:** execute PipeFunc user functions inside a dedicated activity (`act_run_func`) when in Temporal mode.-------## MEDIUM/LOW — command-argument drift (silent skew, not hangs)--Temporal does not compare input payloads on replay, so these cannot TMPRL1100 — but recorded vs re-emitted command attributes diverge, and any not-yet-completed activity rescheduled after a config edit runs with different routing/args/policies than history recorded.--| Site | Worker-local input | What drifts | Fix |-|---|---|---|---|-| `tprl_content_generation/content_generator_in_workflow.py:113` (same pattern in every `make_*`: 154, 195, 265, 311, 350, 391, 429, 458, 491, 525) | `get_config().temporal.worker_config.resolve_dispatch` + `temporal.queue_options` at command-emission time | task_queue, all four timeouts, full RetryPolicy of every `ScheduleActivityTask` | move-to-payload: resolve `DispatchOptions` at the submission boundary and carry the per-activity dispatch bundle in `PipeJob`. Note `wf_pipe_run.py` already states and follows this exact principle for its child dispatch — the content generator violates it |-| `pipe_operators/llm/pipe_llm.py:196` (same pattern: `pipe_structure.py:125`, img-gen/extract operators) | worker-local model deck singleton (`get_model_deck()`; deck/config files, not in the crate) | resolved handle is both the activity arg (`LLMAssignment.llm_setting`) **and** the `routing_key` for `resolve_dispatch` → task-queue selection via `by_handle` | move-to-payload |-| `pipe_operators/llm/pipe_llm.py:219` | `pipelex.prompting_config` **plus mutation of `self.llm_prompt_spec` on the library-resident shared pipe object** (the `not templating_style` guard makes the first run on a worker win — sticky, order-dependent) | inline Jinja rendering → the LLM activity's prompt argument | move-to-payload for the style; separately, stop mutating library-resident pipe objects per-run |-| `pipe_operators/llm/pipe_llm.py:275` + `pipe_operators/structure/pipe_structure.py:132` | `cogt.llm_config` structure-prompt flag + template text | whether/how the structure-schema section is rendered into the activity-arg prompt | move-into-activity |-| `tprl_content_generation/content_generator_in_workflow.py:256` (and 302) | `cogt.img_gen_config` defaults | baked into `ImgGenAssignment` activity argument | move-into-activity |-| `tprl_content_generation/content_generator_in_workflow.py:385` (and 452) | `cogt.extract_config.default_page_views_dpi` (the default path always has `page_views_dpi=None`, so the read is always live) | `RenderPageViewsAssignment` activity argument | move-into-activity |-| `tprl/observability.py:104` | `temporal.search_attributes` (enabled flag + attribute subset) | `search_attributes` on `StartChildWorkflowExecution` (called from `wf_pipe_run.py:63` and `temporal_pipe_router.py:88`; the helper's docstring claims purity from `pipe_job` — the config read breaks that claim) | move-to-payload |-| `pipe_operators/compose/pipe_compose.py:317` | `pipeline_execution_config.is_mock_inputs` | raise vs mock-and-continue in the dry-run construct-mode except path — diverges the remaining command stream (low: needs a `StructuredContentComposerValueError` first) | move-to-payload |--### Randomness / clock cluster--Root cause: `pipelex/core/stuffs/stuff_factory.py:44` — `make_stuff_code()` is `shortuuid.uuid()[:5]`, and every operator's output stuff is minted inline on the workflow thread. The codes reach command arguments two ways: working memory serialized into child `PipeJob` args (`pipe_batch.py:138` branch codes → `final_stuff_code` on each branch's `PipeRunParams`; `sub_pipe.py:119` synthetic batch stuff; `pipe_batch.py:218` aggregate output), and the IOSpec digest inside trace events passed literally as the `act_flush_trace_events` argument. Adjacent instances:--- `pipelex/core/pipes/pipe_abstract.py:627` — `PipelineFactory.make_pipe_run_id()` is plain `uuid.uuid4()` per pipe run, stamped into the `child_metadata` that rides in every activity and child-workflow argument. Note the *child workflow id* was already fixed with `workflow.uuid4` (`temporal_pipe_router.py:72`) — the run id needs the same treatment.-- `pipelex/core/pipes/pipe_abstract.py:650` — OTel span creation inline: worker-local telemetry singleton decides span-vs-None and the SDK's random span-id generator feeds `child_metadata.otel_context` (low).-- `pipelex/graph/graph_tracer.py:250` — `datetime.now(timezone.utc)` timestamps in every trace event → flush activity argument (low; timestamps never gate emission).--**Fix shape for the whole cluster:** an injectable replay-safe id/clock provider that returns `workflow.uuid4()` / `workflow.now()` when inside a Temporal workflow, plain `shortuuid`/`uuid4`/`datetime.now` outside.-------## LOW — test workflows (`test_extras/`, flaky-test risk only)--- `wf_test_child_pipe.py:43` — mutates the worker-global hub pipe library inline with hardcoded pipe codes; `add_new_pipe` raises on duplicates, so any replay or second run on the same worker fails inline before recorded commands. One verifier found these two workflows are currently **orphaned** (registered in `PIPELEX_TEMPORAL_TEST_WORKFLOWS` but not exercised by any test), which caps urgency.-- `wf_test_child_pipe.py:48` (and the other test workflows) — `JobMetadata` constructed inline, so `started_at`'s `default_factory=datetime.now` (`pipeline/job_metadata.py:91`) fires on the workflow thread and rides into every activity argument. The root is the naive `default_factory` itself — already tracked as a P0.2 follow-on in [README.md](README.md).-- `wf_test_content_generator_child.py:63,71` — worker-local model-deck alias resolution (`$testing-text` / `$testing-structured`) inline; a worker missing the alias raises before any downstream activity is scheduled.-------## Refuted candidates (verified non-breaches — don't re-chase)--- **`temporal_error.py:198` config read of non-retryable name lists** — reachable workflow-side via `from_app_error`, but only sets log severity and the `non_retryable` flag on an unconditionally-raised error; no command divergence.-- **Kajson global `ClassRegistry` accumulation (`wf_pipe_router.py:89`)** — the only writes to the global registry happen at deterministic worker boot; per-workflow registries are seeded fresh.-- **`conditional_worker.py` INTERNAL branch (shortuuid + instance state)** — unreachable inside workflow bodies; the hub router that workflow-thread code resolves is always built EXTERNAL.-- **`wf_pipe_router.py:241` teardown raising on missing library** — `open_library` is idempotent and runs at line 96 on every replay before teardown can be reached.-- **`SubPipe` hub pipe lookup (`sub_pipe.py:45`)** — payload-derived on every supported crate-backed dispatch (the genuine gap is the `PipeCondition` no-crate path, M4).-- **`pipe_run_params_factory.py:19` `pipe_stack_limit` fallback** — production params are fully payload-derived; only the orphaned test workflows hit the config fallback.-- **`pipe_compose.py:292` dry generator selection** — the chosen generator is never invoked on any `PipeCompose` path.-- **`submitter_hydration.py:58` `uuid4`** — names a worker-local scoped library only; never reaches a command.-- **`pipe_abstract.py:474` `datetime.now` in `_run_pipe_traced`** — timestamps land only in trace-event activity args, never gate emission (subsumed by the clock-cluster note above).-------## Suggested fix order--1. ~~**H1 + M1 together**~~ — **DONE**: `act_flush_trace_events` scheduling is a pure function of `trace_context` (always scheduled when present; activity no-ops on empty events), `open_tracer` is collision-proof, and activity-side usage events never route through the in-sandbox buffer. This finishes PR #984.-2. ~~**H2**~~ — **DONE**: `open_fresh_library` force-teardown of leaked predecessor state, plus eviction-safe cleanup ordering in the `finally` (worker-local cleanup before the awaited flush).-3. **Dispatch options to payload** — resolve `DispatchOptions` at the submission boundary and carry them in `PipeJob` (`ContentGeneratorInWorkflow` then reads only the payload), closing M2's sibling and the biggest argument-drift surface in one move.-4. **Replay-safe id/clock provider** for the randomness cluster.-5. **M3/M4/M5** individually as the dry-run-as-activity and PipeFunc-as-activity work lands.diff --git a/wip/doctor-output-ansi-under-force-color.md b/wip/doctor-output-ansi-under-force-color.mddeleted file mode 100644index 09db34c21..000000000--- a/wip/doctor-output-ansi-under-force-color.md+++ /dev/null@@ -1,45 +0,0 @@-# `pipelex doctor` — ANSI in captured output under `FORCE_COLOR`, and whether to offer a plain mode--**Status:** deferred to a follow-up PR — investigation done, decision pending. **Low priority.**-**Origin:** surfaced while bringing `pipelex-worker` onto the keyword-only branch pin (PR #23). Its `pipelex doctor` config-health smoke test failed because the captured output contained ANSI codes that defeated a section regex.--## TL;DR (corrected conclusion)--**This is not a pipelex defect.** `pipelex doctor` already honors rich's standard, env-driven color detection: when stdout is not a TTY it emits **plain text**, and it respects both `NO_COLOR` and `FORCE_COLOR`. The ANSI codes we saw in captured `doctor` output came from **`FORCE_COLOR=3` set in the agent/session environment**, which rich (correctly) obeys even when stdout is piped. In a vanilla CI/piped context with no `FORCE_COLOR`, `doctor` output is already plain and the worker's original regex would have matched.--So there is **likely nothing to fix in pipelex**. The only open question is an optional ergonomics one (below).--## Evidence--`pipelex doctor` run against the pipelex-worker `.pipelex/` config, output redirected to a file (non-TTY), counting ANSI escape sequences (`\x1b[…m`):--| Condition | ANSI escapes | `Configuration Files ✓` regex matches? |-| --- | ---: | --- |-| `FORCE_COLOR=3` set (this agent env) | 114 | no |-| `FORCE_COLOR` unset, piped (vanilla CI-like) | 0 | **yes** |-| `FORCE_COLOR` unset + `NO_COLOR=1` | 0 | yes |--With `FORCE_COLOR` set, adding `NO_COLOR=1` dropped the color codes but left bold/reset (`\x1b[1m`/`\x1b[0m`) — rich still considers the stream a terminal, so it keeps non-color styling. That residual reset between `Configuration Files` and the `✓` is what broke the regex.--The console is built without any `force_terminal`: `pipelex/cli/commands/doctor_cmd.py` renders via `get_console()`, which in `pipelex/hub.py` returns a plain `Console(file=sys.stdout)` / `Console(stderr=True)`. Rich's env-driven detection does the rest. (For contrast, `pipelex/tools/misc/pretty.py` already pins `force_terminal=False` on its export consoles — the right call for capture.)--## What we already did (no further action needed for it)--`pipelex-worker/tests/test_config_health.py` now strips ANSI before matching (PR #23). That is a good **defensive** fix and should stay regardless of the decision below: agent/CI runners sometimes set `FORCE_COLOR`, and a smoke test that asserts on captured CLI text should normalize ANSI so it isn't hostage to the runner's color env.--## The genuine (optional) question for the next PR--Should a **diagnostic** command like `pipelex doctor` — whose output is frequently captured (CI smoke tests, support bundles, log scrapers) — produce deterministic plain output even when `FORCE_COLOR` is set in the environment?--- **(a) Do nothing.** Current behavior is standard and idiomatic; consumers that capture `doctor` output normalize ANSI themselves (the worker now does). **Recommended baseline** unless we get bitten again.-- **(b) Add a `--plain` / `--no-color` flag** to `doctor` (and perhaps other diagnostic commands) that forces a non-styled console for the command's duration — e.g. `Console(file=sys.stdout, no_color=True, force_terminal=False)`. Small, self-contained, nicer for CI/scripting; lets callers get clean output without env juggling. Pick this if we want first-class machine-friendly diagnostics.-- **(c) Make `doctor` ignore `FORCE_COLOR` and key purely off `isatty`.** **Rejected** — it overrides the user's explicit `FORCE_COLOR` and is surprising.--**Recommendation:** ship **(a)** as the baseline (i.e. nothing), and reach for **(b)** only if/when we want first-class capturable diagnostics or hit this again.--## Where to look if we do (b)--- `pipelex/cli/commands/doctor_cmd.py` — the doctor command; renders via `get_console()` with rich markup.-- `pipelex/hub.py` — `get_console()` (~`:255`), `set_console_print_target` / `set_console` (~`:148`). A `--plain` flag would set a `Console(no_color=True, force_terminal=False, file=sys.stdout)` on the hub for the command, then restore.-- `pipelex/cli/agent_cli/commands/doctor_cmd.py` — the agent-CLI twin; keep behavior consistent if we add a flag.-- Precedent for capture-safe consoles: `pipelex/tools/misc/pretty.py` (`force_terminal=False`).diff --git a/wip/dry-run-refactor/00-questions.md b/wip/dry-run-refactor/00-questions.mddeleted file mode 100644index 844a4cc85..000000000--- a/wip/dry-run-refactor/00-questions.md+++ /dev/null@@ -1,50 +0,0 @@-> **Historical origin of the audit.** These are the three questions that kicked off the investigation. All three answers still hold today (see `C-synthesis.md` / `A-taxonomy.md`): `DryPipeRouter` is still dead code, the taxonomy is unchanged, the load profile is still light. One thing to note for context: the audit's *motivation* — a dry-run regression in `validate_bundle` — has since been closed by the signature-validation feature. Kept verbatim as the record of what was originally asked.--# Open questions from user--## Q1 — Is DryPipeRouter useless?-User's claim: dry-run dispatch happens at the pipe level (`PipeAbstract._run_pipe_traced`-match on `pipe_run_params.run_mode`). So a router that calls `pipe.dry_run_pipe()`-instead of `pipe.run_pipe()` adds nothing — the regular router would route to the-same dry path via the in-pipe match.--Need to verify: is `pipe.run_pipe()` the one that does the match, and does it call-`dry_run_pipe` itself when mode==DRY? If yes, DryPipeRouter is dead code.--## Q2 — Taxonomy of PipeRouter / PipeRun / WfPipeRouter / WfPipeRun-User is lost. Need a clean map:-- What is each class-- Who calls whom-- What's the swap mechanism in `Pipelex.make()` / hub when `is_temporal_enabled=True`-- Is there a way to inject DIFFERENT runners for different cases (e.g. dry-run- stays local even when temporal is on)?--## Q3 — CPU/memory load of a dry-run in FastAPI-The use case: a FastAPI endpoint receives thousands of concurrent dry-run requests-(different workflows, different users). Question: is dry-run heavy enough that we'd-need to push it through Temporal too, or is it light enough to stay in-process in-the API?--Need to characterize:-- What does dry-run actually DO computationally- - Parses MTHDS bundle (CPU)- - Builds blueprints / Pydantic models (CPU + allocation)- - Loads concept classes (CPU + import side-effects?)- - Mocks inputs (cheap)- - Walks the pipe graph (recursion, async coroutines)- - At leaves: ContentGeneratorDry (returns canned text — cheap, no network)- - Validates Jinja2 templates (CPU)-- What it does NOT do- - LLM calls (no)- - HTTP / external API (no, modulo ContentGeneratorDry behavior — verify)- - File I/O for storage (no — unless DeliveryExecutor is on)-- Is it bound by:- - Single-thread CPU? (Python GIL — yes if pure-Python computation)- - Event loop blocking? (any sync CPU work blocks the loop)- - Memory? (Pydantic model graphs can be heavy — bundle size matters)- - File system? (bundle loading from disk)--## Plan-- Dispatch Agent A → Q1 + Q2 (taxonomy + DryPipeRouter verification)-- Dispatch Agent B → Q3 (load characterization)-- Synthesize findings into a recommendationdiff --git a/wip/dry-run-refactor/A-taxonomy.md b/wip/dry-run-refactor/A-taxonomy.mddeleted file mode 100644index fc30fe74e..000000000--- a/wip/dry-run-refactor/A-taxonomy.md+++ /dev/null@@ -1,366 +0,0 @@-> **Background — still structurally accurate; line numbers are as of ~v0.29.1.** The four-class taxonomy, the call chains, the hub swap, and the `DryPipeRouter` dead-code proof were re-verified against the live branch and still hold (the hub swap is now at `pipelex.py:446–461`; `PipelexRunner(pipe_run=…)` injection is unchanged). What this doc predates is the **signature-validation** feature — see the new [§7 Signature-validation deltas](#section-7-signature-validation-deltas-added-since-this-doc) at the end for the surface it added to the dry-run path.--# Runtime Classes Taxonomy: PipeRouter, PipeRun, and Temporal Variants--## Section 1: Four Classes Side by Side--### PipeRouter (`pipelex/pipe_run/pipe_router.py:9`)-**What it does:** Direct-execution router; calls `pipe.run_pipe()` inline (with all modes—LIVE and DRY—baked into the pipe's logic).--**Public interface:**-```python-async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput-async def run(self, pipe_job: PipeJob) -> PipeOutput # inherited from protocol-```--**Execution:** Calls `pipe.run_pipe()` at line 18, passing `pipe_job` and delegating to the pipe's implementation. No redirection; the pipe itself interprets `pipe_run_params.run_mode`.-------### PipeRun (`pipelex/pipe_run/pipe_run.py:21`)-**What it does:** Orchestrator for direct execution; wraps `PipeRouter.run()` with delivery, graph assembly, and error handling.--**Public interface:**-```python-async def run(self, pipe_job: PipeJob, delivery_assignment: DeliveryAssignment | None = None) -> PipeOutput-```--**Execution:** At line 40, calls `self._pipe_router.run(pipe_job)`, then executes delivery at line 71 if assignment is provided. Centralizes tracing cleanup (lines 47–58) and graph assembly (line 61).-------### WfPipeRouter (`pipelex/temporal/tprl_pipe/wf_pipe_router.py:26`)-**What it does:** Temporal workflow that executes the pipe; dispatched as a child workflow by `TemporalPipeRouter`. Handles per-workflow library setup, tracing, and event flushing.--**Public interface:**-```python-async def run(self, workflow_arg: PipeJob) -> PipeOutput-```--**Execution:** At line 126, calls `pipe.run_pipe()` directly (same as `PipeRouter`). All the heavyweight setup (library hydration, tracing manager, event buffering) wraps it, then the workflow dehydrates the output for transit (line 180).-------### WfPipeRun (`pipelex/temporal/tprl_pipe/wf_pipe_run.py:22`)-**What it does:** Parent workflow orchestrating pipe execution + delivery; runs `WfPipeRouter` as a child, then runs delivery as an activity. Failure-safe: delivery fires even if pipe fails.--**Public interface:**-```python-async def run(self, workflow_arg: PipeRunArg) -> PipeOutput-```--**Execution:** At line 53, executes `WfPipeRouter.run()` as a child workflow; at line 83, runs graph assembly as an activity; at line 116, runs delivery as an activity. Preserves original error for correct failure attribution (line 132).-------### DryPipeRouter (`pipelex/pipe_run/dry_pipe_router.py:9`)-**What it does:** Dead code; calls `pipe.dry_run_pipe()` at line 18. No production usage.--**Public interface:**-```python-async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput-```-------### Protocols--**`PipeRouterProtocol` (`pipelex/pipe_run/pipe_router_protocol.py:11`):**-```python-observer: ObserverProtocol-async def run(self, pipe_job: PipeJob) -> PipeOutput-async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput # abstract-```--Implemented by `PipeRouter`, `DryPipeRouter`, and `TemporalPipeRouter`.--**`PipeRunProtocol` (`pipelex/pipe_run/pipe_run_protocol.py:12`):**-```python-async def run(- self,- pipe_job: PipeJob,- delivery_assignment: DeliveryAssignment | None = None,-) -> PipeOutput-```--Implemented by `PipeRun` and `TemporalPipeRun`.-------## Section 2: Call Chains--### Local (Direct) Mode--```-PipelexRunner.execute_pipeline()- → pipeline_run_setup() [builds PipeJob]- → effective_pipe_run.run(pipe_job, delivery_assignment)- [calls get_pipe_run() if not overridden; PipeRun instance]--PipeRun.run() [pipelex/pipe_run/pipe_run.py:29–91]- → self._pipe_router.run(pipe_job) [line 40]-- PipeRouter.run() [from PipeRouterProtocol.run]- → self._before_run(pipe_job) [observer hook, line 52]- → self._run_pipe_job(pipe_job) [line 55]-- PipeRouter._run_pipe_job() [pipelex/pipe_run/pipe_router.py:14–24]- → pipe_job.pipe.run_pipe() [line 18]-- PipeAbstract.run_pipe() [pipelex/core/pipes/pipe_abstract.py:409]- → pipe_run_params.push_pipe_to_stack() [line 421]- → self._run_pipe_traced() [line 423]-- PipeAbstract._run_pipe_traced() [pipelex/core/pipes/pipe_abstract.py:434]- → validate_before_run() [line 501]- → match pipe_run_params.run_mode: [line 504]- case PipeRunMode.LIVE: live_run_pipe() [line 506]- case PipeRunMode.DRY: dry_run_pipe() [line 514]- → pipe_run_params.pop_pipe_from_stack() [line 431]-- → self._after_successful_run() [observer hook, line 74]-- → self._delivery_executor.execute() [line 71]- → assemble_graph_on_output() [line 61]- → GraphTracerManager.close_tracer() [line 50]-```--**Key:** The `run_mode` decision point is inside `pipe._run_pipe_traced()` at line 514, not at the router level. `DryPipeRouter` is an **unreachable alternative.**-------### Temporal Mode--```-PipelexRunner.execute_pipeline()- → pipeline_run_setup()- → effective_pipe_run.run(pipe_job, delivery_assignment)- [calls make_temporal_pipe_run(); TemporalPipeRun instance]--TemporalPipeRun.run() [pipelex/temporal/tprl_pipe/temporal_pipe_run.py:49–81]- → stamp_submitter_session_id() [line 57]- → WorkflowExecutorFactory.create_executor() [line 64]- → executor.execute_workflow(workflow_class=WfPipeRun, ...) [line 69]-- [Temporal server executes WfPipeRun]-- WfPipeRun.run() [pipelex/temporal/tprl_pipe/wf_pipe_run.py:31–139]- → workflow.execute_child_workflow(WfPipeRouter.run, pipe_job) [line 53]-- [Temporal server executes WfPipeRouter]-- WfPipeRouter.run() [pipelex/temporal/tprl_pipe/wf_pipe_router.py:29–183]- [library hydration, tracing setup, per-workflow state initialization]- → pipe.run_pipe() [line 126]- [SAME pipe execution logic as local mode; run_mode baked in]- [event flushing, tracer close]-- → workflow.execute_activity(act_assemble_graph) [line 83]- → workflow.execute_activity(act_deliver) [line 116]-- → rehydrate_pipe_output_with_crate() [line 81]-```--**What swaps:**-- `PipeRun` ↔ `TemporalPipeRun`: delivery orchestration layer (hub selection at line 456–461 in `pipelex.py`)-- `PipeRouter` ↔ `TemporalPipeRouter`: pipe dispatch mechanism (hub selection at line 447–452 in `pipelex.py`)-- **Graph assembly, delivery:** move from direct sync calls → Temporal activities (lines 83, 116 in `WfPipeRun`)-- **Per-workflow tracing:** added inside the workflow (lines 72–123 in `WfPipeRouter`)--**What stays the same:**-- `pipe.run_pipe()` still dispatches to `PipeRunMode.DRY` vs `.LIVE` at the pipe level (no router-level decision)-- `pipe_job` structure and semantics-------## Section 3: The Hub Swap--### Config Check-`Pipelex.setup()` at `pipelex/pipelex.py:162`, checks temporal enablement at line 189:-```python-# Line 188–192-if temporal_enabled is not None:- config = get_config()- updated_temporal = config.temporal.model_copy(update={"is_enabled": temporal_enabled})- config.temporal = updated_temporal-```--Then at line 428:-```python-if get_config().temporal.is_enabled:- # [Temporal task manager setup]-```--### Pipe Router Registration-**Lines 443–452** (`pipelex.py`):-```python-# --- Pipe Router -------------------------------------------------------------------------if pipe_router:- self.pipelex_hub.set_pipe_router(pipe_router)-elif get_config().temporal.is_enabled:- from pipelex.temporal.tprl_pipe.temporal_pipe_router import make_temporal_pipe_router # noqa: PLC0415- self.pipelex_hub.set_pipe_router(make_temporal_pipe_router())-else:- self.pipelex_hub.set_pipe_router(PipeRouter(observer=multi_observer))-```--**Injection order:** explicit parameter > temporal enabled > default (direct).--### Pipe Run Registration-**Lines 454–461** (`pipelex.py`):-```python-# --- Pipe Run ----------------------------------------------------------------------------if get_config().temporal.is_enabled:- from pipelex.temporal.tprl_pipe.temporal_pipe_run import make_temporal_pipe_run # noqa: PLC0415- self.pipelex_hub.set_pipe_run(make_temporal_pipe_run())-else:- self.pipelex_hub.set_pipe_run(PipeRun(pipe_router=self.pipelex_hub.get_required_pipe_router()))-```--**Note:** `PipeRun` takes the already-registered router as a constructor parameter (line 461).-------## Section 4: DryPipeRouter Verification--### Question-When `PipeRouter.run()` calls `pipe.run_pipe()` with `pipe_run_params.run_mode == PipeRunMode.DRY`, does the pipe execute its dry-run path anyway?--### Answer: YES (Confirmed Dead Code)--The decision happens inside the pipe, not at the router.--**`PipeAbstract._run_pipe_traced()` at `pipelex/core/pipes/pipe_abstract.py:504–520`:**-```python-match pipe_run_params.run_mode:- case PipeRunMode.LIVE:- pipe_output = await self.live_run_pipe(- job_metadata=job_metadata,- working_memory=working_memory,- pipe_run_params=pipe_run_params,- output_name=output_name,- library_crate=library_crate,- )- case PipeRunMode.DRY:- pipe_output = await self.dry_run_pipe(- job_metadata=job_metadata,- working_memory=working_memory,- pipe_run_params=pipe_run_params,- output_name=output_name,- library_crate=library_crate,- )-```--**Therefore:**-- `PipeRouter` with `run_mode=DRY` → `pipe.run_pipe()` → `dry_run_pipe()` ✓ Works-- `DryPipeRouter` → `pipe.dry_run_pipe()` → Bypasses the pipe's own mode dispatch--### Usage of DryPipeRouter--Search results for non-test production code:-```-pipelex/pipe_run/dry_pipe_router.py:class DryPipeRouter(PipeRouterProtocol):-```--**No production instantiation found.** Only appears in its own definition. **Dead code confirmed** (re-verified on the live branch — `grep -rn DryPipeRouter` still finds only the class definition).-------## Section 5: Per-Request Runner Injection--### Can We Inject Per-Request Runners?--Yes. `PipelexRunner` accepts an optional `pipe_run` parameter.--**`PipelexRunner.__init__()` at `pipelex/pipeline/runner.py:53–72`:**-```python-def __init__(- self,- library_id: str | None = None,- library_dirs: list[str] | None = None,- bundle_uris: list[str] | None = None,- pipe_run_mode: PipeRunMode | None = None,- search_domain_codes: list[str] | None = None,- user_id: str | None = None,- execution_config: PipelineExecutionConfig | None = None,- pipe_run: PipeRunProtocol | None = None, # <-- HERE-):- # ...- self._pipe_run = pipe_run-```--**Usage at line 152:**-```python-effective_pipe_run = self._pipe_run or get_pipe_run()-pipe_output = await effective_pipe_run.run(pipe_job, delivery_assignment=delivery_assignment)-```--### Implication-The hub design **is compatible** with per-request injection. A user can:-1. Create a `PipelexRunner` with `pipe_run=direct_runner` while temporal is globally enabled-2. That call uses the injected direct runner, not the temporal one from the hub--**Limitation:** The router is not similarly injectable into `PipelexRunner`. Only the top-level `PipeRun` is. To swap the router per-request, you'd need to either:-- Inject the entire `PipeRun` (which carries a router)-- Or extend `PipelexRunner` to accept `pipe_router` and pass it to `PipeRun` construction-------## Section 6: Smells and Inconsistencies--1. **Naming asymmetry: `Wf` prefix**- - `WfPipeRouter` and `WfPipeRun` are Temporal workflows; `TemporalPipeRouter` and `TemporalPipeRun` are submitter-side executors.- - The naming does not reflect this tier distinction. `Wf*` blurs whether you're looking at a workflow (server-side) or an executor (client-side).- - **Suggestion:** Rename `WfPipeRouter` → `WorkflowPipeRouter` and `WfPipeRun` → `WorkflowPipeRun` for clarity; or rename the temporal executors to make the distinction explicit.--2. **Protocol enforcement gap**- - `PipeRouterProtocol` and `PipeRunProtocol` exist but are not enforced as boundaries. Both protocols have implementations in direct and temporal tracks, yet there's no factory abstraction to select between them.- - Swap logic is hardcoded in `Pipelex.setup()` (lines 447–461) rather than delegated to a factory that respects the protocol.- - **Suggestion:** Add `RouterFactory` and `RunFactory` protocols that return `PipeRouterProtocol` and `PipeRunProtocol` based on config.--3. **DryPipeRouter is unreachable**- - Dead code sitting in the codebase; no path calls it.- - Run mode is a pipe-level concern, not a router-level one.- - **Suggestion:** Delete it or move the logic into a test-only module.--4. **Library and tracing setup scattered**- - Library hydration lives in `WfPipeRouter` (lines 51–65).- - Tracing setup lives in `WfPipeRouter` (lines 72–123).- - Graph assembly lives in `WfPipeRun` (lines 83–97).- - These are orthogonal concerns (library, tracing, graph) but their setup/teardown is mixed into the pipe execution workflow.- - **Suggestion:** Extract library and tracing setup into reusable context managers (e.g., `async with setup_workflow_library()`, `async with setup_workflow_tracing()`).--5. **Observable inconsistency**- - `PipeRun` receives `self._pipe_router` as a constructor dependency (line 24).- - `TemporalPipeRun` does not receive a router; it hardcodes `WfPipeRouter` (line 54 in temporal_pipe_run).- - This breaks symmetry: if you want to swap the router in temporal mode, you can't without rewriting `TemporalPipeRun`.- - **Suggestion:** `TemporalPipeRun` should accept a workflow class parameter (defaulting to `WfPipeRouter`).--6. **`PipelexRunner` vs. hub coupling**- - `PipelexRunner` can inject `pipe_run` but not `pipe_router`.- - The router is chosen at the hub level, not at the runner level.- - Per-request router swapping (e.g., dry-runs stay local even when temporal is global) requires injecting the entire `PipeRun`, not just the router.- - **Suggestion:** Add `pipe_router` parameter to `PipelexRunner.__init__()` and construct `PipeRun` locally if either is overridden.--7. **Magic string comparison in delivery**- - Both `PipeRun` and `WfPipeRun` check `if delivery_assignment is not None` to decide whether to deliver.- - This is ad-hoc; there's no delivery mode enum (e.g., `DeliveryMode.ENABLED` vs `DeliveryMode.SKIPPED`).- - **Suggestion:** Add a delivery mode parameter to `PipeRunParams` or `PipeJob` for consistency with `run_mode`.-------## Section 7: Signature-validation deltas (added since this doc)--The original taxonomy predates the **signature-validation** feature, which landed on the live branch and re-wired the dry-run path. None of it changes the four-class taxonomy above (routers/runs are untouched), but it heavily changes `pipelex/pipe_run/dry_run.py` — the module the consolidation plan wants to delete. What's new:--- **A third pipe category, `PipeSignature`** (`pipelex/pipe_signature/`). A contract-only pipe (`type = "PipeSignature"`) declaring `inputs` / `output` / `description` with no body. Its `_dry_run_pipe` **mints a mock `Stuff`** from the declared output concept via `WorkingMemoryFactory.make_mock_content`; its `_live_run_pipe` raises `PipeSignatureNotExecutableError`.-- **`dry_run.py` is now the home of the strict signature pre-check.** `dry_run_pipe` and `dry_run_pipes` gained an `allow_signatures` parameter. When `allow_signatures=False` (the default — both CLIs are strict by default), `dry_run_pipes` walks every pipe with `collect_signature_refs` / `collect_signature_paths` (free functions in `pipe_signature/signature_walk.py`, kept off `PipeAbstract` to avoid an import cycle) and raises a **single aggregated `SignaturesNotAllowedError`** listing every reachable signature plus the controller dep-chain. This aggregation-into-one-error is a deliberate UX property the consolidation must preserve.-- **`dry_run.py` now exports a helper that `pipe_signature.py` imports.** `convert_stuff_spec_to_typed_named` (used to resolve a signature's declared output to a typed mock) lives in `dry_run.py` and is imported by `pipelex/pipe_signature/pipe_signature.py`. So `dry_run.py` is no longer a leaf the plan can simply delete — there is a hard dependency from the signature runtime into it.-- **`validate_bundle` threads `allow_signatures` and `dry_run_pipe_codes`.** The latter (with the `_pipes_to_dry_run` selector) implements the `--pipe` single-slice path: load the whole bundle so deps resolve, but dry-run only the selected pipe. `ValidateBundleError` gained `signature_check_error` to carry the strict refusal up to the CLI for friendly rendering.-- **Callsites that now thread `allow_signatures`:** `cli/commands/validate/_validate_core.py`, `cli/agent_cli/commands/validate/_validate_core.py`, `builder/operations/validate_ops.py`, `builder/operations/runner_code_ops.py`, plus `validate_bundle.py` itself. These are the same callsites the consolidation wants to migrate to `PipelexRunner` — they now carry an extra parameter each.--**Implication for the plan:** routing dry-run through `PipelexRunner` means `run_mode == DRY` is no longer the only mode-shaped concern on that path — `allow_signatures` (strict vs lenient) and `dry_run_pipe_codes` (which pipes to dry-run) ride alongside it, and the batch-level strict pre-check needs a home that runs *before* per-pipe execution. See `D-plan.md` for how this reshapes the design.diff --git a/wip/dry-run-refactor/B-load-profile.md b/wip/dry-run-refactor/B-load-profile.mddeleted file mode 100644index 4ee2d470d..000000000--- a/wip/dry-run-refactor/B-load-profile.md+++ /dev/null@@ -1,275 +0,0 @@-> **Background — still valid.** The CPU-only / no-network / no-disk characterization of the dry-run path holds. One addition since this was written: signature pipes now mint their declared mock output via `WorkingMemoryFactory.make_mock_content` (polyfactory, same cheap cost class as the mock-input generation profiled in Stage 3). No new I/O, no new locks. The verdict — dry-run is safe in-process even under high concurrency — is unchanged.--# Dry-Run Load Profile: CPU/Memory/IO Analysis--## Executive Summary--A dry-run in Pipelex is **CPU-moderate, memory-light, with zero external I/O**. It is safe to keep in-process even under thousands of concurrent requests on a single FastAPI worker. The main cost is Pydantic validation and Jinja2 template rendering, both of which are synchronous but brief. Exceeding `uvicorn` worker limits will be a concurrency/GIL concern, not a throughput one.-------## Stage 1: MTHDS Parsing + Blueprint Construction--**Entry Point:** `PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=...)`-**File:** `pipelex/core/interpreter/interpreter.py:21-60`--```python-@classmethod-def make_pipelex_bundle_blueprint(cls, bundle_path: Path | None = None, mthds_content: str | None = None) -> PipelexBundleBlueprint:- blueprint_dict: dict[str, Any]- try:- if bundle_path is not None:- blueprint_dict = load_toml_from_path(path=str(bundle_path))- blueprint_dict[PIPELEX_BUNDLE_BLUEPRINT_SOURCE_FIELD] = str(bundle_path)- elif mthds_content is not None:- blueprint_dict = load_toml_from_content(content=mthds_content)- # ...- pipelex_bundle_blueprint = PipelexBundleBlueprint.model_validate(blueprint_dict)-```--**What happens:**-1. **TOML parsing** (synchronous): `load_toml_from_content()` parses the TOML string using the standard library `tomllib`. This is pure Python, no I/O beyond the input string. **Cost:** proportional to bundle size (typically <1 MB for a bundle).-2. **Pydantic validation** (CPU-bound): `PipelexBundleBlueprint.model_validate()` validates the dict against ~327 lines of nested Pydantic models. No disk I/O, no network. **Cost:** O(bundle size), typically 1-5 ms for a small bundle.-3. **Bundle elaboration** (CPU): `BundleElaborator.elaborate()` rewrites `preliminary_text` structuring into synthetic pipes. **Cost:** O(number of pipes), typically <1 ms.--**Classification:** LIGHT (pure Python text parsing + Pydantic validation, single-digit milliseconds)--**I/O Check:** ✅ No disk reads, no network calls, no subprocess invocations.-------## Stage 2: `pipeline_run_setup` — Library and Concept Loading--**File:** `pipelex/pipeline/pipeline_run_setup.py:50-336`--```python-async def pipeline_run_setup(- execution_config: PipelineExecutionConfig,- library_id: str | None = None,- # ... many params ...-) -> tuple[PipeJob, str, str]:- library_manager = get_library_manager()- set_current_library(library_id=library_id)- library_manager.open_library(library_id=library_id)-- if mthds_contents:- all_blueprints = [PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=content) for content in mthds_contents]- # ... load blueprints into library ...- library_manager.load_from_blueprints(library_id=library_id, blueprints=blueprints_to_load)-```--**What happens:**-1. **Per-request library scope**: Each dry-run call gets its own `library_id` (defaults to `pipeline_run_id`), so libraries are isolated per request. No global cache contention. (Line 139-144)-2. **Bundle parsing**: Blueprints are constructed from MTHDS contents (already profiled above).-3. **Library loading**: `library_manager.load_from_blueprints()` registers concepts and pipes into the current library instance. **Cost:** O(bundle complexity), typically <5 ms.-4. **Pipe lookup**: `get_required_pipe(pipe_code=...)` retrieves the compiled pipe from the library. **Cost:** O(1) dictionary lookup.-5. **Graph tracer setup** (if `is_generate_graph=True`): Creates a `GraphContext` with metadata. (Line 224-233) **Cost:** lightweight object construction, <1 ms.-6. **Telemetry**: Empty on dry-run (conditional on `is_live`), so no I/O. (Line 292)--**Memory Impact:** Each library instance holds a copy of the loaded bundle's pipe specs and concepts. For a typical 10–20 pipe bundle, ~0.5–2 MB per instance. The library lives for the duration of the request and is cleaned up on exit.--**Classification:** LIGHT to MODERATE (dominated by blueprint parsing already measured; library loading is in-memory only)--**Shared State Check:** ✅ Libraries are **scoped per library_id**. No global contamination from concurrent requests.-------## Stage 3: Mock Input Construction--**File:** `pipelex/pipe_run/dry_run.py:61-62` and `pipelex/core/memory/working_memory_factory.py:91-196`--```python-needed_inputs_for_factory = convert_to_working_memory_format(needed_inputs_spec=pipe.needed_inputs())-working_memory = WorkingMemoryFactory.make_mock_inputs(needed_inputs=needed_inputs_for_factory)-```--**What happens:**-1. **Concept class registry lookup**: `convert_to_working_memory_format()` calls `get_class_registry().get_class(name=structure_class_name)` (Line 167). This is a dictionary lookup in an in-memory registry. **Cost:** O(1), <0.1 ms.-2. **Mock object generation via polyfactory**: `DryRunFactory.make_dry_run_factory()` (file: `pipelex/cogt/content_generation/dry_run_factory.py`) creates a `polyfactory` ModelFactory that generates mock Pydantic objects with smart field strategies (e.g., snake_case fields, PascalCase, Literal choices, etc.). **Cost:** per-object factory creation ~1 ms, per-object instantiation (`.build()`) ~1–5 ms depending on nesting.-3. **Mock data is pure objects**: No file I/O, no network. For lists, creates N objects (default 3 items per list). (Line 166)--**Memory Impact:** One `WorkingMemory` instance with mock Stuff objects. For a typical dry-run with 2–3 inputs, ~0.1–0.5 MB.--**Classification:** LIGHT (object allocation and Pydantic factory instantiation, single-digit milliseconds per input)-------## Stage 4: Pipe Graph Walk in DRY Mode--**File:** `pipelex/pipe_operators/llm/pipe_llm.py:394-407` (and other operators)--### **PipeLLM (the critical one)**--```python-async def _dry_run_operator_pipe(self, job_metadata: JobMetadata, working_memory: WorkingMemory,- pipe_run_params: PipeRunParams, output_name: str | None = None) -> PipeLLMOutput:- return await self._live_run_operator_pipe(- job_metadata=job_metadata,- working_memory=working_memory,- pipe_run_params=pipe_run_params,- output_name=output_name,- content_generator=ContentGeneratorDry(), # <-- MOCK generator, NOT real LLM- )-```--The dry-run delegates to the live path but replaces the `content_generator` with a mock.--**ContentGeneratorDry:** `pipelex/cogt/content_generation/content_generator_dry.py`--```python-@override-async def make_llm_text(self, job_metadata: JobMetadata, llm_setting_main: LLMSetting, llm_prompt_for_text: LLMPrompt) -> str:- # No network call, just returns a string with truncated prompt info- return f"DRY RUN: make_llm_text • llm_setting={llm_setting_main.desc()} • prompt={prompt_truncated}"--@override-async def make_object(self, job_metadata: JobMetadata, object_class: type[BaseModelTypeVar], ...) -> BaseModelTypeVar:- object_factory = DryRunFactory.make_dry_run_factory(object_class)- return object_factory.build() # Polyfactory generation, no LLM call-```--**Classification:**-- **Prompt templating (Jinja2):** ~1–5 ms. Synchronous, Python-only.-- **Mock object generation:** ~1–5 ms per object (same as Stage 3).-- **No network call:** ✅ Verified—no `requests`, `aiohttp`, or external I/O.-- **No telemetry I/O:** Dry-run telemetry is synthetic and in-memory. (See `_report_dry_llm_job()`: creates a zero-token LLMJob and reports it locally.)--### **Other Operators (PipeOCR, PipeImg, PipeFunc, PipeJinja2, PipeCompose, PipeSearch, PipeExtract, PipeStructure)**--All follow the same pattern: dry-run methods short-circuit expensive operations.--- **PipeFunc**: Validates input/output types but does NOT execute the user function. Returns a mock result. **Cost:** TRIVIAL.-- **PipeOCR/PipeExtract**: Returns mock `PageContent` objects with example text. **Cost:** LIGHT (object construction).-- **PipeImg/PipeImgGen**: Returns fake `ImageContent` with example URLs. **Cost:** TRIVIAL.-- **PipeJinja2/PipeCompose**: Renders template syntax but not against real backends. **Cost:** LIGHT (Jinja2 parse, typically <2 ms per template).-- **PipeSearch**: Mock search—returns hardcoded results. **Cost:** TRIVIAL.--### **Controllers (PipeSequence, PipeParallel, PipeBatch, PipeCondition)**--All delegate to their `_live_run_controller_pipe()` which recursively runs child pipes in dry mode. No special optimization.--**Cost:** Inherited from child pipes + minimal overhead (O(number of children)).--**Classification:** LIGHT across all operators (Jinja2 + object construction dominate; no network, no file I/O).-------## Stage 5: Graph Tracing (Optional)--**File:** `pipelex/graph/graph_tracer_manager.py`--If `execution_config.is_generate_graph=True`, a `GraphContext` is opened (line 224-233 in `pipeline_run_setup.py`). The graph tracer records node entry/exit and data shapes during the pipe walk.--**Cost per node:** Each traced node emits a dictionary entry with metadata (node ID, pipe name, input/output types, timing). **Cost:** O(1) per node, <0.1 ms per event, no I/O.--**Memory:** One `GraphSpec` dict structure in memory for the run duration (~0.1 MB for 10–50 nodes).--**Classification:** TRIVIAL (in-memory event buffering)-------## Stage 6: Async / Event Loop Concerns--### **Synchronous Blocking Sections**--1. **Pydantic validation (stages 1–2, 3):** Synchronous, CPU-bound. Runs in the event loop thread. **Duration:** 1–10 ms total.-2. **Jinja2 template rendering (stage 4):** Synchronous, CPU-bound. Typical MTHDS prompts are <1000 chars, render time <2 ms.-3. **DryRunFactory.make_dry_run_factory() + .build():** Synchronous, CPU-bound. **Duration:** 1–5 ms per object.--**GIL Impact:** All of these are Python CPU work, so multiple async tasks running on the same event loop will experience GIL contention. However, each individual dry-run is brief (<50 ms typical), so task-switching is acceptable.--### **No Blocking I/O**--✅ **Verified:**-- No `time.sleep()` in dry-run code.-- No `.read()` / `.write()` / `open()` file operations on the dry-run path.-- No subprocess calls (`subprocess.Popen`, `os.system`).-- No blocking network calls (`requests.get()`, `httpx.request()`).--### **Locking**--One lock encountered: `_cache_lock` in `SchemaToModelFactory` (file: `pipelex/cogt/content_generation/schema_to_model_factory.py`).--```python-_cache_lock: ClassVar[threading.Lock] = threading.Lock()--with cls._cache_lock:- if cache_key in cls._schema_cache:- cls._schema_cache.move_to_end(cache_key)- return cls._schema_cache[cache_key]- # ... codegen and exec ...- cls._schema_cache[cache_key] = reconstructed_class-```--**Impact:** This lock protects schema-to-Pydantic-model code generation, which happens when LLM output needs to be structured into a dynamic class. On the dry-run path, `DryRunFactory.make_object()` calls this under the hood **only if a structured output is requested**. For typical dry-runs with text or simple mocks, this is not hit. If hit, contention is brief (milliseconds) and limited to the first miss for a given schema (post-warmup, lock is uncontended).--**Classification:** LIGHT (post-warmup contention is minimal; first-miss contention is acceptable for occasional dry-run calls).-------## Stage 7: Memory Profile--**Per-dry-run memory footprint:**--1. **Bundle blueprint + library instance:** ~1 MB (10–20 pipe bundle).-2. **WorkingMemory (mock inputs):** ~0.1–0.5 MB.-3. **Traversal stack + intermediate results:** ~0.1 MB.-4. **Graph tracer (if enabled):** ~0.1 MB.--**Total per request:** ~1.5–2 MB--**Shared/Global State:**-- ✅ **Concept/pipe registry per library:** Scoped per `library_id`, so no cross-request pollution.-- ✅ **Schema cache:** Global but append-only (LRU, capped at 1024 entries). Safe concurrent access with lock.-- ✅ **Class registry:** Global but immutable after initialization. Safe concurrent read-only access.--**Conclusion:** Memory is lightweight and well-isolated. Holding 1000 concurrent dry-runs in flight would consume ~1.5–2 GB, which is reasonable for a modern FastAPI process.-------## Stage 8: Bottom Line for FastAPI--### **When Could Thousands of Concurrent Dry-Runs Degrade the Process?**--1. **GIL contention from Pydantic + Jinja2:** If you have N concurrent dry-runs on a single-threaded event loop, each paying ~5–50 ms of synchronous CPU work, you can sustain roughly `1000 / 0.05 = 20,000 dry-run/sec` throughput on one event loop. Standard `uvicorn` workers (4–16 per machine) can therefore handle **thousands of concurrent requests** distributed across the worker pool. However, a **single worker** with 100+ concurrent tasks will experience GIL-induced tail latency (p99 > 100 ms) because the event loop cannot context-switch freely.--2. **Memory pressure:** 1000 concurrent dry-runs = ~1.5–2 GB. This is within the memory footprint of a typical container (4–8 GB) but will trigger GC pressure. For a cloud-native setup, this is fine; for a tiny edge instance, consider worker limits.--3. **No I/O bottlenecks:** The dry-run path is I/O-free, so scaling is CPU-limited, not I/O-limited.--4. **Interaction with uvicorn:** `uvicorn` defaults to using the number of CPU cores as worker count. Each worker runs a single asyncio event loop. With 8 cores, you get 8 event loops, each capable of interleaving 50+ concurrent requests (given the brief per-request latency). This is sufficient for typical dry-run workloads.--### **Verdict: Is a Dry-Run "Free"?**--**"Free"** = negligible latency when called in isolation. Not quite.--- **Single dry-run latency:** 5–50 ms (dominated by Pydantic parsing and Jinja2 rendering). This is fast, but not "free."-- **Throughput per worker:** ~20,000 dry-run/sec (assuming 5 ms per dry-run and plenty of CPU cores). Excellent.-- **Memory per dry-run:** ~1.5–2 MB. Negligible compared to a full pipeline run (which may allocate temporary tensors or LLM context, consuming 100 MB+).--**Recommendation:**--✅ **Dry-runs can safely stay in-process** if your API has:-- A reasonable request timeout (5 sec is plenty; 10 sec is conservative).-- Worker limits matching your hardware (standard uvicorn defaults are fine).-- No unusual Pydantic models with deep nesting or custom validators in the critical path.--❌ **Consider offloading to a worker pool / Temporal only if:**-- You expect >10,000 concurrent dry-run requests (risk of GIL-induced latency spikes).-- Your dry-run path includes user-defined pipes with blocking I/O (e.g., `PipeFunc` calling a sync library that does network I/O).-- You need strict SLA tail latency (p99 < 10 ms) under >1000 concurrent load.--**For the current Pipelex use case (1K–100K DAU, typical API load), keeping dry-runs in-process is the right choice.** The latency is acceptable, memory is cheap, and CPU scaling is linear. Offloading would add complexity (Temporal task distribution, serialization) with minimal benefit unless the workload changes dramatically.-------## Unknowns & Caveats--1. **Custom concept classes:** If a user defines a concept with a custom `structure_class` that includes expensive validators or post-processors, mock data generation could be slow. However, Pydantic validation happens regardless of the dry-run path, so this is a bundle-design issue, not a dry-run issue.--2. **Jinja2 template complexity:** The analysis assumes templates are simple (<1000 chars, minimal filter chains). Pathological templates (e.g., deeply nested loops, regex operations) could push render time to 10+ ms. However, benchmarking is needed to quantify.--3. **Library I/O during warm-up:** The analysis assumes the concept/pipe registries are warm (post-import). If Pipelex does lazy-load concept classes from disk, the first dry-run would be slower. Code review did not reveal disk I/O, so this is likely not an issue, but monitoring is recommended.--4. **Concurrency limits on threading.Lock:** If thousands of dry-runs hit the schema codegen cache on first-miss simultaneously, the lock will serialize codegen, potentially bottlenecking. Real-world risk is low (schemas are typically cached), but profiling under load would be wise.-diff --git a/wip/dry-run-refactor/C-synthesis.md b/wip/dry-run-refactor/C-synthesis.mddeleted file mode 100644index 3f31abb8c..000000000--- a/wip/dry-run-refactor/C-synthesis.md+++ /dev/null@@ -1,115 +0,0 @@-> **Background — condensed answers to the founding questions; still accurate, with one update.** The `DryPipeRouter`-is-dead verdict, the taxonomy, the load profile, and the per-request injection mechanism all still hold. The one thing that changed: §5's last bullet (the three abandoned `# TODO` dry-run lines in `validate_bundle.py`) is **no longer true** — that regression was closed by signature-validation. Updated in place below.--# Synthesis — answers to user's questions--## 1. Is DryPipeRouter dead? YES — confirmed.--- `PipeAbstract._run_pipe_traced` (core/pipes/pipe_abstract.py:504-520) does the LIVE/DRY- match. Any router calling `pipe.run_pipe(...)` with `run_mode=DRY` will end up in- `dry_run_pipe()` automatically.-- `DryPipeRouter._run_pipe_job` (pipe_run/dry_pipe_router.py:18) calls- `pipe.dry_run_pipe(...)` directly — same destination, different door.-- Grep finds no production instantiation. The class is defined and never used.-- **Delete it.** It's also conceptually wrong: it says "dry-run is a routing concern"- when it's actually a pipe-execution concern.--## 2. Taxonomy of the four classes--| Class | Role | Side | What it does |-|---|---|---|---|-| `PipeRouter` | router | local | Calls `pipe.run_pipe()`. That's it. Mode-agnostic. |-| `PipeRun` | orchestrator | local | Wraps router + delivery + graph assembly + tracing cleanup. |-| `WfPipeRouter` | router | **server-side workflow** | Same job as PipeRouter, but it IS a Temporal child workflow. Adds library hydration + per-workflow tracing inside the workflow boundary. |-| `WfPipeRun` | orchestrator | **server-side workflow** | Same job as PipeRun, but is a Temporal parent workflow that runs WfPipeRouter as a child and delivery as an activity. |-| `TemporalPipeRouter` | router | **client-side launcher** | Submits WfPipeRouter to Temporal. |-| `TemporalPipeRun` | orchestrator | **client-side launcher** | Submits WfPipeRun to Temporal. |--The naming confusion: `Wf*` = server-side (runs IN the Temporal worker),-`Temporal*` = client-side (the thing that KICKS OFF a workflow from the API).-Worth renaming to `WorkflowPipeRouter` vs `TemporalSubmitterPipeRouter` or similar.--### Hub swap--`Pipelex.setup()` at pipelex.py:443-461:-- Line 447-452: pipe_router → TemporalPipeRouter if `config.temporal.is_enabled`, else PipeRouter-- Line 454-461: pipe_run → TemporalPipeRun if enabled, else PipeRun(router)--**One swap per process at boot.** Not per-request.--### Per-request override--`PipelexRunner.__init__(pipe_run=...)` (runner.py:62) lets you inject a PipeRun.-Line 152: `effective_pipe_run = self._pipe_run or get_pipe_run()`.--So you CAN say "for this request, use the local PipeRun even though the hub default-is the Temporal one." But you can only inject the whole PipeRun, not the router-separately. That's enough for the dry-run case.--## 3. Dry-run load profile — verdict: SAFE IN-PROCESS--| Resource | Per dry-run |-|---|---|-| Latency | 5-50 ms (Pydantic + Jinja2 dominate) |-| Memory | 1.5-2 MB |-| Network | 0 |-| Disk I/O | 0 |-| Subprocess | 0 |-| Throughput/worker | ~20k/sec theoretical |-| Locks | 1 (schema codegen cache, uncontended post-warmup) |--Bottlenecks:-- GIL (Pydantic + Jinja2 are sync CPU, block the event loop briefly)-- Memory: ~2GB for 1000 concurrent in flight-- Tail latency starts mattering above ~100 concurrent per worker--Caveats:-- If user pipes wire in a `PipeFunc` that does blocking I/O, all bets off.-- Pathological Jinja2 templates could push render to 10+ ms.--## 4. Architectural implication--Under the user's philosophy ("a run is a run, all paths the same"), and combined-with the load profile (dry-run is cheap), here's the design that falls out:--**Dry-runs go through the same PipelexRunner → PipeRun → DeliveryExecutor as live-runs, but with `pipe_run_mode=DRY`. In a FastAPI process where Temporal is the-hub default for live runs, the dry-run endpoint injects a local `PipeRun` per-request.** Something like:--```python-# Live endpoint — hub default (TemporalPipeRun)-runner = PipelexRunner(pipe_run_mode=PipeRunMode.LIVE, ...)-await runner.execute_pipeline(..., delivery=user_delivery)--# Dry endpoint — explicit local PipeRun-runner = PipelexRunner(- pipe_run_mode=PipeRunMode.DRY,- pipe_run=PipeRun(pipe_router=PipeRouter(...)), # force local- ...-)-await runner.execute_pipeline(..., delivery=fs_delivery)-```--This collapses ALL the dead/duplicate paths:-- `dry_run.py` / `dry_run_pipe` / `dry_run_pipes` — gone (validators try/except the runner)-- `dry_run_pipeline.py` — gone (callers use runner directly)-- `dry_run_with_graph.py` — gone (set generate_graph=True on the runner)-- `dry_pipe_router.py` — gone (mode is pipe-level, not router-level)-- `_run_core.py` artifact block — gone (DeliveryExecutor handles it via a filesystem target)--## 5. Smells worth noting (carryover from agents)--- TemporalPipeRun hardcodes `WfPipeRouter` — can't inject workflow class.-- PipelexRunner can't inject a router separately — must wrap in a PipeRun.-- "delivery_assignment is None" as the implicit "skip delivery" — ad-hoc.-- Library/tracing setup scattered across WfPipeRouter (lines 51-65, 72-123) and- WfPipeRun (lines 83-97). Could be context managers.-- ~~Three `# TODO: wip - restore or refactor dry run` in validate_bundle.py~~ —- **CLOSED.** Since this synthesis was written, signature-validation restored- dry-on-load: `validate_bundle` now calls `dry_run_pipes(...)` on every load- path. But it did so through the **old** `dry_run.py` (not the PipelexRunner- consolidation), and then extended that module with the strict signature- pre-check. So the duplicate paths in §4 are still uncollapsed — the- consolidation goal stands — but `dry_run.py` is now load-bearing for- signatures (see `A-taxonomy.md` §7), which makes the deletion bigger than this- synthesis assumed.diff --git a/wip/dry-run-refactor/D-plan.md b/wip/dry-run-refactor/D-plan.mddeleted file mode 100644index df84af872..000000000--- a/wip/dry-run-refactor/D-plan.md+++ /dev/null@@ -1,267 +0,0 @@-# Dry-Run Refactor — Plan (FINALIZED)--> **Status: design finalized 2026-06-01; revised same day (code-grounded review), then expanded same day with the execution-backend dimension (decisions D4–D5).** Decisions **D1–D3** cover the in-process validation consolidation; **D4–D5** add the backend dimension the first cut ignored: *run mode (LIVE/DRY) is orthogonal to execution backend (direct in-process vs Temporal distributed)*. This was driven by three user requirements — (1) DRY must be testable through the **real distributed path** (Temporal workers, no shortcuts; mock at the operation/activity leaf), (2) production pipeline validation runs as a **single standalone Temporal activity** dispatched by the API, (3) direct in-process execution must still work unchanged. The expansion **retires the "DRY → local in-process" shortcut** that `E-parity-gate.md` was written to justify (see the banner there) and splits the work into three parts: **A** in-process consolidation (D1–D3), **B** run-mode/backend orthogonality at the cogt leaf (D4), **C** distributed validation as a Temporal activity (D5). The background docs ([`A-taxonomy.md`](./A-taxonomy.md), [`B-load-profile.md`](./B-load-profile.md), [`C-synthesis.md`](./C-synthesis.md), [`E-parity-gate.md`](./E-parity-gate.md)) remain accurate except where this status note supersedes the parity-gate conclusion.--> **▶ Part A (the in-process consolidation) shipped in #956.** Its as-built record is [`consolidation-as-built.md`](./consolidation-as-built.md); the open follow-ups (Parts B/C + cleanups) are in [`README.md`](./README.md). This doc is the **design reference** (the *why*, and the full Part B/C design). **Line numbers in this doc are indicative** (pinned to the branch state when written) — always **verify by symbol** (grep the function/class), never edit by line number.--## 1. Context--Pipelex executes MTHDS pipelines — graphs of pipes (LLM calls, controllers like sequence/parallel/batch/condition, operators, and `PipeSignature` contracts). DRY mode swaps real LLM/IO calls for mock outputs so a pipeline can be validated end-to-end without spending money or making network calls. The hosted deployment runs LIVE pipelines on Temporal (workflows on remote workers); DRY is meant to stay cheap and in-process.--Today DRY mode has accumulated parallel code paths that do not go through the same orchestration as LIVE, plus a confirmed dead `DryPipeRouter`. This plan consolidates them onto a single execution primitive.--## 2. North-star principle (user-stated)--> "A run is a run, whether it's dry or live or else, it's the SAME THING. The delivery, the preparation, should go through the SAME thing."->-> "Everything should go through the PipelexRunner. Runner protocol. JUST FUCKIN DROP those dry-run functions."->-> "The CLI should ONLY CALL THE PIPELEXRUNNER with the right configuration. There SHOULDN'T BE A dry-run CLI; it should be the RUN cli with a dry mode."--One entry point for *running a pipe* (`PipelexRunner`), `pipe_run_mode=DRY` as the only switch, no bespoke dry-run *execution* functions, no router-level mode dispatch.--## 3. The reframe that drives the design: two operations, not one--The thing the codebase calls "dry-run" is in fact **two distinct operations** wearing one name. Conflating them is what made the north-star feel slippery. Separating them is what makes the consolidation clean.--**A — Execution-dry-run (already through the runner).** `PipelexRunner.execute_pipeline(pipe_run_mode=DRY)` loads the library, builds a `PipeJob`, runs it through `PipeRun → PipeRouter → pipe.run_pipe()` (which dispatches DRY at the pipe level via `PipeAbstract._run_pipe_traced`), optionally delivers/graphs, **raises `PipelineExecutionError` on the first failure**, and tears the library down in `finally`. Single pipe, loads-from-scratch, strict. `pipe_run/dry_run_pipeline.py` (graph path) already does exactly this. **This is the north-star and it already exists.**--**B — Validation-sweep (the bespoke `dry_run.py` path).** `dry_run_pipes(pipes=[...])` takes **already-loaded** `PipeAbstract` objects, shares one library across all of them, and for each pipe mocks inputs + runs DRY **tolerantly**: it collects `SUCCESS / FAILURE / SKIPPED` per pipe, honors `allowed_to_fail_pipes`, skips cross-package unresolved deps (`PipeNotFoundError → SKIPPED`), and does a **single aggregated** signature pre-check across the whole batch *before* running anything. Consumed by `validate --all`, `validate_bundle`, `builder/operations/validate_ops.py`, and the agent CLI. **This is a quality gate, not a run.**--The honest conclusion: a validation sweep is not a run — it is a batch *policy* that *uses* runs. So "drop the dry-run functions and route everything through the runner" is exactly right for the *execution primitive* (A), while the *batch-validation semantics* (B) need a clean, explicit home that **composes** the runner rather than forking it.--## 3.5 The second axis: run mode is orthogonal to execution backend--The first cut treated "dry" as if it implied "local in-process" (the whole point of `E-parity-gate.md` was to justify shortcutting DRY to a local `PipeRun` even under a Temporal hub). **That conflation is wrong and is now retired.** Two independent axes:--- **Run mode** — `LIVE` (real operations) vs `DRY` (mock the operation, e.g. mint a fake LLM result instead of calling the model). Carried on `PipeRunParams.run_mode` (`pipe_run_params.py:138`), serialized inside the `PipeJob` across the worker boundary.-- **Execution backend** — `direct` (in-process; hub gives `PipeRun`/`PipeRouter` + inline `ContentGenerator`) vs `Temporal` (distributed; hub gives `TemporalPipeRun` → `WfPipeRun`/`WfPipeRouter` workflows + `ContentGeneratorInWorkflow` dispatching `act_llm_gen_*` activities). Chosen by config at boot (`pipelex.py:443-461`, `366-381`), **not** by run mode.--These compose into a matrix, and all four cells must work:--- **direct + LIVE** — in-process, real ops. (today)-- **direct + DRY** — in-process, mock ops. The CLI/local validation case; req 3.-- **Temporal + LIVE** — full workflow path, real ops as activities. (today)-- **Temporal + DRY** — full workflow path, **mock ops at the activity leaf**. req 1 — "dry-run through the same workers, no shortcuts, mock instead of the LLM call." Today this cell is *broken by a shortcut*: the pipe-level dry path hardcodes the inline `ContentGeneratorDry` (`PipeLLM._dry_run_pipe` → `ContentGeneratorDry()`), so DRY never dispatches an activity even under a Temporal hub. **D4 fixes this.**--**Where the LIVE/DRY decision belongs.** Today it is made twice and in the wrong place: at the pipe level (`_run_pipe_traced` → `dry_run_pipe()`) and by swapping in `ContentGeneratorDry` *above* the activity boundary. Per the user's requirement, the decision must move **down to the cogt leaf** — `llm_gen_text` / `llm_gen_object[_list]` / extract / img-gen (`pipelex/cogt/content_generation/`, wrapped by `act_llm_gen_*` etc.). The leaf honors `run_mode`: DRY → mint a mock, LIVE → real call. The content generator (inline vs in-workflow) is then **run-mode-agnostic** — it only decides *where* the leaf runs (here vs in an activity), while the leaf decides *real vs mock*. Result, with no further branching:--- Temporal + DRY → `ContentGeneratorInWorkflow` dispatches `act_llm_gen_text` (as always) → the activity runs `llm_gen_text(run_mode=DRY)` → mock **inside the activity** (so activity scheduling + arg/result serialization are exercised — req 1, "mock inside activities").-- direct + DRY (and the validation sweep) → inline `ContentGenerator` runs `llm_gen_text(run_mode=DRY)` → mock inline, no activity.--This is **D4** (§4.8). The validation sweep (B) is then just "run the bundle's pipes DRY on the *direct* backend, in-process" — and because that is backend-independent, it can itself be hosted *inside* one Temporal activity for production (**D5**, §4.9) without any pipe re-dispatching through Temporal.--## 4. Finalized design--### Decisions--> **Two decision schemes — don't conflate.** The decisions here (**D1–D6**) are *architecture* decisions; this doc is their source. Some follow-up docs also cite *eng-review* decisions, **separately numbered D1–D8** (e.g. eng-review D6 = the two-lifecycle split, eng-review D8 = the checkpoint split) — a different scheme. When a bare number is ambiguous, qualify it ("D-plan D6" vs "eng-review D6").--- **D1 — Architecture.** A first-class `BundleValidator` domain service owns the validation sweep (signature pre-pass, per-pipe loop, tolerant aggregation) and composes the shared execution seam (§4.1 / D2) — the same seam `PipelexRunner` composes for single runs. The runner stays a pure single-pipe execution primitive. (Chosen over "runner absorbs a batch method," which would push `SKIPPED` / `allowed_to_fail` / signature-precheck — all irrelevant to LIVE — into the runner and make it a god-object.)-- **D2 — Shared execution seam (no flags on the runner).** Decompose `pipeline_run_setup` into two reusable seams — `acquire_library(...)` (set current, open, load dirs/contents/bundle; owns its own load-failure teardown) and `prepare_pipe_job(library_id, pipe_code, ...)` (resolve the pipe, build mock/real working memory, run params, job metadata, crate → a `PipeJob`; **pure**: no telemetry, no pipeline registration, no library mutation). Both the self-contained runner path and `BundleValidator` compose these seams; the runner gains **no** `borrowed_library` method and **no** suppression flags. `BundleValidator` acquires the library **once**, then per pipe calls `prepare_pipe_job` + a **direct, in-process** execution primitive (see D5 — *not* the hub's `get_pipe_run()`, which would be the Temporal runner under a Temporal hub), and owns library teardown once via `try/finally`. (Chosen over threading ≥3 validation-only flags through `execute_pipeline`/`pipeline_run_setup`, which would have branched the monolith and contradicted D1's "keep the runner pure": pipeline-registration, report-registry, and `PIPELINE_EXECUTE/COMPLETE` telemetry are side effects of the single-run *wrapper*, not of building-and-running a job — so the sweep simply never invokes the wrapper, and there is nothing to "suppress." Also avoids reload-per-pipe churn on `validate --all`.)-- **D3 — Validation model.** The sweep stays **tolerant**: per-pipe `SUCCESS / FAILURE / SKIPPED`, with `SKIPPED` preserved for cross-package unresolved deps and `allowed_to_fail_pipes` kept but **fixed to namespaced `domain.pipe_code` refs** (closing the live bare-code multi-domain-collision `# TODO`). One validation telemetry event, not one per pipe.-- **D4 — Run mode ⟂ backend; mock at the cogt leaf** (§3.5, §4.8). The LIVE/DRY decision moves out of the pipe level and the pipe-level `ContentGeneratorDry` swap, **down to the cogt leaf** (`llm_gen_*`/extract/img-gen). The leaf honors `run_mode` (DRY → mock, LIVE → real); the content generator (inline `ContentGenerator` vs `ContentGeneratorInWorkflow`) becomes run-mode-agnostic and only decides *where* the leaf runs. This makes DRY honor the configured backend — so DRY-on-Temporal goes through the real workers and mocks **inside** `act_llm_gen_*` (req 1) — and **retires the "DRY → local in-process" shortcut** entirely. (Chosen over keeping the pipe-level inline-mock shortcut, which by construction can never exercise the distributed path.)-- **D5 — Validation hosting: one in-process service, optionally one Temporal activity** (§4.9). `BundleValidator` is a **self-contained, fully in-process DRY executor** — direct router + inline content generator, never the hub's Temporal components — so the *same* code runs (a) in-process for CLI / local / tests / direct-mode API, and (b) inside a **single standalone Temporal activity** (`act_validate_bundle`) dispatched by the API for production. Per-pipe runs never re-dispatch through Temporal (choice #4 — "in-process within the sweep"); one activity validates the whole bundle. Production dispatch uses a **true standalone activity**, which requires bumping `temporalio` (pinned `1.23.0` predates the feature) and verifying Temporal server/Cloud support (§4.9, §8).-- **D6 — Per-pipe error *classification* stays in `BundleValidator`; the run/validator outcome seam is a deferred tightening.** A fair question about the boundary: shouldn't the per-pipe `except`/classify clauses "belong to the run, in the Runner's sense," rather than the validation layer? Resolution — the error-handling is **three altitudes**, not two layers fighting over one home: **(L1) the run primitive** owns single-pipe execution *and* failure classification — `PipeRun.run` already builds a structured `ErrorReport` (`pipe_run.py:51-54`), and `WfPipeRun` recovers the same one via `recover_error_report`, so classification is backend-symmetric; **(L2) `BundleValidator`** interprets a run's outcome under *batch policy* — `SKIPPED` (a cross-package missing dep, which a LIVE run *must* hard-fail on; "skipped" is only the sweep's reading of that raise), `allowed_to_fail`, aggregation, the signature pre-pass; **(L3) `_translate_to_validate_bundle_error`** shapes the user-facing `ValidateBundleError`. The batch policy (L2) is irreducible and must **not** move into the run — that recreates the god-object D1 rejected, and `SKIPPED` would corrupt the strict LIVE contract. **Valid kernel conceded:** classification *is* an L1 output, but the run *discards* the `ErrorReport` it computed by re-raising the raw exception (`pipe_run.py:99-100`), forcing the validator to re-derive it with a *second* `try/except` + the `_root_cause_is` `__cause__`/`__context__` walk (`bundle_validator.py:272-295`, `:335-353`) — a real seam smell. **Why not fixed now:** the clean fix is **not** "move the excepts into the run" but giving the run an outcome-*returning* sibling (`run_to_outcome` on `PipeRunProtocol`, in both `PipeRun` and `WfPipeRun`) the validator consumes, with `execute_pipeline` staying the strict raise-on-failure wrapper. Bounded payoff — `ErrorReport` is *wrapper-wins* and collapses the cause chain (a `PipeRunError` wrapping a `PipeNotFoundError` reports `error_type="PipeRunError"`), so the validator would **still** need the raw chain for `SKIPPED` — plus cross-backend churn and it reopens the freshly-hardened signature path. Deferred to [`followup-run-outcome-seam.md`](./followup-run-outcome-seam.md). Boundary affirmed; tightening deferred, not denied. (Chosen over relocating the per-pipe classification into the run, which conflates irreducible batch policy with the execution primitive.)--### 4.1 The shared execution seam (extracted from `pipeline_run_setup`)--`PipelexRunner.execute_pipeline(...)` keeps its current public API and behavior — DRY or LIVE, raises `PipelineExecutionError` on failure, owns its library open/teardown and per-run telemetry. It gains **no** `borrowed_library` method, **no** suppression flags, and **no** knowledge of signatures, `allowed_to_fail`, or `SKIPPED`. Instead we factor the *reusable middle* of `pipeline_run_setup` into two seams that both the runner and `BundleValidator` compose:--- **`acquire_library(library_id, library_dirs, mthds_contents, bundle_uris, ...) -> str`.** Sets the current library, opens it, loads dirs + blueprints + bundle into it. Owns its own load-failure teardown (open, then load in a `try`, teardown on failure). This is the "set_current / open_library / load_libraries / load_from_blueprints" block of today's `pipeline_run_setup` (`pipeline_run_setup.py:150-189`).-- **`prepare_pipe_job(library_id, pipe_code, *, execution_config, pipe_run_mode, inputs=None, graph_context=None, otel_context=None, pipeline_run_id, output_name=None, ...) -> PipeJob`.** Against an already-open library: resolves the pipe (or the bundle's `main_pipe`), builds working memory — generating mock inputs when `execution_config.is_mock_inputs` — resolves run mode, and assembles `PipeRunParams` / `JobMetadata` / `library_crate` into a `PipeJob`. **Pure:** no pipeline-manager registration, no report-registry open, no `PIPELINE_EXECUTE` emission, no library mutation. (Exact signature is settled in Phase 1 and recorded at Checkpoint A.)--`pipeline_run_setup` is then recomposed as a thin wrapper the self-contained runner path uses unchanged: `add_new_pipeline` → `acquire_library` → open graph tracer (if requested) → `prepare_pipe_job` → open report registry + otel trace-start (live) + emit `PIPELINE_EXECUTE` → return. `execute_pipeline`'s `finally` still owns teardown for this path. **Net behavior of the LIVE / self-contained DRY path is identical to today — it is the same code, re-expressed through the seams.**--`BundleValidator` composes the *same* seams without the wrapper: `acquire_library` once, then per pipe `prepare_pipe_job(...)` + a **direct, in-process** execution primitive — a locally-constructed `PipeRun`/`PipeRouter` (or `pipe.run_pipe()` reached through one), explicitly **not** `get_pipe_run()`. Under a Temporal hub `get_pipe_run()` returns `TemporalPipeRun`, which would spawn a workflow *per pipe* (and the sweep itself may already be running inside a Temporal activity, where dispatching a workflow is illegal). The validator must also force the **inline** content generator, not the hub's `ContentGeneratorInWorkflow` (see §4.9 for the mechanism). It owns the single library teardown via `try/finally`. Because it never invokes the runner wrapper, it inherits none of the per-run registration/telemetry — there is nothing to suppress.--> **North-star, refined.** The shared execution *core* is `prepare_pipe_job` → `PipeRouter` → `pipe.run_pipe()` → cogt leaf. `PipelexRunner.execute_pipeline` reaches it via the hub-selected `get_pipe_run()` (so the **backend** — direct or Temporal — is honored; a top-level DRY run on Temporal really runs on the workers, req 1). `BundleValidator` reaches the *same core* via an explicitly-**direct** primitive (so the batch sweep stays in-process and can be hosted in one activity). Both share `pipe.run_pipe()` and the run-mode-aware cogt leaf (D4); they differ only in backend selection and lifecycle (single-run vs batch). This is a stronger realization of "a run is a run" than flag-threading — the execution core is literally shared code — and it keeps run mode orthogonal to backend (§3.5).--### 4.2 `BundleValidator` — the batch validation service (D1)--A first-class service (proposed home: `pipelex/pipeline/bundle_validator.py`, next to the existing `validate_bundle.py`, reusing `_translate_to_validate_bundle_error` as the error-translation boundary). All validation entry points route through it — both batch and single-pipe. Responsibilities, in order:--1. **Acquire the library once** via `acquire_library(...)` (loads dirs / contents / bundle once), inside a `try/finally` that owns teardown for the whole sweep (restore-then-teardown ordering, matching `validate_bundle` today).-2. **Select pipes to dry-run.** Whole bundle, or the `--pipe` slice via the existing `_pipes_to_dry_run` selector (keeps its `PipeNotFoundError` typo-guard so a misspelled `--pipe` fails loudly).-3. **Signature pre-pass** (D-plan §4.3). Walk the selected pipes with `collect_signature_refs` / `collect_signature_paths`; if `allow_signatures=False` and any signature is reached, raise the single aggregated `SignaturesNotAllowedError` (longest dep-chain per signature, as today). In strict mode, exclude signature pipes from the sweep itself (validating a signature directly would always trip the check); in lenient mode keep them (they dry-run trivially by minting a mock output). This is the same signature-aware filtering `validate --all` does today, now owned by the service.-4. **`validate_with_libraries()` pass.** Per selected pipe — this is a static library-wiring check that is *more* than a dry run and the runner's DRY path does not perform it (the runner only triggers `validate_before_run` inside `_run_pipe_traced`). Keeping it here preserves coverage. (It removes today's redundancy where both `validate --all` and `dry_run_pipe` call it.)-5. **Dry-run sweep.** For each selected pipe, build the job against the open library and run it through the **direct, in-process** primitive (D5) — `prepare_pipe_job(library_id, pipe_code=pipe.code, execution_config=<DRY, is_mock_inputs=True>, pipe_run_mode=DRY, pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED)` then `await direct_pipe_run.run(pipe_job)` (a locally-constructed `PipeRun`, **never** `get_pipe_run()` — see §4.1/§4.9) — classifying the outcome:- - success → `SUCCESS`- - failure whose `__cause__` chain contains `PipeNotFoundError` (cross-package unresolved dep) → `SKIPPED`. The reclassification must **walk the chain recursively** (a `_root_cause_is(exc, PipeNotFoundError)` helper), *not* check only the top-level exception: post-refactor the error no longer arrives as the bare `PipeNotFoundError` that today's `dry_run_pipe` `except` catches — `PipeRun.run` re-raises the original (`pipe_run.py:99-100`) and the router may add `__cause__` links. Pinned by a cross-package partial-validation test.- - any other failure → `FAILURE`, unless the pipe's **namespaced** ref (`pipe.pipe_ref`) is in `allowed_to_fail_pipes`. Catch the project base `PipelexError` from the run (a legitimately broad domain-failure surface — not `except Exception`), classify by cause.-6. **Aggregate + report.** Build the per-pipe status map, emit one validation telemetry event (`PIPE_DRY_RUN` with `NB_PIPES`, relocated from `_validate_core.py:87`), raise a single aggregated error if there are unexpected failures, else return the result.--Mock inputs are **not** built by the validator — it passes a DRY + `is_mock_inputs=True` execution config and lets `prepare_pipe_job` generate them (§4.5).--### 4.3 `allow_signatures` is a validation gate, not a run parameter--Key finding that simplifies the surface: in DRY, a `PipeSignature` **always** mints its mock output (`PipeSignature._dry_run_pipe`), regardless of `allow_signatures`. The flag changes exactly one thing — whether the **batch pre-check raises**. The per-pipe *run* is identical either way. Therefore `allow_signatures` does **not** ride on `PipeRunParams` or `execute_pipeline`; it is a `BundleValidator` parameter that controls only the pre-pass (§4.2 step 3). This corrects the earlier assumption (old §6) that `allow_signatures` had to thread through the runner alongside `run_mode`.--`ValidateBundleError.signature_check_error` + `handle_signatures_not_allowed_error` (honoring `--traceback`) are preserved: the pre-pass raises `SignaturesNotAllowedError`, the translation boundary wraps it, the CLI renders it.--### 4.4 Tolerant result model (D3)--`DryRunStatus` (`SUCCESS / FAILURE / SKIPPED`) and `DryRunOutput` survive, but **relocate** out of `pipe_run/dry_run.py` into the `BundleValidator` module — they are validation-report types, not execution types. The CLI / builder JSON consumers (`builder/operations/validate_ops.py`, the agent CLI) keep reading the per-pipe status map.--`allowed_to_fail_pipes` stays in `DryRunConfig` + `pipelex.toml`, **migrated to namespaced refs** (`domain.pipe_code`), and the matching in `BundleValidator` keys off `pipe.pipe_ref`, not `pipe.code`. Concretely: `infinite_loop_1` → `failing_pipelines.infinite_loop_1` (defined in `tests/integration/pipelex/pipes/pipelines/failing_pipelines.mthds`); **`pipe_builder` is obsolete and long gone — drop the entry entirely** (no pipe with that code exists in the libraries or fixtures; it is dead config). This is a breaking config change (allowed — no backward-compat requirement) and closes the bare-code collision risk.--### 4.5 Mock inputs flow through the runner--Mock-input generation lives in the `prepare_pipe_job` seam (§4.1): when `execution_config.is_mock_inputs` is true it runs `convert_to_working_memory_format` → `WorkingMemoryFactory.make_mock_inputs` for inputs the caller didn't provide (the logic at `pipeline_run_setup.py:256-272` today). `BundleValidator` therefore stops building working memory itself and simply passes a DRY + `is_mock_inputs=True` execution config. The helper `convert_to_working_memory_format` **relocates** to `WorkingMemoryFactory` (or a `mock_inputs.py` beside it), since `prepare_pipe_job` / the signature runtime — not the deleted `dry_run.py` — are its real owners. (Minor polish: `with_graph_config_overrides` already carries the `mock_inputs` override but is graph-named; consider a clearer `with_overrides`.)--### 4.6 Graph path--- `pipe_run/dry_run_pipeline.py` already routes through the runner (`execute_pipeline(generate_graph=True, mock_inputs=True, DRY)`) and is the sole path used by `graph/graph_rendering.py`. **Keep** it as a thin helper (or inline into `graph_rendering.py`); it is already north-star-compliant.-- `pipe_run/dry_run_with_graph.py` (`dry_run_pipe_with_graph`, single pre-loaded pipe + graph, direct `pipe.run_pipe()`) has **no consumers** within `pipelex/`. **Delete** it (verify `pipelex-api` and the test tree first).--### 4.7 `convert_stuff_spec_to_typed_named` relocation (unblocks the deletion)--`pipe_signature/pipe_signature.py` imports `convert_stuff_spec_to_typed_named` from `dry_run.py` — a hard dependency from the signature runtime into the module we want to delete. Relocate it alongside `convert_to_working_memory_format` (§4.5) into `WorkingMemoryFactory`, and rewire `pipe_signature.py` + `pipeline_run_setup.py`. This is **step 0** of the migration so `dry_run.py` becomes a deletable leaf.--### 4.8 Leaf-level run-mode mock — DRY honors the backend (D4, Part B)--**The problem.** Today the DRY mock is minted *above* the activity boundary, so it can never travel through the distributed path:--- `PipeAbstract._run_pipe_traced` (`pipe_abstract.py:522-538`) dispatches `dry_run_pipe()` vs `live_run_pipe()` at the pipe level.-- Each operator's `_dry_run_pipe` hardcodes the inline mock generator — e.g. `PipeLLM._dry_run_pipe` runs its shared body with `content_generator=ContentGeneratorDry()` (`pipe_llm.py:406`), while `_live_run_pipe` uses `get_content_generator()` (the hub's — `ContentGeneratorInWorkflow` under Temporal).-- So under a Temporal hub, a DRY run **mocks inline and never dispatches `act_llm_gen_*`** — the worker/activity path is bypassed. This is the shortcut req 1 forbids.--**Why the leaf is the right chokepoint (verified).** Both content generators funnel through the *same* assignment → leaf-function pair, differing only in where the leaf runs:--- `ContentGenerator.make_llm_text` (direct, `content_generator.py`) builds an `LLMAssignment` and calls `await llm_gen_text(llm_assignment)` **inline**.-- `ContentGeneratorInWorkflow.make_llm_text` (`content_generator_in_workflow.py:65`) builds the *same* `LLMAssignment` and `await workflow.execute_activity(act_llm_gen_text, llm_assignment, …)`; `act_llm_gen_text` (`act_llm_generate.py`) calls `await llm_gen_text(llm_assignment)` **inside the activity**.--So `llm_gen_text` / `llm_gen_object` / `llm_gen_object_list` / `img_gen_*` / `extract_gen_pages` / `templating_gen_text` are the single point both paths cross. Put the DRY branch there and it is honored identically inline and in-activity. (`llm_gen_text` calls `get_llm_worker(...)` only *after* the branch, so DRY never needs inference configured — preserving today's "dry works with no API keys.")--**Where `run_mode` rides — carrier decision.** The leaf receives only the assignment, so `run_mode` must be reachable from it. Two options:--- **(recommended) On `JobMetadata`.** `JobMetadata` is *already* a field on every assignment (`LLMAssignment`, `ImgGenAssignment`, `ExtractAssignment`, `TemplatingAssignment`, `RenderPageViewsAssignment`; `ObjectAssignment` via its inner `LLMAssignment`) **and** a parameter of every `ContentGeneratorProtocol` method, and it already serializes across the Temporal boundary. Adding `run_mode` to it is near-zero plumbing: no protocol-signature changes, no new assignment fields, no new activity args. Cost: it duplicates `PipeRunParams.run_mode`, so enforce **single-writer discipline** — `prepare_pipe_job` (or `JobMetadata` construction) sets `JobMetadata.run_mode = pipe_run_params.run_mode` once; `PipeRunParams.run_mode` stays the authoritative switch `_run_pipe_traced` matches on; `JobMetadata.run_mode` is *only* the carrier to the leaf.-- **(alternative) Explicit field on each assignment.** More honest typing, but touches every `ContentGeneratorProtocol` method signature + every call site + every assignment model. Higher churn for the same effect. Pick this only if overloading `JobMetadata` proves objectionable in review.--**Per-leaf mock inventory** (what each leaf's DRY branch produces — lift from today's `ContentGeneratorDry`):--- `llm_gen_text` → a `"DRY RUN: …"` string **and** a synthetic zero-token `LLMJob` report via `_report_dry_llm_job → get_report_delegate().report_inference_job(...)` (so the runner-side usage-emission path stays observable in dry e2e — keep this).-- `llm_gen_object` / `llm_gen_object_list` → `DryRunFactory.make_dry_run_factory(...).build()` + synthetic report. **Schema wrinkle (real):** the direct generator mocks from the live `object_class`, but the leaf only has `ObjectAssignment.object_class_schema` (the class can't cross the activity boundary — `make_for_class` ships `model_json_schema()`). To keep ONE mock site, mock from the schema-reconstructed model in both cases — but verify `DryRunFactory`'s constraint detection survives the round-trip (`json_schema_extra={"mock_format": …}` and field `examples` survive in JSON Schema; `Literal` → `enum`; nested models reconstruct). Accept marginally lower fidelity for direct-dry object mocks, or keep a class-based mock in `make_object` for the direct path and a schema-based one at the activity — at the cost of two sites. **Decide in Phase B1; default = single schema-based site + a fidelity test on a representative `StructuredContent`.**-- `img_gen_*` → fake `ImageContent` (example URL / requested size). `extract_gen_pages` → mock `Page`/`PageContent` list.-- `templating_gen_text` → **runs the real `check_jinja2_parsing`** then returns a mock string. DRY templating is a *validation*, not a skip — preserve the parse check at the leaf.--**Operators stop swapping generators by mode.** Each *operator*'s `_dry_run_pipe`/`_live_run_pipe` collapses toward one path that uses the hub content generator and threads `run_mode` (via `JobMetadata`) to the leaf. Net per cell: direct+DRY → inline leaf mocks (same result, new path); Temporal+DRY → `act_llm_gen_*` **is dispatched** and the leaf mocks **inside the activity** (req 1 — scheduling + arg/result serialization exercised); LIVE cells unchanged.--**Keep at the pipe level only what has no leaf op:** `PipeSignature._dry_run_pipe` (mints its declared output — no LLM/activity) and the controller dry behavior (controllers orchestrate sub-pipes; the leaves they reach handle DRY). The `_dry_run_pipe` override boundary does **not** disappear — it shrinks to the no-operation cases. Refines §5's "only place `run_mode` should matter": `run_mode` now matters at the **cogt leaf** (operations) **and** at `PipeSignature`/controllers (mock-minting / orchestration with no op).--**`ContentGeneratorDry`'s fate.** Its per-method mock bodies move into the leaf DRY branches (or a shared `cogt/content_generation/dry_mock.py` the leaves call). The boot-time `not needs_inference → ContentGeneratorDry()` fallback (`pipelex.py:366-368`) is re-expressed as "force `run_mode=DRY`" (no inference configured ⇒ everything mocks) so there is one mock path, not a parallel generator class. `DryRunFactory` stays (it's the mock-building engine the leaf calls). **Decide in Phase B2** whether `ContentGeneratorDry` is deleted outright or kept as a thin "force-DRY" facade for the boot fallback.--> **Scope honesty.** This is a real refactor of the operator/cogt dry path (assignments, the four leaf families, every operator's `_dry_run_pipe`, the boot fallback), independent of the validation-sweep consolidation. It is the price of "no shortcuts." Part B is sequenced so it can land on its own and be verified by a Temporal-DRY e2e arm before Part C builds on it.--### 4.9 Distributed validation as a Temporal activity (D5, Part C)--Production flow (req 2): the web app calls the Pipelex API to validate a bundle; the API dispatches the validation to a Temporal worker as a **single standalone activity**; the worker runs the whole sweep in-process and returns the per-pipe status map.--- **`BundleValidator` is backend-independent and self-contained.** It constructs its own **direct** stack — a local `PipeRun`/`PipeRouter` and the **inline** `ContentGenerator` — and runs every pipe DRY in-process (D5, §4.1). It never touches `get_pipe_run()` or `ContentGeneratorInWorkflow`. So the exact same object works in the API process (direct mode), in a CLI/test, or inside an activity.-- **Why it must force the inline generator.** Inside a Temporal *activity* there is no workflow context, so `ContentGeneratorInWorkflow.execute_activity(...)` is illegal; and even from a workflow we do **not** want each validated pipe to spawn its own activity (choice #4 — in-process within the sweep). The validator therefore runs the inline leaf (which, post-D4, mocks on `run_mode=DRY`).-- **Mechanism — a content-generator `ContextVar` override (mirror the library pattern).** The operators read the generator from the hub *inside* `_live_run_pipe` (`content_generator or get_content_generator()`), so the override must intercept `get_content_generator()`. The hub already stores the content generator as a plain process-global attribute (`PipelexHub._content_generator`, `set_content_generator`), so a bare `set_content_generator(inline)` swap is **racy** across concurrent activities on one worker. **But the repo already solved this exact problem for the library:** `_library_id: ContextVar` + the `scoped_current_library(library_id)` save/restore context manager (`hub.py:479-514`), with `get_class_registry()` preferring the scoped value. **Recommendation: add the parallel construct for the content generator** — a `_content_generator_override: ContextVar[ContentGeneratorProtocol | None]`, a `scoped_content_generator(generator)` context manager, and make `get_content_generator()` return the override when set, else the hub default. `BundleValidator` (and therefore `act_validate_bundle`) wraps its sweep in `with scoped_content_generator(inline_dry_generator):`. This is coroutine-local (each concurrent activity gets its own value — concurrency-safe), needs no change to operator code, and never touches the serialized `PipeJob`. *(The earlier "thread it through `PipeJob`" idea is wrong: `PipeJob` is the serialized Temporal workflow arg and cannot carry a live generator object.)* *Risk if missed:* the sweep tries to dispatch nested activities/workflows and fails (§8).-- **`act_validate_bundle` activity.** A thin activity whose body is `BundleValidator.validate(...)`. Inputs (serializable): `mthds_contents` / bundle + `library_dirs`, `allow_signatures`, the optional `--pipe` selection. Output (serializable): the `{pipe_ref: DryRunOutput}` map (+ aggregated signature-check error if any). Wrap with the existing `convert_pipelex_errors` activity-error boundary (`tprl/activity_error_boundary.py`) so failures cross back as structured `ErrorReport`s like other activities.-- **Standalone dispatch (choice #2).** Dispatched as a **true standalone activity** (no workflow). This requires **bumping `temporalio`** beyond `1.23.0` to a version that supports standalone-activity execution, and verifying our Temporal server / Temporal Cloud version supports it. The SDK bump is a prerequisite task with its own regression surface (it is the same SDK the whole worker/runtime uses) — call it out in Part C and run the full Temporal e2e suite after bumping. *(If the bump proves blocked, the fallback is a minimal one-step wrapper workflow that runs the single `act_validate_bundle` and returns — functionally identical to the caller — but the chosen direction is standalone.)*-- **API wiring (cross-repo, `pipelex-api`).** The validate route (`pipelex-api/api/routes/pipelex/validate.py` and `build/runner.py`) stops calling `dry_run_pipes` in-process and instead, when Temporal is enabled, dispatches `act_validate_bundle` and awaits the result; in direct mode it calls `BundleValidator` in-process. This supersedes the §7 "let it break on the pin bump" note for `build/runner.py` (that consumer is now actively migrated as part of Part C, not deferred).-- **Parity.** `E-parity-gate.md`'s evidence (API and worker boot identical registries, user classes loaded per-request from the MTHDS payload) now supports a *different* conclusion: hosting the sweep in-process (API) vs in a worker activity is **correctness-equivalent**, because both load the same bundle per request. The gate's original "so we can shortcut DRY to local" conclusion is retired (D4); its parity evidence is repurposed here.--## 5. What changes — delete / relocate / keep--**Delete**--- `pipe_run/dry_pipe_router.py` (`DryPipeRouter`) — confirmed dead code; mode is a pipe-level concern, not a router-level one.-- `pipe_run/dry_run_with_graph.py` — no consumers (§4.6).-- `pipe_run/dry_run.py`'s execution functions `dry_run_pipe` / `dry_run_pipes` — their semantics move into `BundleValidator`. Once the helpers and types relocate, `dry_run.py` is fully removed.--**Relocate**--- `convert_to_working_memory_format`, `convert_stuff_spec_to_typed_named` → `WorkingMemoryFactory` (the `prepare_pipe_job` seam + signature runtime are the real owners).-- `DryRunStatus`, `DryRunOutput` → `BundleValidator` module (validation-report types).--**Extract (new seams)**--- `acquire_library(...)` and `prepare_pipe_job(...)` out of `pipeline_run_setup` (§4.1 / D2) — composed by both the runner's recomposed `pipeline_run_setup` wrapper and `BundleValidator`.--**Refactor — leaf-level run-mode mock (Part B / D4, §4.8)**--- Carry `run_mode` on the cogt assignments (`assignment_models.py`); make the leaf generators (`llm_gen_*`, extract, img-gen) mock when `run_mode.is_dry`.-- Collapse each *operator*'s `_dry_run_pipe`/`_live_run_pipe` toward one path that threads `run_mode` to the hub content generator (stop hardcoding `ContentGeneratorDry()`).-- Fold `ContentGeneratorDry` / `DryRunFactory` mock logic into the leaf DRY branch; re-express the boot-time `not needs_inference` fallback as "force `run_mode=DRY`".--**Add (Part C / D5, §4.9)**--- `act_validate_bundle` Temporal activity (thin wrapper over `BundleValidator.validate`) + standalone dispatch + `temporalio` bump.--**Keep**--- `PipeRouterProtocol`, `PipeRunProtocol`, the Router/Run two-layer split (mirrors the Temporal parent/child workflow shape).-- The pipe-level `_dry_run_pipe` override boundary, **narrowed by D4** to the cases with no operation to dispatch: `PipeSignature._dry_run_pipe` (mints its declared output) and the controller dry behavior (orchestrate sub-pipes; leaves handle DRY). Operator `_dry_run_pipe` swaps are removed (Part B). `run_mode` now matters at the **cogt leaf** (operations) plus these no-op cases.-- `pipe_run/dry_run_pipeline.py` — already runner-based (§4.6).--## 6. Migration sequencing (phased)--Three parts, sequenced by dependency and risk. **Part A** (in-process consolidation, D1–D3) is the branch's original goal and lands first — it is independent of where the mock is minted. **Part B** (leaf-level run-mode mock, D4) delivers req 1 and is a separable cogt refactor. **Part C** (Temporal validation activity, D5) delivers req 2 and depends on Part A (`BundleValidator` exists) and benefits from Part B (clean inline leaf mock). Each part ends green; checkpoints mark session-handoff points.--### Part A — In-process validation consolidation (D1–D3, reqs 3)--Ordered to keep every intermediate state compiling and green, given `dry_run.py` is imported by `pipe_signature.py`.--**Phase 0 — Unblock the leaf.** Relocate `convert_to_working_memory_format` + `convert_stuff_spec_to_typed_named` to `WorkingMemoryFactory`; rewire `pipe_signature.py` and `pipeline_run_setup.py`. No behavior change. Run the suite.--**Phase 1 — Extract the execution seams** (§4.1 / D2). Factor `acquire_library(...)` and `prepare_pipe_job(...)` out of `pipeline_run_setup`, then recompose `pipeline_run_setup` as the thin self-contained wrapper on top of them. **Pure refactor — no new capability, no behavior change.** The runner's public API is untouched. Verify the LIVE + self-contained DRY paths are unchanged (existing runner / pipeline tests stay green); add a focused test that `prepare_pipe_job` builds an equivalent `PipeJob` against a pre-opened library.--> **Checkpoint A (after Phase 1).** The seams exist and the self-contained path is recomposed on them with no behavior change; the signature runtime no longer depends on `dry_run.py` (Phase 0); nothing yet composes the seams for batch use — clean handoff point. Update this doc with the final `acquire_library` / `prepare_pipe_job` signatures and any `pipeline_run_setup` shape changes.--**Phase 2 — Build `BundleValidator`** (§4.2, D1/D3/D5). Implement the service composing the seams against a **direct, in-process** primitive: `acquire_library` once → signature pre-pass → `validate_with_libraries` pass → per-pipe `prepare_pipe_job` + `direct_pipe_run.run` loop (a locally-constructed `PipeRun`, **not** `get_pipe_run()`) with `SUCCESS/FAILURE/SKIPPED` classification (recursive `__cause__` walk for the `PipeNotFoundError → SKIPPED` case), namespaced `allowed_to_fail`, single `PIPE_DRY_RUN` telemetry event, `try/finally` library teardown. Build it behind no callers yet, against the still-present `dry_run.py`. **Verify** a direct DRY run with no `delivery_assignment` and no opened tracer needs no open report registry — if it does, open **one** lightweight per-sweep registry (§8). Relocate `DryRunStatus` / `DryRunOutput`. Port `tests/unit/pipelex/pipe_run/test_dry_run.py` onto the service.--**Phase 3 — Migrate the callers.** Point `validate_bundle` / `validate_bundles_from_directory`, both CLI `_validate_core.py` files, `builder/operations/validate_ops.py`, and `builder/operations/runner_code_ops.py` at `BundleValidator`. Rewire the **tests that import soon-to-be-deleted symbols** (`dry_run_pipe` / `dry_run_pipes` / `DryRunStatus` / `convert_to_working_memory_format`) onto the new surfaces: `tests/integration/pipelex/pipe_signature/test_dry_run_strict_mode.py`, `.../test_pipe_signature.py`, `tests/integration/pipelex/pipes/controller/pipe_sequence/test_pipe_sequence_dry_run.py`, `.../test_pipe_sequence_list_output_bug.py`, `tests/e2e/test_signature_validation_mthds.py`. Migrate `pipelex.toml`'s `allowed_to_fail_pipes` to namespaced refs (`infinite_loop_1` → `failing_pipelines.infinite_loop_1`; **delete the obsolete `pipe_builder` entry**). Verify the single-pipe `validate <pipe>` / `--pipe` slice and the friendly `SignaturesNotAllowedError` rendering still fire.--> **Checkpoint B (after Phase 3).** All validation traffic now goes through `BundleValidator` → the shared seam; `dry_run.py`'s execution functions are unreferenced *inside this repo* (the external `pipelex-api` consumer is handled separately — §7). Re-run the signature e2e + integration suites (`tests/e2e/test_signature_validation_mthds.py`, `tests/integration/pipelex/pipe_signature/*`) and the full `make agent-test`. This is the natural place to split into a fresh session if context has grown.--**Phase 4 — Delete dead code.** Remove `dry_pipe_router.py`, `dry_run_with_graph.py`, and the now-unreferenced `dry_run.py`. Settle `dry_run_pipeline.py` (keep thin or inline into `graph_rendering.py`). Final `make agent-check` + `make agent-test`.--> **Checkpoint C (after Part A).** The in-process consolidation is complete and the branch's original goal is met. DRY still mocks at the pipe level (pre-D4) — that's fine; Part B changes *where* the mock is minted, not the validation outcomes. Natural point to ship Part A on its own if desired, then open a fresh session for Part B.--### Part B — Run-mode/backend orthogonality at the cogt leaf (D4, req 1)--A separable cogt/operator refactor (§4.8). Can run independently of Part A, but is described after it because Part C builds on both.--**Phase B1 — Thread `run_mode` to the leaf, add the leaf DRY branch.** Carry `run_mode` on `JobMetadata` (§4.8 carrier decision; single-writer from `prepare_pipe_job`/`PipeRunParams`). Add the DRY branch to each leaf (`llm_gen_text` / `llm_gen_object` / `llm_gen_object_list` / `img_gen_*` / `extract_gen_pages` / `templating_gen_text`), lifting the mock bodies from `ContentGeneratorDry` into the leaves (or a shared `dry_mock.py`): the `"DRY RUN: …"` string + `_report_dry_llm_job`, the `DryRunFactory` object build, the fake image/pages, and the preserved `check_jinja2_parsing` for templating. Settle the **object-mock-from-schema** question (§4.8 — default: single schema-based site + a fidelity test on a representative `StructuredContent`). Keep operators calling `ContentGeneratorDry` for now — no behavior change yet; unit-test each leaf DRY branch directly (incl. the synthetic report fires and needs an open report registry — see §8).--**Phase B2 — Collapse the operator dry path.** Remove the pipe-level `ContentGeneratorDry()` swap from each operator's `_dry_run_pipe` (`PipeLLM` `:406`, plus `PipeImgGen`/`PipeExtract`/`PipeOcr`/templating operators); route DRY through the hub content generator with `run_mode` threaded via `JobMetadata`. Re-express the boot-time `not needs_inference` fallback (`pipelex.py:366-368`) as force-`run_mode=DRY`; decide `ContentGeneratorDry`'s disposition (delete vs thin force-DRY facade). `PipeSignature` and controllers keep their pipe-level dry behavior. Verify **direct + DRY** results are unchanged (full `make agent-test`).--> **Checkpoint D (after Phase B2).** Direct DRY is re-expressed through the leaf with identical outcomes; the Temporal cell is not yet verified. Update this doc with the final assignment/`run_mode` shape and `ContentGeneratorDry` disposition.--**Phase B3 — Verify Temporal + DRY end-to-end.** With a Temporal server (per the `temporal-e2e-validate` topology), run a pipeline `run_mode=DRY` and assert: `act_llm_gen_*` (and extract/img-gen activities) **are dispatched** and **mock inside the activity**; LibraryCrate propagation, cross-process serialization, and graph tracing behave as in LIVE; no real LLM/IO occurs. Add a DRY arm to the Temporal e2e suite. **This is the req-1 acceptance gate.**--> **Checkpoint E (after Part B).** Run mode is now orthogonal to backend across all four cells. Foundation for Part C is in place.--### Part C — Distributed validation as a Temporal activity (D5, req 2)--Depends on Part A (`BundleValidator`) and Part B (inline leaf mock). Cross-repo (`pipelex-api`).--**Phase C0 — Bump `temporalio` + verify standalone-activity support.** Upgrade the SDK beyond `1.23.0` to a version supporting standalone activities; verify our Temporal server / Cloud version supports it. Run the **full** Temporal e2e suite after the bump (the SDK underpins the whole worker/runtime — treat this as a self-contained, separately-reviewable step). If blocked, fall back to the one-step wrapper-workflow dispatch (§4.9) and note it here.--**Phase C1 — `scoped_content_generator` + `act_validate_bundle`.** Add the `_content_generator_override` `ContextVar` + `scoped_content_generator(...)` context manager mirroring `_library_id`/`scoped_current_library` (`hub.py:479-514`), and make `get_content_generator()` prefer the override. Have `BundleValidator` wrap its sweep in `with scoped_content_generator(inline_dry_generator):` so per-pipe runs use the inline leaf and never dispatch nested activities (concurrency-safe across concurrent activities — §4.9). Then wrap `BundleValidator.validate` as a standalone activity (`tprl_pipe/act_validate_bundle.py`) with `convert_pipelex_errors`. Test: a worker runs the activity; assert zero activity/workflow dispatches occur during the sweep (§8 nested-dispatch risk) and that concurrent invocations don't cross-contaminate the override.--**Phase C2 — API dispatch (cross-repo).** In `pipelex-api`, switch the validate route + `build/runner.py` to dispatch `act_validate_bundle` when Temporal is enabled, and call `BundleValidator` in-process otherwise. Removes the last `dry_run_pipes` consumer (supersedes the §7 deferral). Test both backends against the API.--> **Checkpoint F (after Part C).** All three requirements met: direct in-process (req 3), distributed DRY testing with activity-level mocks (req 1), production validation as a standalone Temporal activity (req 2).--## 7. Out of scope (follow-ups)--- ~~`pipelex-api` consumer migration as a deferred follow-up~~ — **now in scope as Part C / Phase C2** (the API route is actively migrated to dispatch `act_validate_bundle`, removing the `dry_run_pipes` consumer rather than letting it break on a pin bump). Note for sequencing: if Part A ships *before* Part C, then between them `pipelex-api`'s `build/runner.py:64` still imports the (now-deleted) `dry_run_pipes` and would break on its next `pipelex` pin bump — so either land Part C in the same release train, or temporarily keep a `dry_run_pipes` shim until C2.-- API endpoint unification (`/validate` + `/execute` → `/run` in `pipelex-api`).-- CLI artifact dedup — `_run_core.py` now gates artifact writing via `save_main_stuff` / `save_working_memory` against an `output_path`, in two copies (main + agent CLI). Re-scope before touching.-- Renaming `WfPipeRouter` / `WfPipeRun` for clarity ([`A-taxonomy.md` §6 smell #1](./A-taxonomy.md#section-6-smells-and-inconsistencies)).--## 8. Invariants & risks--- **Parity invariant, repurposed** ([`E-parity-gate.md`](./E-parity-gate.md)): the API process and the Temporal worker register identical class registries (user concept classes loaded per-request from the MTHDS payload, not at boot). The original conclusion ("so route DRY to a local run even under a Temporal hub") is **retired** (D4 — DRY now honors the backend). The same evidence now supports D5: hosting the validation sweep in-process (API) vs in a worker activity is **correctness-equivalent**. Re-opens only if one side gains a boot-time library preload the other lacks.-- **Load profile is safe in-process** ([`B-load-profile.md`](./B-load-profile.md)): a dry run is CPU-cheap (Pydantic + Jinja2), no network, no disk — fine to loop in the validation sweep (and cheap enough to host the whole sweep in one activity).-- **Report-delegate registry — confirmed needed (not just a maybe).** DRY is **not** report-silent: the LLM leaf's DRY branch emits a synthetic zero-token job (`_report_dry_llm_job → get_report_delegate().report_inference_job(...)`, lifted from `ContentGeneratorDry`). So `BundleValidator`, running each pipe through a direct `PipeRun.run`, *will* hit the report delegate. It must open **one** report registry per *sweep* (single open/clear, discard contents — validation doesn't surface usage), never per pipe. Covered by a test asserting exactly one `PIPE_DRY_RUN` event per sweep and **no** stray `PIPELINE_EXECUTE`/`PIPELINE_COMPLETE`. (If we'd rather validation be fully report-silent, gate `_report_dry_llm_job` on a "report synthetic dry usage" flag that the sweep turns off and the runner-emission e2e turns on — decide in Phase B1/B2.)-- **Risk — object-mock fidelity from schema (Part B / Phase B1).** Moving the object mock to the leaf means building it from `ObjectAssignment.object_class_schema` (JSON Schema), not the live class (the class can't cross the activity boundary). `DryRunFactory`'s `mock_format`/`examples`/`Literal`/nested-model detection mostly survives the schema round-trip, but not provably for every structured output. Pin with a fidelity test on a representative `StructuredContent`; if a class relies on signals lost in JSON Schema, either enrich the schema or accept a two-site mock (class-based direct, schema-based activity).-- **Risk — `SKIPPED` classification.** Routing through `PipeRun.run` no longer surfaces a bare `PipeNotFoundError`: the run layer re-raises the original (`pipe_run.py:99-100`) and the router may add `__cause__` links. `BundleValidator` must **walk the `__cause__` chain recursively** to re-classify a cross-package unresolved dep as `SKIPPED`, not just check the top-level exception — otherwise partial-bundle validation regresses to hard failure. Covered by a cross-package partial-validation test.-- **Risk — nested-dispatch from the validation sweep (D5/Part C).** If `BundleValidator` fails to force the **inline** content generator and direct runner, then under a Temporal hub its per-pipe runs would try to dispatch activities/workflows — illegal from inside the `act_validate_bundle` activity, and a per-pipe workflow explosion even from a workflow. The inline-generator mechanism (§4.9) is load-bearing; cover with a test that runs the sweep with a Temporal-enabled hub and asserts zero activity/workflow dispatches occur during the sweep.-- **Risk — `temporalio` bump regression surface (Part C / Phase C0).** Standalone activities need an SDK beyond `1.23.0`; that SDK underpins the entire worker/runtime, so the bump can perturb LIVE Temporal execution. Gate it behind a full Temporal e2e pass before building C1. Fallback: wrapper-workflow dispatch (§4.9).-- **Risk — req-1 fidelity regressions (Part B / Phase B3).** Moving the mock to the leaf changes the DRY code path for every operator. The acceptance gate is the Temporal+DRY e2e arm asserting activities are dispatched and mock internally with no real IO; also re-run direct+DRY to confirm identical validation outcomes (Checkpoint D).diff --git a/wip/dry-run-refactor/E-parity-gate.md b/wip/dry-run-refactor/E-parity-gate.mddeleted file mode 100644index 037784168..000000000--- a/wip/dry-run-refactor/E-parity-gate.md+++ /dev/null@@ -1,62 +0,0 @@-> **⚠️ Conclusion superseded (2026-06-01) — read this first.** This doc was written to justify routing DRY to a **local in-process** run even under a Temporal hub. That conclusion is **retired** by decision **D4** in [`D-plan.md`](./D-plan.md): run mode (LIVE/DRY) is now orthogonal to execution backend, so DRY honors the configured backend (DRY-on-Temporal really runs on the workers, mocking at the activity leaf — req 1). **The parity *evidence* below is still correct and still useful** — it now supports a *different* conclusion (decision **D5**): because the API process and the Temporal worker load the same MTHDS payload per request and register identical class registries, hosting the validation sweep **in-process (API) vs inside a worker activity is correctness-equivalent**. Read the rest of this doc as the parity proof, not as a routing recommendation.->-> **Background — parity evidence still holds.** The boot-parity argument (API process and Temporal worker register the same class registry, user concept classes are loaded per-request not at boot) was re-checked against the live branch and is unchanged. Signature-validation does not alter this: signature pipes are loaded from the same per-request MTHDS payload as any other pipe. The `PARITY_HOLDS` finding stands; only the conclusion drawn from it changed (see banner above).--# E. Parity Gate — API/Worker Library Preload Verification--## Why this gate exists--The plan in `D-plan.md` routes DRY runs to a local in-process `PipeRun` **even when the hub default is the Temporal-backed runner** (i.e. in the hosted `pipelex-api-deploy` runner). The original outside-voice review flagged one concrete risk:--> Edge case: user-defined dynamic concepts loaded only in the Temporal worker — would a local dry-run fail to resolve them?--If the Temporal worker process pre-loads things (extensions, plugins, library dirs, Python concept classes) that the API process does not, then a local DRY run inside the API would fail with "class not found" errors that the equivalent LIVE-on-Temporal run would not see. That would invalidate "DRY → always local" and the routing decision would have to change.--**Gate rule:** Verify parity BEFORE writing any implementation code. If parity holds, proceed. If parity diverges, redesign.--## Verdict: PARITY_HOLDS--Both API and Temporal worker boot through the same `Pipelex.make()` path with no pre-loaded `library_dirs`. User concept structure classes are not registered at process boot — they are dynamically generated and registered **per request** from the MTHDS payload via `ConceptFactory.make_from_blueprint() → register_class()`. The same MTHDS payload that the worker would receive over a workflow argument is the same payload the API receives in an HTTP request, so both processes end up with identical class registries for any given pipeline run.--## Evidence--### API boot (`pipelex-api` → FastAPI lifespan)--`pipelex-api/api/lifespan.py` calls `Pipelex.make(integration_mode=IntegrationMode.FASTAPI)` with no custom `library_dirs`. `Pipelex.setup()` (`pipelex/pipelex.py:162`) registers only:--- `CoreRegistryModels` (line 412) — built-in concept structures-- `TestRegistryModels` (line 415) — only when `unit_testing=True`-- Plugin manager (line 323) — loads plugins declared in config-- Template registries (lines 298–316)--No per-request user concept classes at this point. Global class registry starts essentially empty for user types.--### Worker boot (`pipelex-worker` → CLI `worker` command)--`pipelex/cli/commands/worker_cmd.py` calls `make_pipelex_for_cli(context=..., temporal_enabled=True)`, which invokes `Pipelex.make(integration_mode=IntegrationMode.CLI, temporal_enabled=True)`. Same `setup()` path as the API — same registries, no library preload.--**Identical boot sequence.**--### Per-request loading (where user classes appear)--`pipelex/pipeline/pipeline_run_setup.py` line 164: `PipelexInterpreter.make_pipelex_bundle_blueprint()` parses MTHDS content from the request. Line 182: `library_manager.load_from_blueprints()` loads parsed blueprints. Inside `library_manager.py:669`, `ConceptFactory.make_from_blueprint()` is called for each concept; inside `concept_factory.py` (lines 332, 373, 421, 452), generated structure classes are registered via `_get_class_registry().register_class(the_generated_class)`.--`hub.py:384–395`: `get_class_registry()` returns a per-workflow-scoped registry when one is set (Temporal contexts), otherwise the singleton KajsonManager class registry. **User concept structure classes are NOT preloaded at process boot.** They are registered dynamically per request from the MTHDS payload — same path on API and worker.--## Justification (under 150 words)--The API and worker:--- Boot with identical `Pipelex.setup()` paths-- Register only core/test models at startup-- Defer user-concept-class registration to per-request MTHDS loading-- Share the same global class registry (KajsonManager)--When a DRY run is dispatched locally in the API process, it loads the same MTHDS contents the corresponding LIVE Temporal run would, triggering the same `ConceptFactory.make_from_blueprint() → register_class()` flow. Structure classes resolve identically.--The only way parity could break: if `pipelex-api-deploy` introduced a `.pipelex/` override that pre-loads bundles the worker doesn't (or vice versa). Today it doesn't. Adding one would re-open this gate — anyone adding boot-time library preloads on one side must add the matching preload on the other, or the "DRY → always local" routing decision is no longer safe.--## Implication for future work--- If we ever ship "user-defined Python concept classes preloaded as a worker plugin" without an equivalent API-side load, this gate fails and DRY-in-API must route back through Temporal (or block).-- The gate is essentially a parity invariant on the boot path. Worth a CI check if drift becomes likely.diff --git a/wip/dry-run-refactor/README.md b/wip/dry-run-refactor/README.mddeleted file mode 100644index f40e56c55..000000000--- a/wip/dry-run-refactor/README.md+++ /dev/null@@ -1,68 +0,0 @@-# Dry-run / validation consolidation--All dry-run / validation traffic now flows through one service — **`BundleValidator`**, composing the same `acquire_library` / `prepare_pipe_job` execution seam the runner uses. This replaced the four parallel dry-run code paths (`dry_run.py`, `dry_run_with_graph.py`, the dead `dry_pipe_router.py`, kept `dry_run_pipeline.py`).--The work is framed as three parts (D-plan **A/B/C**). **Part A — the in-process consolidation — shipped on `main` in #956.** Parts B and C are deferred to their own branches and are the bulk of the open work below.--- **What exists now** → [`consolidation-as-built.md`](./consolidation-as-built.md) (the as-built architecture + the decisions that hold).-- **Why it's shaped this way + the full Part B/C design** → [`D-plan.md`](./D-plan.md) (the design source).-- **Rendered PR recap** → [`../history/PR-bundle-validator.html`](../history/PR-bundle-validator.html) (a point-in-time snapshot of the consolidation PR).-------## Open follow-ups — do not forget--Everything below is **not yet done**. Each deferred workstream has its own tracker (phases, pre-flight gates, handoff blocks); the smaller items are listed inline here because this is their only home.--### Deferred workstreams — own branches--> **Parts B and C are the two dry-run modes** (full-distribution-with-leaf-mocks, and one-in-process-activity-in-memory). How they relate, what they share, and who builds the common `scoped_content_generator` seam → [`dry-run-modes-master-plan.md`](./dry-run-modes-master-plan.md).--| Workstream | What it delivers | Status / gate | Tracker |-|---|---|---|---|-| **Part B — leaf-level run-mode mock** | Move the LIVE/DRY decision down to the cogt leaf so DRY *honors the backend* (DRY-on-Temporal dispatches `act_llm_gen_*` and mocks inside the activity). Retires the "DRY → local" shortcut. (req 1) | **COMPLETE (2026-06-10)** on `feature/Mock-activities` (PR #983) — all phases done, Checkpoint E reached, Tier 17 GREEN+RED. | [`followup-leaf-run-mode-mock.md`](./followup-leaf-run-mode-mock.md) |-| **Unified dry run — retire `is_mock_inference` as a mode** | One non-live mode: **dry run** ("no effects": every leaf mocked, no PipeFunc execution, no storage IO). `is_mock_inference` becomes `CogtRunParams.is_mock_usage`, an **internal** sub-flag of DRY (no public CLI) that flips usage reporting from zero-token/suppressed to non-zero/rendered — what the cost-assembly tests (incl. `temporal-e2e-validate` Tier 8b) consume. **Supersedes eng-review decision D8.** | **IMPLEMENTED (2026-06-10)** on `feature/Mock-activities` — plan + as-built: [`plan-unified-dry-run.md`](./plan-unified-dry-run.md). | [`design-unified-dry-run.md`](./design-unified-dry-run.md) |-| **Part C — dry-run + validation as one in-process Temporal activity** | The API dispatches the whole job — validation sweep **+** graph-producing dry-run — to a worker as one activity that runs in-process and traces the graph **in memory** (no DynamoDB round-trip, no NDJSON, no usage/cost). Returns the status map **+** `GraphSpec`. (req 2) | **Built** on `feature/Dry-run-as-temporal-activity` (+ `pipelex-api` `feature/Update-dry-run-api`) — all phases through Checkpoint F (phase record in that branch's `TODOS.md`); verified by Mode-1 isolation tests and Tier 2d of `temporal-e2e-validate` (activity + API arms, GREEN+RED on a real 3-process stack). As-built record: [`consolidation-as-built.md` § Part C](./consolidation-as-built.md). The `pipelex-api` branch needs a pipelex release/rev bump (> v0.32.1) before merge. Deferred: Phase G0 standalone activity (`temporalio` bump); retiring the old worker-workflow graph path; moving `pipe_structures` into the activity result. | [`followup-temporal-validation-activity.md`](./followup-temporal-validation-activity.md) |-| **Run-outcome seam** *(optional)* | Have the run primitive *return* its classified outcome so `BundleValidator` consumes a typed outcome instead of re-catching + chain-walking the re-raised exception (D-plan D6). | Not started. **Optional tightening, not a bug** — gated on "is it worth the cross-backend churn?" Default: leave it. | [`followup-run-outcome-seam.md`](./followup-run-outcome-seam.md) |--### Cross-repo--- **MTHDS skills — namespaced `validate` identity.** The single-pipe `validate pipe` agent-CLI surfaces now emit the namespaced `domain.code` (the whole-set surfaces were already namespaced). Skills that exact-match a bare code against the returned `pipe_code` need updating. Self-contained SWE-agent handoff: [`handoff-skills-validate-namespaced-identity.md`](./handoff-skills-validate-namespaced-identity.md). Could not be done from this repo.--### Verify (at/after the #956 merge)--- **`pipelex-api` pin flip.** During co-development the `pipelex-api` `pyproject.toml` carried an editable `pipelex = { path = "../_sig" }` pin (keeps its CI red). It must be flipped to the merged `pipelex` git rev now that #956 has landed. **Confirm this happened** — if `pipelex-api` CI is still red on the editable pin, this is the cause.--### Hardening & cleanups — anytime, small--Part-C operational hardening (heartbeat/cancellation, CPU-bound activity body, worker-vs-API config drift, double dry-run of the main pipe, `_pipe_source_map` scoping, selected-pipe wire field, per-queue dispatch tuning) has its own tracker: [`followup-dry-validate-hardening.md`](./followup-dry-validate-hardening.md).--| Item | Where | Note |-|---|---|---|-| **Per-sweep unique `pipeline_run_id`** | `bundle_validator.validate_pipes` | The report registry + per-pipe graph-trace read are keyed by the **constant** `SpecialPipelineId.DRY_RUN_UNTITLED`, so two concurrent in-process sweeps collide (2nd `open_registry` raises "already exists"; interleaved `close_registry` can drop the other's registry). Mint a unique id per sweep and thread it through `open_registry` + `prepare_pipe_job` + `close_registry` — one fix also scopes the per-pipe trace read to an empty dir. **Rides with Part C** (no concurrent sweeping exists in-process today). `close_registry` is already hardened to `pop(..., None)`. |-| **Shared `acquired_library(...)` helper** | ~6 sites | The "restore outer current-library first, then teardown" `finally` is copied across `validate_bundle.py` (×4), `execution_seams.acquire_library`, and `bundle_validator.acquire_and_validate` (now via `clear_current_library`). Extract one context manager / helper so the load-bearing ordering lives in one place. |-| **`acquire_and_validate` self-teardown guard** | `bundle_validator.acquire_and_validate` | If a caller passes an explicit `library_id` equal to the already-current library, the `finally` restores-then-tears-down the same library (dangling current pointer). Guard `prev_library_id != acquired_id`, or document that `library_id` must not alias the caller's current library. Default `library_id=""` is safe, so latent. |-| ~~**`__init__` content-generator injection seam**~~ | `bundle_validator.BundleValidator.__init__` | **Resolved by Part C** — no constructor seam was needed: `validate_pipes` wraps its sweep loop in `scoped_content_generator(ContentGenerator.make_inline())` (ContextVar scope alongside `scoped_pipe_router`), forcing the inline generator under any hub. |-| **Real-config FAILURE / `allowed_to_fail` integration test** | `tests/integration/.../test_bundle_validator.py` | Integration coverage is SUCCESS-path + real cross-package SKIPPED. The namespaced `allowed_to_fail` match is pinned only at unit level (mocked config). Add a real-config FAILURE-match test (e.g. load `failing_pipelines.mthds`). |-| **Success-path report-registry leak** *(pre-existing)* | `runner.execute_pipeline` finally | Production never closes the report registry on the **success** path either (the `finally` tears down tracer / event-log / library but not the registry). Predates this work; a registry-ownership change spanning setup→run→report. Out of this consolidation's scope, but worth tracking. |-| **Test-style nits** *(very low)* | `pipelex-api` route tests + pipelex `test_agent_output` | Opportunistic: share the duplicated `_build_client()` across the two route-test modules; fold the overlapping body asserts in the runner-with-signatures test; trim the cosmetic domain assertion. None blocking. |--### Out of scope — future (D-plan §7)--- API endpoint unification in `pipelex-api` (`/validate` + `/execute` → `/run`).-- CLI artifact-write dedup — `_run_core.py` gates artifact writing via `save_main_stuff` / `save_working_memory` in two copies (main + agent CLI); re-scope before touching.-- Rename `WfPipeRouter` / `WfPipeRun` for clarity (see `A-taxonomy.md` §6).-------## Background — original audit (historical)--These predate the consolidation and the signature-validation feature; each carries a banner noting what's still accurate. Kept because D-plan's invariants (load-profile-is-safe, API/worker parity) for Parts B/C rest on them. Read only for context.--| File | What |-|---|---|-| [`00-questions.md`](./00-questions.md) | The three founding questions of the audit (is `DryPipeRouter` dead? class taxonomy? FastAPI load profile?). Answers still hold. |-| [`A-taxonomy.md`](./A-taxonomy.md) | `PipeRouter` / `PipeRun` / `WfPipeRouter` / `WfPipeRun` side-by-side; hub swap; `DryPipeRouter` dead-code proof; signature-validation delta. |-| [`B-load-profile.md`](./B-load-profile.md) | CPU/memory/I/O audit — the basis for "dry-run is safe in-process." |-| [`C-synthesis.md`](./C-synthesis.md) | One-page condensed answers to the founding questions. |-| [`E-parity-gate.md`](./E-parity-gate.md) | API-process vs Temporal-worker class-registry parity. Conclusion repurposed by D4/D5 (see its banner); the evidence still holds. |diff --git a/wip/dry-run-refactor/consolidation-as-built.md b/wip/dry-run-refactor/consolidation-as-built.mddeleted file mode 100644index 723868f10..000000000--- a/wip/dry-run-refactor/consolidation-as-built.md+++ /dev/null@@ -1,77 +0,0 @@-# Dry-run / validation consolidation — as built--> **Status: shipped.** The in-process consolidation (D-plan **Part A**) landed on `main` in **#956**, with its `pipelex-api` companion. This doc describes **what exists now** — the architecture, the decisions that hold, and where the code lives. It is a current-state reference, not a build log.->-> - **Still to do** → see the open follow-ups in [`README.md`](./README.md). Nothing here is open.-> - **Design rationale (the *why*, plus Parts B & C)** → [`D-plan.md`](./D-plan.md). Line numbers there are indicative — verify by symbol.--## The model: two operations, not one--"Dry-run" is really two distinct operations, and the consolidation keeps them separate:--- **Execution dry-run** — a single pipe through `PipelexRunner`, in DRY run mode. Raises on failure. This is the normal run primitive with the leaves mocked; it is the north-star ("a run is a run").-- **Validation sweep** — a *batch*, *tolerant* pass over a set of already-loaded pipes that classifies each as `SUCCESS` / `FAILURE` / `SKIPPED` and aggregates the result. This is what `validate --all`, the build/validate CLIs, the builder ops, and `pipelex-api`'s validate/build routes need.--Before the consolidation the sweep lived in a bespoke `dry_run.py` path (plus `dry_run_with_graph.py` and a dead `dry_pipe_router.py`) that bypassed the runner orchestration. Now the sweep is a first-class service — `BundleValidator` — that **composes the same execution seam the runner uses**.--## The shared execution seam--`pipeline_run_setup` was factored into two reusable, side-effect-scoped seams, both in `pipelex/pipeline/execution_seams.py`:--- **`acquire_library(library_id, *, library_dirs, mthds_contents, bundle_uris) -> (library_id, qualified_main_pipe)`** — load-only. Opens + loads a library and returns the domain-qualified main pipe. Owns load-failure teardown: on failure it restores the caller's previous current-library (or clears it) and tears down the partially-opened library. Sync.-- **`prepare_pipe_job(pipe, *, library_id, execution_config, pipe_run_mode, …) -> PipeJob`** — pure. Builds an equivalent `PipeJob` (run mode, mock input, run id, crate) against a pre-opened library. No registration, telemetry, graph-open, or library mutation.--`pipeline_run_setup` is now the thin wrapper composing them (`add_new_pipeline` → `acquire_library` → resolve pipe / tracer / run-mode / registry / otel inside one `try/finally` → `prepare_pipe_job` → emit `PIPELINE_EXECUTE`). Its public signature is unchanged. `BundleValidator` composes the **same** seams without importing the runner wrapper.--A behavior fix landed with the extraction: load/resolve failures now tear down the library. Previously a pre-`try` failure (e.g. an absent pipe code) **leaked** the library; `acquire_library` owns that teardown now, matching the already-hardened `validate_bundle` idiom.--## `BundleValidator` — the validation-sweep service--`pipelex/pipeline/bundle_validator.py`. Two lifecycles (D-plan **D6**):--- **`acquire_and_validate(*, library_dirs, mthds_contents, bundle_uris, library_id="", allow_signatures=False)`** — owns `acquire_library` once + always tears down in `finally`. This is the standalone `validate --all` sweep (sweeps every loaded pipe). Wired to the agent-CLI `validate_all` and builder `validate_all`.-- **`validate_pipes(pipes, *, library_id, allow_signatures=False)`** — the **public inner sweep**. Classifies a caller-supplied pipe list against an **already-open** library and **never tears down**, preserving the loaded-on-success contract that `validate_bundle` and its callers depend on (they read the loaded library afterward via `get_required_pipe`). Wired to `validate_bundle` / `validate_bundles_from_directory`, main-CLI `do_validate_all`, the build CLIs, the builder single-pipe/runner-code ops, and `pipelex-api`'s `/build/runner`.--Internals:--- **Direct in-process primitive.** `__init__` constructs a *local* `PipeRun(PipeRouter(observer=ObserverNoOp()))` — deliberately **not** the hub's `get_pipe_run()`. So a validation sweep runs in-process **regardless of `temporal.is_enabled`** and never dispatches a workflow/activity. (Making DRY *honor* the backend is a separate goal — Part B.)-- **Order (D-plan D7):** `validate_with_libraries` wiring pass → signature pre-pass → per-pipe sweep. The wiring pass runs first so an unresolved **cross-package** sub-pipe in a controller is recorded `SKIPPED` and dropped from the remaining passes instead of aborting the sweep; same-package typos still hard-fail at library load. The signature pre-pass runs over the **full** pipe list (not the post-SKIPPED-drop list) so a `PipeSignature` hidden behind a cross-package wiring gap can't slip past strict validation.-- **Classification catch:** `except (PipelexError, ValidationError, FactoryException)` around both `prepare_pipe_job` and the run. `SKIPPED` is decided by a recursive `_root_cause_is(exc, PipeNotFoundError)` walk over `exc` itself and its `__cause__`/`__context__` chain — necessary because `PipeRun.run` re-raises the original and the router wraps a cross-package `PipeNotFoundError` (itself a `PipelexError`) inside another `PipelexError`. A non-dependency `PipelexError` raised mid-run classifies as a per-pipe `FAILURE`, not a sweep abort.-- **`allowed_to_fail`:** a single aggregate match on the namespaced `pipe.pipe_ref`; the per-pipe step classifies only (no early abort). A sweep with multiple non-allowed failures reports them all. `pipelex.toml`'s `allowed_to_fail_pipes` uses namespaced refs (e.g. `failing_pipelines.infinite_loop_1`).-- **`DryRunStatus` / `DryRunOutput`** live here (canonical home). `DryRunOutput` carries the namespaced `pipe_ref` alongside the bare `pipe_code`.-- **Report registry:** one per sweep, keyed by `SpecialPipelineId.DRY_RUN_UNTITLED`, closed in `finally`. (The DRY LLM leaf emits a synthetic zero-token report, so the registry is genuinely used.)--## `allow_signatures` is a validation gate--Not a runner parameter. It threads through `validate_pipes` / `validate_bundle` and is exposed as a request-body field (`default=False`, strict) on every `pipelex-api` validation route (`/validate`, `/build/runner`, `/build/inputs`, `/build/output`). `SignaturesNotAllowedError` carries `error_domain = ErrorDomain.INPUT`, so a strict signature rejection is a **422** (caller-fixable), not a 500.--## Validate-result identity--Every `validate` surface reports each pipe by its **namespaced `pipe_ref`** (`domain.code`) — agent `validate all`/`bundle`/`pipe`/`pipe --bundle` and all builder validate ops. `build_validated_pipes` (`pipelex/pipeline/validate_bundle.py`) returns `list[ValidatedPipeEntry]` (a `TypedDict` with `pipe_code` + `status`); the `pipe_code` **key name** is preserved for the published JSON contract but its **value** is the qualified ref. The MTHDS skills repo consumes this JSON — its cross-repo update is the one open tail (see [`handoff-skills-validate-namespaced-identity.md`](./handoff-skills-validate-namespaced-identity.md)).--## What was deleted / kept--- **Deleted:** `pipe_run/dry_run.py`, `pipe_run/dry_run_with_graph.py`, `pipe_run/dry_pipe_router.py` (the latter was always dead code), and `tests/unit/pipelex/pipe_run/test_dry_run.py` (coverage ported to `test_bundle_validator.py`).-- **Kept:** `pipe_run/dry_run_pipeline.py` — a still-live "dry-run a whole bundle from MTHDS content → GraphSpec" helper (callers: `pipelex/graph/graph_rendering.py` and `pipelex-api`'s validate route). It already uses the unified DRY-mode `PipelexRunner` path. Not inlined into `graph_rendering.py` (that would break the cross-repo import).-- The two relocated mock helpers (`convert_to_working_memory_format`, `convert_stuff_spec_to_typed_named`) now live as classmethods on `WorkingMemoryFactory`.--## Part C as built — dry-run + validation as ONE in-process Temporal activity (Mode 1)--Shipped on `feature/Dry-run-as-temporal-activity` (+ the `pipelex-api` branch `feature/Update-dry-run-api`). Full phase-by-phase record in the branch's `TODOS.md`; design in [`followup-temporal-validation-activity.md`](./followup-temporal-validation-activity.md).--- **The two-instance fix:** `pipelex.hub.scoped_event_log(event_log)` + `get_event_log_override()` — both the tracer write side (`pipeline_run_setup.py`) and the assembly read side (`tracing_assembly.py`) prefer the scoped instance over `make_event_log`, so a run traces into ONE shared `InMemoryEventLog`. A set override implies tracing-enabled; the scope owner keeps the instance's lifecycle.-- **The in-process graph dry-run:** `dry_run_pipe_in_process(pipe, *, library_id)` in `pipe_run/dry_run_pipeline.py` — `prepare_pipe_job` (DRY + mock_inputs + generate_graph) + a local `PipeRun` under `scoped_event_log` + `scoped_pipe_router` + the new `hub.scoped_content_generator` (inference leaves resolve an inline `ContentGeneratorDry` instead of `ContentGeneratorInWorkflow`; `get_content_generator()` prefers the override). `BundleValidator.validate_pipes` also carries the content-generator scope, so the sweep stays in-process post-Part-B everywhere.-- **The activity:** `temporal/tprl_pipe/act_dry_validate.py` — composes `validate_bundle` (the SAME function the direct route calls, so both backends share the categorized `ValidateBundleError` 422 contract) + `dry_run_pipe_in_process` against the same once-loaded library; graph is best-effort behind a narrow expected-failure catch (`DryRunError`/`PipeRunError`/`PipeRouterError`/`ValidationError`/`FactoryException` → `graph_spec=None`; everything else propagates). Dispatched via the one-step wrapper `wf_dry_validate.py::WfDryValidate` (explicit activity timeout + non-retryable validation error types) through the submitter helper `dry_validate_dispatch.py::dispatch_dry_validate` (workflow-tier `maximum_attempts=1` — a deterministic validation failure must not re-run). Registered in `temporal/tasks.py` (`PackName.PIPE`).-- **The API route** (`pipelex-api/api/routes/pipelex/validate.py`): Temporal-enabled `/validate` dispatches FIRST (one round-trip, identical 422 via the recovered `ErrorReport`), then re-parses blueprints and builds `pipe_structures` from a load-only `acquire_library`. Direct mode unchanged.-- **Verification:** Mode-1 isolation tests (`tests/integration/pipelex/temporal/test_dry_validate_activity_in_memory.py` — zero nested dispatch pinned on the Temporal history) · Tier 2d in the `temporal-e2e-validate` skill (activity + API arms, GREEN + RED-proven on a real 3-process stack) · `pipelex-api/tests/unit/test_validate_temporal_dispatch.py` (both backends).-- **Deferred:** Phase G0 standalone activity (`temporalio` bump) · retiring the old worker-workflow graph path for `/validate` · moving `pipe_structures` into the activity result (drops the API-side load-only acquisition).--## Where the code lives--- `pipelex/pipeline/execution_seams.py` — `acquire_library`, `prepare_pipe_job`.-- `pipelex/pipeline/pipeline_run_setup.py` — the thin runner-setup wrapper over the seams.-- `pipelex/pipeline/bundle_validator.py` — `BundleValidator` (both lifecycles), `DryRunStatus`, `DryRunOutput`.-- `pipelex/pipeline/validate_bundle.py` — `validate_bundle` / `validate_bundles_from_directory` (delegate the sweep to `BundleValidator.validate_pipes`; keep only library lifecycle + error presentation), `build_validated_pipes`, `ValidatedPipeEntry`.-- `pipelex/core/memory/working_memory_factory.py` — the two relocated mock helpers.-- Tests: `tests/unit/pipelex/pipeline/test_bundle_validator.py`, `tests/integration/pipelex/pipeline/{test_bundle_validator,test_execution_seams,test_acquire_and_validate,test_pipeline_run_setup_characterization}.py`.-- `pipelex-api`: `api/routes/pipelex/build/runner.py` (+ `validate.py`, `build/inputs.py`, `build/output.py` for the `allow_signatures` flag).diff --git a/wip/dry-run-refactor/deferred-unified-dry-run-review.md b/wip/dry-run-refactor/deferred-unified-dry-run-review.mddeleted file mode 100644index 5dcc84c0f..000000000--- a/wip/dry-run-refactor/deferred-unified-dry-run-review.md+++ /dev/null@@ -1,27 +0,0 @@-# Deferred items — unified dry run review rounds (PR #986)--> Findings from the PR #986 review rounds (greptile/cubic bots + the `/review` specialist/adversarial passes, 2026-06-10) that were **verified real but deliberately NOT fixed** — each is a design tradeoff or a guard for a currently-unreachable path, not a silent bug in shipped behavior. Fixed-on-the-spot findings are not listed (see the PR's commits).--## Run-mode safety & contracts--- **Typed-error wrap of the `cogt_run_params` mint + factory contract check.** The derived property and `PipeRunParamsFactory` raise pydantic `ValidationError` / `ValueError` on the LIVE+`is_mock_usage` violation. Inside Temporal *workflow* code an unconverted exception retries forever (the known submitter-hang class) — but post-hardening the illegal state requires deliberately bypassing validators (`model_copy(update=...)` on exactly those fields; attribute writes are blocked by `Field(frozen=True)`). Guarding that is guarding a near-impossible path; revisit only if a real producer of the illegal state appears. Same family: the factory's `ValueError` surfacing as an untyped 500 through `pipelex-api` once that repo adopts `is_mock_usage` — decide a typed INPUT-domain error at that adoption point, not before.-- **Keyless-boot trust boundary narrowed (D4 consequence).** Pre-Part-B, a keyless boot installed a mock generator globally, so even direct `ContentGeneratorProtocol` callers got mocks. Now generator selection is backend-keyed only and forced-DRY lives at `make_run_params` — an embedder calling the protocol directly with a hand-built LIVE carrier under a keyless boot whose lenient backend load found credentials *can* reach a real provider. This is the submitter-side-contract-only design decided in eng review D4; flagging the contract change here for visibility. A `is_dry_run_forced()` guard at `get_content_generator()` would restore the old blanket but re-couples generator selection to run mode — rejected for now.-- **Forced-DRY silently downgrades an explicit LIVE request (warning only).** A misconfigured production boot (keyless by accident) returns synthetic structured outputs to clients with only a server-side `log.warning`. Options: fail loud when the LIVE request was *explicit* (coerce only defaulted modes), or stamp `run_mode` into client-visible output metadata. Interacts with the already-deferred `PIPELEX_FORCE_DRY_RUN_MODE` env-var cleanup (see `wip/dry-run-refactor/README.md` open table) — decide together.-- **`hub` forced-DRY flag is process-global, not ContextVar-scoped.** Consistent with the other hub singletons, but it guards spend; if embedded re-boot-while-running ever becomes a supported pattern, move it to the ContextVar style of `scoped_content_generator`.--## Dry-leaf behavior--- **`stamp_mock_main_coordination` can corrupt a user item class that happens to have a constrained `pipe_code` field** (direct attribute write, no validation), and the resulting `DryRunObjectFidelityError` remedy ("declare `examples`/`mock_format`") cannot fix a framework-injected value. Niche (requires a user output class with a constrained `pipe_code`), and the stamp is load-bearing for bundle dry-validation (D3). Candidate fix: pin the field through `build_mock_object(pipe_code="mock_main")` for the first item (runs validators) or scope the stamp to the builder's spec classes.-- **Ambient `dry_run_config.nb_extract_pages` read at two leaves** (extract + render-page-views) — under Temporal these run in different activities; heterogeneous worker configs would mismatch counts and trip the generator's length check. Already tracked since the B1 review round (`followup-leaf-run-mode-mock.md`); the D11-style fix (carry the resolved count on the assignments) stands. Note: the "both `image_uri` and `document_uri`" variant of this finding is impossible — `ExtractInput` validates exactly-one.-- **Inline page-view count mismatch raises bare `ValueError`; the in-workflow twin raises `ContentGenerationError`.** Cross-backend error-type inconsistency on the same fault (pre-existing). Aligning changes error classification on a live path (`ValueError` is in `non_retryable_error_types`); do it together with the nb_extract_pages fix above.-- **Dry image mocks hardcode `mime_type="image/jpeg"` + 1024×1024** while `dry_run_config.image_urls` includes a `.png` — mock metadata can contradict the URL it points at (pre-existing, carried from `ContentGeneratorDry`). Derive mime from the URL suffix if a consumer ever keys on it.--## Reporting & surfaces--- **Synthetic usage is discriminated only by sentinel model-name strings** (`dry_run` / `mock_usage`, `unit_costs={}`). Cost stays $0 by construction, but a downstream aggregator summing `nb_tokens_by_category` without filtering the sentinels would mix synthetic tokens into real usage metrics. A structural `is_synthetic` marker on the usage record is the durable fix — wire-format change, do it when a real aggregator consumer appears (Temporal not in prod).-- **Validation sweeps now reach the `exec()` codegen path.** `BundleValidator` / `dry_run_pipe_in_process` swapped `ContentGeneratorDry` (original-class mocks, no codegen) for the inline `ContentGenerator`, whose dry object leaves run schema-reconstruction (datamodel-code-generator → `exec()`). The hosted `/validate` surface — keyless, typically less gated than execution — now exercises the codegen pipeline on user-bundle-derived schemas. The factory's own docs say its containment is defense-in-depth, not a sandbox; its existing hardening TODO (programmatic `create_model()` or subprocess sandbox) gains priority from this surface expansion.--## Test consolidation candidates (nice-to-have)--- **Fidelity re-validation pattern exists in 4 hand-rolled copies** (`_revalidate_against_object_class` ×2 + the inline search-structured variants in both generators, which already disagree on live-arm behavior — a deliberate boundary decision recorded in `followup-leaf-run-mode-mock.md` deferred items). Consolidating crosses the cogt/temporal layering; revisit if a fifth copy appears.-- **DRY + `should_include_page_views` combined path untested** — both leaves currently agree on length only because they read the same config key; rides with the nb_extract_pages assignment-carried-count fix.diff --git a/wip/dry-run-refactor/design-unified-dry-run.md b/wip/dry-run-refactor/design-unified-dry-run.mddeleted file mode 100644index b64326e87..000000000--- a/wip/dry-run-refactor/design-unified-dry-run.md+++ /dev/null@@ -1,113 +0,0 @@-# Design — Unified dry run (retire `is_mock_inference` as a mode; add internal `is_mock_usage` flag)--> **Status: IMPLEMENTED (2026-06-10) — see [`plan-unified-dry-run.md`](./plan-unified-dry-run.md) for the plan and as-built record.** Design agreed with the user (2026-06-10). This design **supersedes eng-review decision D8** of [`followup-leaf-run-mode-mock.md`](./followup-leaf-run-mode-mock.md) ("is_mock_inference KEPT as thin reportable-mock"): the *mode* is retired; the *capability* (non-zero synthetic usage) survives as a secondary flag on dry run. It also resolves the "fate of `is_mock_inference`" item tracked in [`../history/registry/deferred-followups.md`](../history/registry/deferred-followups.md).->-> **Baseline:** the as-built state of `feature/Mock-activities` (PR #983, Part B complete through Checkpoint E) + the Part C validation activity (`feature/Dry-run-as-temporal-activity`). Read [`followup-leaf-run-mode-mock.md`](./followup-leaf-run-mode-mock.md) and [`followup-temporal-validation-activity.md`](./followup-temporal-validation-activity.md) for how that baseline works.--## 1. How we got here (why two modes exist today)--- **The old dry run** predated the leaf-mock work: `--dry-run` swapped in `ContentGeneratorDry` *pre-dispatch*, so a dry run never entered Temporal activities. That mode is dead — Part B deleted `ContentGeneratorDry` and moved the mock to the cogt leaf.-- **`--mock-inference`** was added as an interim trigger (registry branch, `fix/For-API-update` Phase 5) because tests needed a *dispatching* fake: a LIVE run whose LLM leaves fake the call with non-zero synthetic usage, so the cross-worker cost-report assembly could be validated cheaply. At the time it was the only non-live mode that reached the activities.-- **Part B (this branch) unified the foundation**: `run_mode` now rides `CogtRunParams` on every assignment, every inference leaf branches at the leaf, and `--dry-run` on Temporal genuinely dispatches and mocks inside activities (Tier 17). Result: **two modes that both dispatch and both mock at the leaf**, separated only by usage reporting, leaf coverage (mock-inference is LLM-only), and a few pipe-tier dry behaviors. Two names for nearly one thing — actively confusing, as the run-modes doc work surfaced.--## 2. The design--### 2.1 One non-live mode: **dry run**--There is exactly one non-live run mode for pipeline runs: `run_mode=DRY` (`--dry-run`). Its defining contract is **no effects**:--- no AI provider calls, no API cost, no API keys needed;-- no storage IO (mocks built above the `*_and_store` step — unchanged D10);-- no user-code execution: `PipeFunc` does not execute the user's Python; `PipeCompose` keeps its dry arm (exact compose-dry details deliberately deferred — see §5);-- zero-token synthetic usage, cost report suppressed — *by default* (see §2.3).--The name is "dry run" (not "mock inference") precisely because the contract is broader than inference mocking: it is the no-effects mode.--### 2.2 Dry run is orthogonal to backend (unchanged)--Part B's deliverable holds as-is:--- **DRY × direct**: leaves mock inline in-process.-- **DRY × Temporal**: the run dispatches the *real* workflow/activity tree (child workflows, `act_*_gen_*` on workers) and the leaf mocks inside each activity. This is the mode that exercises the distribution machinery (dispatch, serialization, routing, cross-worker graph tracing) at zero cost with zero credentials.-- **The validation sweep** (`validate_bundle` / `act_dry_validate`) is *not* a third mode: it is dry run with the backend pinned in-process by the existing ContextVar scopes (`scoped_pipe_router`, `scoped_content_generator`, `scoped_event_log`), batched over all pipes, with mock inputs always on and the graph traced in memory. Unchanged by this design.--### 2.3 `is_mock_usage` — a secondary, internal flag on dry run--`CogtRunParams.is_mock_inference` is replaced by `CogtRunParams.is_mock_usage: bool = False`, with new semantics:--- It is a **sub-flag of DRY**, not a mode: it only has meaning when `run_mode=DRY`. Setting it with `run_mode=LIVE` is a contract violation (proposed: model-validator rejection — fail loud; confirm at plan time).-- `is_mock_usage=False` (default): leaves report **zero-token** synthetic jobs → `AggregatedCosts.has_reportable_usage` is False → cost report suppressed. Exactly today's dry-run reporting.-- `is_mock_usage=True`: leaves report **non-zero** synthetic usage (the deterministic sentinel counts, e.g. INPUT 100 / OUTPUT 50, `unit_costs={}` so cost is $0) → the cost report **renders**. This is what cross-worker cost-report assembly tests consume — they assert on non-zero aggregated values, not just on report presence.-- **Internal only — no public CLI flag.** Exposed through the Python surface (`PipelexRunner` / `execute_pipeline` / `PipeRunParamsFactory.make_run_params`) for tests and embedders. No documented CLI option; CLI users get `--dry-run` and nothing else.-- **Tests must still trigger it end-to-end**, including the `temporal-e2e-validate` skill's Mode-2 (3-process, CLI-driven) tiers, with non-zero values asserted. Since Mode 2 drives the real CLI, the design proposes a **hidden CLI option** (typer `hidden=True`, e.g. `--mock-usage`, requires `--dry-run`) as the test trigger: present for scripts, absent from `--help` and docs. Alternative if hidden flags are unwanted: a tiny Python submitter under `temporal/test_extras`. Decide at plan time; the hidden flag is the lowest-friction proposal.--### 2.4 Leaf coverage becomes uniform--Mock-inference's LLM-only restriction dies with the mode. Dry run already mocks **every** leaf (LLM text/object/list, img-gen, extract, render-page-views, search, templating). Consequently:--- `MockInferenceUnsupportedError` and its fail-loud guards at the img/extract/search leaves are **deleted** — their entire reason was mock-inference's partial coverage. With `is_mock_usage` riding on DRY, a "reportable" run covers every operator a dry run covers.-- Non-LLM leaves keep their current dry behavior (synthetic outputs, no usage reporting). Whether `is_mock_usage=True` should ALSO emit synthetic usage for non-LLM leaves (img-gen/extract jobs have their own usage records in live runs) is left to plan time; the minimal design keeps reportable synthetic usage LLM-only, matching what the cost-assembly tests actually assert today.--## 3. Delta inventory — what changes relative to the as-built baseline--### 3.1 Carrier and factory--- `CogtRunParams`: `is_mock_inference` → `is_mock_usage` (semantics per §2.3; frozen/extra-forbid/required-run_mode unchanged). The `is_mock_built` property collapses — mock-built is now simply `run_mode.is_dry`; keep or inline it (the fidelity guards key off it).-- `PipeRunParamsFactory.make_run_params`: parameter re-keyed; stays the single writer. The boot forced-DRY path simplifies — the "forced-DRY swallows an explicit `--mock-inference`" warning disappears (there is nothing to swallow anymore).-- `PipelexRunner` / `execute_pipeline` / `prepare_pipe_job`: `is_mock_inference` param → `is_mock_usage`.--### 3.2 Leaves (`pipelex/cogt/content_generation/`)--- Each LLM leaf's **dual branch collapses to one**: `if cogt_run_params.run_mode.is_dry → dry helper`; the dry helper selects its report func on `is_mock_usage` (zero-token `report_dry_llm_job` vs non-zero `report_mock_usage_llm_job`, renamed from `report_mock_inference_llm_job`). The object/list variants are already parameterized via `_leaf_gen_object(report_func=...)`; the text variant adopts the same shape.-- **Delete** `mock_llm_gen_text` / `mock_llm_gen_object` / `mock_llm_gen_object_list` / `_mock_text` and the `MOCK INFERENCE` marker string. One marker family remains (`DRY RUN: …`). Tests grepping the mock marker must re-key.-- Sentinel identifiers: `dry_run` model name stays for zero-token; `mock_inference` model name/id → `mock_usage` (or fold to a single sentinel with two token payloads — plan-time detail; tests that count/filter by sentinel follow).-- `stamp_mock_main_coordination`: the "mock-inference deliberately does NOT stamp" arm dies with the mode; the dry object-list leaf stamps regardless of `is_mock_usage` (harmless — the stamp only matters to bundle dry-validation, and its docstring sheds the mock-inference caveat).-- `dry_mock.py` module docstring rewritten for the one-mode world.--### 3.3 Errors--- `MockInferenceUnsupportedError`: **deleted** (guards at img/extract/search leaves removed). Error page removed on regeneration.-- `MockInferenceObjectFidelityError`: fires on mock-built objects, which is now DRY-only — **rename candidate** (e.g. `DryRunObjectFidelityError`); message/remedy text already generalized in B1. Decide at plan time; regenerate error pages either way.-- `DryRunMockBuildError`: unchanged.--### 3.4 CLI--- `--mock-inference` removed from `run pipe` / `run bundle` / `run method`.-- `validate_run_flag_combination` loses the mutual-exclusion arm; only "`--mock-inputs` requires `--dry-run`" remains (plus, if the hidden-flag proposal is taken, "`--mock-usage` requires `--dry-run`").--### 3.5 Tests--- Leaf unit tests: dual-arm (dry vs mock) tests merge into dry tests parameterized over `is_mock_usage`; reporting assertions cover both token payloads (zero suppressed / non-zero rendered).-- `test_mock_inference_temporal.py` (wire-survival of the flag): re-key to `is_mock_usage` on the carrier.-- Fidelity tests (D6 arms): re-key from `is_mock_built`'s two triggers to DRY-only.-- Keyless-boot tests: drop the swallow-warning arm.-- **`temporal-e2e-validate` skill**: Tier 8b (the `--mock-inference` cost-assembly arms — cross-worker usage aggregated into one rendered submitter report) re-points to the dry-run + `is_mock_usage` trigger (hidden flag or submitter script per §2.3) and keeps asserting **non-zero** aggregated values. Tier 17 prose drops the "distinct from `--mock-inference`" contrast and instead states the `is_mock_usage` sub-flag. Routing battery / scope manifests mentioning `--mock-inference` updated.--### 3.6 Docs--- `docs/building-methods/pipes/run-modes-and-backends.md`: the 3×2 matrix collapses to **one mode axis** (live / dry) × two backends, plus the validation-sweep section; mock inference disappears as a user-facing concept (the internal flag is deliberately undocumented, or mentioned only as an internal testing affordance).-- `docs/under-the-hood/dry-run-mock-generation.md`, `docs/features/validation-dry-run.md`, `docs/features/cost-tracking.md`, `docs/tools/cli/run.md`: purge `--mock-inference`.-- Error pages regenerated (`make gep`).-- CHANGELOG entry under `[Unreleased]` (breaking: `--mock-inference` removed; no deprecation period per our principles).--### 3.7 Non-concerns--- **Wire/backward compatibility: none.** The Temporal integration has never shipped to prod; the `CogtRunParams` field rename needs no migration shims.-- **`pipelex-api` / platform repos**: grep for `mock_inference` at plan time; the public runner API params carry `is_mock_inference` today via `_run_core` → runner — the public surface change must ride a pipelex version bump like any other.--## 4. What does NOT change--- The leaf-mock foundation: `CogtRunParams` carrier (frozen, single-writer, stamped on every assignment, serializes across the activity boundary), the `dry_*` helpers, schema-built object mocks (`SchemaToModelFactory` + `DryRunFactory`), `DryRunMockBuildError`, the fidelity re-validation.-- Backend orthogonality and Tier 17's meaning (DRY-on-Temporal dispatches and mocks inside activities).-- The validation sweep and `act_dry_validate` (Part C) — shape, scopes, contracts all untouched.-- The forced-DRY keyless boot (`Pipelex.make(needs_inference=False)` → every run forced DRY) — it only loses the swallow-warning edge case.-- `--dry-run` and `--mock-inputs` CLI surface.--## 5. Deliberately deferred to plan time--- **PipeCompose dry-arm specifics** — the user explicitly parked this ("we'll see about those details later"). Current behavior (construct-mode `PipeComposeError` fallback, direct template rendering) stands until then.-- The exact e2e trigger mechanism for `is_mock_usage` (hidden CLI flag vs test submitter script) — §2.3 proposal: hidden flag.-- Whether `is_mock_usage=True` + `run_mode=LIVE` is a validation error (proposed) or silently ignored.-- `MockInferenceObjectFidelityError` rename.-- Sentinel-model naming consolidation (§3.2).-- Whether non-LLM leaves emit synthetic usage under `is_mock_usage=True` (§2.4 — minimal design says no).diff --git a/wip/dry-run-refactor/dry-run-modes-master-plan.md b/wip/dry-run-refactor/dry-run-modes-master-plan.mddeleted file mode 100644index f830dfacb..000000000--- a/wip/dry-run-refactor/dry-run-modes-master-plan.md+++ /dev/null@@ -1,69 +0,0 @@-# Master plan — two dry-run modes on one foundation--> **What this is:** the overview tying together the two dry-run modes we want, how each is achieved, what they share, and who builds what. Neither per-mode tracker owns the *relationship* — this doc does. The per-mode plans:->-> - **Mode 1 (in-memory activity)** → design [`followup-temporal-validation-activity.md`](./followup-temporal-validation-activity.md) · executable plan [`../../TODOS.md`](../../TODOS.md) · branch `feature/Dry-run-as-temporal-activity` (active).-> - **Mode 2 (full distribution, leaves mocked)** → [`followup-leaf-run-mode-mock.md`](./followup-leaf-run-mode-mock.md) (done — landed on this branch via PR #986).-> - **Design source:** [`D-plan.md`](./D-plan.md) §3.5 (run-mode ⟂ backend) + §4.8 (leaf-level mock) + §4.9 (validation activity).--## The two modes, in one sentence each--1. **In-memory activity (req 2).** The whole dry-run + validation runs **in-process inside a single Temporal activity** on a worker — nothing nested is dispatched — tracing the graph **in memory**. Purpose: let the hosted API offload "validate this bundle + give me its graph" to a worker, cheaply, with no DynamoDB round-trip and no AI calls.--2. **Full distribution, leaves mocked (req 1).** A dry-run goes through the **real** Temporal path — `TemporalPipeRun` → `WfPipeRouter` → child workflows → `act_llm_gen_*` / extract / img-gen activities — and the **leaf inside each dispatched activity** mocks instead of calling the model. Purpose: exercise the distribution machinery (dispatch, scheduling, serialization, routing, cross-worker propagation, graph assembly) without AI cost or latency.--## Why they don't conflict — one foundation, two backends--Both are the **same `run_mode=DRY`** run; they differ only in **backend** (where the leaf executes), which is the D-plan §3.5 thesis: *run mode is orthogonal to execution backend*. The foundation that makes both clean is the **leaf-level mock** (Part B / D-plan §4.8): mock at the cogt leaf keyed on `run_mode`; the content generator is run-mode-agnostic and only decides *where* the leaf runs.--| | Controllers (nested sub-pipes) | Cogt leaf (LLM/extract/img-gen) | Graph tracing | Result |-|---|---|---|---|---|-| **Mode 1 — in-memory activity** | forced **in-process** via `scoped_pipe_router` | forced **inline** via `scoped_content_generator` → mocks in-process | `scoped_event_log(InMemoryEventLog())` → in memory | one activity, zero nested dispatch, zero external I/O |-| **Mode 2 — full distribution** | hub default = **Temporal router** → child workflows | hub default = **`ContentGeneratorInWorkflow`** → dispatches `act_llm_gen_*`, mocks **inside** the activity | hub default = `temporal_dynamodb` (cross-worker assembly) | real distributed path, no AI spend |--The **only divergence is which content generator / router the run uses** — and both are selected by **per-call ContextVar overrides** (coroutine-local, restore-on-exit), so concurrent runs of different modes on one worker never cross-contaminate. Mode 1 *sets* the overrides; Mode 2 *leaves the hub defaults*. They are additive, not exclusive.--**Proof they already coexist:** the LLM slice of Mode 2 first shipped as `is_mock_inference` (registry branch, #967) — `run_mode` stayed LIVE so operators dispatched `act_llm_gen_*` normally while the leaf faked the call. The unified dry run (#986) has since retired that flag into the internal `is_mock_usage` DRY sub-flag; the coexistence holds, now keyed on `run_mode=DRY` end to end.--## The shared seam — `scoped_content_generator` (build once)--The single place the two modes touch. Under a Temporal-enabled hub, `get_content_generator()` returns `ContentGeneratorInWorkflow` **globally** (boot-time selection, `pipelex.py:370-385` — not contextual). So:--- **Mode 2 wants that default** — the in-workflow generator *is* how it dispatches leaf activities.-- **Mode 1 must override it to inline** — otherwise, once Part B routes the DRY mock through `get_content_generator()` at the leaf, Mode 1's in-process activity would dispatch `act_llm_gen_*` and break.--So `scoped_content_generator` (a hub ContextVar + context manager, mirroring `scoped_pipe_router` in `hub.py`, with `get_content_generator()` preferring the override) is needed by Mode 1 and is the seam Part C originally specced. **Decision (2026-06-09): build it now, in Mode 1's branch**, so Mode 1 is correct regardless of when Part B lands and the seam exists once for both.--### The ContextVar scope family (`pipelex/hub.py`)--| Scope | Exists? | Used by | Forces |-|---|---|---|---|-| `scoped_current_library` | yes (`hub.py`) | both / general | which library is current |-| `scoped_pipe_router` | yes (`hub.py`, PR #976) | Mode 1 | nested controllers run in-process, not via the Temporal router |-| `scoped_content_generator` | yes (`hub.py`, this branch) | Mode 1 | the leaf uses the inline generator, not the in-workflow one |-| `scoped_event_log` | yes (`hub.py`, this branch) | Mode 1 | graph trace events share one in-memory log across emit + assemble |--## Build order — who builds what--**Mode 1 — branch `feature/Dry-run-as-temporal-activity` (active now).** Full phase plan + checkpoints in [`../../TODOS.md`](../../TODOS.md):--- Phase 1 — **done.** `scoped_event_log` + shared in-memory tracing (fixed the two-instance emit/assemble problem: `pipeline_run_setup.py` write vs `tracing_assembly.py` read each called `make_event_log` separately; the in-memory log now shares one instance).-- Phase 2 — **done.** In-process graph dry-run safe under a Temporal hub (`dry_run_pipe_in_process` in `pipe_run/dry_run_pipeline.py`); built `scoped_content_generator` and wraps the run in it + `scoped_event_log` + `scoped_pipe_router`.-- Phase 3 — **done.** The `act_dry_validate` activity (sweep + best-effort in-memory graph dry-run) + the `WfDryValidate` one-step wrapper workflow + `dispatch_dry_validate` + worker registration (`temporal/tasks.py`) + isolation test + **`temporal-e2e-validate` Tier 2d** (3-process distributed proof).-- Phase 4 — **pending.** API dispatch (cross-repo `pipelex-api` `/validate`), gated on a pipelex release bump.-- Phase G0 *(optional, deferred)* — `temporalio` bump → true standalone activity (a runtime optimization over the wrapper workflow). **All decisions resolved 2026-06-09; no human gate remains** (dispatch = wrapper workflow on current `temporalio`).--**Mode 2 — done (landed on this branch via PR #986)** ([`followup-leaf-run-mode-mock.md`](./followup-leaf-run-mode-mock.md)):--- B1 — **done.** Generalized the leaf mock from `is_mock_inference` (LLM-only) to `run_mode=DRY` across all leaves (LLM, extract, img-gen, templating); shared `cogt/content_generation/dry_mock.py`.-- B2 — **done.** Collapsed each operator's `_dry_run_pipe` so DRY routes through the hub content generator (no more hardcoded `ContentGeneratorDry()`); `is_mock_inference` retired into the internal `is_mock_usage` DRY sub-flag. **Consumes `scoped_content_generator`** (built once by Mode 1).-- B3 — **done.** Temporal + DRY verified e2e via **`temporal-e2e-validate` Tier 17**: activities dispatched, leaves mock inside them, no real IO, no API keys (the req-1 gate).--**Net:** Mode 1 builds the two new scopes and lands the in-memory activity; Mode 2 builds the all-leaf mock and verifies the distributed-DRY path, reusing the content-generator scope. Neither blocks the other; the shared seam is built once.--## Acceptance — each mode gated by a specific distributed `temporal-e2e-validate` tier--Both modes must be proven in the real 3-process topology (server + split workers + submitter), not only in unit/integration. Each adds a tier to the repo's `temporal-e2e-validate` skill following the Tier 2c precedent (a `references/mode-2-tiers.md` scenario with GREEN/RED + worker-log strong-check, a Mode-1 pytest companion, a Step-7 master-table row).--- **Mode 1 (req 2) — Tier 2d.** A Temporal-dispatched `/validate` runs the whole sweep + graph dry-run inside **one in-process activity** (zero nested dispatch — strong worker-idle check), traces the graph **in memory** (no NDJSON/DDB), and returns `{status map, GraphSpec}` in one round-trip; best-effort graph → `None` on failure; direct mode unchanged (req 3). Spec in [`../../TODOS.md`](../../TODOS.md) § Distributed verification.-- **Mode 2 (req 1) — Tier 17.** A `--temporal --dry-run` run **dispatches** `act_llm_gen_*` (+ extract/img-gen) to the worker and **mocks inside** each activity — no real IO, no API keys, zero-token suppressed usage, cross-worker graph still assembles. Flips today's Tier 8 "dry-run never dispatches" note. Spec in [`followup-leaf-run-mode-mock.md`](./followup-leaf-run-mode-mock.md) § Distributed verification.diff --git a/wip/dry-run-refactor/followup-dry-validate-hardening.md b/wip/dry-run-refactor/followup-dry-validate-hardening.mddeleted file mode 100644index b0b2d3316..000000000--- a/wip/dry-run-refactor/followup-dry-validate-hardening.md+++ /dev/null@@ -1,60 +0,0 @@-# Follow-up: operational hardening of the dry-validate activity path--Deferred items from the post-implementation review of `feature/Dry-run-as-temporal-activity` (Part C, `act_dry_validate` + `wf_dry_validate` + `dispatch_dry_validate`). Each was verified against the code at review time; none blocks shipping — they are design tradeoffs or pre-existing shapes that deserve a deliberate pass, not a reflexive patch. Cheap fixes from the same review (teardown suppress-secondary, `workflow_execution_timeout=12min`, close-contract docs, tracer-open ordering, graph-arm parity, dropping `library_dirs`) were applied on the branch and are NOT listed here.--## 1. No heartbeat: a hung validation cannot be cancelled and outlives its timeout--`act_dry_validate` never calls `activity.heartbeat()` and `WfDryValidate` sets no `heartbeat_timeout`. In the Temporal Python SDK, cancellation is only delivered to an async activity through heartbeating — so when the 5-minute `start_to_close_timeout` fires, the *workflow* gives up but the worker coroutine keeps running to completion: the activity slot, the loaded library, and the `GraphTracerManager` entry are held until the run finishes on its own (or forever, if a pathological bundle spins), and the retry attempt starts a second copy of the same work on top. Validation input is user-authored pipe graphs; with recursive design in flight, whether a cyclic controller graph can spin indefinitely in DRY mode needs confirmation.--**Direction:** periodic `activity.heartbeat()` pulses (between the sweep and the graph arm, and inside `BundleValidator.validate_pipes`' per-pipe loop) + a `heartbeat_timeout` on the `execute_activity` call. Alternatively wrap the body in `asyncio.wait_for` matched to `start_to_close`.--## 2. CPU-bound activity body on the worker event loop--The activity body is almost entirely CPU work (TOML interpretation, pydantic construction, polyfactory mock minting, DRY sweep + graph trace) with essentially no real awaits — `validate_bundle` yields exactly once (`await asyncio.sleep(0)`). On a worker that co-hosts inference activities, a large bundle blocks the event loop for long stretches: co-resident activities' heartbeat/start-to-close timers keep ticking, risking spurious timeouts that re-run real, billed LLM calls.--**Direction:** run the validation core off the loop (`asyncio.to_thread`, or a sync `@activity.defn` using the worker's activity executor), or at minimum add a per-pipe `await asyncio.sleep(0)` in the sweep loop. Interacts with item 1 (heartbeats need the loop free to fire).--## 3. Validation verdict depends on worker-side config (drift surface)--`BundleValidator._aggregate` reads `dry_run_config.allowed_to_fail_pipes` and `dry_run_pipe_in_process` reads `pipeline_execution_config` / `graph_config.data_inclusion` from the process-local config. Direct mode resolves these from `pipelex-api-deploy`'s `.pipelex` toml; Temporal mode from `pipelex-worker`'s separately-pinned one. Config drift between the two deploy repos silently changes which pipes are allowed to fail (same bundle passes on one backend, 422s on the other) and how much data the returned `GraphSpec` carries.--**Direction:** either carry the decision-affecting knobs in `DryValidateArg` so the submitter's config is authoritative, or declare worker config the single source of truth for `/validate` and add a deploy-time drift check between the two repos' configs.--## 4. The main pipe is dry-run twice per /validate--The sweep dry-runs every sweepable pipe (untraced, `generate_graph=False`), then `dry_run_pipe_in_process` re-runs the main pipe's whole subtree to capture the `GraphSpec`. Inherited from the direct-mode shape (sweep and graph were always two runs), but now that both arms share one process and one library, a single traced run of the main pipe could feed both its `DryRunOutput` classification and the graph.--**Direction:** fold with the existing D-plan §7 endpoint-unification follow-up; exclude the main pipe from the untraced sweep loop and derive its classification from the traced run.--## 5. `LibraryManager._pipe_source_map` is process-global, keyed by bare `pipe_ref` (pre-existing)--One dict for all libraries: load overwrites entries, teardown pops by `pipe_ref` regardless of owner. Two concurrent `act_dry_validate` invocations whose bundles share a `pipe_ref` overwrite each other's entries, and the first teardown deletes the survivor's. Consumers are currently low-stakes (`get_pipe_source` → `which` CLI), but this feature makes concurrent multi-library validation the norm on workers.--**Direction:** scope the map per-library (key by `(library_id, pipe_ref)` or move it onto `Library`).--## 6. No selected-pipe validation field on the wire--`validate_bundle` supports `dry_run_pipe_codes` (used by `validate_pipe_in_bundle` for builder flows that validate one implemented pipe while siblings may fail); `DryValidateArg` doesn't carry it, so the Temporal path can't preserve that behavior if the API ever exposes pipe selection. Not needed today — the API validates whole bundles — but it's the first field to add when the builder flows move server-side.--## 7. Per-queue dispatch tuning doesn't reach `act_dry_validate`--`WfDryValidate` hardcodes `start_to_close_timeout=5min` / `maximum_attempts=2` in workflow code (per D-C5, precedented by `WfPipeRun`'s own hardcoded utility-activity bounds), so the v2 `resolve_dispatch` overlays (`queue_options`, per-handle options) cannot tune the validation activity. Fine while one bound fits all deployments; revisit if a deployment needs a larger validation budget (big bundles WILL hit the 5-minute cap — see item 1 for what happens then).--## 8. Payload-converter exceptions in workflow tasks hang the submitter (retry-forever), and aware datetimes were the live trigger--Root-caused 2026-06-10 (CI py3.11–3.14 hang on the three GraphSpec-crossing tests): an exception raised by the data converter while decoding an activity result fails the *workflow task*, which Temporal retries indefinitely — the submitter's `execute_workflow` never returns. The trigger was environmental (`ZoneInfoNotFoundError`: no IANA tzdata on the runner for py3.11+), and kajson 0.7.0 structurally eliminated that trigger class: its self-sufficient timezone wire format carries the UTC offset, so decoding degrades gracefully to a fixed offset instead of raising, and kajson declares `tzdata` itself. But the failure *shape* is general: any deterministic decode bug (kajson registry miss, schema drift, future enum rename) will present as an infinite hang on every host, not a failed run. `dispatch_dry_validate`'s 12-minute `workflow_execution_timeout` bounds the API path; direct `execute_workflow` callers (tests, scripts) have no bound.--**Direction (design tradeoff, do not reflexively patch):** consider making deterministic conversion errors fail-fast — e.g. catch `KajsonDecoderError` in `BaseModelPayloadConverter.from_payload` and re-raise as a non-retryable shape, or document that every submitter must set `workflow_execution_timeout`. Temporal's retry-forever default exists so a code redeploy can heal a bad task; trading that away deserves a deliberate decision.--## 9. The library-window teardown dance is hand-mirrored across two files (and already drifts)--Surfaced during the review of the validation-api extraction (`feature/Tweaks-for-validation-api`): the in-process validate orchestration moved out of `PipelexMTHDSProtocol.validate` into the shared `validate_bundles_in_process` (`pipelex/pipeline/validate_in_process.py`), so the local runtime and the upcoming `ApiRunner.validate` share one teardown implementation. But `act_dry_validate` (`pipelex/temporal/tprl_pipe/act_dry_validate.py`) still carries its own copy of the same restore-current-library + teardown-in-`try` + suppress-teardown-error-on-body-failure block — the comments literally say "mirrors act_dry_validate" / "Mirrors act_dry_validate's finally". So the load-bearing teardown semantics now live in *two* places (the new module deliberately unified runner + API but left the Temporal activity as a by-hand mirror).--The two copies have already diverged at the edges — none documented as intentional:--- `validate_in_process` gates the whole block on `validation_library_id is not None and validation_library_id != prev_library_id`, then tears down unconditionally inside.-- `act_dry_validate` gates the restore on `prev_library_id is not None and prev_library_id != library_id` (with an `else: clear_current_library()` branch the other site never executes), and gates only the teardown on `if library_id is not None`.--The behavior difference is benign today and the divergence is **pre-existing** (the extracted block is byte-identical to the old `runner.py` code — the extraction introduced no new drift), but "mirrors act_dry_validate" is now aspirational, which is exactly the failure mode the docstring's "lives once, so the two backends cannot drift" claim is meant to prevent.--**Direction (design tradeoff, do not reflexively patch):** collapse both sites onto one implementation — a `teardown_validation_library(*, validation_library_id, prev_library_id, body_succeeded, log_context)` helper, or a `validation_library_window(...)` context manager that owns the restore-first / teardown / suppress-on-error contract. The activity keeps its own `body_succeeded` flag; only the finally body is shared. Worth doing when `ApiRunner.validate` lands (third consumer), so the unification is proven against all three call shapes at once rather than guessed at now.diff --git a/wip/dry-run-refactor/followup-leaf-run-mode-mock.md b/wip/dry-run-refactor/followup-leaf-run-mode-mock.mddeleted file mode 100644index a862a4587..000000000--- a/wip/dry-run-refactor/followup-leaf-run-mode-mock.md+++ /dev/null@@ -1,190 +0,0 @@-# Follow-up — Leaf-level run-mode mock (DRY honors the backend)--> **⚠ Decision D8 superseded (2026-06-10, post-completion):** `is_mock_inference` will NOT survive as a thin reportable-mock mode — it becomes an internal `is_mock_usage` sub-flag of DRY. See [`design-unified-dry-run.md`](./design-unified-dry-run.md). Everything else in this file stands as the as-built record.->-> **Status: COMPLETE (2026-06-10) — built on `feature/Mock-activities`, all phases done, Checkpoint E reached.** Implements **D-plan Part B / D4 / req 1**. Split out of [`consolidation-as-built.md`](./consolidation-as-built.md) on 2026-06-01 (eng-review D1) so the in-process consolidation ships alone.->-> **Design rationale:** [`D-plan.md`](./D-plan.md) §3.5 (run mode ⟂ backend) and §4.8 (leaf-level mock). **Risks:** D-plan §8 (object-mock fidelity, req-1 fidelity regressions).->-> **Depends on:** the consolidation ([`consolidation-as-built.md`](./consolidation-as-built.md)) is *not* a hard prerequisite (this is a separable cogt/operator refactor). **Branch off the same D-plan.**->-> **Relationship to the in-memory activity follow-up (corrected 2026-06-09):** this follow-up (req 1) is **orthogonal** to [`followup-temporal-validation-activity.md`](./followup-temporal-validation-activity.md) (req 2), **not** a prerequisite for it — see "Still relevant — a distinct dry-run mode" below.->-> **⚠️ Coordinate with the registry branch's Phase 5 (`fix/For-API-update`, decided 2026-06-06).** That branch ships `--mock-inference` / `is_mock_inference` as an *interim* trigger, deliberately built **leaf-first so it lands B1's core ahead of time**: a shared `cogt/content_generation/dry_mock.py` + a per-leaf dry branch keyed on a per-run flag carried on `JobMetadata`. So when this follow-up runs, **B1 collapses to "re-key that helper from `is_mock_inference` → `run_mode`"** (carrier + helper already exist — verify before building from scratch), and **B2 settles the fate of `is_mock_inference`** (retire vs keep a thin reportable-mock — see B2). One distinction to preserve when re-keying: `--mock-inference` emits *non-zero* synthetic usage so a cost report renders; `--dry-run` stays zero-token and its report is *suppressed*. See the registry feature's [`../history/registry/deferred-followups.md`](../history/registry/deferred-followups.md) (`--mock-inference` coverage + the `is_mock_inference` fate decision).--Goal: move the LIVE/DRY decision **down to the cogt leaf** so DRY honors the configured backend — DRY-on-Temporal dispatches `act_llm_gen_*` and mocks **inside** the activity, retiring the "DRY → local in-process" shortcut. Separable cogt/operator refactor (§4.8). Pre-flight items are decided; eng-review decisions (2026-06-10, D2–D11) are folded into the phase checklists below.--The threading route after Part B:--```-PipeRunParams.cogt_run_params (run_mode lives ONLY here; .run_mode = delegating property)- └─ built once at PipeRunParams construction (prepare_pipe_job resolves boot force-DRY)- └─ operator (one path, no Dry-swap) passes cogt_run_params into generator methods- └─ hub ContentGenerator | ContentGeneratorInWorkflow ← backend choice ONLY- └─ assignment.cogt_run_params ← serializes across the wire- └─ [direct: inline] | [temporal: act_*_gen_* on worker]- └─ leaf: run_mode==DRY → dry_mock (no provider, no storage)- LIVE → real call (+ store for img/extract)-```--## Still relevant — a distinct dry-run mode (do NOT drop this file)--This describes a **different** dry-run mode from the in-memory in-process activity ([`followup-temporal-validation-activity.md`](./followup-temporal-validation-activity.md)). Both are wanted; they're orthogonal `run_mode × backend` combinations of one foundation (mock at the leaf; the backend decides where the leaf runs — D-plan §3.5):--- **This file (req 1) — full distribution, leaf-only mocks.** A DRY run goes through the **real** Temporal path (`WfPipeRouter` → child workflows → `act_llm_gen_*` activities) and the **leaf inside each activity** mocks instead of calling the model. Purpose: **test the distribution machinery** (dispatch, scheduling, serialization, routing, cross-worker propagation) without AI cost or latency.-- **The other file (req 2) — one in-process activity, in-memory tracing.** The whole dry-run + validation runs **in-process inside a single activity** (nothing nested is dispatched), tracing the graph in memory. Purpose: offload validate+graph to a worker cheaply.--**Already partly shipped:** `is_mock_inference` (the registry branch's interim trigger) **is** the LLM slice of this mode — `run_mode` stays LIVE so operators dispatch `act_llm_gen_*` normally, but the leaf fakes the call (`JobMetadata.is_mock_inference` → the `llm_generate.py` leaf branch). So req-1 behavior already coexists, today, alongside the pipe-level DRY path — concrete proof the two modes don't conflict. B1/B2 generalize it from LLM-only + `is_mock_inference` to all leaves + `run_mode=DRY`.--**Shared seam with the in-memory activity — `scoped_content_generator` (RESOLVED: it exists, consume it).** B2 routes the DRY mock through `get_content_generator()` at the leaf. Under a Temporal-enabled hub that returns `ContentGeneratorInWorkflow` **globally** (boot-time, `pipelex.py:370-385`). The in-process activity (req 2) landed first and **built the inline-content-generator scope** — `scoped_content_generator` is live (`bundle_validator.py:260` uses it, see `TODOS.md` Phase 2). This follow-up consumes it: B2's Mode-1 scopes (`validate_pipes`, `dry_run_pipe_in_process`) swap `ContentGeneratorDry()` for an inline `ContentGenerator` inside the existing scope.--## Status at a glance--| Phase | Title | Status | Commit |-|---|---|---|---|-| B1 | Thread `run_mode` to cogt leaf + leaf DRY branch | ☑ done | `feat(dry-run): thread run_mode to the cogt leaf via CogtRunParams` (2026-06-10, review-fixed) |-| B2 | Collapse operator dry path | ☑ done | `feat(dry-run): collapse the operator dry path onto the leaf mock` (2026-06-10) |-| | **⛔ CHECKPOINT D — cleared 2026-06-10** | | |-| B3 | Verify Temporal + DRY e2e (**req-1 gate**) | ☑ done | `test(temporal): Tier 17 — DRY honors the Temporal backend` (2026-06-10) |-| | **⛔ CHECKPOINT E — follow-up complete — REACHED 2026-06-10** | | |--Status legend: ☐ not started · ◐ in progress · ☑ done.--## Pre-flight — ALL DECIDED (2026-06-10, with the user)--- [x] **(DECIDED, refined by eng review D2 2026-06-10) `run_mode` carrier = a new `CogtRunParams` class, nested INSIDE `PipeRunParams`, holding the only copy of `run_mode`.** The cogt-tier params class `CogtRunParams` (`pipelex/cogt/content_generation/cogt_run_params.py`, a `BaseModel` — it crosses the Temporal wire) becomes a field `PipeRunParams.cogt_run_params`. **`run_mode` lives ONLY in `CogtRunParams` — zero duplication**: `PipeRunParams.run_mode` becomes a delegating property (`return self.cogt_run_params.run_mode`) so existing pipe-tier readers (`_run_pipe_traced` match, factories) keep working unchanged. Single writer: wherever `PipeRunParams` is constructed (`make_run_params` factory / `prepare_pipe_job`, which also resolves the boot force-DRY flag into it). Route to the leaf: operators slice `pipe_run_params.cogt_run_params` off and pass it into the content-generator protocol methods (**12-method signature change** across `ContentGeneratorProtocol` + every implementation — `ContentGenerator`, `ContentGeneratorInWorkflow`, `ContentGeneratorDry` until B2 deletes it, the `temporal/test_extras` workflow callers, and test doubles); generators stamp it as a field on each cogt assignment (`LLMAssignment`, `ObjectAssignment`, img-gen/extract/templating/search/render); the leaf keys on `cogt_run_params.run_mode == DRY`. Temporal converter work is ~zero (activities take the assignment models whole; a new pydantic field rides free). Rationale: type-explicit on the execution contract (can't be missed when building an assignment), keeps `JobMetadata` pure tracing/reporting identity, and gives `is_mock_inference` a natural home to migrate into (one flag-transport style, not two) — the class starts with one attribute by design; it is the cogt slice of the run params and will grow. **Rejected:** the `JobMetadata` mirror (the doc's old default — drifts metadata toward a behavior-flag grab-bag), threading `PipeRunParams` into cogt (layer violation — drags pipe-tier concepts like multiplicity/batch params into a deliberately pipe-agnostic layer), and a stored duplicate `run_mode` on both `PipeRunParams` and `CogtRunParams` (copy-drift risk). Consequence for B1: not a pure re-key — B1 introduces `CogtRunParams`, the `PipeRunParams` nesting + property, the protocol/call-site threading, and the assignment field, then re-keys the leaf branch; `is_mock_inference` migrates into `CogtRunParams` in B2 (fate decided: see B2).-- [x] **(DECIDED) Object mock = single schema-based site + fidelity test.** Both direct and dispatched leaves build via `SchemaToModelFactory` from the JSON schema — one code path, identical mock on both backends (the backend-parity this follow-up exists to deliver); fidelity bugs surface in cheap local unit tests instead of only on a worker. Accepted cost: direct mode gives up the in-hand original class — exotic format constraints (`json_schema_extra` hints dropped on round-trip) must declare `examples`/`mock_format`; the mandated fidelity test on a representative `StructuredContent` pins this.-- [x] **(DECIDED) Synthetic dry LLM report = keep, into a throwaway per-sweep registry.** The leaf stays unconditional (no "am I validating?" flag threading into cogt); the validation sweep opens one throwaway `UsageRegistry` so events have a home and die with it. Zero-token suppression already guarantees no rendered report; the runner-emission e2e keeps its cheap observable signal.-- [x] **(DECIDED) `ContentGeneratorDry` = delete outright in B2.** One mock mechanism remains: `run_mode=DRY` at the leaf. The boot `not needs_inference` fallback (`pipelex.py:371`) is re-expressed as a forced-DRY flag consumed at `prepare_pipe_job` (B2 mandated this anyway); Mode-1's scopes (`validate_pipes`, `dry_run_pipe_in_process`) pass the inline `ContentGenerator` instead (their callers already run DRY); tests using it as a double switch to the run_mode-threaded equivalent.--## Phase B1 — Thread `run_mode` to the leaf + add the leaf DRY branch (§4.8)--> **B1's core LANDED on the registry branch's Phase 5 (`fix/For-API-update`).** Already built: `JobMetadata.is_mock_inference`, the shared `cogt/content_generation/dry_mock.py` (synthetic-job reporting parameterized zero-vs-non-zero, plus the leaf mocks + `build_mock_object`), and the per-leaf branch in `llm_generate.py` keyed on `is_mock_inference`. So B1 reduces to: (1) carry `run_mode` on `JobMetadata` alongside `is_mock_inference` (same single-writer, `prepare_pipe_job`), and (2) re-key the leaf branch from `is_mock_inference` → `run_mode==DRY`. **Don't rebuild the helper.** Two caveats to finish what Phase 5 deliberately scoped out: it covers the **LLM leaf only** — `img_gen_*` / `extract_gen_pages` / `templating_gen_text` leaf branches are still TODO here (Phase 5 deferred them because their output is stored above the leaf; see that branch's §7); and `ContentGeneratorDry` still owns its method bodies (only its *reporting* delegates to `dry_mock`), so B2's collapse is still real work.--> **Interim hard guard already landed (F1, registry branch).** `img_gen_single_image` / `img_gen_image_list` / `extract_gen_pages`, plus the search leaf, now **raise `MockInferenceUnsupportedError`** under `is_mock_inference` instead of silently calling the real provider. So when B1 adds the real synthetic-output leaf branch for img-gen/extract/search here, it is **replacing a fail-loud guard, not patching a silent-spend path** — delete the guard at that leaf as you add the DRY branch (under decision 8B the guard's `is_mock_inference` arm stays wherever no reportable-mock is built). See [`../history/registry/deferred-followups.md`](../history/registry/deferred-followups.md) (`--mock-inference` coverage).->-> **Stale-premise correction (eng review D5, 2026-06-10): search HAS a cogt leaf now.** `search_generate.py` landed as a framework-agnostic leaf (sibling of `llm_generate`, with `act_search_gen_sourced_answer` / `act_search_gen_structured` activities), so the old "web search has no `content_generation` leaf" claim is outdated. **Decision: full uniform treatment** — the search leaf and the `render_page_views` leaf both get `run_mode` DRY branches in B1 (lift `make_search_sourced_answer` / `make_search_structured` / `make_render_page_views` mock bodies from `ContentGeneratorDry:299-319,222-244` into `dry_mock.py`), `PipeSearch` joins B2's swap-removal list (`pipe_search.py:177`), and Tier 17 extends to the search activities.->-> **Mock typing is leaf-shaped, not generator-shaped (outside-voice fold, 2026-06-10).** The img/extract leaves return `GeneratedImageRawDetails` / `ExtractOutput`, not `ImageContent` / `PageContent` — "lift bodies from `ContentGeneratorDry`" means lift the *semantics* and build mocks at each leaf's own return type; the generator-level conversion (`make_image_content` / `make_page_contents`) stays where it is.->-> **DRY short-circuits storage (eng review D10, 2026-06-10).** The img/extract Temporal activities store outputs via `GeneratedContentFactory` + `get_storage_provider()` (`act_img_gen_generate.py`, `*_and_store` functions). The DRY branch sits **above the store step**: mock-and-return URL-only `ImageContent`/`PageContent` (what `ContentGeneratorDry`'s fakes already are) without touching the storage provider — dry means dry: no storage IO, no storage credentials, no mock artifacts in real buckets. Name this asymmetry explicitly (the DRY branch for img/extract lives at the `*_and_store` layer, one step above the raw provider leaf) — do not "unify" it downward. Tier 17 asserts no storage writes.--- [x] Confirm the funnel by grep: `ContentGenerator.make_llm_text` and `ContentGeneratorInWorkflow.make_llm_text` both build `LLMAssignment` and converge on `llm_gen_text(assignment)` (inline vs via `act_llm_gen_text`). ✓ confirmed.-- [x] **Introduce `CogtRunParams`** (`pipelex/cogt/content_generation/cogt_run_params.py`, `BaseModel`) nested as `PipeRunParams.cogt_run_params`; `run_mode` lives ONLY there; `PipeRunParams.run_mode` becomes a delegating property so existing readers are untouched. Single writer at `PipeRunParams` construction (`make_run_params` / `prepare_pipe_job`). *As built:* `PipeRunParams` got `extra="forbid"` so a stale `run_mode=` kwarg fails loudly; `PipeRunMode` importers were repointed from the `pipe_run_params` re-export to `pipe_run_mode` directly; the redundant `branch_pipe_run_params.run_mode = …` writer in `pipe_batch.py` (a no-op after `model_copy(deep=True)`) was deleted.-- [x] **Thread it through the protocol**: leaf-dispatching `ContentGeneratorProtocol` methods gain a `cogt_run_params` parameter; all implementations + operator call sites updated; generators stamp it as a field on every assignment model. *As built (deviation from the "12-method" framing):* 8 raw-param methods gained the parameter; the two search methods take a pre-built assignment, so `PipeSearch` stamps `cogt_run_params` at `SearchAssignment` construction (no redundant param); `make_image_content` / `make_page_contents` are post-leaf conversion steps with no leaf dispatch and were deliberately left un-threaded. `ObjectAssignment` / `SearchObjectAssignment` expose `cogt_run_params` as delegating properties over their nested assignment (zero duplication).-- [x] **Carry `nb_items: int | None` on `ObjectAssignment`** (eng review D11): both generators pass it through; the dry AND mock leaf list mocks honor `nb_items or config.nb_list_items`.-- [x] *Tests:* per-leaf DRY-branch unit tests (`test_*_dry_branch.py` per leaf module); object-mock-from-schema fidelity pin on a representative `StructuredContent` + generator-level D6 DRY-arm tests (`test_leaf_dry_object_mocks.py`); fixed `nb_items` regression (leaf + through `ContentGenerator`); templating DRY still raises on a bad jinja2 template; `LLMAssignment` + `PipeRunParams` wire round-trip with `cogt_run_params` through the Temporal payload converter. *Deferred to B2:* the builder-bundle dry-validation `mock_main` pin (it lands with B2's `stamp_mock_main_coordination()` helper).-- [x] Add the DRY branch to each leaf, keyed on `cogt_run_params.run_mode.is_dry` — `llm_gen_text` / `llm_gen_object` / `llm_gen_object_list` (zero-token `report_dry_llm_job` + schema-built mocks), `img_gen_*_and_store` / `extract_gen_pages_and_store` (**above the store step** — D10), `render_page_views_and_store` (new shared leaf `render_generate.py`, now also used by `act_render_page_views` AND direct `ContentGenerator` — direct img/extract/render converge on the same `*_and_store` leaves as the activities), `templating_gen_text` (preserves `check_jinja2_parsing`), `search_gen_*` (lifted mocks — D5). Semantics lifted from `ContentGeneratorDry` into `dry_mock.py` `dry_*` helpers. *Note:* the `MockInferenceUnsupportedError` guards stay as-is — they are keyed on `is_mock_inference` only (which still has no non-LLM reportable mock per D8), so there was nothing to delete at the DRY arm.-- [x] **Typed terminal mock-build errors** (eng review D7): `build_mock_object` wraps polyfactory/validation failures into `DryRunMockBuildError` naming the class and the remedy; unit test pins the type (`test_dry_run_mock_build_error.py`). Error pages regenerated.-- [x] **Re-key the object-fidelity wrapper** (eng review D6): both `_revalidate_against_object_class` copies (direct + in-workflow) fire on `is_mock_built = run_mode.is_dry or is_mock_inference`; `MockInferenceObjectFidelityError` docs/message generalized (remedy = `examples`/`mock_format`, no longer "use --dry-run"); DRY-arm unit + integration tests added.-- [x] **Comment/doc maintenance**: `dry_mock.py` module docstring rewritten for the post-B1 world; `JobMetadata.is_mock_inference` comment updated (D8 migration noted); stale "search has no leaf mock" comment fixed.-- [x] Operators still call `ContentGeneratorDry` for now — **no behavior change yet**. `make agent-check` + `make agent-test` green (2026-06-10).--## Deferred items from the B1 code review (2026-06-10)--Findings from the post-B1 review round that were verified real but deliberately NOT fixed in B1 — each is a design tradeoff or a later-phase concern, not a silent bug in shipped behavior:--- **Storing activities resolve `get_storage_provider()` before the leaf's DRY branch** (`act_img_gen_generate.py`, `act_extract_generate.py`, `act_render_page_views.py`): D10's "no storage credentials under DRY" holds only because boot always constructs a provider from config (provider construction needs no credentials; only real IO does). If B3's Tier 17 no-keys arm ever boots a worker without a storage provider, move acquisition inside the leaf's LIVE arm. Verify in B3.-- **Sequential page-view storage uploads** in `render_page_views_and_store` (pre-existing behavior, relocated): each rendered page is stored one `await` at a time; a large PDF pays serialized upload latency inside the activity. Candidate `asyncio.gather` with a bound — perf-only, touch when a real workload hurts.-- **Ambient `dry_run_config.nb_extract_pages` read at two leaves** (`dry_extract_page_contents`, `dry_render_page_views`): under Temporal these run in different activities, so heterogeneous worker configs could mismatch page counts and trip the generator's length check on a dry run. D11-style fix = carry the resolved count on the assignments. Revisit in B3 (Tier 17 would surface it).-- **No cross-call memoization of `DryRunFactory.make_dry_run_factory`**: list mocks now build the factory once per call (`build_mock_objects`), but repeated leaf calls for the same class still rebuild it. An lru-cache keyed by class adds lifetime/invalidation surface for dynamically-created classes — skipped as over-engineering until dry-run profiling says otherwise.-- **Schema-reconstruction failures stay untyped** (`_reconstruct_object_class` → `SchemaToModelFactory`): D7 wrapped the polyfactory build; codegen failures (ValueError / datamodel-code-generator errors) share their surface with the LIVE leaf path (same function, same schema), so typing them only on the mock path would add asymmetry. Belongs to the broader "retry-forever on unconverted exceptions" hardening family already tracked.--Additional deferred items from the B2 code review (2026-06-10):--- **`PIPELEX_FORCE_DRY_RUN_MODE` env var vs the hub forced-DRY flag**: two run-mode-forcing mechanisms now coexist — the env var (resolved in `pipeline_run_setup`, only a *default* despite its name: an explicit run mode wins, per its own `# TODO: rethink this, it's not forcing`) and the hub flag (genuinely forcing, resolved at `make_run_params`). Nothing in the repo sets the env var; folding it into the hub flag (or deleting it) is a user-facing-surface decision — defer.-- **Mock-input list factory not hoisted** (`working_memory_factory.make_mock_inputs` builds a `DryRunFactory` per item): NOT a pure hoist — `_get_mockable_class` picks a *random concrete subclass per item*, so hoisting would change mock diversity. Left as is.-- **`ContentGeneratorInWorkflowFactory` is now a param-less wrapper**: kept as the deliberate lazy-import construction seam (boot imports it deferred); revisit if it never grows params.-- **Mock-inference guard not consolidated**: the `MockInferenceUnsupportedError` guards at the img/extract/search leaves stay inline (3 small sites with distinct operation labels); consolidation would save little.--Deferred items from the PR #983 review-bot round (2026-06-10):--- **Direct structured-search live validation stays a raw `ValidationError`** (greptile P1 / cubic P2): the Temporal arm wraps it in `ContentGenerationError` because an unconverted exception in workflow code retries forever and hangs the submitter — a boundary-specific failure mode the direct path doesn't have (the raw error propagates and is reported normally). Unifying would require moving/duplicating a temporal-tier error class into cogt; not worth the layering churn for a cosmetic consistency gain. Mock-built failures ARE unified (typed `MockInferenceObjectFidelityError` on both arms, D6).-- **Raw img/extract leaves carry no DRY fail-loud guard** (flagged independently by two adversarial passes): only the `*_and_store` wrappers branch on DRY; a future direct caller of `img_gen_single_image`/`img_gen_image_list`/`extract_gen_pages` with a DRY assignment would hit the real provider. Today no such caller exists (grep-verified) and the D10 docstrings forbid unifying downward — kept documentary rather than adding guards for an unreachable path; revisit if a new raw-leaf caller appears.-- **Per-call schema hashing + global codegen cache lock** (perf, /review final gate): every dry object mock pays `json.dumps`+`sha256` of the full schema for the cache lookup, and `SchemaToModelFactory.make_from_json_schema` holds one process-global lock even on cache hits (documented tradeoff in that module) — now on the DRY path too. Candidates: precomputed schema hash on `ObjectAssignment`, double-checked locking. Touch when dry-sweep profiling says so.-- **Assignment models are extra-ignore while the nested carrier is extra-forbid**: stale keys on `LLMAssignment` etc. are silently dropped. Broader wire-hygiene sweep deferred (belongs with the decode-error hardening family).-- **`Pipelex.make()` failure poisons the test worker** (observed in CI): a boot that raises mid-`setup()` leaves the singleton registered with `LogConfig` set, and `teardown()` of the half-initialized instance raises before `reset_config()` — every later module-scoped boot on that xdist worker then fails with "LogConfig is already set". Pre-existing test-infra hardening item (make `teardown()` tolerate a half-setup instance, or have `make()` clean up on failure); surfaced by the keyless-boot test's control arm booting PYTEST mode in CI (fixed by mirroring the conftest's CI-mode selection).--## Phase B2 — Collapse the operator dry path (§4.8)--- [x] Remove the pipe-level `ContentGeneratorDry()` swap from each operator's `_dry_run_operator_pipe` (`PipeLLM`, `PipeCompose`, `PipeImgGen`, `PipeExtract`, `PipeStructure`, `PipeSearch` — D5). DRY now routes through the hub content generator; `cogt_run_params` (carrying DRY) rides into every assignment and the leaf mocks. *As built (post-review):* the delegation is the base-class default (`PipeOperator._dry_run_operator_pipe`); only `PipeFunc` and `PipeCompose` override; the dead `content_generator` parameter was removed from the operators' `_live_run_operator_pipe` and from `StructuredContentComposer` (the scoped contextvar is the only generator-override seam); the in-process scopes use `ContentGenerator.make_inline()`; `MOCK_MAIN_PIPE_CODE` is a shared constant imported by `bundle_header_spec`. Also: `PipeCompose`'s dry path kept its construct-mode `PipeComposeError` fallback but dropped the generator selection entirely (the composer renders templates directly and never calls the generator), which made `dry_run_config.apply_to_jinja2_rendering` fully dead — the config key was removed (configs.py + pipelex.toml).-- [x] **Consolidate the `mock_main` coordination into one shared helper** (eng review D3): `stamp_mock_main_coordination()` lives in `dry_mock.py` with the one WHY docstring (coordination with `bundle_header_spec.py` `examples=["mock_main"]`); called by `working_memory_factory`, `pipe_batch`, and the dry object-list leaf (`dry_llm_gen_object_list`) — `--mock-inference` keeps not stamping. Regression tests pin both arms (`test_leaf_dry_object_mocks.py`).-- [x] Re-express the boot `not needs_inference` fallback as force-`run_mode=DRY`: `PipelexHub.set_dry_run_forced(...)` set at boot, consumed at `PipeRunParamsFactory.make_run_params` — the single writer of `run_mode` — so EVERY entry point that builds run params is covered (pipeline API, runtime bridge, `PipeJobFactory` defaults); a warning logs when forced-DRY swallows an explicit `--mock-inference`. **Generator selection backend-keyed unconditionally** (D4). Regression tests in `test_keyless_boot_forced_dry.py`: keyless+temporal ⇒ in-workflow generator + forced DRY; keyless direct ⇒ inline + forced DRY; control arm; LIVE-requested job under keyless boot prepared as DRY. `ContentGeneratorDry` **deleted**; `bundle_validator` + `dry_run_pipeline` scopes pass an inline `ContentGenerator`; `temporal/test_extras` workflows always use the in-workflow generator (the dry arm now dispatches and mocks inside activities — leaf parity in action); its remaining test doubles repointed (`test_scoped_content_generator`, `test_dry_report_tz` (renamed), `test_dry_run_graph_in_process` simplified to real post-Part-B behavior); redundant `test_content_generator_dry_reporting.py` deleted (covered by the leaf dry-branch tests).-- [x] **`is_mock_inference` migrated from `JobMetadata` into `CogtRunParams`** (D8): single writer `PipeRunParamsFactory.make_run_params` (fed by `prepare_pipe_job` off the CLI flag); all leaf branches/guards and both fidelity wrappers re-keyed; `JobMetadata` field deleted; tests migrated (the temporal mock-inference test now injects the flag via `pipe_run_params.cogt_run_params`, pinning wire survival on the new carrier). The CLI surface (`--mock-inference`, runner/pipeline params) is unchanged, so Tier 8b's CLI-level arms keep working as written; its prose gets the Tier-8-note update in B3.-- [x] **Usage isolation** (pre-flight decision 3, resolved as-built): there is no ambient `UsageRegistry` anymore — usage events are dropped unless a trace context with `emit_usage_events=True` is registered, and validation sweeps never register one. Pinned by the new isolation assertion in `test_dry_run_graph_in_process` (dry run emits graph events but zero `UsageReportEvent`s through the scoped in-memory transport).-- [x] `PipeSignature._dry_run_pipe` and controller dry behavior unchanged (no leaf op) — untouched.-- [x] Verify **direct + DRY** outcomes: full `make agent-test` green (2026-06-10) + spot-checked `pipelex validate --all` and a dry `run bundle native_text_sequence --dry-run --mock-inputs` (output carries the leaf marker `DRY RUN: llm_gen_text`, proving the leaf mock served the run); fixed-`nb_items` pinned by `test_generator_dry_object_list_keeps_fixed_length`; builder-bundle dry-validation covered by the suite's builder tests + the `mock_main` leaf pins.--> ### ⛔ CHECKPOINT D — after Phase B2 — **REACHED 2026-06-10**->-> Direct DRY re-expressed through the leaf with identical outcomes; Temporal cell not yet verified.->-> **Verified:** `make agent-check` clean · full `make agent-test` green · dry `validate --all` + dry bundle run spot-checked (leaf marker confirmed in output) · committed.->-> **Handoff:**->-> - **Carrier shape (final):** `CogtRunParams { run_mode: PipeRunMode = LIVE, is_mock_inference: bool = False }` in `pipelex/cogt/content_generation/cogt_run_params.py`. Nested as `PipeRunParams.cogt_run_params` (only copy; `PipeRunParams.run_mode` is a read-only delegating property; `extra="forbid"`). Single writer: `PipeRunParamsFactory.make_run_params(pipe_run_mode=…, is_mock_inference=…)` fed by `prepare_pipe_job`; the factory itself consumes the hub's forced-DRY flag (`PipelexHub.set_dry_run_forced` set at boot when `needs_inference=False`), covering every entry point incl. the runtime bridge. `CogtRunParams.is_mock_built` is the derived fidelity-guard predicate (DRY wins over `is_mock_inference` at every leaf). Every cogt assignment carries `cogt_run_params` (required field; `ObjectAssignment`/`SearchObjectAssignment` delegate via property to their nested assignment).-> - **`ContentGeneratorDry`: deleted.** One mock mechanism remains: the leaf branches keyed on `cogt_run_params` (`run_mode.is_dry` for DRY; `is_mock_inference` for the thin reportable LLM mock). The in-process scopes (`bundle_validator`, `dry_run_pipeline`) pass an inline `ContentGenerator`; under DRY its leaves mock without storage IO.-> - **Object-mock decision (held):** single schema-based site — both backends build via `SchemaToModelFactory` + polyfactory (`_leaf_gen_object(_list)` in `dry_mock.py`); fidelity failures surface as `MockInferenceObjectFidelityError` on both the mock and DRY arms; deterministic build failures as `DryRunMockBuildError` (non-retryable on Temporal via `non_retryable_error_types`).-> - **Next entry point: Phase B3 — Temporal+DRY e2e (Tier 17)** — needs a real Temporal server (3-process Mode 2); use the `temporal-e2e-validate` skill. Also update the skill's Tier 8 note + Tier 8b prose there.--## Phase B3 — Verify Temporal + DRY end-to-end (**req-1 acceptance gate**)--The acceptance gate is a **specific distributed scenario** in the repo's `temporal-e2e-validate` skill — **Tier 17** — built to the Tier 2c/2d precedent (a 3-process scenario in `references/mode-2-tiers.md`, a Mode-1 pytest, a Step-7 master-table row). Full spec below.--- [x] **Add `temporal-e2e-validate` Tier 17 — "DRY honors the Temporal backend (leaf mock inside the activity)".** Added as Step 5g in `mode-2-tiers.md` (GREEN LLM + extract/img-gen/search arms + no-keys arm + RED mutate-the-leaf procedure) + Step-7 master-table row + SKILL.md routing entry; the Tier 8 / Tier 8b / routing-battery stale notes were already corrected in B2's review round. Mode-1 pytest: `tests/integration/pipelex/temporal/tracing/test_dry_run_dispatches_and_mocks.py` (under `tracing/` so the directory conftest applies the CI-xdist quarantine and the tmp-dir trace redirect) — asserts dispatch via workflow history across the parent + child workflow tree (the LLM activity is scheduled by the controller's child workflows, not the root), the `DRY RUN:` leaf output, and zero-token suppressed usage under the `dry_run` sentinel.-- [x] `make agent-check` + `make agent-test` green; **Temporal e2e RUN AND GREEN (2026-06-10, real `temporal server start-dev` + split router/runner workers)**: GREEN LLM arm (exit 0, `act_llm_gen_text` ×2 executed on the runner, zero provider/storage lines, `DRY RUN: llm_gen_text` in output, no cost table, `reactflow.html` assembled) · GREEN extract (`act_extract_gen_extract_pages` + `act_render_page_views` dispatched) · GREEN img-gen (`act_img_gen_images`) · GREEN search (`act_search_gen_sourced_answer`) · **no-keys GREEN** (runner booted with invalid credentials — dry run still exit 0) · **RED-proven** (DRY branch disabled in `llm_gen_text` + keyless runner → exit 1, dispatched activity failed loud with HTTP 401 carried back to the submitter) · recovery confirmed (leaf restored → exit 0).--## Distributed verification — `temporal-e2e-validate` (Mode 2 / Tier 17)--This tier **flips the current Tier 8 note** ("dry-run instantiates `ContentGeneratorDry` on the router and never dispatches `act_llm_gen_text`"): after Part B, DRY honors the backend and DOES dispatch, mocking inside the activity. That behavior change is the req-1 deliverable, so its proof is a new tier.--**Tier 17 — DRY honors the Temporal backend (leaf mock inside the activity).** New sequential tier (Step 3 family).--**Mode 2 (3-process) GREEN** — split workers up (`mode-2-setup.md`), Temporal-enabled; run a multi-step LLM bundle DRY:--```-pipelex run bundle .../library_crate/native_text_sequence.mthds --pipe native_text_sequence --temporal --dry-run --mock-inputs --no-logo --graph-```--- exit 0; **the worker DID dispatch `act_llm_gen_text`** (grep the runner session / `WorkflowHandle.fetch_history()` shows `ActivityTaskScheduled` for it) — the runner actually ran the activity. *(Pre-Part-B this dispatches nothing — that's the behavior we're changing.)*-- **The leaf mocked inside the activity:** no real provider call, **runs with NO API keys** (the "dry works with no inference configured" invariant must hold even though the activity dispatches), zero real spend, output is the DRY mock (`"DRY RUN: …"` / minted object), and usage is **zero-token / report suppressed** — distinct from `--mock-inference`, which keeps non-zero synthetic usage for a rendered cost report (kept per decision 8B).-- **No storage writes** (eng review D10): the DRY img/extract path never touches the storage provider — assert no storage IO occurred (no bucket/object writes in the worker session).-- cross-worker graph tracing assembled (`reactflow.html`); LibraryCrate propagation + serialization behaved as LIVE.-- **Extend to non-LLM leaves once B1 adds them:** rerun against extract (`pdf_extract_page_views`), img-gen (`generate_image`), and **search** (eng review D5) and assert `act_extract_gen_extract_pages` / `act_img_gen_images` / `act_search_gen_*` are dispatched and mocked inside — this **replaces** the `MockInferenceUnsupportedError` fail-loud guard at those leaves.--**Mode 2 RED (prove it bites):** temporarily key the leaf branch to LIVE (mutate the `run_mode` check in `llm_generate.py`) → the dispatched activity attempts a real call and **fails loud on missing keys / would spend** — the **dispatched + mocked-inside + no-spend + no-keys** quartet flips. Restore. *(Outside-voice fold: post-B2 there is no router-level mock to "revert to", so the RED arm is mutate-the-leaf, not revert-the-re-key.)*--**Mode 1 (pytest) companion** — `tests/integration/pipelex/temporal/tracing/test_dry_run_dispatches_and_mocks.py` (the DRY analogue of the existing `test_mock_inference_temporal.py`): assert `run_mode=DRY` over the in-process server dispatches `act_llm_gen_*` and the leaf mints the DRY mock with **zero-token, suppressed** usage, no real inference.--**Coordination with Tier 8b.** The existing `--mock-inference` cost arms (Tier 8b) are the LLM **cost-rendering** slice of this same mode (LIVE run mode + leaf mock, non-zero usage). **Decided (D8): the thin reportable-mock survives**, migrating into `CogtRunParams` — Tier 8b stays rendered-report-based, re-pointed at the migrated flag; update its scope manifest in B2. Also **update the Tier 8 note** in the `temporal-e2e-validate` skill as part of B3 — Tier 17 flips its meaning ("dry-run never dispatches" becomes pre-Part-B history).--**Step-7 master-table row to add:** `Tier 17: DRY honors the backend | a --temporal --dry-run run dispatches act_llm_gen_* (+ extract/img-gen) to the worker and mocks INSIDE the activity — no real IO, no API keys needed, zero-token suppressed usage, cross-worker graph still assembles | PASS/FAIL | path | — `.--> ### ⛔ CHECKPOINT E — after Phase B3 — **Follow-up complete** — **REACHED 2026-06-10**->-> Run mode is now orthogonal to backend across all four cells. **Req 1 satisfied.** Foundation for the Temporal-validation follow-up is in place.->-> **Verified:** `temporal-e2e-validate` **Tier 17 GREEN and RED-proven** (all arms, incl. no-keys; see B3 checklist for the run log summary) · full `make agent-test` green · committed.->-> **Handoff:** the four `run_mode × backend` cells all behave: LIVE/direct (unchanged), LIVE/temporal (unchanged), DRY/direct (leaf mock inline, no storage), DRY/temporal (real dispatch, leaf mock inside the activity — Tier 17). The `--mock-inference` reportable-mock coexists on the LIVE row (Tier 8b unchanged at the CLI surface, flag carried by `CogtRunParams`). Branch `feature/Mock-activities` → PR against `feature/Dry-run-as-temporal-activity`. — **Next:** [`followup-temporal-validation-activity.md`](./followup-temporal-validation-activity.md) (HARD GATE — get the `temporalio` answer first).--## GSTACK REVIEW REPORT--| Review | Trigger | Why | Runs | Status | Findings |-|--------|---------|-----|------|--------|----------|-| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | — |-| Codex Review | `/codex review` | Independent 2nd opinion | 1 | issues→folded | 12 points: 5 cross-model agreements, 3 precision folds, 2 new adopted (DRY storage short-circuit, nb_items carrier), 1 verification note, 1 dismissed |-| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 1 | CLEAR (clean) | 16 issues, 0 critical gaps open |-| Design Review | `/plan-design-review` | UI/UX gaps | 0 | — | — |-| DX Review | `/plan-devex-review` | Developer experience gaps | 0 | — | — |--Decisions folded into the phases above (2026-06-10): **D1** full CogtRunParams scope confirmed · **D2** CogtRunParams nests inside PipeRunParams holding the ONLY run_mode copy (delegating property; 12-method protocol threading named) · **D3** mock_main consolidates into one shared helper, leaf stamp conditional on DRY · **D4** boot generator selection backend-keyed only · **D5** search + render_page_views get full leaf DRY branches · **D6** fidelity wrapper re-keys to DRY-or-mock · **D7** typed terminal DryRunMockBuildError · **D8** is_mock_inference KEPT as thin reportable-mock, migrating into CogtRunParams · **D10** DRY short-circuits storage (mock above the *_and_store step) · **D11** nb_items rides ObjectAssignment.--**CODEX:** outside voice confirmed the search/render/mock_main/carrier-consistency findings independently and surfaced the two storage/nb_items issues this review had missed; its `make agent-test` complaint was wrong (that target is the repo standard) and was dismissed.--**CROSS-MODEL:** agreement on 5 of 8 in-review findings; zero unresolved tensions — both new Codex findings were verified in code and adopted (10A, 11A).--**VERDICT:** ENG CLEARED — ready to implement (B1 entry point; Tier 17 GREEN+RED is the req-1 acceptance gate).--NO UNRESOLVED DECISIONSdiff --git a/wip/dry-run-refactor/followup-run-outcome-seam.md b/wip/dry-run-refactor/followup-run-outcome-seam.mddeleted file mode 100644index 6db3bd405..000000000--- a/wip/dry-run-refactor/followup-run-outcome-seam.md+++ /dev/null@@ -1,76 +0,0 @@-# Follow-up — Run-outcome seam (the validator consumes the run's outcome, not its raised exception)--> **Status: deferred — own branch, not yet started. OPTIONAL tightening, not a bug.** Implements the deferred half of **D-plan [`D-plan.md`](./D-plan.md) D6**. Unlike the leaf-mock and Temporal-activity follow-ups, this is **not** tied to a user requirement — it's a seam cleanup. Only pursue if the seam smell is judged worth the cross-backend churn.->-> **Design rationale:** D-plan **D6** (the three-altitude model: L1 run primitive / L2 batch policy / L3 presentation). **Depends on:** the consolidation ([`consolidation-as-built.md`](./consolidation-as-built.md)) has landed (`BundleValidator`, `PipeRun`). Backend symmetry means touching `WfPipeRun` too. **Branch off the same D-plan.**--## What and why--Today the run primitive **discards** the classification it already computes: `PipeRun.run` catches the failure (`pipe_run.py:43`), builds a structured `ErrorReport` for the FAILED-delivery webhook (`pipe_run.py:51-54`), then **re-raises the raw exception** (`pipe_run.py:99-100`). The Temporal run is symmetric — `WfPipeRun` recovers the same `ErrorReport` via `recover_error_report`.--Because the run re-raises raw, `BundleValidator._classify_pipe` wraps the run in a **second** `try/except (PipelexError, ValidationError, FactoryException)` and re-derives the classification by walking the `__cause__`/`__context__` chain (`_root_cause_is`, `bundle_validator.py:272-295`, `:335-353`). Two `try/except` around one execution, the second reverse-engineering what the first already had.--**Goal:** give the run an outcome-**returning** sibling so the validator consumes a typed outcome and does pure L2 policy with **zero** execution `try/except`. The strict `run()` contract stays untouched — `PipelexRunner.execute_pipeline` keeps raising `PipelineExecutionError` on failure (it just raises on the failed-outcome variant).--**Explicitly NOT this:** moving `SKIPPED` / `allowed_to_fail` / aggregation / the signature pre-pass into the run. Those are irreducible **batch policy** (L2) — moving them down recreates the god-object D1 rejected, and `SKIPPED` would corrupt the strict LIVE contract (a LIVE run with a missing cross-package dep *must* hard-fail). See D6.--## Bounded payoff — read before committing--- `ErrorReport` is **wrapper-wins** and **collapses the cause chain** (`base_exceptions.py:459-534`): a `PipeRunError` wrapping a `PipeNotFoundError` reports `error_type="PipeRunError"`, so the not-found identity is **lost** on the report. The validator's `SKIPPED` rule needs "`PipeNotFoundError` *anywhere* in the chain" — which only the **raw exception chain** carries. So the outcome must hand back the **raw exception** (not just the `ErrorReport`); consuming the report alone would *not* let the validator drop the `_root_cause_is` walk. This is the crux: the win is "one catch instead of two", **not** "no chain walk".-- Cross-backend surface: `PipeRunProtocol` + `PipeRun` + `WfPipeRun` all change. Touches the freshly-hardened signature path (`36914511` / `76b96992`) — re-run the signature e2e.--## Status at a glance--| Phase | Title | Status | Commit |-|---|---|---|---|-| | **⛔ HUMAN GATE — is this worth doing?** | | |-| R0 | Decide the outcome type + method shape (pre-flight) | ☐ not started | |-| R1 | `PipeRun.run_to_outcome` (direct) + `run()` becomes raise-on-failed-outcome | ☐ | |-| R2 | `WfPipeRun` parity (Temporal) | ☐ | |-| R3 | Migrate `BundleValidator._classify_pipe` to consume the outcome | ☐ | |-| | **⛔ CHECKPOINT — seam tightened, suite green** | | |--Status legend: ☐ not started · ◐ in progress · ☑ done.--## Pre-flight — needs human input--Resolve **before** R1. Record in the R0 Handoff.--- [ ] **(GATE) Is the cleanup worth the cross-backend churn?** This is a design-tradeoff, not a bug — the current chain-walk is correct and well-tested. Default if unanswered: **leave it** (do not start; the seam smell is documented in D6 and that may be enough).-- [ ] **(before R1) Outcome type shape.** A small result type — e.g. `PipeRunOutcome = PipeOutput | PipeRunFailure(raw_exc, error_report)`. Confirm it carries the **raw exception** (required for the `SKIPPED` chain-walk — see "Bounded payoff"). Default: carry both the raw exception and the `ErrorReport`.-- [ ] **(before R1) Method shape.** New sibling `run_to_outcome(...)` on `PipeRunProtocol` (recommended — keeps the strict `run()` contract for `execute_pipeline` untouched) vs change `run()` to return the union and make `execute_pipeline` raise on the failed variant (higher blast radius). Default: **new sibling**.--> ### ⛔ HUMAN GATE — before R0 — **STOP, confirm intent**->-> This follow-up is optional. Do not start R1 until the GATE pre-flight item is answered. If the answer is "leave it", close this tracker as *won't-do (documented in D6)* and stop.--## Phase R0 — Decide the outcome type + method shape--- [ ] Confirm the two pre-flight answers (outcome shape carries the raw exception; sibling method vs union return).-- [ ] Sketch the `PipeRunOutcome` / `PipeRunFailure` type next to `PipeRun` (a `pydantic dataclass` or small `BaseModel` — pick the lowest-churn correct form per the off-wire-shape rule; it stays in-process for validation, so no Temporal serialization constraint unless it ever crosses an activity boundary).--## Phase R1 — `PipeRun.run_to_outcome` (direct)--- [ ] *Tests first:* `run_to_outcome` returns the `PipeOutput` variant on success and the `PipeRunFailure` variant (carrying the raw exception + `ErrorReport`) on failure — **without raising**; delivery/tracer side effects still fire exactly as in `run()`.-- [ ] Factor the existing sole `except Exception` (`pipe_run.py:43`) so the failure path can **return** the outcome. Re-express `run()` as a thin wrapper: call `run_to_outcome`, raise the carried exception on the failed variant (preserves the strict contract `execute_pipeline` depends on).-- [ ] `make agent-check` + targeted `pipe_run` / runner suites green. No behavior change for `execute_pipeline`.--## Phase R2 — `WfPipeRun` parity (Temporal)--- [ ] Mirror the sibling on `WfPipeRun` (the Temporal run already builds the total `ErrorReport` via `recover_error_report` — surface it on the same outcome type). Keeps "a run is a run" backend-symmetric.-- [ ] Temporal integration tests green (the run path is exercised by the crate / e2e suites).--## Phase R3 — Migrate `BundleValidator._classify_pipe`--- [ ] *Tests first:* the existing `BundleValidator` classification tests (`tests/unit/pipelex/pipeline/test_bundle_validator.py` — SKIPPED via wrapped cause, widening, collect-all) stay green against the new consume-the-outcome shape.-- [ ] Replace the `try: await self._pipe_run.run(pipe_job) except (...)` with `outcome = await self._pipe_run.run_to_outcome(pipe_job)`; classify off the outcome. **Keep** `_root_cause_is` — now applied to `outcome.raw_exc` (still needed; `ErrorReport` collapses the chain). **Keep** L2 policy (allowed_to_fail, aggregation, signature pre-pass) exactly where it is.-- [ ] Keep the `prepare_pipe_job` call in its own narrow try if it can raise before the run (mock-input build) — that is a *job-build* failure, distinct from a *run* failure; classify both, but don't conflate them under one broad catch.-- [ ] Re-run the signature e2e (`tests/e2e/test_signature_validation_mthds.py`) + `tests/integration/pipelex/pipe_signature/*`.--> ### ⛔ CHECKPOINT — after R3 — **MANDATORY STOP**->-> The validator no longer wraps execution in its own `try/except`; the run owns the single catch and hands back a typed outcome. L2 policy unchanged.->-> **Verify:** `make agent-check` clean · `make agent-test` green · signature e2e green · commit.->-> **Handoff (fill in):** final `PipeRunOutcome` / `PipeRunFailure` shape; `run_to_outcome` signature on `PipeRunProtocol`; confirm `execute_pipeline` behavior identical.diff --git a/wip/dry-run-refactor/followup-temporal-validation-activity.md b/wip/dry-run-refactor/followup-temporal-validation-activity.mddeleted file mode 100644index 4191c605a..000000000--- a/wip/dry-run-refactor/followup-temporal-validation-activity.md+++ /dev/null@@ -1,70 +0,0 @@-# Follow-up — Distributed dry-run + validation as one in-process Temporal activity--> **Status: in progress — branch `feature/Dry-run-as-temporal-activity`.** Implements **D-plan Part C / D5 / req 2**, expanded with the **in-memory tracing** requirement surfaced 2026-06-09. The executable phase-by-phase plan with checkboxes and checkpoints lives in [`../../TODOS.md`](../../TODOS.md) (repo root) — this doc is the durable design reference (the *why* + the decisions).->-> **Design rationale:** [`D-plan.md`](./D-plan.md) §4.9 (distributed validation activity). **Risks:** D-plan §8 (nested-dispatch from the sweep, `temporalio` bump regression surface).->-> **Depends on:** the consolidation (`BundleValidator`, [`consolidation-as-built.md`](./consolidation-as-built.md)) — **shipped (#956)**. Part B ([`followup-leaf-run-mode-mock.md`](./followup-leaf-run-mode-mock.md)) is **no longer a hard prerequisite** — see "Dependency picture, corrected" below.--## What this delivers (req 2)--Production flow: the web app calls the Pipelex API to validate a bundle; when Temporal is enabled the API dispatches the whole job — **validation sweep + graph-producing dry-run** — to a worker as **one activity**; the worker runs it all **in-process, in memory**, and returns the per-pipe status map **and** the `GraphSpec`. No usage/cost reporting (dry runs are free). In direct mode the API keeps running it in-process unchanged (req 3).--## Dependency picture, corrected (2026-06-09)--The original brief said this work "depends on Part B (leaf-level run-mode mock)" and hinged on a new `scoped_content_generator` ContextVar. Both have been overtaken by what shipped since:--- **The sweep already stays in-process under a Temporal hub.** PR #976 wrapped `BundleValidator`'s per-pipe loop in `scoped_pipe_router(self._pipe_router)` (`bundle_validator.py`), so nested controller sub-pipes resolve the local in-process router instead of leaking to the Temporal router. The nested-dispatch risk D-plan §8 worried about — the load-bearing reason for `scoped_content_generator` — is **already handled** for the sweep.-- **DRY already mocks inline at the pipe level.** Pre-Part-B, each operator's dry path uses `ContentGeneratorDry()` inline, so a DRY sweep never dispatches `act_llm_gen_*`. That is exactly what we want *inside* a single activity (everything in-process). Part B is about the **opposite** cell (a top-level DRY run that *should* exercise the worker path and mock inside the activity) — it does not gate hosting the sweep in one activity.-- **The usage registry is gone.** Post-#967 (distributed cost reporting) usage rides on `PipeOutput`, not a process-global report registry. The sweep accumulates no per-run reporting state (`bundle_validator.py` step-4 comment), so overlapping sweeps can't collide on it. The "open one throwaway per-sweep registry" worry from earlier drafts is moot.--**Caveat — `scoped_content_generator` is deferred, not dead.** "Largely superseded" holds *only pre-Part-B*. Today a DRY sweep mocks at the **pipe** level (`ContentGeneratorDry` inline) and never touches `get_content_generator()`, so the hub's global generator is irrelevant and `scoped_pipe_router` alone keeps the sweep in-process. But under a Temporal-enabled hub `get_content_generator()` returns `ContentGeneratorInWorkflow` **globally** (set once at boot — `pipelex.py:370-385`, not contextual). So once **Part B** relocates the DRY mock *down to the leaf* (routing it through `get_content_generator()`), this in-process activity would reach `ContentGeneratorInWorkflow` and try to dispatch `act_llm_gen_*` — breaking the in-process guarantee. **Part B therefore reintroduces the need for the inline content-generator scope here.** This is the same `scoped_content_generator` seam Part C originally specced and the same one Shape C ([`followup-leaf-run-mode-mock.md`](./followup-leaf-run-mode-mock.md)) needs — whichever follow-up lands first should build it; the other consumes it. **Recommendation: add the inline content-generator scope to this activity now**, so it is correct regardless of Part-B ordering (robust over quick).--**Net:** the validation-sweep half is essentially ready to wrap. The real new work is the **graph-producing dry-run half**, specifically **in-memory graph tracing** (so it runs inside one activity without the heavyweight cross-worker DynamoDB round-trip) plus forcing the inline content generator so the leaf never dispatches.--## The graph step today vs. the new design--**Today (the thin-submitter design — see the #976 reviewer note).** `pipelex-api`'s `/validate` route does two things: `validate_bundle()` (the in-process sweep → status map) and, best-effort, `dry_run_pipeline()` → a `GraphSpec`. `dry_run_pipeline` goes through `PipelexRunner.execute_pipeline` → the hub's `get_pipe_run()`, which under a Temporal hub is `TemporalPipeRun` — so it dispatches **one top-level workflow** to the worker. The worker runs the pipeline, traces to the `temporal_dynamodb` event-log backend (`BufferingEventLog` in the workflow → `act_flush_trace_events` → DynamoDB), and `act_assemble_tracing` reads the events back and assembles the `GraphSpec`. The API itself runs with tracing **off** (thin submitter); the worker owns tracing + assembly. This works because the graph step is a *single* dispatch (no concurrent same-id collision — unlike the sweep's fan-out, which #976 fixed). **Do not move this step in-process in the API** — that breaks graph generation (tracing off in the API → empty graph).--**New (this follow-up).** Host the dry-run + validation in **one activity on the worker** that runs entirely in-process and traces the graph in an **in-memory** event log:--- No per-pipe child workflows / activities (the whole pipeline dry-runs in-process inside the one activity — like the sweep does).-- No `temporal_dynamodb` round-trip and no NDJSON files for the graph — the trace events live and die in memory inside the activity, just long enough to assemble the `GraphSpec`.-- No usage/cost events at all (dry run).--This is lighter than the current top-level-workflow + DynamoDB path and is the natural shape for "validate this bundle and show me its graph" — the webapp's actual call.--## The core technical problem — emit and assemble use *two* event-log instances--The blocker that makes "just add an in-memory backend" insufficient: the write side and the read side each construct their **own** event log from config, and only an **external store** bridges them.--- **Write side:** `pipeline_run_setup` (`pipeline_run_setup.py`) calls `make_event_log(tracing_config)` and hands it to the graph tracer, which emits `PipeStart`/`PipeEnd`/edge events into it during the run.-- **Read side:** after the run, `PipeRun.run`'s `finally` calls `assemble_tracing_on_output` → `assemble_tracing` (`tracing_assembly.py`), which calls `make_event_log(tracing_config)` **again** — a brand-new instance — and `read_events(pipeline_run_id)` to rebuild the `GraphSpec`.--For NDJSON the two instances connect through the file on disk; for DynamoDB through the table. For a plain `InMemoryEventLog` they would be **two empty lists** — the reader sees nothing.--`InMemoryEventLog` already exists (`pipelex/tracing/in_memory_event_log.py`, fully implements `EventLogProtocol`, used in unit tests). It is not the missing piece. **The missing piece is a way to share one instance across emit and assemble for the duration of a run.**--### The fix — a scoped event-log override (mirror the existing scope pattern)--Add a hub-level `_event_log_override: ContextVar[EventLogProtocol | None]` + a `scoped_event_log(event_log)` context manager, exactly mirroring `scoped_current_library` and `scoped_pipe_router` (`hub.py`). Make **both** `make_event_log` call sites (`pipeline_run_setup` write, `assemble_tracing` read) prefer the override when set, else fall back to `make_event_log(tracing_config)`. The activity wraps its dry-run in `with scoped_event_log(InMemoryEventLog()):` so emit and assemble hit the **same** instance — fully in memory, no file, no DynamoDB. ContextVar-scoped ⇒ coroutine-local ⇒ concurrent activities on one worker don't cross-contaminate (same property that makes `scoped_pipe_router` concurrency-safe).--**Decision (default): scope-only; no new `TracingBackend.IN_MEMORY` config enum.** A config-selectable in-memory backend is *insufficient on its own* (the two-instance problem) unless the factory caches a per-run singleton keyed by run id — uglier than the scope, and it would change the global default rather than being opt-in per call. The scope is the clean mechanism. (Revisit only if a use case wants in-memory tracing selected purely by config, with no caller scope.)--## The in-process graph dry-run must be safe under a Temporal hub--`dry_run_pipeline` → `PipelexRunner.execute_pipeline` → `get_pipe_run()` → `TemporalPipeRun` under a Temporal hub → dispatches a workflow. Inside an activity that is illegal, and it defeats the in-process goal. The runtime bridge already solved this exact "force in-process even inside a Temporal worker" problem: `runtime_bridge.bridge._run_direct` scopes a local `PipeRouter` via `scoped_pipe_router` + runs a local `PipeRun`, and `run_pipe_via_bridge` already accepts a `trace_context` that is honored in DIRECT mode. So the in-process graph dry-run = open a `GraphTracerManager` tracer against an `InMemoryEventLog` (under `scoped_event_log`), build the `trace_context`, run the main pipe through the **bridge DIRECT path** (not `PipelexRunner`/`get_pipe_run()`), and the `GraphSpec` rides back on `PipeOutput` — assembled from the same in-memory log. Net: GraphSpec produced, zero nested dispatch, zero files/DDB.--## The activity + dispatch--- **`act_dry_validate`** — a thin activity wrapping the validation sweep (`BundleValidator`) **and** the in-memory graph dry-run, wrapped in `scoped_event_log(InMemoryEventLog())` + `scoped_content_generator(inline)` and the existing `convert_pipelex_errors` boundary (`tprl/activity_error_boundary.py`) so failures cross back as structured `ErrorReport`s. The graph dry-run is **best-effort inside the activity** (a graph failure returns `graph=None` with validation still successful — today's `/validate` contract). Inputs (serializable): `mthds_contents` / `library_dirs` / `bundle_uris`, `allow_signatures`, optional `--pipe` selection. Output (serializable): `{pipe_ref: DryRunOutput}` map + `GraphSpec | None` + the aggregated signature-check error. Registered in `pipelex/temporal/tasks.py` (the `PackName.PIPE` lists).-- **Dispatch — DECIDED (2026-06-09): one-step wrapper workflow.** A one-step workflow runs the single activity and returns — works on the current `temporalio` (`1.23.0`), **no SDK bump, no Temporal Cloud/server verification, no hard gate**. Functionally identical to the caller. A true standalone activity (no workflow) is a **later optional optimization** only — deferred, off the critical path.-- **API wiring (cross-repo `pipelex-api`).** When Temporal enabled, `/validate` dispatches the wrapper workflow and awaits `{status map, GraphSpec}`; in direct mode it keeps the current in-process `validate_bundle` + `dry_run_pipeline`. This *replaces* today's "graph as a top-level workflow to the worker + DynamoDB" path with the single-activity in-memory path.--## ✅ Former HUMAN GATE — RESOLVED (2026-06-09)--The `temporalio` standalone-activity question is **resolved**: dispatch uses the **one-step wrapper workflow** on the current `temporalio`. No bump, no infra verification, nothing left to ask — the whole plan (Phases 1→4 in [`../../TODOS.md`](../../TODOS.md)) runs without stopping for human input. Switching to a true standalone activity later is a pure runtime optimization (Phase G0, deferred); only *that* would need the version + Cloud/server-support facts.--## Acceptance (Checkpoint F)--req 1 (distributed DRY testing, activity-level mocks — delivered by Part B, not here) · **req 2 (production dry-run + validation as one in-process Temporal activity, in-memory tracing) — this follow-up** · req 3 (direct in-process unchanged).diff --git a/wip/dry-run-refactor/handoff-skills-validate-namespaced-identity.md b/wip/dry-run-refactor/handoff-skills-validate-namespaced-identity.mddeleted file mode 100644index 557b5e978..000000000--- a/wip/dry-run-refactor/handoff-skills-validate-namespaced-identity.md+++ /dev/null@@ -1,96 +0,0 @@-# SWE-agent handoff — update MTHDS skills for the namespaced `validate` pipe identity--**For:** a SWE agent working in the **skills repo** (`../skills/skills/` relative to the pipelex project root — a separate checkout, not present in the pipelex worktree where this change landed).--**Why you're here:** the `pipelex` runtime changed the identity carried in the agent-CLI `validate` JSON envelope. Skills that consume that JSON may still assume the old form. This handoff tells you exactly what changed, what to hunt for, and how to verify. You can complete it without reading the pipelex diff, but pointers are included so you can confirm the contract yourself.--## The one-sentence change--`pipelex-agent validate pipe <code>` (and `validate pipe --bundle … <code>`) now report each pipe in `validated_pipes` by its **namespaced `pipe_ref`** (`domain.code`) instead of the **bare** `code`.--## What did NOT change (so you don't chase ghosts)--- **`validate all` and `validate bundle` are unchanged.** Those whole-set surfaces *already* emitted namespaced `domain.code` refs before this change. Any skill example or instruction tied to `validate all` / `validate bundle` was already correct — leave it.-- **The JSON shape is identical.** Still `{"success", "validated_pipes": [{"pipe_code", "status"}], "total_pipes"}`. The `pipe_code` **key name is unchanged** — only its **value** for the single-pipe surfaces changed from bare to namespaced.-- **`status` values are unchanged** (`SUCCESS` / `FAILURE` / `SKIPPED`).-- **Namespaced refs are still valid CLI inputs.** `get_required_pipe` resolves a `domain.code` ref directly, so any skill that feeds an emitted `pipe_code` back into a follow-up `validate pipe` / `run pipe` command keeps working.--So the entire blast radius is: **places that assume `validate pipe` echoes the bare code.**--## Concrete before/after--For `pipelex-agent validate pipe extract_invoice --format json`, where `extract_invoice` lives in domain `invoices`:--Before (what a skill may still assume):--```json-{- "success": true,- "validated_pipes": [{ "pipe_code": "extract_invoice", "status": "SUCCESS" }],- "total_pipes": 1-}-```--After (current behavior):--```json-{- "success": true,- "validated_pipes": [{ "pipe_code": "invoices.extract_invoice", "status": "SUCCESS" }],- "total_pipes": 1-}-```--## The highest-risk pattern to find--Skills are `SKILL.md` prose (plus optional `references/`) consumed by an LLM agent, so "breakage" is usually a **stale instruction or a stale example**, not compiled code. The single dangerous pattern is:--> **Exact-matching a bare code the skill itself passed in against the returned `pipe_code`.**--For example, an instruction like *"after `validate pipe <code>`, find the entry whose `pipe_code` equals `<code>` and read its `status`."* That equality now fails, because the skill passed a bare `<code>` but the entry reports `domain.<code>`. Rewrite such instructions to either:--- read the status from the **single** returned entry directly (these single-pipe surfaces return exactly one entry), or-- match by **suffix** (`pipe_code == code or pipe_code.endswith("." + code)`) when a bare code must still be reconciled.--Lower-risk, usually-fine cases (verify, but likely no change): displaying the code to the user (namespaced is clearer), keying an internal status map, or passing the code into a follow-up command.--## Where to look--The consuming skills and shared docs (paths relative to the skills repo root):--- Skill bodies most likely to read validate output: `mthds-check`, `mthds-fix` (they validate and report per-pipe status). Also scan `mthds-build`, `mthds-run`, `mthds-edit` for embedded validate examples.-- Shared reference docs: `shared/` — especially anything documenting the agent-CLI `validate` JSON shape (e.g. `mthds-agent-guide.md`).--Grep starting points:--```bash-grep -rn "validated_pipes" .-grep -rn "validate pipe" .-grep -rn "pipe_code" .-```--For each hit, ask: *does this assume the value is the bare code, specifically for a single-pipe `validate pipe` call?* If yes, update the prose and any embedded example JSON to the namespaced `domain.code` form. If it's a `validate all` / `validate bundle` context, leave it.--## How to verify--1. Run a single-pipe validate against any bundle and confirm the emitted `pipe_code` is namespaced:-- ```bash- pipelex-agent validate pipe <some_pipe> --format json- ```-- Expect `validated_pipes[].pipe_code` to be `domain.<some_pipe>`.--2. Walk each updated skill's instructions against that real output and confirm no step relies on a bare match.-3. If the skills repo has its own test/lint/CI, run it.--## Provenance (pipelex side — for confirming the contract)--Landed on branch `feature/Validate-with-signatures-4-fix-dry-run`.--- Source of truth: `pipelex/pipeline/validate_bundle.py::build_validated_pipes` → returns `ValidatedPipeEntry` whose `pipe_code` is `output.pipe_ref` (namespaced) on every surface.-- Contract doc: the `validate` output section in `pipelex/cli/agent_cli/CLAUDE.md`.-- Rationale + the consolidation it shipped with: `wip/dry-run-refactor/consolidation-as-built.md` ("Validate-result identity"). This handoff is the cross-repo tail.-- Changelog: the `[Unreleased] → Changed` entry naming the `validated_pipes` identity unification.--The decision was to **unify on `pipe_ref`** so the same pipe is never reported under two identifiers by different `validate` commands. This handoff exists because the skills repo could not be inspected or edited from the pipelex worktree where the change was made.diff --git a/wip/dry-run-refactor/plan-unified-dry-run.md b/wip/dry-run-refactor/plan-unified-dry-run.mddeleted file mode 100644index bf3b7e63b..000000000--- a/wip/dry-run-refactor/plan-unified-dry-run.md+++ /dev/null@@ -1,77 +0,0 @@-# Plan — Unified dry run (retire `is_mock_inference`; add internal `is_mock_usage` flag)--> **Status: COMPLETE (2026-06-10) — all phases done, `make agent-check` + full `make agent-test` green.** Implements [`design-unified-dry-run.md`](./design-unified-dry-run.md) (design agreed with the user). Baseline: `feature/Mock-activities` as-built (Part B complete + Part C validation activity merged in).--## Plan-time decisions (resolving design §5)--The design deferred these to plan time; resolved here, taking the design's proposed options:--- **`is_mock_usage=True` + `run_mode=LIVE` is a validation error** — a pydantic model-validator on `CogtRunParams` rejects it (fail loud, per the design's proposal). The CLI never produces the combination (the hidden flag requires `--dry-run`), so the validator only fires on a buggy embedder call.-- **E2E trigger = hidden CLI flag `--mock-usage`** (typer `hidden=True` on `run pipe` / `run bundle` / `run method`, requires `--dry-run`). Present for the `temporal-e2e-validate` Mode-2 scripts, absent from `--help` and docs.-- **`MockInferenceObjectFidelityError` → renamed `DryRunObjectFidelityError`** — mock-built objects are now DRY-only, the old name is a lie; no backward compat per project principles. Error pages regenerated.-- **Sentinel naming**: two sentinels stay — `dry_run` (zero-token) unchanged; `mock_inference` → `mock_usage`. No fold into a single sentinel: tests filter usage events by sentinel name, and two names keep the two reporting payloads grep-able.-- **Non-LLM leaves do NOT emit synthetic usage under `is_mock_usage=True`** (minimal design — matches what the cost-assembly tests assert).-- **`CogtRunParams.is_mock_built` property deleted** (not kept as alias): mock-built is now exactly `run_mode.is_dry`; keeping a second name for one predicate is the disease this refactor cures. Fidelity-guard call sites read `run_mode.is_dry`.-- **PipeCompose dry arm: untouched** (explicitly parked by the user).--## Phases--### Phase 1 — Carrier + factory + entry points--- [x] `CogtRunParams`: `is_mock_inference` → `is_mock_usage` with new semantics docstring; add model-validator rejecting `is_mock_usage=True` with `run_mode.is_live`; delete `is_mock_built` property. Frozen/extra-forbid/required-run_mode unchanged.-- [x] `PipeRunParamsFactory.make_run_params`: param re-keyed `is_mock_inference` → `is_mock_usage`; forced-DRY path simplified (single warning; no swallow arm — `is_mock_usage` rides DRY fine).-- [x] Entry-point threading re-keyed: `PipelexRunner.__init__` / `execute_pipeline`, `pipeline_run_setup`, `prepare_pipe_job` (`execution_seams.py`). Docstrings rewritten for the sub-flag semantics.-- [x] Fidelity guard call sites (`content_generator.py` ×3, `content_generator_in_workflow.py` ×3) re-keyed from `is_mock_built` to `run_mode.is_dry`.--### Phase 2 — Leaves--- [x] `llm_generate.py`: dual branch collapses to one `if run_mode.is_dry` per leaf; dry helpers select report func on `is_mock_usage`.-- [x] `dry_mock.py`: delete `mock_llm_gen_text` / `mock_llm_gen_object` / `mock_llm_gen_object_list` / `_mock_text` and the `MOCK INFERENCE` marker; rename `report_mock_inference_llm_job` → `report_mock_usage_llm_job`, `MOCK_INFERENCE_MODEL_NAME/ID` → `MOCK_USAGE_*` (= `"mock_usage"`), `MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY` → `MOCK_USAGE_NB_TOKENS_BY_CATEGORY`; dry helpers pick report func via a shared `_dry_report_func(cogt_run_params)`; `stamp_mock_main_coordination` docstring sheds the mock-inference caveat (the dry object-list leaf stamps regardless of `is_mock_usage`); module docstring rewritten for the one-mode world.-- [x] Delete the `MockInferenceUnsupportedError` guards: `img_gen_generate.py` (×2), `extract_generate.py`, `search_generate.py` (`_guard_no_mock_inference`).--### Phase 3 — Errors + config lists--- [x] Delete `MockInferenceUnsupportedError` from `cogt/content_generation/exceptions.py`; remove it from `non_retryable_error_types` in all three toml copies (`pipelex/pipelex.toml`, `pipelex/kit/configs/pipelex.toml`, `.pipelex/pipelex.toml`).-- [x] Rename `MockInferenceObjectFidelityError` → `DryRunObjectFidelityError`; message/docstring re-scoped to DRY-only.-- [x] Regenerate error pages (`make gep`): `mock-inference-unsupported-error.md` removed, fidelity page renamed.--### ⛔ CHECKPOINT 1 — core re-key compiles and unit tests pass — **CLEARED 2026-06-10** (targeted cogt/cli/pipe_run/temporal unit + cogt integration runs green)--### Phase 4 — CLI--- [x] Remove `--mock-inference` from `pipe_cmd.py` / `bundle_cmd.py` / `method_cmd.py`; add hidden `--mock-usage` (typer `hidden=True`).-- [x] `validate_run_flag_combination`: mutual-exclusion arm replaced by "`--mock-usage` requires `--dry-run`"; `--mock-inputs` rule unchanged.-- [x] `_run_core.py`: `mock_inference` params → `mock_usage`, threaded to `PipelexRunner(is_mock_usage=...)`.--### Phase 5 — Tests (TDD where behavior is new: validator + CLI flag rule written red-first)--- [x] `test_cogt_run_params_carrier.py`: re-key; add the LIVE+`is_mock_usage` rejection test (red first).-- [x] `test_run_flag_combination.py`: new `--mock-usage`-requires-`--dry-run` arm (red first); exclusion arm dropped.-- [x] Merge `test_llm_generate_mock_branch.py` into `test_llm_generate_dry_branch.py` parameterized over `is_mock_usage` (zero-token suppressed vs non-zero rendered payloads asserted); delete the mock-branch file.-- [x] Delete guard tests: `test_extract_generate_mock_guard.py`, `test_img_gen_generate_mock_guard.py`, `test_pipe_search_mock_inference_guard.py`, `test_mock_inference_cli_guard.py` (folded into flag-combination tests).-- [x] Re-key + rename: `test_mock_inference_temporal.py` → `test_mock_usage_temporal.py` (wire survival on the carrier, DRY + `is_mock_usage`), `test_mock_inference_direct.py` → `test_mock_usage_direct.py`, `test_mock_inference_object_fidelity.py` → `test_dry_run_object_fidelity.py`, `test_content_generator_in_workflow_object_fidelity.py` (error rename).-- [x] Fidelity tests (D6 arms): re-key from two triggers to DRY-only; `test_leaf_dry_object_mocks.py` stamp test now asserts stamping regardless of `is_mock_usage`.-- [x] Keyless-boot tests: drop the swallow-warning arm.-- [x] `test_dry_mock.py`, `test_traceback_flag_run_core.py`, tracing `test_data.py`/`helpers.py`: mechanical re-keys.--### ⛔ CHECKPOINT 2 — full suite green (`make agent-check` + `make agent-test`) — **CLEARED 2026-06-10** (0 lint/type errors, full suite passed)--### Phase 6 — Skill + docs + changelog--- [x] `temporal-e2e-validate` skill: Tier 8b arms re-pointed to `--dry-run --mock-usage` (still asserting non-zero aggregated values); Tier 17 prose drops the "distinct from `--mock-inference`" contrast and states the sub-flag; SKILL.md description + routing battery + scope manifests updated.-- [x] `docs/building-methods/pipes/run-modes-and-backends.md`: matrix collapses to one mode axis × two backends; mock inference disappears as a user-facing concept.-- [x] Purge `--mock-inference` from `docs/features/validation-dry-run.md`, `docs/under-the-hood/dry-run-mock-generation.md`, `docs/building-methods/pipes/executing-pipelines.md`; error pages already regenerated in Phase 3.-- [x] CHANGELOG `[Unreleased]`: breaking — `--mock-inference` removed; `CogtRunParams.is_mock_inference` → internal `is_mock_usage` DRY sub-flag.-- [x] wip docs: this plan's checkboxes updated; `wip/history/registry/deferred-followups.md` "fate of is_mock_inference" item closed; `README.md` updated.--### Phase 7 — Final verification--- [x] `make agent-check` + `make agent-test` green; commit.--## Post-review change (PR #986 round 2, human review)--Reviewer + user direction: `run_mode` belongs on `PipeRunParams` directly. Implemented as an inversion of the D2 carrier shape — `run_mode` and `is_mock_usage` are direct fields of `PipeRunParams` (single stored copy, model-validator enforces the DRY-only sub-flag rule); `cogt_run_params` became a **derived property** minting the frozen cogt-tier carrier on demand. The wire contract is unchanged (`CogtRunParams` still stamped on every assignment, still validates at its own boundary); operators' `pipe_run_params.cogt_run_params` reads work as before. The factory validates the REQUESTED mode before the keyless forced-DRY coercion so the contract violation cannot be masked (greptile round-1 finding, kept).--## Cross-repo note (design §3.7)--`pipelex-api` carries `is_mock_inference` on its public runner API params (fed from `_run_core` → runner). That change rides a pipelex version bump like any other public-surface change — grep `mock_inference` in `pipelex-api/` when bumping the pin there; NOT part of this branch.diff --git a/wip/error-handling/README.md b/wip/error-handling/README.mddeleted file mode 100644index ccb59def2..000000000--- a/wip/error-handling/README.md+++ /dev/null@@ -1,58 +0,0 @@-# Error Handling — Current State and Open Tracks--This directory is the current-state reference for error handling across Pipelex. It is intentionally **track-organized**, not phase-organized: each track is a self-contained concern with its own current state, gaps, and followups. Tracks are independent — there is no implied order between them.--Each `track-*.md` describes **what the code does today**. What is still open is summarized in [What's still open](#whats-still-open).--## Status at a glance--| Track | Status | Doc |-|---|---|---|-| Error metadata model | Landed for all inference workers — inference errors self-describe via `to_error_report()`, and every LLM, img-gen, extract, and search worker raise carries structured `ProviderErrorMetadata` (status_code, request_id, retry_after_seconds, provider_error_code, body) plus `UserAction(kind, detail)` with semantic `UserActionKind` values. `InferenceErrorCategory.UNKNOWN` distinguishes unrecognized fallback paths from genuine `CONTENT`. Each worker family fills `model` / `provider` onto the error at its public-method chokepoint via `CogtError.fill_model_and_provider()`, so a production inference failure's `ErrorReport` is attributable to a model and a backend. A long tail of non-inference `PipelexError` subclasses still depend on string-keyed dicts in `agent_output.py` — see [track-metadata-model.md](track-metadata-model.md) Followups. | [track-metadata-model.md](track-metadata-model.md) |-| Worker classification | Landed for all inference workers — every LLM, img-gen, extract, and search worker (plus AWS Bedrock LLM) maps SDK exceptions to a `CogtError` subclass with `error_category`, unwraps `InstructorRetryException` correctly on structured-gen paths, and carries the structured metadata + semantic user-action upgrades. Response-shape validation failures (non-SDK paths) carry `provider_metadata=None` plus a semantic `UserAction` for uniformity. | [track-worker-classification.md](track-worker-classification.md) |-| Extract / Classify / Render | Landed — every inference worker's `except` block now collapses to `extract → classify → raise render(...) from exc`. **Classify** (`pipelex/cogt/inference/error_classify.py`) and **Render** (`error_render.py`) are single shared functions; only the `extract_*_metadata` functions stay per-provider. A new `tests/unit/pipelex/cogt/inference/test_provider_classification_parity.py` walks every `ProviderName` against the extract-fn registry to catch unwired providers. | [track-extract-classify-render.md](track-extract-classify-render.md) |-| Retry & resilience | Landed — direct (non-Temporal) execution is a single pipeline-level attempt with no application-level retry loop; resilience is the Temporal track's job (activity `RetryPolicy` keyed off `InferenceErrorCategory`). `PipeBatch` bounds fan-out via `gather_bounded` (chunked, `max_concurrency`, default 8) — admission control, not retry. Tier 1 (transport retry) is an explicit, uniform policy: `cogt.transport_max_retries` is wired into every inference SDK client factory, the SDK-less `azure_rest` image-gen path has a `tenacity` transport-retry floor, and `instructor`'s structured-output retry is confined to schema re-ask. | [track-retry-and-resilience.md](track-retry-and-resilience.md) |-| CLI delivery | Landed — agent CLI emits markdown by default for `run`/`validate`/`init` (and `models`/`check-model`/`doctor`), `--format json` for the structured payload; the error path follows the same option via a per-invocation `ContextVar`. `ErrorReport` carries an authoritative `error_domain` → HTTP-status mapping. `display_error_panel` collapses the field-shaped Rich handlers onto one helper. | [track-cli-delivery.md](track-cli-delivery.md) |-| Temporal integration | Landed — `TemporalError.from_message_exception()` derives retryability from `InferenceErrorCategory.is_retryable` for category-carrying `CogtError`s (the `non_retryable_error_types` list is the fallback) and packs the full `ErrorReport` into `ApplicationError.details`. Every in-scope activity is decorated with `@convert_pipelex_errors`, so the bridge runs in production. The symmetric workflow → submitter and child-workflow → parent recovery is landed: `recover_error_report` reads the report back out of `ApplicationError.details`, so a pipe failing on a Temporal worker reaches the CLI and HTTP adapters with the same classification as a local run. A later pass added the **workflow-level fail-safe floor** (an escaping *inline* `PipelexError` — never dispatched through an activity — now fails the workflow terminally and classified instead of hanging on indefinite task-retry); landed on `fix/Temporal-failsafe`. Design rationale + open review backlog live in the three `temporal-failsafe-*.md` docs — see [What's still open](#whats-still-open) item 4. | [track-temporal-integration.md](track-temporal-integration.md) |-| Testing | Landed — worker-level classification tests, the `agent_output.py` dict-drift test, and the full-chain worker→runner→CLI integration test (JSON + markdown) plus Rich-panel snapshot tests are all in place. | [track-testing.md](track-testing.md) |-| API readiness (error-handling companion) | Landed — `PipelexError.title()` / `type_uri()`, `request_id` on `JobMetadata`, `DisclosureMode` with `to_dict(disclosure_mode=)` / `to_problem_document()`, total `recover_error_report`, `ErrorReport` as a `BaseModel` threaded to the webhook, and per-class error doc pages. Webhook signing is the last open item — see [What's still open](#whats-still-open). | [api-readiness.md](api-readiness.md) |--Architectural reference (layer model, class hierarchy, two reporting systems) lives in [architecture.md](architecture.md).--## What's still open--The bulk of the error-handling work has landed. What remains:--1. **Webhook signing** — the last unshipped piece of the API-readiness companion work. Cross-repo: the pipelex side and the API side land in lockstep. Authoritative plan: [../webhook-signing.md](../webhook-signing.md).-2. **Metadata-model long tail** — a handful of `CogtError` subclasses still carry no class-level `error_category`, and several non-inference `PipelexError` subclasses still depend on the `agent_output.py` fallback dicts for `hint` / `error_domain` rather than carrying class-level metadata. See the Followups in [track-metadata-model.md](track-metadata-model.md).-3. **`request_id` correlation on delivery error paths** — `request_id` is on the success-path completion logs but not on the failure-path messages of `StorageDeliveryError` / `WebhookDeliveryError`, nor on the `pipe_run.py:79` debug entry log. Small but real. See [track-delivery-error-path-request-id.md](track-delivery-error-path-request-id.md).-4. **Temporal fail-safe floor — review backlog (parked, gated on Temporal productionization).** The silent-hang hole is *closed* (workflow-level catch-alls + `PipelexError` worker floor, landed on `fix/Temporal-failsafe`), but the code review left follow-ups open: a `SecurityError`-contract decision (Finding 3), small cleanups (5, 8, 10), a product question (12), and two findings from a later `/review` pass — the `_carries_temporal_failure` predicate's third fragility mode (Finding 13) and a pre-existing `build_search_attributes` replay-determinism hazard (Finding 14). **Live backlog:** [temporal-failsafe-review-followups.md](temporal-failsafe-review-followups.md). Fix rationale (✅ resolved, read-only): [temporal-error-handling-failsafe-gap.md](temporal-error-handling-failsafe-gap.md) (the "why it was the real defect" brief; also tracks deferred hardening C/D/F — submitter deadline, CI dispatch-guard, stuck-workflow observability) and [temporal-failsafe-guard-hardening.md](temporal-failsafe-guard-hardening.md) (the guard/consolidation findings 1/6/7/9).--## Suggested read order--Tracks are independent for implementation purposes, but there is a natural onramp for someone reading the directory cold:--1. [architecture.md](architecture.md) — the layer model, class hierarchy, and `ErrorReport` shape that every track refers back to.-2. [track-metadata-model.md](track-metadata-model.md) — the data contract (`error_category`, `error_domain`, `user_action`) everything downstream consumes.-3. [track-worker-classification.md](track-worker-classification.md) — Layer 0 → 1, where errors originate and pick up their category.-4. [track-extract-classify-render.md](track-extract-classify-render.md) — landed refactor: decomposes the per-worker pipeline into one shared `extract → classify → render` chain.-5. [track-cli-delivery.md](track-cli-delivery.md) — Layer 4 → 5, where the classified errors get rendered for humans and agents.-6. [track-retry-and-resilience.md](track-retry-and-resilience.md) — builds on classification and metadata to drive retry decisions.-7. [track-temporal-integration.md](track-temporal-integration.md) — extends the same model across the activity → workflow boundary. For the fail-safe floor that backstops it, the deep dive is the three `temporal-failsafe-*.md` docs (start at [temporal-failsafe-review-followups.md](temporal-failsafe-review-followups.md) for live work; the other two are resolved rationale).-8. [track-testing.md](track-testing.md) — cross-cutting, comes last because it verifies the others.--This is reading order, not implementation order — any track can be picked up independently.--## Cross-cutting principles (what's already true)--These are honored across the codebase today and the tracks below build on them:--- **Single-rooted hierarchy.** Every custom exception inherits from `pipelex.base_exceptions.PipelexError`. `ToolError` (`pipelex/system/exceptions.py`) is folded under it.-- **`msg = "..."; raise XError(msg) from exc`.** Message before raise; `from exc` everywhere a re-raise happens.-- **No bare `except` and no broad `except Exception` in business logic.** Broad catches only at CLI entry points and async task root handlers.-- **Workers transform SDK errors into domain errors.** Every inference worker catches the SDK's typed exceptions, attaches an `InferenceErrorCategory`, and chains the original via `from exc`.-- **`ErrorReport` is the single serialization schema.** `pipelex/base_exceptions.py` defines it; `PipelexError.to_error_report()` is the entry point; CLI JSON, Rich panels, and Temporal details all draw from it. `to_error_report()` enriches from the `__cause__` chain, so a wrapper exception surfaces the inference metadata (`error_category`, `retryable`, `model`, `provider`) of the underlying `CogtError`.--## Conventions used in track docs--Each `track-*.md` opens with **what this track is** (the concern in one paragraph) and a **current state** section — what is true in the code today, with verified file paths. A track with unfinished work then has an **open gaps** / **what's left** section and concrete **followups**; a fully-landed track says so explicitly. Every doc closes with **related tracks** cross-references. The docs describe the current state, not the history of how it got there.diff --git a/wip/error-handling/api-readiness.md b/wip/error-handling/api-readiness.mddeleted file mode 100644index c36d09151..000000000--- a/wip/error-handling/api-readiness.md+++ /dev/null@@ -1,266 +0,0 @@-# API-companion revisions — what we are actually building vs. the original spec--This doc captures the **deviations** the pipelex side is taking from the original spec in `changes-for-api-early-draft.md`, and the **rationale** for each. Its primary audience is the `pipelex-api` agent: when you read `changes-for-api-early-draft.md` to plan how the API will consume the new pipelex primitives, read this first to know what surface to actually expect.--The execution plan that implemented these revisions is now retired.-------## Why this exists--The original spec in `changes-for-api-early-draft.md` is **structurally sound** — every item identifies a real downstream pain and a real upstream primitive to fix it. But on close reading, several items push the very kind of consumer-side duplication, ad-hoc fallback logic, and optional-`None` plumbing that the spec itself argues against. A few items also introduce surface area earlier than it can be used, or hedge with "curate a subset, do the rest later" which leaves the codebase in an inconsistent state.--The revisions below collapse and re-scope the original items (see the mapping table next), eliminate the consumer-side drift sources it introduced, remove the "caller hand-authors a fallback" duplication pattern, drop the speculative item as YAGNI, and stay strictly inside the repo's *no speculative future-proofing* rule (`CLAUDE.md`).--**For the API agent:** wherever the surface here disagrees with `changes-for-api-early-draft.md`, **this doc is authoritative**. The spec is kept for context but is no longer the contract.-------## Mapping: original items → revised items--| Original spec items | Revised item | Stage |-|---|---|---|-| 1 (`title`) + 2 (`type_uri`) | **Item A** — `PipelexError.title()` / `type_uri()` classmethods with auto-derive defaults | 1 |-| 3 (`request_id`) | **Item B** — `request_id` on `JobMetadata` | 1 |-| 4 (`to_strict_dict`) + 6 (`to_problem_document`) | **Item C** — `to_dict(disclosure_mode=)` + `to_problem_document(...)` | 2 |-| 5 (caller-recovers + caller-hand-authors fallback) — *part 1* | **Item D-1** — make `recover_error_report` total | 2 |-| 5 — *part 2* | **Item D-2** — thread `ErrorReport` to the webhook | 3 |-| 7 (per-class doc pages) | **Item E** | 4 |-| **8 (`query_pipeline_state`)** | **dropped** — speculative future work | — |-| 9 (full-payload webhook signature) | **Item F** | 5 |-------## Per-item revisions and rationale--### §A. Items 1+2 — `title` / `type_uri` ownership--**What the spec proposes.** Add `ClassVar[str | None]` attributes defaulting to `None`. When `None`, "downstream consumers fall back to humanizing the class name" / "auto-derive `<base>/<kebab-case classname>` from a pipelex-config-provided base URI."--**Problem.** This is exactly the duplication the spec's own *Why* section argues against ("Curating titles API-side creates drift"). If pipelex doesn't own the auto-derive, every consumer (the API, the CLI, the agent CLI, future SDKs) re-implements humanize-from-classname and kebab-case. The drift moves from "curated copy" to "humanize implementation" — it doesn't go away.--**What we are building instead.**--```python-class PipelexError(Exception):- _declared_title: ClassVar[str | None] = None- _declared_type_uri: ClassVar[str | None] = None-- @classmethod- def title(cls) -> str:- # read only the class's OWN body — a curated title is never inherited- declared = cls.__dict__.get("_declared_title")- return declared if isinstance(declared, str) else _humanize(cls.__name__)-- @classmethod- def type_uri(cls) -> str:- declared = cls.__dict__.get("_declared_type_uri")- if isinstance(declared, str):- return declared- return f"{_base_error_uri()}/{_kebab(cls.__name__)}/"-```--Subclasses override `_declared_title` / `_declared_type_uri` only when the auto-derive is bad copy or the URI needs to point elsewhere — and the override is read from the class's own body, so a subclass never silently inherits an ancestor's curated value (see [What landed in Stage 1](#what-landed-in-stage-1)). **Every class works out of the box.** `ErrorReport` carries both as populated string fields — consumers never see `None`.--**API consumer impact.** The API reads `report.title` and `report.type_uri` directly. No humanize helper, no kebab-case helper, no `_resolve_title_for_class_name(...)`. The `https://docs.pipelex.com/latest/errors` namespace is owned by pipelex via the `URLs.error_docs_base` constant (`pipelex/urls.py`).-------### §B. Item 3 — `request_id`--**Decision taken in planning.** Field lives on `JobMetadata`, not `PipeJob`. `JobMetadata` already carries `pipeline_run_id`, `user_id`, `session_id` and threads through every activity / workflow / submitter hop via `PipeJob.job_metadata` → `PipeRunArg`. Adding it on `PipeJob` (as the spec's literal example shows) would require copying it across into `JobMetadata` anyway to make it visible to the logger context.--**Acceptance.** The API populates `JobMetadata.request_id` from its inbound `X-Request-ID` middleware once at submitter dispatch. From there `request_id` rides on `JobMetadata` across every Temporal hop and is threaded **explicitly** into the worker-side log calls — **not** via a `ContextVar`. A `ContextVar` is process-local and does not survive the Temporal activity/workflow serialization boundary, so it cannot carry a correlation id from the submitter process to a worker running in another process. The worker-side wiring is landed: `WfPipeRun` / `WfPipeRouter` build a per-invocation `WorkflowLog` bound to `job_metadata.request_id` at entry, so every workflow log record they emit carries `request_id` in its `extra` dict. The current `webhook.payload["request_id"]` piggyback becomes obsolete.-------### §C. Items 4+6 — disclosure modes and rendering--**What the spec proposes.** Split into two methods: `ErrorReport.to_strict_dict()` at Stage 2, then `ErrorReport.to_problem_document(disclosure_mode=...)` at Stage 4. The redaction rule for strict mode drops `user_action`, `provider`, `model`, `provider_metadata` *and* replaces `message` with a fixed string.--**Problems.**--1. **The `DisclosureMode` enum is introduced at Stage 2 but its only real consumer is at Stage 4.** Either merge the work or parameterize `to_dict` directly. The spec itself hedges with "Add a method (or parameterize `to_dict`)" — the parameterized version is the right call.-2. **Stage placement of `to_problem_document` is wrong.** The spec defers it to Stage 4 with the reasoning "the API's Phase 1 builder needs ~30 lines anyway." But those 30 lines are about FastAPI Response wiring (headers, content type, status code) — the **dict shape** is the reusable part, and if pipelex doesn't own it, the API builds its own envelope in Phase 1 and creates the very drift item 6 is meant to prevent.-3. **The redaction rule strips `title`.** RFC 7807 §3.1.4: `title` is "the same for occurrences of the problem" — a *stable type identifier*, not sensitive data. Same for `type_uri`, `error_type`, `error_domain`, `error_category`, `retryable`. Redacting these breaks the very spec the API is trying to comply with. The redaction should target `message` (can contain paths, prompts, user data) and implementation details (`provider`, `model`, `provider_metadata`, `user_action.detail` which can contain operator-specific advice).--**What we are building instead.**--```python-class DisclosureMode(StrEnum):- VERBOSE = "verbose"- STRICT = "strict"--class ErrorReport:- def to_dict(self, disclosure_mode: DisclosureMode = DisclosureMode.VERBOSE) -> dict[str, Any]: ...- def to_problem_document(- self,- *,- instance: str | None = None,- request_id: str | None = None,- disclosure_mode: DisclosureMode = DisclosureMode.VERBOSE,- ) -> dict[str, Any]: ...-```--**Strict redaction rule:**-- `provider`, `model`, `provider_metadata` → always dropped, whatever the `error_domain`.-- A report flagged `caller_facing_message` (set by error classes that author caller-facing copy — `PipelexInterpreterError`, `ValidateBundleError`) → keeps its `message` and `user_action`.-- Every other report → `message` replaced with `"An internal error occurred."`, `user_action` dropped.-- **Kept in all modes** (stable identifiers): `error_type`, `error_domain`, `error_category`, `retryable`, `title`, `type_uri`.--> **Why provenance, not inherited classification.** STRICT keys its `message` passthrough on the per-class `caller_facing_message` flag, *not* on `error_domain == INPUT`. `error_domain` is inherited up the `__cause__` chain, so keying on it would let a domain-less wrapper raised `from` an `INPUT` cause leak its own (non-caller-facing) `message`, and `to_problem_document(STRICT)` echo provider metadata for `INPUT` reports. Keying on message *provenance* avoids both; provider metadata is stripped unconditionally.--Both methods land in Stage 2 (was 2+4 in the spec). `to_problem_document` is callable from the API's Phase 1 directly.--**API consumer impact.** The Phase 1 builder is now ~10 lines (FastAPI Response wiring only). The envelope shape — `type`, `title`, `status`, `detail`, `instance`, extension members — comes from `report.to_problem_document(...)`. Strict mode is one kwarg. The "what does strict mode hide" rule lives in one place.-------### §D. Item 5 — async error pipe, totality, and the Temporal data converter--This is the load-bearing item. Three sub-issues; the revisions address all of them.--**Numbering note.** The `§D.x` subsections below are *sub-issues*, not revised items. §D.1 is **Item D-1** (totality, Stage 2); §D.2 and §D.3 together are **Item D-2** (`ErrorReport` threaded to the webhook as a typed field, Stage 3). There is no "Item D-3".--#### §D.1 — "Caller hand-authors a fallback report" is duplication--**What the spec proposes.** `recover_error_report(exc)` returns `Optional[ErrorReport]`. When it returns `None` (malformed details or version skew), the caller — the workflow boundary that catches the failure — hand-authors an `ErrorReport(error_type="WorkflowFailureUnrecoverable", error_domain="runtime", retryable=False, message="...")`.--**Problem.** Every other caller that uses `recover_error_report` (the API's submitter-side recovery, the CLI on async result consumption, anyone else who lands later) duplicates this hand-author logic. The `error_type` string "WorkflowFailureUnrecoverable" doesn't correspond to a real `PipelexError` subclass, so it can't be navigated to a doc page (item 7), can't carry an auto-derived `type_uri`, and can't be matched on by pattern. The fallback is a stringly-typed escape hatch.--**What we are building instead.** A new pipelex error class — `UnrecoverableWorkflowFailureError(TemporalFlowError)`, a `PipelexError` subclass — and `recover_error_report` becomes **total**: signature `def recover_error_report(exc: BaseException) -> ErrorReport`. When no embedded report is found, it synthesizes the report via `UnrecoverableWorkflowFailureError(message_from_exc(exc)).to_error_report()`. One construction site, one error class, one doc page, navigable by pattern. A report dict that *is* found but fails `from_dict` validation is an internal contract bug — within one deploy the activity bridge and the submitter share the schema — and is also synthesized into the same `UnrecoverableWorkflowFailureError` fallback (with an `[error report failed schema validation]` marker on the message) so the failure webhook still fires before the workflow surfaces the contract bug. The Temporal integration ships fresh, so there is no prior on-wire schema to stay compatible with.--All three call sites in `workflow_caller.py` (`execute_workflow` / `execute_child_workflow` / `start_child_workflow`) drop their `if error_report is not None` branches.--**API consumer impact.** When the API recovers from a Temporal workflow failure on the submitter side, it gets a usable `ErrorReport` unconditionally. No `None` to handle, no fallback to author.--#### §D.2 — `error_report_dict: dict[str, Any]` is a Temporal-shim smell--**What the spec implies.** The `DeliveryActivityArg` would carry the error as a dict because "Temporal serialization."--**Problem.** The other fields on `DeliveryActivityArg` (`pipe_output: PipeOutput | None`, `delivery_assignment: DeliveryAssignment`) don't carry `_dict` suffixes because the pipelex Temporal data converter handles them as Pydantic `BaseModel`s. Adding a `_dict` suffix telegraphs an implementation detail and is inconsistent with the rest of the file. `ErrorReport` should cross the activity boundary as a typed field, like every other field on the arg.--**What we built.** `ErrorReport` was converted from a frozen Pydantic `@dataclass` to a `BaseModel` with `model_config = ConfigDict(frozen=True, extra="forbid", arbitrary_types_allowed=True)`. The Temporal data converter then handles it with **no converter surgery**: the already-shipped `BaseModelPayloadConverter._unwrap_optional_base_model` resolves the `ErrorReport | None` field directly. `DeliveryActivityArg.error_report: ErrorReport | None` is a plain typed field — no `_dict` suffix, no shim. A `tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py` case covers the round-trip before `ErrorReport` is wired into the activity arg.--**Why this, not extending the converter.** The alternative considered was teaching the data converter (and `kajson`) to round-trip Pydantic dataclasses generically. The `BaseModel` conversion turned out strictly cleaner: it reuses the converter's existing `BaseModel` path instead of adding a new one, it is a smaller diff, and `extra="forbid"` gives `from_dict` the strict-schema validation the §D.1 recovery path depends on. The dataclass-vs-`BaseModel` ergonomics difference is negligible for a frozen value type.--#### §D.3 — Webhook payload field shape--`payload["error"] = error_report.to_dict()` — verbose disclosure by default. The API chooses strict mode at *its own* surface (HTTP response), not at the pipelex delivery boundary; redacting the payload before the receiver has decided what to do with it would be presumptuous. The webhook target URL is caller-supplied, so a VERBOSE `error` dict reaches an endpoint pipelex does not control — but that endpoint belongs to the run's own caller, who already owns the run and its inputs. VERBOSE is therefore the receiver's data to redact, not pipelex's — accepted by design. Webhook signing (Item F / Stage 5) is orthogonal: it authenticates the webhook's origin so the receiver can trust it, but does not change what the payload discloses.--**API consumer impact.** Receiving the webhook, the API sees a full `error` dict with the same shape as `ErrorReport.to_dict()`. It can call `ErrorReport.from_dict(payload["error"])` to rehydrate, or pass it through, or redact server-side based on `ERROR_DISCLOSURE`.-------### §E. Item 7 — per-class doc pages--No structural change from the spec. The improvement is that `cls.title()` and `cls.type_uri()` from item A are total (always populated), so the generator never has to fall back to placeholder content. Every page has a real title and a stable URL anchor.--**One material deviation, settled at landing time.** The original plan had `base_uri` defaulting to `https://pipelex.dev/errors` — a domain that does not actually serve these pages. Retargeted at landing to `https://docs.pipelex.com/latest/errors`, the live MkDocs + mike docs site. The `/latest/` alias is mike's canonical pointer at current stable, forced via `docs/overrides/main.html` — a `type` URI emitted by any deployed version always lands on up-to-date docs. `PipelexError.type_uri()` now appends a trailing slash (`<base>/<slug>/`) to match the canonical form MkDocs serves with `use_directory_urls: true` — clients dereferencing the URI hit the page directly without a 301. API consumers should treat `type_uri` as opaque and not strip or normalize the trailing slash.-------### Dropped — Item 8 — `query_pipeline_state(...)`--This item is **dropped** from the plan. It is not assigned a revised-item letter — the `§A`–`§F` sections cover `Item A`–`Item F` (`Item D` itself splits into `D-1` / `D-2`; see the [mapping table](#mapping-original-items--revised-items)).--**What the spec proposes.** A pipelex-level async function `query_pipeline_state(pipeline_run_id) -> PipelineState` that reads Temporal workflow history and returns a typed state for a future `GET /api/v1/pipeline/{run_id}` endpoint.--**Problem.** The spec itself flags this as future-facing: *"Not blocking the current API work... The status-query is a different use case (polling, not push) that no current consumer needs yet."* That is the textbook definition of speculative work. Repo `CLAUDE.md` is explicit: *"Don't add features... beyond what the task requires. Don't design for hypothetical future requirements."*--The risk of designing it now without a concrete consumer is that the typed state shape (`PipelineState` fields, naming, semantics for cancelled / timed-out workflows, how to surface partial progress) gets locked in by the API agent's assumptions rather than by a real client's needs.--**What we are doing instead.** Drop from this pass. When the first concrete polling consumer materializes — with a real spec for what fields it needs — the work picks up with clear requirements. The `recover_error_report` primitive (now total per item D-1) is already in place for that future use; only the workflow-history reading and the typed state shape are deferred.--**API consumer impact.** The webhook path is the supported async pattern. A future polling endpoint requires fresh design once a client asks for it.-------### §F. Item 9 — webhook signature--No structural change from the spec. One implementation note: the per-deployment signing secret lives in pipelex config (`get_config().pipeline_execution_config.webhook_signing_secret`) with an environment-variable fallback for ops convenience. The cross-repo PRs land in coordinated lockstep.-------## Cross-cutting themes--These show up in multiple items above; collecting them here so the API agent can recognize the patterns when reviewing pipelex PRs.--- **Pipelex owns the defaults.** If a primitive has an obvious sensible default (humanize a class name, kebab-case a class name, "an internal error occurred" placeholder), the default lives in pipelex and is returned as a populated value, not as `None` that consumers fill in. The only `None`s consumers see are for genuinely-absent data (no provider involved, no `user_action` declared).-- **Total functions over partial ones.** `recover_error_report` returns an `ErrorReport`, not `ErrorReport | None`. `title()` returns a `str`, not `str | None`. Each consumer is one branch simpler.-- **No backward-compat shims.** New optional kwargs default to the natural empty value because *that is the natural empty value*, not to preserve old callers. The project explicitly does not have a deprecation transition period.-- **No speculative surface.** We don't introduce surface that will be used "later." If item 6 is genuinely needed in Stage 2, it lands in Stage 2. If item 8 isn't needed by any current consumer, it doesn't land.-------## Current state--> **For the API agent:** check this section first to know what's actually available.--- [x] **Stage 1 — Foundations.** Items A + B.-- [x] **Stage 2 — Rendering primitives + total recovery.** Items C + D-1.-- [x] **Stage 3 — Async error pipe.** Item D-2 (the unblock for API Phase 4).-- [x] **Stage 4 — DX polish.** Item E.-- [ ] **Stage 5 — Security tightening.** Item F (cross-repo, tracked at [`../webhook-signing.md`](../webhook-signing.md)).--**Net to the API team:** the error-handling refactor on the pipelex side is landed. API Phases 0/1/4/5 are unblocked. One thing remains on the pipelex side:--- **Webhook signing (Stage 5 / Item F)** — the cross-repo security track at [`../webhook-signing.md`](../webhook-signing.md), independent of the rest of this plan, landing on its own schedule.--The post-review follow-ups have all landed: the STRICT-disclosure INPUT-domain leak, the `request_id` log wiring (§B), the webhook reserved-key collision, and the test-coverage backfill.--### What landed in Stage 1--Available now:--- **`PipelexError.title()` / `PipelexError.type_uri()`** (`pipelex/base_exceptions.py`) — classmethods returning a populated `str`. Auto-derive from the class name unless a subclass declares `_declared_title` / `_declared_type_uri` directly in its own body (inheritance is intentionally bypassed). Curated `_declared_title` overrides shipped for high-traffic base classes.-- **`ErrorReport.title` and `ErrorReport.type_uri`** — both required `str`. Every `to_error_report()` call populates them. Round-trips through `to_dict` / `from_dict`. `from_dict` raises `ValidationError` on missing fields (the path that Stage 2 Item D-1's `recover_error_report` synthesizes into `UnrecoverableWorkflowFailureError`).-- **`URLs.error_docs_base`** (`pipelex/urls.py`) — the base URI for every error `type_uri`, a hardcoded constant (`"https://docs.pipelex.com/latest/errors"`). `type_uri()` is a pure function: it reads only this constant, so it is safe to call before Pipelex boot and inside Temporal workflow code. A fork that needs a different host patches the constant or declares a per-class `_declared_type_uri`.-- **`pascal_case_to_kebab`** (`pipelex/tools/misc/string_utils.py`) — acronym-aware kebab conversion (`"APIError" -> "api-error"`, `"V2APIError" -> "v2-api-error"`, `"OAuth2" -> "o-auth2"`).-- **`JobMetadata.request_id: str | None`** (`pipelex/pipeline/job_metadata.py`) — round-trips through `model_dump_json`, distinct from `ProviderErrorMetadata.request_id` (the provider-side request id) which keeps its name.-- **`pipeline_run_setup(..., request_id="...")`** (`pipelex/pipeline/pipeline_run_setup.py`) — kwarg threaded into the constructed `JobMetadata`. The current `webhook.payload["request_id"]` piggyback on the API side is now obsolete; consumers should set `request_id` at dispatch time and read it back off `arg.<path>.job_metadata.request_id`.-- **`WorkflowLog` / `ActivityLog`** (`pipelex/temporal/log_temporal.py`) — every level (`verbose` / `debug` / `dev` / `info` / `warning` / `error` / `critical`) attaches the helper's bound `request_id` to the Temporal log record via `extra={"request_id": ...}`. A per-invocation bound helper — `WfPipeRun` / `WfPipeRouter` build a `WorkflowLog` from `job_metadata.request_id` at entry — means worker log records carry the `request_id` end to end.--**Update — `type_uri()` is now a pure constant.** Stage 1 originally read the base URI through an `ErrorManager` singleton holding an `ErrorsConfig` — a workaround for the import cycle a lazy `get_config()` call inside the method would have created. That whole machinery (`ErrorManager`, `ErrorsConfig`, the `[errors_config]` config block) was later removed: the `type` URI is a stable identifier per RFC 7807, so making it configurable was over-engineering, and the mutable read leaked into Temporal workflow determinism (the base URI rode into workflow history via a synthesized `UnrecoverableWorkflowFailureError`). `type_uri()` now reads the `URLs.error_docs_base` constant directly — a pure function, no boot dependency, no static cycle. API consumers see no difference.--**What `ErrorReport` looks like now.** Still a `@pydantic.dataclasses.dataclass(frozen=True, ...)` at the end of Stage 1; the conversion to `BaseModel` lands in Stage 3 Item D-2 as a prereq for crossing the Temporal activity boundary as a typed field. The Stage 2 work (`to_dict(disclosure_mode=...)` / `to_problem_document(...)`) lands on the dataclass form.--### What landed in Stage 2--Available now:--- **`DisclosureMode` enum** (`pipelex/base_exceptions.py`) — `VERBOSE` / `STRICT`. The contract is pinned on the enum's own docstring: STRICT is a *classification-projection*, **not a path-leak shield**. STRICT always drops `provider` / `model` / `provider_metadata`, then redacts `message` by *provenance*: a report flagged `caller_facing_message` (set by error classes that author caller-facing copy — `PipelexInterpreterError` / `ValidateBundleError`) keeps its `message` and `user_action`; every other report drops `user_action` and replaces `message` with `"An internal error occurred."`, keeping the stable identifiers (`error_type` / `error_domain` / `error_category` / `retryable` / `title` / `type_uri`). The passthrough keys on the per-class flag, not `error_domain` (see the gap note above).-- **`ErrorReport.to_dict(disclosure_mode=DisclosureMode.VERBOSE)`** — projects the report through the disclosure mode. `VERBOSE` round-trips through `from_dict` exactly; `STRICT` is lossy by design. Default is `VERBOSE` so existing internal-trust callers (webhook payloads, Temporal details) are unaffected.-- **`ErrorReport.to_problem_document(*, instance=None, request_id=None, disclosure_mode=VERBOSE)`** — returns a plain dict in RFC 7807 shape. `type` ← `type_uri`, `title` ← `title`, `status` ← `http_status`, `detail` ← `message` (subject to disclosure-mode redaction). The pipelex-native classification fields ride as extension members; `type_uri` and `title` are mapped — NOT echoed — so the returned dict contains exactly one `title` key. `request_id` and `instance` ride as extensions only when the caller supplies them. The runtime stays HTTP-agnostic — no FastAPI/Starlette import.-- **`recover_error_report(exc: BaseException) -> ErrorReport`** (`pipelex/temporal/tprl/temporal_error.py`) — total. When the failure carries no embedded report, the function synthesizes `UnrecoverableWorkflowFailureError(_message_from_exc(exc)).to_error_report()`. A report dict that *is* found but fails `from_dict` validation is an internal contract bug — the activity bridge and the submitter share the schema within one deploy — and is also synthesized into the same `UnrecoverableWorkflowFailureError` fallback, with the recovered message plus an `[error report failed schema validation]` marker, so failure-webhook delivery stays intact (the workflow still fails afterwards, keeping the contract bug visible). `UnrecoverableWorkflowFailureError` lives in `pipelex/temporal/exceptions.py` next to `WorkflowExecutionError` etc., with `error_domain=RUNTIME` and `_declared_title="Unrecoverable workflow failure"`. Callers in `workflow_caller.py` (the three former `if error_report is not None` branches at `execute_workflow` / `execute_child_workflow` / `start_child_workflow`) now treat the return as always-usable.--**One deviation from the spec's `message_from_exc`.** The plan specified "outer Temporal failure's message if non-empty, else `repr(exc)`". Implemented instead as a `__cause__`-chain walk that keeps the deepest non-empty message — because `WorkflowFailureError` always carries the generic outer text `"Workflow execution failed"`, which would defeat the round-3 P2 pin (the test asserts the synthesized message "contains the underlying exception text"). The walk surfaces the worker-side cause text (`RuntimeError("worker crashed hard")` → message contains `"worker crashed hard"`) while still falling back to `repr(exc)` when every node in the chain has an empty message. Same observable contract for callers; strictly more informative messages.--**User-visible behavior change at the unrecoverable path.** After Item D-1, `WorkflowExecutionError.message` on the no-report path changes from the Pipelex-framed `"Failed to execute workflow X"` / `"Application error in child workflow X"` to the synthesized `_message_from_exc(exc)` text. This is intentional — strictly more diagnostic — and is pinned by the rewritten `test_workflow_failure_without_report_synthesizes_unrecoverable` test. The legacy framing strings are gone on the `WorkflowFailureError` and `isinstance(exc.cause, ApplicationError)` paths. They remain on the `WorkflowAlreadyStartedError` / `RPCError` / non-`ApplicationError`-cause paths, because those branches do not call `recover_error_report`.--### What landed in Stage 3--Available now:--- **`ErrorReport` is a `BaseModel`** (`pipelex/base_exceptions.py`) — `model_config = ConfigDict(frozen=True, extra="forbid", arbitrary_types_allowed=True)`. `to_dict` uses `self.model_dump(exclude_none=True)`; `from_dict` uses `cls.model_validate(data)`. The frozen contract now raises `pydantic.ValidationError` (not `dataclasses.FrozenInstanceError`) on mutation attempts — caller-visible if anyone caught the specific dataclass error. Tests updated to match.-- **`DeliveryExecutor.execute(error_report=...)`** (`pipelex/pipe_run/delivery_executor.py`) — accepts `ErrorReport | None`; when non-None, the webhook payload includes `error = report.to_dict(DisclosureMode.VERBOSE)`. Receivers (the API) rehydrate losslessly via `ErrorReport.from_dict(payload["error"])`. The webhook stays the only mechanism for surfacing the report — the storage path does NOT serialize it as a file. The disclosure choice (VERBOSE on the wire) is locked here; the API decides what to re-expose to its own clients (it can render strict via `to_problem_document(disclosure_mode=STRICT)`).-- **`DeliveryActivityArg.error_report: ErrorReport | None`** (`pipelex/temporal/tprl_pipe/act_deliver.py`) — typed BaseModel field. No `_dict` shim. The existing `BaseModelPayloadConverter._unwrap_optional_base_model` (already shipped in Stage 1) handles the `ErrorReport | None` field directly with no converter surgery.-- **`wf_pipe_run.py` recovers and threads the report.** On the `ChildWorkflowError` catch, calls `recover_error_report(exc.cause if exc.cause is not None else exc)` (total since Item D-1), uses it to construct `WorkflowExecutionError("WfPipeRouter failed", error_report=report)` AND threads it into `DeliveryActivityArg(error_report=report)`. The outer Temporal wrap therefore does NOT drop the inner classification — the submitter-side `WorkflowExecutionError.to_error_report()` carries the same `error_type` / `error_category` / `retryable` / `model` / `provider` / `user_action` / `title` / `type_uri` the worker saw. Pinned by the new end-to-end full-chain test.-- **Direct-mode `PipeRun` (`pipelex/pipe_run/pipe_run.py`) threads the report too.** Captures `error_report = exc.to_error_report()` when the caught exception is a `PipelexError`, passes it to `DeliveryExecutor.execute`. The local / Temporal parity now holds at the delivery boundary — a sync-mode failure surfaces the same `error` dict on the webhook as a Temporal-mode failure. (This was an additive deviation from the plan's strict reading: the plan called out only the Temporal path, but the D-2 acceptance criterion implies parity, so the direct path was updated too. Non-`PipelexError` exceptions still surface no report, matching existing in-process semantics.)-**One additive deviation.** The plan localized the threading to the Temporal path. The direct-mode `PipeRun` was also updated for parity (see above). Same observable contract for API consumers in both modes.--**Pre-flight that the activity arg change relies on.** `tests/integration/pipelex/temporal/test_payload_codec_roundtrip.py::TestPayloadCodecRoundTrip::test_error_report_round_trips_through_activity` lands as a new test case that round-trips an `ErrorReport` (with populated `UserAction` + `ProviderErrorMetadata`) through a workflow → activity → return hop, verifying the BaseModel converter handles it directly. The plan's ordering (round-trip test before activity arg change) is preserved within the same commit set.--### What landed in Stage 4--Available now:--- **Error `type_uri` base is `https://docs.pipelex.com/latest/errors`** (`URLs.error_docs_base` in `pipelex/urls.py`) — the `/latest/` alias is mike's canonical pointer at current stable; canonical URLs are forced to `/latest/` via `docs/overrides/main.html`. A fork hosting its own error docs patches the constant.-- **`PipelexError.type_uri()` emits a trailing slash** (`pipelex/base_exceptions.py`) — form is `<base>/<kebab-class-name>/`. Matches the canonical URL MkDocs serves with `use_directory_urls: true` (verified against the built `<link rel="canonical">` of the deployed page). Clients dereferencing the URI now hit the docs page directly — no 301 round-trip. Treat `type_uri` as opaque on the API side; do NOT strip or normalize the trailing slash.-- **`docs/errors/<kebab-class-name>.md`** — one generated page per non-test `PipelexError` subclass. Each page carries the class identity table (`error_type`, `title`, `type_uri`, `error_domain`, defining module, parent-class link, class-level `user_action` when declared), a docstring fragment when present, and a back-link to the Error Model overview. Stamped with `<!-- pipelex:generated -->`. Maintainers can claim a page for hand-editing by adding `<!-- pipelex:authored -->` as a standalone line; the generator then preserves it across runs. A landing `docs/errors/index.md` lists every page grouped by top-level `PipelexError` branch.-- **`pipelex-dev generate-error-pages`** (`pipelex/cli/dev_cli/commands/generate_error_pages_cmd.py`) — internal CLI command to regenerate the pages. Quiet status line via `--quiet`; custom output dir via `--output DIR` (defaults to `docs/errors/`). Bootstraps Pipelex for parity with other dev CLI commands and to surface setup errors; discovery itself is handled by `iter_pipelex_error_subclasses`; tears down on exit.-- **`pipelex/errors/error_pages_generator.py::iter_pipelex_error_subclasses`** — yields every `PipelexError` subclass. Calls `_force_load_all_error_modules()` (a `functools.cache`-d helper) on first invocation to rglob the package for `exceptions.py` / `*_exceptions.py` modules and force-import each. Discovery is complete (AST set = runtime set, enforced by `tests/unit/pipelex/errors/test_error_class_location_convention.py`); the helper runs only inside dev/test-time consumers (the docs generator and the URI uniqueness test), never on production bootstrap.-- **`mkdocs.yml` updates** — added a `not_in_nav` directive whitelisting `errors/*.md` (except `errors/index.md`) and `CLAUDE.md` so `make docs-check` (= `mkdocs build --strict`) finishes with zero INFO/WARNING/ERROR; added "Reference > Error Reference" pointing at `errors/index.md`.-- **Pre-existing bug fix carried in this stage.** `pipelex/core/pipes/inputs/input_stuff_specs_factory.py` was shadowing `InputStuffSpecsFactoryError` with a local declaration that bypassed the canonical class in `pipelex/core/pipes/inputs/exceptions.py`. Two distinct class objects with the same name existed. Consolidated to one — the factory module now re-exports the canonical class via `__all__`; test imports keep working with no callsite changes.--**One deviation from the plan**, both already documented in §E above: `base_uri` defaulted to the live docs host (not `pipelex.dev`), and `type_uri()` emits a trailing slash. Pure URL-shape changes; no other contract shifts. The trailing slash is the only thing the API team needs to be aware of when comparing strings against `type_uri` values.diff --git a/wip/error-handling/architecture.md b/wip/error-handling/architecture.mddeleted file mode 100644index 6a7f29f59..000000000--- a/wip/error-handling/architecture.md+++ /dev/null@@ -1,161 +0,0 @@-# Architecture — Layer Model, Class Hierarchy, Reporting--This doc is the structural reference the tracks build on. It describes the layer model errors flow through, the class hierarchy, and how structured error reporting works today.--## Layer model--```-Layer 5: CLI entry points (catch + format for human/agent)-Layer 4: CLI factories (catch setup errors, route to handlers)-Layer 3: Pipeline runner (catch + wrap as PipelineExecutionError)-Layer 2: Pipe router / operators (catch + wrap with pipe context)-Layer 1: Workers / SDK calls (catch third-party + classify)-Layer 0: Third-party SDKs (raw OpenAI/Anthropic/Google/etc. exceptions)-```--### Layer 0 → 1: SDK to domain--Every provider worker under `pipelex/plugins/*/` catches the SDK's typed exceptions and re-raises a `CogtError` subclass with:--- An `InferenceErrorCategory` value (TRANSIENT, CONFIGURATION, CONTENT, CAPACITY).-- An optional `user_action` hint (set per-instance, or as a class default).-- The model descriptor in the message (`self.inference_model.desc` or `model_id`).-- `from exc` to preserve the original SDK exception in the chain.--Pure classification helpers live in `pipelex/cogt/inference/error_classification.py`: `is_quota_exhaustion_openai`, `is_quota_exhaustion_anthropic`, `is_quota_exhaustion_google`, `is_quota_exhaustion_mistral`, `is_quota_exhaustion_aws`, `is_quota_exhaustion_gateway`, and `is_content_policy_violation`. Per-provider quota and content-policy patterns are defined alongside as module constants.--See [track-worker-classification.md](track-worker-classification.md) for the per-worker inventory.--### Layer 1 → 2: Pipe operators--Pipe operators define thin wrapper exceptions in `pipelex/pipe_operators/*/exceptions.py` and `pipelex/pipe_operators/exceptions.py`. The richest is `PipeOperatorModelAvailabilityError` in `pipelex/pipe_operators/exceptions.py`, which carries `run_mode`, `pipe_type`, `pipe_code`, `pipe_stack`, `model_handle`, and `fallback_list`.--### Layer 2 → 3: Pipeline runner--`PipelexRunner.execute_pipeline()` in `pipelex/pipeline/runner.py` catches:--| Caught | Wrapped as | Context attached |-|---|---|---|-| `PipeRouterError` | `PipelineExecutionError` | run_mode, pipe_code, output_name, pipe_stack |-| Other `PipelexError` | `PipelineExecutionError` | run_mode, pipe_code, output_name, pipe_stack |-| `ValidationError` (pydantic) | `PipeExecutionError` | formatted validation message |--`PipelineExecutionError` is defined at `pipelex/pipeline/exceptions.py`; the original is chained with `from exc` and telemetry events are emitted on failure.--### Layer 3 → 4: CLI factories--- Human: `pipelex/cli/cli_factory.py` catches init errors from `Pipelex.make()` and delegates to handlers in `pipelex/cli/error_handlers.py`.-- Agent: `pipelex/cli/agent_cli/commands/agent_cli_factory.py` catches the same set and sends each through `agent_error()` in `pipelex/cli/agent_cli/commands/agent_output.py`.--### Layer 4 → 5: Delivery--- **Human CLI** — Rich console; each error type has its own `handle_*` function in `error_handlers.py`, all building their panel through the shared `display_error_panel()` helper (red banner, structured fields, tip, doc/Discord links, `raise typer.Exit(1) from exc`).-- **Agent CLI** — Structured JSON or markdown via `agent_error()`, dispatched on the `--format` option (markdown default; carried by a per-invocation `ContextVar`). For `PipelexError` subclasses it reads `to_error_report()` and merges `hint` / `error_domain` / `retryable` from the lookup dicts only when not present on the report.-- **Bundle validation** — `ValidateBundleError` aggregates blueprint, factory, validation, and instantiation errors; `extract_validation_errors()` in `agent_output.py` flattens them into a list per-category.-- **Markdown special case** — `InferenceSetupRequiredError` is rendered as markdown to stdout (exit 0) for first-run guidance, independent of the `--format` option.--## Class hierarchy--`PipelexError` is the single root. The full top-level shape:--```-Exception-└── PipelexError pipelex/base_exceptions.py- ├── PipelexUnexpectedError- ├── PipelexConfigError- ├── PipelexSetupError- ├── SecurityError- ├── ToolError pipelex/system/exceptions.py- │ ├── NestedKeyConflictError- │ ├── StorageError- │ ├── Jinja2TemplateSyntaxError- │ ├── SecretNotFoundError- │ └── ...- ├── MissingDependencyError pipelex/system/exceptions.py- ├── CredentialsError pipelex/system/exceptions.py- ├── CogtError pipelex/cogt/exceptions.py- │ └── (many subclasses, see track-worker-classification.md)- ├── PipeExecutionError pipelex/pipeline/exceptions.py- ├── PipelineExecutionError pipelex/pipeline/exceptions.py- ├── PipeStackOverflowError pipelex/pipeline/exceptions.py- ├── PipeRunError pipelex/pipe_run/exceptions.py- │ ├── PipeRouterError- │ ├── PipeRunParamsError- │ └── PipeJobError- ├── DeliveryError pipelex/pipe_run/exceptions.py- ├── ConceptError pipelex/core/concepts/exceptions.py- ├── StuffError pipelex/core/stuffs/exceptions.py- ├── WorkingMemoryError pipelex/core/memory/exceptions.py- ├── LibraryError pipelex/libraries/exceptions.py- ├── PipeControllerError pipelex/pipe_controllers/exceptions.py- ├── PipelexServiceError pipelex/system/pipelex_service/exceptions.py- │ ├── InferenceSetupRequiredError- │ ├── GatewayTermsNotAcceptedError- │ └── ...- ├── PipelexInterpreterError pipelex/core/interpreter/exceptions.py- ├── GraphSpecError pipelex/graph/exceptions.py- └── KitError pipelex/kit/exceptions.py--TracebackMessageError pipelex/system/exceptions.py-└── FatalError- ├── ConfigValidationError- └── ConfigModelError (also inherits from ValueError)-```--Notes:--- `PipelexError` lives at `pipelex/base_exceptions.py` (top-level), not under `pipelex/system/`. Several older docs reference the old location; treat any `pipelex/system/exceptions.py` reference for the root class as stale.-- Exception modules follow the **one `exceptions.py` per package** convention, with sub-packages (e.g. `pipelex/pipe_operators/llm/exceptions.py`) adding their own as needed.-- `TracebackMessageError` is the one exception type that uses a separate logging-on-construction mechanism (see "Open hierarchy issues" below).--### Three patterns for individual exceptions--- **Plain message-only exception** — used purely for type-based catching. Most subclasses.-- **Exception with structured fields** — carries context as instance attributes. Examples: `PipelineExecutionError`, `PipeOperatorModelAvailabilityError`, `ModelChoiceNotFoundError`, `InferenceBackendCredentialsError`.-- **Non-exception structured error data** — `BaseModel` / dataclass aggregated into a raised exception, not raised itself. Examples: `ErrorReport`, `PipelexBundleBlueprintValidationErrorData`, `PipeValidationErrorData`, `PipeFactoryErrorData`.--## Structured error reporting--`ErrorReport` (`pipelex/base_exceptions.py`) is a frozen `BaseModel` (`ConfigDict(frozen=True, extra="forbid", arbitrary_types_allowed=True)`) — the single source of truth for all error serialization (CLI JSON output, agent output, Temporal error details). Frozen so a cause-enrichment pass rebuilds via `model_copy` rather than mutating an outer wrapper's identity in place.--```-ErrorReport- error_type: str- message: str- title: str- type_uri: str- error_category: str | None- error_domain: str | None- retryable: bool | None- user_action: UserAction | None- model: str | None- provider: str | None- provider_metadata: ProviderErrorMetadata | None- caller_facing_message: bool-```--`title` / `type_uri` are the stable identity pair surfaced to consumers — every `PipelexError` populates them automatically via `PipelexError.title()` / `PipelexError.type_uri()` (auto-derived from the class name unless a subclass declares `_declared_title` / `_declared_type_uri`), so consumers never humanize or kebab-case a class name themselves.--Serialization goes through `to_dict(disclosure_mode=DisclosureMode.VERBOSE)`, not a plain dump. `DisclosureMode` has two values:--- `VERBOSE` — every populated (non-`None`) field; the strict inverse of `from_dict`, so it round-trips. Use on internal-trust boundaries (webhook payloads, internal RPCs) that need round-trip fidelity.-- `STRICT` — a lossy projection for untrusted external surfaces: `provider` / `model` are always dropped, `provider_metadata` is reduced to a curated subset (`status_code`, `retry_after_seconds`), and unless the report sets `caller_facing_message` its `message` is replaced with a generic placeholder and `user_action` dropped — leaving only the stable identifiers. `caller_facing_message` flags reports whose `message` is genuinely caller-facing copy (e.g. a `.mthds` syntax error from `PipelexInterpreterError`), so STRICT keeps it. STRICT keys on message *provenance*, not on `error_domain` (which is inherited up the `__cause__` chain).--`PipelexError.to_error_report()` returns `error_type`, `message`, `title`, `type_uri`, the class-level `error_domain`, `user_action`, and `caller_facing_message`. `CogtError.to_error_report()` overrides to add `error_category`, `retryable` (derived from category), `provider_metadata`, and reads `model_handle` / `backend_name` when present on the instance. `to_error_report()` enriches from the `__cause__` chain, so a wrapper exception surfaces the inference metadata of the underlying `CogtError` — but `title` / `type_uri` are *wrapper-wins* (enrichment never overwrites the outermost wrapper's identity).--`error_category` carries an `InferenceErrorCategory` value (`pipelex/cogt/exceptions.py`). Only `TRANSIENT` is retryable; `CONFIGURATION` / `CONTENT` / `CAPACITY` / `AMBIGUOUS` / `UNKNOWN` are not. The `AMBIGUOUS` / `UNKNOWN` split is deliberate: `AMBIGUOUS` means the error type is known but the outcome is not (e.g. a connection dropped mid-request, so a non-idempotent operation may or may not have committed) — a blind retry is unsafe, so it is non-retryable by design; `UNKNOWN` means classification failed outright. Keeping them separate lets a non-idempotent worker (e.g. `azure_img_gen_worker.py`) pick `AMBIGUOUS` rather than the looser `UNKNOWN`.--See [track-metadata-model.md](track-metadata-model.md) for the remaining gap: `error_domain` is now a class-level attribute and inference errors fully self-describe, but a long tail of non-inference `PipelexError` subclasses still depend on the lookup dicts in `pipelex/cli/agent_cli/commands/agent_output.py` for their `hint` / `error_domain`.--## Open hierarchy issues--Tracked here because they don't fit cleanly into a track:--- **`TracebackMessageError` separate lifecycle.** `TracebackMessageError` (`pipelex/system/exceptions.py`) logs its message on construction and chooses between `logger.error` and `logger.exception` based on a `ClassVar` `error_mode`. It is rooted under `PipelexError` but doesn't participate in `to_error_report()` enrichment. No real cost today — it works fine for fatal startup errors. Low priority.--## Related tracks--- [track-metadata-model.md](track-metadata-model.md) — `ErrorReport`, `error_category`, `error_domain`, `user_action`, dict drift.-- [track-worker-classification.md](track-worker-classification.md) — Layer 0 → 1 details.-- [track-cli-delivery.md](track-cli-delivery.md) — Layer 4 → 5 details.-- [track-temporal-integration.md](track-temporal-integration.md) — how the layer model extends into Temporal workflows.diff --git a/wip/error-handling/temporal-error-handling-failsafe-gap.md b/wip/error-handling/temporal-error-handling-failsafe-gap.mddeleted file mode 100644index 57d957420..000000000--- a/wip/error-handling/temporal-error-handling-failsafe-gap.md+++ /dev/null@@ -1,114 +0,0 @@-# The real defect: the Temporal integration can hang silently instead of failing loud--> **Cold start:** this doc is the *why* — resolved rationale, read-only. The live open backlog is in [`temporal-failsafe-review-followups.md`](./temporal-failsafe-review-followups.md); the directory index is [`README.md`](./README.md). Read here only when you need the rationale behind the fix or the deferred-hardening list (C/D/F) below.--**Status:** ✅ **FIXED** (fixes A + B + E landed on `fix/Temporal-failsafe`). The fail-safe floor is in place; the silent-hang hole is closed. Deferred hardening (C, D, F) is tracked under "Deferred follow-ups" below — none of it is required to close the hole.-**Severity:** high — this is a class of bug, not one bug. The missing Search activity (see [`search-operator-missing-temporal-activity.md`](./search-operator-missing-temporal-activity.md)) is only the first trigger. Any future code path where an exception escapes pipe execution without having been converted to a Temporal terminal failure would hang the runner the same way. Fixing Search closes the instance; the fix below closes the hole.-**Companion brief:** read the Search brief first for the concrete instance; this one is about why the system let that instance become a silent hang.--## Resolution (implemented)--The fix establishes the fail-safe floor the design lacked, scoped to pipelex **domain** errors so transient Temporal/infra errors keep Temporal's (correct) task-retry behavior:--- **Fix A — workflow-level catch-all.** `WfPipeRouter.run()` and `WfPipeRun.run()` each gained a final `except PipelexError` boundary. `WfPipeRouter` converts a *genuine inline* error to a terminal `TemporalError` via `from_message_exception` (rich report in `ApplicationError.details`, exactly as the activity bridge does). `WfPipeRun` routes the inline error through its deferred-delivery path so the FAILED webhook still fires, then chains a details-carrying `TemporalError` so the classification also reaches the submitter. The catch-all converts only genuine inline errors: an escaping `PipelexError` that already carries a Temporal failure in its `__cause__` chain (e.g. a nested `TemporalPipeRouter` child-dispatch that already wrapped a sub-pipe failure as `WorkflowExecutionError`) is already terminal and recoverable, so it propagates untouched (guarded by `_carries_temporal_failure`) rather than being flattened to a generic report.-- **Fix B — worker-level floor.** `workflow_failure_exception_types` is now `[WorkflowExecutionError, PipelexError]` (`pipelex/temporal/temporal_task_manager.py`). Any domain error that slips past the catch-alls fails the workflow terminally instead of hanging — degrading to a synthesized `UnrecoverableWorkflowFailureError` (message preserved) rather than a silent retry loop.-- **Fix E — defensive tests.** `tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py` pins the escape path: (1) an inline `PipelexError` in `WfPipeRouter` surfaces fully classified; (2) an inline `PipelexError` in `WfPipeRun` fires the FAILED webhook and reaches the submitter classified; (3) a raw uncaught `PipelexError` on the *production* worker fails terminally with the message preserved. Each sets a short `workflow_execution_timeout` so a regression that reopens the hole fails fast (on the wrong, timeout-shaped classification) instead of hanging the suite. The two contract tests that mirror the production worker config were updated to the broadened list. Full temporal suite green.--Docs updated: `docs/under-the-hood/error-model.md` gains a "Workflow-Level Fail-Safe Floor" subsection.--## Deferred follow-ups--These harden the system further but are **not** required to close the silent-hang hole (A + B already do). They are deferred because each is a distinct concern — cross-repo, a different bug class, or ops:--- **C — separate short submitter-side deadline for the synchronous API path.** Genuinely cross-repo: the sync `/execute` endpoint lives in `pipelex-api`, and giving it a bounded result-wait distinct from the 1h durable `workflow_execution_timeout` requires a caller-side change there (plus a config knob plumbed from `WorkerConfig` here). With A + B the silent hang is already closed — any domain error now fails terminally and fast — so C is defense-in-depth against *other* causes of slowness (genuinely slow pipes, infra hangs), not this defect. The brief's "no `pipelex-api` change" framing rules C out of a pipelex-only fix.-- **D — CI guard for the activity-dispatch invariant.** This is the regression guard for the *Search-class trigger* (an inference operator forgetting to route its leaf through an activity), tracked by the companion Search brief — not the systemic hole this brief is about. The floor (A + B) already degrades any such omission to a loud, classified failure instead of a hang, so D is a quality guard for the trigger, deferrable independently.-- **F — observability on stuck workflows.** A metric/alert on workflow-task-failure accumulation (or workflows open past an interactive threshold). Ops/infra concern; with A + B nothing should reach the stuck state for a domain error, so this is a backstop-for-the-backstop.--## Where the fix happens--Entirely in the `pipelex` repo (the open-source runtime). This brief is filed in `pipelex-api/wip/` because the symptom showed up there, but no `pipelex-api` change is involved. The next session is expected to run from a `pipelex` worktree.--**Path convention:** every `pipelex/...` path is relative to the **pipelex repo root** = your worktree root. Work inside the worktree; do not reach into a sibling `../pipelex/` checkout.--## One-sentence statement of the defect--The error-handling architecture (`pipelex/wip/error-handling/`) makes **terminal** Temporal failures cross the boundary beautifully — but it has **no backstop for the non-terminal "workflow task failure" mode**, which is exactly what any unconverted exception inside pipe code triggers, and which Temporal's default turns into a silent, ~1-hour hang that finally surfaces as the *wrong* error.--## Why this is the actual problem (not "Search forgot an activity")--The error-handling work invested heavily in the **terminal-failure recovery chain**:--- activity → workflow: `convert_pipelex_errors` makes leaf errors `TemporalError(ApplicationError)`; `from_app_error` recovers them in workflow code.-- workflow → submitter: `recover_error_report` + the `except WorkflowFailureError` clause rebuild the `ErrorReport` on the caller.-- child → parent: `WfPipeRun`'s `except ChildWorkflowError` lifts the report out of `exc.cause`.--Every one of these handlers fires **only when the inner unit fails *terminally* with a recognized failure type** (`ApplicationError` / `FailureError` / `WorkflowExecutionError`). That is the unstated, load-bearing precondition of the entire design.--Temporal has a second failure mode that the design never accounts for: a **workflow task failure**. When workflow code raises any exception that is *not* an `ApplicationError`/`FailureError` and *not* in the worker's `workflow_failure_exception_types`, Temporal does **not** fail the workflow. It assumes a transient/code-deploy bug, and **retries the workflow task** — re-running the workflow code from the last consistent point, indefinitely, bounded only by the workflow execution timeout.--A workflow-task failure reaches **none** of the recovery handlers above (there is no terminal failure to recover). The child never produces a `ChildWorkflowError`; the parent's `await execute_child_workflow(...)` simply never returns; the submitter's `execute_workflow(...)` simply never returns. The polished recovery chain is bypassed wholesale, and the system's behavior degrades to its worst possible quadrant: **invisible + unbounded-ish + resource-consuming.**--## The exact chain (Search as the worked example)--> Pre-fix walkthrough: line numbers are dropped and present-tense handler claims are framed as the pre-fix state, since the fix (the two `except PipelexError` catch-alls + the broadened worker registration) changed exactly the code this describes.--1. `WfPipeRouter.run()` executes the pipe inline (its `pipe.run_pipe` call). At the time its only handler was `except ActivityError`.-2. A `SearchJobFailureError` (a `CogtError`/`PipelexError`) is raised inline — **not** an `ActivityError`, **not** an `ApplicationError`, **not** a `WorkflowExecutionError`. It escapes the child workflow raw.-3. Worker then registered `workflow_failure_exception_types=[WorkflowExecutionError]` only (`pipelex/temporal/temporal_task_manager.py`). The escaping type isn't in it → **child workflow-task failure → child retries its task**, re-executing the inline search call for real on each retry (real provider spend + log spam; no result is recorded in history because there is no activity).-4. The child never fails terminally, so `WfPipeRun`'s `except ChildWorkflowError` (`pipelex/temporal/tprl_pipe/wf_pipe_run.py`) — the handler that would have produced a classified `WorkflowExecutionError` and fired the failure webhook — **never runs**. The parent's `await workflow.execute_child_workflow(...)` blocks.-5. The submitter's `execute_workflow(...)` (`pipelex/temporal/tprl/workflow_caller.py`) is awaiting the parent's result with `execution_timeout = workflow_execution_timeout` (configured **1h**). The child has **no** execution timeout of its own (deliberately omitted for replay determinism — see `WfPipeRun.run`'s child-dispatch comment), so the whole tree is bounded only by the parent's 1h.-6. After ~1h the parent execution times out. The submitter gets a `WorkflowFailureError(TimeoutError)`; `recover_error_report` finds no `ApplicationError` report in a timeout chain and synthesizes an `UnrecoverableWorkflowFailureError` (`error_domain=RUNTIME`). The API returns **that opaque timeout** — not the `SearchJobFailureError` that was plainly in the worker logs the entire hour.--Net: a sync `/execute` request hangs for up to an hour and then returns the wrong, generic error, while the worker silently burns retries. The "improve error handling so it behaves well for the API" goal is fully defeated by this single uncovered mode.--## The systemic gaps (each is independent of the Search feature)--1. **The "every leaf error is converted at an activity boundary" invariant is unenforced.** It holds today only by manual enumeration — the hand-maintained `Tasks` activity list (`pipelex/temporal/tasks.py`) and `@convert_pipelex_errors` applied by hand per activity. Nothing fails when a new operator skips activity dispatch. Enumerated coverage rots; Search is the proof.--2. **No workflow-level catch-all that makes escaping domain exceptions terminal.** `WfPipeRouter` catches only `ActivityError`; `WfPipeRun` catches only `ChildWorkflowError`. Neither has a final clause that wraps an escaping `PipelexError` into a `WorkflowExecutionError` (the one type that *is* terminal here). Such a backstop would have turned the Search hang into an immediate, classified, surfaced failure — i.e. exactly the promised behavior — for any cause, known or not.--3. **`workflow_failure_exception_types` is set to the narrowest possible value and the default is the dangerous one.** It's `[WorkflowExecutionError]`. Everything depends on code paths *manually* arriving at that one type. The safer posture is to make Temporal fail the workflow terminally for any pipelex **domain** error by default (e.g. include `PipelexError`), so a missed conversion degrades to "fails loud with a slightly-less-rich report" instead of "hangs for an hour and lies."--4. **The synchronous API path has no short deadline and mis-prioritizes the surfaced error.** Two distinct faults: (a) a sync HTTP endpoint inherits the **1h durable** workflow execution timeout — there is no separate, short submitter-side result-wait for the interactive path; (b) on timeout the caller surfaces an opaque `UnrecoverableWorkflowFailureError` rather than the real, already-observed failure.--5. **Worst-quadrant failure with no observability.** During the hang there is no terminal event, no surfaced error, and (presumably) no alert — only an accumulating count of workflow-task failures visible in the Temporal UI if someone looks. A silent, resource-consuming, unbounded-until-timeout state is the single worst outcome a durable-execution system can produce, and nothing flags it.--6. **Tests pin the *terminal-classification* path, not the *escape* path.** The parity integration tests (`tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py` ↔ `.../error_handling/test_error_report_local_full_chain.py`) mock an activity to fail and assert `ErrorReport` parity — i.e. they exercise the path where conversion already happened. The defensive case that actually broke — "an unconverted exception escapes the workflow" — is untested. The test suite encoded the architecture's blind spot.--## Recommended fixes (systemic; do these even after Search is wired)--> **Status:** A, B, E are implemented — see [Resolution](#resolution-implemented) above. C, D, F are [deferred](#deferred-follow-ups). The original analysis is kept below for rationale.--Ordered by leverage. A and C are the floor; the rest harden and prevent regression.--**A. Workflow-level fail-safe (highest leverage).** In both `WfPipeRouter.run()` and `WfPipeRun.run()`, add a final boundary that converts an escaping pipelex **domain** exception into a terminal `WorkflowExecutionError` carrying `recover_error_report(...)` classification — mirroring the existing `ActivityError`/`ChildWorkflowError` handlers, but for the "raised inline, never went through an activity" case. This guarantees no pipe-code exception can become a silent workflow-task-retry hang again, for *any* operator.- - **Scope it carefully.** Do **not** blanket-convert `Exception`. Workflow-task retry is the *correct* behavior for genuinely transient Temporal/infra errors and for deterministic-replay glitches — making those terminal would throw away durability's whole point. Convert pipelex **domain** errors (`PipelexError` and friends), and let Temporal-internal exceptions keep their default. Per repo standards, a workflow-root `except` is one of the two sanctioned `except Exception` sites only if you immediately re-raise as terminal *and* you've reasoned about the transient case; prefer `except PipelexError` here.--**B. Broaden `workflow_failure_exception_types` to include `PipelexError`.** Cheap belt-and-suspenders behind A: even a path that slips past the catch-all fails the workflow terminally instead of hanging. (`pipelex/temporal/temporal_task_manager.py`.) Verify interaction with the existing `WfPipeRun` contract test `test_wf_pipe_run_failure_path` (it pins the current `[WorkflowExecutionError]` value).--**C. Separate, short submitter-side deadline for synchronous execution.** Give the sync `/execute` path a bounded result-wait (config, seconds-to-low-minutes) distinct from the 1h durable `workflow_execution_timeout` that the async `/start` path legitimately needs. On deadline, surface a clear, classified timeout — never leave the HTTP request hanging on the durable budget. (Submitter side: `pipelex/temporal/tprl/workflow_caller.py` + the timeout config plumbed from `WorkerConfig`.)--**D. Enforce the activity-dispatch invariant in CI.** A test that walks the inference `PipeOperator` subclasses and asserts each routes its leaf through `get_content_generator()` (equivalently: each inference leaf has a protocol method + a registered activity). The next operator that forgets the wiring then fails CI, not production. (This is the regression guard for the Search-class bug.)--**E. Negative/defensive test for the escape path.** Assert that a raw `PipelexError` raised *inside pipe code* within a workflow (not via an activity) results in a terminal, classified failure promptly surfaced to the submitter — the exact invariant that was silently violated. Pair it across local/Temporal like the existing parity tests.--**F. Observability on stuck workflows.** A metric/alert on workflow-task-failure accumulation (or workflows open past a sane interactive threshold), so a path that still slips through A–E is visible instead of silent.--## The meta-lesson worth recording--The error-handling project optimized the **fail-loud** path to a high polish but never established a **fail-safe floor**. It assumed terminal classification everywhere and built rich recovery on top of that assumption, without a guard for the case where the assumption doesn't hold. In a durable-execution system the default for "unexpected exception in business logic running inside a workflow" is the most dangerous behavior available (retry forever), so "we convert all the errors we know about" is not enough — the system needs a backstop that holds for the errors, and the code paths, that nobody enumerated. Search exposed it; the next gap will too unless the floor exists.--## Reference file map (relative to pipelex repo root / your worktree root)--> Symbolic refs (clause / function names), not line numbers — the fix shifted the lines and the catch-alls below now exist, so line-pinned navigation would mislead. Where the *gap* code differs from current reality it is noted inline.--- Workflow that runs pipe code inline (`WfPipeRouter.run`'s `pipe.run_pipe` call): `pipelex/temporal/tprl_pipe/wf_pipe_router.py` — its `except ActivityError` handler, now backstopped by the `except PipelexError` inline fail-safe this fix added.-- Parent workflow (`WfPipeRun.run`), `except ChildWorkflowError` plus the `except PipelexError` inline fail-safe this fix added; child started without its own execution timeout (deliberate, for replay determinism): `pipelex/temporal/tprl_pipe/wf_pipe_run.py`-- Worker `workflow_failure_exception_types` (now `[WorkflowExecutionError, PipelexError]`; was `[WorkflowExecutionError]` pre-fix): `pipelex/temporal/temporal_task_manager.py`-- Submitter wait + 1h `execution_timeout` applied (`WorkflowExecutor.execute_workflow`): `pipelex/temporal/tprl/workflow_caller.py`-- Timeout config source (`workflow_execution_timeout`, `run_timeout`, `rpc_timeout`): `pipelex/temporal/config_temporal.py` (`WorkerConfig`) + `pipelex/pipelex.toml` temporal section (~line 489)-- Activity error conversion (works for activity-dispatched leaves only): `pipelex/temporal/tprl/activity_error_boundary.py`-- Submitter-side report recovery (terminal-failure path only): `pipelex/temporal/tprl/temporal_error.py` (`recover_error_report`)-- Existing parity tests (terminal path; escape path untested): `tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py`, `tests/integration/pipelex/error_handling/test_error_report_local_full_chain.py`-- Design docs this critiques/extends: `pipelex/wip/error-handling/track-temporal-integration.md`, `.../track-retry-and-resilience.md`-- The concrete first instance: [`search-operator-missing-temporal-activity.md`](./search-operator-missing-temporal-activity.md)diff --git a/wip/error-handling/temporal-failsafe-guard-hardening.md b/wip/error-handling/temporal-failsafe-guard-hardening.mddeleted file mode 100644index ee689ca10..000000000--- a/wip/error-handling/temporal-failsafe-guard-hardening.md+++ /dev/null@@ -1,131 +0,0 @@-# Temporal fail-safe — harden & consolidate the inline-error guard--> **Cold start:** this doc is resolved rationale, read-only (findings 1/6/7/9, all handled). The live open backlog is in [`temporal-failsafe-review-followups.md`](./temporal-failsafe-review-followups.md); the directory index is [`README.md`](./README.md). Note: Finding 1's "resolved — kept the broad predicate" conclusion is *reopened* by Finding 13 in the followups doc (a third fragility mode the analysis here missed).--**Status:** ✅ **RESOLVED** on `fix/Temporal-failsafe`. All four findings handled; the chain-walk and fail-safe suites are green. Summary of decisions (the original analysis is preserved below for the record):--- **Finding 7 (done).** Extracted one cycle-guarded `iter_cause_chain(exc)` primitive in `base_exceptions.py`; the five hand-rolled `__cause__` walks (`_carries_temporal_failure`, `find_inference_error_category_in_chain`, `_find_error_report_dict`, `_message_from_exc`, and the cycle check in `_enrich_error_report_from_cause` — the doc missed this fifth one) now delegate to it.-- **Finding 9 (done).** Added `test_router_already_terminal_error_propagates_untouched_preserving_leaf_classification` for the guard's `True` branch. Proven meaningful: forcing the guard to `False` flips the submitter's `error_type` from the leaf `LLMCompletionError` to the generic `WorkflowExecutionError`, which the new test catches — while the convert-branch sibling stays green, which is exactly why this branch needed its own coverage.-- **Finding 1 (kept, *not* rewritten).** Verification upgraded the doc's own risk note: the guard's `True` branch is **reachable**, not theoretical — a controller sub-pipe failing as a child workflow is wrapped by `TemporalPipeRouter` as `WorkflowExecutionError(msg)` (no `error_report`) and escapes the parent's `pipe.run_pipe`. Its rich leaf report lives only in the child's `ApplicationError.details`, recoverable solely by `recover_error_report` at the submitter; any worker-side `from_message_exception` flattens it. **Both proposed rewrites regress this** — direction (a) (direct-cause check) is equal-or-worse (a deeper `FailureError` in a legitimate nested wrapper would now flatten), and direction (b) (always-convert) flattens the reachable case outright. So the broad "carries a Temporal `FailureError`" predicate is kept — it *is* the definition of "already terminal", not a proxy for a narrower type — refactored onto the primitive, with its docstring now stating the reachable case and the `raise … from` invariant (the over-inclusive "recover-then-rechain" failure requires an anti-pattern no operator does). The Finding 9 test pins it.-- **Finding 6 (light touch).** The two catch-alls' control flow is irreducibly different (router raises immediately; parent defers so `act_deliver` still fires the FAILED webhook), so they were **not** folded into one helper — that would force a dead, unreachable guarded branch into `WfPipeRun`. Instead the duplicated *rationale prose* (Finding 11) was deduped: the full argument lives once in `_carries_temporal_failure`'s docstring (code contract) + `docs/under-the-hood/error-model.md` "Workflow-Level Fail-Safe Floor" (design narrative); the router / parent / worker-registration comments carry a short statement plus a pointer.--**Left open (followups doc):** Finding 4 (parent lacks the guard) stays open but is assessed harmless — `WfPipeRun`'s `except PipelexError` can only catch errors from the pure `build_search_attributes` / `build_static_summary` argument evaluation, which cannot carry a `FailureError`, so a guarded branch there would be dead code. Findings 2, 3, 5, 8, 10, 12 are untouched (out of this doc's scope).-------**Original review follow-up (for the record).** These were the highest-value findings from the code review of the fail-safe floor that landed on `fix/Temporal-failsafe`. None blocked the merge — the silent-hang hole *is* closed — but all four concern the single load-bearing mechanism the fix relies on, and they were best done together as one small refactor.--**Companion docs:** the fix itself is described in [`temporal-error-handling-failsafe-gap.md`](./temporal-error-handling-failsafe-gap.md) (the "why" and "what landed"); the landed behavior is summarized in [`track-temporal-integration.md`](./track-temporal-integration.md) under "Workflow-level fail-safe floor". The other, lower-priority review findings are in [`temporal-failsafe-review-followups.md`](./temporal-failsafe-review-followups.md).--**Finding numbers** (1, 6, 7, 9) are kept from the original review so the two follow-up docs cross-reference cleanly.-------## Background (cold-start)--The fail-safe floor closes a hole where a pipelex **domain** error raised *inline in workflow code* — never dispatched through an activity — was neither an `ActivityError` nor an `ApplicationError`, so Temporal treated it as a *workflow-task* failure and retried it indefinitely: a silent, resource-burning hang surfaced only as a generic timeout after the execution timeout.--The fix has two layers:--- **Workflow-level catch-all.** `WfPipeRouter.run()` (`pipelex/temporal/tprl_pipe/wf_pipe_router.py`) and `WfPipeRun.run()` (`.../wf_pipe_run.py`) each end their boundary handling with an `except PipelexError` clause that converts a *genuine inline* error into a terminal, classified `TemporalError` (an `ApplicationError` carrying the structured `ErrorReport` in `details`).-- **Worker-level floor.** The worker registers `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]` (`pipelex/temporal/temporal_task_manager.py`), so any domain error that slips past the catch-alls still fails the workflow terminally instead of hanging.--The "genuine inline" decision is made by **`_carries_temporal_failure(exc)`** in `wf_pipe_router.py:26`. It walks the exception's `__cause__` chain and returns `True` if any node is a Temporal `FailureError`. `True` ⇒ the error already originated at a Temporal boundary (activity / child workflow), is already terminal, and its report is recoverable from the chain by `recover_error_report` at the submitter — so the catch-all leaves it untouched. `False` ⇒ a genuine inline error that never crossed a boundary — convert it via `TemporalError.from_message_exception`.--**Key fact established during review (why none of these is a live bug yet):** the real inline-leaf path, `pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py`, dispatches activities and, on `ActivityError`, raises `TemporalError.from_app_error(exc=exc.cause) from exc` — a `FailureError`, wired with `from`. So `_carries_temporal_failure` classifies it correctly. Genuine inline errors (the Search-operator-missing-activity case, crate-load, hydration setup) carry no `FailureError` and are converted correctly.--> **Correction (this session).** The original review claimed "no confirmed live trigger" for the guard's `True` branch. That was wrong, and it reshaped Finding 1: the `True` branch **is** reachable — a controller sub-pipe failing as a child workflow is wrapped by `TemporalPipeRouter` (`temporal_pipe_router.py:92-94`) as `WorkflowExecutionError(msg)` and escapes the parent's `pipe.run_pipe`. So the guard is load-bearing on a real path, not just a latent safeguard. What *remains* latent are only the two **misclassification** fragilities below (under-/over-inclusive), each of which requires a `raise … from` discipline violation that no operator currently makes.-------## ✅ Finding 1 — `_carries_temporal_failure` uses chain-membership as a proxy for "already terminal", which is fragile both ways--> **Resolved — kept the guard, not rewritten.** Both proposed directions regress the *reachable* nested-failure case (worker-side conversion flattens a report only the submitter can recover). The broad "carries a Temporal `FailureError`" predicate is correct — it *is* the definition of "already terminal" — and now sits on `iter_cause_chain` with a docstring stating the reachable case + the `raise … from` invariant. Pinned by the Finding 9 test. (Detail in the top summary.)--**Where:** `pipelex/temporal/tprl_pipe/wf_pipe_router.py:26` (the helper), `:195` (the guard call), `:197` (the conversion it gates).--**What:** the rule "any `FailureError` anywhere in the `__cause__` chain ⇒ propagate untouched" is a *proxy* for the real question, which is "is **this** error already a terminal, report-carrying Temporal failure?". The proxy is both over- and under-inclusive:--- **Under-inclusive (flattening).** A Temporal-failure-carrying error wrapped **without** `from` (implicit `__context__` only, not `__cause__`) reads as "no failure in chain" → the guard returns `False` → the already-classified error is re-converted by `from_message_exception`, flattening its rich leaf report (`model` / `provider` / `error_category`) to the wrapper's generic classification. Pipelex's coding standard is `raise … from exc`, so this requires a *missing* `from` somewhere on a workflow-inline path — the guard is silently fragile to that mistake.-- **Over-inclusive (misattribution).** A genuinely fresh inline error that is chained `from` an unrelated, *already-recovered* side failure (pipe code dispatches a side activity, it fails as a `TemporalError`, the code catches and recovers, then later raises a different fresh domain error `from` the recovered one) carries a `FailureError` in its chain → the guard returns `True` → it propagates untouched and carries **no report of its own**. At the submitter, `recover_error_report` walks the chain and surfaces the **side** failure's `error_type` / `error_category` / `model` — i.e. the run is reported as the wrong failure.--**Why it matters:** this guard is the hinge of the whole fix. The failure modes are latent today (see the key fact above), but a single missing `from`, or one operator that recovers-then-rechains, silently degrades the exact classification fidelity the error-handling project exists to deliver — and there is no test on the `True` branch to catch a regression (see Finding 9).--**Proposed direction:** replace the membership heuristic with a decision about *this* error's own terminality/recoverability. Options to weigh:--- Convert only when the error is **not itself** a `FailureError` *and* does not **directly** wrap one as its immediate `__cause__` (rather than anywhere in the chain) — tightens the over-inclusive case.-- Or: always convert (mint a fresh `TemporalError` from `exc`), but make `from_message_exception` *prefer an already-present report in the chain* over re-deriving — so flattening becomes impossible regardless of `from`/`__cause__` shape. This removes the guard's load-bearing role entirely and may be the deeper fix.--**Open questions:**--1. Do we tighten the predicate (direct-cause check) or eliminate it by making the conversion report-preserving (prefer the chain's existing report)?-2. Is "recover-then-rechain" a pattern we ever want in pipe/operator code, or should we forbid it (and lean on `from`-discipline + a lint)? That answer decides whether the over-inclusive case is worth engineering against at all.-------## ✅ Finding 7 — `_carries_temporal_failure` is the 4th hand-rolled `__cause__`-chain walk--> **Resolved — done.** Extracted `iter_cause_chain(exc)` in `base_exceptions.py`; all five `__cause__` walks (the four named here + the cycle check in `_enrich_error_report_from_cause`) plus two unguarded strays found while sweeping (`agent_output._build_error_source`, `pipe_llm._format_llm_error` — both previously missing the cycle guard) now delegate to it.--**Where:** `pipelex/temporal/tprl_pipe/wf_pipe_router.py:26`. Siblings that re-implement the same walk: `find_inference_error_category_in_chain` (`pipelex/cogt/exceptions.py`, ~`:129`), `_find_error_report_dict` (`pipelex/temporal/tprl/temporal_error.py:41`), `_message_from_exc` (`.../temporal_error.py:68`).--**What:** all four are the identical loop — `node = exc; seen: set[int] = set(); while node is not None and id(node) not in seen: <predicate>; seen.add(id(node)); node = node.__cause__` — differing only in the per-node predicate. Each re-rolls the cycle guard (the `id()` set that prevents an infinite spin on a cyclic `__cause__` chain) by hand.--**Why it matters:** on the error-reporting path a cycle-guard bug would itself become a hang — the very failure the fail-safe exists to prevent. Four hand-copies mean a fifth will get it subtly wrong. This is pure reuse debt, but it sits on the most safety-critical path.--**Proposed fix:** extract one primitive — e.g. `find_first_in_cause_chain(exc, predicate) -> BaseException | None` (and/or `any_in_cause_chain`) — in `pipelex/base_exceptions.py` (or a small `pipelex/tools` exception util) with the cycle guard written once. Then:--- `_carries_temporal_failure(exc)` collapses to `any_in_cause_chain(exc, lambda n: isinstance(n, FailureError))`.-- `_find_error_report_dict`, `_message_from_exc`, `find_inference_error_category_in_chain` all delegate to it.--**Open question:** where does the primitive live so both `cogt/` and `temporal/` can import it without a layering inversion or circular import? `base_exceptions.py` is the natural home (it already defines `PipelexError` and a cycle-guarded `_enrich_error_report_from_cause` walk) — confirm that placement.-------## ✅ Finding 6 — Two near-identical catch-alls were copy-pasted into two workflows; the worker-floor registration is the actual hole-closer (altitude)--> **Resolved — light touch.** Catch-alls kept separate (their control flow is irreducibly different: router raises immediately, parent defers so `act_deliver` fires the FAILED webhook). The duplicated *rationale prose* was deduped onto `_carries_temporal_failure`'s docstring (code contract) + `error-model.md` (design); the router / parent / worker-registration comments now carry a statement + pointer. No shared helper — that would have forced a dead guarded branch into `WfPipeRun` (followups Finding 4).--**Where:** the `except PipelexError` clause in `wf_pipe_router.py:174` and in `wf_pipe_run.py:83`; the registration in `temporal_task_manager.py:159`.--**What:** the single line `workflow_failure_exception_types=[…, PipelexError]` already closes the hang for **every** workflow, present and future, with zero per-workflow code — it guarantees an escaping `PipelexError` fails terminally instead of hanging. The two hand-written catch-alls only *upgrade the report* from a synthesized `UnrecoverableWorkflowFailureError` to the original classification. That richness is real and worth having, but it means:--- the actual hole-closer is one config line, while the design doc / CHANGELOG / three multi-paragraph inline comment blocks present the catch-alls as the headline fix;-- the "convert genuine inline error / leave already-terminal alone" decision is split across two ~20-line blocks that are implemented **divergently** — the router guards with `_carries_temporal_failure` and raises immediately; the parent omits the guard and defers via the delivery path (see Finding 4 in the followups doc);-- every future workflow author must remember to copy the clause to get the rich report.--**Why it matters:** maintenance weight and drift. A change to the conversion policy must be applied in N places, and the two existing copies already disagree.--**Proposed fix (right altitude):** hoist the policy into one shared helper, e.g. `convert_inline_pipelex_error(exc) -> TemporalError` that owns the `_carries_temporal_failure` short-circuit (or its Finding-1 replacement) and the `from_message_exception` conversion. Both workflows call it; each chooses only *immediate-raise* (router) vs *deferred-delivery* (parent). The policy then lives once, the guard becomes non-optional, and Finding 4's asymmetry disappears for free.--**Open questions:**--1. Is the per-workflow rich-report richness worth keeping at all, or do we accept the worker floor's synthesized `UnrecoverableWorkflowFailureError` and delete the catch-alls? (The floor already prevents the hang; the question is purely "rich classification vs less code".) Recommendation: keep the richness but centralize it.-2. If we keep it, should the shared helper be a plain function both sites call, or a workflow-root decorator/util applied uniformly to all `@workflow.defn` run methods so new workflows get it automatically?-------## ✅ Finding 9 — The `_carries_temporal_failure` `True` branch is untested--> **Resolved — done.** Added `test_router_already_terminal_error_propagates_untouched_preserving_leaf_classification`. Proven meaningful: forcing the guard to `False` flips the submitter's `error_type` from the leaf `LLMCompletionError` to the generic `WorkflowExecutionError`, which the new test catches while the convert-branch sibling stays green.--**Where:** `tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py`.--**What:** all three new tests inject errors with **no** `FailureError` in the chain (`make_failing_llm_error()` builds a clean `LLMCompletionError`; `WorkflowInputError` is raw). So they exercise only the `False` branch (convert). The `True` branch — "the error already carries a Temporal failure, do **not** re-wrap" — is the trickier, more error-prone arm and has **zero** coverage. Negate or delete the guard and the suite stays green.--**Why it matters:** the guard from Finding 1 is the fix's hinge, and its hardest branch is the one with no regression net. Combined with the retryable-inline gap (Finding 2 in the followups doc), the test suite currently pins the easy paths and leaves the subtle ones open.--**Proposed fix:** add a test that raises, inline in `WfPipeRouter`, a `PipelexError` whose `__cause__` chain already contains a report-carrying `TemporalError`/`ApplicationError` (simulating an already-terminal escapee), and assert it propagates **untouched** — i.e. the submitter recovers the *original* leaf classification, not a re-wrapped generic one. If Finding 1 is reworked to be report-preserving, this test should assert the preserved leaf report either way.--**Open question:** the new tests run **unsandboxed** (`is_not_sandboxed=True`), like the rest of the temporal suite. The catch-all now calls `from_message_exception` (→ `get_config()`, `find_inference_error_category_in_chain`, `ErrorReport.to_dict`, `workflow_log`) inside workflow code under the production `SandboxedWorkflowRunner` — a surface no test covers under the sandbox. Do we add one sandboxed smoke test for the conversion path, or is the existing unsandboxed coverage + the passthrough-module config (`temporal_task_manager.py` `with_passthrough_modules(...)`) sufficient assurance? (This is a pre-existing gap for the whole suite, not introduced here, but the floor is now one more untested-under-sandbox surface.)-------## What was done (final sequencing)--All four landed as one unit of work:--1. **Finding 7** — extracted `iter_cause_chain` (a generator, not `find_first_in_cause_chain`: it also subsumes the accumulator walk `_message_from_exc` and the is-self cycle check, which a first-match primitive could not). Migrated all five named walks plus two unguarded strays.-2. **Finding 1** — *kept* the guard rather than rewriting it: verification showed the `True` branch is reachable and both proposed rewrites regress it. Refactored the predicate onto the primitive and documented the invariant.-3. **Finding 6** — light touch: deduped the rationale prose, kept the two catch-alls separate (no shared helper — see followups Finding 4's resolution).-4. **Finding 9** — added the `True`-branch test and proved it meaningful via a deliberate guard-regression check. The sandboxed-smoke open question (above) was left as the pre-existing suite-wide gap it is.diff --git a/wip/error-handling/temporal-failsafe-review-followups.md b/wip/error-handling/temporal-failsafe-review-followups.mddeleted file mode 100644index 1b0e01781..000000000--- a/wip/error-handling/temporal-failsafe-review-followups.md+++ /dev/null@@ -1,155 +0,0 @@-# Temporal fail-safe — secondary review follow-ups--> **Gated on Temporal productionization — parked, not abandoned.** The silent-hang hole this branch closed is *done and shipped*; everything below is latent because Temporal is not in production (Findings 13/14 have no trigger until workflows actually run; C/D/F are pre-prod hardening; 3/12 are decisions, not bugs). **Do not execute piecemeal now** — run it as one focused pass when Temporal is scheduled to ship, with the system exercisable so the fixes are verifiable rather than speculative. The only "now" exception: fold the opportunistic one-liners (5, 8, 10) in *if* their file is touched for another reason.--**Status:** 🔎 open backlog. Lower-priority findings from the code review of the fail-safe floor that landed on `fix/Temporal-failsafe`. None blocks the merge. **Resolved in the guard-hardening session and removed from this doc:** Finding 4 (catch-all asymmetry — won't fix, harmless: a guarded branch in `WfPipeRun` would be dead code) and Finding 11 (rationale duplication — deduped onto `_carries_temporal_failure`'s docstring + `error-model.md`). Finding 12's comment overstatement was fixed there too; only its product question survives below. **Resolved in the review-agent triage session** (independently re-flagged by codex + cubic): Finding 2 — the inline conversions now force `non_retryable=True` (deterministic + terminal); see its resolution banner below. What remains open: a contract decision (Finding 3), small cleanups (5, 8, 10), and two findings added by a later `/review` pass — **Finding 13** (the `_carries_temporal_failure` predicate floors inline errors chained from a report-less `FailureError` — deepens the Finding 1 fragility analysis) and **Finding 14** (a pre-existing replay-determinism hazard: `build_search_attributes` reads config on the recorded child-start command).--**Companion docs:** the fix is described in [`temporal-error-handling-failsafe-gap.md`](./temporal-error-handling-failsafe-gap.md); the landed behavior in [`track-temporal-integration.md`](./track-temporal-integration.md). The four highest-value findings (the guard's correctness, consolidation, reuse, and missing test) are in [`temporal-failsafe-guard-hardening.md`](./temporal-failsafe-guard-hardening.md) — **read its Background section first**; this doc assumes that context.--**Finding numbers** (2, 3, 5, 8, 10, 12) are kept from the original review. Findings 4 and 11 were resolved and removed (see Status). **Findings 13 and 14** were added by a later `/review` pass (adversarial subagent + verification); they continue this doc's numeric scheme. (The gap doc owns the `A`–`F` letters for its fix-plan items, so there is deliberately no letter/number overlap across the two docs.)-------## Background (cold-start, condensed)--The fail-safe floor makes a pipelex **domain** error raised *inline in workflow code* (never via an activity) fail the workflow **terminally and classified** instead of hanging on indefinite workflow-task retry. Two layers:--- `WfPipeRouter.run()` and `WfPipeRun.run()` each end with an `except PipelexError` clause. The router converts a genuine inline error to a terminal `TemporalError` via `from_message_exception` and raises immediately. The parent (`WfPipeRun`) routes it through its deferred-delivery path so the FAILED webhook still fires, then re-raises terminally.-- The worker registers `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]` as a backstop.--`from_message_exception` (`pipelex/temporal/tprl/temporal_error.py:232`) builds the terminal `TemporalError`: it sets `error_type = exc.__class__.__name__`, packs `exc.to_error_report().to_dict()` into `ApplicationError.details`, and derives `non_retryable` from the inference error category (or the `non_retryable_error_types` name-list fallback). The full background and the `_carries_temporal_failure` guard are explained in the companion hardening doc.-------## Finding 2 — Retryable inline errors still re-run the inline leaf on retry (behavioral tradeoff + test gap)--> **✅ Resolved on `fix/Temporal-failsafe`** (review-agent triage; independently re-flagged by codex + cubic). The inline catch-alls now pass `force_non_retryable=True` to `from_message_exception` (the `wf_pipe_router.py` line-195 conversion and the `wf_pipe_run.py` inline carrier). Open question #1 is answered **"force `non_retryable=True`"**: an inline domain error fails the workflow terminally instead of re-running already-completed inline work on a blunt whole-workflow retry — retry belongs at the activity boundary, where it is scoped and observable. Forcing the flag also short-circuits the config-reading `_is_non_retryable`, so the workflow-side conversion is now **config-free / deterministic** — which also resolves the separate review concern that `from_message_exception` read `get_config()` inside workflow code (a replay-fragility risk). The report's own `retryable` field is config-free and preserved as informational metadata. Open question #3 (test the arm) is closed: `test_temporal_error_bridge.py::TestTemporalErrorBridge::test_force_non_retryable_is_deterministic_and_preserves_report_retryable` pins `non_retryable=True`, the preserved `report.retryable`, and that `_is_non_retryable` is not called when forced. Original analysis kept below for the record.--**Where:** `wf_pipe_router.py:195` (the conversion) → `from_message_exception` retryability logic in `temporal_error.py`.--**What:** `from_message_exception` sets `non_retryable=False` for a transient/uncategorized-not-listed inline domain error. A *retryable* `ApplicationError` raised in workflow code is then subject to the workflow's `RetryPolicy` — configured for top-level workflows via `WorkflowExecutorFactory`, and the child `WfPipeRouter` is dispatched **without** an explicit retry policy (deliberately, for replay determinism — see `wf_pipe_run.py:50-57`), so it inherits Temporal's default. The consequence: a retryable inline error re-runs `pipe.run_pipe` — **including any real provider/inference call it makes inline** — on each attempt, up to the execution timeout.--**Why it matters:** this is strictly *better* than the pre-fix outcome (an unbounded workflow-task-retry hang that also re-ran the inline work), so it is not a regression. But the fix does **not** make retryable inline errors fail fast — it trades an unbounded hang for a bounded-but-still-resource-burning execution retry. And the tests only exercise the **non-retryable** arm (`make_failing_llm_error()` is a non-retryable `CONFIGURATION`-class error), so the retryable path's behavior is unverified.--**Open questions:**--1. Is execution-level retry of an inline leaf acceptable, or should the inline catch-all force `non_retryable=True` regardless of category? Rationale for forcing: if an operator is running its leaf *inline* (rather than via an activity with its own `RetryPolicy`), retrying the whole workflow execution is a blunt, expensive instrument — the operator was supposed to dispatch an activity precisely so that retry is scoped and observable. Rationale against: a genuinely transient inline error (rare) would then fail instead of self-healing.-2. This overlaps with deferred follow-up **C** (a short submitter-side deadline for the synchronous API path) and **D** (CI guard that every inference operator routes its leaf through an activity) in [`temporal-error-handling-failsafe-gap.md`](./temporal-error-handling-failsafe-gap.md). If **D** lands, "retryable inline error" becomes a should-never-happen state and this question is moot. Decide whether to engineer #1 now or defer behind **D**.-3. Add a test for the retryable-inline arm regardless of the above (it is currently a blind spot).-------## Finding 3 — `SecurityError` is now routed through `except PipelexError` and the worker floor, contradicting its own contract--**Where:** the catch-alls (`wf_pipe_router.py:184`, `wf_pipe_run.py:83`) and the registration (`temporal_task_manager.py:147`). The contract: `pipelex/base_exceptions.py:567`.--**What:** `SecurityError`'s docstring states it is "kept distinct from domain errors so security signals are not silently swallowed by domain-level `except` handlers (e.g. `except PipelexError`)." `SecurityError` is a `PipelexError`, so the new `except PipelexError` clauses catch it and the `PipelexError` registration floors it — exactly the pattern the docstring warns against.--**Nuance:** the new code does **not** *silence* it — it re-raises it terminally with the type preserved in the report (`error_type` survives via `from_message_exception`). And making a security violation a terminal failure (rather than an indefinite retry) is arguably *correct*. So this may be a benign — even desirable — interaction. But it is the literal anti-pattern the contract calls out, and a maintainer relying on that contract would be surprised.--**Open questions:**--1. Should the catch-alls add an explicit `except SecurityError: raise` (or `except SecurityError` that converts but flags/audits differently) **before** the `except PipelexError`, to honor the contract literally?-2. Or do we update the `SecurityError` docstring/contract to acknowledge that the Temporal workflow boundary legitimately floors it to a terminal failure (because hanging on a security violation is worse)? If so, is there any place a `SecurityError` *should* get differentiated handling at the boundary (e.g. suppressed from the webhook payload, or routed to an audit sink)?-3. Are there other `PipelexError` subclasses with special "do not swallow" intent that the broad registration now captures? (Audit the hierarchy under `base_exceptions.py` for similar contracts.)-------## Finding 5 — The inline path drops the original exception from the worker-side cause chain--**Where:** `wf_pipe_run.py:99` — `execution_error.__cause__ = TemporalError.from_message_exception(exc=exc)`.--**What:** the minted `TemporalError` itself has `__cause__ = None` (`from_message_exception` returns `cls(...)` and never chains `from exc`). So after `execution_error.__cause__` is set to it, the **original** inline `exc` and its traceback are no longer reachable from `execution_error`. This is asymmetric with:--- the sibling child-failure path (`wf_pipe_run.py:81`: `execution_error.__cause__ = exc`, preserving the `ChildWorkflowError`), and-- the router catch-all (`wf_pipe_router.py:195`: `raise TemporalError.from_message_exception(exc=exc) from exc`, preserving `exc`).--**Why it matters:** low. The classification and message survive (they ride in the `TemporalError.details` report and are logged via `workflow_log.error(f"WfPipeRun inline failure: {exc}")`), and the submitter recovers the report fine. The loss is the original exception object / traceback / any deeper `__cause__` on the worker side — a debuggability and consistency gap, not a behavioral break.--**Open question:** preserve the original by chaining it — set the minted `TemporalError`'s `__cause__` to `exc` before assigning it to `execution_error`, giving `execution_error → TemporalError → exc`. Confirm this does not confuse `recover_error_report`'s walk (it stops at the first report-carrying `ApplicationError`, so a deeper `exc` is harmless). Standalone fix now — the Finding-6 shared helper that would have owned the chaining was not built (the catch-alls stayed separate; see the hardening doc's light-touch resolution).-------## Finding 8 — `exc.to_error_report()` is computed twice on the inline path (efficiency + drift risk)--**Where:** `wf_pipe_run.py:92` (`error_report = exc.to_error_report()`, for the webhook) and `:99` (`from_message_exception`, which internally recomputes `exc.to_error_report().to_dict()` at `temporal_error.py:243`).--**What:** `to_error_report()` is not trivial — it walks and enriches from the `__cause__` chain (`base_exceptions.py:482` + `_enrich_error_report_from_cause`). On every inline failure it is paid twice. Worse, the webhook gets the object from `:92` while the submitter gets the dict packed at `:99`; if the two derivations ever diverge, the webhook and submitter disagree on classification.--**Why it matters:** small wasted work on the failure path; the drift risk is the more interesting part (single-source-of-truth for the report).--**Proposed fix:** compute the report once and have `from_message_exception` accept a precomputed `ErrorReport` (or expose a variant taking the dict), so the webhook payload and the wire details come from one source.--**Open question:** does `from_message_exception`'s signature change ripple to the activity-side caller (`activity_error_boundary.py:52`)? Prefer an additive optional `error_report=` parameter that defaults to recomputing, so the activity path is unaffected.-------## Finding 10 — `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]` is redundant--**Where:** `temporal_task_manager.py:147`.--**What:** `WorkflowExecutionError` is a subclass of `PipelexError` (via `TemporalFlowError`), so the second entry fully subsumes the first. The list is redundant. The previously-misleading comment was already trimmed in the guard-hardening session — it now says `WorkflowExecutionError` is "listed explicitly for intent" rather than claiming it is separately load-bearing — so what remains is purely the cosmetic list decision below.--**Why it matters:** harmless at runtime; a maintenance trap. A future reader who trusts the comment might narrow the broad `PipelexError` entry believing `WorkflowExecutionError` still covers the child-wrap case — when in fact removing `PipelexError` silently reopens the floor for every non-`WorkflowExecutionError` domain error, and removing `WorkflowExecutionError` changes nothing.--**Open question:** drop to `[PipelexError]` and simplify the comment, **or** keep `WorkflowExecutionError` explicit purely for documentation (signaling the originally-intended terminal type) with a comment that says so? Either is defensible; pick one and make the comment match reality.-------## Finding 12 — Should a terminal failure with no delivery target still notify?--**Where:** `wf_pipe_run.py:89` (the webhook comment) and `:146` (`if delivery_assignment is not None:`).--**Comment half — done.** The guard-hardening trim dropped the "a terminal failure must always notify the receiver" overstatement; the comment now reads "fires the FAILED webhook on the failure path," which is accurate. Only the product question below remains.--**What:** when `delivery_assignment is None`, Step 3 is skipped (`wf_pipe_run.py:146`) and no webhook fires. This matches direct-mode `PipeRun` semantics (no target ⇒ no delivery), so the current behavior is intentional. The only test always supplies a webhook target, so the no-assignment branch is unexercised.--**Open question:** is "no target ⇒ no notification" the right product contract, or should a terminal failure always notify *somewhere* even without a `delivery_assignment` (cf. the memory note that a run's terminal failure webhook must always fire)? If the latter, the gap is in the code, not the comment. Current lean: keep as-is (parity with direct mode); revisit only if a "must-always-notify" product invariant is adopted.-------## Finding 13 — `_carries_temporal_failure` floors inline errors chained from a non-`ApplicationError` `FailureError` (deepens Finding 1)--> **Provenance:** surfaced by a `/review` adversarial pass and verified against the live Temporal SDK hierarchy. Latent today, not a merge blocker. Extends [`temporal-failsafe-guard-hardening.md`](./temporal-failsafe-guard-hardening.md) Finding 1 with a third fragility mode its analysis did not consider.--**Where:** `wf_pipe_router.py:56` (`_carries_temporal_failure`), gating the conversion at `:198`.--**What:** the predicate is `any(isinstance(node, FailureError) for node in iter_cause_chain(exc))`. Verified: `TimeoutError`, `CancelledError`, `TerminatedError`, `ActivityError`, and `ChildWorkflowError` are **all** `FailureError` subclasses but **not** `ApplicationError` — and only `ApplicationError` carries the `details` payload where the structured `ErrorReport` rides. So an inline `PipelexError` chained `from` a bare Temporal failure that carries no recoverable report — e.g. pipe code that awaits an activity, the activity times out, and the code does `raise SomeDomainError(...) from timeout_exc` — returns `True` and is **propagated untouched**. At the submitter, `recover_error_report` walks `__cause__` for a report-bearing `ApplicationError`, finds none (a `TimeoutError`/`CancelledError` has no `details`), and synthesizes an `UnrecoverableWorkflowFailureError` — **dropping the domain error's own `error_type` / message** that `from_message_exception` (the convert branch) would have preserved. The guard's docstring assumes the chain's `FailureError` is a report-carrier ("the rich leaf classification lives deeper, in the child's `ApplicationError.details`"); that assumption is false for non-`ApplicationError` `FailureError`s.--**Why it matters:** this is the exact classification-fidelity loss the error-handling project exists to prevent — surfacing neither the timeout nor the domain error, but a generic synthesized report. **Latent today:** the one real inline-leaf path (`content_generator_in_workflow.py`) raises `TemporalError.from_app_error(...)` — an `ApplicationError`-with-report — so it is classified correctly, and genuine inline errors (Search-missing-activity, crate-load, hydration) carry no `FailureError` and convert correctly. The gap opens the moment any operator catches a Temporal timeout/cancel and re-raises a *plain* `PipelexError` `from` it inline.--**Why the obvious fix regresses (and what to do instead):** tightening the predicate to "carries a recoverable report" (`isinstance(node, ApplicationError) and error_report_dict_from_details(node.details) is not None`) **breaks the reachable case** Finding 1 protects: the controller-sub-pipe-as-child-workflow failure escapes as `WorkflowExecutionError` → `ChildWorkflowError`, whose leaf report lives in `ChildWorkflowError.cause` (recovered only by the submitter's `.cause`-normalizing walk), **not** in `__cause__` — so a worker-side `__cause__`-only report check would not find it and would wrongly convert. This is third confirmation that the broad-vs-narrow predicate is fundamentally fragile, and it strengthens the case for **Finding 1's deferred option (b)**: stop using the guard as the load-bearing decision — instead always mint a `TemporalError` but have `from_message_exception` *prefer an already-present report in the chain* over re-deriving, so flattening is impossible regardless of the `from`/`__cause__`/`.cause` shape.--**Open questions:**--1. Adopt Finding 1 option (b) (report-preserving conversion, guard demoted)? That subsumes both this finding and Finding 1's original two modes.-2. Until then, is it worth a narrow guard tweak that treats a chain whose only `FailureError`s are report-less (`TimeoutError`/`CancelledError`/`TerminatedError`) as convert-eligible, while still propagating `ChildWorkflowError`/report-bearing `ApplicationError`? (More surface, partial fix — option (b) is cleaner.)-3. Add a `True`-branch test variant where the chain carries a report-less `FailureError`, asserting the domain error's own classification survives (it currently does **not**).-------## Finding 14 — `build_search_attributes` reads `get_config()` on the recorded child-start command (replay-determinism hazard, pre-existing)--> **Provenance:** surfaced by a `/review` adversarial pass; verified that `build_search_attributes` reads config. **Pre-existing — not introduced by this branch** (`pipelex/temporal/tprl/observability.py` is untouched here). Filed here per the repo's "flag pre-existing bugs" principle because it directly undercuts the fail-safe's anti-hang purpose. Better tracked in a dedicated Temporal wip doc if one is opened.--**Where:** `observability.py:104` (`config = get_config().temporal.search_attributes`) feeding `wf_pipe_run.py:63` and `temporal_pipe_router.py` (`search_attributes=build_search_attributes(pipe_job)` as an argument to `execute_child_workflow`).--**What:** the `search_attributes` value is part of the recorded `StartChildWorkflowExecution` command. Because `build_search_attributes` derives it from `get_config().temporal.search_attributes` (`.enabled`, `.attributes`) — process/deploy state, not workflow input — editing that config while a workflow is live would make a later replay re-derive a *different* attribute set on the child-start command → a Temporal non-determinism mismatch. This contradicts the same file's load-bearing comment (`wf_pipe_run.py:50-57`), which deliberately keeps the child-dispatch command "a pure function of the workflow input" precisely so "replay after a config edit" cannot diverge — the comment enumerates `execution_timeout` / `retry_policy` / `task_queue` but overlooks that `search_attributes` is on the same command and is config-derived. The function's docstring calls itself pure because it "reads everything off `pipe_job`," which is inaccurate.--**Why it matters:** a config-edit-during-live-workflow → stuck/failed replay is the exact silent-hang class the fail-safe exists to eliminate, sitting unguarded on the dispatch path. **Latent:** narrow trigger (a long-running workflow surviving a worker redeploy with edited search-attribute config) and Temporal is not yet in production.--**Open questions:**--1. Stamp the resolved search attributes onto `pipe_job`/`PipeRunArg` at submit time (submitter-side, before the durable boundary) so the workflow reads them off its input instead of `get_config()` — matching how the file already handles `execution_timeout` / `retry_policy` / `task_queue`.-2. Or confirm (with a versioning/patch strategy) that search-attribute config is frozen for a workflow's lifetime, and correct the `build_search_attributes` docstring's "pure" claim either way.-3. Does `build_static_summary` (also on the command) have any config dependency? (Spot-checked: no `get_config()` read — appears input-only, but confirm.)-------## Suggested handling--- **Finding 2** is resolved (see its banner — inline conversions forced non-retryable + deterministic). **Finding 3** remains a genuine decision — surface it before the next Temporal release; it may be answered by the deferred **C/D** follow-ups in the gap doc.-- **Findings 5 and 8** are cheap standalone cleanups (the Finding-6 shared helper that would have absorbed them was not built — see the hardening doc's light-touch resolution). Best folded in opportunistically the next time `wf_pipe_run.py` is touched; **5** is the higher-value of the two.-- **Finding 10** is a one-line cleanup (drop the redundant entry, or keep it for intent) — batch it whenever `temporal_task_manager.py` is next edited. **Finding 12**'s comment is already fixed; only its product question lingers.-- **Finding 13** is the higher-value of the two new ones: it converts Finding 1 from "resolved — kept the broad predicate" to "the broad predicate has a third, unhandled failure mode," and points at Finding 1 option (b) (report-preserving conversion) as the resolution that subsumes all of it. Decide 13 and Finding 1 together, before the next Temporal release. Latent today, so not urgent — but it is a correctness gap, not a cosmetic one.-- **Finding 14** is pre-existing and out of this branch's scope; fold it into a dedicated Temporal wip doc (or fix it at submit time per its open question 1) when `observability.py` / the child-dispatch path is next touched.diff --git a/wip/error-handling/track-cli-delivery.md b/wip/error-handling/track-cli-delivery.mddeleted file mode 100644index f616c287c..000000000--- a/wip/error-handling/track-cli-delivery.md+++ /dev/null@@ -1,101 +0,0 @@-# Track — CLI Delivery--## What this track is--How errors are rendered to the two CLI audiences: humans through the main `pipelex` CLI (Rich panels) and agents through `pipelex-agent` (structured output, JSON or markdown). The track also covers the success-path delivery for the agent CLI, because the error path inherits the format option.--Both delivery paths are landed. Human Rich delivery and agent JSON/markdown delivery both consume `to_error_report()`, the agent CLI defaults to markdown with `--format json` available, and the Rich handlers share one panel helper.--## Current state--### Human CLI — Rich--`pipelex/cli/error_handlers.py` defines per-error-type handler functions, each building its panel through the shared `display_error_panel(console, *, title, fields, error_message, tip, links)` helper:--```python-def display_error_panel(- console: Console,- *,- title: str,- fields: list[tuple[str, str]], # (label, escaped value)- error_message: str | None, # None omits the error body line- tip: str,- links: list[tuple[str, str]], # (label, url)-) -> None:- ...-```--Each `handle_*` function constructs the field list and calls the helper; the exception-specific logic (`if exc.fallback_list:`, `if exc.enabled_backends:`) stays in the handler, but the panel shape — red banner, structured fields, `report.user_action` tip, doc/Discord links, `raise typer.Exit(1) from exc` — lives in one place. Handlers exist for `PipeOperatorModelChoiceError`, `PipeOperatorModelAvailabilityError`, `ModelDeckPresetValidatonError`, `ValidateBundleError`, `InferenceSetupRequiredError`, `TelemetryConfigValidationError`, the gateway-config family, and the remote-config family.--`ErrorContext` (a `StrEnum`) describes where the error happened: `PIPE_RUN`, `VALIDATION`, `BUILD`, plus pre-validation contexts.--### Agent CLI — JSON and markdown--`pipelex/cli/agent_cli/commands/agent_output.py` defines:--- `CliOutputFormat` (`StrEnum`): `JSON`, `MARKDOWN`.-- A module-level `ContextVar` (`_agent_cli_error_format`, default `JSON`) with `set_agent_cli_error_format()` / `get_agent_cli_error_format()` — backs the **error** stream only. Each `--format`-aware command sets it at entry via `set_agent_cli_error_format(error_format or output_format)`, so every downstream `agent_error()` call honors it without threading the format through — including error sites that never see the Typer option (factory init failures in `agent_cli_factory.py`, the unknown-command handler, app-callback validation). The **success** stream is threaded explicitly via an `output_format` argument, not the ContextVar.-- `agent_error(message, error_type, cause, **extra)` — dispatches on the active error format (the `_agent_cli_error_format` ContextVar): JSON to stderr via `_agent_error_json`, or markdown to stderr via `agent_error_markdown`. Both `raise typer.Exit(1) from cause`.-- `agent_success(result)` — JSON success envelope to stdout (also drains any pending setup warnings into a `warnings` array). `agent_success_formatted(result, markdown_renderer, output_format)` wraps it for `--format`-aware commands: JSON via `agent_success`, or markdown via `markdown_renderer(result)`, dispatched on the explicit `output_format` argument.-- `extract_validation_errors(exc: ValidateBundleError)` — flattens blueprint / factory / validation / instantiation errors into a list per-category. This is the most structured error delivery in the codebase and remains the reference shape.--The JSON error payload:--```json-{- "error": true,- "error_type": "LLMCompletionError",- "message": "...",- "hint": "...",- "retryable": true,- "error_domain": "runtime",- "error_category": "transient",- "model": "gpt-4o",- "provider": "openai",- "error_source": ["LLMCompletionError @ .../worker.py:152 (in _gen_text)"]-}-```--`agent_error()` reads `cause.to_error_report()` when `cause` is a `PipelexError` and pulls `user_action`, `retryable`, `error_category`, `error_domain`, `model`, `provider` from the report. The string-keyed dicts (`AGENT_ERROR_HINTS`, `AGENT_ERROR_DOMAINS`, `RETRYABLE_ERROR_TYPES`) are consulted only as a fallback when the report has no value — see [track-metadata-model.md](track-metadata-model.md) for the migration of dict entries onto the classes.--`_build_error_source(exc)` walks the `__cause__` chain and emits `"ExceptionType @ filename:line (in function)"`, prioritizing frames inside `/pipelex/` over `.venv/` frames.--### Agent CLI — format coverage--`--format markdown|json` (success, stdout) is wired on `run`, `validate`, `init`, `models`, `doctor`, and `check-model`, with **markdown as the default**. A companion `--error-format markdown|json` (errors, stderr) is accepted by the same commands and, when omitted, inherits `--format`'s value — it is the option backed by the `_agent_cli_error_format` ContextVar. `inputs` / `concept` / `pipe` / `fmt` / `lint` / `accept-gateway-terms` have neither and stay JSON / raw passthrough. The contract is documented in `pipelex/cli/agent_cli/CLAUDE.md`.--`InferenceSetupRequiredError` is a markdown-by-default special case independent of `--format`: it renders setup-guidance to stdout with exit 0 (used by agent skills on first run).--### Validation error special case--`ValidateBundleError` (defined in `pipelex/pipeline/validate_bundle.py`) aggregates four error sources:--1. Blueprint validation errors (from interpreter).-2. Pipe factory errors (e.g. missing concepts).-3. Pipe validation errors (e.g. missing inputs, type mismatches).-4. Pipe / concept instantiation errors (pydantic validation during factory).--`extract_validation_errors` returns a flat list of dicts, each with at minimum `category`, `error_type`, and `message`, plus contextual fields per category. The Rich path uses `_display_validation_error_details` in `pipelex/cli/error_handlers.py` to render the same structure as styled blocks.--## HTTP-status mapping (authoritative)--`pipelex` is a library — there is no API server in the package. But the HTTP API repos (`pipelex-relay`, `pipelex-back-office`) must render an `ErrorReport` as an HTTP response, and the `error_domain` → status mapping was being reinvented per repo. The mapping lives in the library, in `pipelex/base_exceptions.py`:--- `error_domain_to_http_status(error_domain)` — the pure domain → status table: `INPUT` → 422, `CONFIG`/`RUNTIME` → 500, `None` (or an unrecognized string) → 500.-- `ErrorReport.http_status` — the full property: a provider 429 (`provider_metadata.status_code == 429`) takes precedence so the API can emit a `Retry-After` header from `provider_metadata.retry_after_seconds`; otherwise it follows `error_domain`.--The library stays HTTP-agnostic — no web-framework import, only the mapping table. Downstream FastAPI exception handlers call `ErrorReport.http_status` and are a trivial adapter; they must not redefine the contract.--## Open gaps--Delivery itself is landed — the Rich handlers and the agent JSON/markdown path faithfully consume `to_error_report()`. The Temporal hole in that data source is now closed: the submitter-side recovery (`recover_error_report` → `WorkflowExecutionError(error_report=...)`, owned by [track-temporal-integration.md](track-temporal-integration.md)) rebuilds the structured report from `ApplicationError.details`, so a worker-side failure now reaches delivery with the same `error_category` / `retryable` / `model` / `provider` / `user_action` classification as a local run. The fix lived entirely in the Temporal bridge — delivery and `pipelex/cli/agent_cli/` stay Temporal-agnostic.--The recovered report is carried on `WorkflowExecutionError` as an optional attribute, with a `to_error_report()` override — option (a) from the eng review. Option (b) (a `RemotePipelexError` carrier in the `__cause__` chain) was rejected: (a) keeps `raise WorkflowExecutionError(msg) from exc` so the Temporal `WorkflowFailureError` stays in the traceback for free, and the override mirrors the 3-line pattern `PipelineExecutionError.to_error_report()` already uses.--The remaining delivery-adjacent work is the metadata migration tracked in [track-metadata-model.md](track-metadata-model.md): while delivery reads `to_error_report()` first, several `PipelexError` subclasses still depend on the fallback dicts because they carry no class-level `error_domain` / `user_action`.--## Related tracks--- [track-metadata-model.md](track-metadata-model.md) — `to_error_report()` is the canonical data source for both delivery paths; the dict fallback it discusses is the metadata that feeds these renderers.-- [track-temporal-integration.md](track-temporal-integration.md) — owns the Temporal error bridge; its submitter-side recovery is what keeps delivery's `to_error_report()` data fully classified for a Temporal-run pipe.-- [track-testing.md](track-testing.md) — the full-chain integration snapshot and the Rich-panel snapshot tests that cover this track.diff --git a/wip/error-handling/track-delivery-error-path-request-id.md b/wip/error-handling/track-delivery-error-path-request-id.mddeleted file mode 100644index f7d6e7749..000000000--- a/wip/error-handling/track-delivery-error-path-request-id.md+++ /dev/null@@ -1,95 +0,0 @@-# track: request_id correlation on delivery error paths--**Status:** open. Decision: TBD.--**TL;DR:** Earlier work threaded `request_id` through `DeliveryExecutor` and into the **success-path** completion log lines (Temporal + direct-mode dispatchers). The **failure-path** log/error messages in the same executor still don't carry the field — same correlation gap the feature was meant to close, just on the unhappy path. Three lines in `delivery_executor.py`, plus one debug log in `pipe_run.py`.--## Context--A review's "Delivery logs lose request" comment was scoped to "the storage/webhook completion logs don't carry the inbound `X-Request-ID`." We addressed it twice:--- Temporal path — `DeliveryActivityArg.request_id` plumbed through to `DeliveryExecutor.execute` and into the **two success log lines** in `_store_results` and `_notify_webhook`.-- Direct mode — `PipeRun.run` forwards `pipe_job.job_metadata.request_id` to `DeliveryExecutor.execute` so non-Temporal dispatchers don't lose correlation.--Both commits stopped at the success-path log lines. The failure-path messages — which is exactly where operators most want the correlation — still emit `pipeline_run_id=...` without `request_id`.--## The open gap--Three error-message constructions in `pipelex/pipe_run/delivery_executor.py`, and one debug log in `pipelex/pipe_run/pipe_run.py`:--| Site | Current shape (no request_id) | Trigger |-|---|---|---|-| `delivery_executor.py:243` | `f"Storage delivery failed for pipeline_run_id={pipeline_run_id}"` | Any exception inside `_store_results` (S3 write, key generation, serialization) |-| `delivery_executor.py:282` | `f"Webhook delivery failed for pipeline_run_id={pipeline_run_id}: HTTP {exc.response.status_code}"` | Webhook returns 4xx/5xx |-| `delivery_executor.py:285` | `f"Webhook delivery failed for pipeline_run_id={pipeline_run_id}: {exc}"` | Network-level failure (`httpx.RequestError`) |-| `pipe_run.py:79` | `f"Executing delivery for pipeline_run_id={pipeline_run_id}, status={status}"` (debug-level entry log) | Every delivery, both modes |--All four lines have `request_id` (or could have it) in local scope — `_store_results` and `_notify_webhook` already accept the kwarg; `pipe_run.py:79` runs in a function where `pipe_job.job_metadata.request_id` is in scope.--## Case for closing it--- **Completeness of the stated feature.** The point of threading `request_id` was operator correlation across the delivery phase. Operators most need that correlation when something fails — failed webhooks are exactly when they grep logs for the inbound request id. Leaving the failure messages uncorrelated defeats half the value.-- **Consistency.** Success-path log lines carry the field; failure-path messages don't. Any operator who reads logs will notice the asymmetry and learn the wrong heuristic ("request_id is on success lines only — guess I have to correlate by pipeline_run_id when things break").-- **Cheap.** The exception-path messages already have `pipeline_run_id` interpolated; adding the same conditional suffix that the success path uses is mechanical. No new arguments to thread (the kwarg is already in scope at all three executor sites). Test changes are small.-- **The feature isn't done until this is done.** The review's #3 finding is still open — closing it with "we did half" invites a re-flag.--## Case for leaving it--- **Out of strict scope.** Greptile's comment quoted the `completion log` lines specifically. Failure-path error messages were not in their report. The discipline we just enforced on fix #2 (match the reviewer's scope, no scope creep) argues for parsimony.-- **The failure already raises a typed exception.** `WebhookDeliveryError` / `StorageDeliveryError` propagate; the receiving code in `pipe_run.py:89-96` and the Temporal activity boundary both log the failure with their own (already-bound) context. The local `msg=` on the exception is one of several ways the failure surfaces — operators have other paths to the correlation.-- **Severity.** No data loss, no incorrect behavior. Worst case: operators have one extra hop to correlate (look up the pipeline_run_id → request_id mapping elsewhere). Compared to the original bug (delivery completion logs uncorrelated at all), the marginal value is smaller.-- **Pre-existing.** These four lines were as they are before this PR. Fixing them is feature-extension, not regression-repair. Could ride a separate, narrower PR.-- **Altitude flag.** The previous code review noted that per-line manual threading doesn't scale; a request-bound contextvar or structured-log adapter would close all these lines at once without three more manual edits. If we're going to fix the failure-path lines, the question of "do it the same manual way OR do the higher-altitude fix" reopens.--## My read--**Close the three executor error-path lines in a small follow-up commit; defer the `pipe_run.py:79` debug log and the altitude refactor.**--Rationale:--- The three executor lines are the operator-visible asymmetry. Closing them removes the "request_id is only on success" footgun and makes the feature contract symmetric.-- They cost almost nothing — the `request_id_suffix = f", request_id={request_id}" if request_id is not None else ""` helper line already exists in both `_store_results` and `_notify_webhook` for the success log; reusing the same variable on the failure message is two characters of insertion per site (plus passing through to `_store_results`'s raise — already in scope there).-- The debug entry-point log in `pipe_run.py:79` is debug-level and lives in the dispatcher, not the executor. Lower value, different file. Drop from this follow-up.-- The altitude refactor (contextvar) is the right long-term answer but is a separate, larger change. Threading the field once more is acceptable when the fix is mechanical and the long-term answer is still on the open list.--## If we do it--Single commit scoped to `pipelex/pipe_run/delivery_executor.py`:--- `_store_results` line 243 — append the existing `request_id_suffix` variable to the error message. (Already constructed at line 238 in the same function.)-- `_notify_webhook` lines 282 and 285 — both error-message constructions. Lift `request_id_suffix` (currently defined at line 279 after the success branch) to the top of the function so both the success and both failure messages can reuse it.-- One new test in `tests/unit/pipelex/pipe_run/test_delivery_executor.py` pinning that the storage-failure and webhook-failure exception messages carry `request_id=...` when set. Mirror the existing success-path test scaffolding (`mocker.spy(pipelex_log, "info")` won't help here since the field is on the **exception** `msg`, not a log line — assert on `str(exc)` from the raised exception inside `pytest.raises(WebhookDeliveryError) as exc_info`).--Sketch commit message:--```-fix(delivery): include request_id in DeliveryExecutor failure messages--Closes the asymmetric correlation gap left by the success-path threading:-success-path completion logs carry request_id, but the matching-exception messages on storage/webhook failure paths still did not.-Operators correlating failed deliveries to inbound X-Request-ID had-to fall back to pipeline_run_id. Three error messages in-delivery_executor.py now carry the same conditional ", request_id={...}"-suffix the success lines use.--Deferred: the pipe_run.py:79 debug entry-point log and the altitude-refactor to a request-bound contextvar.-```--## Decision criteria--Push toward **close now** if:-- The review thread on the "Delivery logs lose request" comment is still open and we don't want to leave it for a re-flag.-- Operators have already raised an issue about asymmetric correlation in delivery failures.-- Cost of the fix stays as small as sketched (single-file, single-commit, one regression test).--Push toward **defer** if:-- We're racing a merge and any further change widens the audit surface.-- We want to land the altitude refactor (contextvar / structured-log adapter) first and not pay the per-line cost twice.-- The review thread is already closed and a separate PR for the failure-path is acceptable.--## References--- The success-path threading in `DeliveryExecutor` (Temporal + direct-mode) — the prior work this builds on.-- The code-review error-path findings that surfaced this gap.diff --git a/wip/error-handling/track-extract-classify-render.md b/wip/error-handling/track-extract-classify-render.mddeleted file mode 100644index fd3e8c319..000000000--- a/wip/error-handling/track-extract-classify-render.md+++ /dev/null@@ -1,40 +0,0 @@-# Extract / Classify / Render — as built--**Status: landed.** Every inference worker's `except` block resolves an SDK exception through one shared three-step chain — **extract → classify → render** — instead of hand-coding metadata extraction, categorization, and error construction inline. Only the Extract step is per-provider; Classify and Render are single shared functions.--## How it works--The chain lives in `pipelex/cogt/inference/`:--- **Extract** (per-provider) — `extract_<provider>_metadata(exc) -> ProviderErrorMetadata` in `error_classification.py`, one function per provider. It pulls the structured fields out of the SDK exception — status code, request id, retry-after, provider error code, message, body — into a `ProviderErrorMetadata` (aliased `SDKErrorEnvelope`). This is the only place provider-specific attribute access lives.-- **Classify** (shared, pure) — `classify_inference_error(metadata: SDKErrorEnvelope) -> ClassificationResult` in `error_classify.py`. Provider-agnostic: it maps the envelope — status code, the computed `@property` flags `is_quota_exhaustion` / `is_content_policy_violation` / `is_network_error`, and a status-less-exception-name table — to a `ClassificationResult` carrying `category: InferenceErrorCategory`, `user_action_kind: UserActionKind`, and `is_model_not_found: bool`. Because it reads only the envelope, it is tested against synthetic envelopes with no SDK installed.-- **Render** (shared) — `render_inference_error(metadata, classification, family, model_desc, model_handle) -> CogtError` in `error_render.py`. It builds the concrete `CogtError` subclass selected by the worker `family` (`InferenceErrorFamily`: `LLM` / `IMG_GEN` / `EXTRACT` / `SEARCH`), composes the message and `UserAction`, and attaches the metadata.--A worker `except` block is three lines:--```python-except anthropic.APIError as exc:- metadata = extract_anthropic_metadata(exc)- classification = classify_inference_error(metadata)- raise render_inference_error(- metadata, classification, InferenceErrorFamily.LLM, self.inference_model.desc, model_handle,- ) from exc-```--## What this buys--- Adding a provider — or a new SDK exception type — is an Extract-only change; Classify and Render are untouched.-- Classification rules live in one provider-agnostic function, tested exhaustively against synthetic envelopes (no SDK imports).-- Human-facing message formatting lives in one Render function rather than being duplicated per worker.--## Adding a provider--Write one `extract_<provider>_metadata` function in `error_classification.py` and wire it at the worker's `except` site. `tests/unit/pipelex/cogt/inference/test_provider_classification_parity.py` walks every `ProviderName` against the extract-function registry, so a provider added without a wired Extract function fails fast.--## Related tracks--- [track-worker-classification.md](track-worker-classification.md) — where SDK exceptions are caught; the per-worker classification this chain consolidated.-- [track-metadata-model.md](track-metadata-model.md) — the `ProviderErrorMetadata` / `error_category` / `user_action` data contract this chain produces.-- [track-retry-and-resilience.md](track-retry-and-resilience.md) — consumes the uniform `error_category` and `retry_after_seconds` for retry decisions.-- [track-testing.md](track-testing.md) — the provider-blind Classify tests and the cross-provider parity meta-test.-- [track-cli-delivery.md](track-cli-delivery.md) — Render is the single place human-facing message formatting is shaped.diff --git a/wip/error-handling/track-metadata-model.md b/wip/error-handling/track-metadata-model.mddeleted file mode 100644index 76569258c..000000000--- a/wip/error-handling/track-metadata-model.md+++ /dev/null@@ -1,84 +0,0 @@-# Track — Error Metadata Model--## What this track is--The contract for **what every exception carries** so it can be rendered identically to humans, agents, and Temporal: `error_type`, `message`, `title`, `type_uri`, `error_category` (TRANSIENT / CONFIGURATION / CONTENT / CAPACITY / AMBIGUOUS / UNKNOWN), `error_domain` (input / config / runtime), `user_action`, `retryable`, optional `model`, `provider`, and `provider_metadata`. The metadata lives on the exception class itself so there is a single source of truth.--The model is **landed for inference errors and the major pipeline/interpreter/service exceptions** — they self-describe via class-level attributes consumed through `to_error_report()`. The remaining work is closing the long tail: a handful of `CogtError` subclasses still carry no class-level `error_category`, and several `PipelexError` subclasses still depend on the fallback dicts in `agent_output.py` rather than carrying their own `error_domain` / `user_action`.--## Current state--### `ErrorReport` — the serialization schema--`ErrorReport` (`pipelex/base_exceptions.py`) is a frozen `BaseModel` (`ConfigDict(frozen=True, extra="forbid", arbitrary_types_allowed=True)`):--```-ErrorReport- error_type: str- message: str- title: str- type_uri: str- error_category: str | None- error_domain: str | None- retryable: bool | None- user_action: UserAction | None- model: str | None- provider: str | None- provider_metadata: ProviderErrorMetadata | None- caller_facing_message: bool-```--`to_dict(disclosure_mode=VERBOSE)` returns the non-`None` fields and round-trips through `from_dict`; `STRICT` is a lossy projection for untrusted external surfaces (see [architecture.md](architecture.md) for `DisclosureMode`). `http_status` maps the report to an HTTP status for downstream API adapters (see [track-cli-delivery.md](track-cli-delivery.md)).--`PipelexError.to_error_report()` returns `error_type`, `message`, `title`, `type_uri`, the class-level `error_domain`, `user_action`, and `caller_facing_message`. `CogtError.to_error_report()` overrides it to layer on `error_category`, `retryable` (derived from category via `InferenceErrorCategory.is_retryable`), `provider_metadata`, and the `model` / `provider` it reads from the instance's `model_handle` / `backend_name` when present. `to_error_report()` also enriches from the `__cause__` chain, so a wrapper exception surfaces the inference metadata of the underlying `CogtError` (`title` / `type_uri` stay wrapper-wins).--### Inference categories--`InferenceErrorCategory` (`pipelex/cogt/exceptions.py`) is a `StrEnum` with values `TRANSIENT`, `CONFIGURATION`, `CONTENT`, `CAPACITY`, `AMBIGUOUS`, `UNKNOWN` and an `is_retryable` property — `True` only for `TRANSIENT`. Implemented with `match/case` per the project's enum style. `AMBIGUOUS` and `UNKNOWN` are both non-retryable but distinct: `AMBIGUOUS` means the error type is known yet the outcome is not (e.g. a connection dropped mid-request on a non-idempotent call, so a blind retry is unsafe), while `UNKNOWN` means classification failed outright.--### Error domains--`ErrorDomain` (`StrEnum`) has values `INPUT`, `CONFIG`, `RUNTIME`. `PipelexError` declares an optional class-level `error_domain: ErrorDomain | None`. Class-level domains are set on:--- `PipelexSetupError`, `PipelexConfigError` (`pipelex/base_exceptions.py`) → `CONFIG`.-- `PipeExecutionError` (`pipelex/pipeline/exceptions.py`) → `RUNTIME`; `PipelineExecutionError` injects `RUNTIME` + a `user_action` in its `to_error_report()` override.-- `ValidateBundleError` (`pipelex/pipeline/validate_bundle.py`) → `INPUT` + a `user_action`.-- `PipelexInterpreterError` (`pipelex/core/interpreter/exceptions.py`) → `INPUT`.-- `PipelexServiceError` and its subclasses (`pipelex/system/pipelex_service/exceptions.py`) → `CONFIG`.--### Inference class-level defaults--Many `CogtError` subclasses declare `error_category` as a class attribute:--- Routing / backend / model-deck families default to `CONFIGURATION` (e.g. `RoutingProfileDisabledBackendError`, `InferenceBackendCredentialsError`, `LLMSettingsValidationError`, `ModelNotFoundError` and its subclasses).-- Handle-not-found family (`LLMHandleNotFoundError`, etc.), capability errors, choice errors, `LLMConfigError`, `SdkTypeError`, `InferenceBackendLibraryError` — all categorized.-- The prompt / parameter family (`LLMPromptSpecError`, `LLMPromptTemplateInputsError`, `LLMPromptParameterError`, `PromptImageFactoryError`, `PromptImageFormatError`, `PromptDocumentFactoryError`, `ImgGenPromptError`, `ImgGenParameterError`, `ImageContentError`) → `CONTENT`.--`InferenceBackendCredentialsError` also declares a class-level `user_action` — the canonical pattern.--The four "outcome" exceptions stay **intentionally** uncategorized at the class level and take their category per-instance from the worker that raised them: `LLMCompletionError`, `ImgGenGenerationError`, `ExtractJobFailureError`, `SearchJobFailureError`.--### How delivery consumes this--`agent_error()` (`pipelex/cli/agent_cli/commands/agent_output.py`) calls `cause.to_error_report()` when `cause` is a `PipelexError` and reads `user_action`, `retryable`, `error_category`, `error_domain`, `model`, `provider` from the report **first**; the string-keyed dicts (`AGENT_ERROR_HINTS`, `AGENT_ERROR_DOMAINS`, `RETRYABLE_ERROR_TYPES`) are a fallback only. The Rich handlers in `error_handlers.py` likewise build panels from `report.user_action`. The `agent_output.py` dicts are guarded by a drift-detection test (see [track-testing.md](track-testing.md)).--## Open gaps--These are the genuine remainder — the metadata model is otherwise landed.--- **A few `CogtError` subclasses still have no class-level `error_category`.** `CostRegistryError`, `ReportingManagerError`, `ExtractOutputError`, `GeneratedImageError`, `LLMAssignmentError`, and parts of the routing / model-deck family (`RoutingProfileLibraryError`, `ModelManagerError`, `ModelDeckNotFoundError`, `ModelDeckValidationError`, and similar). When raised without an instance-level override they produce an `error_category`-less report. Decide a default per class.-- **Several `PipelexError` subclasses still depend on the fallback dicts.** Types caught by the agent CLI — `PipeOperatorModelChoiceError`, `ModelDeckPresetValidatonError`, the gateway-config family, `PipelexInterpreterError`'s hint, etc. — have their `hint` (and sometimes `error_domain`) only in `AGENT_ERROR_HINTS` / `AGENT_ERROR_DOMAINS`, not as class-level `user_action` / `error_domain`. The migration moves those onto the classes; the dicts then keep only built-in / third-party types (`FileNotFoundError`, `JSONDecodeError`, `ValidationError`, …) that can't carry attributes.--## Followups--The work below closes the two gaps. Either can be tackled independently.--1. **Finish class-level `error_category` on the uncategorized `CogtError` subclasses** listed above. Leave the four outcome exceptions (`LLMCompletionError` / `ImgGenGenerationError` / `ExtractJobFailureError` / `SearchJobFailureError`) as `None` — workers set them per-instance.-2. **Migrate the remaining `PipelexError`-keyed dict entries onto the classes** as class-level `user_action` (from `AGENT_ERROR_HINTS`) and `error_domain` (from `AGENT_ERROR_DOMAINS`). Keep dict entries only for non-`PipelexError` types. The drift-detection test ([track-testing.md](track-testing.md)) already enforces that every `PipelexError` subclass is covered one way or the other.--## Related tracks--- [track-worker-classification.md](track-worker-classification.md) — where per-instance `error_category` and `user_action` are set on the four outcome exceptions.-- [track-cli-delivery.md](track-cli-delivery.md) — how the human Rich handlers and the agent JSON/markdown path consume `to_error_report()`.-- [track-testing.md](track-testing.md) — the drift-detection test that keeps the dicts and the class-level metadata in sync.-- [track-temporal-integration.md](track-temporal-integration.md) — packs `to_error_report()` into `ApplicationError.details`.diff --git a/wip/error-handling/track-retry-and-resilience.md b/wip/error-handling/track-retry-and-resilience.mddeleted file mode 100644index f57d3a840..000000000--- a/wip/error-handling/track-retry-and-resilience.md+++ /dev/null@@ -1,64 +0,0 @@-# Track — Retry and Resilience--## What this track is--Where retry legitimately lives in Pipelex, and how the direct (non-Temporal) path stays honest about being the non-resilient one.--Resilience is Temporal's job. Pipelex integrates Temporal precisely so that durability, redelivery, and retry-under-failure are handled by a system built for it. Direct execution does **not** reproduce that — it makes one pipeline-level attempt, on top of a transport that already shrugs off brief blips. The value of the direct path is simplicity and zero infrastructure, not resilience.--This track is landed. The state below is what the code does today.--## Decisions--- **No application-level transient-retry loop in direct execution.** There is no `PipeRouter` retry loop. Direct execution makes a single pipeline-level attempt. (An earlier loop existed; it only ever ran on the direct path, re-ran at the wrong granularity, and carried a per-run/global config bug — it was removed rather than fixed for a path that is deliberately not the resilient one.)--- **Tier 1 — transport retry — is the provider SDKs' own retry, configured by Pipelex.** The provider SDKs retry transient transport failures (connection errors / 408 / 409 / 429 / 5xx, honoring `Retry-After`). Pipelex makes that an explicit, uniform, configured policy via `cogt.transport_max_retries` instead of inheriting each SDK's silent default — and extends the same floor to the workers that do not ride a retrying SDK. It does **not** add a Pipelex retry layer on top, which would duplicate well-tested SDK behavior.--- **`instructor`'s structured-output retry is confined to schema re-ask.** `instructor` wraps the factory-built SDK client, so Tier 1's configured retry already reaches structured calls. `instructor`'s *own* retry is given a validation-only predicate, so it re-asks the model on a malformed-output failure but does **not** re-run the completion on transport errors — Tier 1 is the sole transport-retry layer.--- **Bounded fan-out stays.** `PipeBatch`'s `gather_bounded` / `max_concurrency` is admission control, not retry — it stops a large batch from causing a self-inflicted rate-limit storm. It stays as-is.--## The model — where retry lives--| Layer | What it does | Scope | Owned by |-|---|---|---|---|-| **Tier 1 — transport retry** | Retries connection errors / 408 / 409 / 429 / 5xx, honoring `Retry-After`; bounded (`cogt.transport_max_retries`, default 2) | inside the SDK / HTTP client; both paths | the provider SDKs, configured by Pipelex |-| `instructor` structured-output retry | Re-asks the model on schema-validation failure only | structured-output path; both paths | `instructor`, configured by Pipelex |-| **Tier 2 — Temporal durability** | Activity retry keyed off `InferenceErrorCategory.is_retryable`; workflow-level durability and redelivery | Temporal path only | platform (worker / queue config) |-| Bounded fan-out | Caps simultaneous branches in `PipeBatch` so a large batch does not storm the provider | both paths | platform (`max_concurrency`) |--Between Tier 1 and Tier 2 there is nothing on the direct path, by design. Direct execution retries transient *transport* failures (via the SDK) and then surfaces the error — it does not retry at the pipeline level, and it does not survive a crash. That is the honest contract.--## Current state--**Tier 1 — transport retry.** `cogt.transport_max_retries` (default 2) is wired explicitly into every inference SDK client factory under `pipelex/plugins/*/` — Anthropic, OpenAI / Azure OpenAI, the Portkey-backed gateway OpenAI clients, Mistral, and Google. The two families that defaulted to *no* transport retry are brought up to the floor: the Mistral client is built with a bounded-backoff `RetryConfig` (`retry_connection_errors=True`), the Google `genai` client with `HttpOptions(retry_options=...)`. The genuinely SDK-less path — the raw-`httpx` `azure_rest` image-gen worker — uses a `tenacity`-based transport-retry wrapper (`pipelex/cogt/inference/transport_retry.py`) that retries connection failures and transient HTTP statuses and honors `Retry-After`; for a non-idempotent submit-style POST it narrows the retry to failures that prove the request did no work (declining an ambiguous 5xx, a 409, and a post-delivery timeout). FAL rides the `fal_client` SDK, which has its own internal transport retry; the `portkey-ai` gateway SDK exposes no `max_retries` knob and carries its own internal retry — both are left as-is. `fal_poller.py`'s `tenacity` is job-status *polling*, not error-retry, and is also left as-is.--**`instructor` structured-output retry.** Each `instructor` call site passes a `tenacity.AsyncRetrying` built by `pipelex/cogt/llm/instructor_retry.py`, whose predicate matches only validation failures (`pydantic.ValidationError`, `json.JSONDecodeError`, and `instructor`'s own validation-error types). A transport error therefore propagates immediately as the raw SDK exception rather than being wrapped in `InstructorRetryException`. Consequence handled: the Google and Mistral structured workers carry an `httpx.TransportError` `except` clause, since those SDKs let raw connection / timeout errors propagate outside their own exception hierarchies.--**Tier 2 — Temporal durability.** `TemporalError.from_message_exception()` sets `non_retryable = not InferenceErrorCategory.is_retryable`; every in-scope activity is wrapped by `@convert_pipelex_errors`; `RetryPolicyConfig` composes baseline + per-queue + per-handle `non_retryable_error_types`. See [track-temporal-integration.md](track-temporal-integration.md).--**Bounded fan-out.** `gather_bounded` in `pipelex/tools/misc/async_utils.py`, `max_concurrency` on `PipelineExecutionConfig`, consumed by `PipeBatch`.--## The honest contract--- **Direct execution** — retries transient *transport* failures via the provider SDK (bounded, `Retry-After` honored), then surfaces the error. Structured-output calls additionally re-ask the model on schema-validation failure (`instructor`) — output-shaping, not resilience. No pipeline-level retry, no durability, no crash survival. Intended, not a gap.-- **Temporal execution** — the same transport retry, plus Tier 2 (activity `RetryPolicy` + workflow durability) for retry-under-failure and crash survival. The resilient path.-- The product pitch stays clean: *direct for simplicity, Temporal for resilience* — with no half-measure in between pretending otherwise.--## Open gaps--None for this track. The one open-ended item is a review (not a change) of whether Temporal's Tier 2 defaults — which categories retry, attempt caps, backoff — are right for the hosted product.--## Out of scope — dependencies, not part of this track--These are real and belong to the hosted product, but they are platform/product concerns, not the engine's retry behavior. They are tracked in the internal enterprise-readiness plans.--- **Multi-tenant admission control and per-tenant quotas / rate limits.** Tier 1 obeys a rate limit *after* hitting it; proactive pacing keyed to a provider account is separate and platform-global, not per-run.-- **Caller-supplied run deadline / budget** — a ceiling on total time or spend that bounds every tier. Belongs with the API submission envelope.-- **Idempotency model** — the prerequisite for any future "re-submit the whole run" feature.-- **Circuit breaking** on a provider that is down.--## Related tracks and docs--- [track-temporal-integration.md](track-temporal-integration.md) — Tier 2: how `InferenceErrorCategory` drives Temporal's retry decision.-- [track-metadata-model.md](track-metadata-model.md) — `ProviderErrorMetadata` carries `retry_after_seconds` for the error report; the SDK does its own `Retry-After` handling for retries.diff --git a/wip/error-handling/track-temporal-integration.md b/wip/error-handling/track-temporal-integration.mddeleted file mode 100644index f5dc2ad97..000000000--- a/wip/error-handling/track-temporal-integration.md+++ /dev/null@@ -1,75 +0,0 @@-# Track — Temporal Integration--## What this track is--How Pipelex errors cross the Temporal boundary: activities raise Pipelex exceptions, the bridge converts them to `TemporalError(ApplicationError)`, and Temporal's retry policy uses the error to decide whether to retry. Temporal's retry decision flows from the same `InferenceErrorCategory.is_retryable` signal that classification produces — not from a static class-name list.--The bridge is landed end-to-end: the conversion is category-aware, the full `ErrorReport` travels in `ApplicationError.details`, every in-scope activity routes through the conversion decorator, and the symmetric workflow → submitter recovery reads that report back out of `details` once the failure returns to the process that submitted the workflow. See "Submitter-side report recovery (landed)" below.--## Current state--### TemporalError--`pipelex/temporal/tprl/temporal_error.py` defines `TemporalError(ApplicationError)`. Two things travel with the error across the activity → workflow boundary:--- **`non_retryable`** — for a `CogtError` carrying an `InferenceErrorCategory`, the flag is derived from `category.is_retryable` (`non_retryable = not is_retryable`). For category-less exceptions the bridge falls back to the configured `non_retryable_error_types` class-name list — the union of the worker, per-queue, and per-handle levels (`all_non_retryable_error_types` in `pipelex/temporal/config_temporal.py`).-- **`error_report`** — `exc.to_error_report().to_dict()` is packed into `ApplicationError.details`, so workflow code keeps `error_category`, `error_domain`, `user_action`, `model`, and `provider` rather than just the message string.--`from_message_exception` converts a `PipelexError` raised inside an activity; `from_app_error` re-wraps an `ApplicationError` observed in workflow code, recovering the `non_retryable` flag and the details payload the activity-side bridge set. `_log_critical` / `_log_error` select `activity_log` vs `workflow_log` via `activity.in_activity()`, since the two entry points run on opposite sides of the boundary (`workflow.logger` raises outside a workflow event loop).--### Activity error boundary--A shared decorator `convert_pipelex_errors` (`pipelex/temporal/tprl/activity_error_boundary.py`) is applied beneath `@activity.defn` on every in-scope activity:--```python-@convert_pipelex_errors-async def act_llm_gen_text(llm_assignment: LLMAssignment) -> str:- return await llm_gen_text(llm_assignment=llm_assignment)-```--The decorator catches `PipelexError` only (never the generic `Exception`) and re-raises `TemporalError.from_message_exception(exc=exc) from exc`. Wired on `act_llm_gen_text` / `act_llm_gen_object` / `act_llm_gen_object_list`, `act_img_gen_images`, `act_extract_gen_extract_pages`, `act_jinja2_gen_text`, `act_render_page_views`, `act_deliver`, `act_flush_trace_events`. Deliberately **not** wired on `act_assemble_graph` — it is best-effort observability that swallows every failure and degrades to `None`, so no error crosses its boundary.--### Retry policy config--`pipelex/temporal/config_temporal.py` defines `RetryPolicyConfig` with `non_retryable_error_types: list[str]` (baseline list) and `non_retryable_error_types_extra: list[str]` (additive overrides per queue / per handle). `RetryPolicyConfig.make_retry_policy(merged_non_retryable_types)` builds the Temporal `RetryPolicy`. The `all_non_retryable_error_types` helper unions the three layers for the fallback retry decision (category-less exceptions) and the matching log severity. The composition is **additive** across worker → queue → handle layers.--The class-name list is documented as a **fallback only**: a `CogtError` carrying an `InferenceErrorCategory` is decided by category; the list applies to non-`CogtError` `PipelexError` subclasses, a `CogtError` raised without a category, and per-queue overrides.--## Submitter-side report recovery (landed)--The bridge moves the `ErrorReport` activity → workflow, and `from_app_error` recovers it for workflow code. The symmetric workflow → submitter recovery — reading the report back out of `ApplicationError.details` once the failure returns to the process that submitted the workflow — is now landed:--- `recover_error_report` (`pipelex/temporal/tprl/temporal_error.py`) walks the failure's `__cause__` chain for an `ApplicationError`, pulls the details-packed report dict via the now-public `error_report_dict_from_details`, and rebuilds an `ErrorReport` through `ErrorReport.from_dict` (the strict inverse of `to_dict` in VERBOSE mode). It is **total** — it always returns an `ErrorReport`, never `None`. When the chain carries no report payload (a non-Pipelex exception, a worker crash, a heartbeat timeout) it synthesizes an `UnrecoverableWorkflowFailureError` report (`error_domain=RUNTIME`) carrying the most informative message recoverable from the chain. `from_dict` is `extra="forbid"`, so a found-but-malformed payload is an internal contract bug within one deploy (the activity bridge and submitter share the schema); rather than raise — which would abort the caller before it delivers the failure webhook — it synthesizes the same `UnrecoverableWorkflowFailureError` fallback, tagged with an `[error report failed schema validation]` marker so the contract bug stays visible on the wire. The error path therefore never crashes and the receiver is always notified.-- `WorkflowExecutor.execute_workflow` (`pipelex/temporal/tprl/workflow_caller.py`) splits the combined handler into a dedicated `except WorkflowFailureError` clause that recovers the report and a sibling `except (WorkflowAlreadyStartedError, RPCError)` clause that stays generic. The recovered report is carried on `WorkflowExecutionError(error_report=...)`, and the original failure message replaces the generic `"Failed to execute workflow ..."`.-- `WorkflowExecutionError.to_error_report()` returns the recovered report when present; with no report it falls through to base `__cause__`-chain enrichment. `PipelineExecutionError` then inherits the classification natively, since `WorkflowExecutionError` is a `PipelexError`.--Net effect: a pipe failure on a Temporal worker now reaches every `to_error_report()` consumer — agent CLI JSON/markdown, the Rich human CLI, the `ErrorReport.http_status` mapping for HTTP adapters — with the same `error_category` / `retryable` / `model` / `provider` / `user_action` classification as the identical failure run locally.--**Coverage:** beyond the unit tests that pin `recover_error_report` and `WorkflowExecutionError` against synthetic failures, an integration parity pair exercises the full chain: `tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py` runs a real `WfPipeRouter` workflow on the in-process Temporal server through `WorkflowExecutor.execute_workflow` with the LLM call mocked to fail — the genuine activity → workflow → submitter serialization round-trip — and `tests/integration/pipelex/error_handling/test_error_report_local_full_chain.py` runs the same pipe locally. Both assert the same `ErrorReportParityTestData` constants, so local/Temporal `ErrorReport` parity is locked.--**Design:** `WorkflowExecutionError` holds the `ErrorReport` as an optional attribute and overrides `to_error_report()` — option (a). This keeps `raise WorkflowExecutionError(msg) from exc` so the Temporal `WorkflowFailureError` stays in the traceback for free. Rejected option (b) (a `RemotePipelexError` carrier in the `__cause__` chain) for forcing a new public exception class plus manual `__cause__` wiring.--### Child-workflow boundary recovery (landed)--The child-workflow boundary (`execute_child_workflow` / `start_child_workflow` in `pipelex/temporal/tprl/workflow_caller.py`) now applies the same recovery. `ChildWorkflowError` exposes the deserialized failure via `.cause`; when that cause is an `ApplicationError`, the handler runs `recover_error_report(exc.cause)` and, on a recovered report, raises `WorkflowExecutionError(error_report.message, error_report=...)` — mirroring `execute_workflow`. A cause with no report payload, malformed details, or a non-`ApplicationError` cause all fall back to the prior generic error. These wrappers are not on Pipelex's traced execution paths (in-workflow child spawns call `workflow.execute_child_workflow` directly to stay replay-deterministic), so this closes a latent gap on the public surface.--### Workflow-level fail-safe floor (landed)--The bridge above converts every error that crosses an *activity* boundary. The remaining gap was an error raised **inline in workflow code** that never went through an activity — neither an `ActivityError` nor an `ApplicationError`, so Temporal treated it as a *workflow-task* failure and retried it indefinitely (a silent hang surfaced only as a generic timeout after the execution timeout). The floor closes that:--- `WfPipeRouter.run()` and `WfPipeRun.run()` each end with an `except PipelexError` boundary. `WfPipeRouter` converts a genuine inline error to a terminal `TemporalError` via `from_message_exception`; `WfPipeRun` routes it through deferred delivery (FAILED webhook still fires) and chains a details-carrying `TemporalError` so the submitter stays classified. Both convert only *genuine* inline errors — an escaping `PipelexError` already carrying a Temporal failure in its `__cause__` chain (e.g. a nested `TemporalPipeRouter` child-dispatch wrapping a sub-pipe failure as `WorkflowExecutionError`) is already terminal and recoverable, so it propagates untouched (`_carries_temporal_failure` guard) rather than being flattened.-- The worker now registers `workflow_failure_exception_types=[WorkflowExecutionError, PipelexError]` (`pipelex/temporal/temporal_task_manager.py`). Anything that slips past the catch-alls still fails terminally (degrading to a synthesized `UnrecoverableWorkflowFailureError`) instead of hanging. Scoped to `PipelexError`, not `Exception`: transient Temporal/infra errors keep Temporal's default task-retry.--**Coverage:** `tests/integration/pipelex/temporal/test_workflow_inline_error_failsafe.py` pins the escape path across the router catch-all, the `WfPipeRun` inline-delivery path, and the worker-level floor, each bounded by a short execution timeout so a regression fails fast instead of hanging. The systemic analysis and deferred hardening (sync-API deadline, activity-dispatch CI guard, stuck-workflow observability) live in [`temporal-error-handling-failsafe-gap.md`](./temporal-error-handling-failsafe-gap.md).--### Deferred review followups--Two optional refinements from the activity-boundary review (tracked in the deferred backlog) remain deliberate scope choices: the integration test verifies the converted payload rather than Temporal's retry-engine behavior, and `error_report_dict_from_details` identifies the report by dict shape. Optional to revisit.--## Related tracks--- [track-retry-and-resilience.md](track-retry-and-resilience.md) — this is Tier 2; direct execution has no pipeline-level retry tier below it.-- [track-worker-classification.md](track-worker-classification.md) — every inference worker classifies before this bridge is meaningful.-- [track-metadata-model.md](track-metadata-model.md) — the `error_category` / `user_action` / `model` / `provider` on `to_error_report()` are what the details payload carries.-- [track-cli-delivery.md](track-cli-delivery.md) — the delivery surfaces consume `to_error_report()`; with submitter-side recovery now landed, Temporal-run pipes preserve the same structured error data path as local runs.diff --git a/wip/error-handling/track-testing.md b/wip/error-handling/track-testing.mddeleted file mode 100644index ba24c9261..000000000--- a/wip/error-handling/track-testing.md+++ /dev/null@@ -1,44 +0,0 @@-# Track — Testing--## What this track is--The test coverage that proves the error-handling system behaves as specified end-to-end, and the drift-detection that keeps the metadata-on-classes contract honest as the codebase grows.--Coverage is landed at every level: per-worker classification, the classification helpers, the `instructor` unwrap, `to_error_report()` shape, the full-chain runner→CLI→JSON snapshot, the local/Temporal `ErrorReport` parity pair, the agent-CLI dict drift-detection, and the Rich-panel snapshots.--## Current state--### Inference layer--- **Per-worker classification** — test modules under `tests/unit/pipelex/plugins/<provider>/` mock the SDK to raise each typed exception and assert the resulting domain exception has the expected `error_category`, `user_action`, `provider_metadata`, and model descriptor. Each LLM worker also has a raw-SDK-transport-error case pinning the post-`instructor`-confinement behavior (a transport error propagates raw and classifies correctly).-- **Classification helpers** — parametrized tests in `tests/unit/pipelex/cogt/inference/` cover the message-pattern discriminators (`is_quota_exhaustion_*`, `is_content_policy_violation`) for every provider.-- **`instructor` unwrap** — each provider has one end-to-end test against real `instructor.from_<provider>(...)` (e.g. `tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_object_error_handling.py`) so the unwrap can't silently rot if `instructor`'s wrapping shape changes; the remaining categorization cases use the synthetic `wrap_in_instructor_retry` helper in `tests/helpers/instructor_test_utils.py`.-- **`instructor` schema-retry helper** — `tests/unit/pipelex/cogt/llm/test_instructor_retry.py` asserts a transport-style exception is not retried (the predicate excludes it) and a `pydantic.ValidationError` is retried up to `stop_after_attempt`.-- **Transport retry** — `tests/unit/pipelex/cogt/inference/test_transport_retry.py` covers the `tenacity`-based SDK-less wrapper; `tests/unit/pipelex/plugins/test_transport_retry_wiring.py` asserts each SDK client factory builds its client with the configured `transport_max_retries`.--### Reporting and serialization--- **`to_error_report()` shape** — unit tests in `tests/unit/pipelex/cogt/exceptions/` confirm the report includes the right fields per `CogtError` subclass, that `to_dict()` drops `None` fields, and that JSON serialization round-trips cleanly.--### Full-chain and delivery--- **Full-chain runner→CLI→JSON snapshot** — `tests/integration/pipelex/cli/agent_cli/test_run_error_chain.py` builds a minimal pipeline where one pipe fails with a deterministic worker error, runs the agent CLI, and asserts the emitted JSON carries `error_category`, `retryable`, `model`, `provider`, and an `error_source` chain in the expected order (worker → pipe operator → router → runner → CLI). This is the test that catches a wrapper exception silently swallowing `error_category`, `agent_error()` regressing on forwarded fields, or `error_category` arriving as a Python repr instead of the StrEnum value.-- **Drift-detection for `agent_output.py` dicts** — `tests/unit/pipelex/cli/test_agent_output_drift.py` walks the `PipelexError` hierarchy and asserts every key in `AGENT_ERROR_HINTS` / `AGENT_ERROR_DOMAINS` / `RETRYABLE_ERROR_TYPES` corresponds to a real exception class, and that a `PipelexError` subclass is either covered by class-level metadata or has a fallback dict entry. This closes the "silent breakage when classes are renamed" failure mode.-- **Rich-panel snapshots** — `tests/unit/pipelex/cli/test_error_handlers_snapshot.py` snapshots representative `display_error_panel` outputs so the shared-helper refactor can't drift the rendered layout.--### Temporal boundary--- `tests/unit/pipelex/temporal/test_temporal_error_bridge.py` pins `from_message_exception`'s retry-flag derivation from `InferenceErrorCategory`; `tests/integration/pipelex/temporal/test_activity_error_boundary.py` drives a real `CogtError` from a real activity through a real worker and asserts what `from_app_error` receives on the workflow side.-- **Submitter-boundary recovery** — `tests/unit/pipelex/temporal/test_workflow_caller_error_recovery.py` and `test_workflow_caller_child_error_recovery.py` pin `recover_error_report` and `WorkflowExecutionError` against synthetic `WorkflowFailureError` / `ChildWorkflowError` inputs (G1–G7 edge cases: malformed details, version skew, no-report fallback).-- **Full-chain local/Temporal parity** — a parity pair runs the same `native_text_sequence` pipe with the LLM call mocked to fail and asserts the resulting `ErrorReport` carries the same `error_category` / `retryable` / `model` / `provider` / `user_action` whether run locally or via Temporal. The Temporal arm (`tests/integration/pipelex/temporal/test_workflow_error_report_full_chain.py`) runs a real `WfPipeRouter` workflow on the in-process Temporal server through `WorkflowExecutor.execute_workflow`, exercising the genuine activity → workflow → submitter serialization round-trip the unit tests only fake; the local arm (`tests/integration/pipelex/error_handling/test_error_report_local_full_chain.py`) runs the same pipe through the local `PipeRouter`. Both assert the shared `ErrorReportParityTestData` constants, so parity holds by construction.--## Open gaps--None load-bearing. Two optional refinements (tracked in the deferred backlog) are deliberate scope choices: the Temporal integration tests verify the converted payload rather than Temporal's retry-engine behavior, and a couple of search/extract worker tests omit non-status `APIError` subtypes.--## Related tracks--- [track-metadata-model.md](track-metadata-model.md) — the drift-detection test enforces the class-level metadata contract.-- [track-cli-delivery.md](track-cli-delivery.md) — the full-chain snapshot and Rich-panel snapshots exercise the delivery paths.-- [track-worker-classification.md](track-worker-classification.md) — the shared `instructor` test utilities live in `tests/helpers/instructor_test_utils.py`.-- [track-temporal-integration.md](track-temporal-integration.md) — the activity → workflow boundary tests.diff --git a/wip/error-handling/track-worker-classification.md b/wip/error-handling/track-worker-classification.mddeleted file mode 100644index dbb28d4c3..000000000--- a/wip/error-handling/track-worker-classification.md+++ /dev/null@@ -1,109 +0,0 @@-# Track — Worker Classification--## What this track is--Every inference worker that calls a third-party SDK must catch the SDK's typed exceptions and re-raise a `CogtError` subclass with an `InferenceErrorCategory`, a `user_action` hint, the model descriptor in the message, and `from exc` to preserve the cause chain. This is the foundation that everything downstream depends on — retry policy, agent hints, Temporal `non_retryable` decisions, human Rich panels.--The work to lift every worker to this standard **has landed across all worker kinds** (LLM, img-gen, extract, search), including the four LLM workers that previously mis-categorized `instructor`-wrapped errors. OpenAI Completions, OpenAI Responses, Mistral, and Google all now unwrap `InstructorRetryException` and dispatch through the same categorization helper their `_gen_text` paths use. Every worker additionally carries the beyond-reference upgrades: `InferenceErrorCategory.UNKNOWN` for unrecognized-underlying fallbacks (instead of mis-categorizing as `CONTENT`), structured `ProviderErrorMetadata` (status_code, request_id, retry_after_seconds, provider_error_code, body) on every raised inference error, and structured `UserAction(kind, detail)` with semantic `UserActionKind` values.--## Current state--### What every worker does--Each provider worker under `pipelex/plugins/*/` follows the same shape:--```python-except RateLimitError as exc:- if is_quota_exhaustion_<provider>(str(exc)):- msg = f"{provider} quota exhausted for model '{self.inference_model.desc}': {exc}"- raise LLMCompletionError(- message=msg,- error_category=InferenceErrorCategory.CAPACITY,- user_action="Your {provider} account has exceeded its quota — check your billing dashboard at {billing_url}",- ) from exc- msg = f"Rate limited by {provider} for model '{self.inference_model.desc}': {exc}"- raise LLMCompletionError(- message=msg,- error_category=InferenceErrorCategory.TRANSIENT,- user_action=f"Rate limited by {provider} — the system will retry automatically",- ) from exc-```--The discriminator helpers are pure functions in `pipelex/cogt/inference/error_classification.py`:--- `is_quota_exhaustion_openai`-- `is_quota_exhaustion_anthropic`-- `is_quota_exhaustion_google`-- `is_quota_exhaustion_mistral(error_message, status_code)` — HTTP 402 is a definitive signal; 429 requires message inspection.-- `is_quota_exhaustion_aws`-- `is_quota_exhaustion_gateway(error_message, status_code)` — same 402 / 429 split.-- `is_content_policy_violation`--Per-provider message patterns are module constants alongside the helpers (`_OPENAI_QUOTA_PATTERNS`, `_ANTHROPIC_QUOTA_PATTERNS`, `_GOOGLE_QUOTA_PATTERNS`, `_MISTRAL_QUOTA_PATTERNS`, `_AWS_QUOTA_PATTERNS`, `_GATEWAY_QUOTA_PATTERNS`, `_CONTENT_POLICY_PATTERNS`).--### Per-worker coverage--All workers below catch their SDK's typed exceptions and assign `error_category`. Quota-vs-rate-limit and content-policy-vs-bad-request discriminators run where applicable.--| Worker | File |-|---|---|-| OpenAI Completions LLM | `pipelex/plugins/openai/openai_completions_llm_worker.py` |-| OpenAI Responses LLM | `pipelex/plugins/openai/openai_responses_llm_worker.py` |-| OpenAI img-gen | `pipelex/plugins/openai/openai_img_gen_worker.py` |-| OpenAI Completions img-gen | `pipelex/plugins/openai/openai_completions_img_gen_worker.py` |-| Anthropic LLM | `pipelex/plugins/anthropic/anthropic_llm_worker.py` |-| Google LLM | `pipelex/plugins/google/google_llm_worker.py` |-| Google img-gen | `pipelex/plugins/google/google_img_gen_worker.py` |-| Mistral LLM | `pipelex/plugins/mistral/mistral_llm_worker.py` |-| Mistral extract | `pipelex/plugins/mistral/mistral_extract_worker.py` |-| AWS Bedrock LLM | `pipelex/plugins/bedrock/bedrock_llm_worker.py` |-| Azure img-gen | `pipelex/plugins/azure_rest/azure_img_gen_worker.py` |-| FAL img-gen | `pipelex/plugins/fal/fal_img_gen_worker.py` |-| HuggingFace img-gen | `pipelex/plugins/huggingface/huggingface_img_gen_worker.py` |-| Docling extract | `pipelex/plugins/docling/docling_extract_worker.py` |-| Linkup extract | `pipelex/plugins/linkup/linkup_extract_worker.py` |-| Linkup search | `pipelex/plugins/linkup/linkup_search_worker.py` |-| Gateway extract | `pipelex/plugins/gateway/gateway_extract_worker.py` |-| Gateway img-gen | `pipelex/plugins/gateway/gateway_img_gen_worker.py` |-| Gateway search | `pipelex/plugins/gateway/gateway_search_worker.py` |-| pypdfium2 | `pipelex/plugins/pypdfium2/pypdfium2_worker.py` |--Each provides at minimum: catch SDK errors, classify, assign category, attach model descriptor, chain via `from exc`. The four dynamic-category exception types (`LLMCompletionError`, `ImgGenGenerationError`, `ExtractJobFailureError`, `SearchJobFailureError`) carry `error_category` per instance from the worker that raised them.--### Class hierarchy snapshot--`CogtError` (`pipelex/cogt/exceptions.py`) is the inference branch root. Two notable subclasses with structured fields:--- `ModelChoiceNotFoundError` carries `model_type`, `model_choice`, `reference_kind`, `available_options`, `suggestions`, `wrong_sigil_hints`, `cross_collection_suggestions`. Builds a "Did you mean: …" message automatically.-- `InferenceBackendCredentialsError` carries `credentials_error_type`, `backend_name`, `key_name`, and declares class-level `error_category = CONFIGURATION` and `user_action = "Check that the required API key environment variable is set"`.--`ModelNotFoundError` is the parent of `LLMModelNotFoundError`, `ImgGenModelNotFoundError`, and `ModelWaterfallError`. The waterfall variant adds `fallback_list`.--### The `instructor` unwrap--`extract_underlying_sdk_exception(instructor_exc)` (shared, in `pipelex/cogt/inference/error_classification.py`) recovers the SDK exception from `failed_attempts[-1].exception`, falling back to `__cause__.last_attempt._exception` (tenacity's storage). Each LLM worker's `_gen_object` path catches `InstructorRetryException`, unwraps it through that helper, and routes the result through the same per-provider categorization helper its `_gen_text` path uses; only truly unrecognized inner exceptions keep the `UNKNOWN` fallback.--Note the interaction with the retry track: since `instructor`'s retry predicate is now confined to schema re-ask ([track-retry-and-resilience.md](track-retry-and-resilience.md)), a transport error is no longer wrapped in `InstructorRetryException` — it propagates as the raw SDK exception and is caught by the worker's SDK-exception `except` clauses directly. `InstructorRetryException` now carries only genuine schema-validation failures. The end-to-end test `tests/unit/pipelex/plugins/anthropic/test_anthropic_worker_object_error_handling.py::test_real_instructor_propagates_transport_error_raw` locks that behavior in against real `instructor.from_anthropic(...)`.--## Open gaps--The classification + beyond-reference upgrades have landed across every worker kind — LLM, img-gen, extract, and search (plus AWS Bedrock LLM). The only residual note:--- **`pydantic.ValidationError` now routes to `UNKNOWN`.** When the LLM returns JSON that doesn't match the schema, `instructor` raises `InstructorRetryException` with a `ValidationError` (or `JSONDecodeError`) at `failed_attempts[-1].exception`. The unwrap branch returns it, the per-provider SDK categorization helpers don't recognize it, and the `UNKNOWN` fallback (introduced by Phase 2's upgrade A) catches it — distinguishing schema-mismatch from genuine `CONTENT`-policy violations. If we ever want a dedicated SCHEMA_MISMATCH category, the categorization helpers can be extended to recognize `pydantic.ValidationError` explicitly. Not a regression.--## What's left--Worker classification itself is complete — there is no remaining per-worker work. The deeper **Extract / Classify / Render** refactor that consolidates the now-uniform per-worker pipeline into one per-provider Extract function plus a shared Classify + Render has also landed — see [track-extract-classify-render.md](track-extract-classify-render.md). The consistency gap noted earlier (LLM workers using `classify_*_sdk_error()` free functions while img-gen workers used inline `_raise_categorized_*` methods) is largely closed: classification is unified behind the shared `classify_inference_error()` (`pipelex/cogt/inference/error_classify.py`) and the per-provider `*_error_classification.py` files are gone. One inline categorizer survives by design: the Azure REST img-gen worker keeps `_raise_categorized_azure_status_error` (`azure_img_gen_worker.py:65`, called from its generate path at `:159`). It is not a wholesale bypass — 4xx statuses route through the shared `classify_inference_error` + `render_inference_error` — but it forces 5xx statuses to `AMBIGUOUS` rather than the classifier's verdict, because image generation is a non-idempotent POST and the operation-agnostic classifier would mark a 5xx `TRANSIENT`, letting the Temporal bridge auto-retry and duplicate a billed generation.--### Risks and gotchas (for future similar work)--- The categorization helpers expect a raw SDK exception, not the wrapped one. The unwrap step in each worker preserves this contract.-- Chain via `from instructor_exc` so the traceback shows `instructor → tenacity → SDK`. Don't chain from the bare SDK exc — it loses retry context useful for debugging.-- Real-instructor + AsyncMock'd SDK tests are slightly slow (~0.5s each) because `instructor` still does its full attempt loop. Each provider has one such test (kept as an end-to-end lock-in); the remaining categorization cases use the synthetic `wrap_in_instructor_retry` helper in `tests/helpers/instructor_test_utils.py`.--## Related tracks--- [track-metadata-model.md](track-metadata-model.md) — the `error_category` / `user_action` contract the classified errors fill in.-- [track-retry-and-resilience.md](track-retry-and-resilience.md) — once workers correctly classify TRANSIENT vs CAPACITY, the router-level retry can act on those signals.-- [track-cli-delivery.md](track-cli-delivery.md) — how Rich panels and agent JSON render the classified errors.-- [track-temporal-integration.md](track-temporal-integration.md) — uses `is_retryable` to drive Temporal's `non_retryable` flag.diff --git a/wip/executive.md b/wip/executive.mddeleted file mode 100644index 728462a05..000000000--- a/wip/executive.md+++ /dev/null@@ -1,39 +0,0 @@-# Executive Priorities--This is the top-level decision sheet for the current `wip/` tree. It intentionally favors what to do next over full historical context.--## Strategic Priorities--1. **Finish the plugin/orchestrator architecture.**- The plugin track is the highest-leverage architecture work: it removes core vendor coupling, gives Temporal and MISTRAL_NATIVE a proper seam, and makes installed integrations discoverable instead of hard-coded. Close the execution-mode/delivery split, version-gating semantics, Temporal extraction, and downstream MISTRAL_NATIVE reconciliation before expanding the plugin surface further.--2. **Productionize distributed execution.**- P0/P1 tracing and cross-worker cost reporting shipped, so the remaining value is operational: local cross-package dependencies in the crate, remote dependencies from GitHub, nondeterminism decisions, request-scoped tracing state, and the split-worker test coverage gaps. This is what makes worker pools stateless enough to scale.--3. **Close the API/error-handling security tail.**- Webhook body signing is the clearest security item. After that, finish the structured error long tail: class-level metadata for remaining errors, request ids on delivery failures, and the parked Temporal fail-safe review backlog when Temporal productionization resumes.--4. **Make batch execution harder to overwhelm.**- The concurrency work is not urgent until batch-at-scale pain returns, but the diagnosis is solid: fan-out scheduling and partial failure should be decided together; rate limiting is separate and needs mode-specific design for direct vs Temporal.--5. **Improve validation and graph ergonomics for tools.**- The next useful tooling lifts are explicit validation graph targets, registry source paths in `GraphSpec`, parse-level source attribution for malformed TOML, and eventually stuffs as first-class `NodeSpec`s.--## Quick Wins--- **Add `pipelex init --yes`.** The handoff is scoped and mostly plumbing: short-circuit existing prompts to defaults, avoid silently accepting Gateway terms, and unblock downstream CI without vendored `.pipelex/`.-- **Thread `request_id` into delivery failure logs.** Small error-handling tail item; closes an observability asymmetry on webhook/storage failure paths.-- **Categorize missing pipe `type`.** Add `union_tag_not_found` beside `union_tag_invalid` in structured validation error categorization and test a `[pipe.foo]` block without `type`.-- **Add parse-level source attribution test/fix.** More involved than the previous item, but contained and important for editor diagnostics on multi-file validation.-- **Clean up additive multi-file polish.** The recursivity review notes are P3-only but cheap when nearby: soften an overclaiming docstring, comment the `or ""` coercion, and add one populated `LibraryLoadingError` forwarding test if touching validation.-- **Fix/retire runtime-bridge small leftovers.** Decide whether to consume or remove unused execution-mode property helpers; fix the dev helper that glob-selects the latest cost CSV instead of the current run.-- **Tidy plugin follow-ups.** Drop the dead `bedrock` model-lister key with its exact-set test update, and make plugin API version gating use a literal built-against version plus a semver compatibility rule.-- **Patch `temporal-e2e-validate` stale notes.** Remove the vestigial `act_jinja2_gen_text` reference and stale Mode 1 known-xfail text.-- **Add optional `doctor --plain` only if needed.** Current ANSI behavior is standard; a flag is a small ergonomics win for captured diagnostics if this bites again.--## Lower Priority / Parked--- CSV `.xlsx`, formula escaping, delimiter/encoding config, and remote tabular URLs are all feature expansion, not active debt.-- Structured logging is worth doing after the error-handling stack stops moving.-- `stuffs-as-nodespec`, blueprint elaboration directives, and the pathlib workflow are future-shaping tracks; pick them up only with a concrete product/tooling need.-- Historical records live in [`history/`](history/). Do not promote them back into active `wip/` unless they contain a concrete unowned follow-up.diff --git a/wip/graph/graphspec-source-map-enrichment.md b/wip/graph/graphspec-source-map-enrichment.mddeleted file mode 100644index 3394025fd..000000000--- a/wip/graph/graphspec-source-map-enrichment.md+++ /dev/null@@ -1,185 +0,0 @@-# GraphSpec Registry Source Paths--## Question--Can `GraphSpec` include the source bundle path for each pipe and concept by enriching `pipe_registry` and `concept_registry` entries from `LibraryCrate.source_map`?--Short answer: yes, this is feasible, and the lowest-churn path is to add a `source` field to registry payload dictionaries at graph event creation time.--## Current State--`GraphSpec` has:--- `nodes[]`: execution nodes with `pipe_code`, `pipe_type`, `description`, `domain_code`, IO, timing, status, metrics, and execution data.-- `pipe_registry`: `dict[str, dict[str, Any]]`, keyed by `domain.pipe_code`.-- `concept_registry`: `dict[str, dict[str, Any]]`, keyed by `domain.ConceptCode`.--The registry entries are optional. They are emitted only when `graph_config.data_inclusion.pipe_and_concept_registry` is true, which is true in the shipped config.--Today those registry entries do not include bundle source paths:--- pipe registry entries come from `PipeAbstract.model_dump(mode="json")`;-- concept registry entries come from `Concept.model_dump(mode="json")` plus `json_schema`;-- runtime `PipeAbstract` has no `source` field;-- runtime `Concept` does not receive the blueprint `source` when created.--The source data exists elsewhere:--- `PipeBlueprint.source` and `ConceptBlueprint.source` exist at blueprint level.-- `LibraryCrate.source_map` maps both `pipe_ref` and `concept_ref` to the source file path.-- `LibraryManager.get_pipe_source()` exposes pipe source for diagnostics, backed by the crate source map.-- `PipeJob.library_crate` is already attached by `prepare_pipe_job()` and crosses the Temporal boundary.--## Recommended Design--Enrich registry payloads before they are handed to the tracer:--- pipe registry entry: add `source` from `library_crate.source_map[self.pipe_ref]`;-- concept registry entry: add `source` from `library_crate.source_map[concept.concept_ref]`;-- if no crate or no source map entry exists, omit the field rather than emitting `source: null`.--This keeps source paths attached to the existing registry artifacts and avoids changing `NodeSpec`, `PipeAbstract`, or `Concept`.--Example resulting shape:--```json-{- "pipe_registry": {- "research.summarize": {- "code": "summarize",- "domain_code": "research",- "type": "PipeLLM",- "description": "Summarize notes",- "inputs": {},- "output": {},- "source": "/workspace/research/summarize.mthds"- }- },- "concept_registry": {- "research.Summary": {- "code": "Summary",- "domain_code": "research",- "description": "A summary",- "structure_class_name": "...",- "json_schema": {},- "source": "/workspace/research/concepts.mthds"- }- }-}-```--## Why This Is Feasible--The graph event creation point has access to both sides:--- `PipeAbstract._run_pipe_traced(..., library_crate: LibraryCrate | None = None)` already receives the crate.-- The same function currently builds `pipe_data` and `concept_data` before calling `tracer_manager.on_pipe_start(...)`.-- `PipeAbstract._make_single_concept_data_for_registry()` already centralizes concept registry payload construction.-- `PipeStartEvent.pipe_data` and `PipeStartEvent.concept_data` are generic dictionaries, so serialized Temporal graph events can carry the additional `source` field without schema changes.-- `GraphTracer` and `GraphSpecAssembler` already copy those dictionaries into `GraphSpec.pipe_registry` / `concept_registry` unchanged.--No `GraphSpec` schema migration is required because the registries are already `dict[str, Any]` payloads.--## Implementation Plan--1. Thread source-map awareness into registry serialization.-- Suggested helpers in `PipeAbstract`:-- ```python- def _make_pipe_data_for_registry(self, *, library_crate: LibraryCrate | None) -> dict[str, Any]:- pipe_data = self.model_dump(mode="json")- if library_crate is not None:- source = library_crate.source_map.get(self.pipe_ref)- if source:- pipe_data["source"] = source- return pipe_data- ```-- Update concept helpers similarly:-- ```python- def _make_single_concept_data_for_registry(- self,- concept: Concept,- *,- library_crate: LibraryCrate | None,- ) -> dict[str, Any]:- concept_dict = concept.model_dump(mode="json")- if library_crate is not None:- source = library_crate.source_map.get(concept.concept_ref)- if source:- concept_dict["source"] = source- ...- ```--2. Use those helpers from `_run_pipe_traced()`.-- Replace:-- ```python- pipe_data = self.model_dump(mode="json")- concept_data = self._make_concept_data_for_registry()- ```-- with source-aware helper calls using the `library_crate` argument already present in `_run_pipe_traced()`.--3. Preserve current behavior when source is unavailable.-- This happens for:-- - in-memory protocol validation without `mthds_sources`;- - tests or internal call sites that build `PipeJob` without `library_crate`;- - native concepts or concepts loaded from code rather than a `.mthds` declaration;- - dependency scenarios where the active crate does not include a source-map entry for a cross-package ref.-- In all of those cases, omit `source`.--4. Keep source on registry entries, not nodes.-- A node is an invocation. A registry entry is the declaration. Source path is declaration metadata, so the registry is the right layer. Renderers can join `node.domain_code + "." + node.pipe_code` to `pipe_registry` when they need to show source.--## Tests--Add coverage for both direct and assembled graph paths.--1. Unit-level helper test in the graph or pipe area:-- - construct a pipe with a fake `LibraryCrate(source_map={pipe_ref: "...", concept_ref: "..."})`;- - run with `pipe_and_concept_registry=True`;- - assert registry payloads include `source`.--2. Direct/in-memory graph integration:-- - use a real `.mthds` bundle loaded from disk;- - run graph generation;- - assert `graph_spec.pipe_registry["domain.pipe"]["source"]` is the bundle path;- - assert declared concepts in `concept_registry` include their source path.--3. Temporal/event-assembled path:-- - ensure `PipeStartEvent.pipe_data` / `concept_data` carry `source`;- - assert `GraphSpecAssembler` preserves the fields into `pipe_registry` / `concept_registry`.--4. Sourceless path:-- - validate/run from raw in-memory `mthds_contents` without `mthds_sources`;- - assert graph generation still succeeds and registry entries simply omit `source`.--## Edge Cases--- Multi-file same-domain bundles: `LibraryCrate.source_map` already tracks the winning declaration source for each `pipe_ref` / `concept_ref`, including signature/concrete reconciliation, so use it as the source of truth.-- Duplicate declarations: no new logic needed; the crate factory already rejects or reconciles before execution.-- Cross-package dependencies: confirm whether the `PipeJob.library_crate` used by the executing pipe includes dependency source-map entries. If not, source enrichment will be incomplete for dependency pipes. That is acceptable for a first pass, but it should be documented and pinned with a test or follow-up.-- Privacy: source paths can be absolute filesystem paths. That is already true in validation diagnostics and `LibraryCrate.source_map`, but exposing them through API graph responses is a product/API decision. If public hosted responses should not leak worker filesystem paths, add a normalization/redaction policy before shipping this on hosted surfaces.--## Files To Touch--- `pipelex/core/pipes/pipe_abstract.py`-- tests under `tests/unit/pipelex/graph/` or `tests/unit/pipelex/core/pipes/`-- graph integration tests under `tests/integration/pipelex/pipeline/` or existing graph tracing tests-- Temporal tracing tests if the event-assembled path needs explicit coverage--## Recommendation--Implement this as a registry enrichment, not as a structural `GraphSpec` model change. It is additive, works for direct and Temporal graph assembly, and uses the already-canonical `LibraryCrate.source_map` instead of duplicating source-tracking state onto runtime pipe/concept models.diff --git a/wip/graph/validate-explicit-graph-target.md b/wip/graph/validate-explicit-graph-target.mddeleted file mode 100644index 00d7e6169..000000000--- a/wip/graph/validate-explicit-graph-target.md+++ /dev/null@@ -1,150 +0,0 @@-# Validate Graphs With an Explicit Pipe Target--## Problem--Validation can only attach a `graph_spec` when the submitted batch declares a `main_pipe`. The current path is deliberate:--- `select_primary_blueprint()` picks the first blueprint declaring `main_pipe`; if none does, it returns the first blueprint with `main_pipe_ref=None`.-- `validate_bundles_in_process()` passes that `main_pipe_ref` to `best_effort_graph_spec()`.-- `best_effort_graph_spec()` returns `None` when `pipe_ref` is `None`.-- The CLI graph helpers use `dry_run_pipeline()`, which is stricter: no `main_pipe` raises `PipelexInterpreterError("Bundle does not declare a main_pipe, cannot generate graph")`.--This makes sense for the default report because `GraphSpec` is currently an execution trace for one root pipe, not a static diagram of every declared pipe in the bundle. But it blocks a useful case: a valid bundle with no declared `main_pipe`, or a partially built bundle where the caller wants to validate and graph one specific pipe.--## Proposed Lift--Support an explicit graph target pipe for validation graph generation.--The rule becomes:--- If the caller provides a graph target pipe, dry-run that pipe for `graph_spec`.-- Otherwise keep today's default: use the batch's selected `main_pipe`.-- If neither exists, keep returning `graph_spec=None` on protocol/API validation and keep the CLI graph command's explicit "cannot generate graph" error when a graph was requested.--This is intentionally not "graph the whole bundle." Whole-bundle static graphing is a different artifact with different semantics. This lift stays inside the existing execution-trace model by choosing one pipe to execute.--## Existing Machinery To Reuse--Most of the lower-level plumbing already supports this:--- `best_effort_graph_spec(pipe_ref=...)` can dry-run any resolved pipe ref against the already-open validation library.-- `act_dry_validate.DryValidateArg.pipe_code` already exists and is used as an override before falling back to `select_primary_blueprint(...).main_pipe_ref`.-- `pipeline_run_setup()` already supports `pipe_code` overriding `main_pipe` when executing MTHDS contents.-- `validate_bundle(..., dry_run_pipe_codes=[pipe_code])` already validates a single selected pipe within a bundle.--The missing work is making the direct/public validation surfaces carry the graph target consistently, and making CLI graph/view generation use it.--## Implementation Shape--1. Add an optional graph target parameter to the shared in-process validate orchestrator.-- Suggested signature:-- ```python- async def validate_bundles_in_process(- *,- mthds_contents: list[str],- mthds_sources: list[str] | None = None,- library_dirs: Sequence[Path] | None = None,- allow_signatures: bool = False,- graph_pipe_code: str | None = None,- log_context: str = "validate",- ) -> PipelexValidationReport:- ```-- The graph arm should call:-- ```python- graph_target_ref = graph_pipe_code or select_primary_blueprint(result.blueprints).main_pipe_ref- graph_spec = await best_effort_graph_spec(- pipe_ref=graph_target_ref,- library_id=validation_library_id,- log_context=log_context,- )- ```-- This preserves today's `None` behavior when no target exists. Unknown `graph_pipe_code` should degrade to `graph_spec=None` on the best-effort report path, because `best_effort_graph_spec()` intentionally catches pipe-resolution domain failures.--2. Thread the same option through `PipelexMTHDSProtocol.validate()` if the protocol layer has an extension argument for it.-- Today local protocol `validate()` rejects every `extra` key. Two reasonable choices:-- - accept `extra={"graph_pipe_code": "..."}` for Pipelex-specific local runtime usage;- - leave protocol `validate()` unchanged and expose the option only on API/CLI surfaces that already have concrete request fields.-- If the hosted MTHDS Protocol surface needs this, prefer a typed request field in the API schema and map it into the shared orchestrator rather than relying on untyped `extra`.--3. Wire the Temporal path using the field that already exists.-- `DryValidateArg.pipe_code` already chooses:-- ```python- arg.pipe_code or select_primary_blueprint(validate_result.blueprints).main_pipe_ref- ```-- Check the API dispatch layer and ensure the request's graph target field is passed into `DryValidateArg.pipe_code`. Rename to `graph_pipe_code` later if clarity is worth the wire break; no compatibility constraint blocks that, but avoid churn unless the surrounding API field also uses that name.--4. Update agent CLI `validate bundle --pipe X --graph/--view`.-- Today `--pipe` narrows validation, but `--graph` and `--view` still call `generate_graph_for_bundle()` / `generate_view_for_bundle()`, which eventually call `dry_run_pipeline()` with no pipe override and therefore require `main_pipe`.-- Add a pipe override to the graph rendering helpers:-- ```python- async def _dry_run_bundle(- bundle_path: Path,- *,- library_dirs: list[str] | None = None,- pipe_code: str | None = None,- ) -> tuple[GraphSpec, str]:- ```-- Then either:-- - extend `dry_run_pipeline(..., pipe_code: str | None = None)` and let it pass the override into `runner.execute(pipe_code=pipe_code or main_pipe_ref, ...)`; or- - bypass the pre-parse `main_pipe` requirement in `_dry_run_bundle()` when `pipe_code` is supplied.-- The first option is cleaner because it keeps one graph-producing bundle execution helper.--5. Decide bare vs qualified pipe target rules.-- Current pipe library lookup accepts either a unique bare code or a qualified `domain.code` ref. For multi-domain batches, qualified refs are safer. Recommended behavior:-- - accept both bare and qualified targets;- - document that bare targets must be unique in the loaded library;- - let existing `PipeLibrary.get_required_pipe()` ambiguity errors flow through the best-effort degradation path for report graph generation and through explicit CLI graph errors when the user requested graph output.--## Tests--Add tests at the existing pin points:--- `tests/integration/pipelex/pipeline/test_protocol_validate.py`- - no `main_pipe` plus explicit graph target produces a non-null `graph_spec`;- - explicit graph target overrides declared `main_pipe`;- - unknown explicit graph target degrades to `graph_spec=None` on the validation report path.--- Agent CLI validate bundle tests- - `validate bundle no_main.mthds --pipe some_pipe --view` succeeds and includes a `graphspec`;- - `validate bundle no_main.mthds --view` still errors because graph was explicitly requested and no target exists;- - `validate bundle bundle_with_main.mthds --pipe other_pipe --view` graphs `other_pipe`, not the declared `main_pipe`.--- Temporal dry-validate/API path- - request with explicit graph target maps to `DryValidateArg.pipe_code`;- - direct and Temporal modes produce the same `graph_spec` target behavior.--## Non-Goals--- Do not infer a root from "the only pipe in the bundle" unless product explicitly wants that policy. It sounds convenient but creates a hidden second default alongside `main_pipe`.-- Do not create a static all-pipes `GraphSpec` under the same field. That would overload an execution-trace artifact with non-execution semantics.-- Do not require `main_pipe` to validate a bundle. Current validation correctly supports concept-only files, membership-only files, and top-down partial bundles.--## Files To Touch--- `pipelex/pipeline/validate_in_process.py`-- `pipelex/pipeline/runner.py` if protocol `extra` or a typed protocol extension should expose it-- hosted API validate request/dispatch mapping in `pipelex-api`-- `pipelex/temporal/tprl_pipe/act_dry_validate.py` only if renaming or tightening `pipe_code`-- `pipelex/pipeline/dry_run_pipeline.py`-- `pipelex/graph/graph_rendering.py`-- `pipelex/cli/agent_cli/commands/validate/bundle_cmd.py`-- tests under `tests/integration/pipelex/pipeline/` and agent CLI validate coveragediff --git a/wip/history/PR-bundle-validator.html b/wip/history/PR-bundle-validator.htmldeleted file mode 100644index 4b51ed607..000000000--- a/wip/history/PR-bundle-validator.html+++ /dev/null@@ -1,695 +0,0 @@-<!DOCTYPE html>-<html lang="en">-<head>-<meta charset="UTF-8">-<meta name="viewport" content="width=device-width, initial-scale=1.0">-<title>PR Story — Dry-Run Consolidation: BundleValidator + Execution Seams</title>-<style>- :root {- --bg: #0f1117;- --panel: #181b24;- --panel-2: #1f2330;- --ink: #e8eaf0;- --muted: #9aa3b6;- --line: #2c3142;- --accent: #6ea8fe;- --green: #5fd38f;- --amber: #f0c060;- --red: #f08a8a;- --violet: #b89cf0;- --mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;- }- * { box-sizing: border-box; }- body {- margin: 0;- background: var(--bg);- color: var(--ink);- font: 16px/1.65 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;- -webkit-font-smoothing: antialiased;- }- .wrap { max-width: 960px; margin: 0 auto; padding: 56px 28px 120px; }- header.hero { border-bottom: 1px solid var(--line); padding-bottom: 32px; margin-bottom: 40px; }- .eyebrow { font: 600 12px/1 var(--mono); letter-spacing: .14em; text-transform: uppercase; color: var(--accent); }- h1 { font-size: 34px; line-height: 1.2; margin: 14px 0 10px; letter-spacing: -.01em; }- .sub { color: var(--muted); font-size: 17px; max-width: 760px; }- .meta { margin-top: 18px; display: flex; flex-wrap: wrap; gap: 8px; }- .chip {- font: 600 12px/1 var(--mono); padding: 6px 10px; border-radius: 999px;- background: var(--panel-2); border: 1px solid var(--line); color: var(--muted);- }- .chip.a { color: var(--accent); border-color: #2f4368; }- .chip.g { color: var(--green); border-color: #2c5340; }- h2 {- font-size: 22px; margin: 56px 0 16px; letter-spacing: -.01em;- padding-left: 12px; border-left: 3px solid var(--accent);- }- h3 { font-size: 16px; margin: 28px 0 10px; color: var(--ink); }- p { margin: 12px 0; }- .muted { color: var(--muted); }- strong { color: #f3f5fa; }- code {- font: 13.5px/1.5 var(--mono); background: var(--panel-2);- padding: 1.5px 6px; border-radius: 5px; border: 1px solid var(--line);- }- .panel {- background: var(--panel); border: 1px solid var(--line);- border-radius: 12px; padding: 20px 22px; margin: 18px 0;- }- .callout {- border-left: 3px solid var(--amber); background: var(--panel);- border-radius: 0 10px 10px 0; padding: 14px 18px; margin: 18px 0;- }- .callout.bug { border-left-color: var(--red); }- .callout.note { border-left-color: var(--accent); }- .callout.win { border-left-color: var(--green); }- .callout .lbl { font: 600 11px/1 var(--mono); letter-spacing: .1em; text-transform: uppercase; color: var(--amber); }- .callout.bug .lbl { color: var(--red); }- .callout.note .lbl { color: var(--accent); }- .callout.win .lbl { color: var(--green); }-- .cards { display: grid; grid-template-columns: repeat(3, 1fr); gap: 14px; margin: 18px 0; }- .cards.two { grid-template-columns: repeat(2, 1fr); }- .card { background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 18px; }- .card .n { font: 700 13px/1 var(--mono); color: var(--accent); }- .card h4 { margin: 8px 0 6px; font-size: 15px; }- .card p { margin: 0; font-size: 14px; color: var(--muted); }-- .figure { margin: 22px 0; background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 18px; }- .figure .mermaid { display: flex; justify-content: center; }- .figcap { color: var(--muted); font-size: 13px; margin-top: 12px; text-align: center; }-- table { width: 100%; border-collapse: collapse; margin: 16px 0; font-size: 13.5px; }- th, td { text-align: left; padding: 9px 11px; border-bottom: 1px solid var(--line); vertical-align: top; }- th { font: 600 11px/1.3 var(--mono); letter-spacing: .06em; text-transform: uppercase; color: var(--muted); }- td .dot { font-weight: 700; }- .dot.g { color: var(--green); }- .dot.a { color: var(--amber); }- .dot.r { color: var(--red); }- tbody tr:hover { background: var(--panel); }-- .diff {- background: #11141c; border: 1px solid var(--line); border-radius: 10px;- overflow-x: auto; margin: 14px 0; font: 12.5px/1.7 var(--mono);- }- .diff .fname {- padding: 8px 14px; border-bottom: 1px solid var(--line);- color: var(--muted); background: var(--panel-2); font-weight: 600;- }- .diff pre { margin: 0; padding: 12px 14px; }- .diff .add { color: var(--green); display: block; }- .diff .del { color: var(--red); display: block; }- .diff .ctx { color: #8b93a7; display: block; }- .diff .hdr { color: var(--accent); display: block; }- .diff .cmt { color: #7a8195; }-- ul.tight { margin: 10px 0; padding-left: 22px; }- ul.tight li { margin: 6px 0; }- ol.phases { margin: 10px 0; padding-left: 22px; }- ol.phases li { margin: 7px 0; }- .files { columns: 2; column-gap: 26px; font: 12.5px/1.9 var(--mono); color: var(--muted); }- .files .new { color: var(--green); }- .files .del { color: var(--red); }-- details { margin: 14px 0; border: 1px solid var(--line); border-radius: 10px; background: var(--panel); overflow: hidden; }- details > summary {- cursor: pointer; padding: 13px 18px; font: 600 14px/1.3 -apple-system, sans-serif;- background: var(--panel-2); list-style: none; user-select: none;- }- details > summary::-webkit-details-marker { display: none; }- details > summary::before { content: "▸ "; color: var(--accent); font-family: var(--mono); }- details[open] > summary::before { content: "▾ "; }- details > summary:hover { color: var(--accent); }- details .body { padding: 4px 20px 16px; }-- .tabs { margin: 18px 0; }- .tab-bar { display: flex; flex-wrap: wrap; gap: 6px; border-bottom: 1px solid var(--line); margin-bottom: 0; }- .tab-btn {- font: 600 12.5px/1 var(--mono); padding: 9px 14px; cursor: pointer;- background: transparent; border: none; border-bottom: 2px solid transparent; color: var(--muted);- }- .tab-btn:hover { color: var(--ink); }- .tab-btn.active { color: var(--accent); border-bottom-color: var(--accent); }- .tab-panel { display: none; padding: 6px 2px; }- .tab-panel.active { display: block; }-- .kbd { font: 600 12px/1 var(--mono); color: var(--violet); }- footer { margin-top: 70px; padding-top: 24px; border-top: 1px solid var(--line); color: var(--muted); font-size: 13px; }- @media (max-width: 720px) {- .cards, .cards.two { grid-template-columns: 1fr; }- .files { columns: 1; }- }-</style>-<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>-<script>- mermaid.initialize({- startOnLoad: true,- theme: "dark",- themeVariables: {- background: "#181b24", primaryColor: "#1f2330", primaryTextColor: "#e8eaf0",- primaryBorderColor: "#2c3142", lineColor: "#6ea8fe", fontFamily: "ui-monospace, monospace",- fontSize: "13px"- },- flowchart: { curve: "basis", htmlLabels: true }- });- function showTab(group, idx, btn) {- document.querySelectorAll('[data-tabgroup="' + group + '"] .tab-btn').forEach((b, i) => b.classList.toggle('active', i === idx));- document.querySelectorAll('[data-tabgroup="' + group + '"] .tab-panel').forEach((p, i) => p.classList.toggle('active', i === idx));- }-</script>-</head>-<body>-<div class="wrap">--<header class="hero">- <div class="eyebrow">Pull Request Story · #956</div>- <h1>Dry-Run Consolidation</h1>- <p class="sub">Four parallel dry-run code paths (plus a confirmed dead router) collapse into one first-class <code>BundleValidator</code> service composed over a shared <code>acquire_library</code> / <code>prepare_pipe_job</code> execution seam. Validation stops forking the runner — it now <em>composes the same execution core</em> a real run uses.</p>- <div class="meta">- <span class="chip a">branch · feature/Validate-with-signatures-4-fix-dry-run</span>- <span class="chip">base · feature/Validate-with-signatures-3 (stacked)</span>- <span class="chip g">Part A of 3 · in-process consolidation</span>- <span class="chip">~4.9k +/− · 70 files</span>- </div>-</header>--<!-- ───────────────────────────────── TL;DR ───────────────────────────────── -->-<h2>TL;DR</h2>-<div class="cards">- <div class="card">- <div class="n">THE PROBLEM</div>- <h4>Four doors to one room</h4>- <p>DRY mode had grown <code>dry_run.py</code>, <code>dry_run_with_graph.py</code>, <code>dry_pipe_router.py</code> and a pile of bespoke setup — each bypassing the real <code>PipelexRunner</code> orchestration. <code>DryPipeRouter</code> was dead code.</p>- </div>- <div class="card">- <div class="n">THE INSIGHT</div>- <h4>It's two operations, not one</h4>- <p>"Dry-run" was secretly <strong>two</strong> things wearing one name: a single-pipe <em>execution</em> (already the runner's job) and a tolerant batch <em>validation sweep</em> (a quality gate that <em>uses</em> runs).</p>- </div>- <div class="card">- <div class="n">THE FIX</div>- <h4>Compose, don't fork</h4>- <p>A new <code>BundleValidator</code> owns the sweep and <em>composes</em> the shared execution seam — the exact same core the runner reaches. The runner stays a pure single-pipe primitive with zero validation flags bolted on.</p>- </div>-</div>--<div class="callout note">- <div class="lbl">Scope</div>- <p style="margin:8px 0 0">This PR is <strong>Part A</strong> — the in-process consolidation. The design (<code>wip/dry-run-refactor-draft/D-plan.md</code>) also defines <strong>Part B</strong> (move the LIVE/DRY mock down to the cogt leaf so DRY honors the Temporal backend) and <strong>Part C</strong> (run the whole sweep as one standalone Temporal activity). Both are deferred to their own branches — see <a href="#deferred" style="color:var(--accent)">What's deferred</a>.</p>-</div>--<!-- ───────────────────────────────── RUN ⟷ VALIDATOR BOUNDARY (D6) ───────────────────────────────── -->-<h2>The run ⟷ validator boundary <span class="muted" style="font-size:15px">· clarification (D6)</span></h2>-<div class="callout note">- <div class="lbl">Design note · the run ⟷ validator seam</div>- <p style="margin:8px 0 0">A fair question one could raise: <em>shouldn't all those <code>except</code> clauses belong to the run, in the Runner's sense — rather than the validation layer?</em> Resolution (<code>D-plan.md</code> <strong>D6</strong>): the boundary holds — <strong>the validator <em>uses</em> the run</strong> — but the question has a valid kernel worth naming. The error-handling isn't two layers fighting over one home; it's <strong>three altitudes</strong>.</p>-</div>--<div class="cards">- <div class="card" style="border-left:3px solid var(--accent)">- <div class="n" style="color:var(--accent)">L1 · RUN PRIMITIVE</div>- <h4>Outcome + classification</h4>- <p><code>PipeRun.run</code> executes one pipe and <strong>already classifies</strong> its failure — it builds a structured <code>ErrorReport</code> (and <code>WfPipeRun</code> recovers the same one via <code>recover_error_report</code>, so it's backend-symmetric). ← the valid kernel: <em>classification is a run output.</em></p>- </div>- <div class="card" style="border-left:3px solid var(--green)">- <div class="n" style="color:var(--green)">L2 · BUNDLEVALIDATOR</div>- <h4>Batch policy over outcomes</h4>- <p><code>SKIPPED</code> (a missing cross-package dep — which a LIVE run <em>must</em> hard-fail on; “skipped” is only the <em>sweep's</em> reading of that raise), <code>allowed_to_fail</code>, aggregation, the signature pre-pass. <strong>Irreducibly batch</strong> — moving it into the run rebuilds the god-object D1 rejected and corrupts the strict LIVE contract.</p>- </div>- <div class="card" style="border-left:3px solid var(--violet)">- <div class="n" style="color:var(--violet)">L3 · TRANSLATE</div>- <h4>User-facing shaping</h4>- <p><code>_translate_to_validate_bundle_error</code> turns the run/load surface into one <code>ValidateBundleError</code> the CLI renders. A presentation concern of the entry point — neither run nor policy.</p>- </div>-</div>--<div class="callout">- <div class="lbl">The valid kernel — and why it's deferred, not denied</div>- <p style="margin:8px 0 0">The run <strong>discards</strong> the <code>ErrorReport</code> it already computed (it re-raises the raw exception), so <code>BundleValidator</code> re-derives classification with a <em>second</em> <code>try/except</code> + the <code>_root_cause_is</code> <code>__cause__</code>/<code>__context__</code> walk — a real seam smell. The clean fix is <strong>not</strong> “move the excepts into the run”; it's giving the run an outcome-<em>returning</em> sibling (<code>run_to_outcome</code> on <code>PipeRunProtocol</code>, in both <code>PipeRun</code> and <code>WfPipeRun</code>) the validator consumes, with <code>execute_pipeline</code> staying the strict raise-on-failure wrapper. Deferred because the payoff is bounded — <code>ErrorReport</code> is <em>wrapper-wins</em> and collapses the cause chain (a <code>PipeRunError</code> wrapping a <code>PipeNotFoundError</code> reports <code>error_type="PipeRunError"</code>), so the validator would <em>still</em> need the raw chain for <code>SKIPPED</code> — and it reopens the freshly-hardened signature path. Tracked in <code>followup-run-outcome-seam.md</code>.</p>-</div>--<!-- ───────────────────────────────── NORTH STAR ───────────────────────────────── -->-<h2>The north-star: "a run is a run"</h2>-<div class="panel">- <p style="margin-top:0" class="muted">The whole refactor is anchored on a user-stated principle:</p>- <p style="font-size:17px; color:var(--ink); border-left:3px solid var(--violet); padding-left:16px; margin:0">- “A run is a run, whether it's dry or live — it's the <strong>same thing</strong>. The delivery, the preparation, should go through the <strong>same thing</strong>. Everything should go through the <code>PipelexRunner</code>. <strong>Just drop those dry-run functions.</strong>”- </p>-</div>-<p>Taken literally that says "delete the dry-run functions and route everything through the runner." But a validation sweep is <em>tolerant</em> (collect <code>SUCCESS/FAILURE/SKIPPED</code>, honor <code>allowed_to_fail</code>, skip cross-package gaps) and the runner is <em>strict</em> (raise on first failure). You can't make one method be both without turning the runner into a god-object. The reframe below is what made the north-star achievable without that compromise.</p>--<!-- ───────────────────────────────── THE REFRAME ───────────────────────────────── -->-<h2>The reframe that drives the design</h2>-<div class="cards two">- <div class="card" style="border-left:3px solid var(--accent)">- <div class="n" style="color:var(--accent)">A · EXECUTION-DRY-RUN</div>- <h4>Already through the runner</h4>- <p><code>PipelexRunner.execute_pipeline(pipe_run_mode=DRY)</code> — one pipe, loads from scratch, <strong>raises on first failure</strong>, owns its library teardown. This is the north-star and it already existed. We don't touch its public behavior.</p>- </div>- <div class="card" style="border-left:3px solid var(--green)">- <div class="n" style="color:var(--green)">B · VALIDATION-SWEEP</div>- <h4>The bespoke <code>dry_run.py</code> path</h4>- <p>Takes <strong>already-loaded</strong> pipes, shares one library, runs each DRY <strong>tolerantly</strong>, does a single aggregated signature pre-check. This is a quality gate, <em>not</em> a run — it needs its own explicit home.</p>- </div>-</div>-<p>The honest conclusion: <strong>a validation sweep is not a run — it's a batch <em>policy</em> that <em>uses</em> runs.</strong> So "route everything through the runner" is exactly right for the execution primitive (A), while the batch semantics (B) need a clean home that <em>composes</em> the runner instead of forking it. That home is <code>BundleValidator</code>.</p>--<!-- ───────────────────────────────── BEFORE / AFTER ───────────────────────────────── -->-<h2>Before & after</h2>--<div class="tabs" data-tabgroup="ba">- <div class="tab-bar">- <button class="tab-btn active" onclick="showTab('ba',0,this)">Before — four parallel paths</button>- <button class="tab-btn" onclick="showTab('ba',1,this)">After — one seam, two composers</button>- </div>- <div class="tab-panel active">- <div class="figure">- <div class="mermaid">-flowchart TD- subgraph callers["validate callers"]- VA["validate --all"]- VB["validate_bundle"]- AG["agent CLI validate"]- BO["builder validate_ops / runner_code_ops"]- end- VA --> DR["dry_run_pipes()"]- VB --> DR- AG --> DRP["dry_run_pipe()"]- BO --> DR- DR --> SETUP["bespoke library setup<br/>+ mock inputs<br/>(duplicated in each caller)"]- DRG["dry_run_with_graph.py<br/>(no consumers)"]:::dead- DPR["DryPipeRouter<br/>(never instantiated)"]:::dead- SETUP --> RUN["pipe.run_pipe(DRY)"]- classDef dead fill:#2a1a1f,stroke:#f08a8a,color:#f08a8a;- </div>- <div class="figcap">Each caller hand-rolled its own library open / load / mock-input setup, then funneled into <code>dry_run_pipe(s)</code> — none of which shared a path with the real runner. Two modules were outright dead.</div>- </div>- </div>- <div class="tab-panel">- <div class="figure">- <div class="mermaid">-flowchart TD- subgraph seam["execution_seams.py — the shared core"]- AL["acquire_library()<br/>open + load dirs/blueprints<br/>owns load-failure teardown"]- PJ["prepare_pipe_job()<br/>working memory + mock inputs<br/>run params + crate → PipeJob<br/><i>pure: no telemetry/registry</i>"]- end- RUNNER["PipelexRunner.execute_pipeline<br/>(single run · strict · LIVE or DRY)"] --> WRAP["pipeline_run_setup<br/>(thin wrapper)"]- WRAP --> AL- WRAP --> PJ- BV["BundleValidator<br/>(batch sweep · tolerant)"] --> AL- BV --> PJ- BV --> DPR["direct in-process PipeRun<br/>(NOT get_pipe_run)"]- classDef seamcls fill:#16241a,stroke:#5fd38f,color:#cfeede;- class AL,PJ seamcls;- </div>- <div class="figcap">Both the runner's thin <code>pipeline_run_setup</code> wrapper and <code>BundleValidator</code> compose the <em>same two seams</em>. The runner gains no flags; the validator gains no fork. "A run is a run" is now literal shared code.</div>- </div>- </div>-</div>--<!-- ───────────────────────────────── THE SEAMS ───────────────────────────────── -->-<h2>The shared execution seam (D2)</h2>-<p>The reusable middle of <code>pipeline_run_setup</code> was factored into two building blocks in <code>pipelex/pipeline/execution_seams.py</code>. The key discipline: <code>prepare_pipe_job</code> is <strong>pure</strong> — it builds a job and nothing else (no pipeline-manager registration, no report-registry open, no telemetry, no library mutation). Those side effects are properties of the single-run <em>wrapper</em>, not of building-and-running a job — which is exactly why the sweep can skip them with nothing to "suppress."</p>--<div class="cards two">- <div class="card">- <div class="n">acquire_library(...)</div>- <h4>Set current → open → load</h4>- <p>Loads dirs + blueprints into a fresh library. Owns its <strong>own</strong> load-failure teardown (restore the caller's outer current-library, then tear the just-opened one down) so a failed load never leaks a <code>Library</code>. Returns <code>(library_id, qualified_main_pipe)</code>.</p>- </div>- <div class="card">- <div class="n">prepare_pipe_job(...)</div>- <h4>Build a <code>PipeJob</code>, purely</h4>- <p>Working memory (user inputs + mock inputs when <code>is_mock_inputs</code>), run params, job metadata, library crate. Mock-input generation now lives <em>here</em>, so callers stop building working memory themselves.</p>- </div>-</div>--<details>- <summary>Minimal diff — <code>pipeline_run_setup</code> recomposed onto the seams</summary>- <div class="body">- <p class="muted">~130 lines of inline library-open + mock-input logic became two seam calls. Net behavior of the LIVE / self-contained DRY path is identical — it's the same code, re-expressed.</p>- <div class="diff">- <div class="fname">pipelex/pipeline/pipeline_run_setup.py</div>-<pre><span class="del">- library_manager = get_library_manager()</span>-<span class="del">- set_current_library(library_id=library_id)</span>-<span class="del">- library_manager.open_library(library_id=library_id)</span>-<span class="del">- effective_dirs, source_label = resolve_library_dirs(library_dirs)</span>-<span class="del">- if effective_dirs:</span>-<span class="del">- library_manager.load_libraries(library_id=library_id, library_dirs=effective_dirs)</span>-<span class="del">- if mthds_contents:</span>-<span class="del">- all_blueprints = [PipelexInterpreter.make_...(content) for content in mthds_contents]</span>-<span class="del">- ... # ~40 lines: filter already-loaded URIs, load_from_blueprints, resolve main_pipe</span>-<span class="del">- # ...then ~30 more lines of mock-input + working-memory assembly inline</span>-<span class="ctx"> </span>-<span class="add">+ # Seam 1: open the library and load dirs + blueprints. Owns its own load-failure teardown.</span>-<span class="add">+ library_id, qualified_main_pipe = acquire_library(</span>-<span class="add">+ library_id=library_id, library_dirs=library_dirs,</span>-<span class="add">+ mthds_contents=mthds_contents, bundle_uris=bundle_uris,</span>-<span class="add">+ )</span>-<span class="ctx"> ...</span>-<span class="add">+ # Seam 2 (later, against the now-open library): build the job purely.</span>-<span class="add">+ pipe_job = await prepare_pipe_job(pipe=pipe, library_id=library_id, ...)</span></pre>- </div>- </div>-</details>--<!-- ───────────────────────────────── BUNDLEVALIDATOR ───────────────────────────────── -->-<h2>BundleValidator — the batch validation service (D1)</h2>-<p>A first-class domain service in <code>pipelex/pipeline/bundle_validator.py</code>. It owns the sweep semantics and runs every pipe through a <strong>locally-constructed, direct in-process <code>PipeRun</code></strong> — explicitly <em>not</em> the hub's <code>get_pipe_run()</code>, which under a Temporal hub would spawn a workflow <em>per pipe</em> (and is illegal from inside an activity). Keeping it direct is what lets the whole sweep later be hosted in one Temporal activity (Part C).</p>--<div class="diff">- <div class="fname">pipelex/pipeline/bundle_validator.py — the direct primitive</div>-<pre><span class="ctx">class BundleValidator:</span>-<span class="ctx"> def __init__(self) -> None:</span>-<span class="add"> # Locally-constructed, direct execution primitive — NOT get_pipe_run().</span>-<span class="add"> # ObserverNoOp keeps the sweep observation-silent; validation surfaces its own report.</span>-<span class="add"> self._pipe_run: PipeRunProtocol = PipeRun(pipe_router=PipeRouter(observer=ObserverNoOp()))</span></pre>-</div>--<h3>Two lifecycles (D6)</h3>-<p>The subtlety that an outside-voice review pass caught: <code>validate --all</code> wants the library torn down afterward, but <code>validate_bundle</code> and the build CLIs depend on the library staying <em>loaded</em> so they can keep using it. One method couldn't satisfy both — so there are two:</p>--<div class="tabs" data-tabgroup="lc">- <div class="tab-bar">- <button class="tab-btn active" onclick="showTab('lc',0,this)">acquire_and_validate — owns teardown</button>- <button class="tab-btn" onclick="showTab('lc',1,this)">validate_pipes — loaded-on-success</button>- </div>- <div class="tab-panel active">- <p><strong>The standalone <code>validate --all</code> lifecycle.</strong> Opens a fresh library, loads dirs + contents, sweeps <em>all</em> loaded pipes, and <strong>always</strong> tears the library down in a <code>finally</code> — restoring the caller's outer current-library first so the guarantee survives a teardown raise.</p>- <div class="diff">- <div class="fname">acquire_and_validate(...)</div>-<pre><span class="ctx">prev_library_id = get_current_library_id_or_none()</span>-<span class="ctx">acquired_id, _ = acquire_library(library_id=library_id, library_dirs=library_dirs, ...)</span>-<span class="ctx">try:</span>-<span class="ctx"> all_pipes = get_pipe_library().get_pipes()</span>-<span class="cmt"> # strict mode excludes signatures; lenient keeps them (they mint a mock trivially)</span>-<span class="ctx"> pipes = all_pipes if allow_signatures else [p for p in all_pipes if not p.is_signature]</span>-<span class="ctx"> return await self.validate_pipes(pipes, library_id=acquired_id, allow_signatures=allow_signatures)</span>-<span class="ctx">finally:</span>-<span class="add"> if prev_library_id is not None: set_current_library(library_id=prev_library_id)</span>-<span class="add"> else: clear_current_library()</span>-<span class="add"> get_library_manager().teardown(library_id=acquired_id)</span></pre>- </div>- </div>- <div class="tab-panel">- <p><strong>The public inner sweep.</strong> Classifies a caller-supplied list of pipes against the caller's <em>already-open</em> library and <strong>never tears it down</strong> — preserving the loaded-on-success contract (pinned by <code>test_validate_bundle_loaded_on_success</code>). This is what <code>validate_bundle</code>, <code>validate_pipe</code> and <code>build_runner_code_for_pipe</code> call.</p>- <p class="muted">Order matters (D7): the static <code>validate_with_libraries</code> wiring check runs <em>before</em> the signature pre-pass, so a wiring error keeps its precedence over a signature error — exactly as before.</p>- </div>-</div>--<h3>The sweep, step by step</h3>-<div class="figure">- <div class="mermaid">-flowchart TD- START([validate_pipes pipes, library_id, allow_signatures]) --> WIRE- WIRE["1 · wiring check<br/>pipe.validate_with_libraries()"] -->|PipeNotFoundError| SKIP1[["record SKIPPED<br/>drop from sweep"]]- WIRE -->|ok| COLLECT[collect sweepable_pipes]- SKIP1 --> COLLECT- COLLECT --> SIG{2 · allow_signatures?}- SIG -->|false| PRE["signature pre-pass<br/>aggregated SignaturesNotAllowedError<br/>lists every reached signature"]- SIG -->|true| TEL- PRE --> TEL["3 · one PIPE_DRY_RUN telemetry event"]- TEL --> REG[open ONE report registry for the sweep]- REG --> LOOP["4 · per pipe: prepare_pipe_job DRY<br/>+ self._pipe_run.run(job)"]- LOOP --> CLS{classify}- CLS -->|success| OK[SUCCESS]- CLS -->|cause chain has PipeNotFoundError| SK[SKIPPED]- CLS -->|other PipelexError / ValidationError| FAIL[FAILURE]- OK --> AGG- SK --> AGG- FAIL --> AGG["5 · aggregate<br/>allowed_to_fail match on namespaced pipe_ref<br/>collect ALL unexpected failures → one DryRunError"]- AGG --> DONE([per-pipe status map])- </div>- <div class="figcap">The tolerant sweep: classify, never abort early. Unexpected failures are collected into a single aggregated error, not raised on the first one.</div>-</div>--<div class="callout bug">- <div class="lbl">Subtle bug avoided · SKIPPED classification</div>- <p style="margin:8px 0 0">Routing each pipe through <code>PipeRun.run</code> means a cross-package unresolved dependency no longer arrives as a bare <code>PipeNotFoundError</code> — the run layer re-raises the original and the router may add <code>__cause__</code> links. If <code>BundleValidator</code> only checked the top-level exception, partial-bundle validation would regress from <span class="dot a">SKIPPED</span> to a hard <span class="dot r">FAILURE</span>. So <code>_root_cause_is</code> walks the entire <code>__cause__</code> / <code>__context__</code> chain (with a seen-set guard against cyclic contexts).</p>-</div>--<details>- <summary>Minimal diff — <code>_classify_pipe</code> and the recursive cause walk</summary>- <div class="body">- <div class="diff">- <div class="fname">pipelex/pipeline/bundle_validator.py</div>-<pre><span class="ctx">try:</span>-<span class="ctx"> pipe_job = await prepare_pipe_job(pipe=pipe, library_id=library_id,</span>-<span class="ctx"> execution_config=execution_config, pipe_run_mode=PipeRunMode.DRY,</span>-<span class="ctx"> pipeline_run_id=SpecialPipelineId.DRY_RUN_UNTITLED, ...)</span>-<span class="ctx"> await self._pipe_run.run(pipe_job)</span>-<span class="cmt">except (PipelexError, ValidationError, FactoryException) as exc:</span>-<span class="cmt"> # broad DOMAIN surface — never `except Exception`</span>-<span class="add"> if self._root_cause_is(exc=exc, exc_type=PipeNotFoundError):</span>-<span class="add"> return DryRunOutput(..., status=DryRunStatus.SKIPPED, ...) # cross-package gap</span>-<span class="ctx"> return DryRunOutput(..., status=DryRunStatus.FAILURE, ...)</span>-<span class="ctx">return DryRunOutput(..., status=DryRunStatus.SUCCESS)</span>-<span class="ctx"> </span>-<span class="add">@classmethod</span>-<span class="add">def _root_cause_is(cls, *, exc, exc_type) -> bool:</span>-<span class="add"> seen: set[int] = set(); stack = [exc]</span>-<span class="add"> while stack:</span>-<span class="add"> current = stack.pop()</span>-<span class="add"> if id(current) in seen: continue # cyclic __context__ guard</span>-<span class="add"> seen.add(id(current))</span>-<span class="add"> if isinstance(current, exc_type): return True</span>-<span class="add"> for linked in (current.__cause__, current.__context__):</span>-<span class="add"> if linked is not None: stack.append(linked)</span>-<span class="add"> return False</span></pre>- </div>- </div>-</details>--<!-- ───────────────────────────────── LIBRARY LIFECYCLE ───────────────────────────────── -->-<h2>Library lifecycle hardening</h2>-<p>A recurring, easy-to-get-wrong pattern shows up across every code path that opens a temporary library: <strong>the current-library binding is a <code>ContextVar</code>, and naive teardown clobbers the caller's outer binding.</strong> The fix is one disciplined idiom, applied everywhere.</p>--<div class="figure">- <div class="mermaid">-sequenceDiagram- participant C as Caller (has outer library X)- participant H as hub _library_id ContextVar- participant V as validate path- C->>H: outer current = X- C->>V: enter- V->>H: prev = get_current_library_id_or_none() (= X)- V->>H: set_current = Y (the temp/run library)- Note over V: ...do the work (may raise)...- rect rgb(42,26,31)- Note over V,H: finally / teardown- V->>H: restore prev FIRST → set_current = X- V->>H: teardown(Y)- end- V->>C: return (outer current still = X)- </div>- <div class="figcap">Restore the outer binding <em>before</em> teardown, so the safety guarantee holds even if teardown itself raises. <code>set_current_library</code> can't take <code>None</code>, so the "no outer was set" case routes through the renamed <code>clear_current_library()</code>.</div>-</div>--<div class="callout note">- <div class="lbl">Rename · teardown_current_library → clear_current_library</div>- <p style="margin:8px 0 0">The old name implied it freed the <code>Library</code>. It doesn't — it only resets the <code>_library_id</code> <em>pointer</em> to <code>None</code>. Releasing the library is a separate <code>library_manager.teardown(library_id=...)</code> call. The rename makes the two distinct operations honest at every call site.</p>-</div>--<details>- <summary>Minimal diff — runner restores the outer binding, with a collision guard</summary>- <div class="body">- <p class="muted">The recent review fix in <code>PipelexRunner.execute_pipeline</code>: capture the outer binding up front, and on teardown restore it — <em>unless</em> the caller's outer library <em>is</em> this run's library (in which case "restoring" would leave the ContextVar pointing at the library we're about to tear down, so we clear instead).</p>- <div class="diff">- <div class="fname">pipelex/pipeline/runner.py</div>-<pre><span class="add">+ prev_library_id = get_current_library_id_or_none() # before pipeline_run_setup overwrites it</span>-<span class="ctx"> try:</span>-<span class="ctx"> pipe_job, pipeline_run_id, library_id_resolved = await pipeline_run_setup(...)</span>-<span class="ctx"> ...</span>-<span class="ctx"> finally:</span>-<span class="ctx"> if library_id_resolved is not None:</span>-<span class="del">- get_library_manager().teardown(library_id=library_id_resolved)</span>-<span class="del">- teardown_current_library()</span>-<span class="add">+ # restore outer FIRST so the guarantee survives a teardown raise</span>-<span class="add">+ if prev_library_id is not None and prev_library_id != library_id_resolved:</span>-<span class="add">+ set_current_library(library_id=prev_library_id)</span>-<span class="add">+ else:</span>-<span class="add">+ clear_current_library() # collision OR no outer → clear, don't dangle</span>-<span class="add">+ get_library_manager().teardown(library_id=library_id_resolved)</span></pre>- </div>- </div>-</details>--<details>- <summary>Minimal diff — builder ops now own their library lifecycle</summary>- <div class="body">- <p class="muted"><code>build_runner_code_for_pipe</code> keeps the library loaded on success (it generates runner code against it) but restores + tears down on failure. <code>validate_pipe</code> owns the full lifecycle on both paths since it doesn't need the library after returning. Both pinned by new lifecycle tests.</p>- <div class="diff">- <div class="fname">pipelex/builder/operations/runner_code_ops.py</div>-<pre><span class="del">- set_current_library(library_id)</span>-<span class="del">- for pipe in pipes: pipe.validate_with_libraries()</span>-<span class="del">- await dry_run_pipes(pipes=pipes, raise_on_failure=True)</span>-<span class="del">- pipe = get_required_pipe(pipe_code)</span>-<span class="del">- return generate_runner_code(pipe=pipe)</span>-<span class="add">+ prev_library_id = get_current_library_id_or_none()</span>-<span class="add">+ success = False</span>-<span class="add">+ try:</span>-<span class="add">+ set_current_library(library_id=library_id)</span>-<span class="add">+ pipes = library_manager.load_from_blueprints(library_id=library_id, blueprints=blueprints)</span>-<span class="add">+ # validate (static wiring + dry run) — never tears the library down (D6 inner sweep)</span>-<span class="add">+ await BundleValidator().validate_pipes(pipes=pipes, library_id=library_id)</span>-<span class="add">+ runner_code = generate_runner_code(pipe=get_required_pipe(pipe_code))</span>-<span class="add">+ success = True</span>-<span class="add">+ return runner_code</span>-<span class="add">+ finally:</span>-<span class="add">+ if not success: # loaded-on-success; this caller owns cleanup on failure</span>-<span class="add">+ if prev_library_id is not None: set_current_library(library_id=prev_library_id)</span>-<span class="add">+ else: clear_current_library()</span>-<span class="add">+ library_manager.teardown(library_id=library_id)</span></pre>- </div>- </div>-</details>--<div class="callout win">- <div class="lbl">Win · idempotent close_registry</div>- <p style="margin:8px 0 0"><code>ReportingManager.close_registry</code> now pops with a default instead of raising <code>KeyError</code> on a miss. It runs from sweep / runner <code>finally</code> blocks, where a <code>KeyError</code> would <em>mask the in-flight exception</em> — exactly the kind of silent-failure swap we want gone. (Mirrors the existing <code>clear_event_log</code> behavior.)</p>-</div>--<!-- ───────────────────────────────── SIGNATURES ───────────────────────────────── -->-<h2>Signature validation fixes</h2>-<div class="cards two">- <div class="card">- <div class="n">CLASSIFICATION</div>- <h4><code>SignaturesNotAllowedError</code> → INPUT domain</h4>- <p>An unimplemented placeholder in a submitted bundle is a <em>caller</em> mistake, not a server fault. Now marked <code>error_domain = ErrorDomain.INPUT</code> + <code>_authors_caller_facing_message = True</code> → the API renders it <strong>422</strong> (not 500), and the CLI logs it as a caller error. Matches the agent-CLI's existing classification.</p>- </div>- <div class="card">- <div class="n">IDENTITY</div>- <h4><code>validated_pipes</code> unified on namespaced <code>pipe_ref</code></h4>- <p>Previously <code>validate all</code> reported <code>domain.code</code> while <code>validate pipe</code> reported the bare <code>code</code> — the same pipe under two identities depending on the command. Now <strong>every</strong> surface emits the qualified ref, which can't collide across domains.</p>- </div>-</div>--<details>- <summary>Minimal diff — the caller-facing classification</summary>- <div class="body">- <div class="diff">- <div class="fname">pipelex/pipe_signature/exceptions.py</div>-<pre><span class="del">- from pipelex.base_exceptions import PipelexError</span>-<span class="add">+ from pipelex.base_exceptions import ErrorDomain, PipelexError</span>-<span class="ctx"> </span>-<span class="ctx"> class SignaturesNotAllowedError(PipelexError):</span>-<span class="add">+ error_domain = ErrorDomain.INPUT # 422, not 500 — caller fixes their bundle</span>-<span class="add">+ _authors_caller_facing_message = True</span></pre>- </div>- <p class="muted">The shared projection helper <code>build_validated_pipes</code> dropped its <code>use_ref</code> parameter and now always emits <code>output.pipe_ref</code>; its entry type is a named <code>ValidatedPipeEntry</code> <code>TypedDict</code>. Allowed-to-fail FAILUREs and cross-package SKIPPEDs are reported truthfully rather than flattened to SUCCESS.</p>- </div>-</details>--<!-- ───────────────────────────────── BREAKING CHANGES ───────────────────────────────── -->-<h2>Breaking changes</h2>-<p class="muted">No backward-compat period (per project policy) — note them and move.</p>-<table>- <thead><tr><th>Change</th><th>What breaks</th><th>Migration</th></tr></thead>- <tbody>- <tr>- <td><code>validated_pipes</code> identity</td>- <td>The JSON <code>pipe_code</code> key now carries the namespaced <code>domain.code</code> ref on every surface (it used to be bare on some).</td>- <td>Consumers parsing <code>pipe_code</code> (e.g. MTHDS skills) must match on the qualified ref.</td>- </tr>- <tr>- <td><code>allowed_to_fail_pipes</code></td>- <td>Config now keys off namespaced refs. <code>infinite_loop_1</code> → <code>failing_pipelines.infinite_loop_1</code>; obsolete <code>pipe_builder</code> entry dropped.</td>- <td>Update <code>pipelex.toml</code> <code>dry_run_config.allowed_to_fail_pipes</code> to qualified refs.</td>- </tr>- <tr>- <td><code>teardown_current_library</code></td>- <td>Renamed to <code>clear_current_library</code>.</td>- <td>Rename the call. Remember: it only clears the pointer; <code>teardown(library_id=...)</code> frees the library.</td>- </tr>- <tr>- <td>Deleted dry-run functions</td>- <td><code>dry_run_pipe</code>, <code>dry_run_pipes</code>, <code>dry_run_pipe_with_graph</code>, <code>DryPipeRouter</code> are gone.</td>- <td>Use <code>BundleValidator</code> (sweep) or <code>PipelexRunner(..., pipe_run_mode=DRY)</code> (single run / graph).</td>- </tr>- <tr>- <td>Moved conversion helpers</td>- <td><code>convert_to_working_memory_format</code> & <code>convert_stuff_spec_to_typed_named</code> left <code>dry_run.py</code>.</td>- <td>Import from <code>WorkingMemoryFactory</code>.</td>- </tr>- <tr>- <td><code>DryRunStatus</code> / <code>DryRunOutput</code></td>- <td>Relocated (they're validation-report types, not execution types).</td>- <td>Import from <code>pipelex.pipeline.bundle_validator</code>.</td>- </tr>- </tbody>-</table>--<div class="callout bug">- <div class="lbl">Cross-repo follow-up · pipelex-api</div>- <p style="margin:8px 0 0"><code>pipelex-api</code>'s <code>build/runner.py</code> still imports the now-deleted <code>dry_run_pipes</code> and will break on its next <code>pipelex</code> pin bump. Either land Part C (which migrates that route to <code>act_validate_bundle</code>) in the same release train, or keep a temporary <code>dry_run_pipes</code> shim until then.</p>-</div>--<!-- ───────────────────────────────── DELETED ───────────────────────────────── -->-<h2>What got deleted (Phase 4)</h2>-<div class="panel">- <div class="files">- <div class="del">− pipe_run/dry_run.py <span style="color:var(--muted)">(277)</span></div>- <div class="del">− pipe_run/dry_run_with_graph.py <span style="color:var(--muted)">(82, no consumers)</span></div>- <div class="del">− pipe_run/dry_pipe_router.py <span style="color:var(--muted)">(24, dead code)</span></div>- <div class="del">− tests/unit/.../pipe_run/test_dry_run.py <span style="color:var(--muted)">(113, ported)</span></div>- </div>- <p class="muted" style="margin:14px 0 0">Phase 0 first relocated the two conversion helpers (so the signature runtime no longer imported <code>dry_run.py</code>), which turned the module into a deletable leaf. <code>dry_run_pipeline.py</code> survives — it was already runner-based and north-star-compliant.</p>-</div>--<!-- ───────────────────────────────── TESTS ───────────────────────────────── -->-<h2>Test coverage</h2>-<div class="cards">- <div class="card">- <div class="n">SERVICE</div>- <h4>BundleValidator</h4>- <p><code>test_bundle_validator</code> (unit + integration), <code>test_acquire_and_validate</code> — the sweep, classification, signature pre-pass, aggregation.</p>- </div>- <div class="card">- <div class="n">SEAMS</div>- <h4>Execution seams</h4>- <p><code>test_execution_seams</code>, <code>test_pipeline_run_setup_characterization</code> — proves the recomposition is behavior-identical.</p>- </div>- <div class="card">- <div class="n">LIFECYCLE</div>- <h4>Library & registries</h4>- <p><code>test_validate_pipe_library_lifecycle</code>, <code>test_runner_code_ops_library_lifecycle</code>, <code>test_runner_library_lifecycle</code>, <code>test_validate_bundle_loaded_on_success</code>, <code>test_no_orphan_registries</code>.</p>- </div>-</div>-<p class="muted">Existing dry-run / signature / library-crate tests were rewired onto the new surface (importing from <code>bundle_validator</code> / <code>WorkingMemoryFactory</code> instead of the deleted modules).</p>--<!-- ───────────────────────────────── DEFERRED ───────────────────────────────── -->-<h2 id="deferred">What's deferred (and why the design looks the way it does)</h2>-<p>The whole conceptual frame is <strong>run mode (LIVE/DRY) is orthogonal to execution backend (direct vs Temporal)</strong>. Four cells, all must eventually work — and that's <em>why</em> <code>BundleValidator</code> is built as a self-contained in-process executor that never touches the hub's Temporal components.</p>--<table>- <thead><tr><th></th><th>direct (in-process)</th><th>Temporal (distributed)</th></tr></thead>- <tbody>- <tr><td><strong>LIVE</strong></td><td><span class="dot g">●</span> works today</td><td><span class="dot g">●</span> works today</td></tr>- <tr><td><strong>DRY</strong></td><td><span class="dot g">●</span> this PR (the sweep)</td><td><span class="dot a">◐</span> Part B — mock at the activity leaf</td></tr>- </tbody>-</table>--<div class="cards two">- <div class="card" style="border-left:3px solid var(--amber)">- <div class="n" style="color:var(--amber)">PART B · own branch</div>- <h4>Leaf-level run-mode mock (req 1)</h4>- <p>Today the DRY mock is minted <em>above</em> the activity boundary (operators hardcode <code>ContentGeneratorDry()</code>), so DRY on a Temporal hub never dispatches <code>act_llm_gen_*</code>. Part B moves the LIVE/DRY decision down to the cogt leaf (<code>llm_gen_*</code> / extract / img-gen, carried on <code>JobMetadata.run_mode</code>) so DRY honors the backend and mocks <em>inside</em> the activity. Retires the "DRY → local" shortcut.</p>- </div>- <div class="card" style="border-left:3px solid var(--violet)">- <div class="n" style="color:var(--violet)">PART C · own branch, gated</div>- <h4>Validation as a Temporal activity (req 2)</h4>- <p>The API dispatches the whole sweep as one standalone <code>act_validate_bundle</code> activity. Needs a <code>scoped_content_generator</code> ContextVar (mirroring <code>scoped_current_library</code>) to force the inline generator, a <code>temporalio</code> bump beyond 1.23.0 for standalone activities, and the cross-repo <code>pipelex-api</code> route migration.</p>- </div>-</div>--<p class="muted">A third deferred item is non-backend: the <strong>run-outcome seam</strong> (D6) — have the run <em>return</em> its classified outcome so the validator stops re-catching it. Optional tightening, not a req; see <code>followup-run-outcome-seam.md</code>.</p>--<p class="muted">Design source: <code>wip/dry-run-refactor-draft/D-plan.md</code> (architecture decisions <strong>D1–D6</strong>; eng-review decisions D1–D8 live separately in <code>TODOS.md</code>). Execution tracker for Part A: <code>TODOS.md</code> (checkpoints A / A2 / B / C, all cleared).</p>--<footer>- <p><strong>PR #956</strong> · <code>refactor(dry-run): consolidate validation into BundleValidator + execution seams</code> · Part A of the dry-run refactor.</p>- <p>Key files: <code>pipelex/pipeline/bundle_validator.py</code> · <code>pipelex/pipeline/execution_seams.py</code> · <code>pipelex/pipeline/pipeline_run_setup.py</code> · <code>pipelex/pipeline/validate_bundle.py</code> · <code>pipelex/builder/operations/{validate_ops,runner_code_ops}.py</code> · <code>pipelex/pipeline/runner.py</code>.</p>-</footer>--</div>-</body>-</html>diff --git a/wip/history/README.md b/wip/history/README.mddeleted file mode 100644index 6f12f2cbb..000000000--- a/wip/history/README.md+++ /dev/null@@ -1,16 +0,0 @@-# `wip/history/`--This folder holds shipped project records, old audits, and point-in-time recaps that are useful for archaeology but should not compete with active `wip/` priorities.--The active priority view is [`../executive.md`](../executive.md). If a historical doc still contains a real pickup item, surface that item in `executive.md` instead of moving the whole record back to the active tree.--## Contents--- [`archive/`](archive/) - former `wip/archive/` records.-- [`csv-support/`](csv-support/) - shipped CSV input/output feature notes and deferred expansion items.-- [`keyword-only-arguments/`](keyword-only-arguments/) - completed keyword-only refactor, audits, and downstream notes.-- [`registry/`](registry/) - shipped cost-reporting/usage-registry work and remaining deferred design decisions.-- [`signature-based-validation/`](signature-based-validation/) - completed signature validation feature record.-- [`tests/`](tests/) - completed test-grind notes.-- [`recursivity/`](recursivity/) - completed additive multi-file/recursive-design handoffs.-- Top-level HTML files - rendered PR or design recaps kept as static snapshots.diff --git a/wip/history/archive/plugin-system-implementation-tracker.md b/wip/history/archive/plugin-system-implementation-tracker.mddeleted file mode 100644index 22f200f44..000000000--- a/wip/history/archive/plugin-system-implementation-tracker.md+++ /dev/null@@ -1,596 +0,0 @@-> **ARCHIVED 2026-06-21.** This tracked the in-`pipelex` plugin-system work (Phases 0–5: invert every core↔integration seam, then externalize Temporal to the private `pipelex-temporal` plugin). Phases 0–4 are complete and committed; Phase 5's local cut-over is done, with only the publish/release-gated downstream pin-flips outstanding. The active tracker is now [`../../../TODOS.md`](../../../TODOS.md) — the **consumer-side** migration (orchestrator-agnostic `pipelex-api` base + deployment flavors). Kept here for the as-built history and the decision log (D1–D9).--# Pipelex plugin system — implementation TODOS--**Branch:** `refactor/Plugins-2` (worktree `_plugins`) · **Status:** Phases 0–3 complete + **Option A landed** (post-Phase-3). Phase 3 (orchestrators through the seam) = `5f61db323`. **Option A** — drop the CLI-command harvest entirely; Temporal `worker`/`setup-namespace` → standalone `pipelex-temporal` console script — = commit **`989c9beed`**; all gates green (`make agent-check`, `make tb`, `make agent-test` "All tests passed", Temporal suite **156 passed / 4 xpassed**, both `--help` smokes). The former **D3 is superseded** (see D1–D7). Full as-built + open follow-ups: [`../../plugins/option-a-drop-cli-command-seam.md`](../../plugins/option-a-drop-cli-command-seam.md) and the **Option A — as-built** note in the log below. **Phase 4 done** (model_lists, the 5th seam, per D6 — full record in `### Phase 4 — as-built` below; all gates green, committing at this checkpoint). With it, **core names no integration across every enumerated seam**; three pre-existing *unenumerated* core→vendor couplings remain, dispositioned in [`../../plugins/phase-4-residual-core-vendor-couplings.md`](../../plugins/phase-4-residual-core-vendor-couplings.md). **Phase 5 Step 0b (C6) DONE** — the Temporal config *schema* relocated out of `pipelex.temporal` into core (`pipelex/system/configuration/config_temporal.py` + its two exceptions in `…/exceptions.py`); `configs.py` no longer imports `pipelex.temporal` (**the last hard core→temporal import is gone**). In-`pipelex`, behavior-neutral (byte-equivalent exception semantics), zero cross-repo blast radius; all gates green (`make tb` · `make agent-check` · `make agent-test` "All tests passed"). Record in `### Phase 5 — Step 0b (C6) as-built` below. **Phase 5 LOCAL cut-over DONE = 2026-06-20** (user go/no-go for the reversible local work only): `pipelex-temporal` scaffolded + populated (top-level `pipelex_temporal`, D8) + **green** (`agent-check` 0/0, `agent-test` "All tests passed"); **core green without temporal** (`agent-check` clean, `tb` green, full `agent-test` 7678 passed); Option-A config-loader dead code removed; docs hygiene (install instructions → `pipelex-temporal`). Downstream pin flips **DEFERRED — publish/release-gated** (cut-list in the `### Phase 5 — cross-repo cut-over as-built` note). Local commits both repos, **NOT pushed**. **Next = downstream wiring** — **CORRECTION 2026-06-21: `pipelex-temporal` is PRIVATE / not-open-source → NOT PyPI.** It's distributed by a pinned `git+ssh` ref (precedent: `pipelex-shared @ git+ssh://…/infra-python-tools.git@<ref>` in `pipelex-platform`). Forced split: public MIT consumers (`pipelex-api`, `pipelex-mistralai-workflows`) **DROP** the `temporal` extra; only **private** infra pins `pipelex-temporal` via git+ssh: `pipelex-worker` **and `pipelex-api-hosted`** (confirmed 2026-06-21 — the hosted runner enqueues Temporal, so its child image needs the client too). Core `pipelex` stays public → normal release. Gated on the **hosted-Temporal rollout**; **user chose to DEFER all wiring (docs-only)** 2026-06-21. **Finalize-local pass (session 2, 2026-06-20):** core cleaned of the moved Temporal dev-tooling (both `temporal-e2e-validate` *and* the not-previously-listed `temporal-test-crate` skills relocated to `pipelex-temporal`; dead Makefile temporal targets removed), `pipelex-temporal` additions-side `/code-review` → **SHIP/clean**, review NITs triaged (stray `.pipelex/traces` untracked+gitignored; `test_extras→tests` back-import deferred), all gates green both repos; **local, uncommitted, still paused before publish.** Record: [`../../plugins/phase-5-cutover-review-followups.md`](../../plugins/phase-5-cutover-review-followups.md) → *Finalize-local pass*.--> **Delivery model for this branch.** Per the driving goal, all phases land on `refactor/Plugins` as **one commit per checkpoint** (not one PR per phase), and a **single PR to `dev`** opens once the tracker is done. Each phase's "PR: …" box below is satisfied by its checkpoint commit; the standalone-PR-per-phase wording in the original plan is superseded.--This is the **execution tracker**. The *why* and *how* live in [`../../plugins/implementation-plan.md`](../../plugins/implementation-plan.md) (the reviewed, decision-locked plan), built from [`../../plugins/design.md`](../../plugins/design.md) (the single decided design). **Read the implementation-plan before starting any phase** — this file tracks progress against it, it does not replace it. Background assessment: [`../../plugins/README.md`](../../plugins/README.md), plus the per-area notes (`inference-backends-as-plugins.md`, `orchestrators-as-plugins.md`, `temporal-as-plugin.md`).--> **How to use this file.** Tick boxes as you land work. **Do not skip a `🛑 CHECKPOINT`** — each one is a hard stop with three mandatory actions (verify · capture cold-start context · fan-out `/code-review`). The plan is one PR per phase; each phase lands green on its own.-------## Cold-start primer (read this first if you're new to the session)--**Goal of the whole effort:** invert the *last* coupling between core and its optional integrations so **core names no integration — not by import, not by string — anywhere**. Two families ride one shared seam: inference *driver* plugins (the SDK `match` statements) and orchestrator *strategy* plugins (Temporal, Mistral).--**The seams to invert** (verified present; line numbers drift, locations hold):--| Seam | File | Today |-|---|---|---|-| Inference dispatch (×4) | `cogt/{llm,img_gen,extract,search}/*_worker_factory.py` | `match plugin.sdk:` → lazy import + `find_spec` guard + client cache + build worker |-| ~~Model listing (5th seam)~~ | `cogt/model_backends/model_lists.py` | **INVERTED — Phase 4.** `match sdk` → `get_model_lister_registry().get_optional(sdk=…)`; each backend plugin registers an optional `ListModelsFn` via `add_model_lister`. File names no integration. |-| Orchestrator dispatch | `runtime_bridge/bridge.py` (`match` ~`:147`, `_run_mistral_native` `:408`, `_run_direct` `:270`, `_run_temporal_*` `:297/:330`) | `match execution_mode:` → hard Mistral import + lazy Temporal + per-mode install messages |-| Boot/teardown hub swap | `pipelex.py` (~`:377-479`) | four `if get_config().temporal.is_enabled:` blocks + inlined teardown |-| ~~Config~~ | `system/configuration/configs.py:14` | **RELOCATED — Phase 5 Step 0b (C6).** Schema → `pipelex/system/configuration/config_temporal.py`; its two exceptions (`TemporalConfigError`, `WorkerTaskQueueUnknownError`) → `pipelex/system/configuration/exceptions.py` (re-based on `PipelexError`, byte-equivalent). `configs.py` imports from core — the one hard `pipelex.temporal` import in core config is gone. |-| ~~CLI~~ | `pipelex/temporal/temporal_cli.py` | **REMOVED — Option A.** No CLI-command seam/harvest at all; Temporal's `worker` + `setup-namespace` ship as the standalone `pipelex-temporal` console script ([doc](../../plugins/option-a-drop-cli-command-seam.md)). |-| Naming overload | `plugins/{plugin,plugin_manager,plugin_sdk_registry,plugin_factory_abstract}.py`, `hub.py` `"PluginManager2 is not initialized"` | three-way "plugin" overload to kill |--**Locked decisions (D1–D7)** — do not re-litigate, see plan for full text:--- **D1** — inference factory = a typed callable `MakeWorkerFn`, not a one-method Protocol object (kills ~30 classes).-- **D2** — Phase 1 PR = seam machinery **merged** with the LLM family (contract meets a real backend before locking).-- **D3** — ~~plugin CLI commands harvested by running the pure `build_registrar` at CLI-build~~ **SUPERSEDED → Option A** ([`../../plugins/option-a-drop-cli-command-seam.md`](../../plugins/option-a-drop-cli-command-seam.md), landed post-Phase-3): the CLI-command *contribution path* was **removed**, not hardened. Temporal's `worker` + `setup-namespace` are now a standalone `pipelex-temporal` console script (`[project.scripts]`), so `pipelex --help` no longer loads config or scans entry points and a broken/colliding plugin can't brick the host CLI. The inference-backend / orchestrator / boot-slot seams and `pipelex plugins list` are **unchanged**; `add_cli_command` / `CliCommand` are gone from the registrar. Downstream `pipelex-worker` Dockerfile/Makefile flip is release-gated (see the plan's cross-repo section).-- **D4** — Phase 0 keeps `SdkClientManager` a **pure rename**; collapsing it into the registry is a **P3 follow-up commit**, not bundled.-- **D5** — hub slot claims are **deferred thunks** (`claim_*(factory: Callable[[], Impl])`), invoked only at the setup apply-point → `register` stays import-light (never imports `temporalio`).-- **D6** — the 5th seam (`model_lists.py`) is **inverted too**, as Phase 4, via a model-listing capability on the inference contract.-- **D7** — control surface: discovery is the source of truth for *presence*; add `pipelex plugins list` (observability) + a narrow `plugins.disabled` denylist; **no** redundant allowlist.--**Enablement model** (so you don't add a flag that shouldn't exist): *presence* = in `BUILTIN_PLUGINS` or an installed dist with the `pipelex.plugins` entry point (zero config). *Activation* is decomposed by registry shape — inference backends activate by **data** (a model names the `sdk`); orchestrators select per-call by `execution_mode`; boot-global orchestrators are claimed only when the plugin's own config slice says so (`temporal.is_enabled` = "boot *this process* as a Temporal runtime", a per-process gate, **not** a generic plugin on/off).--**Invariants that must survive every phase:**--- `register(registrar)` is **side-effect-free** — may only call registrar menu methods (no hub touch, no I/O, no client/SDK construction). This is what makes `build_registrar` safely repeatable (D3 runs it at CLI-build *and* boot).-- **Import-light boot** — booting with built-ins registered must not import any optional SDK (`anthropic`, `mistralai`, `google.genai`, `boto3`, `fal_client`, …) or `temporalio`. Enforced by the subprocess import-blocker guard (lands Phase 1).-- **Fail loud** — duplicate `(family, sdk)` / duplicate mode / double-claimed slot / version mismatch / broken entry point each raise a named, contextual error. A missing backend says "install X", never derefs `None`.--**Commands** (from `_plugins/CLAUDE.md`): `make agent-check` (lint/types — always after changes) · `make agent-test` (full suite, silent on success — at end of every phase) · `make tb` (boot/config sanity — fast, run after config or seam edits) · `make agent-test-debug` / `make atd` (when the suite hangs). Always use the venv: `.venv/bin/...`.--**PR boundaries:** Phase 0 · Phase 1 (merged seam+LLM) · Phase 2 (one sub-PR per family) · Phase 3 · Phase 4 — all `pipelex`-only, each lands green. Phase 5 is cross-repo, gated on Checkpoint 3.-------## Phase 0 — Rename (free the word "plugin")--**Goal:** kill the three-way "plugin" overload. Pure rename, **no behavior change**. Own PR.--- [x] **Grep the blast radius first** (codex C11 — it's bigger than the four factories): `grep -rn 'import Plugin\b' pipelex/` and `grep -rn 'plugin_sdk_registry\|plugin_manager\|PluginManager\|PluginSdkRegistry\|PluginFactoryAbstract' pipelex/`. Note every adapter module (`openai`, `bedrock`, `anthropic`, `gateway`, `portkey` list+factory) and `search_worker_factory.py` (uses the *correct* `plugins.plugin` path while llm/img_gen/extract use the *wrong* `plugins.plugin_sdk_registry` path).-- [x] `Plugin` (`plugins/plugin.py`) → `ModelHandle` (`plugins/model_handle.py`). Keep `make_for_inference_model`. It's a backend *selector* (`sdk`+`backend`+`variant`+`sdk_handle`), not a plugin.-- [x] `PluginManager` (`plugins/plugin_manager.py`) → `SdkClientManager` (`plugins/sdk_client_manager.py`).-- [x] `PluginSdkRegistry` (`plugins/plugin_sdk_registry.py`) → `SdkClientRegistry` (`plugins/sdk_client_registry.py`). Rename `get_sdk_instance`/`set_sdk_instance` → `get`/`set` (the DRY `get_or_create` lands in Phase 1, **not here**).-- [x] `PluginFactoryAbstract` (`plugins/plugin_factory_abstract.py`, the `make_extras` helper — unrelated to the new system) → `BackendExtrasFactory` (`plugins/backend_extras_factory.py`). Name-hygiene; it collides with the new vocabulary.-- [x] `hub.py`: `get_plugin_manager`/`set_plugin_manager` → `get_sdk_client_manager`/`set_sdk_client_manager`; **fix the `"PluginManager2 is not initialized"` message** → `"SdkClientManager is not initialized"`.-- [x] `pipelex.py`: update setup/teardown call sites.-- [x] Fix the wrong-path imports (llm/img_gen/extract → corrected registry path) — free win during the rename.-- [x] **D4 guard:** `SdkClientManager` stays a thin wrapper holding one `SdkClientRegistry`. Do **not** collapse it here — that's a tracked P3 follow-up (see "Deferred / out of scope").--**Tests:** no new tests. Existing suite + `make tb` stay green.--### 🛑 CHECKPOINT 0 — hard stop--- [x] **Verify:** `make agent-check` clean · `make tb` green · `make agent-test` green. Confirm `PluginManager2` artifact gone (`grep -rn PluginManager2 pipelex/` → empty) and wrong-path `Plugin` imports fixed.-- [x] **Capture cold-start context:** see `### Phase 0 — as-built` in the As-built log.-- [x] **Fan-out `/code-review`:** sub-agent ran `/code-review --fix` on the Phase 0 diff (pure-rename scope). Findings triaged in the as-built note.-- [x] **PR:** superseded — committed on `refactor/Plugins` (single PR to `dev` opens after the tracker is done).-------## Phase 1 — Seam machinery + LLM family (merged, per D2)--**Goal:** build discovery + contract + registry machinery **and** migrate the LLM family through it in one PR, so the contract is proven against a real, messy family before it locks. No behavior change. Own PR.--> ✅ **DONE — committed on `refactor/Plugins`.** Full record in **`### Phase 1 — as-built`** (As-built log). The unchecked boxes below are the original plan items; all are delivered (or noted as deferred to Phase 2/3) in the as-built.--**New modules (all import-light — must not import any backend SDK or `temporalio`):**--- [ ] `plugins/contract.py` — `PipelexPlugin` (`@runtime_checkable Protocol`: `name: str`, `targets_api: int`, `register(self, registrar) -> None`) + `PLUGIN_API_VERSION: int = 1`. Document the **side-effect-free `register`** invariant here.-- [ ] `plugins/registrar.py` — `PluginRegistrar` accumulator: inference registry, orchestrator registry, **deferred** hub-slot-claim thunks, CLI-command list, teardown-callback list, read-only `config: ConfigRoot`. Menu methods:- - [ ] `add_inference_backend(*, family, sdk, make_worker: MakeWorkerFn)` (D1 — callable).- - [ ] `add_orchestrator(*, mode, orchestrator)`.- - [ ] `claim_content_generator | pipe_router | pipe_run | task_manager(factory: Callable[[], Impl])` (D5 — **thunk, not instance**).- - [ ] `add_cli_command(*, name, help, command)`, `add_teardown(callback)`.-- [ ] `plugins/inference_backend_registry.py` — `InferenceFamily` (StrEnum: `LLM`/`IMG_GEN`/`EXTRACT`/`SEARCH`), `MakeWorkerFn` TypeAlias, `InferenceBackendRegistry` keyed by `(family, sdk)`. Signature: `make_worker(*, inference_model, backend, sdk_clients, reporting_delegate)`.-- [ ] `plugins/orchestrator_registry.py` — `OrchestratorProtocol` (`async def run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput`), `OrchestratorRegistry` keyed by `PipelexExecutionMode`.-- [ ] `plugins/discovery.py` — `build_registrar(*, config) -> PluginRegistrar`: **pure function** iterating `BUILTIN_PLUGINS` + `importlib.metadata.entry_points(group="pipelex.plugins")`, version-checking each, calling `plugin.register(registrar)`, skipping any `name` in `config.plugins.disabled` (logged; raise if a core-unconditional plugin is denylisted), fail-loud. Pure ⇒ safe at CLI-build and boot.-- [ ] `plugins/builtins.py` — `BUILTIN_PLUGINS`. Phase 1 fills it with the LLM drivers.-- [ ] `plugins/exceptions.py` — `PluginApiVersionMismatchError`, `DuplicateInferenceBackendError`, `DuplicateOrchestratorError`, `HubSlotAlreadyClaimedError`, `BrokenPluginError`. (`MissingOrchestratorError` lives in `runtime_bridge/exceptions.py` — added in Phase 3.)--**DRY the two repeated blocks (design §6.2):**--- [ ] `SdkClientRegistry.get_or_create(*, handle, build)` replaces the `get_sdk_instance(...) or set_sdk_instance(...)` dance in every arm.-- [ ] `require_sdk(*, spec, extra, msg)` (raises `MissingDependencyError` if `find_spec(spec) is None`) replaces the repeated guard — moved **into** `make_worker` so a missing extra fails at use, not boot.--**LLM family migration:**--- [ ] Collapse `llm_worker_factory.py`'s `match` to `get_inference_backend_registry().lookup(family=LLM, sdk=model_handle.sdk)(...)`. A registry miss → distinct "backend `<sdk>` not registered; is its plugin installed?" error.-- [ ] Each LLM driver plugin's `register` calls `add_inference_backend(family=LLM, sdk="anthropic", make_worker=<closure>)`. Add them to `BUILTIN_PLUGINS`.--**Boot wiring:**--- [ ] In `Pipelex.setup()`, after config is fully resolved (incl. the `temporal_enabled` override at `pipelex.py:193-196`) and before the hub setup points, call `build_registrar(config=get_config())`. Store the two keyed registries on the hub (`set_inference_backend_registry` / `set_orchestrator_registry`). Hold slot-claim thunks, CLI commands, teardown callbacks for their apply-points.-- [ ] **Conflict & version policy (fail loud):** duplicate `(family, sdk)` / duplicate mode / double-claimed slot each raise naming **both** contributors; `targets_api != PLUGIN_API_VERSION` → `PluginApiVersionMismatchError` with the remedy; a broken entry point → `BrokenPluginError` (the one sanctioned `except Exception` site — annotate it).--**Control surface (D7):**--- [ ] Add a core-owned `plugins: PluginsConfig` section (`disabled: list[str]`) to `configs.py`; default `[]` in `pipelex/pipelex.toml` **and** `.pipelex/pipelex.toml` (never a class-def default — repo rule). Apply the denylist in `build_registrar` (skip + log; error on a core-unconditional name).-- [ ] Add the core `pipelex plugins list` command — lists discovered plugins (built-in vs external), what each registered, denylist state.--**Tests (red-green):**--- [ ] Contract conformance (synthetic plugins): protocol satisfaction; `targets_api` mismatch fails loud; duplicate `(family, sdk)` / mode / slot each raise naming both; broken entry point → `BrokenPluginError`; **`build_registrar` idempotent** (two calls, no side effect — pins the side-effect-free invariant).-- [ ] Denylist (D7): a discovered non-core plugin in `plugins.disabled` is skipped + logged; denylisting a core-unconditional plugin raises at startup; `pipelex plugins list` shows built-ins + synthetic externals with contributions + denylist state.-- [ ] **Import-light boot** (codex C12): subprocess with a `sys.meta_path` finder that *raises* on `anthropic`/`mistralai`/`google.genai`/`boto3`/`fal_client`/… → boot with built-ins → assert no raise.-- [ ] Registry round-trip (LLM): register → `lookup(LLM, sdk)` → right `LLMWorkerAbstract`.-- [ ] Error parity: an Anthropic model with the `anthropic` extra absent raises the *same* `MissingDependencyError` text (lib + extra + alternative).--### 🛑 CHECKPOINT 1 — hard stop (contract locks here)--- [x] **Verify:** `make agent-check` clean · `make tb` green · `make agent-test` green. Import-light subprocess guard green; the LLM `match` is gone (replaced by a registry lookup).-- [x] **Capture cold-start context:** see `### Phase 1 — as-built` in the As-built log.-- [x] **Fan-out `/code-review`:** sub-agent ran `/code-review --fix` on the Phase 1 diff (see as-built for triage).-- [x] **PR:** superseded — committed on `refactor/Plugins` (single PR to `dev` opens after the tracker is done).-------## Phase 2 — Remaining inference families (ImgGen / Extract / Search)--**Goal:** migrate the other three families onto the proven contract. **One family per sub-PR** (lanes A/B/C are independent — can run in parallel worktrees).--> **Pre-work analysis (done — read this before starting; the three factories were already audited so you don't have to re-read them).** Each closure follows the Phase 1 LLM shape: `def _make_x(*, inference_model, backend, sdk_clients, reporting_delegate) -> InferenceWorkerAbstract`, derives `model_handle` from `inference_model`, lazy-imports its factory/worker (`# noqa: PLC0415`), guards optional SDKs with `require_sdk(...)` inside the closure, and caches via `sdk_clients.get_or_create(handle=model_handle, build=lambda: ...)`. Then the family factory (`ImgGen/Extract/Search WorkerFactory`) collapses its `match` to `get_inference_backend_registry().lookup(family=<FAM>, sdk=model_handle.sdk)(...)` exactly like `llm_worker_factory.py` already does, passing `sdk_clients=get_sdk_client_manager().sdk_client_registry` and `cast`ing the result to the family worker abstract. Registry-miss `NotImplementedError` message comes from `InferenceBackendRegistry.lookup` (the per-family "is not supported" / "for image generation" / "for search" wording in the old `case _:` arms goes away — update those families' `test_*_worker_factory.py` unknown-sdk asserts to the new "Is its plugin installed?" wording, as was done for LLM).->-> **SDK → (vendor plugin · worker · client build) map:**->-> *IMG_GEN* (`img_gen_worker_factory.py`):->-> - `gateway_img_gen` → **gateway** plugin · `GatewayImgGenWorker` · `GatewayFactory.make_portkey_client(backend=backend)`-> - `gateway_completions` → **gateway** · `OpenAICompletionsImgGenWorker(openai_completions_factory=GatewayCompletionsFactory(is_http_url_enabled=False))` · `GatewayCompletionsFactory.make_portkey_openai_client_for_completions(model_handle, backend)`-> - `openai_img_gen` → **openai** · `OpenAIImgGenWorker` · `OpenAIClientFactory.make_openai_client(model_handle, backend)`-> - `google` → **google** · `GoogleImgGenWorker` · `GoogleFactory.make_google_client(backend)` · `require_sdk(spec="google.genai", dependency_name="google-genai", extra="google", msg="The google-genai SDK is required in order to use Google Gemini Image models.")`-> - `fal` → **NEW `fal` plugin** · `FalImgGenWorker` · `fal_client.AsyncClient(key=backend.api_key)` (import `from fal_client import AsyncClient`) · `require_sdk(spec="fal_client", dependency_name="fal-client", extra="fal", msg="The fal-client SDK is required in order to use FAL models (generation of images).")`-> - `huggingface_img_gen` → **NEW `huggingface` plugin** · `HuggingFaceImgGenWorker` · `AsyncInferenceClient(provider=provider_literal, token=backend.api_key)` where `provider_literal = HuggingFaceFactory.make_huggingface_inference_provider(provider_str=model_handle.variant)` if `model_handle.variant` else `"auto"` · **C10 BUG FIX: add `require_sdk(spec="huggingface_hub", extra="huggingface", msg="The huggingface_hub SDK is required in order to use HuggingFace image generation models.")`** (the old arm had no guard) — and add the huggingface error-parity test.-> - `blackboxai_img_gen` → **NEW `blackboxai` plugin** · `OpenAICompletionsImgGenWorker(openai_completions_factory=BlackboxaiCompletionsFactory(is_http_url_enabled=True))` · `OpenAIClientFactory.make_openai_client(model_handle, backend)`-> - `openrouter_img_gen` → **NEW `openrouter` plugin** · `OpenAICompletionsImgGenWorker(openai_completions_factory=OpenRouterCompletionsFactory(is_http_url_enabled=True))` · `OpenAIClientFactory.make_openai_client(model_handle, backend)`-> - `azure_rest_img_gen` → **NEW `azure_rest` plugin** · `AzureImgGenWorker(model_handle=model_handle, inference_model=..., reporting_delegate=...)` — **DIRECT construction: no `sdk_clients`/`get_or_create`** (its `test_img_gen_worker_factory.py` asserts `registry.root == {}`).->-> *EXTRACT* (`extract_worker_factory.py`) — workers take `extra_config=backend.extra_config`:->-> - `gateway_extract` → **gateway** · `GatewayExtractWorker` · `GatewayFactory.make_portkey_client(backend)`-> - `mistral` → **mistral** · `MistralExtractWorker` · `MistralFactory.make_mistral_client(backend)` · `require_sdk(spec="mistralai", extra="mistral", msg="The mistralai SDK is required in order to use Mistral OCR models through the mistralai client.")`-> - `docling_sdk` → **NEW `docling` plugin** · `DoclingExtractWorker` · `DoclingFactory.make_docling_sdk()` · `require_sdk(spec="docling", extra="docling", msg="The docling library is required in order to use Docling for PDF and image text extraction.")`-> - `pypdfium2` → **NEW `pypdfium2` plugin** · `Pypdfium2Worker(extra_config=..., inference_model=..., reporting_delegate=...)` — **STATELESS: no `get_or_create`, no `require_sdk`** (built directly).-> - `linkup_fetch` → **`linkup` plugin** · `LinkupExtractWorker(extra_config=...)` — **STATELESS (no `get_or_create`)** · `require_sdk(spec="linkup", extra="linkup", msg="The linkup SDK is required in order to use Linkup Fetch extraction models.")`->-> *SEARCH* (`search_worker_factory.py`) — **C9 normalization: `make_search_worker` currently takes NO `reporting_delegate` and pulls `get_report_delegate()` from the hub twice. Change its signature to accept `reporting_delegate` and pass it through (drop the hub coupling); update its callers.** Closures then receive `reporting_delegate` like every family.->-> - `linkup` → **`linkup` plugin** · `LinkupSearchWorker(inference_model=..., reporting_delegate=...)` — STATELESS (no client/`get_or_create`).-> - `gateway_search` → **gateway** · `GatewaySearchWorker` · `GatewayFactory.make_portkey_client(backend)`->-> **NEW vendor plugin modules to create:** `fal`, `huggingface`, `blackboxai`, `openrouter`, `azure_rest`, `docling`, `pypdfium2`, `linkup` (each `pipelex/plugins/<vendor>/<vendor>_plugin.py`, added to `BUILTIN_PLUGINS`). **EXTEND existing plugins** (one `register` adds across families — the cross-family-vendor coordination point): `openai` (+IMG_GEN), `gateway` (+IMG_GEN ×2, +EXTRACT, +SEARCH), `google` (+IMG_GEN), `mistral` (+EXTRACT), `linkup` (EXTRACT + SEARCH — one new plugin serving two families). None of these new SDKs are core-unconditional (only `openai` is). **Build-time check:** `azure_rest`/`pypdfium2`/`linkup`/blackboxai/openrouter closures must stay import-light at module load (lazy factory/worker imports), and the import-light subprocess guard's BLOCKED set should grow to cover `fal_client`, `huggingface_hub`, `docling`, `linkup` so the guard actually proves it.--- [x] **ImgGen (Lane A).** Collapse the `match`. **Fix the pre-existing huggingface bug (codex C10):** `img_gen_worker_factory.py` imports `huggingface_hub` with **no `find_spec` guard` → a missing extra raises raw `ImportError` instead of `MissingDependencyError`. Add `require_sdk(spec="huggingface_hub", extra="huggingface", …)` during migration. Provider-literal handling (`model_handle.variant` → provider) goes inside the closure. Substrate-reuse workers (`blackboxai`/`gateway_completions`/`openrouter` → `OpenAICompletionsImgGenWorker`) become closures capturing the per-vendor completions factory.-- [x] **Extract (Lane B).** Collapse the `match`. Stateless arms (`pypdfium2`, `linkup_fetch`) skip `get_or_create`.-- [x] **Search (Lane C) — normalize the call surface (codex C9).** `search_worker_factory.py` takes **no `reporting_delegate` param** and pulls `get_report_delegate()` from the hub; the uniform `MakeWorkerFn` passes it explicitly. Normalize `make_search_worker` + callers to accept `reporting_delegate` (removes search's hidden hub coupling — a strict improvement), so search fits the uniform signature.-- [x] Add each migrated driver to `BUILTIN_PLUGINS`. OpenAI is an always-on built-in driver, no privileged arm. The OpenAI-compat *substrate* stays an in-tree library the gateway/portkey/openrouter/blackboxai closures import (substrate extraction into a named module is deferred).--**Tests:**--- [x] Registry round-trip per family.-- [x] Import-light subprocess guard now covers all optional SDKs.-- [x] **huggingface error-parity** (missing `huggingface` extra now raises `MissingDependencyError` — proves the bug fix).-- [x] **Cross-family vendor** (e.g. `mistral` registers into both LLM and Extract from one `register`).--**Cross-cutting deliverable (ships with this phase):**--- [x] **Inference SPI reference** (design §9.1) + **plugin-authoring guide** + **minimal example backend plugin** — delivered as one comprehensive page `docs/under-the-hood/inference-backend-plugins.md` (seam walkthrough + SPI module/symbol table + authoring guide with a copy-pasteable `acme` example, the entry-point declaration, the denylist, and the fail-loud table), added to both mkdocs nav blocks under "Under the Hood". A standalone runnable example *package* was judged disproportionate for this checkpoint (the embedded example is complete and copy-pasteable) — flag as a follow-up if a separate example repo is wanted.--### 🛑 CHECKPOINT 2 — hard stop (natural session handoff — inference done)--- [x] **Verify:** `make agent-check` clean · `make agent-test` green (exit 0). All four worker-factory `match` statements gone, every backend goes via the registry, the huggingface guard bug is fixed (require_sdk added), and search's hub coupling is removed (factory takes `reporting_delegate`, caller supplies `get_report_delegate()`).-- [x] **Capture cold-start context:** `### Phase 2 — as-built` note below — vendor plugins created/extended, cross-family coordination points, the search C9 normalization, test + docs locations.-- [x] **Fan-out `/code-review`:** `pr-review-toolkit:code-reviewer` sub-agent reviewed the full Phase 2 working-tree diff, comparing every new closure against the pre-Phase-2 `match` arm via `git show HEAD:…`. **Verdict: clean — no blockers, no should-fix, no nits.** It confirmed byte-equivalent parity (worker classes, ctor kwargs, `is_http_url_enabled` flags), the C10 require_sdk fix + TYPE_CHECKING-only `PROVIDER_OR_POLICY_T`, import-light (no SDK at any plugin module top), every `# noqa: ARG001` justified by the old arm, C9 production-safety (real caller still supplies the delegate; the two integration tests assert on results not reporting so `None` is harmless), and cross-family non-collision (distinct `(family, sdk)` keys + distinct closure names).-- [x] **Commit:** single checkpoint commit on `refactor/Plugins` (per the branch's one-commit-per-checkpoint delivery model). **Good point to end the session** before opening the orchestrator phase.-------## Phase 3 — Orchestrators through the seam (in-tree) — HIGHEST RISK--**Goal:** collapse the bridge `match` + the four `temporal.is_enabled` boot blocks into registry/slot lookups; move Temporal's modes/CLI/boot-swap/teardown behind its (still-in-repo) plugin; flip Mistral's hard import to discovery; publish the orchestrator SPI. No behavior change. Own PR.--> **Spike the CLI-timing + thunk integration (D3+D5) FIRST** — it's the one genuinely novel integration point. Prove `build_registrar` at CLI-build harvests `worker` into `--help` without constructing a Temporal impl, before doing the rest.--**Extract orchestrators (verbatim bodies):**--- [x] `DirectOrchestrator.run` from `_run_direct` → `runtime_bridge/direct_orchestrator.py`. **Landmine kept verbatim:** `with scoped_pipe_router(PipeRouter())`. Registered in core by `plugins/direct/direct_plugin.py` (`name="direct"`, in `CORE_UNCONDITIONAL_PLUGIN_NAMES`).-- [x] `TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` from `_run_temporal_*` → `temporal/temporal_orchestrators.py`, keeping the `WorkflowExecutionError` catch + `make_workflow_id` recompute. Each `run` first calls `_require_temporal_extra(mode=...)` (find_spec → `MissingOrchestratorError` if the extra is absent).-- [x] (`MistralWorkflowsOrchestrator` authored in `pipelex-mistralai-workflows` — Phase 5.)--**Bridge + Mistral:**--- [x] Collapsed the bridge `match` → `get_orchestrator_registry().get_optional(mode=...)` → `orchestrator.run`, else `MissingOrchestratorError(mode)`. `bridge.py` names no integration (only docstrings/comments mention temporal/mistral). Serialize helpers extracted to `runtime_bridge/serialization.py` (breaks the `bridge→bootstrap→pipelex` cycle that the orchestrators would otherwise re-form); `serialize_pipe_output` re-exported from bridge for back-compat.-- [x] **Per-mode error quality (C7):** `MissingOrchestratorError(*, mode)` maps mode→exact message via match/case (temporal modes → `pip install 'pipelex[temporal]'`; mistral → `pip install pipelex-mistralai-workflows`; direct → boot-problem msg). `_authors_caller_facing_message=True`.-- [x] **Mistral → discovery:** the `_run_mistral_native` hard import is gone. MISTRAL_NATIVE has no in-core orchestrator → registry miss → `MissingOrchestratorError(MISTRAL_NATIVE)`. Suite proves dispatch with a fake orchestrator (`test_orchestrator_dispatch.py`).--**The Temporal plugin (`temporal/temporal_plugin.py`), its `register`:**--- [x] **always** `add_orchestrator(TEMPORAL_BLOCKING|TEMPORAL_FIRE_AND_FORGET, …)` + `add_cli_command(worker|setup-temporal-namespace, import_path=…)`.-- [x] **if `config.temporal.is_enabled`:** `claim_content_generator|task_manager|pipe_router|pipe_run(thunk)` + `add_teardown(thunk)` — each a module-level zero-arg thunk that lazy-imports `temporalio`; `register` imports no temporalio (proven by the import-light guard with `is_enabled=True`).--**Boot/teardown collapse (C8 — explicit injection precedence):**--- [x] The four `if temporal.is_enabled:` boot blocks → `_resolve_hub_slot(slot, default)` (generic over `_HubSlotImplT`) at each ordered point. Precedence: explicit `setup()` param > slot-claim thunk > core default. TASK_MANAGER runs its thunk (full temporal-hub wiring); no core default/param. Pinned by `test_hub_slot_injection_precedence.py`.-- [x] The inlined teardown → registered teardown callbacks run **LIFO** (`for cb in reversed(self._plugin_registrar.teardown_callbacks)`). `pipelex.py` teardown names no integration. `temporal_hub.get_optional_task_manager()` added so the temporal teardown thunk re-fetches the tm.--**CLI collapse (C4 + D3):**--- [x] The two hard temporal-command imports + static registrations + hardcoded `list_commands` entries are gone. `_cli.py` harvests plugin commands at module load via `build_registrar` (`_register_discovered_cli_commands`), importing each `import_path` lazily; `list_commands` returns `_CORE_COMMAND_ORDER + _PLUGIN_COMMAND_NAMES`.-- [x] CLI-build config load is side-effect-free (`config_manager.load_config(ensure_global_if_missing=False)`, new param) with a bulletproof `load_base_config_dict()` fallback so a broken/absent user config can't brick `--help`/`init`. D5 thunks ⇒ harvest constructs no temporal impl (proven: `worker` harvested with temporalio import-blocked).--**SPI:**--- [x] Orchestrator SPI published: `docs/under-the-hood/orchestrator-plugins.md` (seam, contract, boot-orchestrator slot claims, injection precedence, CLI-by-import-path, the SPI module/symbol table from design §9.2, the Temporal worked example, out-of-tree authoring) — added to both mkdocs nav blocks. `MissingOrchestratorError` added to `runtime_bridge/exceptions.py`; `MissingPipelexTemporalExtraError` + `MissingMistralWorkflowsPluginError` removed (error pages regenerated).--**Tests:**--- [x] Bridge dispatch by mode (fake orchestrator) — `test_orchestrator_dispatch.py`.-- [x] **Per-mode error parity** — `test_orchestrator_dispatch.py` (per-mode miss) + `test_exceptions_disclosure.py` (per-mode hint survives STRICT).-- [x] DIRECT router scoping (`test_direct_router_scoping.py`, unchanged + green) + DIRECT parity (`test_dispatch.py`, `test_bridge_direct.py`).-- [x] **Injection-precedence** (explicit param wins over slot claim — C8) — `test_hub_slot_injection_precedence.py`.-- [x] Boot-via-slots (temporal-enabled → content-generator slot resolves to `ContentGeneratorInWorkflow`) — existing `test_keyless_boot_forced_dry.py::test_keyless_temporal_boot_*` (green through the new seam) + slot-claim arm in the precedence test; teardown LIFO pinned there.-- [x] **CLI-command contribution** (`worker`/`setup-temporal-namespace` harvested, in order) — `test_plugin_cli_command_harvest.py`. Import-light guard extended with `temporalio` + `is_enabled=True` slot-claim arm.-- [x] **Full Temporal suite green in-tree** (§14.5 — the extraction gate) — `.venv/bin/pytest tests/integration/pipelex/temporal/` → **156 passed, 4 xpassed** (the xpassed are pre-existing xdist class-registration flakiness markers, unrelated to this phase); also green inside the full `make agent-test`.--### 🛑 CHECKPOINT 3 — hard stop (THE gate before externalization)--- [x] **Verify:** `make agent-check` clean · `make agent-test` green ("All tests passed", exit 0) · **full Temporal suite green through the seam** (`.venv/bin/pytest tests/integration/pipelex/temporal/` → 156 passed, 4 xpassed). bridge + boot/teardown + CLI name **no** integration (only docstrings/comments mention temporal/mistral; dispatch/boot/teardown go via the registry + slot thunks + config).-- [x] **Capture cold-start context:** `### Phase 3 — as-built` note above — dispatch seam, verbatim extractions, the cycle-break (serialization.py + `import_path` CLI), injection-precedence ordering + where slot thunks apply, the CLI harvest mechanism, the orchestrator SPI doc, and the §14.5 green evidence.-- [x] **Fan-out `/code-review`:** `pr-review-toolkit:code-reviewer` reviewed the full Phase 3 diff against `HEAD` — **clean, no BLOCKERs**; byte-equivalence + all landmines verified. S1/N1/N2 applied, N3 deliberate (see the CHECKPOINT-3 `/code-review` note in the as-built).-- [x] **Commit:** checkpoint commit `19e6ca66b` on `refactor/Plugins-2` (one-commit-per-checkpoint delivery model). **Do NOT begin Phase 5** until this Temporal-green gate is recorded (it is, above). Phase 4 (model_lists) may start next.-------## Phase 4 — Invert model listing (the 5th seam, per D6)--**Goal:** close the goal. `model_lists.py`'s `match sdk:` (the `list-models` CLI path) still hardcodes `pipelex.plugins.{openai,anthropic,mistral,google,bedrock}.*_list` imports. Invert it so core names no integration *anywhere*. Own PR.--- [x] Add a **model-listing capability** to the inference contract: `add_model_lister(*, sdk, lister: ListModelsFn)` (callable mirroring `MakeWorkerFn` — import-light, lazy). A backend plugin that lists models registers its lister alongside its worker factory.-- [x] Collapse `ModelLister.list_models` (`model_lists.py`) to a registry lookup keyed by `sdk`; a miss → the same friendly "is its plugin installed?" guidance. Move the `find_spec` guards (already in this file) into each lister.-- [x] The contract grows by one **optional** method — backends without listing simply don't register a lister (progressive disclosure preserved).--**Tests:**--- [x] Round-trip (a registered lister is invoked for its `sdk`).-- [x] A backend with no lister → the friendly miss.-- [x] Import-light subprocess guard still green (listers are lazy).--### 🛑 CHECKPOINT 4 — hard stop ("core names no integration" holds without an asterisk)--- [x] **Verify:** `make agent-check` clean (pyright 0 over 2243 files / mypy 0 / keyword-only pass) · `make agent-test` green ("All tests passed.") · targeted plugins+cogt suite 715 passed. `model_lists.py` names no integration; the final sweep returns only **pre-existing, out-of-scope** vendor refs (`config_cogt.py` typed config = by-design D7; `img_gen_args_factory.py` + `backend_factory.py` = genuine but unenumerated couplings) — captured in [`../../plugins/phase-4-residual-core-vendor-couplings.md`](../../plugins/phase-4-residual-core-vendor-couplings.md). No dispatch-path leak remains.-- [x] **Capture cold-start context:** `### Phase 4 — as-built` note below.-- [x] **Fan-out `/code-review`:** `pr-review-toolkit:code-reviewer` reviewed the Phase 4 working-tree diff against HEAD. **Verdict: byte-equivalent and clean — no BLOCKER / no SHOULD-FIX.** It walked all five correctness questions against `git show HEAD:…model_lists.py` and confirmed: each lister's kwargs match its old `match` arm (mistral's no-`backend` + sync `# noqa: RUF029`, bedrock sync, the rest async); the deleted `find_spec` guards survive **byte-for-byte** inside each `list_*_models` fn (same lib/extra/message; openai has none in old or new); the anthropic `AnthropicSDKUnsupportedError`→`ModelListingUnsupportedError` translation is correct and core imports no anthropic symbol; `any_listed` threads identically (not set on the unsupported `continue`); the registered sdk keys are exactly the old arms' set. Plus import-light, `get_optional`-vs-`lookup`, keyword-only, and the Case-2 boundary (with `except PipelexCLIError: raise` ordering preserved so internal auth `PipelexCLIError`s aren't re-wrapped). One **NIT** (doc recommended `require_sdk` while the built-in listers reuse their `list_*_models` inline `find_spec` guard) — **applied**: the "Listing models" doc section now tells one story (recommends `require_sdk`, notes the built-ins reuse the existing inline guard). The wip residual-couplings doc was confirmed accurate. A follow-up **xhigh workflow `/code-review`** (10 finder angles → adversarial verify) re-confirmed byte-equivalence and surfaced reuse/altitude follow-ups: the two cheap ones were **applied** (a shared `PluginRegistrar._add` helper mirroring `_claim`; the hardcoded item-counts removed from this as-built), the four design-tradeoffs **deferred** — all recorded in [`../../plugins/phase-4-review-followups.md`](../../plugins/phase-4-review-followups.md).-- [x] **PR:** checkpoint commit on `refactor/Plugins-2` (one-commit-per-checkpoint). Phases 0–4 complete — the in-`pipelex` work is done.-------## Phase 5 — Externalize Temporal → `pipelex-temporal` (cross-repo, GATED on Checkpoint 3)--**Goal:** lift `pipelex/temporal/` into a separate dist. A *packaging* move — but only after a prerequisite the design missed. **Cross-repo; the only phase that can break downstream deploys. Separate effort.**--> **Destination repo exists.** The user created an **empty** `pipelex-temporal` repo (sibling dir at `../pipelex-temporal`, remote `[email protected]:Pipelex/pipelex-temporal.git`, branch `main`, **no commits / no scaffolding** yet) ahead of this phase, so the move has a target from the first commit. **Do this in a fresh session** (cross-repo; not a normal in-`pipelex` checkpoint).->-> **Progress:** the in-`pipelex` prerequisite **Step 0b (C6) is DONE** (committed; behavior-neutral; gates green). What remains is the **cross-repo cut-over** — Step 0a (scaffold the empty repo), the breaking move of `pipelex/temporal/` out, the downstream pin flips, and publishing. Those are outward-facing/deploy-breaking and were intentionally **paused for user go/no-go** after Step 0b.--> **Cut-over in progress (local-only, paused before publish — user go/no-go given 2026-06-20 for the *reversible* local work; publish/merge/deploy still gated).** Two foundational decisions locked this session:-> - **D8 — package layout = top-level `pipelex_temporal`** (NOT a `pipelex.temporal` namespace stitch). Rationale: `pipelex` is a *regular* package (owns `pipelex/__init__.py`, hatchling `packages=["pipelex"]`), so two wheels can't cleanly co-own `pipelex/`; the family convention is top-level (`pipelex-api`→`api`, the existing plugin→`pipelex_mistralai_workflows`); and two *editable* installs both mapping into `pipelex/` is fragile, which would break the local cross-pin testing this scope needs. Cost: mechanical `pipelex.temporal.*` → `pipelex_temporal.*` rewrite across the moving code + its tests (which travel anyway) + ~9 import lines in `pipelex-mistralai-workflows` tests + the entry point. Supersedes the design's "keep the import path / lowest-churn" lean (written before the regular-package packaging reality was confronted).-> - **D9 — scaffold tool-configs are sourced from `pipelex`'s own `pyproject.toml`** (ruff/pyright/mypy/pytest/coverage — the moved code passes these verbatim today), while repo *shape* (Makefiles/, metadata, `.gitignore`, LICENSE, family feel) mirrors `pipelex-api`. Avoids lint churn on the moved code.->-> **Recon facts (three agents, 2026-06-20):** core→temporal coupling is a *single* hard import (`pipelex/plugins/builtins.py:18,40` → `TemporalPlugin`); everything else core-side is config (`temporal: Temporal` field, `[temporal]` toml, `--temporal/--no-temporal` flags) that correctly stays. temporal→core spans ~46 modules (`cogt.*`, `core.*`, `runtime_bridge.*`, `pipe_run.*`, `pipeline.*`, `hub`, `tools.*`) ⇒ `pipelex-temporal` depends on base `pipelex`. Downstream: `pipelex-worker` = pin + `pipelex worker` CLI (no temporal imports); `pipelex-mistralai-workflows` = `temporal` extra + ~9 `pipelex.temporal.*` test imports; `pipelex-api-hosted` = config-only wrapper, **no Python pin** (temporal reaches it through the publish-gated `pipelex-api` image). `pipelex-api` itself pins `pipelex[…,temporal]==X` — a publish-gated flip, out of this local scope.->-> **Local cross-pin strategy:** `pipelex-temporal` depends on editable `pipelex` via `[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }` (mirrors how `pipelex-mistralai-workflows` pins `../_workflows`); downstream repos pin editable `../pipelex-temporal`. The published `==Y` pins are the publish-time flip (deferred). Order constraint: drop the `pipelex-temporal` console script from `_plugins/pyproject.toml` **before** any `uv sync` in the new repo (else two `pipelex-temporal` scripts collide).--- [x] **Step 0a — scaffold `pipelex-temporal` to match the sibling Python repos' conventions. ✅ DONE** (D9; see cut-over as-built). Don't invent a new toolchain — mirror the habits already used across the workspace's Python repos (`pipelex-platform` / `pipelex-admin-api` / `pipelex-api` are the references): same Makefile targets (`agent-check`, `agent-test`, `format`, `lint`), ruff (format + lint) + pyright + mypy strict, pytest layout, `uv` for deps, hatchling build backend, the standard `pyproject.toml` shape + CHANGELOG. The scaffold should feel like it belongs in the family before any Temporal code lands in it.-- [x] **Step 0b — relocate the Temporal config schema to a core-owned module (codex C6 — the prerequisite). ✅ DONE** (in-`pipelex`, behavior-neutral, committed at this checkpoint). Schema → `pipelex/system/configuration/config_temporal.py`; the two exceptions it needs (`TemporalConfigError`, `WorkerTaskQueueUnknownError`) → `pipelex/system/configuration/exceptions.py`, re-based on `PipelexError` (byte-equivalent: same `error_domain=None`, `ValueError` mixin intact, subclasses preserved). All importers repointed; `configs.py:14` imports from core; the `if TYPE_CHECKING: … RetryPolicy = Any` placeholder kept. Gates green (`make tb` · `make agent-check` pyright 0/mypy clean · `make agent-test` "All tests passed"); zero external importers of the moved symbols. Full record + the now-fixed `(ValueError, PipelexError)` `.message` latent bug (reordered to `(PipelexError, ValueError)` in an `xhigh`-review follow-up): `### Phase 5 — Step 0b (C6) as-built` below. *(Was: low-risk, could be pulled into Phase 3.)*-- [x] Move `pipelex/temporal/` (impl) + `temporal_plugin.py` + tests + the `temporal` marker + the `--temporal-server` conftest option into `pipelex-temporal`; declare `[project.entry-points."pipelex.plugins"]`. **✅ DONE** (top-level `pipelex_temporal`, D8).-- [x] Keep protocol-level conformance tests in core; Temporal's behavioral suite travels. **✅ DONE** (incl. the 5 mixed-test splits + 2 config tests kept in core — see cut-over as-built).-- [ ] **Flip downstream pins** (cross-repo blast radius): `pipelex-worker`, `pipelex-api-hosted` from `pipelex[temporal]==X` → `pipelex-temporal==Y`. **⏸ DEFERRED — publish/release-gated** (precise cut-list in the cut-over as-built; per Option-A the `pipelex-worker` flip lands in the same commit as the release pin bump).-- [ ] **Repoint `pipelex-mistralai-workflows`** (design §15): its `temporal` extra + `pipelex.temporal.*` imports → `pipelex-temporal`. **⏸ DEFERRED — publish-gated + coordinate with that repo's in-flight mistralai-2.x work.**--**Tests:**--- [ ] Re-run the relocated Temporal suite against published `pipelex`.-- [ ] Run downstream consumers' suites against the new pins **before** publishing.--### 🛑 CHECKPOINT 5 — hard stop (final)--- [ ] **Verify:** Temporal is an external plugin; core has no `temporal` extra; downstream pins flipped; consumer suites green.-- [ ] **Capture cold-start context:** `### Phase 5 — as-built` note — the published `pipelex-temporal` version, the pins flipped in each downstream repo, and the consumer-suite green evidence.-- [ ] **Fan-out `/code-review`:** core *deletion* side reviewed (session 1, on `a8382d1a1`); `pipelex-temporal` *additions* side reviewed (session 2) → **SHIP/clean** (no BLOCKER/SHOULD-FIX; wheel built offline, pin-no-leak + import-light proven at runtime). 2 NITs triaged (traces fixed; `test_extras→tests` back-import deferred). Remaining: re-review downstream pin-flip diffs at publish. Triage: [`../../plugins/phase-5-cutover-review-followups.md`](../../plugins/phase-5-cutover-review-followups.md).-- [ ] **Ship:** coordinate the cross-repo release. Effort complete.-------## Deferred / out of scope (considered, parked — do NOT do these in 0–5)--- [ ] **P3 follow-up commit:** collapse `SdkClientManager` into `SdkClientRegistry` (hub holds the registry directly) — D4 keeps it out of the Phase 0 rename. Land as its own commit after Phase 0.-- Third-party-defined execution modes (open string-keyed space) — enum stays closed (design D5).-- Generic per-plugin typed-config namespace — Temporal config stays typed in core (design D7).-- Deliberately *overriding* a built-in (third party replacing core's OpenAI driver) — design §5.3.-- Per-SPI versions / semver-range matching — single coarse `int` (design §5.4).-- Vendor inference dists + OpenAI-substrate extraction into a named module — design §6.4/§6.6/§11.-- Distribution/CI for `pipelex-temporal` — part of Phase 5's cross-repo effort.-------## Sequencing--Phase 0 (rename) → Phase 1 (seam + LLM, merged) → Phase 2 (ImgGen/Extract/Search, parallel sub-PRs) → Phase 3 (orchestrators) → Phase 4 (model_lists) → **[gate: Checkpoint 3 §14.5 Temporal-green]** → Phase 5 (externalize Temporal, cross-repo). Phases 0–4 are independent PRs in `pipelex`; each lands green. Inference SPI + authoring guide ship with Phase 2; orchestrator SPI with Phase 3. Phases 0 and 1 are strictly sequential; after Phase 1 lands, lanes A (ImgGen) / B (Extract) / C (Search) / D (Phase 3) / E (Phase 4) can run in parallel worktrees — **conflict flag:** lanes A/B/C and E both touch `plugins/<vendor>/` for multi-family vendors; coordinate the per-vendor plugin object.-------## As-built log (append per phase at each checkpoint — keep this current for cold starts)--> Each checkpoint appends an `### Phase N — as-built` subsection here with: final names/signatures, divergences from plan, test evidence, and anything a cold resume needs.--### Phase 0 — as-built--**Status:** done, committed on `refactor/Plugins`. `make agent-check` clean (ruff/plxt/pyright 0/mypy 0/keyword-only pass) · `make tb` green · `make agent-test` green.--**Final module → class map** (`pipelex/plugins/`):--| Old module / symbol | New module / symbol |-|---|---|-| `plugin.py` · `Plugin` | `model_handle.py` · `ModelHandle` |-| `plugin_sdk_registry.py` · `PluginSdkRegistry` (`PluginSdkRegistryRoot`) | `sdk_client_registry.py` · `SdkClientRegistry` (`SdkClientRegistryRoot`) |-| `plugin_manager.py` · `PluginManager` | `sdk_client_manager.py` · `SdkClientManager` |-| `plugin_factory_abstract.py` · `PluginFactoryAbstract` | `backend_extras_factory.py` · `BackendExtrasFactory` |--- `SdkClientRegistry.get_sdk_instance`/`set_sdk_instance` → **`get(self, model_handle)`** / **`set(self, *, model_handle, sdk_instance)`** (subject `model_handle` positional; the DRY `get_or_create` is still Phase 1, not added here).-- `SdkClientManager.plugin_sdk_registry` attribute → **`sdk_client_registry`** (the D4 thin-wrapper shape preserved; collapse into the registry remains the deferred P3 commit).-- `hub.py`: `get_plugin_manager`/`set_plugin_manager` → `get_sdk_client_manager`/`set_sdk_client_manager` (both the `PipelexHub` methods and the module-level functions); field `_plugin_manager` → `_sdk_client_manager`; the `"PluginManager2 is not initialized"` message is now `"SdkClientManager is not initialized"`.-- `pipelex.py`: `self.plugin_manager` → `self.sdk_client_manager` at construction/setup/teardown call sites.--**Scope beyond the literal rename (kept it a clean "free the word plugin"):** the `Plugin` *type* rename was carried through to the variables/params that held it — `plugin` → `model_handle` — across the four worker factories and every in-tree adapter (`openai`, `bedrock`, `anthropic`, `gateway`, `portkey`, `azure_rest`). The four wrong-path imports (`llm`/`img_gen`/`extract` imported `Plugin` from `plugin_sdk_registry`; `search` used the correct path) are all now `from pipelex.plugins.model_handle import ModelHandle`. Adapter error messages `f"Plugin '{...}'…"` → `f"ModelHandle '{...}'…"`. Prose/comment uses of "plugin" that mean the *future plugin concept* (e.g. `pipe_llm.py`, `provider_name.py`, `model_deck.py`, `cogt/exceptions.py`, `error_pages_generator.py`) were left untouched (one capital-P comment in `pipe_llm.py` was lowercased to read as the concept, not the type).--**Tests touched:** symbol/kwarg renames in the worker-factory routing tests, the gateway/transport/azure error-handling tests, and the temporal teardown call sites. The `plugin=`→`model_handle=` kwarg flip and `client_kwargs["plugin"]`→`["model_handle"]` assertions track the renamed param. Fixtures `plugin_for_openai`/`plugin_for_anthropic` → `model_handle_for_openai`/`model_handle_for_anthropic`, and `tests/integration/pipelex/fixtures/plugin_fixtures.py` → `model_handle_fixtures.py` (conftest import + `__all__` updated). The `tests/integration/pipelex/plugins/` dir name was kept — it reads correctly as backend tests under the new vocabulary.--**No behavior change.** Pure rename; the dispatch `match` statements, caching dance, and error surfaces are byte-equivalent modulo identifiers.--**Deferred (still open):** P3 follow-up — collapse `SdkClientManager` into `SdkClientRegistry` (hub holds the registry directly), own commit after Phase 0. Not done here by D4.--### Phase 1 — as-built--**Status:** done, committed on `refactor/Plugins`. `make agent-check` clean (pyright 0 / mypy 0 over 2219 files / keyword-only pass) · `make tb` green · `make agent-test` fully green. `/code-review --fix` ran clean. This is the locked contract Phases 2–4 build against.--**New seam modules** (all in `pipelex/plugins/`, all import-light — importing them imports no backend SDK or `temporalio`):--- `contract.py` — `PipelexPlugin` (`@runtime_checkable Protocol`: `name: str`, `targets_api: int`, `register(self, registrar) -> None`) + `PLUGIN_API_VERSION: int = 1`. Docstring states the **side-effect-free `register`** invariant.-- `inference_backend_registry.py` — `InferenceFamily(StrEnum)` = LLM/IMG_GEN/EXTRACT/SEARCH; `MakeWorkerFn: TypeAlias = Callable[..., InferenceWorkerAbstract]`; `InferenceBackendRegistry` keyed by `(family, sdk)` with `.lookup(*, family, sdk)` (miss → `NotImplementedError` "No inference backend registered for sdk '<sdk>' in the <family> family. Is its plugin installed?"), `.has(...)`, `.keys`. Also hosts **`require_sdk(*, spec, extra, msg, dependency_name=None)`** — `spec` accepts a `str` or a `Sequence[str]` (bedrock needs both `boto3`+`aioboto3`); `dependency_name` overrides the displayed package when it differs from the import name (google: spec `google.genai` / dep `google-genai`); raises `MissingDependencyError`.-- `orchestrator_registry.py` — `OrchestratorProtocol` (`async def run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput`) + `OrchestratorRegistry` keyed by `PipelexExecutionMode` (`.get_optional`, `.has`, `.modes`). **Skeleton only — not consumed until Phase 3.**-- `registrar.py` — `PluginRegistrar` accumulator + `PluginDiscovery` (mutable BaseModel: name/origin/status/targets_api/contributions/detail), `HubSlot`/`PluginOrigin`/`PluginStatus` StrEnums, `CliCommand` NamedTuple. Menu surface: `add_inference_backend(*, family, sdk, make_worker)`, `add_orchestrator(*, mode, orchestrator)`, `claim_content_generator|pipe_router|pipe_run|task_manager(factory)` (D5 thunks), `add_cli_command(*, name, help, command)`, `add_teardown(callback)`. `begin_plugin(...)` (driven by `build_registrar`) sets the "active" discovery so contributions are attributed and **duplicate errors name both contributors**. Duplicate `(family,sdk)`/mode/slot each raise their fail-loud error.-- `discovery.py` — **`build_registrar(*, config: PipelexConfig) -> PluginRegistrar`** (pure, idempotent, D3-safe). Iterates `BUILTIN_PLUGINS` then `importlib.metadata.entry_points(group="pipelex.plugins")` (external entry points resolve to a plugin instance or a zero-arg factory); version-checks each (`targets_api != PLUGIN_API_VERSION` → `PluginApiVersionMismatchError`); skips + logs names in `config.plugins.disabled` (raises `CoreUnconditionalPluginDisabledError` if the name is in `CORE_UNCONDITIONAL_PLUGIN_NAMES`); wraps any other failure in `BrokenPluginError` (the two sanctioned `except Exception` sites, annotated "Case 2"). The `_external_entry_points()` helper is the monkeypatch seam for tests.-- `builtins.py` — `BUILTIN_PLUGINS` = the seven LLM driver plugin instances (OpenAI, Gateway, Portkey, Anthropic, Mistral, Bedrock, Google); `CORE_UNCONDITIONAL_PLUGIN_NAMES = frozenset({"openai"})`.-- `exceptions.py` — `PluginError` base (CONFIG domain) + `PluginApiVersionMismatchError`, `DuplicateInferenceBackendError`, `DuplicateOrchestratorError`, `HubSlotAlreadyClaimedError`, `CoreUnconditionalPluginDisabledError`, `BrokenPluginError`. (`MissingOrchestratorError` is Phase 3, in `runtime_bridge/exceptions.py`.)--**Driver plugins** (one per vendor, in `pipelex/plugins/<vendor>/<vendor>_plugin.py`): each is a tiny class (`name`, `targets_api`, `register`) whose `register` calls `add_inference_backend(family=LLM, sdk=..., make_worker=<module-level closure>)`. The closures do the lazy SDK imports (`# noqa: PLC0415`), the `require_sdk` guard (anthropic/mistral/bedrock/google), and `sdk_clients.get_or_create(handle=model_handle, build=lambda: ...)`. They are byte-equivalent to the old `match` arms. **Phase 2 extends these same vendor plugins** to register IMG_GEN/EXTRACT/SEARCH backends (the cross-family-vendor coordination point).--**DRY helpers (design §6.2):** `SdkClientRegistry.get_or_create(*, handle, build)` (uses `is not None`, not truthiness) replaced the `get(...) or set(...)` dance; `require_sdk(...)` replaced the per-arm `find_spec` guard, moved **into** the closures so a missing extra fails at use.--**`MakeWorkerFn` call shape (the locked contract):** `make_worker(*, inference_model, backend, sdk_clients, reporting_delegate)` → `InferenceWorkerAbstract`. The closure derives its own `model_handle` from `inference_model`. The dispatcher (`LLMWorkerFactory.make_llm_worker`) builds `model_handle`, resolves `backend` via the models manager, `lookup`s the registry, calls the closure with `sdk_clients=get_sdk_client_manager().sdk_client_registry`, and `cast`s the result to `LLMWorkerInternalAbstract`.--**Boot wiring (`pipelex.py:setup()`):** `build_registrar(config=get_config())` is called **after** the gateway/model-setup checks (so `Pipelex.__new__(...).setup()` tests that expect `InferenceSetupRequiredError`/`GatewayTermsNotAcceptedError` still raise first) and **before** the content-generator/router/run hub setup points (right before `set_dry_run_forced`). The two registries are stored on the hub via `set_inference_backend_registry` / `set_orchestrator_registry`; the registrar is held on `self._plugin_registrar` for Phase 3's slot-thunk/CLI/teardown apply-points (empty in Phase 1).--**Hub:** added `_inference_backend_registry` / `_orchestrator_registry` fields, `set_/get_inference_backend_registry`, `set_/get_orchestrator_registry` (raise if unset), and module-level `get_inference_backend_registry()` / `get_orchestrator_registry()`. Registry types imported under `TYPE_CHECKING` (hub is imported everywhere — avoids cycles).--**Import-cycle break (not in the original plan, required):** `PipelexPipeRunInput` / `PipelexPipeRunOutput` were extracted from `runtime_bridge/bridge.py` into a new import-light **`runtime_bridge/payloads.py`**; `bridge.py` re-imports them (so `from ...bridge import PipelexPipeRunInput` still works for the existing tests). Without this, `orchestrator_registry → bridge → bootstrap → pipelex` was a `reportImportCycles` error.--**Control surface (D7):** core-owned `PluginsConfig` (`disabled: list[str]`) added to `configs.py` as `PipelexConfig.plugins`; `[plugins] disabled = []` added to **both** `pipelex/pipelex.toml` and `.pipelex/pipelex.toml` (real default, not commented). New `pipelex plugins list` command (`cli/commands/plugins_cmd.py`, registered in `_cli.py` + `list_commands`) renders `build_registrar(...).discoveries` as a Rich table.--**Tests:** `tests/unit/pipelex/plugins/test_plugin_discovery.py` (protocol satisfaction, version mismatch, duplicate backend/mode/slot naming both, broken plugin + broken entry point, **idempotent `build_registrar`**, denylist skip + core-unconditional raise, discoveries describe built-ins) and `tests/unit/pipelex/plugins/test_import_light_boot.py` (subprocess `sys.meta_path` import-blocker). The existing `test_llm_worker_factory.py` was updated to build the real registry from `BUILTIN_PLUGINS` and patch `get_inference_backend_registry` — so it exercises the real closures through the real lookup (registry round-trip + caching + huggingface-style missing-dependency parity + the registry-miss `NotImplementedError`). New error classes → `pipelex-dev generate-error-pages` regenerated `docs/errors/`.--**Behavior unchanged** in LLM dispatch except the deliberate registry-miss error wording (was `"ModelHandle '<...>' is not supported"`, now `"No inference backend registered for sdk '<sdk>' ... Is its plugin installed?"`).--**Deferred / notes for later phases:** docs (Inference SPI reference + plugin-authoring guide + example backend plugin) ship with **Phase 2** per plan — not written in Phase 1. The orchestrator registry/protocol is a skeleton; Phase 3 wires the bridge `match` collapse, slot-thunk apply-points (with injection-precedence), CLI-command harvesting, and adds `MissingOrchestratorError`.--### Phase 2 — as-built--**Status:** done, uncommitted on `refactor/Plugins`. `make agent-check` clean (ruff/plxt, pyright **0 errors**, mypy 0 over 2228 files, keyword-only pass) · `make tb` green · targeted plugins+cogt unit suite green (977 passed) · **full `make agent-test` green (exit 0, "All tests passed.")**. All four worker-factory `match` statements are gone — every inference backend now resolves through `get_inference_backend_registry().lookup(family, sdk)`.--**Registry shape after Phase 2** — `BUILTIN_PLUGINS` now has 15 plugins contributing **30** `(family, sdk)` backends: LLM 14, IMG_GEN 9, EXTRACT 5, SEARCH 2.--**New vendor plugin modules** (`pipelex/plugins/<vendor>/<vendor>_plugin.py`, each added to `BUILTIN_PLUGINS`):--- `fal` → IMG_GEN `fal` (require_sdk `fal_client`/extra `fal`, dep name `fal-client`).-- `huggingface` → IMG_GEN `huggingface_img_gen`. **C10 bug fixed:** added `require_sdk(spec="huggingface_hub", extra="huggingface")` (the old arm imported `huggingface_hub` with no guard). The `PROVIDER_OR_POLICY_T` annotation is a module-level `if TYPE_CHECKING` import (import-light at runtime); the `model_handle.variant → provider` resolution lives in the closure.-- `blackboxai` → IMG_GEN `blackboxai_img_gen` (OpenAI-completions substrate, `BlackboxaiCompletionsFactory(is_http_url_enabled=True)`; no require_sdk — reuses OpenAI client).-- `openrouter` → IMG_GEN `openrouter_img_gen` (same substrate shape, `OpenRouterCompletionsFactory(is_http_url_enabled=True)`).-- `azure_rest` → IMG_GEN `azure_rest_img_gen`. **DIRECT construction** — `AzureImgGenWorker(model_handle=…)`, no `sdk_clients`/`get_or_create`; `backend` + `sdk_clients` params carry `# noqa: ARG001`.-- `docling` → EXTRACT `docling_sdk` (require_sdk `docling`; `get_or_create(build=DoclingFactory.make_docling_sdk)`).-- `pypdfium2` → EXTRACT `pypdfium2`. **STATELESS** — built directly, no `get_or_create`, no require_sdk; `sdk_clients` param `# noqa: ARG001`.-- `linkup` → EXTRACT `linkup_fetch` (require_sdk `linkup`, stateless) **and** SEARCH `linkup` (stateless, **no** require_sdk — matches the original search arm exactly; raw `ImportError` if linkup absent, a deliberate no-behavior-change choice). One plugin, two families.--**Extended existing vendor plugins** (one `register` now spans families — the cross-family coordination point):--- `openai` += IMG_GEN `openai_img_gen`.-- `gateway` += IMG_GEN `gateway_img_gen` + `gateway_completions`, EXTRACT `gateway_extract`, SEARCH `gateway_search` (now serves **all four** families). Closures named `_make_gateway_{img_gen,completions_img_gen,extract,search}_worker` (distinct from the existing LLM `_make_gateway_{completions,responses}_worker`).-- `google` += IMG_GEN `google` (own missing-msg constant `_GOOGLE_IMG_GEN_MISSING_MSG`, distinct from the LLM one).-- `mistral` += EXTRACT `mistral` (own `_MISTRAL_EXTRACT_MISSING_MSG`).--Note: `gateway_completions` and `google` are SDK strings registered in **two** families each — distinct `(family, sdk)` keys, no conflict. The OpenAI-compat substrate (`OpenAICompletionsImgGenWorker`, the per-vendor `*CompletionsFactory`) stays an in-tree library the substrate-reuse closures import; extraction into a named module remains deferred.--**Factory collapses** (`cogt/{img_gen,extract,search}/*_worker_factory.py`): each is now the LLM shape — derive `model_handle`, resolve `backend` via the models manager, `lookup(family=<FAM>, sdk=model_handle.sdk)`, call with `sdk_clients=get_sdk_client_manager().sdk_client_registry`, `cast` to the family worker abstract. Kept `@classmethod` (call sites unchanged). The per-family `case _:` wording ("is not supported …") is gone — registry miss now raises the uniform `NotImplementedError("… Is its plugin installed?")` from `InferenceBackendRegistry.lookup`.--**Search C9 normalization:** `make_search_worker` now takes `*, reporting_delegate: ReportingProtocol | None = None` (default `None`, so existing test callers that omit it still work) and passes it through — the factory no longer pulls `get_report_delegate()` from the hub. The hub coupling moved to the caller: `cogt/content_generation/search_generate.py::_make_search_worker` now passes `reporting_delegate=get_report_delegate()` (import added). No production behavior change (the real caller still supplies the delegate); integration tests that call the factory directly get `None`, which the workers already accept.--**Tests:** `tests/unit/pipelex/cogt/img_gen/test_img_gen_worker_factory.py` updated to build the real registry from `BUILTIN_PLUGINS` (new `build_builtin_inference_backend_registry` + patches `get_inference_backend_registry` in `patch_hub_getters`), the unknown-sdk assert switched to "Is its plugin installed?", and `huggingface_img_gen` added to the missing-dependency parametrize (proves the C10 fix). New `tests/unit/pipelex/plugins/test_inference_backend_coverage.py` pins the full 30-key `(family, sdk)` round-trip + the cross-family vendors (mistral LLM+EXTRACT, google/openai LLM+IMG_GEN, linkup EXTRACT+SEARCH, gateway all four). Import-light guard BLOCKED set grew `docling` + `linkup` (fal_client/huggingface_hub were already there) — proves the new vendor plugin modules import no optional SDK at boot.--**Docs:** `docs/under-the-hood/inference-backend-plugins.md` (added to both mkdocs nav blocks) — the seam walkthrough, the `PipelexPlugin` contract, `MakeWorkerFn` + import-light/fail-at-use invariants, a complete copy-pasteable `acme` example (built-in + `[project.entry-points."pipelex.plugins"]` form), the `plugins.disabled` denylist, the SPI module/symbol table, and the fail-loud table.--**No factory test modules for extract/search** existed before and none added — those families are covered by the new registry-coverage test + the integration suites (`test_search.py`, extract integration). Only img_gen had a routing test to update.--#### Post-checkpoint xhigh `/code-review` (2026-06-20, on PR #997 = the whole Phases 0–2 diff)--Beyond the per-phase CP reviews above, an **xhigh** whole-PR pass ran (10 finder angles → 35 candidates → independent verify → 13 kept). **No live correctness bug.** Three CONFIRMED pre-merge gaps fixed + committed as `94fa908df` (`make agent-check` green, 78 affected tests pass). These **supersede specifics in the CP2 record above**:--- **Lookup-miss error (supersedes the line about `NotImplementedError("… Is its plugin installed?")`):** `InferenceBackendRegistry.lookup` now raises a structured **`InferenceBackendNotFoundError(PluginError)`** (new in `plugins/exceptions.py`, carries `family`/`sdk`, message ends "Is its plugin installed **and enabled?**"). Both factory miss-tests assert the structured type + `.sdk`.-- **Import-light blocklist (supersedes "BLOCKED grew docling+linkup"):** added `openai`, `portkey_ai`, `pypdfium2` — the migrated plugins' deferred SDKs the guard wasn't covering. (The review said "openrouter," but that closure rides the `openai` SDK; verified by probe that none of the three are imported at registration time, so the invariant holds.) The guard docstring now says "any backend SDK (optional extra **or** heavy core dep)."-- **Multi-spec `require_sdk`:** on a partial miss it named ALL specs; now collects + names only the absent one(s) (so a user with `boto3` but not `aioboto3` isn't told to reinstall `boto3`). New `test_require_sdk` multi-spec case.--**Deferred follow-ups → `../../plugins/phase-2-review-deferred.md`** (latent footguns reachable only via external entry-point plugins / future callers + quality cleanups). **Two bear on Phase 3 — read that doc before starting:**--- `build_registrar` accumulates `cli_commands` / `slot_claims` / `teardown_callbacks` but boot only consumes `inference_backends` + `orchestrators` — i.e. those three menus are **wired for the first time in Phase 3** (CLI collapse / Boot-teardown collapse). The doc frames the current state as a teardown-leak footgun for *external* plugins; closing it is literally Phase 3 work. Until then, an external plugin contributing those is silently inert.-- search `reporting_delegate` defaults to `None` (the C9 normalization above) — fine for the one real caller, but a future Temporal search-activity path that omits it gets silent zero-reporting. Relevant when Phase 3/5 routes search through a worker.--(The `ModelHandle` double-construction cleanup is cross-referenced there to the existing `phase-1-review-deferred.md`; it dissolves if Phase 3 threads `model_handle` through `MakeWorkerFn`.)--### Phase 3 — as-built--**Status:** done, checkpoint commit `19e6ca66b` on `refactor/Plugins-2`. `make agent-check` clean (ruff/plxt, **pyright 0**, mypy 0 over 2239 files, keyword-only pass) · `make tb` green · **full `make agent-test` green** ("All tests passed", exit 0) · **Temporal integration suite green** (`tests/integration/pipelex/temporal/` → 156 passed) · `/code-review` clean (no BLOCKERs; S1/N1/N2 applied).--**The dispatch seam.** `bridge.py`'s `match execution_mode:` is gone. `run_pipe_via_bridge` now does `get_orchestrator_registry().get_optional(mode=…)` → `orchestrator.run(...)`, else `MissingOrchestratorError(mode=…)`. `bridge.py` names no integration (only docstrings/comments). `build_pipe_job_from_input`, `_validate_input`, `_decode_*` stay; `serialize_pipe_output` is re-exported (`# noqa: F401`/`TC001`) for host-runtime back-compat.--**Orchestrators (verbatim extraction):**--- `runtime_bridge/direct_orchestrator.py::DirectOrchestrator` — body of the old `_run_direct`, **`scoped_pipe_router(PipeRouter())` landmine kept verbatim**. Core, always-on.-- `temporal/temporal_orchestrators.py::TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` — bodies of `_run_temporal_*`, lazy `temporalio`/`make_temporal_pipe_run`/`WorkflowExecutionError` inside `run`; each `run` first calls `_require_temporal_extra(mode=…)` (`importlib.util.find_spec("temporalio")` → `MissingOrchestratorError` on absence — no relabelling of deeper import bugs).--**Cycle break (the one non-obvious structural move).** Two extractions were forced by `reportImportCycles` (pyright counts **function-level** imports too — verified empirically; lazy-import does NOT break a pyright cycle, only a runtime one):--1. `runtime_bridge/serialization.py` — `PIPE_DISPATCH_ERRORS`, `serialize_pipe_output`, `resolve_main_stuff_root_key`, `serialize_completed_output` (import-light: core memory + pipe/pipeline exceptions + payloads). Both DIRECT and Temporal orchestrators serialize through it, so they never import `bridge.py` (which would re-form `bridge→bootstrap→pipelex→discovery→builtins→orchestrator→bridge`).-2. **CLI commands declared by `import_path` string, not by importing the callable.** A builtin Temporal plugin contributing `worker`/`setup-temporal-namespace` (whose modules boot Pipelex) would cycle `builtins→temporal_plugin→worker_cmd→pipelex→discovery→builtins`. Fix: `CliCommand.command: Callable` → `CliCommand.import_path: str` (`"module:attr"`); `_cli.py` resolves it via `importlib.import_module` at harvest (dynamic ⇒ off pyright's static graph). The two command modules are unchanged from `main`. Temporal stays a **builtin** (not an entry point) — the plan's Phase-3 intent.--**Plugins:** `plugins/direct/direct_plugin.py::DirectOrchestratorPlugin` (`name="direct"`) + `temporal/temporal_plugin.py::TemporalPlugin` (`name="temporal"`) added to `BUILTIN_PLUGINS`; `CORE_UNCONDITIONAL_PLUGIN_NAMES = {"direct", "openai"}`. `TemporalPlugin.register`: always two `add_orchestrator` + two `add_cli_command(import_path=…)`; iff `registrar.config.temporal.is_enabled`, four `claim_*` thunks + `add_teardown` (all module-level zero-arg thunks lazy-importing temporalio).--**Boot/teardown (`pipelex.py`):** the four `temporal.is_enabled` blocks → injection-precedence at each ordered point via `_resolve_hub_slot(*, slot, default: Callable[[], _HubSlotImplT]) -> _HubSlotImplT` (explicit param > slot thunk > core default). TASK_MANAGER just runs its thunk (full temporal-hub wiring inside it). `teardown` runs `reversed(self._plugin_registrar.teardown_callbacks)` (LIFO) in place of the old temporal block. `_temporal_task_manager` attribute removed; `_plugin_registrar` now declared `PluginRegistrar | None` in `__init__` and assigned via a local in `setup` (narrowing). `temporal_hub.get_optional_task_manager()` added (+ `teardown()` promoted onto the `TaskManager` protocol with `@override` on the impl).--**CLI (`_cli.py`):** `_CORE_COMMAND_ORDER` (core block) + harvested `_PLUGIN_COMMAND_NAMES`; `list_commands` returns their concatenation. `_register_discovered_cli_commands()` runs `build_registrar(config=_config_for_cli_harvest())` at module load and `app.command(...)`-registers each resolved callable. `_config_for_cli_harvest` = `load_config(ensure_global_if_missing=False)` (new side-effect-free param) with `(TomlError, ValidationError)` → `load_base_config_dict()` fallback (new method; package-default only, always valid).--**Exceptions:** `MissingOrchestratorError(*, mode)` (mode→message match/case, `_authors_caller_facing_message=True`) replaces `MissingPipelexTemporalExtraError` + `MissingMistralWorkflowsPluginError` (both removed). `pipelex-dev generate-error-pages` regenerated `docs/errors/` (the two old pages removed, `missing-orchestrator-error.md` added; also picked up `inference-backend-not-found-error.md` from CP2).--**Tests:** new `test_orchestrator_dispatch.py` (fake-orchestrator dispatch + per-mode miss parity), `test_hub_slot_injection_precedence.py` (explicit>slot>default at content-generator/pipe-router + teardown LIFO, fake registrar via patched `build_registrar`), `test_plugin_cli_command_harvest.py` (worker/setup-temporal-namespace harvested, in order, after core). Updated: `test_dispatch.py` (mistral → `MissingOrchestratorError`), `test_exceptions_disclosure.py` (per-mode `MissingOrchestratorError`), `test_output_serialization.py` (import from `serialization`), `test_import_light_boot.py` (BLOCKED += `temporalio`; config stub gains `temporal.is_enabled=True` to exercise the slot-claim branch import-light), `test_inference_backend_coverage.py` + `test_plugin_discovery.py` (stub configs gain `temporal` so the now-builtin TemporalPlugin's `register` can read `config.temporal.is_enabled` — closes deferred-doc item 4 for these two tests). Boot-via-slots for the temporal content generator is already covered end-to-end by `test_keyless_boot_forced_dry.py` (green through the new seam).--**Deferred-doc items closed by Phase 3:** the three previously-unconsumed registrar menus are now wired — `cli_commands` (consumed by `_cli.py` harvest), `slot_claims` + `teardown_callbacks` (consumed by `pipelex.py` boot/teardown). So an external plugin contributing those is no longer silently inert.--**Notes for Phase 4/5:** Phase 4 (model_lists 5th seam) is independent. Phase 5 externalizes `pipelex/temporal/` → `pipelex-temporal`: the Temporal plugin + orchestrators + commands are already a self-contained unit behind the seam; the move is (a) relocate the Temporal config schema out of `pipelex.temporal` (codex C6 — `configs.py:14` still imports `pipelex.temporal.config_temporal`), then (b) ship `pipelex/temporal/` + declare the `pipelex.plugins` entry point in the new dist (today Temporal is an in-tree builtin, not yet an entry point). Gate: this CHECKPOINT-3 Temporal-green evidence.--**Post-checkpoint xhigh `/code-review` follow-ups (separate from the CHECKPOINT-3 reviewer pass above).** A later xhigh `/code-review` over the Phase 3 commit surfaced a dominant correctness cluster plus medium/low items, captured for triage:--- 🔴 **Critical — read before Phase 5 / before externalizing any plugin:** [`../../plugins/phase-3-critical-cli-harvest-fragility.md`](../../plugins/phase-3-critical-cli-harvest-fragility.md). The import-time CLI plugin-command harvest (D3) is an unguarded surface: a broken/incompatible/colliding **external** plugin or an unreadable user config can brick *every* `pipelex` command — including the `--help`/`doctor`/`plugins`/`init` recovery commands — and `add_cli_command` has no collision guard, so a plugin can silently shadow a core command (`run`, …). Latent for in-tree-only installs (suite green), live the moment Phase 5 ships an external dist. **Cold-start problem statement only — solution to be explored in a fresh session.**-- 🟡 **Medium/low deferred items → [`../../plugins/phase-3-review-deferred.md`](../../plugins/phase-3-review-deferred.md):** the misleading `MISTRAL_NATIVE` "install a package you already have" hint (dead mode until the Mistral plugin lands), the bridge dropping its orchestrator error-wrap (external-orchestrator contract), `teardown()` with no `try`/`finally`, `_teardown_temporal` re-fetching from the global hub, the wrong SPI doc about Temporal entry-point discovery, an enum identity comparison, double-discovery/import-time cost (coupled to the critical fix), a dead `serialize_pipe_output` re-export, and a hardcoded slot count.--#### CHECKPOINT-3 `/code-review` (pr-review-toolkit:code-reviewer, on the Phase 3 working-tree diff)--**Verdict: clean — no BLOCKERs.** The reviewer compared every extracted body against `git show HEAD:…bridge.py` and confirmed **byte-equivalence** of all four orchestrator bodies (modulo the `_PIPE_DISPATCH_ERRORS`→`PIPE_DISPATCH_ERRORS` rename + the added `_require_temporal_extra` guard) and verified all named landmines: DIRECT `scoped_pipe_router` preserved, temporal-blocking `make_workflow_id` recompute + `WorkflowExecutionError` catch verbatim, per-mode error parity (exact deleted hint strings, `_authors_caller_facing_message=True`), injection precedence (explicit>slot>default at all four points), import-light (no module-top temporalio; register reads only `config.temporal.is_enabled`), CLI determinism, teardown LIFO + no-integration.--Applied: **S1** (the one should-fix) — added `test_harvest_config_falls_back_to_base_on_broken_user_config` pinning the malformed-user-config fallback in `_config_for_cli_harvest` (named Phase-3 risk #6, previously untested); **N1** — fixed the two stale `runtime_bridge.bridge._run_direct` doc references in `bundle_validator.py` → `runtime_bridge.direct_orchestrator.DirectOrchestrator.run`; **N2** — re-anchored the harvest-order test on "plugin commands are the last registrations" (`names[-2:]`) rather than the brittle `which` positional. **N3** (unused `OrchestratorRegistry.has`/`.modes`) left as a deliberate read-API symmetry surface mirroring `InferenceBackendRegistry`. The reviewer also confirmed the `if pipe_router:`-truthy vs `if content_generator is None:` asymmetry is **pre-existing** (identical in HEAD), not a Phase-3 regression.--### Option A — as-built (post-Phase-3, commit `989c9beed`)--**Status:** done. `make agent-check` clean (pyright 0 / mypy 0 over 2240 files / keyword-only) · `make tb` green · `make agent-test` "All tests passed." · `pytest tests/integration/pipelex/temporal/` → **156 passed, 4 xpassed** (the 4 xpassed = pre-existing xdist class-registration flakiness markers, matching the Phase-3 baseline) · both `--help` smokes pass. Full record + rationale: [`../../plugins/option-a-drop-cli-command-seam.md`](../../plugins/option-a-drop-cli-command-seam.md).--**Why.** The post-Phase-3 xhigh review found a critical fragility: the Phase-3 CLI-command harvest (the former D3) ran `build_registrar` at *import* of `pipelex/cli/_cli.py` on **every** `pipelex` invocation, so a broken/colliding external plugin or an unreadable user config could brick every command — including the `--help`/`doctor`/`plugins`/`init` recovery commands — and `add_cli_command` had no collision guard, so a plugin could silently shadow a core command. **Option A resolves it by removing the surface, not hardening it.** CLI commands are not a plugin contribution type any more.--**What changed:**--- **New:** `pipelex/temporal/temporal_cli.py` — a grouped `typer.Typer()` app registering `worker` + `setup-namespace`; `pipelex-temporal = "pipelex.temporal.temporal_cli:app"` in `pyproject.toml` `[project.scripts]`. Import-light at module top (callables pull `temporalio` lazily).-- **Moved (git renames):** `pipelex/cli/commands/worker_cmd.py` → `pipelex/temporal/worker_cmd.py`; `…/setup_temporal_namespace_cmd.py` → `pipelex/temporal/setup_namespace_cmd.py` (callable `setup_temporal_namespace_cmd` → `setup_namespace_cmd`). They now travel with Temporal into Phase 5.-- **Removed:** the harvest in `_cli.py` (`_config_for_cli_harvest`, `_register_discovered_cli_commands`, `_PLUGIN_COMMAND_NAMES`; `list_commands` now returns `list(_CORE_COMMAND_ORDER)`); `add_cli_command` + `CliCommand` + `cli_commands` in `plugins/registrar.py`; the two `add_cli_command` calls in `temporal/temporal_plugin.py` (docstrings rewritten; the hardcoded "four hub slots" count also dropped while there).-- **Strings/docs:** `PIPELEX_SETUP_CLI_COMMAND` + all docstrings/examples → `pipelex-temporal setup-namespace` / `pipelex-temporal worker`; 5 distributed-execution docs swapped; `docs/under-the-hood/orchestrator-plugins.md` CLI section rewritten ("Operational commands ship as console scripts") + SPI table + worked-example + the factually-wrong "self-entry-point" claim fixed; `docs/errors/search-attribute-registration-error.md` regenerated (not hand-edited).-- **Tests:** deleted `test_plugin_cli_command_harvest.py`; added `tests/unit/pipelex/temporal/test_temporal_cli.py` (console-script smoke); updated `test_setup_temporal_namespace_cmd.py` (import + use-site mock targets + callable); fixed `test_cli_entrypoint_smoke.py` (dropped the now-invalid `pipelex worker --help` case) and `test_search_attribute_bootstrap_check.py` (assert the new string).--**Untouched (deliberately out of scope — do NOT unwind in later phases):** the four inference-family registry lookups, the bridge → `OrchestratorRegistry` dispatch, the `temporal.is_enabled` boot-slot claims + LIFO teardown, `build_registrar` / `BUILTIN_PLUGINS` / the `pipelex.plugins` entry-point group for *runtime* contributions, `pipelex plugins list`, the denylist.--**Decision lineage:** the former **D3** (harvest at CLI-build) is **superseded** by Option A (see the D1–D7 list). [`phase-3-critical-cli-harvest-fragility.md`](../../plugins/phase-3-critical-cli-harvest-fragility.md) → RESOLVED-BY-REMOVAL; two items in [`phase-3-review-deferred.md`](../../plugins/phase-3-review-deferred.md) (double-discovery/import-time cost; factually-wrong SPI doc) → resolved.--**Open follow-ups (NOT done):**--1. **Config-loader cleanup (flagged, low-risk):** `_config_for_cli_harvest` was the *sole* caller of `config_manager.load_base_config_dict()` and `load_config(ensure_global_if_missing=False)`. Both are now dead (the method has zero callers; the `False` branch is unreachable) in `pipelex/system/configuration/config_loader.py`. Left in place to keep Option A surgical — a follow-up should delete `load_base_config_dict()` and simplify `load_config` to always-ensure-global.-2. **Cross-repo, RELEASE-GATED (do NOT do ahead of the pipelex release shipping `pipelex-temporal`):** flip `pipelex-worker/Dockerfile` (`CMD ["pipelex","worker","--no-sandbox"]` → `["pipelex-temporal","worker","--no-sandbox"]`) and `pipelex-worker/Makefile` (`pipelex worker …` → `pipelex-temporal worker …`) in the **same** commit that bumps `pipelex-worker`'s `pipelex` pin to that release. `pipelex-api-hosted` / `sandbox` / `pipelex-api` were checked — they do **not** invoke the worker. Until that bump, the old pinned pipelex still has `pipelex worker`, so nothing breaks. (Mirror this into the Phase-5 downstream-pin step.)-3. **Legacy `pipelex/temporal/worker_cli.py`** (the `python -m pipelex.temporal.worker_cli` / `configure` entrypoint) is a unification candidate with the new `temporal_cli.py` — noted, not addressed.--**Env gotchas for the next session:** (a) `uv sync --all-extras` materializes the `pipelex-temporal` console script into `.venv/bin/`; it also brings the venv up to the already-committed `uv.lock` (which had drifted on `cryptography`/`mthds`) — `uv.lock` itself unchanged. (b) `make cleanderived` deletes the gitignored `tests/integration/pipelex/fixtures/_generated_model_sets.py`; run `make regenerate-test-models-quiet` (alias `rtm`) before `make agent-check` or pyright fails on the missing import.--### Phase 4 — as-built (the 5th seam: model listing)--**Status:** done (working-tree, pre-commit at time of writing). `make agent-check` clean (ruff/plxt, **pyright 0 over 2243 files**, mypy 0, keyword-only pass) · `make tb` green · **full `make agent-test` green** ("All tests passed.") · targeted `tests/unit/pipelex/plugins/` + `tests/unit/pipelex/cogt/model_backends/` → 715 passed. `model_lists.py`'s `match sdk:` is gone — `pipelex show models <backend>` now dispatches through a registry on the hub.--**The seam.** `cogt/model_backends/model_lists.py` `ModelLister.list_models` no longer branches on `match sdk:` with per-arm hardcoded `from pipelex.plugins.{openai,anthropic,mistral,google,bedrock}.*_list import …`. It now does `get_model_lister_registry().get_optional(sdk=sdk)` → `await lister(...)`, else (miss) → `unsupported_sdks`. The file names **no integration** (imports only `ModelListingUnsupportedError`/`ModelManagerError` from `cogt.exceptions`, `PipelexCLIError`, and the three hub getters).--**New module** `plugins/model_lister_registry.py` (mirrors `inference_backend_registry.py` / `orchestrator_registry.py`, dependency-free):-- `ListModelsFn: TypeAlias = Callable[..., Awaitable[None]]` — the uniform lister callable. Always `async` (the loop awaits it); import-light to reference, lazy inside. Call shape: `await lister(*, sdk, backend_name, backend, flat, any_listed)`.-- `ModelListerRegistry` keyed by `sdk` alone (listing is per-SDK, not per-`(family, sdk)`). `get_optional(*, sdk) -> ListModelsFn | None` (a miss is a **soft** "unsupported-for-listing" outcome, mirroring `OrchestratorRegistry.get_optional` — **not** the inference registry's raising `lookup`), `has`, `sdks`.--**Registrar** (`plugins/registrar.py`): new `add_model_lister(*, sdk, lister)` menu method + `model_listers: dict[str, ListModelsFn]` accumulator + `_model_lister_sources`. Duplicate `sdk` → `DuplicateModelListerError` naming both plugins. Contributions line `f"model lister {sdk}"` (so `pipelex plugins list` shows listers automatically).--**Exceptions:** `DuplicateModelListerError(PluginError)` in `plugins/exceptions.py`; **`ModelListingUnsupportedError(CogtError)`** in `cogt/exceptions.py` — the core soft signal the loop catches (carries `sdk`). `pipelex-dev generate-error-pages` wrote new pages (`duplicate-model-lister-error.md`, `model-listing-unsupported-error.md`) + the `inference-and-providers.md` index; the rest unchanged.--**Hub + boot:** `_model_lister_registry` field + `set_/get_model_lister_registry` + module-level `get_model_lister_registry()`. `pipelex.py setup()` builds `ModelListerRegistry(plugin_registrar.model_listers)` and sets it on the hub, right after the inference-backend registry. The `show models` CLI path boots Pipelex via `make_pipelex_for_cli(needs_inference=True)`, so the registry is always set before `list_models` runs.--**Vendor listers** — each of the 5 plugins grew one `async` lister closure that lazy-imports its `list_*_models` fn (`# noqa: PLC0415`) and registers it:-- `openai` → `_list_openai_models` for sdks `openai`, `azure_openai`, `openai_responses`, `azure_openai_responses` (one closure across those keys; async underlying).-- `anthropic` → `_list_anthropic_models` for `anthropic`. **The one non-trivial case:** the closure catches the vendor `AnthropicSDKUnsupportedError` and re-raises core `ModelListingUnsupportedError(sdk=…)` — so core names no Anthropic-specific symbol while preserving the old "bedrock-backed Anthropic client can't list → unsupported_sdks" behavior.-- `mistral` → `_list_mistral_models` for `mistral`. Underlying fn is **sync and takes no `backend`** → closure is `async def` with `# noqa: RUF029` and `backend: … # noqa: ARG001`.-- `google` → `_list_google_models` for `google` (async underlying).-- `bedrock` → `_list_bedrock_models` for `bedrock`, `bedrock_aioboto3` (**sync** underlying → `# noqa: RUF029`).--**Behavior preserved byte-for-byte** (verified against `git show HEAD:…model_lists.py`): the per-sdk `find_spec` guards removed from `model_lists.py` were **already duplicated inside each `list_*_models` fn** (same lib/extra/message for anthropic/mistral/google/bedrock; openai has none — always installed), so a missing extra still raises `MissingDependencyError` inside the lister and is wrapped by the loop's Case-2 `except Exception → PipelexCLIError` exactly as before. The `list_*_models` fns themselves are **unchanged**. `any_listed` threads identically (set `True` only after a successful `await`; not reached when the lister raises `ModelListingUnsupportedError`, matching the old `continue`). The unsupported-SDK display path is untouched.--**Tests:** `tests/unit/pipelex/plugins/test_model_lister_coverage.py` (registry built from `BUILTIN_PLUGINS`: every expected sdk key resolves to a lister + exact-set assertion + soft-miss cases) · `tests/unit/pipelex/cogt/model_backends/test_model_lister_dispatch.py` (behavioral: lister invoked with expected kwargs; `any_listed` progresses False→True across two SDKs; unknown sdk → unsupported message + lister not invoked; `ModelListingUnsupportedError` translate path → unsupported; generic lister failure → wrapped `PipelexCLIError`). `test_import_light_boot.py` grew an `assert registrar.model_listers` (proves listers register import-light).--**Docs:** `docs/under-the-hood/inference-backend-plugins.md` gained a "Listing models — an optional capability" section (the `add_model_lister` call, the `ListModelsFn` shape, import-light/fail-at-use rules, the `ModelListingUnsupportedError` soft-signal); SPI table += `ListModelsFn` / `ModelListingUnsupportedError`; fail-loud table += `DuplicateModelListerError` (and fixed the adjacent stale `NotImplementedError` row → `InferenceBackendNotFoundError`).--**CHECKPOINT-4 sweep result — "core names no integration" now holds for every enumerated seam.** The broad grep surfaces three **pre-existing, unchanged** `pipelex.plugins.<vendor>` refs that were never in the Phase 0–4 seam list: `cogt/config_cogt.py` (vendor typed-config models — **by design**, design D7), `cogt/img_gen/img_gen_args_factory.py` (`OpenAIImgGenFactory`) and `cogt/model_backends/backend_factory.py` (`VertexAIFactory`, lazy, vertexai auth) — the latter two genuine but **unenumerated** couplings (each would need its own contract capability to invert). Recorded with disposition in [`../../plugins/phase-4-residual-core-vendor-couplings.md`](../../plugins/phase-4-residual-core-vendor-couplings.md). **No dispatch-path integration remains.**--**Notes for Phase 5:** Phase 4 is independent of the Temporal externalization. No new cross-repo surface. The model-lister registry is a pure additive capability — an external backend plugin can now contribute a lister via the same `register` it uses for `add_inference_backend`.--### Phase 5 — Step 0b (C6) as-built (the config-schema relocation prerequisite)--**Status:** done, committed at this checkpoint on `refactor/Plugins-2`. **In-`pipelex` only, behavior-neutral, zero cross-repo blast radius.** Gates: `make tb` green · `make agent-check` clean (ruff/plxt, **pyright 0/0**, mypy 0 over 2244 source files, keyword-only pass) · targeted `tests/unit/pipelex/{temporal,system,errors}/` → 463 passed · **full `make agent-test` "All tests passed."** This is the prerequisite that makes the rest of Phase 5 a packaging move: **core config no longer imports `pipelex.temporal`.**--**What moved.**--- `git mv pipelex/temporal/config_temporal.py → pipelex/system/configuration/config_temporal.py` (history preserved; the whole schema module — it was already import-light via the `if TYPE_CHECKING: from temporalio.common import RetryPolicy / else: RetryPolicy = Any` placeholder, kept verbatim). Its internal `from pipelex.temporal.exceptions import …` → `from pipelex.system.configuration.exceptions import …`.-- **New** `pipelex/system/configuration/exceptions.py` holds the two exceptions the schema raises: `TemporalConfigError(ValueError, PipelexError)` + `WorkerTaskQueueUnknownError(TemporalConfigError)` *(base order later corrected to `(PipelexError, ValueError)` by the `.message` fix — see the flagged-bug note below)*. **Re-based** from `TemporalFlowError` to `PipelexError` directly — deliberate: `TemporalFlowError` is the base of the *runtime/workflow* errors that stay with (and externalize with) `pipelex/temporal/`, so dragging it into core would be wrong for the end-state. `TemporalFlowError` is **never imported/caught anywhere** (only mentioned in comments — verified), so dropping it from these two classes' ancestry is invisible. `error_domain` stays `None` (both old `TemporalFlowError` and new `PipelexError` leave it unset — chose `PipelexError` over the domain-setting `PipelexConfigError` to preserve exact behavior).--**What stayed (in `pipelex/temporal/exceptions.py`, travels external in the cut-over).** `TemporalFlowError` + the runtime errors (`WorkflowExecutionError`, `UnrecoverableWorkflowFailureError`, `WorkflowInputError`, `ContentGenerationError`, `TemporalServerError`) + the worker-boot config subclasses (`WorkerScopeConfigError`, `WorkerProfileConfigError`, `SearchAttributeRegistrationError`), which now subclass the **core** `TemporalConfigError` (imported at the module top).--**Importers repointed** (mechanical `perl` rename `pipelex.temporal.config_temporal` → `pipelex.system.configuration.config_temporal` across `pipelex/` + `tests/`, then hand-split the lines that mixed a moved exception with a staying symbol): source = `configs.py:14`, `temporal_tasks.py`, `temporal_connect.py`, `temporal_manager.py`, `temporal_task_manager.py`, `task_manager.py` (TYPE_CHECKING), `codec/codec_server_cli.py`, and the `namespace_check.py` docstring; tests = the ~20 temporal config/exception test modules. The two regression tests got their hardcoded source path / import updated: `test_config_temporal_optional_dep.py` (now AST-scans `pipelex/system/configuration/config_temporal.py`) and `test_dispatch_options_no_temporalio.py`. The `temporal-e2e-validate` skill doc's `BUILTIN_SEARCH_ATTRIBUTES` reference was updated. `pipelex-dev generate-error-pages` rewrote 4 pages (the two moved errors' `Defined in`/`Parent` rows + the `execution-and-runtime`/`platform-and-tooling` macro listings, which correctly regroup the two from "Temporal execution" → "System & configuration").--**Behavior preserved byte-for-byte** (the relocation is a pure move): the schema's validators/methods (`resolve_dispatch`, `make_retry_policy`, `validate_task_queue_known`, the orphan-queue validators) are unchanged; the exception MRO/`error_domain`/`isinstance(_, ValueError)`/`str()`/`.args` are identical to pre-move (verified empirically). The error-class-location convention test passes (new module is a proper `exceptions.py` importing only `PipelexError`).--**Flagged at C6, FIXED in follow-up (`xhigh` review action):** the `(ValueError, PipelexError)` MRO meant `ValueError.__init__` won, so `.message` was never set and `to_error_report()` raised `AttributeError`. Fixed by reordering bases so the message-setting `PipelexError` base resolves first — `TemporalConfigError(PipelexError, ValueError)` and `ConfigModelError(FatalError, ValueError)` (the `Worker*` / `SearchAttribute*` subclasses inherit the fix transitively; the two `(PipeComposeError, ValueError/TypeError)` compose mixins already had the safe order). `isinstance(_, ValueError)` is unaffected by base order, so Pydantic still wraps validator raises; `str()`/`.args` unchanged. Behavior change at one site: `ConfigModelError` now runs the restored `TracebackMessageError.__init__`, so it logs at its single misuse raise site (`ConfigModel.transform_dict_str_to_enum`) — appropriate fatal-error behavior, matching its `ConfigValidationError` / `FatalError` siblings. Guard + regression locked in by `tests/unit/pipelex/test_pipelex_error_message_init.py` (MRO sweep over every loaded `PipelexError` subclass asserting `PipelexError` precedes `ValueError`/`TypeError`, plus `.message` / `to_error_report()` assertions on the reordered classes). `make agent-check` clean, full `make agent-test` green.--**Cross-repo readiness for the cut-over:** scanned `pipelex-worker` / `pipelex-mistralai-workflows` / `pipelex-api` / `pipelex-api-hosted` / `pipelex-platform` / `pipelex-admin-api` / `sandbox` / `cocode` — **none** import the moved config schema or the two moved exceptions (the only external `pipelex.temporal.*` import is `pipelex-api` pulling the *runtime* `WorkflowExecutionError`, which stays in `pipelex/temporal/` and is a normal cut-over pin concern, not a C6 concern). So Step 0b ships with no downstream coordination.--**Remaining Phase 5 (cross-repo, paused for go/no-go):** Step 0a scaffold `../pipelex-temporal` (empty repo, remote `Pipelex/pipelex-temporal`, mirror `pipelex-api`/`pipelex-platform` conventions) → move `pipelex/temporal/` + `temporal_plugin.py` + temporal tests + the `temporal` marker + `--temporal-server` conftest option out, declare the `pipelex.plugins` entry point → flip `pipelex-worker` / `pipelex-api-hosted` pins → repoint `pipelex-mistralai-workflows` → run consumer suites before publishing. Also fold in the Option-A follow-up (delete the now-dead `load_base_config_dict()` / `ensure_global_if_missing=False` in `config_loader.py`).--### Phase 5 — cross-repo cut-over as-built (LOCAL work, pre-publish; 2026-06-20)--**Status: the reversible local cut-over is DONE and green on both sides.** User gave go/no-go for the *reversible local work only* (scaffold + move + flip-to-editable + suites green), explicitly **stopping before any PyPI publish / merge / deploy**. Nothing pushed. Two decisions locked this session: **D8** (package layout = top-level `pipelex_temporal`, not a `pipelex.temporal` namespace stitch — see the header note) and **D9** (scaffold tool-configs sourced from `pipelex`, repo shape from `pipelex-api`).--**Green evidence.**-- Core (`_plugins`): **`make agent-check` clean** (pyright 0 / mypy 0 over 2077 src / keyword-only ✓), **`make tb` green**, **full `make agent-test` = 7678 passed** (after fixing 2 error-handling bundle couplings, below; 4 xfailed + 2 skipped are pre-existing/expected).-- `pipelex-temporal`: **`make agent-check` clean** (ruff/pyright **0** / mypy 0/208), **`make agent-test` "All tests passed"** (≈509 pass, 4 xpassed = the same pre-existing xdist class-registration flake markers as pipelex's baseline). Entry point, `pipelex-temporal` console script (`worker`/`setup-namespace`), and import smokes verified.--**The new dist (`/Users/lchoquel/repos/Pipelex/pipelex-temporal`, branch `main`, NO commits yet at time of writing — local only).** Top-level package `pipelex_temporal/` (64 modules, `pipelex.temporal.*`→`pipelex_temporal.*` rewrite). `pyproject.toml`: deps `pipelex>=0.35.0` + `temporalio==1.24.0` + `aiohttp>=3.14.0`; **`[tool.uv.sources] pipelex = { path = "../_plugins", editable = true }`** (local-dev cross-pin → flips to a published `pipelex==Y` pin at release); `[project.scripts] pipelex-temporal = "pipelex_temporal.temporal_cli:app"`; `[project.entry-points."pipelex.plugins"] temporal = "pipelex_temporal.temporal_plugin:TemporalPlugin"`; tool-configs copied from pipelex; **`[tool.ruff.lint.isort] known-first-party = ["pipelex","pipelex_temporal"]`** so the workflow modules' deliberate `# noqa: TC001` on passed-through `pipelex.*` imports keep working (without it `pipelex` is third-party → TC002 → noqa stripped). `Makefile`+`Makefiles/Makefile_basics.mk` (trimmed from pipelex-api: env/install/lock/format/lint/pyright/mypy/agent-check/agent-test/clean*). README, CHANGELOG (`[Unreleased]`), LICENSE, `.gitignore`, `pipelex_temporal/py.typed`.--**Core (`_plugins`) cut.** `TemporalPlugin` removed from `BUILTIN_PLUGINS` (+ comment that temporal is now an external entry-point dist); `CORE_UNCONDITIONAL_PLUGIN_NAMES` unchanged (`{direct, openai}`). `pyproject.toml`: dropped the `temporal` extra, the `pipelex-temporal` console script (so the new repo's editable install doesn't collide), and the `temporal` pytest marker. `git rm`'d `pipelex/temporal/` + `tests/{unit,integration}/pipelex/temporal/` + 2 scattered moved tests. Core source names no integration (only doc-comments, repointed to `pipelex_temporal`). **`MissingOrchestratorError` install hint changed** `pip install 'pipelex[temporal]'` → `pip install pipelex-temporal` (+ docstring; + the 2 asserting tests `test_exceptions_disclosure.py`/`test_orchestrator_dispatch.py`). Stale `pipelex[temporal]` docstrings in `reporting_manager.py`/`execution_mode.py`/`test_temporal_activity_gate_lazy_import.py` → `pipelex-temporal`. **Option-A fold-in done:** deleted dead `load_base_config_dict()` + the `ensure_global_if_missing` param/branch in `config_loader.py`.--**Test reconciliation (the fiddly part).** Temporal behavioral suite (~110 files) + shared fixtures (`tests/integration/fixtures/` + `tests/integration/conftest.py`), root `tests/conftest.py`, and shared `error_handling/test_data.py` copied into `pipelex-temporal/tests/` (layout `tests/{unit,integration}/pipelex_temporal/`; the blanket rewrite maps both `pipelex.temporal`→`pipelex_temporal` and the test path `tests.integration.pipelex.temporal`→`tests.integration.pipelex_temporal`). **5 mixed core bridge/boot tests split** (core keeps the DIRECT/core-mode methods, the temporal-mode methods move to new pipelex-temporal files): `test_dispatch.py` (3 methods → `test_bridge_temporal_dispatch.py`), `test_temporal_blocking_workflow_id.py` (wholesale), `test_trace_context_contract.py` (→ `test_trace_context_temporal.py`), `test_keyless_boot_forced_dry.py` (→ `test_keyless_temporal_boot.py` under `tests/integration/system/` to escape the temporal conftest's autouse `boot_temporal`), `test_error_type_uri.py` (wholesale). **`test_cv_batch_screening.py` stayed** (DIRECT e2e) — decoupled from the temporal tree via a local `test_data.py` + its own `cv_batch_screening.mthds`. **`test_config_temporal_optional_dep` + `test_non_retryable_baseline_pins` relocated BACK to core** `tests/unit/pipelex/system/` (they validate the core-relocated `config_temporal.py` / `pipelex.toml`, were mis-swept). **2 shared bundles restored to core** `tests/integration/pipelex/error_handling/bundles/{native_text_sequence,native_search}.mthds` (the core local-parity error tests read them; they were mislocated under the temporal tree). `test_import_light_boot.py` reworded (temporalio stays BLOCKED; the temporal slot-claim arm moved to the pipelex-temporal suite — core guarantee "boot imports no temporalio even with is_enabled=True" preserved). pipelex-temporal needed its own `tests/__init__.py` rootdir anchor + a `.pipelex/` project config + a rootdir `conftest.py` (`pytest_plugins` + placeholder creds) for hermetic standalone boot.--**Docs hygiene done:** `tests/CLAUDE.md` temporal row removed; all `pipelex[temporal]` install instructions in `docs/` → `pipelex-temporal`; the stale `MissingPipelexTemporalExtraError`/`MissingMistralWorkflowsPluginError` doc refs → `MissingOrchestratorError`.--> **⚠ CORRECTION 2026-06-21 — read before using this cut-list.** `pipelex-temporal` is **PRIVATE / not-open-source → NOT PyPI.** The `pipelex-temporal==Y` pins written below are the wrong (public-PyPI) form. The correct form is a **`git+ssh` pin** (`pipelex-temporal @ git+ssh://[email protected]/Pipelex/pipelex-temporal.git@<tag>`), matching `pipelex-shared` in `pipelex-platform`. And only **private** repos may pin it: `pipelex-worker` and (newly) `pipelex-api-hosted`'s child image (confirmed 2026-06-21: the hosted runner enqueues Temporal) — the "`pipelex-api-hosted` = no-op" line below is now WRONG. The **public MIT** repos `pipelex-api` and `pipelex-mistralai-workflows` must **DROP** the `temporal` extra instead of repinning it. Reconcile the bullets below to git+ssh when the wiring is actually done.--**DEFERRED — publish/release-gated downstream cut-list (do NOT execute ahead of the pipelex release that ships `pipelex-temporal`):**-- **`pipelex-worker`** — `pyproject.toml:8` `pipelex[dynamodb,s3,temporal]==0.34.0` → `pipelex[dynamodb,s3]==X` + `pipelex-temporal==Y`; **Dockerfile:25** `CMD ["pipelex","worker","--no-sandbox"]` → `["pipelex-temporal","worker","--no-sandbox"]`; **Makefile:37** `$(PIPELEX) worker …` → `pipelex-temporal worker …`; **Makefile:52** editable install `-e "$(PIPELEX_REPO)[dynamodb,s3,temporal]"` → drop `temporal`, add `pipelex-temporal`. Per the Option-A note this MUST land in the same commit as the `pipelex` pin bump (until then the old pinned pipelex still has `pipelex worker`, so nothing breaks).-- **`pipelex-mistralai-workflows`** — `pyproject.toml:41` `temporal = ["pipelex[temporal]>=0.27.0"]` → `temporal = ["pipelex-temporal>=Y"]`; rewrite the ~9 `pipelex.temporal.*` imports in `tests/integration/test_bridge_temporal_{blocking,fire_and_forget}.py` → `pipelex_temporal.*`. Gated on published `pipelex-temporal` AND coordinate with that repo's in-flight mistralai-2.x work (don't disrupt it).-- **`pipelex-api`** pins `pipelex[…,temporal]==0.35.0` (PyPI) → drop `temporal`, add `pipelex-temporal` — publish-gated. **`pipelex-api-hosted`** = config-only wrapper, no Python pin (temporal reaches it through the `pipelex-api` image) — no-op.--**DEFERRED — other follow-ups:**-- `docs/errors/` temporal error pages (`unrecoverable-workflow-failure-error.md`, `workflow-execution-error.md`, `temporal-flow-error.md`, etc.) left in core ON PURPOSE: they are the `type_uri` dereference targets on `docs.pipelex.com`; deleting them before pipelex-temporal's docs exist would 404 those URLs. Moving them (and repointing/aggregating the type_uris) is a docs-deploy task for the publish phase. Do NOT `generate-error-pages` to prune them yet. (`temporal-config-error.md` stays — that error is core.)-- Deeper `docs/under-the-hood/` internal-path references (`temporal-integration.md`, `orchestrator-plugins.md`, `distributed-content-generation.md`, `pipe-routing-and-execution.md`, `error-model.md`) still say `pipelex.temporal` in prose — accurate-keeping pass deferred (the system behaves identically; coupled to the docs-deploy story). `orchestrator-plugins.md` should eventually present temporal as the worked **external** plugin example.-- Cross-repo `error_handling/test_data.py` parity constants are now duplicated in core + pipelex-temporal (can drift; inherent to externalization — no shared-file enforcement across repos).-- Local commits made on both sides (NOT pushed): `_plugins` = Phase 5 checkpoint commit on `refactor/Plugins-2`; `pipelex-temporal` = initial commit on `main`.-- **Post-local-cut-over `/code-review` triage (2026-06-20)** → [`../../plugins/phase-5-cutover-review-followups.md`](../../plugins/phase-5-cutover-review-followups.md). xhigh review of the core-side cut-over commit. **Fixed this session** (reachable/core-hygiene that stays in core): kit `commands.md` temporal-test section removed + `CLAUDE.md` regen; `uv.lock` temporal-extra drop; `bridge.py` + `test_config_temporal_optional_dep` docstrings; 5 plugin/cogt test stubs (dropped vestigial `temporal` stub field); **restored two core-coverage tests** — `test_trace_context_contract::test_non_direct_mode_nulls_host_trace_context` (core-side guard for the bridge's non-DIRECT trace_context nulling — distinct from the temporal-mode test that moved to pipelex-temporal) + `test_dispatch_options_no_temporalio.py` (DispatchOptions constructible without temporalio — the `RetryPolicy = Any` forward-ref contract the AST scan can't catch). **Still deferred** (in the wip doc): skill `temporal-e2e-validate` relocation, Makefile temporal targets (`tw`/`ttm`/…, CI-isolated), `debugging-hanging-pytest-runs.md`, boot eager-validation design call, `non_retryable_error_types` cross-repo class-name desync, dead `cv_batch_screening_inputs.json` mirror.-- **Finalize-local pass (session 2, 2026-06-20)** → same wip doc, *Finalize-local pass* section. Closed two of the prior bullet's "Still deferred" items: **both** Temporal skills relocated to `pipelex-temporal` (`temporal-e2e-validate` *and* `temporal-test-crate` — the latter was never on the list, also dead in core) + removed from core, and the **core** Makefile temporal targets deleted (help + `.PHONY` + `test-temporal`/`ttm` + `temporal-server`/`-stop`/`-worker`/`-run`; the `pipelex-temporal` Makefile *add* stays open — unverifiable without a running server). Plus the `pipelex-temporal` *additions*-side `/code-review` → **SHIP/clean**; NITs: stray `.pipelex/traces/*.ndjson` untracked+gitignored, shipped `test_extras→tests` back-import deferred (pre-existing/dormant). Gates: core `make tb` + full `make agent-test` green (HEAD `27da150a5`); pipelex-temporal `make agent-check` clean. **Local, uncommitted; still paused before publish.**diff --git a/wip/history/archive/signature-based-validation-tdd-plan.md b/wip/history/archive/signature-based-validation-tdd-plan.mddeleted file mode 100644index 8a3b617ce..000000000--- a/wip/history/archive/signature-based-validation-tdd-plan.md+++ /dev/null@@ -1,610 +0,0 @@-# Signature-Based Validation — TDD implementation plan--> **Superseded as the live tracker.** The current-state reference and reviewer verification map now live at [`../signature-based-validation/signature-based-validation.md`](../signature-based-validation/signature-based-validation.md). This file is kept as the phase-by-phase TDD build record (historical); where it disagrees with the live tracker on present behavior, the live tracker wins. Internal references to "TODOS.md" below mean *this* file under its original name.--Status: Phases 1–7.3 landed (plus Phase 5.2 lenient-summary polish) (2026-05-14); Phase 7.4 open and tracked as a merge-gate. **The branch is in a consistent, reviewable state** — the runtime accepts `type = "PipeSignature"`, and `.pipelex/plxt.toml` declares the matching schema source of truth (`derived/mthds_schema.json`) via a `[[rule]] / [rule.schema] path = ...` block scoped to `**/*.mthds`. That block is the intended configuration on this branch, not a workaround: the bundled `plxt` schema doesn't yet describe `PipeSignature`, so the local schema is the only correct one to validate against here. At merge time, once `vscode-pipelex` ships an updated bundled schema, the `[[rule]]` block is removed in the same commit and the bundled schema takes over again — that swap is the Phase 7.4 merge-gate item. Docs (new `signature-pipes.md` page + nav + cross-links + CLI reference + tightened spec docstring) and CHANGELOG (`[Unreleased]` Added/Changed) are in. `make agent-check` and `make agent-test` are green on this branch. Reviewers: please treat the `[[rule]]` block in `.pipelex/plxt.toml` as load-bearing for the branch — not as a smell to fix.--Current state: A `PipeSignature` can be authored as a spec (`pipelex/builder/pipe/pipe_signature_spec.py`), compiled to a blueprint (`pipelex/pipe_signature/pipe_signature_blueprint.py`), instantiated as a runtime (`pipelex/pipe_signature/pipe_signature.py`) via the registered `PipeSignatureFactory`, dry-run (yields a mock `Stuff` from the declared output `StuffSpec`), and rejected on live-run (`PipeSignatureNotExecutableError`). Signatures slot into `PipeBlueprintUnion` and `PipeSpecUnion`. Strict-mode gating is live: `collect_signature_refs(pipe=...)` walks the dependency graph (see the post-archive update note below — it is a free function in `pipelex/pipe_signature/signature_walk.py`, not a `PipeAbstract` method); `dry_run_pipe(..., allow_signatures=False)` (the default) raises `SignaturesNotAllowedError` carrying both `signature_refs` and the qualified `dep_paths` chain. `dry_run_pipes` does a pre-pass that aggregates signature refs across all pipes in the batch so the user sees every offender in one error (post-review fix). `validate_bundle` and `validate_bundles_from_directory` catch the error and surface it via `ValidateBundleError.signature_check_error`. CLI surface: `pipelex validate pipe`/`bundle` now take `--allow-signatures` (default off); `pipelex validate --all` filters signature pipes out of the strict iteration; the agent CLI's `validate_*_core` functions default to strict as well (same as the main CLI — an earlier draft of this plan said lenient; the shipped default is strict, pinned by `test_agent_validate_defaults_strict.py`). Signatures still raise `PipeSignatureNotExecutableError` on live run. End-to-end `.mthds` parsing works: the interpreter happily round-trips `type = "PipeSignature"` through `PipelexBundleBlueprint` → library → runtime → dry-run, in both strict and lenient mode. The schema generator was updated to strip `pipe_category` from `PipeSignatureBlueprint`, but the JSON Schema file lives in `vscode-pipelex/` (not this repo) so no schema artifact needed regeneration here. Surprise: `pipe_abstract.py` cannot import from `pipelex.hub` — the cycle pipe_abstract → hub → libraries.library → libraries.pipe.pipe_library → pipe_abstract is real and breaks at runtime, not just under pyright's `reportImportCycles`. The dependency-graph walk therefore lives as free functions in `pipelex/pipe_signature/signature_walk.py` (a module downstream of `hub`), not as methods on `PipeAbstract` — see the post-archive update note below. The walk iterates `sorted(pipe_dependencies())` so the dep chain rendered in `SignaturesNotAllowedError` is deterministic (post-review fix). Phase 6 surfaced one design gap: `dry_run_pipes` aggregated dep paths with a "first wins" rule, which lost the informative controller chain when the signature pipe was iterated first; the aggregator now prefers the longest known dep chain so the error message always shows the most useful path.--**Post-archive update (2026-05-18) — signature graph walk moved off `PipeAbstract`.** After this plan was archived, the `pipe_lookup`-callable workaround was removed. `collect_signature_refs` / `collect_signature_paths` are no longer methods on `PipeAbstract` taking a `pipe_lookup` parameter — they are now free functions in a new module `pipelex/pipe_signature/signature_walk.py`, which sits downstream of `pipelex.hub` and imports `get_optional_pipe` directly. `PipeAbstract` keeps only `pipe_dependencies()` and `is_signature`. Call sites read `collect_signature_refs(pipe=...)` / `collect_signature_paths(pipe=...)`. The recursion, visited-set cycle protection, sorted iteration, and longest-dep-chain aggregation are unchanged. The phase-by-phase notes below (notably the Phase 4.2 deviations) predate this refactor and are kept verbatim as the historical implementation record — wherever they say `PipeAbstract.collect_signature_refs(pipe_lookup=get_optional_pipe)`, read `signature_walk.collect_signature_refs(pipe=...)`. The refactor landed in commit `66a584dc`.--Known gaps after Phase 6 — **closed out at merge time via Phase 7.4**:--- The bundled `plxt` schema (shipped from `vscode-pipelex/`) does not yet know about `type = "PipeSignature"`. On this branch the canonical schema for `*.mthds` is the locally-regenerated `derived/mthds_schema.json`, wired in via a `[[rule]]` block in `.pipelex/plxt.toml` — that is the intended configuration on this branch and `plxt lint` is green against it. At merge time, the `[[rule]]` block is removed in lock-step with the `vscode-pipelex` release that bundles the updated schema; tracked as Phase 7.4. (Reviewers: this is the deliberate setup, not a workaround to flag.)-- The fast-path PostToolUse hook (`validate-mthds.sh`) uses the same `plxt` toolchain, so it picks up the branch-local schema rule automatically. `Write`/`Edit` on `PipeSignature` fixtures works through the hook on this branch with no special handling. Same Phase 7.4 cleanup applies after merge: the rule comes off when the bundled schema is updated.--**Pre-flight note:** `validate_bundle` was updated on this branch to actually call `dry_run_pipes` (the previously-commented-out calls in all three branches of `validate_bundle.py` were uncommented). This means the strict signature check sitting inside `dry_run_pipe` fires for **every** `validate_bundle` caller, not just direct CLI usage — no separate pre-flight helper is needed.--This plan is **strict TDD**: in every phase, failing tests are written first (red), then implementation makes them green. Integration tests construct blueprints directly in Python; end-to-end tests load real `.mthds` files. Each phase ends with `make agent-check` and a targeted test run; the final phase ends with `make agent-test`.--**Tick boxes as you go.** Mark each checkbox the moment its item lands — don't wait for the next checkpoint. The boxes are the working log: if a session is interrupted mid-phase, the next session resumes from the first unticked box. Checkpoints are for halting, not for batch-ticking.--**Stop checkpoints** are marked ⛔. At each one, coding must halt, the status block at the top of this file must be updated (and any deviations recorded), and a fresh session picks up at the next phase.-------## Quick start (cold session)--Working directory: `/Users/lchoquel/repos/Pipelex/_sig` (a worktree of the `pipelex/` runtime repo).--Read in this order:--1. This file (`TODOS.md`) — phase by phase.-2. `wip/signature-based-validation.md` — full design, rejected alternatives, edge cases. The authoritative source for "why" decisions.-3. `pipelex/builder/pipe/pipe_signature.py` — the existing spec class that needs corrections.-4. `pipelex/core/pipes/pipe_blueprint.py` — `PipeType` / `PipeCategory` enums and `PipeBlueprint` base.-5. `pipelex/core/pipes/pipe_abstract.py` — `PipeAbstract` (gains `is_signature`, promoted `pipe_dependencies`).-6. `pipelex/pipe_run/dry_run.py` — `dry_run_pipe` / `dry_run_pipes` (gains `allow_signatures` parameter).-7. `pipelex/pipeline/validate_bundle.py` — entry point that threads `allow_signatures` down.--Project conventions that apply (see `CLAUDE.md` for the full list):--- One `TestClass` per test module.-- Use pytest-mock; never `unittest.mock`.-- Tests live under `tests/unit/`, `tests/integration/`, `tests/e2e/` per the test type.-- After every code change: `make agent-check` (lint + type) before moving on.-- After every phase: targeted `pytest` run for the affected area. After the last phase: `make agent-test` (full).-- Never hardcode counts in comments or docstrings.-- Don't hard-wrap markdown.--If `make` isn't available, run the agent-check steps manually (see Makefile's `agent-check` target).-------## Architecture at a glance (for the impatient)--```-PipeType.PIPE_SIGNATURE ← new enum value, category PipeCategory.PIPE_SIGNATURE--spec PipeSignatureSpec — corrected (literal type, multiplicity inputs, no `result`)-blueprint PipeSignatureBlueprint — new-runtime PipeSignature — new; _dry_run mocks, _live_run raises--PipeAbstract gains: is_signature (property), pipe_dependencies() (default empty)--signature_walk.py: collect_signature_refs() / collect_signature_paths() — free-function graph walk- (see post-archive note: originally PipeAbstract methods)--Validation:- strict (default): dry_run_pipe(... allow_signatures=False) → SignaturesNotAllowedError- if collect_signature_refs(pipe) is non-empty- lenient: dry_run_pipe(... allow_signatures=True) → signatures mock outputs--CLI:- pipelex validate <pipe> [--allow-signatures]- pipelex validate --all [--allow-signatures]- pipelex-agent validate → defaults allow_signatures=True-```-------## Phase 1 — Type system foundations--Lay down the enum values, the `is_signature` property, and the promoted `pipe_dependencies` method. Small, foundational, no behavior change for existing pipes.--### Phase 1.1 — Tests first (red)--All tests use direct construction; no fixtures needed beyond pytest-mock when applicable.--- [x] `tests/unit/pipelex/core/pipes/test_pipe_blueprint_signature_enums.py` — `class TestPipeSignatureEnums`- - `test_pipe_type_pipe_signature_value` — `PipeType.PIPE_SIGNATURE` equals `"PipeSignature"`.- - `test_pipe_type_pipe_signature_in_value_list` — `"PipeSignature" in PipeType.value_list()`.- - `test_pipe_type_pipe_signature_category` — `PipeType("PipeSignature").category is PipeCategory.PIPE_SIGNATURE`.- - `test_pipe_category_pipe_signature_value` — `PipeCategory.PIPE_SIGNATURE` equals `"PipeSignature"`.- - `test_pipe_category_signature_is_not_controller` — `PipeCategory.is_controller_by_str("PipeSignature") is False`.-- [x] `tests/unit/pipelex/core/pipes/test_pipe_abstract_signature_surface.py` — `class TestPipeAbstractSignatureSurface`- - Build a minimal in-memory `PipeAbstract` subclass (or use an existing PipeLLM with stubbed validators) to assert:- - `is_signature` returns `False` for any operator/controller.- - `pipe_dependencies()` default returns `set()` on `PipeAbstract` (i.e. operators inherit the empty default).- - `is_signature` returns `True` for a category-`PIPE_SIGNATURE` instance (deferred — full assertion happens in Phase 2 once the runtime class exists; here, just assert the property reads from `self.pipe_category` correctly).- > Deviation: used a descriptor-on-stub pattern (resolving `__dict__["is_signature"]` / `__dict__["pipe_dependencies"]`) rather than constructing a full `PipeAbstract` subclass — building one would require a fully-formed `StuffSpec` + `Concept` and pull unrelated machinery into a unit test. Plan explicitly allowed the alternative.-- [x] Run: `.venv/bin/pytest -q tests/unit/pipelex/core/pipes/test_pipe_blueprint_signature_enums.py tests/unit/pipelex/core/pipes/test_pipe_abstract_signature_surface.py`. Confirm: red for the new value-existence assertions; legacy tests unaffected.--### Phase 1.2 — Implementation (green)--- [x] `pipelex/core/pipes/pipe_blueprint.py`:- - Add `PipeType.PIPE_SIGNATURE = "PipeSignature"`.- - Add `PipeCategory.PIPE_SIGNATURE = "PipeSignature"`.- - Update `PipeType.category` match to map `PIPE_SIGNATURE → PipeCategory.PIPE_SIGNATURE`. (Linter will catch missing cases.)- - Update `PipeCategory.is_controller` match to return `False` for `PIPE_SIGNATURE`.- - Add `is_signature` property to `PipeBlueprint` returning `PipeCategory(self.pipe_category) is PipeCategory.PIPE_SIGNATURE`.-- [x] `pipelex/core/pipes/pipe_abstract.py`:- - Add `is_signature` property mirroring the blueprint's.- - Add a default `pipe_dependencies(self) -> set[str]: return set()` method (matches the blueprint-side default).-- [x] `pipelex/pipe_controllers/pipe_controller.py`:- - `PipeController.pipe_dependencies` stays `@abstractmethod` — every controller must implement it; the linter enforces. Do not turn it concrete just because `PipeAbstract` now has a default. The default exists for operators and signatures, not for controllers.- > Deviation: also decorated `PipeController.pipe_dependencies` with `@override` so pyright's `reportImplicitOverride` accepts that it overrides the new `PipeAbstract` default. Linter forced this once the parent method became concrete.--### Phase 1.3 — Lint and targeted tests--- [x] `make agent-check` clean.-- [x] `.venv/bin/pytest -q tests/unit/pipelex/core/pipes/` — all green (new + legacy).--> Phase 1 deviation summary: `pipelex/core/pipes/output/output_renderer.py` and `tests/unit/pipelex/core/pipes/test_pipe_blueprint.py` both contained non-exhaustive `match` statements that needed a new `PIPE_SIGNATURE` arm to satisfy pyright. Added them.-------## Phase 2 — Blueprint, runtime, factory--The runtime class is where the dry-run mock generation lives. Tests construct `PipeSignatureBlueprint` directly in Python and verify end-to-end runtime behavior via `PipeFactory.make_from_blueprint` and `dry_run_pipe`.--### Phase 2.1 — Tests first (red)--Use blueprints constructed in Python — no `.mthds` parsing in this phase.--- [x] `tests/integration/pipelex/pipe_signature/conftest.py` — shared fixtures:- - `pipelex_for_signatures` — Pipelex setup with a tiny library containing a `Text` concept and one PipeSignature pipe (used by multiple tests).- - `make_signature_blueprint(...)` factory helper.- > Deviation: the per-test setup fixture was named `setup_signature_library` (rather than `pipelex_for_signatures`) and shaped as a callable, mirroring the existing `load_empty_library` pattern. The concepts registered are `SigTestDoc` / `SigTestSummary` rather than `Document` / `Summary` because both `Document` and `Summary` collide with native concept names.-- [x] `tests/integration/pipelex/pipe_signature/test_pipe_signature.py` — `class TestPipeSignature` (one class only — split into multiple test methods, parametrized where useful):- - `test_factory_produces_runtime_from_blueprint` — `PipeFactory.make_from_blueprint` on a `PipeSignatureBlueprint` returns a `PipeSignature`.- - `test_is_signature_true` — `runtime.is_signature is True`; `is_controller is False`.- - `test_needed_inputs_returns_declared` — runtime with declared inputs `{"doc": "Text"}` returns that set from `needed_inputs()`.- - `test_required_variables_returns_input_names` — same, `required_variables()` returns `{"doc"}` (no dotted paths).- - `test_dry_run_produces_mock_text` — runtime with `output = "Text"`, `dry_run_pipe` writes a `TextContent` stuff to working memory.- - `test_dry_run_produces_mock_variable_list` — runtime with `output = "Text[]"`, dry-run produces a `ListContent[TextContent]`.- - `test_dry_run_produces_mock_fixed_list` — runtime with `output = "Text[3]"`, dry-run produces a `ListContent` of length 3.- - `test_live_run_raises_signature_error` — `await pipe.run_pipe(..., run_mode=LIVE)` raises `PipeSignatureNotExecutableError` whose message contains the pipe's `pipe_ref`.- - `test_input_multiplicity_in_blueprint` — blueprint accepts `inputs = {"docs": "Document[]"}` without validation error.- - `test_validators_are_noops` — `validate_inputs_static`, `validate_inputs_with_library`, `validate_output_static`, `validate_output_with_library` all return without raising on a well-formed signature.-- [x] `tests/integration/pipelex/pipe_signature/test_pipe_signature_in_blueprint_union.py` — `class TestPipeSignatureBlueprintUnion`:- - `test_bundle_blueprint_accepts_signature_pipe` — a `PipelexBundleBlueprint` built with a `pipe = {"foo": PipeSignatureBlueprint(...)}` validates cleanly.- - `test_bundle_blueprint_rejects_unknown_pipe_type` — sanity guard, asserts the discriminator union still rejects garbage.-- [x] Run: `.venv/bin/pytest -q tests/integration/pipelex/pipe_signature/`. Confirm: all red with `ImportError` or `AttributeError` (the runtime class doesn't exist yet).--### Phase 2.2 — Implementation (green)--- [x] `pipelex/pipe_signature/` — new package directory.- - `pipelex/pipe_signature/__init__.py` — empty.- - `pipelex/pipe_signature/pipe_signature_blueprint.py`:- - `PipeSignatureBlueprint(PipeBlueprint)` with `type: Literal["PipeSignature"]`, `pipe_category: Literal["PipeSignature"]`, `signature_for: PipeType | None = None`.- - [x] `pipelex/pipe_signature/pipe_signature.py`:- - `PipeSignature(PipeAbstract)` with `type: Literal["PipeSignature"]`, `pipe_category: Literal["PipeSignature"]`, `signature_for: PipeType | None = None`.- - `validate_inputs_static`, `validate_inputs_with_library`, `validate_output_static`, `validate_output_with_library` → no-ops.- - `needed_inputs(visited_pipes=None)` → returns `self.inputs` (mirrors operator pattern).- - `required_variables()` → `set(self.inputs.variables)`.- - `_live_run_pipe(...)` → `raise PipeSignatureNotExecutableError(pipe_ref=self.pipe_ref)`.- - `_dry_run_pipe(...)`:- 1. Convert `self.output` (a `StuffSpec`) into a `TypedNamedStuffSpec` via a new `convert_stuff_spec_to_typed_named` helper in `pipelex/pipe_run/dry_run.py` (sibling of the existing `convert_to_working_memory_format`, but for a single output `StuffSpec` instead of `InputStuffSpecs`).- 2. Mint a single `Stuff` via a new `WorkingMemoryFactory.make_mock_stuff(typed_named_stuff_spec)` (a refactor — see below).- 3. Write to `working_memory.set_new_main_stuff(...)` and return `PipeOutput(working_memory=working_memory, pipeline_run_id=job_metadata.pipeline_run_id)`.- - `_validate_before_run`, `_validate_after_run` → no-ops.- - **Error-handling rule for new code:** do NOT introduce new `except Exception` catches in `PipeSignature`, the new helpers in `WorkingMemoryFactory`, or `convert_stuff_spec_to_typed_named`. Let mock-construction errors propagate (e.g. `DryRunFactory` failures, `ClassRegistryNotFoundError`). A separate worktree owns the error-handling refactor of the pre-existing `except Exception` in `WorkingMemoryFactory.make_mock_inputs:190` — do not replicate that pattern.- - [x] `pipelex/pipe_signature/pipe_signature_factory.py`:- - `PipeSignatureFactory(PipeFactoryProtocol[PipeSignatureBlueprint, PipeSignature])` with a `make(...)` classmethod that builds the runtime from the blueprint + parsed inputs/output.- - [x] `pipelex/pipe_signature/exceptions.py`:- - `PipeSignatureNotExecutableError(PipelexError)` with a `pipe_ref` field and a clear message.-- [x] Register `PipeSignatureFactory` in the class registry so `PipeFactory.make_from_blueprint` dispatches via the existing `f"{pipe_type.value}Factory"` convention. (Look at how other factories are registered — probably in `pipelex/__init__.py` or via auto-import; mirror.)- > Registered alongside `PipeSignature` in `CoreRegistryModels.PIPE_SIGNATURES` / `PIPE_SIGNATURES_FACTORY`. `RegistryModels.get_all_models()` walks every list-valued class attribute, so the new lists are auto-discovered.-- [x] `pipelex/core/bundles/pipelex_bundle_blueprint.py`:- - Add `PipeSignatureBlueprint` to `PipeBlueprintUnion`.-- [x] `pipelex/core/memory/working_memory_factory.py`:- - Add `make_mock_stuff(typed_named_stuff_spec: TypedNamedStuffSpec) -> Stuff` — extracts the per-iteration body of `make_mock_inputs` (both the no-multiplicity branch and the multiplicity branch combined into one helper returning a single `Stuff`). `make_mock_inputs` then becomes a thin loop over `make_mock_stuff`.- - Preserve the existing `except Exception` fallback inside `make_mock_inputs` (separate worktree owns that refactor). Do NOT add a new `except Exception` catch inside `make_mock_stuff` — let `make_mock_content` errors propagate. The existing fallback continues to wrap only the multi-stuff loop.-- [x] `pipelex/pipe_run/dry_run.py`:- - Add `convert_stuff_spec_to_typed_named(stuff_spec: StuffSpec, name: str) -> TypedNamedStuffSpec` — sibling of `convert_to_working_memory_format` that operates on a single output `StuffSpec` instead of `InputStuffSpecs`. Same class-registry lookup, same fallback-to-`TextContent` behavior on missing structure class (matches the existing behavior inside `convert_to_working_memory_format`). Do NOT add a new `except Exception`; mirror the specific exception types the existing function catches.--### Phase 2.3 — Lint and targeted tests--- [x] `make agent-check` clean.-- [x] `.venv/bin/pytest -q tests/integration/pipelex/pipe_signature/` — all green.-------## Phase 3 — Spec layer--Wire the existing `PipeSignature` spec into the `PipeSpecUnion`, fix the three corrections (literal `type`, multiplicity inputs, drop `result`), and implement `to_blueprint()`.--### Phase 3.1 — Tests first (red)--- [x] `tests/unit/pipelex/builder/pipe/test_pipe_signature_spec.py` — `class TestPipeSignatureSpec`:- - `test_type_literal_is_pipe_signature` — `PipeSignature(...).type == "PipeSignature"`.- - `test_signature_for_optional` — `PipeSignature(...)` with no `signature_for` field validates; setting `signature_for = PipeType.PIPE_LLM` validates.- - `test_signature_for_rejects_pipe_signature` — `PipeSignature(..., signature_for=PipeType.PIPE_SIGNATURE)` raises. A signature standing in for a signature is nonsensical; the validator rejects it.- - `test_inputs_accept_multiplicity` — `inputs = {"docs": "Document[]"}` validates; `inputs = {"images": "Image[3]"}` validates.- - `test_inputs_reject_invalid_concept_syntax` — `inputs = {"bad": "lowercase"}` raises.- - `test_no_result_field` — `assert "result" not in PipeSignature.model_fields`.- - `test_to_blueprint_returns_signature_blueprint` — `PipeSignature(...).to_blueprint()` returns a `PipeSignatureBlueprint` with matching `code`/`description`/`inputs`/`output`/`signature_for`.- - `test_to_blueprint_preserves_input_multiplicity` — `inputs = {"docs": "Document[]"}` round-trips through `to_blueprint()` unchanged.-- [x] `tests/unit/pipelex/builder/pipe/test_pipe_spec_union_signature.py` — `class TestPipeSpecUnionDispatch`:- - `test_union_dispatches_pipe_signature` — `pydantic.TypeAdapter(PipeSpecUnion).validate_python({"type": "PipeSignature", ...})` returns a `PipeSignature` instance, not another spec type.- - `test_union_dispatches_pipe_llm_unchanged` — regression guard: existing `{"type": "PipeLLM", ...}` still dispatches to `PipeLLMSpec`.- > Deviation: the test on bad concept syntax accepts both `ValidationError` and the underlying `ConceptStringError`. PipeSpec's inherited `validate_inputs` re-raises `ConceptStringError` without wrapping it in `ValueError`, so pydantic does not convert it to `ValidationError`. That looks like a pre-existing rough edge in PipeSpec; I did not widen its scope to address it.-- [x] Run: `.venv/bin/pytest -q tests/unit/pipelex/builder/pipe/test_pipe_signature_spec.py tests/unit/pipelex/builder/pipe/test_pipe_spec_union_signature.py`. Confirm red.--### Phase 3.2 — Implementation (green)--- [x] `pipelex/builder/pipe/pipe_signature.py`:- - Change `type: PipeType | str = Field(...)` to `type: Literal["PipeSignature"] = "PipeSignature"` (use `SkipJsonSchema` if needed to match sibling pattern).- - Change `pipe_category` to `SkipJsonSchema[Literal["PipeSignature"]] = "PipeSignature"`.- - Add `signature_for: PipeType | None = None` with description "Intended downstream pipe type when this signature is implemented (optional hint for agents)."- - Add a `@field_validator("signature_for", mode="after")` that rejects `PipeType.PIPE_SIGNATURE` with a clear message ("a PipeSignature cannot have signature_for=PipeSignature").- - Remove the `set_pipe_category` and `validate_type` validators (the literal handles both).- - **Remove `result` field entirely.**- - Allow multiplicity in inputs: replace the inputs description and add a `validate_inputs` validator that mirrors `PipeSpec.validate_inputs` (reuse via shared helper or copy).- - Add `to_blueprint(self) -> PipeSignatureBlueprint` building the blueprint from the spec fields.- - Update `rendered_pretty` for the new field surface (drop `result`).- > Deviation: `PipeSignature` now inherits from `PipeSpec` (was `StructuredContent`), so it gets `pipe_code`/`description`/`inputs`/`output`/`validate_inputs`/`validate_output` for free and slots naturally into `pipe_type_to_spec_class: dict[str, type[PipeSpec]]`. As a consequence the spec uses `pipe_code` rather than the old `code` field name (matching sibling specs).-- [x] `pipelex/builder/pipe/pipe_spec_union.py`:- - Add `PipeSignature` to the union.-- [x] `pipelex/builder/pipe/pipe_spec_map.py`:- - Add `"PipeSignature": PipeSignature`.--### Phase 3.3 — Lint and targeted tests--- [x] `make agent-check` clean.-- [x] `.venv/bin/pytest -q tests/unit/pipelex/builder/pipe/ tests/integration/pipelex/pipe_signature/` — all green.--> Phase 3 deviation summary: `pipe_signature.py` now extends `PipeSpec` (not `StructuredContent`) so it inherits `pipe_code` and the shared inputs/output validators. The hardcoded `len(PipeType.value_list()) == 10` assertion in `tests/unit/pipelex/language/test_mthds_schema.py` was changed to compare against the expected-blueprint set size; this also caught two other places that needed updates after the new `PipeType.PIPE_SIGNATURE` value: `tests/unit/pipelex/core/pipes/test_execution_data_coverage.py` (filters signature pipes out of the execution-data coverage walk) and `pipelex/language/mthds_schema_generator.py` (adds `PipeSignatureBlueprint` to `_PIPE_DEFINITION_NAMES` so `pipe_category` is stripped from its schema).-------⛔ **CHECKPOINT A — STOP HERE**--**Coding must stop after Phase 3.** Do **not** start Phase 4 in the same session.--Before handing back:--1. Confirm every box above is ticked (boxes should have been ticked as each item landed, not now). For any item that diverged from the plan, add a `> Deviation:` blockquote inline next to the box.-2. Update the top "Status" line of this file to read: `Status: Phases 1–3 landed (YYYY-MM-DD). Signatures construct end-to-end in Python (spec → blueprint → runtime → dry-run mock). Strict-mode gating, CLI wiring, e2e, and docs are open.`-3. Add a "Current state" paragraph near the top summarizing: signatures are mockable but not strict-checked; live execution raises but is unreachable through validate (validate doesn't yet know to refuse them); .mthds files cannot yet declare a `PipeSignature` end-to-end because the interpreter path hasn't been exercised. Note any unexpected surprises (e.g. if a refactor of `make_mock_content` was larger than planned).-4. Run `make agent-test` (full suite) to confirm no regressions. Note the result.-5. Hand back to the human.--Phase 4 picks up by reading this updated "Current state" plus the Phase 4 section below.-------## Phase 4 — Strict pre-check--Add `collect_signature_refs`, `SignaturesNotAllowedError`, and thread `allow_signatures` through the dry-run + bundle-validation surface.--### Phase 4.1 — Tests first (red)--- [x] `tests/integration/pipelex/pipe_signature/test_collect_signature_refs.py` — `class TestCollectSignatureRefs`:- - Fixtures construct mini libraries in Python with mixed pipe types.- - `test_operator_returns_empty` — a real `PipeLLM`'s `collect_signature_refs()` returns `set()`.- - `test_signature_returns_self` — a `PipeSignature`'s `collect_signature_refs()` returns `{self.pipe_ref}`.- - `test_controller_sequence_walks_steps` — a `PipeSequence` whose step references a signature returns `{sig.pipe_ref}`.- - `test_controller_parallel_walks_branches` — a `PipeParallel` whose branch references a signature returns `{sig.pipe_ref}`.- - `test_controller_condition_walks_outcomes` — a `PipeCondition` whose `outcomes` map (and `default_outcome`) references signatures returns the union of those signature `pipe_ref`s.- - `test_controller_batch_walks_branch` — a `PipeBatch` whose `branch_pipe_code` references a signature returns `{sig.pipe_ref}`.- - `test_nested_controller_walks_deeply` — `PipeSequence(steps=[PipeSequence(steps=[signature])])` — outer returns the leaf signature.- - `test_cycle_protection` — two pipes whose `pipe_dependencies()` mutually reference each other; walk terminates and returns the signatures found (if any).- - `test_unresolved_cross_package_dep_skipped` — controller references a cross-package pipe not in the library; walk does not raise and returns the signatures found in the resolvable subgraph.- > Deviation: tests pass `pipe_lookup=get_optional_pipe` to `collect_signature_refs` because the walker takes a required `pipe_lookup` callable (see Status block for the cycle reason).-- [x] `tests/integration/pipelex/pipe_signature/test_signatures_not_allowed_error_message.py` — `class TestSignaturesNotAllowedErrorMessage`:- - `test_dep_paths_keys_are_qualified_pipe_refs` — `error.dep_paths` keys are fully-qualified pipe_refs (`domain.code`), not bare codes; the values are ordered lists of pipe_refs naming the controllers traversed to reach the signature.- - `test_message_lists_each_signature_with_dep_path` — `str(error)` (or `error.message`) contains one human-readable line per signature, naming the signature pipe_ref and the dep chain that reached it.- - `test_message_includes_fix_suggestion` — message tells the user how to recover ("replace with a real implementation, or re-run with `--allow-signatures`").-- [x] `tests/integration/pipelex/pipe_signature/test_dry_run_strict_mode.py` — `class TestDryRunStrictMode`:- - `test_strict_default_passes_when_no_signatures` — dry-run of a real `PipeSequence` of real pipes with `allow_signatures=False` (the default) succeeds.- - `test_strict_fails_on_signature_in_sequence` — dry-run with a signature step + `allow_signatures=False` raises `SignaturesNotAllowedError`; the error's `signature_refs` includes the leaf signature's `pipe_ref`.- - `test_lenient_succeeds_on_signature_in_sequence` — same setup with `allow_signatures=True` succeeds; the working memory after dry-run contains a mock stuff for the signature's output.- - `test_strict_error_lists_all_signatures` — a controller depending on two signatures; the error lists both qualified `pipe_ref`s in `signature_refs`.- - `test_strict_error_includes_dep_paths` — the error's payload includes the dep-chain that reached each signature, with qualified pipe_refs throughout (e.g. `{"sigs.summarize_doc": ["pipelines.process_doc", "pipelines.inner_seq"]}`).- - `test_validate_bundle_strict_fails_on_signature` — `validate_bundle(blueprints=[...], allow_signatures=False)` raises `ValidateBundleError`; the raised error has a non-None `signature_check_error: SignaturesNotAllowedError` carrying the leaf signature's pipe_ref in `signature_refs` and the dep chain in `dep_paths`.- - `test_validate_bundle_lenient_passes_on_signature` — same `allow_signatures=True` passes; the returned `ValidateBundleResult.dry_run_result` contains a `SUCCESS` entry for every loaded pipe (including the signature itself).-- [x] Run: `.venv/bin/pytest -q tests/integration/pipelex/pipe_signature/`. Confirm red on the new tests; previously-green tests stay green.--### Phase 4.2 — Implementation (green)--- [x] `pipelex/core/pipes/pipe_abstract.py`:- - Add `collect_signature_refs(self, visited: set[str] | None = None) -> set[str]` — walks `self.pipe_dependencies()` via `get_optional_pipe`, accumulates signatures, short-circuits on `visited`. Track visited by `self.pipe_ref` (qualified, e.g. `domain.code`). `pipe_dependencies()` returns bare codes; resolution via `get_optional_pipe(pipe_code=...)` is naive (first match wins for ambiguous bare codes) — that mirrors existing behavior across the codebase. The walk does not attempt to disambiguate. (Pseudocode in `wip/signature-based-validation.md`.)- - Add `collect_signature_paths(self, visited: set[str] | None = None, current_path: list[str] | None = None) -> dict[str, list[str]]` — companion of `collect_signature_refs` that returns `dict[signature_pipe_ref, list[controller_pipe_refs]]`. Used by `SignaturesNotAllowedError` to render dep-chain UX. Keys and path entries are all qualified pipe_refs.- > Deviation: both methods take a required `pipe_lookup: Callable[[str], PipeAbstract | None]` parameter rather than calling `get_optional_pipe` internally. Reason: pyright's `reportImportCycles` flagged even a function-local import (`from pipelex.hub import get_optional_pipe`) because the cycle pipe_abstract → hub → libraries.library → libraries.pipe.pipe_library → pipe_abstract is real. Callers pass `get_optional_pipe`; recursion passes the same callable through. A `PipeLookupCallable: TypeAlias` lives under `TYPE_CHECKING` for the type annotation.-- [x] `pipelex/pipe_signature/exceptions.py`:- - Add `SignaturesNotAllowedError(PipelexError)` with fields `pipe_ref: str` (the entry-point pipe being validated), `signature_refs: set[str]` (qualified pipe_refs of every reachable signature), `dep_paths: dict[str, list[str]]` (signature pipe_ref → ordered list of controller pipe_refs naming the path to the signature).- - `__str__` / message format: one line per signature, naming its qualified pipe_ref and the dep chain, ending with the suggested fix ("replace with a real implementation, or re-run with `--allow-signatures`").-- [x] `pipelex/pipe_run/dry_run.py`:- - Update `dry_run_pipe` signature to `async def dry_run_pipe(pipe: PipeAbstract, *, allow_signatures: bool = False, raise_on_failure: bool = False) -> DryRunOutput`.- - At the top of the body, if `not allow_signatures`: call `sig_refs = pipe.collect_signature_refs()`; if non-empty, build `dep_paths = pipe.collect_signature_paths()` and raise `SignaturesNotAllowedError(pipe_ref=pipe.pipe_ref, signature_refs=sig_refs, dep_paths=dep_paths)`.- - Update `dry_run_pipes` similarly (the same flag, threaded into each `dry_run_pipe` call).- > Deviation: `dry_run.py` passes `pipe_lookup=get_optional_pipe` to the walker (see above).- > Deviation: also updated `tests/unit/pipelex/pipe_run/test_dry_run.py` mock-based tests to set `mock_pipe.collect_signature_refs.return_value = set()`; otherwise `MagicMock` returns a truthy mock and the strict pre-check fires.-- [x] `pipelex/pipeline/validate_bundle.py`:- - Add `allow_signatures: bool = False` to `validate_bundle` and `validate_bundles_from_directory`. Thread to `dry_run_pipes` in all three branches of `validate_bundle` (now that the `dry_run_pipes` calls are uncommented) and in `validate_bundles_from_directory`.- - Add a dedicated `except SignaturesNotAllowedError as sig_error:` branch to both functions' try/except chains. Wrap into `ValidateBundleError` (extend the error with a `signature_check_error: SignaturesNotAllowedError | None = None` field so dep_paths and signature_refs are preserved end-to-end). The CLI's pipe_cmd / bundle_cmd surfaces should display the rendered message from `signature_check_error` when present (Phase 5 detail).- - **Why this works:** `validate_bundle` was updated on this branch to actually invoke `dry_run_pipes` end-to-end. The strict check inside `dry_run_pipe` therefore fires for every `validate_bundle` caller — no separate `check_no_signatures_reachable` helper is needed. The earlier review concern about routing is moot.-- [x] `pipelex/cli/commands/validate/_validate_core.py`:- - Thread `allow_signatures` from the CLI into `dry_run_pipe`/`dry_run_pipes`/`validate_bundle`. (CLI flag wiring is Phase 5; here just thread the parameter so the function plumbing is in place.)- > Deviation: also renamed `_validate_pipe_or_bundle` to `validate_pipe_or_bundle` (drop the leading underscore) so the Phase 5 CLI tests can import it without tripping `PLC2701` everywhere. The behavior is unchanged; the only call site (`execute_validate`) was updated in lock-step.--### Phase 4.3 — Lint and targeted tests--- [x] `make agent-check` clean.-- [x] `.venv/bin/pytest -q tests/integration/pipelex/pipe_signature/ tests/unit/pipelex/pipe_run/` — all green.-------## Phase 5 — CLI surface--Surface the strict/lenient choice at the CLI. The agent CLI defaults to lenient because that's its primary use case.--### Phase 5.1 — Tests first (red)--CLI tests should drive the typer command directly via `CliRunner` rather than spawning subprocesses (matches existing CLI test patterns in this repo).--- [x] `tests/integration/pipelex/cli/test_validate_signatures_cli.py` — `class TestValidateSignaturesCli`:- - Fixtures load a tiny in-memory library directory with a signature-containing bundle.- - `test_validate_pipe_strict_default_fails` — `pipelex validate <pipe>` (no flag) on a pipe that reaches a signature exits non-zero with the signature listed in stderr.- - `test_validate_pipe_allow_signatures_passes` — `pipelex validate <pipe> --allow-signatures` exits zero on the same bundle.- - `test_validate_all_strict_default_passes_with_orphan_signature` — `pipelex validate --all` succeeds even when the library contains an orphan signature (no caller); confirms "iterate non-signature pipes only" semantics.- - `test_validate_all_strict_fails_with_caller_of_signature` — `pipelex validate --all` fails when a non-signature pipe reaches a signature.- - `test_validate_all_allow_signatures_passes` — same setup, lenient flag, passes.- > Deviation: tests drive `validate_pipe_or_bundle` and `do_validate_all_libraries_and_dry_run` directly (no CliRunner in this repo). Rich console output isn't captured by pytest's `capsys` for nested writers, so the "signature listed in stderr" assertion was dropped in favor of asserting the `typer.Exit(1)` exit code; the message-in-output check is exercised by Phase 4's `test_signatures_not_allowed_error_message.py`.-- [x] `tests/integration/pipelex/cli/test_agent_validate_defaults_lenient.py` — `class TestAgentValidateDefaultsLenient`:- - `test_agent_validate_defaults_to_lenient` — calling the agent CLI's validate against a signature-bundle exits zero without an explicit flag.-- [x] Run: `.venv/bin/pytest -q tests/integration/pipelex/cli/`. Confirm red.--### Phase 5.2 — Implementation (green)--- [x] `pipelex/cli/commands/validate/app.py` (and any sibling typer command modules):- - Add `--allow-signatures` boolean option, default `False`. Thread to `execute_validate`.- > Deviation: the option lives on `validate_pipe_cmd` and `validate_bundle_cmd` (the two surfaces that actually take subcommand args). `app.py` only wires the `_ValidateGroup` and forwards args to subcommands, so it didn't need editing.-- [x] `pipelex/cli/commands/validate/_validate_core.py`:- - Already threads `allow_signatures` from Phase 4. Surface it to typer.- - `do_validate_all_libraries_and_dry_run` — iterate non-signature pipes only when `allow_signatures=False` (use `pipe.is_signature`); when `allow_signatures=True`, iterate everything (signatures dry-run trivially).-- [x] `pipelex/cli/agent_cli/commands/validate/_validate_core.py`:- - Default `allow_signatures=True` for the agent CLI. Add an `--strict` flag if needed (defer if not required by current tests).- > Deferred: `--strict` flag — not needed by current tests.-- [x] Validation summary: when lenient and signatures were involved, print a final line like `"Validated N pipes (M signatures)."`. Add to whatever summary line currently exists.- > Deviation: instead of a separate trailing line, the suffix is **appended in-place** to the existing summary lines via a `_format_signatures_summary_suffix` helper in `pipelex/cli/commands/validate/_validate_core.py`. Applied to all three lenient surfaces — `validate pipe --all` ("Setup sequence passed OK, …"), `validate pipe <code>` ("Successfully validated pipe '…'"), and `validate bundle <path>` ("Successfully validated bundle '…'"). Suffix is `" (1 signature)"` for one and `" (M signatures)"` for more; empty when no signature was involved so fully-implemented bundles read naturally. Strict mode never appends. Covered by `tests/integration/pipelex/cli/test_validate_signatures_summary.py`.--Additional touches landed in Phase 5.2 (not enumerated above but required for the tests to pass):--- [x] `pipelex/cli/error_handlers.py` — `_display_validation_error_details` now renders `exc.signature_check_error` under an "Unimplemented Signatures:" heading so the strict-mode UX surfaces the dep chain.--### Phase 5.3 — Lint and targeted tests--- [x] `make agent-check` clean.-- [x] `.venv/bin/pytest -q tests/integration/pipelex/cli/ tests/integration/pipelex/pipe_signature/` — all green.-------⛔ **CHECKPOINT B — STOP HERE**--**Coding must stop after Phase 5.** Do **not** start Phase 6 in the same session.--Before handing back:--1. Confirm every box above is ticked. Add `> Deviation:` notes inline for anything that differed from the plan.-2. Update the top "Status" line: `Status: Phases 1–5 landed (YYYY-MM-DD). Strict / lenient validation works end-to-end through Python and CLI. E2E .mthds tests and docs are open.`-3. Add to the "Current state" paragraph: e2e .mthds parsing not yet exercised; some surprise may surface there (interpreter or schema generator might need touching).-4. Run `make agent-test` — full suite green.-5. Hand back to the human.--Phase 6 picks up by reading the updated "Current state" plus the Phase 6 section below.-------## Phase 6 — End-to-end tests with real `.mthds`--This phase exercises the full path: `.mthds` file → `PipelexInterpreter` → `PipelexBundleBlueprint` → library load → factory → runtime → dry-run. No new production code is expected; if something breaks, the design surfaces a real bug rather than a test issue.--### Phase 6.1 — Tests first (red)--Build `.mthds` fixtures under `tests/e2e/fixtures/signature_bundles/` and reference them from the tests.--- [x] `tests/e2e/fixtures/signature_bundles/signature_only.mthds` — a bundle whose `main_pipe` is a `PipeSignature`. Single domain, single concept, single pipe.-- ```toml- domain = "signature_demo"- main_pipe = "summarize_doc"-- [concept]- Document = "A document concept used for testing signatures."-- [pipe.summarize_doc]- type = "PipeSignature"- description = "Produces a summary of a document (contract only)."- inputs = { doc = "Document" }- output = "Text"- ```-- > Deviation: the concept is named `SigDocument` (not `Document`) because `Document` collides with the native concept name. The bundled `plxt` schema doesn't yet know about `PipeSignature`, so `Write`/`Edit` calls trigger the `validate-mthds.sh` PostToolUse hook (blocked once during the session — the file was still written, and remaining fixtures were authored via `Bash` heredoc to avoid spam). Same constraint forced `tests/e2e/fixtures/signature_bundles/**` to be added to `.pipelex/plxt.toml` excludes so `plxt-lint` (part of `make agent-check`) stays green.--- [x] `tests/e2e/fixtures/signature_bundles/mixed_with_signature_step.mthds` — a `PipeSequence` whose second step is a signature.-- ```toml- domain = "signature_mixed"- main_pipe = "process_doc"-- [concept]- Document = "A document concept."- Summary = "A summary concept."-- [pipe.extract_doc]- type = "PipeLLM"- description = "Extract text from a document."- inputs = { doc = "Document" }- output = "Text"- prompt = "Extract text from $doc."-- [pipe.summarize_extracted]- type = "PipeSignature"- description = "Summarize extracted text (contract only)."- inputs = { extracted = "Text" }- output = "Summary"-- [pipe.process_doc]- type = "PipeSequence"- description = "Extract then summarize."- inputs = { doc = "Document" }- output = "Summary"- steps = [- { pipe = "extract_doc", result = "extracted" },- { pipe = "summarize_extracted", result = "summary" },- ]- ```-- > Deviation: concepts renamed `MixDoc` / `MixSummary` (same native-collision reason as above). The PipeLLM `prompt` uses `$doc` (inline value) rather than `@doc` because PipeLLM rejects inline `@` sigils.--- [x] `tests/e2e/fixtures/signature_bundles/multi_input_multiplicity.mthds` — signature with `Document[]` and `Image[3]` inputs.-- ```toml- domain = "signature_multiplicity"- main_pipe = "fuse_docs_and_images"-- [concept]- Document = "A document."- Image = "An image."- Report = "A report."-- [pipe.fuse_docs_and_images]- type = "PipeSignature"- description = "Combine docs and exactly 3 images (contract only)."- inputs = { docs = "Document[]", images = "Image[3]" }- output = "Report"- ```-- > Deviation: concepts renamed `FuseDoc` / `FuseImage` / `FuseReport` because `Document`, `Image`, and the singular concept naming would collide with native concepts. Multiplicity (`FuseDoc[]`, `FuseImage[3]`) round-trips through the interpreter unchanged.--- [x] `tests/e2e/test_signature_validation_mthds.py` — `class TestSignatureValidationE2E`:- - `test_signature_only_bundle_strict_fails` — `validate_bundle(mthds_file_path=signature_only_path)` with default strict raises `ValidateBundleError` wrapping `SignaturesNotAllowedError`.- - `test_signature_only_bundle_lenient_passes` — same call with `allow_signatures=True` returns a `ValidateBundleResult` with a populated `dry_run_result`.- - `test_mixed_bundle_strict_fails_with_dep_path` — strict mode on `mixed_with_signature_step.mthds` fails; the error message includes both the leaf signature's `pipe_ref` and the controller's `pipe_ref` in the dep path.- - `test_mixed_bundle_lenient_passes_and_produces_mock` — lenient mode succeeds; dry-run output for `process_doc` contains a `Summary` mock stuff.- - `test_multiplicity_inputs_lenient_passes` — lenient mode on `multi_input_multiplicity.mthds` succeeds.- - `test_live_run_signature_pipeline_fails` — running the bundle live (via `PipelexRunner.execute_pipeline` with `pipe_run_mode=LIVE`) raises `PipelineExecutionError` whose underlying cause is `PipeSignatureNotExecutableError`.- > Deviation: `DryRunOutput` is a thin status object (no `working_memory`), so `test_mixed_bundle_lenient_passes_and_produces_mock` asserts that both `process_doc` and `summarize_extracted` reach `DryRunStatus.SUCCESS` rather than reaching into the working memory for the mocked stuff. Other tests adjusted similarly.-- [x] `tests/e2e/test_signature_validation_cli.py` — `class TestSignatureValidationCli`:- - `test_cli_validate_signature_bundle_strict_fails` — `pipelex validate signature_only.mthds` exits non-zero with the signature listed in stderr.- - `test_cli_validate_signature_bundle_lenient_passes` — same with `--allow-signatures` exits zero.- > Deviation: tests drive `_validate_pipe_or_bundle` directly (matches the integration CLI tests) rather than spawning a subprocess. The "stderr signature listed" check is exercised by Phase 4's `test_signatures_not_allowed_error_message.py`; here we assert the non-zero exit only.-- [x] Run: `.venv/bin/pytest -q tests/e2e/`. Confirm: most red because interpreter / schema generator hasn't been exercised on the new `type = "PipeSignature"` shape yet.- > Deviation: the interpreter happily produced `PipeSignatureBlueprint` from `type = "PipeSignature"` on the first try — no fix required there. The only failing test was `test_mixed_bundle_strict_fails_with_dep_path`, and the root cause was on the runtime side (`dry_run_pipes` aggregation), not the interpreter.--### Phase 6.2 — Make e2e tests green--Whatever this phase surfaces is a real bug in the previous phases. Expected categories:--- `PipelexInterpreter` round-trip: confirm TOML parsing of `type = "PipeSignature"` produces a `PipeSignatureBlueprint`. If the discriminator dispatch fails in `PipelexBundleBlueprint.pipe`, fix it.-- Library load: confirm `LibraryCrateFactory.make_from_blueprints` + `library_manager.load_from_blueprints` accept the new blueprint shape. If `concept_codes_from_the_same_domain` filtering doesn't accommodate signature concepts, fix.-- Bundle-level validation: `validate_local_pipe_references` already key-based, should pass. Confirm.-- Schema generator: `pipelex-dev generate-mthds-schema` should include `PipeSignature` in the union. Verify and regenerate the schema file (`derived/mthds_schema.json`) if it exists in the repo.--- [x] Make every Phase 6.1 test green.- > Deviation: only one production code change was needed. In `pipelex/pipe_run/dry_run.py::dry_run_pipes`, the strict-mode pre-check aggregated dep paths with a "first wins" rule. When the signature pipe itself was iterated first, the empty `[]` dep chain was recorded and never replaced by the longer, controller-rooted chain discovered later. Switched the aggregation to prefer the longest known dep chain so the error message always shows the most informative path. No interpreter, library, or schema generator fixes were required.-- [x] If the JSON schema is checked in: run `.venv/bin/pipelex-dev generate-mthds-schema` and commit the diff.- > Deviation: no JSON schema is checked in to this repo (the artifact lives in `vscode-pipelex/`), so nothing to regenerate here. The bundled `plxt` schema therefore stays unaware of `PipeSignature` until `vscode-pipelex` ships a new release — captured in "Known gaps after Phase 6" above.--### Phase 6.3 — Lint and full tests--- [x] `make agent-check` clean.-- [x] `make agent-test` — full suite green.-------⛔ **CHECKPOINT C — STOP HERE**--**Coding must stop after Phase 6.** Do **not** start Phase 7 in the same session.--Before handing back:--1. Confirm every box above is ticked. Add `> Deviation:` notes for anything that surprised you (especially in Phase 6.2 — the e2e phase is the most likely to surface design gaps).-2. Update the top "Status" line: `Status: Phases 1–6 landed (YYYY-MM-DD). End-to-end signature validation works from .mthds. Docs and CHANGELOG are open.`-3. Append a "Known gaps after Phase 6" subsection listing anything noticed but deferred (e.g. "schema generator output diff visible; not committed").-4. Hand back to the human.--Phase 7 picks up by reading the updated "Status" plus the Phase 7 section below.-------## Phase 7 — Docs, CHANGELOG, polish--### Phase 7.1 — Docs--- [x] `docs/` MTHDS authoring guide: add a section "Signature pipes" describing the `type = "PipeSignature"` shape, listing all valid fields, with the three example fixtures from Phase 6 inlined as illustrations.- > Landed as a new top-level page under `building-methods/pipes/`: `docs/building-methods/pipes/signature-pipes.md`. Wired into `mkdocs.yml` in both nav placements (after the controllers list, before `Optimize Cost & Quality`). Cross-linked from `docs/building-methods/pipes/index.md` and `docs/tools/cli/validate.md`. The three Phase 6 fixtures are inlined verbatim.-- [x] CLI help text — confirm `pipelex validate --help` mentions `--allow-signatures` with a one-line description ("Accept PipeSignature placeholders in the dependency graph (lenient mode).").- > Already wired by Phase 5 on both `validate_pipe_cmd` and `validate_bundle_cmd` with that exact help string. Also added the flag to `docs/tools/cli/validate.md` under both subcommands' Options blocks with example invocations.-- [x] `pipelex/builder/pipe/pipe_signature.py`: tighten the class docstring with the new contract (drop the `result` line, mention `signature_for`, mention strict-vs-lenient).- > Rewrote the docstring to lead with the design intent (top-down sketching, replace each signature with a real operator), then call out validation behavior (strict / lenient / live), multiplicity, and the optional `signature_for` field. The pre-existing docstring already had no `result` reference, so nothing to drop.--### Phase 7.2 — CHANGELOG--- [x] `CHANGELOG.md` under `## [Unreleased]`:- - `### Added` — `PipeSignature` pipe type, `--allow-signatures` flag, agent CLI lenient default, `collect_signature_refs` graph walk.- - `### Changed` — `dry_run_pipe`/`dry_run_pipes`/`validate_bundle` now accept `allow_signatures: bool = False`.- - Cross-link `wip/signature-based-validation.md`.- > New `## [Unreleased]` section inserted above `[v0.28.0]`, with `### Added` (signature pipe type + factory + runtime, `--allow-signatures` flag with lenient agent-CLI default, `collect_signature_refs`/`collect_signature_paths` graph walk) and `### Changed` (the `allow_signatures` parameter threaded through `dry_run_pipe`/`dry_run_pipes`/`validate_bundle`/`validate_bundles_from_directory`, plus the new `signature_check_error` field on `ValidateBundleError`). Cross-link to `wip/signature-based-validation.md` included in the Added bullet.--### Phase 7.3 — Final lint and tests--- [x] `make agent-check` clean.-- [x] `make agent-test` — full suite green.--### Phase 7.4 — Merge-gate cross-repo cleanup (do NOT skip)--The branch carries a deliberate, scoped schema configuration: `.pipelex/plxt.toml` declares `derived/mthds_schema.json` as the schema source for `**/*.mthds` via a `[[rule]] / [rule.schema] path = ...` block. That is the correct configuration *on this branch* — the bundled `plxt` schema doesn't yet describe `PipeSignature`, so the local schema is the only one that accurately validates the runtime change introduced here. The branch is internally consistent and `plxt lint` is green.--The merge-gate is to land the matching `vscode-pipelex` release and then *retire* the branch-local schema rule in the same merge, so the bundled schema takes over again on `main`:--- [ ] In the `vscode-pipelex/` repo: regenerate the bundled MTHDS JSON Schema so `type = "PipeSignature"` is a valid pipe type, and cut a release of `plxt` / the VS Code extension. (Open a tracking PR there if not done; link the PR URL here when filed.)-- [ ] Once the new `plxt` is installable: in **this** repo, in the merge commit, remove the `[[rule]] / [rule.schema] path = "derived/mthds_schema.json"` block at the bottom of `.pipelex/plxt.toml`. Decide in the same commit whether `derived/mthds_schema.json` should stay as a regenerable artifact or be deleted; nothing else in this repo reads it today.-- [ ] Re-run `make agent-check` after the rule is removed and the new `plxt` is installed: `plxt lint` must accept the signature fixtures *using the bundled schema*, with no per-rule rule. If it doesn't, the `vscode-pipelex` release is incomplete and the merge should not land yet.-- [ ] Verify the `validate-mthds.sh` PostToolUse hook accepts `Write`/`Edit` on `PipeSignature` fixtures with the new `plxt` installed and the local rule removed.--**Cross-repo coordination reminder:** the `vscode-pipelex` schema regen is the gating step. Local `plxt` results during the branch lifetime are valid (the branch-local rule is intentional). The cross-check that matters is "does `plxt lint` stay green *after* the branch-local rule is removed and the new `plxt` is installed" — that is the actual ship-readiness signal. The Phase 7.4 box is closed only when that cross-check is green at merge time.-------⛔ **CHECKPOINT D — FINAL**--**Coding stops here.** Before handing back:--1. Confirm every box is ticked; add deviation notes for anything that didn't match the plan.-2. **Double-check Phase 7.4 is complete.** If the `vscode-pipelex` schema update has not landed yet, the branch is NOT ready to ship — do not update the Status line to "Ready to ship". Instead, leave it as "Phases 1–7 landed, blocked on vscode-pipelex schema update" and hand back with that explicit blocker noted.-3. Update the top "Status" line: `Status: All phases landed (YYYY-MM-DD). Ready to ship via /release.` (only if Phase 7.4 is fully done).-4. Hand back to the human.-------## Out of scope / explicit non-goals--- Signature concepts (concept stubs analogous to pipe signatures). Bigger design, separate plan.-- `signature_for`-driven mock specialization (e.g. `signature_for = "PipeImgGen"` mints an actual ImageContent URL). Deferred — flagged in the design doc as a phase-2 enhancement.-- Migrating existing bundles to use signatures. Authors opt in.-- Parallel `[signature.foo]` TOML table as syntactic sugar. Possible later, not a structural alternative.-- Runtime-time validation of signatures during live execution. Live-run on a signature must raise; that's the only enforcement.-------## Reference: file targets at a glance--Production code:--- `pipelex/core/pipes/pipe_blueprint.py` — enum additions, `is_signature` on `PipeBlueprint`.-- `pipelex/core/pipes/pipe_abstract.py` — `is_signature`, promoted `pipe_dependencies`.-- `pipelex/pipe_signature/signature_walk.py` — free-function dependency-graph walk: `collect_signature_refs` / `collect_signature_paths` (see post-archive update note).-- `pipelex/core/bundles/pipelex_bundle_blueprint.py` — `PipeBlueprintUnion` gains `PipeSignatureBlueprint`.-- `pipelex/pipe_signature/` — new package: blueprint, runtime, factory, exceptions.-- `pipelex/builder/pipe/pipe_signature.py` — three corrections, `to_blueprint()`.-- `pipelex/builder/pipe/pipe_spec_union.py` — `PipeSignature` added to union.-- `pipelex/builder/pipe/pipe_spec_map.py` — `PipeSignature` added to map.-- `pipelex/pipe_run/dry_run.py` — `allow_signatures` parameter, strict pre-check, new `convert_stuff_spec_to_typed_named` helper.-- `pipelex/pipeline/validate_bundle.py` — `allow_signatures` threaded through all three branches and into `validate_bundles_from_directory`.-- `pipelex/core/memory/working_memory_factory.py` — `make_mock_stuff` helper extracted from `make_mock_inputs`.-- `pipelex/cli/commands/validate/app.py` + `_validate_core.py` — `--allow-signatures` flag.-- `pipelex/cli/agent_cli/commands/validate/_validate_core.py` — lenient default.--Tests:--- `tests/unit/pipelex/core/pipes/test_pipe_blueprint_signature_enums.py`-- `tests/unit/pipelex/core/pipes/test_pipe_abstract_signature_surface.py`-- `tests/unit/pipelex/builder/pipe/test_pipe_signature_spec.py`-- `tests/unit/pipelex/builder/pipe/test_pipe_spec_union_signature.py`-- `tests/integration/pipelex/pipe_signature/conftest.py`-- `tests/integration/pipelex/pipe_signature/test_pipe_signature.py`-- `tests/integration/pipelex/pipe_signature/test_pipe_signature_in_blueprint_union.py`-- `tests/integration/pipelex/pipe_signature/test_collect_signature_refs.py`-- `tests/integration/pipelex/pipe_signature/test_signatures_not_allowed_error_message.py`-- `tests/integration/pipelex/pipe_signature/test_dry_run_strict_mode.py`-- `tests/integration/pipelex/cli/test_validate_signatures_cli.py`-- `tests/integration/pipelex/cli/test_agent_validate_defaults_lenient.py`-- `tests/e2e/fixtures/signature_bundles/signature_only.mthds`-- `tests/e2e/fixtures/signature_bundles/mixed_with_signature_step.mthds`-- `tests/e2e/fixtures/signature_bundles/multi_input_multiplicity.mthds`-- `tests/e2e/test_signature_validation_mthds.py`-- `tests/e2e/test_signature_validation_cli.py`diff --git a/wip/history/csv-support/README.md b/wip/history/csv-support/README.mddeleted file mode 100644index b8dbc46ed..000000000--- a/wip/history/csv-support/README.md+++ /dev/null@@ -1,97 +0,0 @@-# CSV Support — Feature Guide & PR-Review Reference--Branch `feature/Support-csv` (worktree `_csv`), targeting `dev`. This document describes the shipped CSV input/output feature so a PR-review agent can review the diff against intent without re-deriving the design. Design rationale lives in [`design.md`](design.md); deferred follow-ups in the other `*.md` files in this folder.--**Status: v1 complete — PR [#955](https://github.com/Pipelex/pipelex/pull/955) finalized, ready to merge.** All phases done; `make agent-check` green (pyright 0 / mypy 0 / ruff + plxt clean); full `make agent-test` green; docs + changelog landed. The PR went through 9 automated review rounds (greptile / cubic / codex) to convergence, then a pre-landing gstack `/review` (5 specialists + Codex adversarial) — which caught 3 issues the bot rounds missed (BOM/Excel input, a coercion-error data leak under STRICT, and `--save-csv` error framing) — and the bot re-review of those fixes also converged. All review threads resolved; CI green; mergeable. Deferred items are documented in this folder (security-followups, phase2/phase3-4 followups).-------## What this feature does--A CSV (or, behind a not-yet-built seam, an Excel `.xlsx`) file maps to a `ListContent[YourConcept]`: each row is one instance of a flat concept, each column is one of its fields. The MTHDS language surface is unchanged — there is **no new native content type or concept**. You describe the row concept exactly as any structured concept.--- **Input:** an `inputs.json` reference whose `content.url` ends in a tabular suffix, under a flat structured concept, is read into a typed `ListContent` (one file → one list; the concept names the *row* type). Reachable from the CLI, the runner API, and programmatic callers, because the hook lives in core, not the CLI.-- **Output:** `--save-csv <path>` on `pipelex run pipe|bundle|method` writes the main stuff to a literal, cwd-relative CSV path when that output is a flat list.--The headline acceptance test is a full round trip: `people.csv` → `PipeBatch` → `PipeSequence`(`PipeLLM` + `PipeCompose`) → `summaries.csv`, green in dry mode via both the in-process runner and a real CLI subprocess.--User-facing docs: [`docs/building-methods/pipes/csv-input-and-output.md`](../../../docs/building-methods/pipes/csv-input-and-output.md) + `--save-csv` in [`docs/tools/cli/run.md`](../../../docs/tools/cli/run.md); CHANGELOG `[Unreleased]`.-------## Code map (what to review, where)--**Codec — `pipelex/tools/tabular/csv_codec.py`** (stdlib `csv`, one module per CQ2; no codec/binding split until `.xlsx` earns it):--- `read_rows` / `write_rows` — low-level primitives over an internal `_read_table` (explicit header + raw rows) / `_row_to_dict` (pads short/blank rows). `read_rows` has no suffix gate.-- `flat_field_names(row_model)` — the **single shared flatness classifier** used by BOTH read and write. `_is_flat_annotation` unwraps `Optional`/PEP-604 unions, ACCEPTS `str/int/float/bool/date/datetime` + **string-valued** `Literal`/`Enum` (only those round-trip through a CSV cell), REJECTS genuine multi-arm unions, non-string `Literal`/`Enum`, list/dict/nested-model/`Any`. Raises `CsvFlatnessError` naming field + concept; returns declared field order.-- `list_content_from_csv(path, row_model, *, delimiter, encoding)` — read path. Maps `""`→`None` before validate; extra/missing-required column → `CsvColumnError`; missing-optional column → field absent → pydantic default (`None`); per-row `ValidationError` → `CsvCoercionError` (1-based row + field).-- `csv_from_list_content(list_content, row_model, path, *, delimiter, encoding)` — write path. Headers from the **declared `row_model`** (so an empty list still writes a header-only file); serializes via `model_dump(mode="json")`; `_to_cell` maps `None`→`""` and bool→`true`/`false`.-- `is_tabular_path(path)` — suffix membership in `_TABULAR_SUFFIXES` (`.csv`, `.xlsx`).-- `assert_supported_table_suffix(path)` — the format seam: `.csv` ok; `.xlsx` → `CsvError` naming `pipelex[tabular]`; else generic `CsvError`.-- `delimiter`/`encoding` are function params (`,` / `utf-8`); no main-config touched (D1).--**Error taxonomy — `pipelex/tools/tabular/exceptions.py`:** `CsvError(ToolError)` base (`ErrorDomain.INPUT`) + `CsvReadError` / `CsvFlatnessError` / `CsvColumnError` / `CsvCoercionError`. All are `PipelexError`s, so the CLI prints them cleanly (no traceback) via `execute_run`'s `except PipelexError`. The codec catches every raw `OSError` / `UnicodeDecodeError` / `csv.Error` / pydantic `ValidationError` at its boundary and re-raises typed, carrying file path + 1-based row + concept + field — no raw third-party exception escapes into core/runner.--**Input hook — `pipelex/core/stuffs/stuff_factory.py`:** `_try_make_csv_list_stuff`, called from **Case 2.5** of `make_stuff_from_stuff_content_or_data` BEFORE the record `model_validate`. Detection: content has a `url` str key whose **path** (via `urlsplit(url).path`, so a query-string/fragment can't hide the suffix) is tabular AND the concept is **non-native** (`Concept.is_native_concept` False — native file concepts like Image/PDF keep their own url handling). Remote guard: `resolve_uri(url)` must be a `ResolvedLocalPath` AND `"://" not in resolved.path` (the `://` clause catches `s3://`/`gs://`, which `resolve_uri` currently mislabels as local) — else `CsvError` "local file paths only in v1". Row model = `concept.get_structure_class()` (wrapped so an unregistered structure class raises a typed `CsvError`, not a raw `ConceptValueError`). Returns `None` for ordinary record dicts (no url / non-tabular suffix / native) → falls through to normal Case 2.5.--**Output hook — `pipelex/cli/commands/run/_run_core.py`:** `_execute_run` / `execute_run` take keyword-only `save_csv: str | None = None`. The three run commands (`pipe_cmd.py` / `bundle_cmd.py` / `method_cmd.py`) declare `--save-csv` and forward it. The save block (after the working-memory save): rejects empty path, no-main-stuff, and non-`ListContent` main stuff with `typer.secho` + `Exit(1)`; creates the parent dir; writes via `csv_from_list_content(...)` at the literal cwd path (CQ1 — NOT under `--output-dir`); the codec's `CsvFlatnessError` covers non-flat rows.-------## Test map--- **Unit — `tests/unit/pipelex/tools/tabular/test_csv_codec.py`:** read→dicts, custom delimiter, missing-file/bad-encoding → `CsvReadError`, duplicate/blank header → `CsvColumnError`, write+read round-trip, empty→header-only; flatness classifier (declared order + `Literal` accept; list/dict/nested/`Union`/`Any` reject); coercion incl. `date`→`datetime.date`; empty-cell→`None` (optional) / required→`CsvCoercionError`; **T3 coercion accept/reject tables** (bool `true/false/1/0/yes/no/on/off`, int, float, ISO date / reject `abc`, `1,5`, `31/12/2020`); strict columns (extra/missing-required error, missing-optional all-`None`); `None`↔`""` both sides.-- **Integration — `tests/integration/pipelex/csv/test_csv_roundtrip.py`** (split per T1): (a) input-codec WorkingMemory build → `ListContent[Person]` + `death_year is None` for Vint Cerf; (b) output-codec hand-built `ListContent[PersonSummary]` → header + rows; (c) full pipeline via `PipelexRunner(...).execute_pipeline("summarize_people", ...)` → 3 items, names/countries from real rows (dry-mode-safe).-- **Integration — `tests/integration/pipelex/csv/test_csv_input_detection.py`:** A3/CT1 pins (`.csv`-url column reads as a table; non-`.csv` url stays a single record), remote-url rejection (`http(s)`/`s3://`/`gs://`, incl. query-string/fragment), eager-IO refresh-after-mutation.-- **E2E — `tests/e2e/pipelex/cli/test_csv_run.py`** (`@pytest.mark.gha_disabled`, subprocess): `run pipe --save-csv --dry-run` → exit 0 + file at literal cwd path + header + one row per person; **T2 cheap wiring checks** that all three run commands declare `save_csv` and forward it; clean missing-`.csv` error (non-zero exit, no traceback, names the file).-- **Fixtures — `tests/integration/pipelex/csv/`:** the runnable demo lives in `csv_demo/` (`csv_demo.mthds` with flat `Person`/`PersonSummary`, `people.csv`, `inputs.json`); the url-field detection fixture is split out into `url_field_concept/` (`url_field_concept.mthds` with the pipe-less `Link` concept that declares its own `url` field, plus `links.csv`). Inline `[concept.X.structure]` blocks auto-generate registered `StructuredContent` subclasses (no `test_structures.py`).-------## The contract (input/output semantics — do not silently change)--- **Flat concepts only.** Scalar fields: `text`→`str`, `integer`→`int`, `number`→`float`, `boolean`→`bool`, `date`→`datetime.datetime` (the MTHDS `date` type generates `datetime.datetime`; the codec also accepts a hand-built `datetime.date`). Optionals allowed; string-valued `Literal`/`StrEnum` choices allowed. Nested/list/dict/`Union`/concept-typed/`Any`/non-string `Literal`/`Enum` → `CsvFlatnessError` naming the field. No silent flatten, no JSON-in-cell, no dropped cells.-- **Input headers:** required; must match field names exactly (no implicit remap). **Extra column → error; missing required column → error; missing *nullable* column → that field is `None` for all rows; missing non-nullable-but-defaulted column → that field keeps its own default** (CT2 lenient). Empty cell → `None` before pydantic (field must be optional).-- **Coercion:** through pydantic lax validation; the accept/reject set is pinned by T3 (see test map). Don't hand-roll coercion.-- **Output:** `--save-csv` triggers explicitly (no auto-emit). Header from the declared row model (empty list → header-only). `None`→`""` (never `"None"`); bool→lowercase. Literal cwd-relative path, parent dir created.-- **CSV is an I/O codec, not a prompt-render format** — it must NOT join the `rendered_plain/markdown/html/json` `TextFormat` router.-------## Locked design decisions (do not re-litigate — see design doc §6 + plan-eng-review)--- **Typed lists only**; full round-trip in v1; **no new native `Table` concept**.-- **stdlib `csv` in core; `.xlsx` deferred** behind `pipelex[tabular]`, reached through the thin suffix seam (`assert_supported_table_suffix`). Don't add openpyxl or build xlsx now.-- **Input hook in CORE** (`stuff_factory` Case 2.5), not the CLI, so it's reachable from the runner API. Eager local-disk read accepted there; codec stays a pure module.-- **v1 = local filesystem paths only.** Remote tabular `url` (`http(s)`/`s3`/`gs`/signed) actively rejected, never opened.-- **No main-config in v1 (D1).** Delimiter/encoding are codec params, never guessed; no TOML/CLI surface yet. `make tb` not needed.-- **`--save-csv` is literal/cwd-relative** (CQ1), keyword-only in `_execute_run`. No `--csv-path`.-- **A3/CT1:** keep `url` + tabular-suffix detection; accept the narrow residual (a flat concept with a `url` field given a `.csv` value reads as a table) — pinned by tests, documented as a limitation.-------## Documented limitations (written up in the guide page)--- **Local paths only** — remote tabular URLs rejected.-- **`.xlsx` recognized but not implemented** — routed to the `pipelex[tabular]` message; backend not built.-- **Empty string indistinguishable from `None`** — empty cell ⇄ `None` both directions.-- **Single-key `{"url": "x.csv"}` residual (A3/CT1)** — a bare `{"url": "x.csv"}` reference under a flat concept reads as a table, not a single record. Detection is gated to the exact single-key wrapper, so a record with sibling keys (`{"label": "Home", "url": "x.csv"}`) stays a record — its fields are never dropped (was a silent-drop; fixed in PR #955 review round 1).-- **No formula-injection escaping (CT3)** — cells written verbatim for data fidelity; opt-in escape deferred.-------## Deferred follow-ups (NOT in v1 — captured, do not expand scope into them)--Out of scope for v1 (design §10): `.xlsx` via openpyxl; streaming/out-of-core; Google Sheets/Parquet; delimiter/dialect auto-detection; a native `Table` concept; remote `.csv` URLs.--Captured review follow-ups (decisions deferred, not bugs to fix in this PR):--- [`phase3-4-review-followups.md`](phase3-4-review-followups.md): (A) `resolve_uri` doesn't classify `s3://`/`gs://` — a cross-caller `tools/uri` gap the `"://"` heuristic papers over; fix is its own change. The same root cause is the `?`/`#`-in-local-filename mis-strip (niche, graceful). (B) factory-altitude eager I/O + cwd-relative paths for non-CLI callers — design decision. **(C) RESOLVED in PR #955 review round 1** — detection tightened to the single-key `{"url": ...}` wrapper, so a multi-field record no longer silently drops siblings. Plus minor cleanups (output-side error framing, suffix-vocabulary drift, header-only-vs-empty-list parity, redundant casts).-- [`phase2-review-followups.md`](phase2-review-followups.md): non-string `Literal`/`Enum` flatness — **RESOLVED in PR #955 review round 3** (Option A: the gate now accepts only string-valued `Literal`/`Enum`, rejecting `Literal[int]`/`IntEnum` that serialize but don't coerce back).--Also addressed in PR #955 review rounds (not deferred): error row numbers now point at the physical CSV line (incl. across multiline quoted cells, via `csv.reader.line_num`); CSV detection gated to the single-key `{"url": ...}` wrapper; `--save-csv` validates before `mkdir` and frames its own failures as "Failed to --save-csv" (not a pipeline failure); newline + quote delimiters rejected with a typed `CsvError`; strict CSV parsing (malformed quotes → `CsvReadError`); `write_rows` wraps `UnicodeEncodeError`; an omitted *nullable* column is `None` for all rows regardless of the row-model default while a non-nullable defaulted column keeps its own default (CT2); `CsvError` opts into `_authors_caller_facing_message` so STRICT disclosure keeps its caller-facing message, and that message sanitizes the offending url (strips query/fragment/userinfo, guards malformed port + bad-IPv6-bracket) so a signed/token-bearing url never leaks; the CLI `s3://` remote-rejection bypass (`is_relative_local_path`) is fixed; **reads default to `utf-8-sig` so Excel-exported BOM CSVs parse**; and the coercion error no longer echoes the raw cell value (no data leak under STRICT).--Deferred (documented): local paths containing URL-reserved `?`/`#` (cubic agreed; entangled with `resolve_uri` scheme-classification, item A); nullable-but-required columns stay mandatory (phase2-followups #2). Security review follow-ups (gstack `/review`) in [`security-followups.md`](security-followups.md): platform-wide untrusted-`url` local-file-read confinement (pre-existing input model, not CSV-specific), CSV formula-injection (locked CT3 document-only decision), embedded-NUL path error-hygiene (niche).--Deferred feature TODOs: user-facing delimiter/encoding config (D1); opt-in CSV/Excel formula-escape flag (CT3); `SaveOptions` struct for `_execute_run`'s param clump; multiplicity-gated CSV detection (CT1 option C, only if the residual proves real).diff --git a/wip/history/csv-support/design.md b/wip/history/csv-support/design.mddeleted file mode 100644index 188a3ba58..000000000--- a/wip/history/csv-support/design.md+++ /dev/null@@ -1,145 +0,0 @@-# CSV Support — Design--Status: **design / pre-plan**. This document frames the problem and the design space so we can choose an approach before writing an implementation plan. It deliberately leaves the key decisions open (see "Decisions to make").--## 1. Goal--Let Pipelex methods take **CSV files as input** and produce **CSV files as output**, so tabular data flows in and out of pipelines as a first-class citizen rather than being shoehorned into a generic document blob or a hand-rolled `PipeFunc`.--Concrete user stories that should "just work":--- "I have a `contacts.csv`; run my enrichment pipe over each row and give me back an enriched CSV."-- "My pipeline produces a list of extracted records; save them as a `.csv` I can open in Excel."-- "Hand this spreadsheet to an LLM step as context."--## 2. Where this lands in the current architecture--Findings from the codebase (file references for the eventual plan):--- **Content/stuff type system** — every value is a `StuffContent` subclass wrapped in a `Stuff` (`pipelex/core/stuffs/`). Concrete types: `TextContent`, `ImageContent`, `DocumentContent`, `JSONContent`, `HtmlContent`, `NumberContent`, `ListContent[T]`, `StructuredContent` (base for custom concepts), `TextAndImagesContent`, `SearchResultContent`, `PageContent`, `MermaidContent`, `DynamicContent`.-- **Common interface** — `rendered_plain / rendered_html / rendered_markdown / rendered_json` (+ async variants), `smart_dump()`, `rendered_pretty()`. Defined on `StuffContent` (`pipelex/core/stuffs/stuff_content.py`).-- **Native concepts** — enumerated in `pipelex/core/concepts/native/concept_native.py` (`NativeConceptCode` → `structure_class`). Native concepts are part of the **MTHDS language surface**, not just runtime sugar.-- **Lists & batch** — `ListContent[T]` implements the Python list protocols; `PipeBatch` (`pipelex/pipe_controllers/batch/pipe_batch.py`) runs a branch pipe over each item of a `ListContent` and recombines outputs into a `ListContent`. `[]` multiplicity (`Person[]`, `Text[3]`) is declared in `.mthds` inputs/outputs.-- **Input path** — `pipelex run --inputs <json|path>` → `inputs.json` may reference local files by `url`; `_inputs_path_resolver.py` resolves relative paths; `working_memory_factory.make_from_pipeline_inputs` builds the `WorkingMemory`. File-backed types (`Document`, `Image`) store a **URL/reference**, not parsed bytes.-- **Output path** — `--save-main-stuff` writes `main_stuff.{json,md,html}`; `--save-working-memory` dumps the whole memory as JSON (`pipelex/cli/commands/run/_run_core.py`).-- **No tabular code exists** — no `csv`/`pandas`/`polars`/`openpyxl` dependency; `.xlsx` is only registered as a MIME type and treated as an opaque document.--**Key reframing:** a CSV table is structurally a `ListContent` of rows. So "CSV support" is less "add a blob type" and more **"a serialization bridge between CSV files and lists of (optionally typed) rows"** — the same role JSON already plays for structured content.--## 3. The central modeling fork--There are two coherent ways to model a CSV in the type system. They are **not** mutually exclusive — they can be layered — but v1 should pick a primary.--### Mode A — Typed rows: CSV ⇄ `ListContent[Concept]`--A CSV binds to a declared concept. `contacts.csv` under concept `Person` becomes `ListContent[Person]`; header cells map to concept fields; cell strings are coerced to field types.--- **Input:** `{ "contacts": { "concept": "myDomain.Person", "content": { "url": "contacts.csv" } } }` → parsed into typed rows.-- **Output:** any `ListContent[StructuredContent]` → CSV by flattening fields to columns.-- **Pros:** deeply native; rows immediately flow through `PipeBatch` and typed pipes; type-safe; round-trips with the rest of the system.-- **Cons:** needs a column↔field mapping rule, type coercion, and a policy for nested/complex fields (flatten with dotted keys / JSON-encode the cell / reject). Header semantics must be pinned down.--### Mode B — Table primitive: CSV ⇄ `TableContent`--A new content type holding the table verbatim: `headers: list[str]` + `rows` (list of string rows, or list of dicts). Renders to markdown/HTML/JSON table. No schema, lossless strings.--- **Pros:** simple; schema-free; ideal for "give the LLM a table" and "carry a spreadsheet through unchanged"; minimal coercion logic.-- **Cons:** untyped (everything is strings); to do typed per-row processing you still need a separate "table → typed list" step. Adds a **new native concept** to the MTHDS language surface (enum + registry + `Stuff` helpers + tests + arguably the `mthds/` spec).--### How they relate--Mode B is the **lossless, schema-free primitive**; Mode A is the **typed bridge**. A clean end state has both: read a CSV as a `Table` when you just want the grid, or bind it to a concept when you want typed rows; write any list/table back out. The decision is which is the *canonical* v1 surface and whether we add a native concept at all.--## 4. I/O surface design space--Independent of the modeling fork, each direction has surface choices (non-exclusive):--**Output**--- **(O1) Render format** — add a CSV rendering (`rendered_csv()` / a `csv` `TextFormat`). Cheapest; makes CSV "just another rendering of a list/table". But CSV doesn't fit the prompt-format router cleanly and is awkward for arbitrarily nested structures.-- **(O2) CLI export** — extend `--save-main-stuff` to also emit `main_stuff.csv` when the main stuff is a list/table, or add `--save-csv <path>`. Best for "run pipeline → get a CSV".-- **(O3) Dedicated pipe operator** — an explicit "write list/table to CSV artifact" operator. Most explicit, most plumbing; probably overkill for v1.--**Input**--- **(I1) `inputs.json` references a `.csv`** — extend the input loader/factory to parse `.csv` URLs into the chosen content (Table or typed list). Most consistent with how `Document`/`Image` inputs work today.-- **(I2) CLI sugar** — e.g. `pipelex run my_pipe --csv data.csv`. Nice-to-have, later.-- **(I3) Programmatic factory** — `TableContent.make_from_csv_path(...)` / `ListContentFactory.from_csv(path, concept=...)` for library users and tests.--## 5. Dependency question--Core has no DataFrame dependency and is intentionally lean. Options:--- **Stdlib `csv` only** — covers comma/delimiter parsing, quoting, headers. No `.xlsx`. Keeps core lean. **Default recommendation.**-- **pandas/polars** — richer (dtypes, Excel, big files) but a heavy core dependency we'd rather not impose. If Excel is wanted, prefer an **optional extra** (`pipelex[tabular]`) over a hard dep.--This choice bounds the Excel/`.xlsx` ambition, so it's coupled to the scope decision below.--## 6. Chosen approach (locked)--Decisions taken (see §7 for the record):--- **Model — typed lists only (Mode A).** A CSV is a **serialization format for `ListContent[Concept]`**, never a blob and never a new content type. No native `Table` concept; the MTHDS language surface is unchanged. Rows are instances of a declared concept; columns are that concept's fields. This reuses `ListContent`, typed concepts, and `PipeBatch` directly.-- **Scope — full round-trip.** CSV in → process (typically batch over rows) → CSV out, both directions in v1.-- **Dependency — stdlib `csv` in core, `.xlsx` behind an optional extra.** v1 reads/writes CSV with the stdlib `csv` module (no DataFrame dep). The reader/writer sits behind a small **format seam** so an `openpyxl`-backed `.xlsx` codec can be added later under `pipelex[tabular]` without touching callers.-- **Nested cells — reject non-flat.** A row concept used for CSV must have **scalar fields only** (`str`/`int`/`float`/`bool`/optional thereof). A nested structure or list field raises a clear error telling the author to project to a flat concept first. No silent flattening, no JSON-in-cell, no dropped data.--**Design principle that falls out of this:** CSV is an **I/O codec**, not a prompt-render format. It does **not** join the `rendered_plain/markdown/html/json` `TextFormat` router (that family is about LLM/template/display rendering). Reading and writing CSV files is a separate codec surface that the CLI and the input loader call into.--## 7. Worked example — the round-trip--The headline "enrich a CSV" flow, end to end:--1. **Input** — `inputs.json` points a row concept at a `.csv`:-- ```json- { "contacts": { "concept": "acme.Contact", "content": { "url": "contacts.csv" } } }- ```-- The input loader sees a `.csv` URL, parses each data row into an `acme.Contact`, and yields `ListContent[Contact]`. (A CSV is inherently a list, so one `.csv` → one list; the named concept is the **row** concept.)--2. **Process** — the pipe declares `contacts = "Contact[]"` and uses `PipeBatch` to run a branch pipe `enrich_contact` over each `Contact`, recombining into `ListContent[EnrichedContact]` as the main stuff.--3. **Output** — the CLI writes the flat `ListContent[EnrichedContact]` back to `enriched.csv` (columns = `EnrichedContact`'s scalar fields).--No new operators, no new concept type — just the CSV codec bolted onto the existing list/batch machinery at the two edges.--## 8. Surfaces & semantics--**Input (I1 + I3).** `inputs.json` `.csv` references are parsed in the input loader; underneath sits a programmatic codec (`from_csv(path, row_concept) -> ListContent`) that tests and library users can call directly. Semantics:--- The `.csv` must have a **header row**; column headers must **match the row concept's field names** (strict — no implicit remapping).-- Cell strings are coerced to field types via pydantic's lax validation (e.g. `"42"` → `int`). An **empty cell → `None`** (and must therefore map to an optional field).-- **Extra columns** (no matching field) and **missing columns** for required fields are errors — predictable, no silent reshaping. (Consistent with the reject-non-flat stance.)--**Output (O2 + codec).** A flat `ListContent[StructuredContent]` is written to CSV; underneath the same codec (`to_csv(list_content, path)`). Trigger surface (finer point, see below): either auto-emit `main_stuff.csv` from `--save-main-stuff` when the main stuff is a flat list, or an explicit `--save-csv <path>`. Semantics:--- Columns are the row concept's scalar fields, in declared order; one CSV row per item.-- A non-flat row concept (any nested/list field) is **rejected with a clear error** at write time.--**Codec & format seam.** A single small tabular-codec module owns read/write, dispatched by file extension/format. CSV is the only built-in in v1; the seam's shape (a reader fn + writer fn keyed by format) is what lets `.xlsx` arrive later as an optional extra with zero changes to the loader or CLI.--## 9. Finer points (proposed defaults — speak up to change)--These don't change the shape of the feature; listed so the plan can lock them.--- **`inputs.json` convention:** the `concept` names the **row** concept; a `.csv` content URL implies list multiplicity (one CSV → `ListContent[row-concept]`). Pipe inputs are therefore declared with `[]`.-- **Output trigger:** prefer an explicit `--save-csv <path>` over auto-emitting from `--save-main-stuff` (explicit, no surprise files). Open to auto-emit if you'd rather.-- **Empty cell → `None`**; requires the field to be optional.-- **Strict columns:** exact header↔field correspondence; extra/missing columns error.-- **Dialect:** UTF-8, standard comma dialect; **delimiter and encoding configurable**, never auto-guessed.--## 10. Non-goals (v1)--- Streaming / out-of-core CSVs too large for memory.-- `.xlsx` in core — it's explicitly designed-for but ships later behind `pipelex[tabular]`; Google Sheets / Parquet are out.-- Delimiter/dialect **auto-detection** (configurable, not guessed).-- Schema/type **inference** beyond what the bound concept declares.-- A native `Table` concept / schema-free table primitive (deliberately not in this design; would be a separate, additive language-surface decision).--## 11. Risks / watch-outs--- **Lossy coercion** — string→typed coercion can change data (e.g. leading zeros, locale numbers). Document the rules; lean on pydantic's well-defined coercion rather than ad-hoc parsing.-- **Header collisions / unnamed columns** — define behavior for duplicate or blank headers up front (recommend: error).-- **Flat-only friction** — real concepts are often nested; the reject-non-flat rule means authors must add a "project to flat row" step. That's intended (predictable), but the error message must name the offending field and suggest the fix.-- **Format-seam over-engineering** — keep the seam minimal (two functions keyed by format); don't build a plugin framework before the second format exists.diff --git a/wip/history/csv-support/phase2-review-followups.md b/wip/history/csv-support/phase2-review-followups.mddeleted file mode 100644index db7b4fed6..000000000--- a/wip/history/csv-support/phase2-review-followups.md+++ /dev/null@@ -1,49 +0,0 @@-# CSV Support — Phase 2 code-review follow-ups (topics to revisit)--Status: **deferred — captured for a deliberate decision later, NOT a blocker for Phase 2.** Produced by an extra-high-recall `/code-review` of the Phase-2 codec implementation (`pipelex/tools/tabular/csv_codec.py`) on branch `feature/Support-csv` (worktree `_csv`).--This file holds review findings that are real but involve a **design tradeoff** rather than a rote fix — they shouldn't be patched reflexively. The mechanical/silent-correctness findings from the same review (blank-line phantom rows, over-wide-row truncation, delimiter/encoding boundary leak, write item-type guard, model-level coercion label) were the "fix-now" set and have since been resolved (see this folder's [`README.md`](README.md)).--Line numbers are approximate anchors — grep the quoted token if they have drifted.-------## #2 — Nullable-but-required column: mandatory vs. backfill-None (DEFERRED, PR #955 review round 6)--cubic flagged that a *nullable required* field (`x: str | None` with **no** default — pydantic reports `is_required() == True`) has a mandatory CSV column: omitting it raises `CsvColumnError` rather than backfilling `None`. cubic argues nullable columns should always be backfillable to `None`.--**Decision: keep current behavior (deferred, not adopted).** The rule "a column is mandatory ⟺ the field is pydantic-required" is intentional and *respects the author's signal*: writing `x: str | None` **without** `= None` is a deliberate "must be provided" choice (pydantic gives `X | None` no implicit default). Backfilling `None` would silently override that. The current code already maps an omitted *non-required* nullable column to `None` (round-4 `_annotation_allows_none`); only the *required* nullable case errors, which is correct. This is also unreachable from `.mthds`: a `required = false` field always compiles to `Optional[T] = None` (nullable WITH a None default → non-required → already `None` on omission), and a `required = true` field is non-nullable. So the distinction only arises for hand-built Python models. Revisit only if a real use-case wants "nullable ⟹ optional column" semantics.-------## #1 — Non-string `Literal` / `Enum` row fields — RESOLVED (Option A, PR #955 review round 3)--**Resolution:** went with Option A (reject). `_is_flat_annotation` now accepts a `Literal` only when every arg is a `str`, and an `Enum` only when every member value is a `str`; a non-string `Literal`/`IntEnum` falls through to `CsvFlatnessError`. This keeps "flat ⇒ round-trippable" honest. Pinned by `IntLiteralRow`/`IntEnumRow` rejection cases + a `StrEnumRow` acceptance case. greptile + cubic both flagged it across rounds. The original tradeoff analysis is kept below for context.-------### Original analysis (for context)--### Non-string `Literal` / `Enum` row fields: reject vs. coerce-on-read--**The finding.** The flatness gate `_is_flat_annotation` (`pipelex/tools/tabular/csv_codec.py:67`, `if origin is Literal: return True`; `:70`, `issubclass(annotation, Enum) → True`) accepts *any* `Literal` and *any* `Enum` as CSV-flat. But the codec can only round-trip **string-valued** ones:--- `_to_cell` writes the value as a bare string (`1`) or lowercased bool (`true`).-- On read, pydantic's `Literal`/`Enum` validator does **not** coerce that string back to the typed member.--Verified: `Literal[1,2,3]` writes `1` → read raises `CsvCoercionError`; `Literal[True,False]` writes `true` → fails; an `IntEnum` likewise. `Literal["low","high"]` (and `StrEnum` / str-valued `Enum`) round-trip fine. So a concept the gate declares "flat" can silently fail its own round-trip.--**Why it's deferred (the tradeoff).** Non-string `Literal`/`Enum` fields are **not reachable from `.mthds` today**, so this is not a live data-corruption risk:--- The structure generator *always* emits **string** `Literal`s — even integer/number "choices" compile to `Literal['1','2',…]`: `pipelex/core/concepts/structure_generation/generator.py:318-320` (`python_type = f"Literal[{', '.join(repr(c) for c in field_blueprint.choices)}]"`, fed by string choices).-- `choices` is validated as `list[str]`: `pipelex/core/concepts/concept_structure_blueprint.py:60` (`choices: list[str] | None`), and is only allowed on text/integer/number types: `pipelex/builder/concept/concept_spec.py:150-162`.-- The generator never produces `Enum`/`StrEnum` subclasses at all.--So non-string `Literal`/`Enum` only arise from hand-built `StructuredContent` subclasses (e.g. tests). Two ways to resolve, with different futures:--- **Option A — reject (simple, safe now).** Tighten the gate to accept only string-valued `Literal` (`all(isinstance(arg, str) for arg in get_args(...))`) and str-valued `Enum` (`all(isinstance(m.value, str) for m in enum)`); non-string members fall through to the existing `CsvFlatnessError`. Breaks no reachable concept, keeps "flat ⇒ round-trippable" honest. **Downside:** forecloses (or makes the codec inconsistent with) a future where integer/number choices materialize as real `Literal[int]` / `IntEnum` and someone wants them in CSV.-- **Option B — coerce-on-read (preserves future support).** For a `Literal`/`Enum` field, coerce the cell string against the member's underlying scalar type (or match by `str(member)`/`member.value`) before/within validation, so `1` → `Literal[1,2,3]` works. More code; partly duplicates pydantic; needs care for mixed-type literals.--**Recommended default if forced to pick now:** Option A (reject) — it's the smallest correct change and matches the current `.mthds` reality. Revisit toward Option B **only if** the structure generator ever emits typed (non-string) `Literal`/`Enum` for choices, at which point CSV support for them should be a deliberate feature with its own coercion + round-trip tests.--**Decision owner:** revisit when (a) touching the choices/structure-generation typing, or (b) a user asks for integer/enum-typed CSV columns. Until then this is a latent gap behind an unreachable code path, intentionally left as-is.diff --git a/wip/history/csv-support/phase3-4-review-followups.md b/wip/history/csv-support/phase3-4-review-followups.mddeleted file mode 100644index 50d5d76fa..000000000--- a/wip/history/csv-support/phase3-4-review-followups.md+++ /dev/null@@ -1,46 +0,0 @@-# CSV support — Phase 3+4 code-review follow-ups--Deferred items from the xhigh code review of the Phase 3 (input hook) + Phase 4 (`--save-csv`) work (2026-06-02). The two confirmed bugs (#1 query-string remote-rejection, #2 `ConceptValueError` wrap) and the cheap UX guards (#7 empty `--save-csv`, #8 parent-dir creation) were **fixed in that session** — see `stuff_factory._try_make_csv_list_stuff` and `_run_core._execute_run`. This file captures the items deliberately **not** patched: design-tradeoff / right-depth calls that deserve a decision rather than a reflexive fix.--## Deferred — design / altitude--### A. `resolve_uri` does not classify non-file remote schemes (`s3://`, `gs://`, …)--`resolve_uri` only special-cases `data:`, `pipelex-storage://`, `file://`, and `http(s)://`; **every other scheme falls through to `ResolvedLocalPath(path=<uri>)`** (`pipelex/tools/uri/uri_resolver.py` final `return`). So `s3://bucket/x.csv` is reported as a *local path* whose `.path` still contains `s3://...`.--The CSV input hook works around this with a string heuristic — `not isinstance(resolved, ResolvedLocalPath) or "://" in resolved.path` — to reject those as remote. That works, but:--- It encodes a fragile assumption about `resolve_uri`'s fallthrough; the `"://"` clause silently becomes dead/contradictory the day a real `ResolvedS3Url` variant is added.-- **The same blind spot lives in other callers** that trust `resolve_uri`'s `ResolvedLocalPath` to mean "local file" — e.g. the input normalizer and the image/document renderers (`load_binary_async(Path("s3://…"))`), and `_inputs_path_resolver.is_relative_local_path` (which would treat `s3://…` as a relative local path and prepend `base_dir`).--**Right-depth fix:** teach `resolve_uri` / `UriKind` to classify an unknown `<scheme>://` as a distinct non-local kind, and expose a single `is_local_uri(resolved) -> bool` predicate. Then the CSV hook and `_inputs_path_resolver` share one authoritative classifier (matching the project's "match/case over a typed enum, no string-sniffing" rule) and the `"://"` heuristic disappears. This is a `tools/uri` change with cross-caller impact → scope and test it on its own, not inside the CSV feature.--**Update (PR #955 review round 9):** the blind spot became a concrete bug — `_inputs_path_resolver.is_relative_local_path` was rewriting `s3://bucket/x.csv` into `<base_dir>/s3:/bucket/x.csv` on the CLI JSON-file path, defeating the CSV remote-url guard (codex/cubic). Fixed by adding the same `"://"`-in-resolved-path stopgap to `is_relative_local_path` (so scheme-bearing refs are left untouched and the remote guard fires). The `"://"` heuristic now lives in **two** callers (the CSV hook + `is_relative_local_path`); item A's typed `is_local_uri` predicate should replace both at once.--**Same root cause — local paths containing URL-reserved `?`/`#` (cubic, PR #955 round 1).** Suffix detection runs on `urlsplit(url).path` so a remote presigned `…csv?X-Amz-…` can't hide its suffix and slip past the local-only guard. The cost: a *local* file whose name literally contains `?` or `#` (POSIX-legal, Windows-illegal) has its suffix mis-stripped, so `data#1.csv` is not detected as tabular and falls through to record handling — a clear validation error, no data loss. The principled fix is the same `resolve_uri` scheme-classification above: resolve first, then take the suffix from the *resolved local path* (`?`/`#` are ordinary filename bytes there) for local refs and from `urlsplit` only for true URLs. Deferred with item A rather than bolted on as a second string heuristic. Niche (pathological local filename) and graceful, so not a v1 blocker.--### B. CSV detection + eager file I/O live in `StuffFactory`, the universal stuff constructor--A1 deliberately put the hook in `StuffFactory.make_stuff_from_stuff_content_or_data` (Case 2.5) so CSV input is reachable from the runner API and programmatic callers, not just the CLI. The consequence to weigh:--- `StuffFactory` is reached by **every** path that builds a stuff — CLI runs, runner-API input building, `WorkingMemory` re-hydration from a serialized payload, mock-input construction. All of them now perform eager local-disk I/O whenever a content dict carries a tabular-suffixed `url`.-- The CLI rewrites relative `url`s to absolute via `_inputs_path_resolver` **before** the factory sees them; non-CLI callers do **not**. A runner/programmatic caller passing `{"url": "data.csv"}` therefore opens a path resolved against the **process cwd**, and a re-hydrated `WorkingMemory` re-opens a path that may not exist on that host. (`test_eager_read_reflects_file_mutation` pins the disk-read-on-every-build behavior as intended.)--**Decision needed:** is eager file I/O at the factory altitude acceptable for all non-CLI callers, or should CSV reference resolution move up into a dedicated input-resolution layer (the design doc §2/§8 originally placed CSV parsing "in the input loader")? If the factory stays, document the relative-path-vs-cwd contract for non-CLI callers and decide whether re-hydration should re-read or carry the parsed list. (Note: Temporal is not shipped — don't over-weight the worker re-hydration angle, but the runner-API relative-path case is real today.)--## Deferred — documentation (fold into the Phase 5 docs page)--### C. `url`-field residual extends to multi-field records (A3/CT1) — RESOLVED (option b, PR #955 review round 1)--The documented A3/CT1 residual was "a single-record concept with a `url`-named field whose value ends `.csv` is read as a table". The hook originally keyed off `content.get("url")` **only**, so a content dict with *other* keys too — `{"label": "Home", "url": "report.csv"}` — was also hijacked into a table read, **silently dropping the sibling keys**.--**Fixed** by tightening detection to the single-key wrapper: `_try_make_csv_list_stuff` now returns `None` unless `set(content) == {"url"}`. A record with any sibling key stays an ordinary record (pinned by `test_record_with_csv_url_field_and_siblings_stays_record`). The residual is now strictly "a one-field `url` concept given `{"url": <tabular>}`", matching the documented limitation exactly. Codex + cubic both flagged the silent-drop; this removes it.--## Deferred — minor / cleanup (low priority, no decision needed)--- **Output-side error UX:** a non-flat / column-mismatched output raises `CsvFlatnessError`/`CsvColumnError` from `csv_from_list_content` **after** the pipeline succeeded; it surfaces via `execute_run`'s generic `except PipelexError` as "Failed to execute pipeline" rather than a tailored "Failed to --save-csv: …". Consider catching `CsvError` around the write block and re-emitting with the save-csv framing.-- **Suffix vocabulary drift:** `_TABULAR_SUFFIXES` (drives `is_tabular_path`) and `assert_supported_table_suffix`'s inline `.csv`/`.xlsx` literals are two independent authorities; express the latter in terms of the former before the `.xlsx`/`.tsv` codec ships.-- **Header-only CSV** yields an empty `ListContent` accepted by the hook, whereas the literal-empty-list path (Case 1.5) rejects empty lists — decide whether an empty CSV is a valid empty batch or should fail fast for parity.-- **Refined native concepts:** a concept that *refines* `Image`/`Document` has a non-native `concept_ref`, so `Concept.is_native_concept` is False and a `.csv`-suffixed `url` triggers the hook → `CsvFlatnessError` on the native content class. Niche; only matters if someone gives such a concept a `.csv` url.-- **Native-row list output:** `--save-csv` on a `Text[]` output writes a degenerate single-column `text` CSV instead of signalling the "flat structured list" contract. Acceptable or worth a clearer message — decide.-- **Small cleanups:** redundant `cast("dict[str, Any]", content)` in Case 2.5 and `cast("ListContent[StuffContent]", …)` in the save block; `get_optional_main_stuff()` vs the `pipe_output.optional_main_stuff` property (two spellings of one lookup); the two `if save_csv is not None:` blocks could be consolidated.diff --git a/wip/history/csv-support/security-followups.md b/wip/history/csv-support/security-followups.mddeleted file mode 100644index 9a4461742..000000000--- a/wip/history/csv-support/security-followups.md+++ /dev/null@@ -1,25 +0,0 @@-# CSV support — security review follow-ups (gstack /review, 2026-06-02)--The pre-landing `/review` (5 Claude specialists + Codex adversarial) confirmed the codec is well-hardened after 9 bot rounds. Three security-flavored items are captured here as deliberate decisions / deferred follow-ups rather than blockers for PR #955. The fixes the review *did* apply (BOM read, coercion-message redaction, `--save-csv` error framing) are in the PR.--## 1. Local-file read via a `url` input is the existing input model, not a CSV-specific hole — DEFERRED (platform-wide)--**Finding (security + Codex):** the CSV input hook opens a local file from an attacker-controllable `url` with no path-traversal/absolute-path confinement. `{"url": "../../some.csv"}` or `{"url": "/abs/some.csv"}` is read. Reachable from `WorkingMemoryFactory.make_from_pipeline_inputs`, the path the hosted runner API uses to build working memory from request inputs.--**Why it's not a CSV-specific blocker:** reading a local file from a `url` is the **pre-existing input model** — native `Image`/`Document` concepts already auto-populate from a local-file `url` path and load it (`image_content.py`/`document_content.py` "Auto-populate filename from url when it is a local file path", and the renderers/extractors call `load_binary` on it). The CSV hook is consistent with that model; it only extends it to flat structured concepts for `.csv`/`.xlsx`. It explicitly defers native concepts to their own handling. So "an untrusted `url` can name a local file" is a platform-wide property that pre-exists this PR, not something CSV introduces.--**The real concern (platform-wide):** *if* the hosted runner API accepts untrusted `url` inputs that reach `make_from_pipeline_inputs` without confinement, then any `url`-bearing input type (Image/Document today, CSV now) is a local-file-read primitive. CSV narrows it to `*.csv`/`*.xlsx` files specifically (the suffix is checked on the real path), but that still includes other tenants' CSVs / config CSVs on the host.--**Recommendation (own change, not this PR):** decide the runner's input-trust model and confine file-path inputs for untrusted callers at the input-resolution layer — reject absolute paths and paths escaping an allowlisted base dir — applied uniformly across **all** `url` inputs (Image/Document/CSV), not just CSV. Doing it CSV-only would be inconsistent and would also break the legitimate CLI use case (the CLI deliberately reads local files by relative/absolute path; `_inputs_path_resolver` resolves relative `url`s against the inputs-file dir). This belongs with the platform/runner input-handling, paired with the `tools/uri` scheme-classification follow-up (item A in `phase3-4-review-followups.md`).--## 2. CSV/Excel formula injection — DEFERRED by design (CT3, documented limitation)--**Finding (Codex, "block merge"):** `csv_from_list_content` writes cell values verbatim, so a value starting with `=`/`+`/`-`/`@` executes when opened in Excel/Sheets.--**Decision: this is the locked CT3 product decision, not a new bug.** plan-eng-review chose document-only on injection: do NOT auto-escape formula-leading cells (escaping mutates data and hurts fidelity), document it as a known v1 limitation, defer an opt-in `escape_formulas` flag. It is already written up in the user docs ("No formula-injection escaping") and the design. Codex flags it because it can't see the product decision. v1's primary surface is the local CLI (the user writes and opens their own CSV). The opt-in escape flag is the planned follow-up for when generated CSVs ship to third parties via the hosted runner. Not a blocker; no change in this PR.--## 3. Embedded-NUL path → raw `ValueError` escapes the codec boundary — DEFERRED (niche)--**Finding (security, sub-threshold):** a `url` whose resolved path contains an embedded NUL byte (`x\x00.csv`) makes `path.open()` raise `ValueError("embedded null byte")`, which `_read_table` doesn't catch (it wraps `OSError`/`UnicodeDecodeError`/`LookupError`/`csv.Error`). No file is opened, no traversal, no sensitive data in the message — it's an error-hygiene gap, not a security issue.--**Decision: defer.** Very niche (embedded NUL in a CSV path), and a broad `except ValueError` in `_read_table` risks masking real bugs. If we tighten it later, catch it narrowly around `path.open()` only. Low priority.diff --git a/wip/history/keyword-only-arguments/README.md b/wip/history/keyword-only-arguments/README.mddeleted file mode 100644index 5ad8f5b94..000000000--- a/wip/history/keyword-only-arguments/README.md+++ /dev/null@@ -1,22 +0,0 @@-# keyword-only-arguments--Track record for the **keyword-only arguments refactor** on `refactor/Function-calling-4`: every non-subject parameter across `pipelex/` source is now keyword-only, mechanically enforced by the `check-keyword-only` AST guard (in `make agent-check`, `make check`, and CI).--## Status--The pipelex-internal refactor is **complete and landed on the branch** — all waves done, the guard hard-blocks on any violation, and the convention is promoted to the canonical contributor doc at [`docs/contribute/keyword-only-arguments.md`](../../../docs/contribute/keyword-only-arguments.md) (in the mkdocs nav). The downstream consumers were updated in lockstep (PRs #19 / #55 / #75 / #153, plus pipelex-worker #23) and **pushed**, all pinned to the branch git rev `0e32c8c0` and green on their test suites. (`pipelex-mistralai-workflows` is the one genuine out-of-scope consumer — it's on its own mistralai-2x hold track.)--**One open item:** at pipelex release, swap each consumer's temporary `[tool.uv.sources]` git pin back to the released `pipelex` version. Tracked in the checklist of [`downstream-consumer-breakage.md`](downstream-consumer-breakage.md).--**Follow-on (separate branch):** an `--fix` auto-fix capability was added to the guard on `feature/Tweaks-for-validation-api`. Its code review surfaced open items (incl. a latent silent-corruption bug) tracked in [`autofix-review-followups.md`](autofix-review-followups.md).--## What's here--- **[autofix-review-followups.md](autofix-review-followups.md)** — review follow-ups for the `--fix` auto-fix addition (the `make fko` / `check-keyword-only --fix` capability). Has open bugs (F1/F2 line-numbering corruption) and tradeoff decisions (F3–F5).-- **[keyword-only-arguments.html](keyword-only-arguments.html)** — self-contained co-developer explainer: TL;DR, what/why, the rule, how it was enforced, the audit→consolidation→downstream-fix arc, and minimal diffs for the critical changes. Open in a browser.-- **[downstream-consumer-breakage.md](downstream-consumer-breakage.md)** — the living cross-repo lockstep record. Which consumer call sites broke at each pipelex SHA, how each was fixed, and the at-release pin-swap checklist. This is the one doc with an open action.-- **[positional-subject-suspects.md](positional-subject-suspects.md)** — RESOLVED. The consolidated shortlist of 59 "positional-subject abuse" suspects (functions that kept their subject positional only to satisfy the carve-out) and the adjudication that turned them all fully keyword-only.-- **audit/** — raw process artifacts from the positional-subject audit (reference, not active):- - [`positional-subject-audit.md`](audit/positional-subject-audit.md) — the full machine+LLM audit of every audited function (large).- - [`review-prompt.md`](audit/review-prompt.md) — the sub-agent review prompt that drove the audit.- - [`findings/`](audit/findings/) — per-package suspect shortlists, one file per `pipelex/` package.diff --git a/wip/history/keyword-only-arguments/audit/findings/base_exceptions.md b/wip/history/keyword-only-arguments/audit/findings/base_exceptions.mddeleted file mode 100644index 43babf5a6..000000000--- a/wip/history/keyword-only-arguments/audit/findings/base_exceptions.md+++ /dev/null@@ -1,5 +0,0 @@-# Suspects — package `base_exceptions`--Reviewed: 0 Section A + 1 primitive lone-subjects. Suspects: 0.--No suspects.diff --git a/wip/history/keyword-only-arguments/audit/findings/builder.md b/wip/history/keyword-only-arguments/audit/findings/builder.mddeleted file mode 100644index 36cbdc98e..000000000--- a/wip/history/keyword-only-arguments/audit/findings/builder.md+++ /dev/null@@ -1,15 +0,0 @@-# Suspects — package `builder`--Reviewed: 20 Section A + 2 primitive lone-subjects. Suspects: 4.--## High confidence--- `pipelex/builder/operations/models_ops.py:36` — `_resolve_preset_backend` — `def _resolve_preset_backend(model_deck: ModelDeck, *, model_handle: str, model_type: ModelType) -> InferenceModelSpec | None` — The docstring says "Resolve a preset's model handle to an InferenceModelSpec": the thing being resolved is `model_handle`; `model_deck` is a lookup registry used as lookup context. Call sites confirm: `_resolve_preset_backend(model_deck, model_handle=setting.model, model_type=model_type)` — the deck reads as an unlabeled blob, while the real semantic target (`model_handle`) is buried in keyword args. Suggested fix: make fully keyword-only (`def _resolve_preset_backend(*, model_deck, model_handle, model_type)`).--- `pipelex/builder/operations/output_ops.py:14` — `build_output_for_pipe` — `async def build_output_for_pipe(mthds_contents: list[str], *, pipe_code: str, output_format: ...) -> dict[str, Any]` — The function name says "for_pipe", signalling that the pipe is the semantic subject. `mthds_contents` is raw bundle source material (context/input) not the object being acted on. A caller reading `build_output_for_pipe(some_list, pipe_code="my.pipe")` must think twice about what that first list is. Pair with `build_runner_code_for_pipe` which has the same shape and semantics. Suggested fix: make fully keyword-only (`def build_output_for_pipe(*, mthds_contents, pipe_code, output_format)`), or reorder so `pipe_code` is the positional subject.--- `pipelex/builder/operations/runner_code_ops.py:17` — `build_runner_code_for_pipe` — `async def build_runner_code_for_pipe(mthds_contents: list[str], *, pipe_code: str) -> str` — Same pattern as `build_output_for_pipe` above: the name says "for_pipe", the pipe is the semantic object, but `mthds_contents` (the source bundle data) was made positional. Suggested fix: make fully keyword-only, or promote `pipe_code` to the positional subject.--## Medium / low confidence--- `pipelex/builder/operations/models_ops.py:101` — `_build_presets_for_category` (and the parallel `_build_aliases_for_category` line 135, `_build_waterfalls_for_category` line 160) — `def _build_presets_for_category(model_deck: ModelDeck, *, category: ModelCategory, backend: str | None)` — The function name says "for_category"; the function matches on `category` to select which field of the deck to read. `model_deck` is the data source/registry, `category` is what drives the behavior. At call sites (`_build_presets_for_category(model_deck, category=ModelCategory.LLM, backend=backend)`) the deck reads as an unnamed blob. Lower confidence than the `_resolve_preset_backend` case because the deck is a richer object (not just a registry lookup), so it has a plausible claim to being the "subject" that is queried. Suggested fix: make fully keyword-only.diff --git a/wip/history/keyword-only-arguments/audit/findings/cli.md b/wip/history/keyword-only-arguments/audit/findings/cli.mddeleted file mode 100644index 971ad58f7..000000000--- a/wip/history/keyword-only-arguments/audit/findings/cli.md+++ /dev/null@@ -1,15 +0,0 @@-# Suspects — package `cli`--Reviewed: 112 Section A + 30 primitive lone-subjects. Suspects: 4.--## High confidence--- `pipelex/cli/cli_factory.py:31` — `make_pipelex_for_cli` — `def make_pipelex_for_cli(context: ErrorContext, *, library_dirs: ..., needs_inference: bool, ...) -> Pipelex` — `context` is error-reporting metadata (an `ErrorContext` enum used to shape error messages), not the object being created. The function builds a `Pipelex` instance; `context` is a side-channel tag. Every call site uses `context=...` as a keyword (confirmed across all call sites). Suggested fix: make fully keyword-only (`def make_pipelex_for_cli(*, context: ErrorContext, library_dirs: ..., ...)`)--- `pipelex/cli/agent_cli/commands/run/_output_helpers.py:14` — `build_run_output` — `def build_run_output(with_memory: bool, *, main_stuff_json: dict, working_memory_dump: dict, compact_result: ..., extra_metadata: ...) -> dict` — `with_memory: bool` is a mode flag, not the semantic object being assembled. Both call sites pass it as `with_memory=with_memory` (keyword). `main_stuff_json` is arguably the more central data. Suggested fix: move `*` before `with_memory` (make it keyword-only too).--- `pipelex/cli/commands/init/command.py:182` — `determine_needs` — `def determine_needs(reset: bool, *, check_config: bool, check_inference: bool, ...) -> tuple[bool, bool, bool, bool]` — `reset: bool` is a mode flag among several other boolean flags; it's not semantically more "the subject" than `check_config`, `check_inference`, etc. The single call site passes it as `reset=reset` (keyword). Suggested fix: move `*` before `reset` (make it keyword-only too, consistent with the other boolean flags).--## Medium / low confidence--- `pipelex/cli/agent_cli/commands/init_cmd.py:173` — `_configure_backends` — `def _configure_backends(config: dict[str, Any], *, backends_toml_path: Path, template_backends_path: Path) -> list[str]` — the function's primary action is modifying `backends_toml_path`; `config: dict` is read-only input/context from which backend selection is extracted. The call site passes it as `_configure_backends(parsed_config, backends_toml_path=..., ...)` (positional use). This is a private helper so the impact is contained, and `config` as "data being acted upon" is defensible. Low confidence — calling convention is actually used positionally here, so no immediate caller-side benefit.diff --git a/wip/history/keyword-only-arguments/audit/findings/cogt.md b/wip/history/keyword-only-arguments/audit/findings/cogt.mddeleted file mode 100644index bdc0e10ef..000000000--- a/wip/history/keyword-only-arguments/audit/findings/cogt.md+++ /dev/null@@ -1,13 +0,0 @@-# Suspects — package `cogt`--Reviewed: 119 Section A + 42 primitive lone-subjects. Suspects: 3.--## High confidence--- `pipelex/cogt/usage/costs_per_token.py:4` — `model_cost_per_token` — `def model_cost_per_token(costs: CostsByCategoryDict, *, cost_category: CostCategory) -> float` — `costs` is a dict used purely as a lookup table; the function dispatches on `cost_category` and returns `costs.get(cost_category) / 1_000_000`. The real "subject" the function acts on is `cost_category`, while `costs` is context/registry. All three real call sites already use `costs=...` as a keyword. Suggested fix: move `*` before `costs` too — `def model_cost_per_token(*, costs: CostsByCategoryDict, cost_category: CostCategory) -> float` — or reorder with `cost_category` first.--- `pipelex/cogt/llm/reasoning_config_base.py:40` — `get_reasoning_level_str` — `def get_reasoning_level_str(effort_to_level_map: EffortToLevelMap, *, effort: ReasoningEffort) -> str | None` — `effort_to_level_map` is a `dict[str, str]` passed as contextual configuration; the function body is simply `effort_to_level_map.get(effort)`. The semantic object is `effort` (the query), while the map is a lookup table. All four call sites pass the map positionally via `self.effort_to_level_map` without a label, which reads opaquely. Suggested fix: fully keyword-only `def get_reasoning_level_str(*, effort_to_level_map: EffortToLevelMap, effort: ReasoningEffort)`.--## Medium / low confidence--- `pipelex/cogt/model_backends/backend_credentials.py:26` — `BackendCredentialsErrorMsgFactory.make_one_variable_missing_error_msg` — `def make_one_variable_missing_error_msg(cls, secrets_provider: SecretsProviderAbstract, *, backend_name: str | None, var_name: str) -> str` — `secrets_provider` acts as a dispatch context (the function `isinstance`-branches on its type to select message wording), while `backend_name` and `var_name` are the real domain objects the message describes. The sole call site already uses `secrets_provider=secrets_provider` as a keyword. This is lower-confidence because `secrets_provider` does meaningfully control branching. Suggested fix: fully keyword-only, or at least keep asymmetry noted; low urgency.diff --git a/wip/history/keyword-only-arguments/audit/findings/core.md b/wip/history/keyword-only-arguments/audit/findings/core.mddeleted file mode 100644index b41b40bbb..000000000--- a/wip/history/keyword-only-arguments/audit/findings/core.md+++ /dev/null@@ -1,13 +0,0 @@-# Suspects — package `core`--Reviewed: 95 Section A + 66 primitive lone-subjects. Suspects: 3.--## High confidence--- `pipelex/core/stuffs/stuff_content.py:64` — `StuffContent.rendered_markdown` — `def rendered_markdown(self, level: int=1, *, is_pretty: bool=False) -> str` — `level` and `is_pretty` are both optional rendering options; neither is "the subject" (self is). All call sites already use `rendered_markdown(level=..., is_pretty=...)` keyword form or the no-arg default. `rendered_markdown(2)` is opaque. Suggested fix: move `*` before `level` so both params are keyword-only.--- `pipelex/core/stuffs/stuff_content.py:113` — `StuffContent.rendered_markdown_async` — `async def rendered_markdown_async(self, level: int=1, *, is_pretty: bool=False) -> str` — same issue as `rendered_markdown`; parallel async signature, all call sites use keyword form. Suggested fix: move `*` before `level`.--## Medium / low confidence--- `pipelex/core/concepts/concept.py:194` — `Concept.render_concept_representation` — `def render_concept_representation(self, output_format: ConceptRepresentationFormat, *, is_multiple: bool=False) -> tuple[dict[str, Any], set[str]]` — `self` (Concept) is the real subject; `output_format` and `is_multiple` are both rendering options of equal standing. All call sites use `render_concept_representation(output_format=..., is_multiple=...)` keyword form. Positional `output_format` adds no clarity. Suggested fix: move `*` before `output_format` (make fully keyword-only).diff --git a/wip/history/keyword-only-arguments/audit/findings/errors.md b/wip/history/keyword-only-arguments/audit/findings/errors.mddeleted file mode 100644index 57dd5150c..000000000--- a/wip/history/keyword-only-arguments/audit/findings/errors.md+++ /dev/null@@ -1,5 +0,0 @@-# Suspects — package `errors`--Reviewed: 5 Section A + 2 primitive lone-subjects. Suspects: 0.--No suspects.diff --git a/wip/history/keyword-only-arguments/audit/findings/graph.md b/wip/history/keyword-only-arguments/audit/findings/graph.mddeleted file mode 100644index 4258018cd..000000000--- a/wip/history/keyword-only-arguments/audit/findings/graph.md+++ /dev/null@@ -1,11 +0,0 @@-# Suspects — package `graph`--Reviewed: 40 Section A + 12 primitive lone-subjects. Suspects: 1.--## High confidence--_(none)_--## Medium / low confidence--- `pipelex/graph/graph_tracer_manager.py:266` — `GraphTracerManager.on_pipe_end_success` — `def on_pipe_end_success(self, lookup_key: str, *, node_id: str | None, ...)` — `lookup_key` is a registry dispatch key used to route to the correct `GraphTracer` instance via `self._get_tracer(lookup_key)`; the semantic subject is `node_id` (the node whose execution ended), which sits in the keyword args. Same pattern applies to `on_pipe_end_error` (line 325), `register_execution_data` (line 304), `add_edge` (line 360), `register_controller_output` (line 389), `register_batch_item_extraction` (line 411), `register_batch_aggregation` (line 440), and `register_parallel_combine` (line 469). All call sites already pass `lookup_key=` as a keyword argument (e.g. `pipe_abstract.py:116`), so no opacity exists today. The case for making `lookup_key` keyword-only too is that it is a dispatch/lookup concern rather than the semantic object of the operation, and the naming asymmetry between these manager methods and the underlying `GraphTracerProtocol` methods (which take `node_id` positionally) is a mild source of confusion. Low confidence overall — the call-site pattern is already keyword-explicit, and `lookup_key` as first-positional is a coherent "which tracer" discriminant. Suggested fix if desired: make `lookup_key` keyword-only across all eight manager dispatch methods (fully keyword-only signatures), mirroring the symmetry with the protocol layer.diff --git a/wip/history/keyword-only-arguments/audit/findings/hub.md b/wip/history/keyword-only-arguments/audit/findings/hub.mddeleted file mode 100644index cc713582f..000000000--- a/wip/history/keyword-only-arguments/audit/findings/hub.md+++ /dev/null@@ -1,5 +0,0 @@-# Suspects — package `hub`--Reviewed: 1 Section A + 14 primitive lone-subjects. Suspects: 0.--No suspects.diff --git a/wip/history/keyword-only-arguments/audit/findings/kit.md b/wip/history/keyword-only-arguments/audit/findings/kit.mddeleted file mode 100644index 365798c02..000000000--- a/wip/history/keyword-only-arguments/audit/findings/kit.md+++ /dev/null@@ -1,11 +0,0 @@-# Suspects — package `kit`--Reviewed: 8 Section A + 0 primitive lone-subjects. Suspects: 2.--## High confidence--- `pipelex/kit/single_file_agent_rules.py:12` — `_read_agent_file` — `def _read_agent_file(agents_dir: Traversable, *, name: str) -> str` — `agents_dir` is a lookup-table/context directory; `name` is the file being read — the real acted-on object. The pattern "read [name] from [agents_dir]" has the semantic subject keyworded and the context positional. Call site (line 79): `_read_agent_file(agents_dir, name=name)` confirms `name` carries the meaning. Suggested fix: make fully keyword-only — `def _read_agent_file(*, agents_dir: Traversable, name: str) -> str`--## Medium / low confidence--- `pipelex/kit/single_file_agent_rules.py:49` — `build_merged_rules` — `def build_merged_rules(kit_index: KitIndex, *, agent_set: str | None=None, file_list: list[str] | None=None) -> str` — `kit_index` is a configuration/registry object used as a data source and lookup table, not the semantic object being acted on. The function "builds merged rules" by consulting the index; `kit_index` reads more like a context/config argument than a subject. Lower confidence because a KitIndex could reasonably be seen as "the thing being merged from". Suggested fix: make fully keyword-only — `def build_merged_rules(*, kit_index: KitIndex, agent_set: str | None=None, file_list: list[str] | None=None) -> str`diff --git a/wip/history/keyword-only-arguments/audit/findings/language.md b/wip/history/keyword-only-arguments/audit/findings/language.mddeleted file mode 100644index 59f3ff0b4..000000000--- a/wip/history/keyword-only-arguments/audit/findings/language.md+++ /dev/null@@ -1,5 +0,0 @@-# Suspects — package `language`--Reviewed: 5 Section A + 3 primitive lone-subjects. Suspects: 0.--No suspects.diff --git a/wip/history/keyword-only-arguments/audit/findings/libraries.md b/wip/history/keyword-only-arguments/audit/findings/libraries.mddeleted file mode 100644index a8afc778b..000000000--- a/wip/history/keyword-only-arguments/audit/findings/libraries.md+++ /dev/null@@ -1,15 +0,0 @@-# Suspects — package `libraries`--Reviewed: 28 Section A + 19 primitive lone-subjects. Suspects: 4.--## High confidence--- `pipelex/libraries/libraries/library_crate.py:39` — `LibraryCrate.compute_fingerprint_from_content` — `def compute_fingerprint_from_content(concepts: dict[str, 'ConceptBlueprint | str'], *, pipes: dict[str, PipeBlueprintUnion]) -> str` — `concepts` and `pipes` are co-equal operands (both dict inputs to a hash); neither is the semantic object of the function. Call sites already keyword both (`compute_fingerprint_from_content(concepts=concepts, pipes=pipes)`). A directional-pair with no natural subject. — suggested fix: make fully keyword-only `def compute_fingerprint_from_content(*, concepts, pipes)`--## Medium / low confidence--- `pipelex/libraries/concept/concept_library.py:191` — `ConceptLibrary.add_dependency_concept` — `def add_dependency_concept(self, alias: str, *, concept: Concept) -> None` — `alias` is a namespace/prefix used to build a composite key; `concept` is what is actually being added to the library. Call sites already use `alias=alias` (keyword). The real "subject" of "add" is the concept, not the alias. — suggested fix: `def add_dependency_concept(self, *, alias: str, concept: Concept)` or reorder to `def add_dependency_concept(self, concept: Concept, *, alias: str)`--- `pipelex/libraries/pipe/pipe_library.py:97` — `PipeLibrary.add_dependency_pipe` — `def add_dependency_pipe(self, alias: str, *, pipe: PipeAbstract) -> None` — identical pattern to `add_dependency_concept` above: `alias` is the namespacing key, `pipe` is the actual subject being added. Call sites already use `alias=alias`. — suggested fix: same as above, make `alias` keyword-only too--- `pipelex/libraries/visibility_utils.py:35` — `make_visibility_checker` — `def make_visibility_checker(manifest: MethodsManifest | None, *, blueprints: list[PipelexBundleBlueprint]) -> PackageVisibilityChecker` — `manifest` is nullable config/filter (often `None`), while `blueprints` is the primary data being inspected. Call sites always pass `manifest=manifest` already. Same pattern for `check_visibility_for_blueprints` at line 53. Low confidence: `manifest` could be argued as the filter subject. — suggested fix: make both args keyword-only for both functionsdiff --git a/wip/history/keyword-only-arguments/audit/findings/observer.md b/wip/history/keyword-only-arguments/audit/findings/observer.mddeleted file mode 100644index dc928af27..000000000--- a/wip/history/keyword-only-arguments/audit/findings/observer.md+++ /dev/null@@ -1,9 +0,0 @@-# Suspects — package `observer`--Reviewed: 1 Section A + 1 primitive lone-subject. Suspects: 2.--## High confidence--- `pipelex/observer/multi_observer.py:14` — `MultiObserver.remove_observer` — `def remove_observer(self, name: str) -> None` — Asymmetric with `add_observer(*, name, observer)` which is already fully keyword-only; `remove_observer("some_name")` passes a bare string with no keyword context at the call site. Convention consistency and readability both point toward `def remove_observer(self, *, name: str) -> None`.--- `pipelex/observer/local_observer.py:22` — `LocalObserver._write_to_jsonl` — `def _write_to_jsonl(self, event_type: str, *, payload: PayloadType) -> None` — `event_type` is a routing/selector string (determines which file to write to and which key to inject into the payload), not the semantic object being written. The payload is the real subject. Call sites pass a named constant (`LocalObserverEventType.BEFORE_RUN`) so readability is acceptable, but the two-parameter shape (`event_type` positional, `payload` keyword) is inconsistent: both parameters are equally descriptive options that label different aspects of the write operation, and `_write_to_jsonl(*, event_type, payload)` would self-document the call site more clearly.diff --git a/wip/history/keyword-only-arguments/audit/findings/pipe_controllers.md b/wip/history/keyword-only-arguments/audit/findings/pipe_controllers.mddeleted file mode 100644index 8f12e7d84..000000000--- a/wip/history/keyword-only-arguments/audit/findings/pipe_controllers.md+++ /dev/null@@ -1,17 +0,0 @@-# Suspects — package `pipe_controllers`--Reviewed: 6 Section A + 3 primitive lone-subjects. Suspects: 3.--## High confidence--- `pipelex/pipe_controllers/parallel/pipe_parallel.py:326` — `PipeParallel._register_branch_outputs_with_graph_tracer` — `def _register_branch_outputs_with_graph_tracer(self, job_metadata: JobMetadata, *, output_stuffs: dict[str, 'Stuff']) -> None` — `job_metadata` is a trace-context carrier / lookup key, not the semantic object of the function; the function acts on `output_stuffs`. All call sites (lines 210 and 309) already pass `job_metadata=job_metadata` by keyword. Suggested fix: make fully keyword-only — `def _register_branch_outputs_with_graph_tracer(self, *, job_metadata: JobMetadata, output_stuffs: dict[str, 'Stuff']) -> None`.--- `pipelex/pipe_controllers/parallel/pipe_parallel.py:369` — `PipeParallel._register_parallel_combine_with_graph_tracer` — `def _register_parallel_combine_with_graph_tracer(self, job_metadata: JobMetadata, *, combined_stuff: 'Stuff', branch_stuffs: dict[str, 'Stuff']) -> None` — same pattern: `job_metadata` is a context/trace carrier; the function acts on `combined_stuff` and `branch_stuffs`. Both call sites (lines 203 and 302) use `job_metadata=job_metadata`. Suggested fix: move `*` before `job_metadata` — `def _register_parallel_combine_with_graph_tracer(self, *, job_metadata: JobMetadata, combined_stuff: 'Stuff', branch_stuffs: dict[str, 'Stuff']) -> None`.--- `pipelex/pipe_controllers/sub_pipe.py:34` — `SubPipe.run_pipe` — `async def run_pipe(self, calling_pipe_code: str, *, working_memory: WorkingMemory, job_metadata: JobMetadata, sub_pipe_run_params: PipeRunParams, library_crate: 'LibraryCrate | None'=None) -> PipeOutput` — `calling_pipe_code` is caller provenance/context, not the semantic object; `self` (the SubPipe) is the object being run. Both real call sites (pipe_parallel.py:152, pipe_sequence.py:195, pipe_condition.py:328) already pass `calling_pipe_code=self.code` by keyword. Suggested fix: make fully keyword-only — `async def run_pipe(self, *, calling_pipe_code: str, working_memory: WorkingMemory, job_metadata: JobMetadata, sub_pipe_run_params: PipeRunParams, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`.--## Medium / low confidence--- `pipelex/pipe_controllers/pipe_controller.py:71` — `PipeController._live_run_controller_pipe` — `async def _live_run_controller_pipe(self, job_metadata: JobMetadata, *, ...)` — `job_metadata` as positional follows the same `_live_run_pipe(self, job_metadata, *, ...)` convention established on `PipeAbstract` (the base class). The single call site (pipe_controller.py:43) already uses keyword form. Medium confidence only because the `job_metadata`-as-first-non-self-param is a deliberate, codebase-wide convention on the run-pipe family; changing only the controller abstract without changing the base class `PipeAbstract` would create inconsistency. If the base family ever moves `job_metadata` to keyword-only, this should follow.--- `pipelex/pipe_controllers/pipe_controller.py:83` — `PipeController._dry_run_controller_pipe` — same reasoning as `_live_run_controller_pipe` above; treat as a pair.diff --git a/wip/history/keyword-only-arguments/audit/findings/pipe_operators.md b/wip/history/keyword-only-arguments/audit/findings/pipe_operators.mddeleted file mode 100644index ee528bfcc..000000000--- a/wip/history/keyword-only-arguments/audit/findings/pipe_operators.md+++ /dev/null@@ -1,15 +0,0 @@-# Suspects — package `pipe_operators`--Reviewed: 40 Section A + 3 primitive lone-subjects. Suspects: 4.--## High confidence--- `pipelex/pipe_operators/img_gen/img_gen_prompt_blueprint.py:269` — `ImgGenPromptBlueprint._render_text` — `async def _render_text(self, context_provider: ContextProviderAbstract, *, template_blueprint: TemplateBlueprint, ...)` — `context_provider` is a lookup environment/registry, not the thing being rendered. `template_blueprint` is the real subject. Call sites always pass `context_provider=` as a keyword: `self._render_text(context_provider=context_provider, template_blueprint=..., ...)`. Suggested fix: make fully keyword-only — `def _render_text(self, *, context_provider, template_blueprint, ...)`.--- `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:350` — `LLMPromptBlueprint._unravel_text` — `async def _unravel_text(self, context_provider: ContextProviderAbstract, *, jinja2_blueprint: TemplateBlueprint, ...)` — identical pattern to `_render_text`: `context_provider` is the resolution environment, `jinja2_blueprint` is what gets unraveled. Call sites always use `context_provider=` as keyword. Suggested fix: make fully keyword-only — `def _unravel_text(self, *, context_provider, jinja2_blueprint, ...)`.--## Medium / low confidence--- `pipelex/pipe_operators/compose/structured_content_composer.py:392` — `StructuredContentComposer._expects_type` — `def _expects_type(self, expected_type: type[Any], *, target_type: type)` — symmetric type-comparison pair (`issubclass(expected_type, target_type)`); neither argument is more "subject" than the other. Call sites always pass `expected_type=expected_type` as keyword (e.g. `self._expects_type(expected_type=expected_type, target_type=StuffContent)`), suggesting both args are directional operands. Suggested fix: make fully keyword-only — `def _expects_type(self, *, expected_type, target_type)`.--- `pipelex/pipe_operators/img_gen/img_gen_prompt_blueprint.py:65` — `ImgGenPromptBlueprint.make_img_gen_prompt` — `async def make_img_gen_prompt(self, context_provider: ContextProviderAbstract, *, extra_params, max_prompt_images)` — `context_provider` is a resolution environment, not what's being generated (the prompt is being built from `self`, the blueprint). Call site uses `context_provider=working_memory` as keyword. Lower confidence because "context to use for building" is a defensible subject role. Suggested fix: make fully keyword-only — `def make_img_gen_prompt(self, *, context_provider, ...)`.diff --git a/wip/history/keyword-only-arguments/audit/findings/pipe_run.md b/wip/history/keyword-only-arguments/audit/findings/pipe_run.mddeleted file mode 100644index 434731859..000000000--- a/wip/history/keyword-only-arguments/audit/findings/pipe_run.md+++ /dev/null@@ -1,13 +0,0 @@-# Suspects — package `pipe_run`--Reviewed: 18 Section A + 3 primitive lone-subjects. Suspects: 3.--## High confidence--- `pipelex/pipe_run/delivery_executor.py:201` — `DeliveryExecutor._try_add_rendered_file` — `async def _try_add_rendered_file(cls, files: dict[str, ResultFile], *, filename: str, render: Awaitable[str], content_type: str) -> None` — `files` is a mutable accumulator dict (output container), not the semantic subject being acted on. The real subjects of the "add" operation are `filename`/`render`/`content_type`. At the call site: `self._try_add_rendered_file(files, filename="main_stuff.json", ...)` — the positional `files` reads as a registry/sink, not the thing the function is about. Suggested fix: make fully keyword-only — `async def _try_add_rendered_file(cls, *, files: dict[str, ResultFile], filename: str, render: Awaitable[str], content_type: str) -> None`.--- `pipelex/pipe_run/delivery_executor.py:219` — `DeliveryExecutor._add_optional_text_file` — `def _add_optional_text_file(cls, files: dict[str, ResultFile], *, filename: str, text: str | None, content_type: str) -> None` — Same pattern as `_try_add_rendered_file`: `files` is the accumulator dict carried positionally, while `filename`/`text`/`content_type` are the content being added (the actual "what"). Call site: `self._add_optional_text_file(files, filename="graphspec.json", text=..., content_type=...)`. Suggested fix: make fully keyword-only — `def _add_optional_text_file(cls, *, files: dict[str, ResultFile], filename: str, text: str | None, content_type: str) -> None`.--## Medium / low confidence--- `pipelex/pipe_run/pipe_run_params.py:28` — `output_multiplicity_to_apply` — `def output_multiplicity_to_apply(base_multiplicity: VariableMultiplicity | None, *, override_multiplicity: VariableMultiplicity | None) -> VariableMultiplicityResolution` — This is a two-operand resolution function: `base_multiplicity` and `override_multiplicity` are both inputs to a priority merge. Neither is clearly "the object the function acts on" — the function computes a result from two equal-standing inputs. The docstring examples (`output_multiplicity_to_apply(None, None)`) also show positional calling with raw `None` values, which is opaque. Suggested fix: make fully keyword-only — `def output_multiplicity_to_apply(*, base_multiplicity: VariableMultiplicity | None, override_multiplicity: VariableMultiplicity | None) -> VariableMultiplicityResolution`.diff --git a/wip/history/keyword-only-arguments/audit/findings/pipe_signature.md b/wip/history/keyword-only-arguments/audit/findings/pipe_signature.mddeleted file mode 100644index 419b12962..000000000--- a/wip/history/keyword-only-arguments/audit/findings/pipe_signature.md+++ /dev/null@@ -1,5 +0,0 @@-# Suspects — package `pipe_signature`--Reviewed: 2 Section A + 0 primitive lone-subjects. Suspects: 0.--No suspects.diff --git a/wip/history/keyword-only-arguments/audit/findings/pipelex.md b/wip/history/keyword-only-arguments/audit/findings/pipelex.mddeleted file mode 100644index 069b648e1..000000000--- a/wip/history/keyword-only-arguments/audit/findings/pipelex.md+++ /dev/null@@ -1,13 +0,0 @@-# Suspects — package `pipelex`--Reviewed: 3 Section A + 1 primitive lone-subject. Suspects: 2.--## High confidence--- `pipelex/pipelex.py:512` — `Pipelex.make` — `def make(cls, integration_mode: IntegrationMode = IntegrationMode.PYTHON, *, ...)` — `integration_mode` is a mode/configuration option, not the semantic subject of the factory (the subject is `cls` / the instance being created). All observed call sites either omit it (accepting the default) or already pass it as `integration_mode=...`. A positional call `Pipelex.make(IntegrationMode.CLI)` reads opaquely — the bare enum value conveys no context. Since this is the primary public API entry point, keyword-only is the right default. Suggested fix: `def make(cls, *, integration_mode: IntegrationMode = IntegrationMode.PYTHON, needs_inference: bool = True, ...)`.--## Medium / low confidence--- `pipelex/pipelex.py:145` — `Pipelex._get_validation_error_msg` — `def _get_validation_error_msg(component_name: str, *, validation_exc: Exception) -> str` — `component_name` is a descriptive string label ("routing profile library", "model deck"), not the object being operated on. It behaves more like a context parameter alongside `validation_exc`. Call sites: `self._get_validation_error_msg("routing profile library", validation_exc=exc)` — the bare positional string reads somewhat opaquely without the label. Private method, low call-site count, but the pattern is inconsistent: `validation_exc` is already keyword-only while `component_name` is not, despite both being descriptive context params. Suggested fix: `def _get_validation_error_msg(*, component_name: str, validation_exc: Exception) -> str`.--- `pipelex/pipelex.py:165` — `Pipelex.setup` — `def setup(self, integration_mode: IntegrationMode, *, ...)` — same logic as `make`: `integration_mode` is a mode-flag, not the subject (`self` is the subject). The sole internal call site (line 588) already uses `integration_mode=integration_mode` as keyword. Part of the public API surface (though `make` is the primary entry point). Suggested fix: `def setup(self, *, integration_mode: IntegrationMode, ...)`.diff --git a/wip/history/keyword-only-arguments/audit/findings/pipeline.md b/wip/history/keyword-only-arguments/audit/findings/pipeline.mddeleted file mode 100644index f29fc1904..000000000--- a/wip/history/keyword-only-arguments/audit/findings/pipeline.md+++ /dev/null@@ -1,11 +0,0 @@-# Suspects — package `pipeline`--Reviewed: 18 Section A + 4 primitive lone-subjects. Suspects: 1.--## High confidence--None.--## Medium / low confidence--- `pipelex/pipeline/job_metadata.py:89` — `JobMetadata.copy_with_update` — `def copy_with_update(self, otel_context: OtelContext | None, *, trace_context: TraceContext | None = None, **updates: Any)` — `otel_context` is not the subject being acted on (that's `self`, the metadata object); it's one of two context fields being updated, asymmetrically split from `trace_context` across the `*` boundary. Call sites always pass it as a keyword (`otel_context=...`) and the docstring justifies the asymmetry semantically (different inheritance semantics), but from a signature-readability standpoint both context args are peers. Suggested fix: make fully keyword-only — `def copy_with_update(self, *, otel_context: OtelContext | None, trace_context: TraceContext | None = None, **updates: Any)`.diff --git a/wip/history/keyword-only-arguments/audit/findings/plugins.md b/wip/history/keyword-only-arguments/audit/findings/plugins.mddeleted file mode 100644index 8ccd439e1..000000000--- a/wip/history/keyword-only-arguments/audit/findings/plugins.md+++ /dev/null@@ -1,15 +0,0 @@-# Suspects — package `plugins`--Reviewed: 49 Section A + 15 primitive lone-subjects. Suspects: 4.--## High confidence--- `pipelex/plugins/gateway/gateway_completions_factory.py:138` — `GatewayCompletionsFactory.make_extract_output_from_response` — `def make_extract_output_from_response(cls, inference_model: InferenceModelSpec, *, response: GenericResponse) -> ExtractOutput` — The function name is "from_response", strongly signalling that `response` is the primary operand (the thing being processed/extracted from). `inference_model` is used only as a lookup key to select which extraction sub-method to call (via `GatewayExtractProtocol.make_from_model_handle`). Swapping `inference_model` and `response` so the response is the subject, or making both keyword-only, would match the semantics. Suggested fix: make fully keyword-only `def make_extract_output_from_response(cls, *, inference_model: InferenceModelSpec, response: GenericResponse)`.--## Medium / low confidence--- `pipelex/plugins/anthropic/anthropic_factory.py:244` — `AnthropicFactory.make_nb_tokens_by_category_from_nb` — `def make_nb_tokens_by_category_from_nb(nb_input: int, *, nb_output: int) -> NbTokensByCategoryDict` — Symmetric directional pair (`nb_input` / `nb_output`); neither operand is more "the subject" than the other. Splitting them across the `*` barrier is arbitrary and reads oddly. Suggested fix: make fully keyword-only `def make_nb_tokens_by_category_from_nb(*, nb_input: int, nb_output: int)`. Note: no external call sites found in the codebase, so risk is low.--- `pipelex/plugins/mistral/mistral_factory.py:365` — `MistralFactory.make_mistral_document_url_chunk_from_uri` — `async def make_mistral_document_url_chunk_from_uri(cls, mistral_client: Mistral, *, uri: str) -> DocumentURLChunkTypedDict` — The function name ends in "from_uri", indicating `uri` is the primary operand. `mistral_client` is a dependency/tool passed to sub-operations (local file upload). The docstring itself labels it "required for local file uploads", framing it as an auxiliary dependency rather than the semantic subject. Suggested fix: make fully keyword-only `def make_mistral_document_url_chunk_from_uri(cls, *, mistral_client: Mistral, uri: str)`.--- `pipelex/plugins/plugin_sdk_registry.py:22` — `PluginSdkRegistry.set_sdk_instance` — `def set_sdk_instance(self, plugin: Plugin, *, sdk_instance: Any) -> Any` — The registry is keyed by `plugin.sdk_handle`; `plugin` acts as an index/key, while `sdk_instance` is the actual object being stored. "Set X instance" semantically implies `sdk_instance` is the subject. Call sites already pass both as keyword args (`set_sdk_instance(plugin=plugin, sdk_instance=...)`). Suggested fix: make fully keyword-only `def set_sdk_instance(self, *, plugin: Plugin, sdk_instance: Any)`.diff --git a/wip/history/keyword-only-arguments/audit/findings/reporting.md b/wip/history/keyword-only-arguments/audit/findings/reporting.mddeleted file mode 100644index 595d4b8cc..000000000--- a/wip/history/keyword-only-arguments/audit/findings/reporting.md+++ /dev/null@@ -1,7 +0,0 @@-# Suspects — package `reporting`--Reviewed: 5 Section A + 1 primitive lone-subject. Suspects: 1.--## Medium / low confidence--- `pipelex/reporting/reporting_manager.py:251` — `ReportingManager._emit_best_effort` — `def _emit_best_effort(event_log: EventLogProtocol, *, event: UsageReportEvent) -> None` — `event` is the semantic object being emitted; `event_log` is the target/sink the function dispatches to via `event_log.emit(event)`. Both call sites already pass `event_log` as a keyword (`_emit_best_effort(event_log=..., event=...)`), suggesting callers don't treat it as an obvious positional subject. The function name "emit best effort" designates the action, not the receiver, so neither arg has a clear claim as the one true subject — suggested fix: make fully keyword-only (`def _emit_best_effort(*, event_log: EventLogProtocol, event: UsageReportEvent)`).diff --git a/wip/history/keyword-only-arguments/audit/findings/runtime_bridge.md b/wip/history/keyword-only-arguments/audit/findings/runtime_bridge.mddeleted file mode 100644index b8de67be9..000000000--- a/wip/history/keyword-only-arguments/audit/findings/runtime_bridge.md+++ /dev/null@@ -1,11 +0,0 @@-# Suspects — package `runtime_bridge`--Reviewed: 12 Section A + 0 primitive lone-subjects. Suspects: 1.--## High confidence--_(none)_--## Medium / low confidence--- `pipelex/runtime_bridge/primitives/hydration.py:17` — `_validate_as_known_class` — `def _validate_as_known_class(item_class: type[StuffContent], *, raw_item: StuffContent | dict[str, Any]) -> StuffContent` — `item_class` is a type/schema used to validate, not the data being acted on; `raw_item` is the actual object being processed. The function "validates raw_item into item_class", so `raw_item` is arguably the semantic subject. Call sites read `_validate_as_known_class(item_class_or_none, raw_item=raw_item)` which is passable but unusual (a type as the positional subject). — suggested fix: `def _validate_as_known_class(*, item_class: type[StuffContent], raw_item: StuffContent | dict[str, Any]) -> StuffContent` (fully keyword-only) or reorder to `def _validate_as_known_class(raw_item: ..., *, item_class: ...)`.diff --git a/wip/history/keyword-only-arguments/audit/findings/system.md b/wip/history/keyword-only-arguments/audit/findings/system.mddeleted file mode 100644index b4b516ad5..000000000--- a/wip/history/keyword-only-arguments/audit/findings/system.md+++ /dev/null@@ -1,15 +0,0 @@-# Suspects — package `system`--Reviewed: 38 Section A + 27 primitive lone-subjects. Suspects: 4.--## High confidence--- `pipelex/system/pipelex_service/gateway_config_merger.py:69` — `GatewayConfigMerger._apply_overrides_to_model` — `def _apply_overrides_to_model(cls, model_name: str, *, gateway_model_specs: BackendModelSpecs, local_model_config: BackendModelSpecs) -> None` — `model_name` is a string identifier used only in log/warning messages; the real operands being acted on are the two keyword args `gateway_model_specs` and `local_model_config`. The sole call site already passes `model_name=model_name` as a keyword. Suggested fix: make fully keyword-only (`def _apply_overrides_to_model(cls, *, model_name, gateway_model_specs, local_model_config)`).--## Medium / low confidence--- `pipelex/system/pipelex_service/pipelex_service_agreement.py:29` — `update_service_terms_acceptance` — `def update_service_terms_acceptance(accepted: bool, *, config_dir: Path | None=None) -> None` — `accepted` is a bare `bool` positional; `update_service_terms_acceptance(True)` is opaque without the keyword. All actual call sites already use `accepted=True/False` as keyword. Suggested fix: move `*` before `accepted` (make fully keyword-only).--- `pipelex/system/pipelex_service/pipelex_service_agreement.py:53` — `update_inference_setup_completed` — `def update_inference_setup_completed(completed: bool, *, config_dir: Path | None=None) -> None` — same pattern as above: bare `bool` positional, always called as `completed=True` at every call site. Suggested fix: make fully keyword-only.--- `pipelex/system/configuration/config_check.py:12` — `check_is_initialized` — `def check_is_initialized(print_warning_if_not: bool=True) -> bool` — Section B: single `bool` param with a descriptive name, but `check_is_initialized(False)` would be opaque; every call site in tests already uses `print_warning_if_not=False/True` or the no-arg default form. Low priority since it has a default and callers always keyword it — but making it keyword-only would enforce that. Suggested fix: `def check_is_initialized(*, print_warning_if_not: bool=True)`.diff --git a/wip/history/keyword-only-arguments/audit/findings/temporal.md b/wip/history/keyword-only-arguments/audit/findings/temporal.mddeleted file mode 100644index 54815c86f..000000000--- a/wip/history/keyword-only-arguments/audit/findings/temporal.md+++ /dev/null@@ -1,17 +0,0 @@-# Suspects — package `temporal`--Reviewed: 39 Section A + 13 primitive lone-subjects. Suspects: 5.--## High confidence--- `pipelex/temporal/task_manager.py:36` — `TaskManager.run_worker` — `async def run_worker(self, is_not_sandboxed: bool, *, is_unit_testing: bool, ...)` — `is_not_sandboxed` is a boolean flag that reads opaquely positionally: `run_worker(True)` reveals nothing. It's not the semantic object the method acts on; the worker it launches is. Both real call sites already pass it as `is_not_sandboxed=...`. The `temporal_task_manager.py` override (which carries `@override` and is thus skip-exempt) would track automatically once the Protocol base is fixed. Suggested fix: make the signature fully keyword-only — `async def run_worker(self, *, is_not_sandboxed: bool, is_unit_testing: bool, ...)`.--- `pipelex/temporal/tprl/workflow_caller.py:298` — `WorkflowExecutorFactory.create_executor` — `def create_executor(cls, task_queue: str | None = None, *, workflow_execution_timeout, retry_policy, ...)` — `task_queue` is one of many configuration options for the executor being created; `create_executor` doesn't designate it as the subject (no "for this queue" or "on this queue" semantic). All four existing call sites pass it as `task_queue=...`. Suggested fix: move `task_queue` after `*` — `def create_executor(cls, *, task_queue: str | None = None, workflow_execution_timeout, ...)`.--- `pipelex/temporal/tprl_pipe/temporal_pipe_router.py:127` — `make_temporal_pipe_router` — `def make_temporal_pipe_router(task_queue: str | None = None, *, workflow_execution_timeout, retry_policy, ...)` — same pattern as `create_executor`: `task_queue` is a configuration option defaulting to `None`, not the semantic object of a "make router" call. Suggested fix: `def make_temporal_pipe_router(*, task_queue: str | None = None, workflow_execution_timeout, ...)`.--- `pipelex/temporal/tprl_pipe/temporal_pipe_run.py:131` — `make_temporal_pipe_run` — `def make_temporal_pipe_run(task_queue: str | None = None, *, workflow_execution_timeout, retry_policy, ...)` — same pattern as `make_temporal_pipe_router`. Suggested fix: `def make_temporal_pipe_run(*, task_queue: str | None = None, workflow_execution_timeout, ...)`.--## Medium / low confidence--- `pipelex/temporal/worker_cli.py:25` — `run_worker` — `async def run_worker(project: str | None = None, *, is_not_sandboxed: bool, is_unit_testing: bool, ...)` — `project` defaults to `None` and is looked up from pyproject.toml when absent. Its role is closer to a context/config override than the semantic subject of "run a worker". The one real call site passes it positionally (`run_worker(project, is_not_sandboxed=...)`) from the Typer command, but a fully keyword-only form would be equally natural. Medium confidence — `project` does describe which project's worker to run, so the positional reading is defensible.diff --git a/wip/history/keyword-only-arguments/audit/findings/tools.md b/wip/history/keyword-only-arguments/audit/findings/tools.mddeleted file mode 100644index 70c9f65a6..000000000--- a/wip/history/keyword-only-arguments/audit/findings/tools.md+++ /dev/null@@ -1,15 +0,0 @@-# Suspects — package `tools`--Reviewed: 128 Section A + 77 primitive lone-subjects. Suspects: 4.--## High confidence--- `pipelex/tools/jinja2/jinja2_required_variables.py:121` — `detect_jinja2_required_variables` — `def detect_jinja2_required_variables(template_category: TemplateCategory, *, template_source: str) -> set[str]` — `template_source` is the object being analyzed; `template_category` is a configuration/mode parameter. All call sites already pass `template_category=` as a keyword — the positional exemption provides zero benefit and buries the real subject keyword-only. Suggested fix: `def detect_jinja2_required_variables(template_source: str, *, template_category: TemplateCategory) -> set[str]`--- `pipelex/tools/jinja2/jinja2_required_variables.py:255` — `detect_jinja2_variable_references` — `def detect_jinja2_variable_references(template_category: TemplateCategory, *, template_source: str) -> list[VariableReference]` — same pattern as above: template_source is the object analyzed, template_category is configuration. All call sites use `template_category=` as keyword. Suggested fix: swap order so `template_source` is the positional subject.--- `pipelex/tools/jinja2/jinja2_filters.py:110` — `apply_tag_style` — `def apply_tag_style(context: Context, *, value: str, tag_name: str | None=None) -> str` — `context` is a Jinja2 `Context` used only as a lookup table to retrieve `TAG_STYLE`; `value` is the string being wrapped. The real subject is `value`, not `context`. Single call site already passes `context` positionally (framework-driven), but conceptually the positional arg is a lookup registry, not the acted-on object. Suggested fix: fully keyword-only `def apply_tag_style(*, context: Context, value: str, tag_name: str | None = None)`.--## Medium / low confidence--- `pipelex/tools/jinja2/jinja2_environment.py:9` — `make_jinja2_env_from_loader` — `def make_jinja2_env_from_loader(template_category: TemplateCategory, *, loader: BaseLoader, enable_async: bool=True) -> Environment` — the function name says "from loader", which points to `loader` as the operative argument; `template_category` is a configuration parameter. Minor — `template_category` is a valid configuration object that drives the env construction, so reasonable to keep it positional. Suggested fix: `def make_jinja2_env_from_loader(loader: BaseLoader, *, template_category: TemplateCategory, enable_async: bool = True)` or fully keyword-only.diff --git a/wip/history/keyword-only-arguments/audit/findings/tracing.md b/wip/history/keyword-only-arguments/audit/findings/tracing.mddeleted file mode 100644index f7e3a7803..000000000--- a/wip/history/keyword-only-arguments/audit/findings/tracing.md+++ /dev/null@@ -1,5 +0,0 @@-# Suspects — package `tracing`--Reviewed: 5 Section A + 4 primitive lone-subjects. Suspects: 0.--No suspects.diff --git a/wip/history/keyword-only-arguments/audit/positional-subject-audit.md b/wip/history/keyword-only-arguments/audit/positional-subject-audit.mddeleted file mode 100644index f555a0d9e..000000000--- a/wip/history/keyword-only-arguments/audit/positional-subject-audit.md+++ /dev/null@@ -1,2062 +0,0 @@-# Keyword-Only Refactor — Positional-Subject Audit--Every `pipelex/` function/method where the **first true parameter (after dropping `self`/`cls`) was kept positional** — i.e. where Exception 1 of the keyword-only convention ("the subject may stay positional") is in effect. This is the population to audit for the worry: *was the first arg really the semantically obvious object of the call, or was it made positional just to satisfy the rule?*--## How this was generated--An AST pass that **reuses the keyword-only guard's own carve-out detection** (`pipelex/cli/dev_cli/commands/keyword_only_guard.py`). Anything the guard skips is excluded here too, because those were never a deliberate "subject" decision:--- dunder/operator methods (`__init__`, `__eq__`, …)-- pydantic `@field_validator` / `@model_validator` / serializers-- framework entrypoints: Typer/click commands, Temporal `@activity.defn`/`@workflow.*`, pytest fixtures, Jinja2 `@pass_context`/… filters-- `@override` implementations (the parent's call convention governs them)-- `# kw-only: ignore` escape-hatch defs--Functions with **no** positional parameter at all (fully keyword-only `def f(*, a, b)`, zero-arg, or staticmethods with only keyword-only params) are also excluded — there is no positional subject to question.--Scope: `pipelex/` source only (tests excluded, matching the refactor's locked scope).--## Summary--**Total positional-subject functions: 1806**--| Category | Count | What it is |-| --- | --- | --- |-| `SUBJECT_THEN_KEYWORDS` | 797 | Subject kept positional **and** one or more other params pushed to keyword-only (`def f(subject, *, ...)`). This is where the "which arg is the subject?" decision was actually made — the **primary review target**. |-| `LONE_SUBJECT` | 1003 | A single positional parameter and nothing else after it (`def f(subject)`, optionally `**kwargs`). No "choice" of subject was possible — only one arg exists. Lowest signal; included for completeness. A bare bool/int/str subject here is the only thing arguably worth a second look (`do_thing(True)` reads opaquely). |-| `SYMMETRIC_ALLOWLIST` | 5 | Exception 2 — the curated symmetric-tuple allowlist (multiple positionals that read better ordered, e.g. `set_env(key, value)`). |-| `MULTI_POSITIONAL` | 1 | Survives the guard with 2+ positionals or a `*args` — i.e. a variadic wrapper/closure, not a deliberate subject choice. |--### By package--| Package | SUBJECT_THEN_KEYWORDS | All categories |-| --- | --- | --- |-| `base_exceptions.py` | 0 | 8 |-| `builder` | 20 | 36 |-| `cli` | 112 | 210 |-| `cogt` | 119 | 291 |-| `core` | 95 | 213 |-| `errors` | 5 | 17 |-| `graph` | 40 | 67 |-| `hub.py` | 1 | 45 |-| `kit` | 8 | 12 |-| `language` | 5 | 21 |-| `libraries` | 28 | 74 |-| `observer` | 1 | 5 |-| `pipe_controllers` | 6 | 11 |-| `pipe_operators` | 40 | 53 |-| `pipe_run` | 18 | 27 |-| `pipe_signature` | 2 | 2 |-| `pipelex.py` | 3 | 4 |-| `pipeline` | 18 | 31 |-| `plugins` | 49 | 132 |-| `reporting` | 5 | 12 |-| `runtime_bridge` | 12 | 22 |-| `system` | 38 | 100 |-| `temporal` | 39 | 88 |-| `test_extras` | 0 | 6 |-| `tools` | 128 | 296 |-| `tracing` | 5 | 23 |-------## Review strategy--**What this audit is asking each reviewer to decide:** for a given `def f(subject, *, ...)`, is `subject` genuinely the semantic object the function acts on (so positional reads naturally — `parse_pipe_spec(spec_data, ...)`), or was it kept positional merely to satisfy Exception 1, when the call would actually read better with that arg keyworded too?--### Model grade--**Sonnet 4.6 is sufficient for the per-line judgment.** This is a naming/API-taste classification, not a reasoning-heavy task — the vast majority of signatures are unambiguous from the `def` line alone. Opus-grade horsepower buys little here.--The real bottleneck is **not** model IQ — it is two other things:--1. **Context per line.** A signature alone settles most cases, but an ambiguous minority cannot be judged without the function body and a call site or two (e.g. `_resolve_preset_backend(model_deck, *, model_handle, model_type)` — is the deck the subject, or is this "resolve a backend *for a handle*" with the deck as a lookup table?). **Give the reviewer Read/Grep so it can open the body — do not feed it only the text line**, or even Opus will just guess.-2. **Calibration.** A cheap pass tends to either rubber-stamp everything or over-flag. Mitigate with a tight rubric (see `docs/contribute/keyword-only-arguments.md` for worked examples) and an instruction to **default to OK and emit only the suspects**, each with a one-line reason and a suggested fix.--### Triage plan (don't sweep all 1806 lines)--1. **Drop most of `LONE_SUBJECT` (1003).** With a single positional param there was no "subject" choice to abuse. The only sub-slice worth a look is **primitive-typed** lone subjects (`do_thing(True)`, `f(count: int)`), which read opaquely at the call site — filter those mechanically (grep), no LLM needed.-2. **Sonnet fan-out across packages** over the 797 `SUBJECT_THEN_KEYWORDS` (+ the primitive lone-subjects), one agent per package, each emitting only a shortlist of "this positional reads wrong" entries with reasoning + suggested fix. Parallel, cheap, consistent.-3. **Adjudicate only the shortlist** (yourself, or Opus on just the contested few dozen). Reserve the expensive judgment for the small contested set rather than spending it uniformly across ~1800 obvious cases.-------## A. `SUBJECT_THEN_KEYWORDS` — primary review target--_Subject kept positional **and** one or more other params pushed to keyword-only (`def f(subject, *, ...)`). This is where the "which arg is the subject?" decision was actually made — the **primary review target**._--Count: 797--### `builder` (20)--- `pipelex/builder/concept/concept_spec.py:198` — `ConceptStructureSpec._raise_type_mismatch_error` — `def _raise_type_mismatch_error(self, expected_type_name: str, *, actual_type_name: str) -> None`-- `pipelex/builder/operations/inputs_ops.py:23` — `build_inputs_for_pipe` — `async def build_inputs_for_pipe(pipe_code: str | None=None, *, mthds_contents: list[str] | None=None, bundle_path: Path | None=None, library_dirs: list[Path] | None=None) -> dict[str, Any]`-- `pipelex/builder/operations/models_ops.py:28` — `_should_include` — `def _should_include(category: ModelCategory, *, categories: list[ModelCategory] | None) -> bool`-- `pipelex/builder/operations/models_ops.py:36` — `_resolve_preset_backend` — `def _resolve_preset_backend(model_deck: ModelDeck, *, model_handle: str, model_type: ModelType) -> InferenceModelSpec | None`-- `pipelex/builder/operations/models_ops.py:46` — `_filter_presets_by_backend` — `def _filter_presets_by_backend(presets_list: list[dict[str, Any]], *, presets_dict: dict[str, Any], model_deck: ModelDeck, model_type: ModelType, backend: str) -> list[dict[str, Any]]`-- `pipelex/builder/operations/models_ops.py:67` — `_filter_aliases_by_backend` — `def _filter_aliases_by_backend(aliases: dict[str, str], *, model_deck: ModelDeck, model_type: ModelType, backend: str) -> dict[str, str]`-- `pipelex/builder/operations/models_ops.py:83` — `_filter_waterfalls_by_backend` — `def _filter_waterfalls_by_backend(waterfalls: dict[str, list[str]], *, model_deck: ModelDeck, model_type: ModelType, backend: str) -> dict[str, list[str]]`-- `pipelex/builder/operations/models_ops.py:101` — `_build_presets_for_category` — `def _build_presets_for_category(model_deck: ModelDeck, *, category: ModelCategory, backend: str | None) -> list[dict[str, Any]]`-- `pipelex/builder/operations/models_ops.py:135` — `_build_aliases_for_category` — `def _build_aliases_for_category(model_deck: ModelDeck, *, category: ModelCategory, backend: str | None) -> dict[str, str]`-- `pipelex/builder/operations/models_ops.py:160` — `_build_waterfalls_for_category` — `def _build_waterfalls_for_category(model_deck: ModelDeck, *, category: ModelCategory, backend: str | None) -> dict[str, list[str]]`-- `pipelex/builder/operations/models_ops.py:185` — `list_models` — `def list_models(categories: list[ModelCategory] | None=None, *, backend: str | None=None) -> dict[str, Any]`-- `pipelex/builder/operations/output_ops.py:14` — `build_output_for_pipe` — `async def build_output_for_pipe(mthds_contents: list[str], *, pipe_code: str, output_format: ConceptRepresentationFormat=ConceptRepresentationFormat.SCHEMA) -> dict[str, Any]`-- `pipelex/builder/operations/pipe_ops.py:47` — `_normalize_sub_pipe_list` — `def _normalize_sub_pipe_list(raw_value: Any, *, field_name: str) -> list[dict[str, Any]]`-- `pipelex/builder/operations/pipe_ops.py:89` — `parse_pipe_spec` — `def parse_pipe_spec(spec_data: Any, *, pipe_type: str) -> PipeSpec`-- `pipelex/builder/operations/pipe_ops.py:191` — `add_type_specific_fields` — `def add_type_specific_fields(pipe_spec: PipeSpec, *, pipe_table: Table) -> None`-- `pipelex/builder/operations/runner_code_ops.py:17` — `build_runner_code_for_pipe` — `async def build_runner_code_for_pipe(mthds_contents: list[str], *, pipe_code: str) -> str`-- `pipelex/builder/operations/validate_ops.py:53` — `validate_bundle_file` — `async def validate_bundle_file(bundle_path: Path, *, library_dirs: list[Path] | None=None) -> dict[str, Any]`-- `pipelex/builder/operations/validate_ops.py:110` — `validate_pipe` — `async def validate_pipe(pipe_code: str, *, library_dirs: list[Path] | None=None) -> dict[str, Any]`-- `pipelex/builder/runner_code.py:65` — `_format_representation_as_python` — `def _format_representation_as_python(representation: dict[str, Any], *, is_multiple: bool=False) -> str`-- `pipelex/builder/runner_code.py:158` — `generate_runner_code` — `def generate_runner_code(pipe: PipeAbstract, *, output_multiplicity: bool=False, library_dir: str | None=None) -> str`--### `cli` (112)--- `pipelex/cli/agent_cli/commands/agent_cli_factory.py:163` — `make_pipelex_for_agent_cli` — `def make_pipelex_for_agent_cli(library_dirs: list[str] | list[Path] | None=None, *, needs_inference: bool=True, needs_model_specs: bool | None=None) -> Pipelex`-- `pipelex/cli/agent_cli/commands/agent_output.py:218` — `_assemble_error_payload` — `def _assemble_error_payload(message: str, *, error_type: str, cause: BaseException | None, extra: dict[str, Any]) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/agent_output.py:311` — `_agent_error_json` — `def _agent_error_json(message: str, *, error_type: str, cause: BaseException | None, extra: dict[str, Any]) -> NoReturn`-- `pipelex/cli/agent_cli/commands/agent_output.py:318` — `agent_error_markdown` — `def agent_error_markdown(message: str, *, error_type: str, cause: BaseException | None=None, **extra: Any) -> NoReturn`-- `pipelex/cli/agent_cli/commands/agent_output.py:338` — `agent_error` — `def agent_error(message: str, *, error_type: str, cause: BaseException | None=None, **extra: Any) -> NoReturn`-- `pipelex/cli/agent_cli/commands/agent_output.py:382` — `agent_success_formatted` — `def agent_success_formatted(result: dict[str, Any], *, markdown_renderer: Callable[[dict[str, Any]], str], output_format: CliOutputFormat) -> None`-- `pipelex/cli/agent_cli/commands/init_cmd.py:153` — `_copy_telemetry_template` — `def _copy_telemetry_template(target_dir: Path, *, for_project: bool) -> None`-- `pipelex/cli/agent_cli/commands/init_cmd.py:173` — `_configure_backends` — `def _configure_backends(config: dict[str, Any], *, backends_toml_path: Path, template_backends_path: Path) -> list[str]`-- `pipelex/cli/agent_cli/commands/init_cmd.py:228` — `_configure_routing` — `def _configure_routing(selected_backend_keys: list[str], *, config: dict[str, Any], target_dir: Path) -> str`-- `pipelex/cli/agent_cli/commands/inputs/_inputs_core.py:13` — `inputs_core` — `async def inputs_core(pipe_code: str | None=None, *, bundle_path: Path | None=None, library_dirs: list[Path] | None=None) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/pipe_cmd.py:74` — `_add_type_specific_fields` — `def _add_type_specific_fields(pipe_spec: PipeSpec, *, pipe_table: tomlkit.TOMLDocument | tomlkit.items.Table) -> None`-- `pipelex/cli/agent_cli/commands/plxt_passthrough.py:11` — `run_plxt` — `def run_plxt(subcommand: str, *, file_path: str) -> None`-- `pipelex/cli/agent_cli/commands/run/_output_helpers.py:14` — `build_run_output` — `def build_run_output(with_memory: bool, *, main_stuff_json: dict[str, Any], working_memory_dump: dict[str, Any], compact_result: dict[str, Any] | None, extra_metadata: dict[str, Any] | None=None) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/run/_output_helpers.py:69` — `format_run_markdown` — `def format_run_markdown(result: dict[str, Any], *, with_memory: bool) -> str`-- `pipelex/cli/agent_cli/commands/run/_run_core.py:21` — `run_pipeline_core` — `async def run_pipeline_core(pipe_code: str, *, mthds_contents: list[str] | None=None, bundle_uris: list[str] | None=None, inputs: dict[str, Any] | None=None, dry_run: bool=False, mock_inputs: bool=False, library_dirs: list[str] | None=None, graph: bool=False, costs: bool=True, with_memory: bool=False) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/run/_run_core_api.py:17` — `run_pipeline_core_api` — `async def run_pipeline_core_api(pipe_code: str, *, mthds_contents: list[str] | None=None, inputs: dict[str, Any] | None=None, with_memory: bool=False) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/run/stdin_resolver.py:116` — `parse_cli_inputs` — `def parse_cli_inputs(inputs_arg: str | None, *, stdin_fallback: bool=True, auto_inputs_path: str | None=None) -> dict[str, Any] | None`-- `pipelex/cli/agent_cli/commands/validate/_validate_core.py:22` — `validate_all_core` — `async def validate_all_core(library_dirs: list[Path] | None=None, *, allow_signatures: bool=False) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/validate/_validate_core.py:57` — `validate_bundle_core` — `async def validate_bundle_core(bundle_path: Path, *, library_dirs: list[Path] | None=None, allow_signatures: bool=False) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/validate/_validate_core.py:97` — `validate_pipe_core` — `async def validate_pipe_core(pipe_code: str, *, library_dirs: list[Path] | None=None, allow_signatures: bool=False) -> dict[str, Any]`-- `pipelex/cli/cli_factory.py:31` — `make_pipelex_for_cli` — `def make_pipelex_for_cli(context: ErrorContext, *, library_dirs: list[str] | list[Path] | None=None, needs_inference: bool=True, temporal_enabled: bool | None=None, needs_model_specs: bool | None=None) -> Pipelex`-- `pipelex/cli/commands/build/inputs/_inputs_core.py:35` — `_generate_inputs_core` — `async def _generate_inputs_core(pipe_code: str | None=None, *, bundle_path: Path | None=None, output_path: Path | None=None, library_dir: list[str] | None=None) -> None`-- `pipelex/cli/commands/build/inputs/_inputs_core.py:116` — `execute_generate_inputs` — `def execute_generate_inputs(pipe_code: str | None, *, bundle_path: Path | None, output_path: Path | None, library_dir: list[str] | None=None, telemetry_command_label: str=f'{COMMAND} {SUB_COMMAND_INPUTS}') -> None`-- `pipelex/cli/commands/build/output/_output_core.py:35` — `_generate_output_core` — `async def _generate_output_core(pipe_code: str | None=None, *, bundle_path: Path | None=None, output_path: Path | None=None, output_format: ConceptRepresentationFormat=ConceptRepresentationFormat.JSON, library_dir: list[str] | None=None) -> None`-- `pipelex/cli/commands/build/output/_output_core.py:130` — `execute_generate_output` — `def execute_generate_output(pipe_code: str | None, *, bundle_path: Path | None, output_path: Path | None, output_format: ConceptRepresentationFormat, library_dir: list[str] | None=None, telemetry_command_label: str=f'{COMMAND} {SUB_COMMAND_OUTPUT}') -> None`-- `pipelex/cli/commands/build/runner/_runner_core.py:52` — `_prepare_runner_core` — `async def _prepare_runner_core(pipe_code: str | None=None, *, bundle_path: Path | None=None, output_path: Path | None=None, library_dirs: list[Path] | None=None) -> None`-- `pipelex/cli/commands/build/runner/_runner_core.py:175` — `execute_prepare_runner` — `def execute_prepare_runner(pipe_code: str | None, *, bundle_path: Path | None, output_path: Path | None, library_dirs: list[Path] | None=None, telemetry_command_label: str=f'{COMMAND} {SUB_COMMAND_RUNNER}') -> None`-- `pipelex/cli/commands/build/structures_cmd.py:50` — `_build_concept_ref_to_class_info` — `def _build_concept_ref_to_class_info(blueprints: list['PipelexBundleBlueprint'], *, output_directory: Path) -> dict[str, ConceptClassInfo]`-- `pipelex/cli/commands/build/structures_cmd.py:105` — `generate_structures_from_blueprints` — `def generate_structures_from_blueprints(blueprints: list['PipelexBundleBlueprint'], *, output_directory: Path, target_path: Path | None=None, skip_existing_check: bool=False, quiet: bool=False) -> list[tuple[str, str]]`-- `pipelex/cli/commands/doctor_cmd.py:291` — `replace_backend_file` — `def replace_backend_file(backend_name: str, *, dry_run: bool=False, config_dir: Path | None=None) -> bool`-- `pipelex/cli/commands/doctor_cmd.py:741` — `setup_doctor_runtime` — `def setup_doctor_runtime(log_config_overrides: Mapping[str, Any] | None=None, *, config_dir: Path | None=None) -> None`-- `pipelex/cli/commands/graph_cmd.py:38` — `_do_graph_render` — `def _do_graph_render(input_file: Path, *, out: str | None, direction: FlowchartDirection | None, mermaidflow: bool, reactflow: bool, subgraphs: bool, open_browser: bool) -> None`-- `pipelex/cli/commands/init/backends.py:31` — `update_backends_in_toml` — `def update_backends_in_toml(toml_doc: Any, *, selected_indices: list[int], backend_options: list[tuple[str, str]]) -> None`-- `pipelex/cli/commands/init/command.py:129` — `prime_remote_config_cache` — `def prime_remote_config_cache(console: Console, *, target_config_dir: Path | None=None) -> None`-- `pipelex/cli/commands/init/command.py:144` — `_check_gateway_terms_if_needed` — `def _check_gateway_terms_if_needed(console: Console, *, backends_toml_path: Path) -> None`-- `pipelex/cli/commands/init/command.py:182` — `determine_needs` — `def determine_needs(reset: bool, *, check_config: bool, check_inference: bool, check_routing: bool, check_telemetry: bool, backends_toml_path: Path, routing_profiles_toml_path: Path, telemetry_config_path: Path, target_config_dir: Path | None=None) -> tuple[bool, bool, bool, bool]`-- `pipelex/cli/commands/init/command.py:219` — `confirm_initialization` — `def confirm_initialization(console: Console, *, needs_config: bool, needs_inference: bool, needs_routing: bool, needs_telemetry: bool, check_credentials: bool, reset: bool, focus: InitFocus) -> bool`-- `pipelex/cli/commands/init/command.py:277` — `execute_initialization` — `def execute_initialization(console: Console, *, needs_config: bool, needs_inference: bool, needs_routing: bool, needs_telemetry: bool, check_credentials: bool, reset: bool, check_inference: bool, check_routing: bool, backends_toml_path: Path, telemetry_config_path: Path, is_first_time_backends_setup: bool, target_config_dir: Path | None=None, for_project: bool=False)`-- `pipelex/cli/commands/init/command.py:478` — `init_cmd` — `def init_cmd(focus: InitFocus=InitFocus.ALL, *, skip_confirmation: bool=False, local: bool=False)`-- `pipelex/cli/commands/init/config_files.py:20` — `init_config` — `def init_config(reset: bool=False, *, dry_run: bool=False, target_dir: Path | None=None) -> int`-- `pipelex/cli/commands/init/credentials.py:48` — `write_env_file` — `def write_env_file(env_path: Path, *, entries: dict[str, str]) -> None`-- `pipelex/cli/commands/init/credentials.py:104` — `prompt_credentials` — `def prompt_credentials(console: Console, *, backends_toml_path: Path) -> None`-- `pipelex/cli/commands/init/ide_extension.py:56` — `_install_extension` — `def _install_extension(ide_name: str, *, cmd: str, console: Console) -> bool`-- `pipelex/cli/commands/init/routing.py:26` — `customize_routing_profile` — `def customize_routing_profile(selected_backend_keys: list[str], *, target_config_dir: Path | None=None) -> None`-- `pipelex/cli/commands/init/ui/backends_ui.py:17` — `get_backend_options_from_toml` — `def get_backend_options_from_toml(template_path: Path, *, existing_path: Path | None=None) -> list[tuple[str, str]]`-- `pipelex/cli/commands/init/ui/backends_ui.py:61` — `get_currently_enabled_backends` — `def get_currently_enabled_backends(backends_toml_path: Path, *, backend_options: list[tuple[str, str]]) -> list[int]`-- `pipelex/cli/commands/init/ui/backends_ui.py:97` — `build_backend_selection_panel` — `def build_backend_selection_panel(backend_options: list[tuple[str, str]], *, currently_enabled: list[int] | None=None, is_first_time_setup: bool=False) -> Panel`-- `pipelex/cli/commands/init/ui/backends_ui.py:155` — `prompt_backend_select` — `def prompt_backend_select(console: Console, *, backend_options: list[tuple[str, str]], currently_enabled: list[int] | None=None, is_first_time_setup: bool=False) -> tuple[list[int], set[str]]`-- `pipelex/cli/commands/init/ui/general_ui.py:6` — `build_initialization_panel` — `def build_initialization_panel(needs_config: bool, *, needs_inference: bool, needs_routing: bool, needs_telemetry: bool, reset: bool, check_credentials: bool=False) -> Panel`-- `pipelex/cli/commands/init/ui/routing_ui.py:13` — `build_primary_backend_panel` — `def build_primary_backend_panel(backend_keys: list[str], *, backend_options: list[tuple[str, str]]) -> Panel`-- `pipelex/cli/commands/init/ui/routing_ui.py:91` — `build_fallback_order_panel` — `def build_fallback_order_panel(remaining_backends: list[str], *, backend_options: list[tuple[str, str]]) -> Panel`-- `pipelex/cli/commands/init/ui/routing_ui.py:127` — `prompt_fallback_order` — `def prompt_fallback_order(console: Console, *, remaining_backends: list[str], backend_options: list[tuple[str, str]]) -> list[str]`-- `pipelex/cli/commands/init/ui/routing_ui.py:198` — `display_routing_profile_result` — `def display_routing_profile_result(console: Console, *, profile_name: str, created: bool=False) -> None`-- `pipelex/cli/commands/login/command.py:105` — `serve_until_callback` — `def serve_until_callback(server: HTTPServer, *, result: dict[str, str | None]) -> None`-- `pipelex/cli/commands/run/_inputs_path_resolver.py:45` — `resolve_url_in_value` — `def resolve_url_in_value(value: Any, *, base_dir: Path) -> Any`-- `pipelex/cli/commands/run/_inputs_path_resolver.py:75` — `resolve_inputs_paths` — `def resolve_inputs_paths(inputs_dict: dict[str, Any], *, base_dir: Path) -> dict[str, Any]`-- `pipelex/cli/commands/run/_run_core.py:74` — `_execute_run` — `async def _execute_run(pipe_code: str | None, *, bundle_path: str | None, inputs: str | None, save_working_memory: bool, working_memory_path: str | None, save_main_stuff: bool, no_pretty_print: bool, graph: bool | None, graph_full_data: bool | None, output_dir: str, dry_run: bool, mock_usage: bool, mock_inputs: bool, library_dir: list[str] | None, costs: bool | None=None, dynamic_output_concept_ref: str | None=None, save_csv: str | None=None) -> None`-- `pipelex/cli/commands/run/_run_core.py:344` — `execute_run` — `def execute_run(pipe_code: str | None, *, bundle_path: str | None, inputs: str | None, save_working_memory: bool, working_memory_path: str | None, save_main_stuff: bool, no_pretty_print: bool, graph: bool | None, graph_full_data: bool | None, output_dir: str, dry_run: bool, mock_usage: bool, mock_inputs: bool, library_dir: list[str] | None, costs: bool | None=None, telemetry_command_label: str=COMMAND, temporal: bool | None=None, dynamic_output_concept_ref: str | None=None, save_csv: str | None=None) -> None`-- `pipelex/cli/commands/update_cmd.py:34` — `update_cmd` — `def update_cmd(local: bool=False, *, yes: bool=False, dry_run: bool=False, no_backup: bool=False) -> None`-- `pipelex/cli/commands/update_cmd.py:102` — `_print_status_table` — `def _print_status_table(report: DeckSyncReport, *, deck_dir: Path) -> None`-- `pipelex/cli/commands/update_cmd.py:138` — `_apply_updates` — `def _apply_updates(deck_dir: Path, *, report: DeckSyncReport, no_backup: bool) -> int`-- `pipelex/cli/commands/update_cmd.py:187` — `_summary_panel` — `def _summary_panel(message: str, *, style: str) -> Panel`-- `pipelex/cli/commands/validate/_validate_core.py:59` — `do_validate_all_libraries_and_dry_run` — `def do_validate_all_libraries_and_dry_run(library_dirs: list[Path] | None=None, *, allow_signatures: bool=False) -> None`-- `pipelex/cli/commands/validate/_validate_core.py:99` — `_validate_pipe_or_bundle` — `async def _validate_pipe_or_bundle(pipe_code: str | None=None, *, bundle_path: Path | None=None, library_dirs: list[Path] | None=None, allow_signatures: bool=False) -> None`-- `pipelex/cli/commands/validate/_validate_core.py:162` — `execute_validate` — `def execute_validate(pipe_code: str | None, *, bundle_path: Path | None, library_dirs: list[Path] | None, telemetry_command_label: str=COMMAND, allow_signatures: bool=False, temporal: bool | None=None) -> None`-- `pipelex/cli/commands/which_cmd.py:33` — `do_which_pipe` — `def do_which_pipe(pipe_code: str, *, library_dirs: list[Path], source_label: str) -> bool`-- `pipelex/cli/dev_cli/commands/check_config_sync_cmd.py:24` — `check_config_sync_cmd` — `def check_config_sync_cmd(show_diff: bool=True, *, leading: LeadingConfig=LeadingConfig.INSTALLED, quiet: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/check_gateway_models_cmd.py:28` — `check_gateway_models_cmd` — `def check_gateway_models_cmd(show_diff: bool=True, *, quiet: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/check_mthds_schema_cmd.py:21` — `check_mthds_schema_cmd` — `def check_mthds_schema_cmd(show_diff: bool=True, *, quiet: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/check_rules_sync_cmd.py:40` — `check_rules_sync_cmd` — `def check_rules_sync_cmd(show_diff: bool=True, *, quiet: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/check_urls_cmd.py:63` — `check_single_url_async` — `async def check_single_url_async(client: httpx.AsyncClient, *, name: str, url: str) -> URLCheckResult`-- `pipelex/cli/dev_cli/commands/check_urls_cmd.py:118` — `check_all_urls_async` — `async def check_all_urls_async(url_pairs: list[tuple[str, str]], *, request_timeout: int) -> list[URLCheckResult]`-- `pipelex/cli/dev_cli/commands/check_urls_cmd.py:145` — `check_urls_cmd` — `def check_urls_cmd(quiet: bool=False, *, timeout: int=DEFAULT_TIMEOUT) -> None`-- `pipelex/cli/dev_cli/commands/gateway_models_generator.py:126` — `generate_pure_markdown_list.sort_by_preferred` — `def sort_by_preferred(items: list[str], *, preferred: list[str]) -> list[str]`-- `pipelex/cli/dev_cli/commands/gateway_models_generator.py:165` — `generate_markdown_table.sort_by_preferred` — `def sort_by_preferred(items: set[str], *, preferred: list[str]) -> list[str]`-- `pipelex/cli/dev_cli/commands/generate_error_pages_cmd.py:16` — `generate_error_pages_cmd` — `def generate_error_pages_cmd(output: Path | None=None, *, quiet: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/generate_mthds_schema_cmd.py:18` — `generate_mthds_schema_cmd` — `def generate_mthds_schema_cmd(output: Path | None=None, *, quiet: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:214` — `_positional_or_keyword_count` — `def _positional_or_keyword_count(node: ast.FunctionDef | ast.AsyncFunctionDef, *, is_method: bool) -> int`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:232` — `_def_line_has_escape_hatch` — `def _def_line_has_escape_hatch(node: ast.FunctionDef | ast.AsyncFunctionDef, *, source_lines: list[str]) -> bool`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:240` — `_qualify` — `def _qualify(name: str, *, class_stack: tuple[str, ...], func_stack: tuple[str, ...]) -> str`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:245` — `_evaluate_def` — `def _evaluate_def(node: ast.FunctionDef | ast.AsyncFunctionDef, *, module_qname: str, relative_path: str, class_stack: tuple[str, ...], func_stack: tuple[str, ...], is_method: bool, source_lines: list[str]) -> Violation | None`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:327` — `find_violations_in_source` — `def find_violations_in_source(source: str, *, module_qname: str, relative_path: str) -> list[Violation]`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:381` — `relative_source_path` — `def relative_source_path(path: Path, *, root: Path | None=None) -> Path | None`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:407` — `collect_violations_for_files` — `def collect_violations_for_files(paths: Iterable[Path], *, root: Path | None=None) -> list[Violation]`-- `pipelex/cli/dev_cli/commands/kit_cmd.py:31` — `_sync_agent_rules` — `def _sync_agent_rules(repo_root: Path | None, *, agent_set: str | None, cleanup: bool, kit_index: KitIndex | None=None, targets_filter: list[AgentTarget] | None=None) -> None`-- `pipelex/cli/dev_cli/commands/kit_cmd.py:91` — `_cleanup_other_targets` — `def _cleanup_other_targets(repo_root: Path, *, kit_index: KitIndex, preferred_targets: list[AgentTarget]) -> None`-- `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:247` — `_process_collections_from_toml` — `def _process_collections_from_toml(collections_raw: Any, *, collections: dict[str, dict[str, list[str]]]) -> None`-- `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:292` — `_resolve_model_list` — `def _resolve_model_list(raw_list: list[str], *, collections: dict[str, list[str]], backend_models: dict[str, list[str]], all_known_models: list[str]) -> list[str]`-- `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:367` — `_filter_models_by_profile` — `def _filter_models_by_profile(availability: dict[str, Any], *, profile: dict[str, Any], collections: dict[str, dict[str, list[str]]]) -> dict[str, list[tuple[str, str]]]`-- `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:498` — `_generate_fixtures_python` — `def _generate_fixtures_python(combo_pairs: dict[str, list[tuple[str, str]]], *, profile_name: str) -> str`-- `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:561` — `_display_summary` — `def _display_summary(availability: dict[str, Any], *, combo_pairs: dict[str, list[tuple[str, str]]], profile_name: str, console: Console) -> None`-- `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:611` — `preprocess_test_models_cmd` — `def preprocess_test_models_cmd(profile: str='dev', *, generate_fixtures: bool=False, output_json: bool=False, quiet: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/refresh_graph_ui_sri_cmd.py:86` — `_validate_version` — `def _validate_version(name: str, *, value: str) -> str`-- `pipelex/cli/dev_cli/commands/refresh_graph_ui_sri_cmd.py:106` — `_fetch` — `def _fetch(url: str, *, timeout: float=30.0) -> bytes`-- `pipelex/cli/dev_cli/commands/refresh_graph_ui_sri_cmd.py:148` — `refresh_graph_ui_sri_cmd` — `def refresh_graph_ui_sri_cmd(mthds_ui_version: str | None=None, *, elkjs_version: str | None=None, output_path: Path | None=None, quiet: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/sync_kit_configs_cmd.py:20` — `_display_result` — `def _display_result(result: MirrorDirResult, *, quiet: bool) -> None`-- `pipelex/cli/dev_cli/commands/sync_kit_configs_cmd.py:86` — `sync_kit_configs_cmd` — `def sync_kit_configs_cmd(quiet: bool=False, *, dry_run: bool=False) -> None`-- `pipelex/cli/dev_cli/commands/sync_main_config_cmd.py:38` — `_display_sync_result` — `def _display_sync_result(result: TomlSyncResult, *, target_label: str, show_diff: bool, quiet: bool) -> None`-- `pipelex/cli/dev_cli/commands/sync_main_config_cmd.py:77` — `sync_main_config_cmd` — `def sync_main_config_cmd(target: SyncTarget=SyncTarget.ALL, *, dry_run: bool=False, quiet: bool=False, show_diff: bool=True) -> None`-- `pipelex/cli/dev_cli/commands/update_gateway_models_cmd.py:41` — `_all_references_up_to_date` — `def _all_references_up_to_date(reference_files: list[tuple[Path, Path]], *, expected_html: str, expected_plain: str) -> bool`-- `pipelex/cli/error_handlers.py:86` — `display_error_panel` — `def display_error_panel(console: Console, *, title: str, fields: list[tuple[str, str]], error_message: str | None, tip: str, links: list[tuple[str, str]]) -> None`-- `pipelex/cli/error_handlers.py:125` — `handle_model_choice_error` — `def handle_model_choice_error(exc: PipeOperatorModelChoiceError, *, context: ErrorContext) -> NoReturn`-- `pipelex/cli/error_handlers.py:156` — `handle_model_availability_error` — `def handle_model_availability_error(exc: PipeOperatorModelAvailabilityError, *, context: ErrorContext) -> NoReturn`-- `pipelex/cli/error_handlers.py:193` — `handle_model_deck_preset_error` — `def handle_model_deck_preset_error(exc: ModelDeckPresetValidatonError, *, context: ErrorContext) -> NoReturn`-- `pipelex/cli/error_handlers.py:248` — `_display_validation_error_details` — `def _display_validation_error_details(console: Console, *, exc: ValidateBundleError) -> None`-- `pipelex/cli/error_handlers.py:325` — `handle_signatures_not_allowed_error` — `def handle_signatures_not_allowed_error(exc: SignaturesNotAllowedError, *, context: ErrorContext) -> NoReturn`-- `pipelex/cli/error_handlers.py:343` — `handle_validate_bundle_error` — `def handle_validate_bundle_error(exc: ValidateBundleError, *, bundle_path: Path | None=None) -> NoReturn`-- `pipelex/cli/installed_methods.py:43` — `discover_installed_methods` — `def discover_installed_methods(include_global: bool=True, *, include_project: bool=True, extra_search_dirs: list[Path] | None=None) -> list[InstalledMethod]`-- `pipelex/cli/installed_methods.py:112` — `discover_method_at` — `def discover_method_at(method_dir: Path, *, seen_dirs: set[Path]) -> InstalledMethod | None`-- `pipelex/cli/installed_methods.py:184` — `find_method_by_full_address` — `def find_method_by_full_address(full_address: str, *, methods: list[InstalledMethod] | None=None, extra_search_dirs: list[Path] | None=None) -> InstalledMethod | None`-- `pipelex/cli/installed_methods.py:216` — `find_method_by_name` — `def find_method_by_name(method_name: str, *, methods: list[InstalledMethod] | None=None, library_dirs: list[str] | None=None) -> InstalledMethod`-- `pipelex/cli/method_resolver.py:227` — `resolve_method_target` — `def resolve_method_target(method_name: str, *, pipe_override: str | None=None, library_dirs: list[str] | None=None) -> tuple[str, list[str], InstalledMethod]`--### `cogt` (119)--- `pipelex/cogt/config_cogt.py:30` — `ImgGenConfig.get_num_inference_steps` — `def get_num_inference_steps(self, model_name: str, *, quality: Quality) -> int`-- `pipelex/cogt/config_cogt.py:96` — `LLMConfig.get_reasoning_budget` — `def get_reasoning_budget(self, prompting_target: str, *, effort: ReasoningEffort) -> int`-- `pipelex/cogt/content_generation/assignment_models.py:87` — `ObjectAssignment.make_for_class` — `def make_for_class(object_class: type[BaseModel], *, llm_assignment: LLMAssignment, nb_items: int | None=None) -> 'ObjectAssignment'`-- `pipelex/cogt/content_generation/assignment_models.py:177` — `SearchObjectAssignment.make_for_class` — `def make_for_class(output_class: type[BaseModel], *, search_assignment: SearchAssignment) -> 'SearchObjectAssignment'`-- `pipelex/cogt/content_generation/content_generator.py:44` — `_revalidate_against_object_class` — `def _revalidate_against_object_class(raw_obj: BaseModel, *, object_class: type[BaseModelTypeVar], is_mock_built: bool) -> BaseModelTypeVar`-- `pipelex/cogt/content_generation/dry_mock.py:143` — `report_dry_llm_job` — `def report_dry_llm_job(job_metadata: JobMetadata, *, llm_setting: LLMSetting, llm_prompt: LLMPrompt) -> None`-- `pipelex/cogt/content_generation/dry_mock.py:155` — `report_mock_usage_llm_job` — `def report_mock_usage_llm_job(job_metadata: JobMetadata, *, llm_setting: LLMSetting, llm_prompt: LLMPrompt) -> None`-- `pipelex/cogt/content_generation/dry_mock.py:186` — `build_mock_objects` — `def build_mock_objects(model_class: type[BaseModelTypeVar], *, count: int) -> list[BaseModelTypeVar]`-- `pipelex/cogt/content_generation/dry_mock.py:240` — `_leaf_gen_object` — `def _leaf_gen_object(object_assignment: ObjectAssignment, *, report_func: _ReportLLMJobFunc) -> BaseModel`-- `pipelex/cogt/content_generation/dry_mock.py:262` — `_leaf_gen_object_list` — `def _leaf_gen_object_list(object_assignment: ObjectAssignment, *, report_func: _ReportLLMJobFunc) -> list[BaseModel]`-- `pipelex/cogt/content_generation/dry_mock.py:343` — `_dry_image_content` — `def _dry_image_content(image_url: str, *, img_gen_assignment: ImgGenAssignment | None=None) -> ImageContent`-- `pipelex/cogt/content_generation/dry_run_factory.py:197` — `DryRunFactory._find_nested_base_model_classes` — `def _find_nested_base_model_classes(cls, object_class: type[BaseModel], *, visited: set[type[BaseModel]] | None=None) -> set[type[BaseModel]]`-- `pipelex/cogt/content_generation/dry_run_factory.py:339` — `DryRunFactory.make_dry_run_factory` — `def make_dry_run_factory(cls, object_class: type[BaseModelTypeVar], *, snake_case_field_names: set[str] | None=None, pascal_case_field_names: set[str] | None=None) -> type[ModelFactory[BaseModelTypeVar]]`-- `pipelex/cogt/content_generation/extract_generate.py:21` — `extract_gen_pages_and_store` — `async def extract_gen_pages_and_store(extract_assignment: ExtractAssignment, *, generated_content_factory: GeneratedContentFactory) -> list[PageContent]`-- `pipelex/cogt/content_generation/generated_content_factory.py:25` — `GeneratedContentFactory._build_storage_key` — `def _build_storage_key(self, primary_id: str, *, secondary_id: str, data: bytes, mime_type: str | None, image_format: ImageFormat | None) -> str`-- `pipelex/cogt/content_generation/generated_content_factory.py:68` — `GeneratedContentFactory.make_image_content` — `async def make_image_content(self, primary_id: str, *, secondary_id: str, raw_details: GeneratedImageRawDetails) -> ImageContent`-- `pipelex/cogt/content_generation/generated_content_factory.py:161` — `GeneratedContentFactory.make_page_contents` — `async def make_page_contents(self, primary_id: str, *, secondary_id: str, extract_output: ExtractOutput) -> list[PageContent]`-- `pipelex/cogt/content_generation/img_gen_generate.py:40` — `img_gen_single_image_and_store` — `async def img_gen_single_image_and_store(img_gen_assignment: ImgGenAssignment, *, generated_content_factory: GeneratedContentFactory) -> ImageContent`-- `pipelex/cogt/content_generation/img_gen_generate.py:66` — `img_gen_image_list_and_store` — `async def img_gen_image_list_and_store(img_gen_assignment: ImgGenAssignment, *, generated_content_factory: GeneratedContentFactory) -> list[ImageContent]`-- `pipelex/cogt/content_generation/render_generate.py:17` — `render_page_views_and_store` — `async def render_page_views_and_store(render_assignment: RenderPageViewsAssignment, *, generated_content_factory: GeneratedContentFactory) -> list[ImageContent]`-- `pipelex/cogt/content_generation/schema_to_model_factory.py:82` — `SchemaToModelFactory.make_from_json_schema` — `def make_from_json_schema(cls, schema: dict[str, Any], *, class_name: str) -> type[BaseModel]`-- `pipelex/cogt/content_generation/schema_to_model_factory.py:126` — `SchemaToModelFactory._collect_unsafe_extension_paths` — `def _collect_unsafe_extension_paths(cls, node: Any, *, path: str, found: list[str]) -> None`-- `pipelex/cogt/content_generation/schema_to_model_factory.py:319` — `SchemaToModelFactory._exec_and_extract_class` — `def _exec_and_extract_class(cls, source_code: str, *, class_name: str) -> type[BaseModel]`-- `pipelex/cogt/document/prompt_document_factory.py:18` — `PromptDocumentFactory.make_prompt_document` — `def make_prompt_document(cls, uri: str | None=None, *, base64_data: str | None=None, raw_bytes: bytes | None=None, mime_type: str | None=None) -> PromptDocument`-- `pipelex/cogt/document/prompt_document_utils.py:30` — `prepare_prompt_document` — `async def prepare_prompt_document(prompt_document: PromptDocument, *, is_http_url_enabled: bool) -> PreparedFile`-- `pipelex/cogt/document/prompt_document_utils.py:96` — `prep_prompt_documents` — `async def prep_prompt_documents(prompt_documents: list[PromptDocument], *, is_http_url_enabled: bool) -> list[PreparedFile]`-- `pipelex/cogt/exceptions.py:68` — `CogtError.fill_model_and_provider` — `def fill_model_and_provider(self, model_handle: str | None, *, backend_name: str | None) -> None`-- `pipelex/cogt/extract/extract_job_factory.py:9` — `ExtractJobFactory.make_extract_job` — `def make_extract_job(cls, extract_input: ExtractInput, *, job_metadata: JobMetadata, extract_job_params: ExtractJobParams | None=None, extract_job_config: ExtractJobConfig | None=None) -> ExtractJob`-- `pipelex/cogt/extract/extract_worker_factory.py:13` — `ExtractWorkerFactory.make_extract_worker` — `def make_extract_worker(cls, inference_model: InferenceModelSpec, *, reporting_delegate: ReportingProtocol | None=None) -> ExtractWorkerAbstract`-- `pipelex/cogt/file/file_preparation_utils.py:24` — `prepare_file_from_uri` — `async def prepare_file_from_uri(uri: str, *, keep_http_url: bool, keep_local_path: bool) -> PreparedFile`-- `pipelex/cogt/image/generated_image.py:61` — `GeneratedImageRawDetails.make_from_pil_image` — `def make_from_pil_image(cls, pil_image: Image.Image, *, image_format: ImageFormat) -> GeneratedImageRawDetails`-- `pipelex/cogt/image/prompt_image_factory.py:18` — `PromptImageFactory.make_prompt_image` — `def make_prompt_image(cls, uri: str | None=None, *, base64_data: str | None=None, raw_bytes: bytes | None=None) -> PromptImage`-- `pipelex/cogt/image/prompt_image_utils.py:30` — `prepare_prompt_image` — `async def prepare_prompt_image(prompt_image: PromptImage, *, is_http_url_enabled: bool) -> PreparedFile`-- `pipelex/cogt/image/prompt_image_utils.py:97` — `prep_prompt_images` — `async def prep_prompt_images(prompt_images: list[PromptImage], *, is_http_url_enabled: bool) -> list[PreparedFile]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:49` — `ImgGenArgsFactory.make_args_for_model` — `async def make_args_for_model(cls, model_rules: ImgGenModelRules, *, img_gen_job: ImgGenJob, nb_images: int, model_id: str, model_name: str) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:199` — `ImgGenArgsFactory.make_args_from_num_images` — `def make_args_from_num_images(cls, num_images_taxonomy: NumImagesTaxonomy, *, nb_images: int) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:208` — `ImgGenArgsFactory.make_args_from_prompt` — `def make_args_from_prompt(cls, prompt_taxonomy: PromptTaxonomy, *, positive_text: str, negative_text: str | None) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:238` — `ImgGenArgsFactory.make_args_from_model_name` — `def make_args_from_model_name(cls, model_name_taxonomy: ModelChoiceTaxonomy, *, model_id: str, model_name: str) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:253` — `ImgGenArgsFactory.make_args_from_background` — `def make_args_from_background(cls, background_taxonomy: BackgroundTaxonomy, *, background: Background, model_name: str) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:270` — `ImgGenArgsFactory.make_args_from_aspect_ratio` — `def make_args_from_aspect_ratio(cls, aspect_ratio_taxonomy: AspectRatioTaxonomy, *, aspect_ratio: AspectRatio, size: ImageSize | None, model_name: str) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:373` — `ImgGenArgsFactory.make_args_from_inference` — `def make_args_from_inference(cls, inference_taxonomy: InferenceTaxonomy, *, num_inference_steps: int | None, quality: Quality | None, guidance_scale: float | None, is_raw: bool | None) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:425` — `ImgGenArgsFactory.make_args_from_safety_checker` — `def make_args_from_safety_checker(cls, safety_checker_taxonomy: SafetyCheckerTaxonomy, *, is_moderated: bool | None, safety_tolerance: int | None) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:454` — `ImgGenArgsFactory.make_args_from_output_format` — `def make_args_from_output_format(cls, output_format_taxonomy: OutputFormatTaxonomy, *, output_format: ImageFormat | None) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:525` — `ImgGenArgsFactory.make_args_from_input_images` — `async def make_args_from_input_images(cls, input_images_taxonomy: InputImagesTaxonomy, *, input_images: list[PromptImage] | None) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:584` — `ImgGenArgsFactory.make_args_from_input_fidelity` — `def make_args_from_input_fidelity(cls, input_fidelity_taxonomy: InputFidelityTaxonomy, *, input_fidelity: InputFidelity | None, model_name: str) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_job_factory.py:10` — `ImgGenJobFactory.make_img_gen_job_from_prompt` — `def make_img_gen_job_from_prompt(cls, img_gen_prompt: ImgGenPrompt, *, job_metadata: JobMetadata, img_gen_job_params: ImgGenJobParams | None=None, img_gen_job_config: ImgGenJobConfig | None=None) -> ImgGenJob`-- `pipelex/cogt/img_gen/img_gen_job_factory.py:32` — `ImgGenJobFactory.make_img_gen_job_from_prompt_contents` — `def make_img_gen_job_from_prompt_contents(cls, positive_text: str, *, negative_text: str | None, job_metadata: JobMetadata, img_gen_job_params: ImgGenJobParams | None=None, img_gen_job_config: ImgGenJobConfig | None=None) -> ImgGenJob`-- `pipelex/cogt/img_gen/img_gen_worker_abstract.py:81` — `ImgGenWorkerAbstract.gen_image_list` — `async def gen_image_list(self, img_gen_job: ImgGenJob, *, nb_images: int) -> list[GeneratedImageRawDetails]`-- `pipelex/cogt/img_gen/img_gen_worker_abstract.py:116` — `ImgGenWorkerAbstract._gen_image_list` — `async def _gen_image_list(self, img_gen_job: ImgGenJob, *, nb_images: int) -> list[GeneratedImageRawDetails]`-- `pipelex/cogt/img_gen/img_gen_worker_factory.py:14` — `ImgGenWorkerFactory.make_img_gen_worker` — `def make_img_gen_worker(cls, inference_model: InferenceModelSpec, *, reporting_delegate: ReportingProtocol | None=None) -> ImgGenWorkerAbstract`-- `pipelex/cogt/inference/error_classification.py:47` — `_resolve_sdk_exception_type` — `def _resolve_sdk_exception_type(exc: BaseException, *, status_code: int | None) -> str`-- `pipelex/cogt/inference/error_classification.py:249` — `_is_quota_exhaustion_mistral` — `def _is_quota_exhaustion_mistral(error_message: str, *, status_code: int) -> bool`-- `pipelex/cogt/inference/error_classification.py:261` — `_is_quota_exhaustion_aws` — `def _is_quota_exhaustion_aws(error_message: str, *, provider_error_code: str | None) -> bool`-- `pipelex/cogt/inference/error_classification.py:276` — `_is_quota_exhaustion_gateway` — `def _is_quota_exhaustion_gateway(error_message: str, *, status_code: int) -> bool`-- `pipelex/cogt/inference/error_classification.py:586` — `extract_azure_metadata_from_response` — `def extract_azure_metadata_from_response(response: Any, *, sdk_exception_type: str, message: str) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:597` — `_build_azure_metadata` — `def _build_azure_metadata(response: Any, *, sdk_exception_type: str, message: str) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:878` — `extract_local_extract_metadata` — `def extract_local_extract_metadata(exc: BaseException, *, provider: ProviderName) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_render.py:53` — `_format_message` — `def _format_message(metadata: SDKErrorEnvelope, *, model_desc: str) -> str`-- `pipelex/cogt/inference/error_render.py:59` — `_render_detail` — `def _render_detail(metadata: SDKErrorEnvelope, *, classification: ClassificationResult) -> str`-- `pipelex/cogt/inference/error_render.py:80` — `render_inference_error` — `def render_inference_error(metadata: SDKErrorEnvelope, *, classification: ClassificationResult, family: InferenceErrorFamily, model_desc: str, model_handle: str) -> CogtError`-- `pipelex/cogt/inference/inference_manager_protocol.py:21` — `InferenceManagerProtocol.set_llm_worker_from_external_plugin` — `def set_llm_worker_from_external_plugin(self, llm_handle: str, *, llm_worker_class: type[LLMWorkerAbstract], should_warn_if_already_registered: bool=True)`-- `pipelex/cogt/inference/transport_retry.py:110` — `request_with_transport_retry` — `async def request_with_transport_retry(send_request: Callable[[], Awaitable[httpx.Response]], *, max_retries: int, retry_on_ambiguous_failure: bool=True) -> httpx.Response`-- `pipelex/cogt/llm/llm_job_factory.py:10` — `LLMJobFactory.make_llm_job` — `def make_llm_job(cls, llm_prompt: LLMPrompt, *, job_metadata: JobMetadata, llm_job_params: LLMJobParams, llm_job_config: LLMJobConfig | None=None) -> LLMJob`-- `pipelex/cogt/llm/llm_prompt_template.py:58` — `LLMPromptTemplate._make_llm_prompt` — `async def _make_llm_prompt(self, system_text: str | None=None, *, user_text: str | None=None, user_images: list[PromptImage] | None=None, is_user_images_append: bool | None=None, template_inputs: LLMPromptTemplateInputs | None=None) -> LLMPrompt`-- `pipelex/cogt/llm/llm_setting.py:95` — `LLMSettingChoices.make_completed_with_defaults` — `def make_completed_with_defaults(cls, for_text: LLMModelChoice | None=None, *, for_object: LLMModelChoice | None=None) -> Self`-- `pipelex/cogt/llm/llm_worker_abstract.py:86` — `LLMWorkerAbstract._start_otel_span_llm` — `def _start_otel_span_llm(self, llm_job: LLMJob, *, output_type: InferenceOutputType, output_class_name: str | None=None) -> Span | None`-- `pipelex/cogt/llm/llm_worker_abstract.py:231` — `LLMWorkerAbstract._end_otel_span_with_completion_text` — `def _end_otel_span_with_completion_text(self, span: Span | None, *, llm_job: LLMJob, completion_text: str) -> None`-- `pipelex/cogt/llm/llm_worker_abstract.py:262` — `LLMWorkerAbstract._end_otel_span_with_completion_object` — `def _end_otel_span_with_completion_object(self, span: Span | None, *, llm_job: LLMJob, completion_object: BaseModel) -> None`-- `pipelex/cogt/llm/llm_worker_abstract.py:294` — `LLMWorkerAbstract._end_otel_span_with_error` — `def _end_otel_span_with_error(self, span: Span | None, *, llm_job: LLMJob, error: BaseException) -> None`-- `pipelex/cogt/llm/llm_worker_abstract.py:398` — `LLMWorkerAbstract.gen_object` — `async def gen_object(self, llm_job: LLMJob, *, schema: type[BaseModelTypeVar]) -> BaseModelTypeVar`-- `pipelex/cogt/llm/llm_worker_abstract.py:436` — `LLMWorkerAbstract._gen_object` — `async def _gen_object(self, llm_job: LLMJob, *, schema: type[BaseModelTypeVar]) -> BaseModelTypeVar`-- `pipelex/cogt/llm/llm_worker_factory.py:13` — `LLMWorkerFactory.make_llm_worker` — `def make_llm_worker(inference_model: InferenceModelSpec, *, reporting_delegate: ReportingProtocol | None=None) -> LLMWorkerInternalAbstract`-- `pipelex/cogt/llm/reasoning_config_base.py:9` — `validate_effort_to_level_map` — `def validate_effort_to_level_map(effort_to_level_map: EffortToLevelMap, *, config_name: str, level_type: type[StrEnum] | None=None) -> EffortToLevelMap`-- `pipelex/cogt/llm/reasoning_config_base.py:40` — `get_reasoning_level_str` — `def get_reasoning_level_str(effort_to_level_map: EffortToLevelMap, *, effort: ReasoningEffort) -> str | None`-- `pipelex/cogt/model_backends/backend_credentials.py:26` — `BackendCredentialsErrorMsgFactory.make_one_variable_missing_error_msg` — `def make_one_variable_missing_error_msg(cls, secrets_provider: SecretsProviderAbstract, *, backend_name: str | None, var_name: str) -> str`-- `pipelex/cogt/model_backends/backend_credentials.py:78` — `BackendCredentialsErrorMsgFactory.make_comprehensive_error_msg` — `def make_comprehensive_error_msg(cls, backend_credential_reports: dict[str, BackendCredentialsReport], *, secrets_provider: SecretsProviderAbstract | None=None) -> str`-- `pipelex/cogt/model_backends/backend_factory.py:39` — `InferenceBackendFactory.make_inference_backend` — `def make_inference_backend(cls, name: str, *, blueprint: InferenceBackendBlueprint, extra_config: dict[str, Any], model_specs: dict[str, InferenceModelSpec]) -> InferenceBackend`-- `pipelex/cogt/model_backends/backend_library.py:58` — `InferenceBackendLibrary.load` — `def load(self, secrets_provider: SecretsProviderAbstract, *, backends_library_path: str, backends_dir_path: str, include_disabled: bool=False, gateway_config: GatewayConfig | None=None, lenient: bool=False)`-- `pipelex/cogt/model_backends/backend_library.py:228` — `InferenceBackendLibrary._load_gateway_model_specs` — `def _load_gateway_model_specs(self, gateway_config: GatewayConfig, *, backends_dir_path: str, substitute_vars_with_provider: Any) -> tuple[BackendModelSpecs, str]`-- `pipelex/cogt/model_backends/backend_library.py:267` — `InferenceBackendLibrary._load_local_model_specs` — `def _load_local_model_specs(self, backend_name: str, *, backends_dir_path: str, substitute_vars_with_provider: Any) -> tuple[BackendModelSpecs, str]`-- `pipelex/cogt/model_backends/backend_library.py:300` — `InferenceBackendLibrary.check_backend_credentials` — `def check_backend_credentials(self, path: str, *, include_disabled: bool=False) -> CredentialsValidationReport`-- `pipelex/cogt/model_backends/model_lists.py:17` — `ModelLister.list_models` — `async def list_models(cls, backend_name: str, *, flat: bool=False) -> None`-- `pipelex/cogt/model_backends/model_lists.py:194` — `ModelLister._display_unsupported_sdks_message` — `def _display_unsupported_sdks_message(any_listed: bool, *, unsupported_sdks: list[str], backend_name: str, models_by_sdk: dict[str, list[str]], flat: bool) -> None`-- `pipelex/cogt/model_backends/model_spec_factory.py:78` — `InferenceModelSpecFactory.make_inference_model_spec` — `def make_inference_model_spec(cls, backend_name: str, *, name: str, blueprint: InferenceModelSpecBlueprint, backend_listed_constraints: list[ListedConstraint], backend_valued_constraints: dict[ValuedConstraint, Any], extra_headers: dict[str, str] | None=None) -> InferenceModelSpec`-- `pipelex/cogt/model_routing/routing_profile.py:25` — `RoutingProfile.get_backend_match_for_model` — `def get_backend_match_for_model(self, enabled_backends: list[str], *, model_name: str) -> BackendMatchForModel | None`-- `pipelex/cogt/model_routing/routing_profile_factory.py:38` — `RoutingProfileFactory.make_routing_profile` — `def make_routing_profile(cls, name: str, *, blueprint: RoutingProfileBlueprint) -> RoutingProfile`-- `pipelex/cogt/model_routing/routing_profile_loader.py:11` — `load_active_routing_profile` — `def load_active_routing_profile(routing_profile_library_path: str, *, enabled_backends: list[str], lenient: bool=False) -> RoutingProfile`-- `pipelex/cogt/models/deck_manifest.py:155` — `write_manifest` — `def write_manifest(manifest: DeckManifest, *, deck_dir: Path) -> None`-- `pipelex/cogt/models/deck_manifest.py:179` — `_is_manifest_older` — `def _is_manifest_older(manifest_version: str, *, current_version: str) -> bool`-- `pipelex/cogt/models/model_deck.py:135` — `ModelDeck.is_model_handle_defined` — `def is_model_handle_defined(self, model_handle: str, *, model_type: ModelType) -> bool`-- `pipelex/cogt/models/model_deck.py:219` — `ModelDeck._raise_handle_not_found_error` — `def _raise_handle_not_found_error(self, ref: ModelReference, *, model_type: ModelType, presets: dict[str, LLMSetting] | dict[str, ExtractSetting] | dict[str, ImgGenSetting] | dict[str, SearchSetting]) -> NoReturn`-- `pipelex/cogt/models/model_deck.py:250` — `ModelDeck.check_llm_choice` — `def check_llm_choice(self, llm_choice: LLMModelChoice, *, is_disabled_allowed: bool=False)`-- `pipelex/cogt/models/model_deck.py:524` — `ModelDeck._validate_llm_setting` — `def _validate_llm_setting(cls, llm_setting: LLMSetting, *, inference_model: InferenceModelSpec)`-- `pipelex/cogt/models/model_deck.py:689` — `ModelDeck._is_model_available_in_backend` — `def _is_model_available_in_backend(self, model_handle: str, *, backend_name: str) -> bool | None`-- `pipelex/cogt/models/model_deck.py:715` — `ModelDeck._resolve_waterfall` — `def _resolve_waterfall(self, waterfall_name: str, *, fallback_list: list[str], model_type: ModelType) -> InferenceModelSpec | None`-- `pipelex/cogt/models/model_deck.py:774` — `ModelDeck.get_optional_inference_model` — `def get_optional_inference_model(self, model_handle: str, *, model_type: ModelType) -> InferenceModelSpec | None`-- `pipelex/cogt/models/model_deck.py:786` — `ModelDeck._get_optional_inference_model` — `def _get_optional_inference_model(self, model_handle: str, *, model_type: ModelType, _visited: frozenset[str]) -> InferenceModelSpec | None`-- `pipelex/cogt/models/model_deck.py:862` — `ModelDeck.is_handle_defined` — `def is_handle_defined(self, model_handle: str, *, model_type: ModelType) -> bool`-- `pipelex/cogt/models/model_deck.py:866` — `ModelDeck.get_required_inference_model` — `def get_required_inference_model(self, model_handle: str, *, model_type: ModelType) -> InferenceModelSpec`-- `pipelex/cogt/models/model_deck_check.py:17` — `_raise_model_choice_not_found` — `def _raise_model_choice_not_found(msg: str, *, model_deck: ModelDeck, model_type: ModelType, raw_choice: str, name: str, reference_kind: ModelReferenceKind, available_options: list[str]) -> NoReturn`-- `pipelex/cogt/models/model_manager.py:100` — `ModelManager._enforce_gateway_model_membership` — `def _enforce_gateway_model_membership(self, gateway_config: GatewayConfig | None, *, gateway_config_source: RemoteConfigSource | None) -> None`-- `pipelex/cogt/models/model_manager.py:193` — `ModelManager._resolve_terminal_candidates` — `def _resolve_terminal_candidates(cls, deck: ModelDeck, *, ref: ModelReference, model_type: ModelType) -> list[str]`-- `pipelex/cogt/models/model_manager.py:226` — `ModelManager._collect_candidates` — `def _collect_candidates(cls, ref: ModelReference, *, aliases: dict[str, str], waterfalls: dict[str, list[str]], is_fallback_enabled: bool, visited: set[tuple[ModelReferenceKind, str]]) -> list[str]`-- `pipelex/cogt/models/model_manager.py:302` — `ModelManager.build_deck` — `def build_deck(self, model_deck_blueprint: ModelDeckBlueprint, *, enabled_backends: list[str]) -> ModelDeck`-- `pipelex/cogt/models/model_manager_abstract.py:22` — `ModelManagerAbstract.setup` — `def setup(self, secrets_provider: SecretsProviderAbstract, *, gateway_config: GatewayConfig | None, gateway_config_source: RemoteConfigSource | None, needs_inference: bool=True) -> None`-- `pipelex/cogt/models/model_manager_abstract.py:33` — `ModelManagerAbstract.get_inference_model` — `def get_inference_model(self, model_handle: str, *, model_type: ModelType) -> InferenceModelSpec`-- `pipelex/cogt/models/model_suggestion.py:33` — `get_collection_keys` — `def get_collection_keys(model_deck: ModelDeck, *, model_type: ModelType, kind: ModelReferenceKind) -> list[str]`-- `pipelex/cogt/models/model_suggestion.py:75` — `suggest_model_alternatives` — `def suggest_model_alternatives(model_deck: ModelDeck, *, model_type: ModelType, name: str, kind: ModelReferenceKind) -> tuple[list[str], list[str], list[str]]`-- `pipelex/cogt/search/search_job_factory.py:8` — `SearchJobFactory.make_search_job` — `def make_search_job(cls, query: str, *, search_setting: SearchSetting, job_metadata: JobMetadata, include_domains: list[str] | None=None, exclude_domains: list[str] | None=None, from_date: str | None=None, to_date: str | None=None) -> SearchJob`-- `pipelex/cogt/search/search_worker_abstract.py:53` — `SearchWorkerAbstract.search_structured` — `async def search_structured(self, search_job: SearchJob, *, schema: type[BaseModelTypeVar]) -> dict[str, Any]`-- `pipelex/cogt/search/search_worker_abstract.py:85` — `SearchWorkerAbstract._search_structured` — `async def _search_structured(self, search_job: SearchJob, *, schema: type[BaseModelTypeVar]) -> dict[str, Any]`-- `pipelex/cogt/templating/template_preprocessor.py:91` — `_validate_at_sigil_alone_on_line` — `def _validate_at_sigil_alone_on_line(template: str, *, declared_inputs: set[str]) -> None`-- `pipelex/cogt/templating/template_preprocessor.py:145` — `validate_template_sigils` — `def validate_template_sigils(template: str, *, declared_inputs: set[str]) -> None`-- `pipelex/cogt/templating/template_preprocessor.py:176` — `preprocess_template` — `def preprocess_template(template: str, *, declared_inputs: set[str] | None=None) -> str`-- `pipelex/cogt/templating/template_rendering.py:10` — `render_template` — `async def render_template(template: str, *, category: TemplateCategory, context: dict[str, Any], templating_style: TemplatingStyle | None=None, finalize: Callable[[Any], Any] | None=None) -> str`-- `pipelex/cogt/usage/cost_registry.py:64` — `CostRegistry.generate_report` — `def generate_report(cls, pipeline_run_id: str, *, tokens_usages: Sequence[TokensUsage], unit_scale: float, cost_report_file_path: Path | None=None, print_to_console: bool=True)`-- `pipelex/cogt/usage/cost_registry.py:88` — `CostRegistry.render_report` — `def render_report(cls, aggregated: AggregatedCosts, *, pipeline_run_id: str, unit_scale: float, cost_report_file_path: Path | None=None, print_to_console: bool=True) -> None`-- `pipelex/cogt/usage/cost_registry.py:195` — `CostRegistry.save_to_csv` — `def save_to_csv(records: list[dict[str, Any]], *, file_path: Path) -> None`-- `pipelex/cogt/usage/costs_per_token.py:4` — `model_cost_per_token` — `def model_cost_per_token(costs: CostsByCategoryDict, *, cost_category: CostCategory) -> float`--### `core` (95)--- `pipelex/core/bundles/pipelex_bundle_blueprint.py:201` — `PipelexBundleBlueprint._collect_local_refs_from_concept` — `def _collect_local_refs_from_concept(self, concept_code: str, *, concept_blueprint: ConceptBlueprint | str) -> list[tuple[str, str]]`-- `pipelex/core/bundles/pipelex_bundle_blueprint.py:224` — `PipelexBundleBlueprint._collect_local_refs_from_pipe` — `def _collect_local_refs_from_pipe(self, pipe_code: str, *, pipe_blueprint: PipeBlueprintUnion) -> list[tuple[str, str]]`-- `pipelex/core/concepts/concept.py:194` — `Concept.render_concept_representation` — `def render_concept_representation(self, output_format: ConceptRepresentationFormat, *, is_multiple: bool=False) -> tuple[dict[str, Any], set[str]]`-- `pipelex/core/concepts/concept_factory.py:88` — `ConceptFactory.make` — `def make(cls, concept_code: str, *, domain_code: str, description: str, structure_class_name: str, refines: str | None=None) -> Concept`-- `pipelex/core/concepts/concept_factory.py:187` — `ConceptFactory.make_domain_and_concept_code_from_concept_ref_or_code` — `def make_domain_and_concept_code_from_concept_ref_or_code(cls, concept_ref_or_code: str, *, domain_code: str | None=None) -> DomainAndConceptCode`-- `pipelex/core/concepts/concept_factory.py:243` — `ConceptFactory.make_refine` — `def make_refine(cls, refine: str, *, domain_code: str) -> str`-- `pipelex/core/concepts/concept_factory.py:274` — `ConceptFactory._handle_structure_with_classname` — `def _handle_structure_with_classname(cls, blueprint: ConceptBlueprint, *, concept_code: str, domain_code: str) -> str`-- `pipelex/core/concepts/concept_factory.py:301` — `ConceptFactory._handle_blueprint_with_structure` — `def _handle_blueprint_with_structure(cls, blueprint: ConceptBlueprint, *, concept_code: str, domain_code: str) -> str`-- `pipelex/core/concepts/concept_factory.py:345` — `ConceptFactory._handle_basic_blueprint` — `def _handle_basic_blueprint(cls, concept_code: str, *, domain_code: str, description: str) -> StructureNameAndRefine`-- `pipelex/core/concepts/concept_factory.py:386` — `ConceptFactory._handle_refines` — `def _handle_refines(cls, blueprint: ConceptBlueprint, *, concept_code: str, domain_code: str) -> StructureNameAndRefine`-- `pipelex/core/concepts/concept_representation_generator.py:50` — `ConceptRepresentationGenerator.generate_representation` — `def generate_representation(self, concept_ref: str, *, structure_class: type[StuffContent], include_optional: bool=True) -> dict[str, Any]`-- `pipelex/core/concepts/concept_representation_generator.py:75` — `ConceptRepresentationGenerator.generate_class_representation` — `def generate_class_representation(self, content_class: type[StuffContent], *, include_optional: bool=True) -> dict[str, Any] | str`-- `pipelex/core/concepts/concept_representation_generator.py:107` — `ConceptRepresentationGenerator._generate_fields_dict` — `def _generate_fields_dict(self, content_class: type[StuffContent], *, include_optional: bool=True) -> dict[str, Any]`-- `pipelex/core/concepts/concept_representation_generator.py:135` — `ConceptRepresentationGenerator.generate_field_value` — `def generate_field_value(self, field_type: Any, *, field_name: str) -> Any`-- `pipelex/core/concepts/concept_representation_generator.py:199` — `ConceptRepresentationGenerator._generate_list_value` — `def _generate_list_value(self, args: tuple[Any, ...], *, field_name: str) -> list[Any]`-- `pipelex/core/concepts/concept_representation_generator.py:238` — `ConceptRepresentationGenerator._generate_literal_value` — `def _generate_literal_value(self, literal_args: tuple[Any, ...], *, field_name: str) -> Any`-- `pipelex/core/concepts/concept_representation_generator.py:253` — `ConceptRepresentationGenerator._generate_enum_value` — `def _generate_enum_value(self, enum_type: type[StrEnum], *, field_name: str) -> str`-- `pipelex/core/concepts/concept_representation_generator.py:293` — `ConceptRepresentationGenerator._generate_basic_value` — `def _generate_basic_value(self, actual_type: Any, *, field_name: str) -> Any`-- `pipelex/core/concepts/concept_representation_generator.py:329` — `ConceptRepresentationGenerator._format_as_python` — `def _format_as_python(self, class_name: str, *, fields: dict[str, Any]) -> str`-- `pipelex/core/concepts/concept_representation_generator.py:394` — `generate_json_representation` — `def generate_json_representation(concept_ref: str, *, structure_class: type[StuffContent]) -> dict[str, Any]`-- `pipelex/core/concepts/concept_representation_generator.py:412` — `generate_python_representation` — `def generate_python_representation(concept_ref: str, *, structure_class: type[StuffContent]) -> tuple[dict[str, Any], set[str]]`-- `pipelex/core/concepts/concept_structure_blueprint.py:188` — `ConceptStructureBlueprint._raise_type_mismatch_error` — `def _raise_type_mismatch_error(self, expected_type_name: str, *, actual_type_name: str) -> None`-- `pipelex/core/concepts/helpers.py:13` — `get_structure_class_name_from_blueprint` — `def get_structure_class_name_from_blueprint(blueprint_or_string_description: ConceptBlueprint | str, *, concept_ref_or_code: str) -> str`-- `pipelex/core/concepts/structure_generation/generator.py:144` — `StructureGenerator.validate_generated_code` — `def validate_generated_code(self, python_code: str, *, expected_class_name: str, base_class_name: str | None=None) -> type`-- `pipelex/core/concepts/structure_generation/generator.py:211` — `StructureGenerator._format_class_docstring` — `def _format_class_docstring(self, docstring: str, *, indent: str=' ') -> str`-- `pipelex/core/concepts/structure_generation/generator.py:249` — `StructureGenerator._generate_class_source_code_from_blueprint` — `def _generate_class_source_code_from_blueprint(self, class_name: str, *, structure_blueprint: dict[str, ConceptStructureBlueprint], base_class_name: str | None=None, description: str='') -> str`-- `pipelex/core/concepts/structure_generation/generator.py:311` — `StructureGenerator._generate_field_from_blueprint` — `def _generate_field_from_blueprint(self, field_name: str, *, field_blueprint: ConceptStructureBlueprint) -> str`-- `pipelex/core/concepts/structure_generation/generator.py:480` — `StructureGenerator._generate_field` — `def _generate_field(self, field_name: str, *, field_def: dict[str, Any] | str) -> str`-- `pipelex/core/concepts/structure_generation/generator.py:530` — `StructureGenerator._get_python_type` — `def _get_python_type(self, field_type: Any, *, field_def: dict[str, Any]) -> str`-- `pipelex/core/concepts/structure_generation/generator.py:614` — `StructureGenerator._validate_execution` — `def _validate_execution(self, python_code: str, *, expected_class_name: str, base_class_name: str | None=None) -> type`-- `pipelex/core/interpreter/bundle_elaborator.py:105` — `BundleElaborator._elaborate_preliminary_text` — `def _elaborate_preliminary_text(cls, pipe_code: str, *, pipe_blueprint: PipeLLMBlueprint, new_pipe_dict: dict[str, PipeBlueprintUnion], elaboration_metadata: dict[str, ElaborationMetadata], existing_codes: set[str]) -> None`-- `pipelex/core/interpreter/interpreter.py:22` — `PipelexInterpreter.make_pipelex_bundle_blueprint` — `def make_pipelex_bundle_blueprint(cls, bundle_path: Path | None=None, *, mthds_content: str | None=None) -> PipelexBundleBlueprint`-- `pipelex/core/interpreter/validation_error_categorizer.py:27` — `_categorize_input_validation_error` — `def _categorize_input_validation_error(message: str, *, domain: str | None, source: str | None, pipe_code: str | None) -> PipelexBundleBlueprintValidationErrorData | None`-- `pipelex/core/interpreter/validation_error_categorizer.py:74` — `_categorize_syntax_validation_error` — `def _categorize_syntax_validation_error(message: str, *, domain: str | None, source: str | None) -> PipelexBundleBlueprintValidationErrorData | None`-- `pipelex/core/interpreter/validation_error_categorizer.py:127` — `_categorize_concept_validation_error` — `def _categorize_concept_validation_error(loc: tuple[int | str, ...], *, message: str, domain: str | None, source: str | None) -> PipelexBundleBlueprintValidationErrorData | None`-- `pipelex/core/interpreter/validation_error_categorizer.py:172` — `categorize_blueprint_validation_error` — `def categorize_blueprint_validation_error(error: ErrorDetails, *, blueprint_dict: dict[str, Any]) -> PipelexBundleBlueprintValidationErrorData | None`-- `pipelex/core/memory/working_memory.py:146` — `WorkingMemory.set_new_main_stuff` — `def set_new_main_stuff(self, stuff: Stuff, *, name: str | None=None)`-- `pipelex/core/memory/working_memory.py:157` — `WorkingMemory.set_alias` — `def set_alias(self, alias: str, *, target: str) -> None`-- `pipelex/core/memory/working_memory.py:167` — `WorkingMemory.add_alias` — `def add_alias(self, alias: str, *, target: str) -> None`-- `pipelex/core/memory/working_memory.py:329` — `WorkingMemory._get_typed_items_from_list_content` — `def _get_typed_items_from_list_content(self, list_content: ListContent[Any], *, wanted_type: type[Any] | None) -> list[Any] | None`-- `pipelex/core/memory/working_memory.py:370` — `WorkingMemory.get_stuff_as` — `def get_stuff_as(self, name: str, *, content_type: type[StuffContentType]) -> StuffContentType`-- `pipelex/core/memory/working_memory.py:374` — `WorkingMemory.get_stuff_as_list` — `def get_stuff_as_list(self, name: str, *, item_type: type[StuffContentType]) -> ListContent[StuffContentType]`-- `pipelex/core/memory/working_memory.py:380` — `WorkingMemory.get_list_stuff_first_item_as` — `def get_list_stuff_first_item_as(self, name: str, *, item_type: type[StuffContentType]) -> StuffContentType`-- `pipelex/core/memory/working_memory_factory.py:39` — `WorkingMemoryFactory.make_from_multiple_stuffs` — `def make_from_multiple_stuffs(cls, stuff_list: list[Stuff], *, main_name: str | None=None, is_ignore_unnamed: bool=False) -> WorkingMemory`-- `pipelex/core/memory/working_memory_factory.py:68` — `WorkingMemoryFactory.make_from_pipeline_inputs` — `def make_from_pipeline_inputs(cls, pipeline_inputs: PipelineInputs, *, search_domain_codes: list[str] | None=None) -> WorkingMemory`-- `pipelex/core/memory/working_memory_factory.py:150` — `WorkingMemoryFactory.convert_stuff_spec_to_typed_named` — `def convert_stuff_spec_to_typed_named(cls, stuff_spec: StuffSpec, *, name: str) -> TypedNamedStuffSpec`-- `pipelex/core/pipes/handle_pipe_errors.py:119` — `_handle_pipe_errors` — `def _handle_pipe_errors(error: ErrorDetails, *, pipe_code: str | None) -> PipesAndConceptValidationErrorData`-- `pipelex/core/pipes/inputs/input_renderer.py:5` — `render_inputs` — `def render_inputs(the_pipe: PipeAbstract, *, indent: int=2) -> str`-- `pipelex/core/pipes/inputs/input_stuff_specs.py:26` — `TypedNamedStuffSpec.make_from_named` — `def make_from_named(cls, named: NamedStuffSpec, *, structure_class: type[StuffContent]) -> 'TypedNamedStuffSpec'`-- `pipelex/core/pipes/inputs/input_stuff_specs.py:84` — `InputStuffSpecs.add_stuff_spec` — `def add_stuff_spec(self, variable_name: str, *, concept: Concept, multiplicity: VariableMultiplicity | None=None)`-- `pipelex/core/pipes/output/output_renderer.py:16` — `_collect_possible_outputs` — `def _collect_possible_outputs(the_pipe: PipeAbstract, *, output_format: ConceptRepresentationFormat=ConceptRepresentationFormat.JSON) -> list[dict[str, Any]]`-- `pipelex/core/pipes/output/output_renderer.py:119` — `render_output` — `def render_output(the_pipe: PipeAbstract, *, indent: int=2, output_format: ConceptRepresentationFormat=ConceptRepresentationFormat.JSON) -> str`-- `pipelex/core/pipes/output/output_renderer.py:154` — `_render_json_output` — `def _render_json_output(the_pipe: PipeAbstract, *, indent: int=2) -> str`-- `pipelex/core/pipes/output/output_renderer.py:240` — `_render_schema_output` — `def _render_schema_output(the_pipe: PipeAbstract, *, indent: int=2) -> str`-- `pipelex/core/pipes/pipe_abstract.py:103` — `PipeAbstract._register_execution_data` — `def _register_execution_data(self, job_metadata: JobMetadata, *, execution_data: dict[str, Any]) -> None`-- `pipelex/core/pipes/pipe_abstract.py:316` — `PipeAbstract.validate_before_run` — `async def validate_before_run(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None)`-- `pipelex/core/pipes/pipe_abstract.py:369` — `PipeAbstract._validate_before_run` — `async def _validate_before_run(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None)`-- `pipelex/core/pipes/pipe_abstract.py:380` — `PipeAbstract.validate_after_run` — `async def validate_after_run(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None)`-- `pipelex/core/pipes/pipe_abstract.py:393` — `PipeAbstract._validate_after_run` — `async def _validate_after_run(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None)`-- `pipelex/core/pipes/pipe_abstract.py:441` — `PipeAbstract.run_pipe` — `async def run_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`-- `pipelex/core/pipes/pipe_abstract.py:467` — `PipeAbstract._run_pipe_traced` — `async def _run_pipe_traced(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`-- `pipelex/core/pipes/pipe_abstract.py:630` — `PipeAbstract.live_run_pipe` — `async def live_run_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`-- `pipelex/core/pipes/pipe_abstract.py:703` — `PipeAbstract.dry_run_pipe` — `async def dry_run_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`-- `pipelex/core/pipes/pipe_abstract.py:723` — `PipeAbstract._live_run_pipe` — `async def _live_run_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`-- `pipelex/core/pipes/pipe_abstract.py:735` — `PipeAbstract._dry_run_pipe` — `async def _dry_run_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`-- `pipelex/core/pipes/pipe_abstract.py:746` — `PipeAbstract._start_pipe_span` — `def _start_pipe_span(self, parent_otel_context: OtelContext, *, pipeline_run_id: str, working_memory: WorkingMemory) -> tuple[Span | None, bool]`-- `pipelex/core/pipes/pipe_abstract.py:862` — `PipeAbstract._end_pipe_span_success` — `def _end_pipe_span_success(self, span: Span | None, *, pipe_output: PipeOutput, is_root_span: bool) -> None`-- `pipelex/core/pipes/pipe_abstract.py:896` — `PipeAbstract._end_pipe_span_error` — `def _end_pipe_span_error(self, span: Span | None, *, error: Exception, is_root_span: bool=False) -> None`-- `pipelex/core/pipes/pipe_factory.py:27` — `PipeFactoryProtocol.make` — `def make(cls, pipe_category: Any, *, pipe_type: str, pipe_code: str, domain_code: str, description: str, inputs: InputStuffSpecs, output: StuffSpec, blueprint: PipeBlueprintType) -> PipeAbstractType`-- `pipelex/core/pipes/validation.py:4` — `is_variable_satisfied_by_inputs` — `def is_variable_satisfied_by_inputs(variable_path: str, *, input_names: set[str]) -> bool`-- `pipelex/core/pipes/validation.py:32` — `is_input_used_by_variables` — `def is_input_used_by_variables(input_name: str, *, variable_paths: set[str]) -> bool`-- `pipelex/core/pipes/variable_multiplicity.py:170` — `format_concept_with_multiplicity` — `def format_concept_with_multiplicity(concept_code_or_string: str, *, multiplicity: VariableMultiplicity | None) -> str`-- `pipelex/core/qualified_ref.py:136` — `QualifiedRef.from_domain_and_code` — `def from_domain_and_code(cls, domain_path: str, *, local_code: str) -> 'QualifiedRef'`-- `pipelex/core/stuffs/image_content.py:71` — `ImageContent.render_with_images` — `def render_with_images(self, registry: ImageRegistry, *, text_format: TextFormat) -> str`-- `pipelex/core/stuffs/image_field_search.py:10` — `search_for_nested_image_fields` — `def search_for_nested_image_fields(content_class: type[StuffContent], *, current_path: str='') -> list[str]`-- `pipelex/core/stuffs/list_content.py:141` — `ListContent.render_with_images` — `def render_with_images(self, registry: ImageRegistry, *, text_format: TextFormat) -> str`-- `pipelex/core/stuffs/structured_content.py:121` — `StructuredContent.render_with_images` — `def render_with_images(self, registry: ImageRegistry, *, text_format: TextFormat) -> str`-- `pipelex/core/stuffs/structured_content.py:150` — `StructuredContent._render_value_with_images` — `def _render_value_with_images(self, value: Any, *, registry: ImageRegistry, text_format: TextFormat) -> str`-- `pipelex/core/stuffs/stuff.py:100` — `Stuff.verify_content_type` — `def verify_content_type(cls, content: StuffContent, *, content_type: type[StuffContentType]) -> StuffContentType`-- `pipelex/core/stuffs/stuff_artefact.py:195` — `StuffArtefact.get` — `def get(self, key: str, *, default: Any=None) -> Any`-- `pipelex/core/stuffs/stuff_artefact.py:350` — `StuffArtefact.render_with_images` — `def render_with_images(self, registry: ImageRegistry, *, text_format: TextFormat) -> str`-- `pipelex/core/stuffs/stuff_content.py:64` — `StuffContent.rendered_markdown` — `def rendered_markdown(self, level: int=1, *, is_pretty: bool=False) -> str`-- `pipelex/core/stuffs/stuff_content.py:113` — `StuffContent.rendered_markdown_async` — `async def rendered_markdown_async(self, level: int=1, *, is_pretty: bool=False) -> str`-- `pipelex/core/stuffs/stuff_content_factory.py:12` — `StuffContentFactory.make_content_from_value` — `def make_content_from_value(cls, stuff_content_subclass: type[StuffContent], *, value: dict[str, Any] | str) -> StuffContent`-- `pipelex/core/stuffs/stuff_content_factory.py:18` — `StuffContentFactory.make_stuff_content_from_concept_required` — `def make_stuff_content_from_concept_required(cls, concept: Concept, *, value: dict[str, Any] | str) -> StuffContent`-- `pipelex/core/stuffs/stuff_content_factory.py:27` — `StuffContentFactory.make_stuff_content_from_concept_with_fallback` — `def make_stuff_content_from_concept_with_fallback(cls, concept: Concept, *, value: dict[str, Any] | str) -> StuffContent`-- `pipelex/core/stuffs/stuff_factory.py:51` — `StuffFactory.make_from_str` — `def make_from_str(cls, str_value: str, *, name: str) -> Stuff`-- `pipelex/core/stuffs/stuff_factory.py:59` — `StuffFactory.make_from_concept_ref` — `def make_from_concept_ref(cls, concept_ref: str, *, name: str, content: StuffContent) -> Stuff`-- `pipelex/core/stuffs/stuff_factory.py:69` — `StuffFactory.make_stuff` — `def make_stuff(cls, concept: Concept, *, content: StuffContent, name: str | None=None, code: str | None=None) -> Stuff`-- `pipelex/core/stuffs/stuff_factory.py:115` — `StuffFactory.combine_stuffs` — `def combine_stuffs(cls, stuff_contents: dict[str, StuffContent], *, concept: Concept, name: str | None=None) -> Stuff`-- `pipelex/core/stuffs/stuff_factory.py:136` — `StuffFactory._try_make_csv_list_stuff` — `def _try_make_csv_list_stuff(cls, concept: Concept, *, content: dict[str, Any], name: str | None, code: str | None) -> Stuff | None`-- `pipelex/core/stuffs/stuff_factory.py:227` — `StuffFactory.make_stuff_from_stuff_content_or_data` — `def make_stuff_from_stuff_content_or_data(cls, stuff_content_or_data: StuffContentOrData, *, name: str | None=None, code: str | None=None, search_domain_codes: list[str] | None=None) -> Stuff`-- `pipelex/core/stuffs/stuff_viewer.py:21` — `render_stuff_viewer` — `async def render_stuff_viewer(stuff: Stuff, *, title: str | None=None, subtitle: str | None=None) -> str`-- `pipelex/core/stuffs/stuff_viewer.py:72` — `render_stuff_content_viewer` — `async def render_stuff_content_viewer(stuff_data: str | dict[str, Any] | list[str] | list[dict[str, Any]], *, stuff_data_text: str, stuff_data_html: str, content_type: str | None=None, title: str='Stuff Content', subtitle: str | None=None) -> str`-- `pipelex/core/stuffs/text_and_images_content.py:76` — `TextAndImagesContent.render_with_images` — `def render_with_images(self, registry: ImageRegistry, *, text_format: TextFormat) -> str`--### `errors` (5)--- `pipelex/errors/error_pages_generator.py:253` — `generate_error_pages` — `def generate_error_pages(output_dir: Path, *, classes: Iterable[type[PipelexError]] | None=None) -> ErrorPagesReport`-- `pipelex/errors/error_pages_generator.py:333` — `_remove_orphans` — `def _remove_orphans(output_dir: Path, *, expected_stems: set[str], report: ErrorPagesReport) -> None`-- `pipelex/errors/error_pages_generator.py:355` — `_commit_page` — `def _commit_page(target: Path, *, new_content: str, report: ErrorPagesReport) -> None`-- `pipelex/errors/error_pages_generator.py:448` — `_subsystems_for_macro` — `def _subsystems_for_macro(macro_slug: str, *, by_subsystem: dict[str, list[type[PipelexError]]]) -> list[tuple[str, list[type[PipelexError]]]]`-- `pipelex/errors/error_pages_generator.py:473` — `render_macro_page` — `def render_macro_page(macro_heading: str, *, sections: list[tuple[str, list[type[PipelexError]]]]) -> str`--### `graph` (40)--- `pipelex/graph/graph_factory.py:45` — `generate_graph_outputs` — `async def generate_graph_outputs(graph_spec: GraphSpec, *, graph_config: GraphConfig, pipe_code: str='', title: str | None=None, direction: FlowchartDirection | None=None, include_subgraphs: bool=True) -> GraphOutputs`-- `pipelex/graph/graph_factory.py:135` — `save_graph_outputs_to_dir` — `def save_graph_outputs_to_dir(graph_outputs: GraphOutputs, *, output_dir: Path) -> dict[str, Path]`-- `pipelex/graph/graph_rendering.py:40` — `render_graph_from_spec` — `async def render_graph_from_spec(graph_spec: GraphSpec, *, graph_config: GraphConfig, include_mermaidflow: bool, include_reactflow: bool, output_dir: Path, pipe_code: str='', title: str | None=None, direction: FlowchartDirection | None=None, include_subgraphs: bool=True) -> dict[str, Path]`-- `pipelex/graph/graph_rendering.py:103` — `_dry_run_bundle` — `async def _dry_run_bundle(bundle_path: Path, *, library_dirs: list[str] | None=None) -> tuple[GraphSpec, str]`-- `pipelex/graph/graph_rendering.py:140` — `generate_graph_for_bundle` — `async def generate_graph_for_bundle(bundle_path: Path, *, graph_format: GraphFormat, library_dirs: list[str] | None=None, direction: FlowchartDirection | None=None, graph_name: str='dry_run.html') -> dict[str, Any]`-- `pipelex/graph/graph_rendering.py:215` — `generate_view_for_bundle` — `async def generate_view_for_bundle(bundle_path: Path, *, library_dirs: list[str] | None=None, direction: FlowchartDirection | None=None) -> dict[str, Any]`-- `pipelex/graph/graph_tracer.py:907` — `GraphTracer.add_selected_outcome_edge` — `def add_selected_outcome_edge(self, condition_node_id: str, *, outcome_node_id: str, outcome_value: str) -> None`-- `pipelex/graph/graph_tracer_manager.py:94` — `GraphTracerManager.open_tracer` — `def open_tracer(self, graph_id: str, *, data_inclusion: DataInclusionConfig, pipeline_ref_domain: str | None=None, pipeline_ref_main_pipe: str | None=None, event_log: 'EventLogProtocol | None'=None, workflow_id: str='direct', pipeline_run_id: str | None=None, tracer_key: str | None=None, emit_graph_events: bool=True, emit_usage_events: bool=True) -> TraceContext`-- `pipelex/graph/graph_tracer_manager.py:218` — `GraphTracerManager.on_pipe_start` — `def on_pipe_start(self, trace_context: TraceContext, *, pipe_code: str, pipe_type: str, node_kind: NodeKind, started_at: datetime, input_specs: list[IOSpec] | None=None, pipe_data: dict[str, Any] | None=None, concept_data: list[dict[str, Any]] | None=None, description: str | None=None, domain_code: str | None=None) -> tuple[str | None, TraceContext | None]`-- `pipelex/graph/graph_tracer_manager.py:266` — `GraphTracerManager.on_pipe_end_success` — `def on_pipe_end_success(self, lookup_key: str, *, node_id: str | None, ended_at: datetime, output_preview: str | None=None, metrics: dict[str, float] | None=None, output_spec: IOSpec | None=None, output_concept_data: dict[str, Any] | None=None) -> None`-- `pipelex/graph/graph_tracer_manager.py:304` — `GraphTracerManager.register_execution_data` — `def register_execution_data(self, lookup_key: str, *, node_id: str | None, execution_data: dict[str, Any]) -> None`-- `pipelex/graph/graph_tracer_manager.py:325` — `GraphTracerManager.on_pipe_end_error` — `def on_pipe_end_error(self, lookup_key: str, *, node_id: str | None, ended_at: datetime, error_type: str, error_message: str, error_stack: str | None=None) -> None`-- `pipelex/graph/graph_tracer_manager.py:360` — `GraphTracerManager.add_edge` — `def add_edge(self, lookup_key: str, *, source_node_id: str, target_node_id: str, edge_kind: EdgeKind, label: str | None=None) -> None`-- `pipelex/graph/graph_tracer_manager.py:389` — `GraphTracerManager.register_controller_output` — `def register_controller_output(self, lookup_key: str, *, node_id: str, output_spec: IOSpec) -> None`-- `pipelex/graph/graph_tracer_manager.py:411` — `GraphTracerManager.register_batch_item_extraction` — `def register_batch_item_extraction(self, lookup_key: str, *, list_stuff_code: str, item_stuff_code: str, item_index: int, batch_controller_node_id: str | None=None) -> None`-- `pipelex/graph/graph_tracer_manager.py:440` — `GraphTracerManager.register_batch_aggregation` — `def register_batch_aggregation(self, lookup_key: str, *, output_list_stuff_code: str, item_stuff_code: str, item_index: int, batch_controller_node_id: str | None=None) -> None`-- `pipelex/graph/graph_tracer_manager.py:469` — `GraphTracerManager.register_parallel_combine` — `def register_parallel_combine(self, lookup_key: str, *, combined_stuff_code: str, branch_stuff_codes: list[str], parallel_controller_node_id: str) -> None`-- `pipelex/graph/graph_tracer_protocol.py:19` — `GraphTracerProtocol.setup` — `def setup(self, graph_id: str, *, data_inclusion: DataInclusionConfig, pipeline_ref_domain: str | None=None, pipeline_ref_main_pipe: str | None=None, event_log: 'EventLogProtocol | None'=None, workflow_id: str='direct', pipeline_run_id: str | None=None, emit_graph_events: bool=True, emit_usage_events: bool=True) -> TraceContext`-- `pipelex/graph/graph_tracer_protocol.py:60` — `GraphTracerProtocol.on_pipe_start` — `def on_pipe_start(self, trace_context: TraceContext, *, pipe_code: str, pipe_type: str, node_kind: NodeKind, started_at: datetime, input_specs: list[IOSpec] | None=None, pipe_data: dict[str, Any] | None=None, concept_data: list[dict[str, Any]] | None=None, description: str | None=None, domain_code: str | None=None) -> tuple[str, TraceContext]`-- `pipelex/graph/graph_tracer_protocol.py:93` — `GraphTracerProtocol.on_pipe_end_success` — `def on_pipe_end_success(self, node_id: str, *, ended_at: datetime, output_preview: str | None=None, metrics: dict[str, float] | None=None, output_spec: IOSpec | None=None, output_concept_data: dict[str, Any] | None=None) -> None`-- `pipelex/graph/graph_tracer_protocol.py:115` — `GraphTracerProtocol.on_pipe_end_error` — `def on_pipe_end_error(self, node_id: str, *, ended_at: datetime, error_type: str, error_message: str, error_stack: str | None=None) -> None`-- `pipelex/graph/graph_tracer_protocol.py:157` — `GraphTracerProtocol.register_controller_output` — `def register_controller_output(self, node_id: str, *, output_spec: IOSpec) -> None`-- `pipelex/graph/graph_tracer_protocol.py:174` — `GraphTracerProtocol.register_batch_item_extraction` — `def register_batch_item_extraction(self, list_stuff_code: str, *, item_stuff_code: str, item_index: int, batch_controller_node_id: str | None=None) -> None`-- `pipelex/graph/graph_tracer_protocol.py:193` — `GraphTracerProtocol.register_batch_aggregation` — `def register_batch_aggregation(self, output_list_stuff_code: str, *, item_stuff_code: str, item_index: int, batch_controller_node_id: str | None=None) -> None`-- `pipelex/graph/graph_tracer_protocol.py:213` — `GraphTracerProtocol.register_parallel_combine` — `def register_parallel_combine(self, combined_stuff_code: str, *, branch_stuff_codes: list[str], parallel_controller_node_id: str) -> None`-- `pipelex/graph/graph_tracer_protocol.py:232` — `GraphTracerProtocol.register_execution_data` — `def register_execution_data(self, node_id: str, *, execution_data: dict[str, Any]) -> None`-- `pipelex/graph/graphspec.py:128` — `_truncate_string` — `def _truncate_string(value: str | None, *, max_length: int) -> str | None`-- `pipelex/graph/mermaidflow/mermaid_html.py:21` — `render_mermaid_html` — `def render_mermaid_html(mermaid_code: str, *, title: str='Pipelex Graph', theme: str='dark') -> str`-- `pipelex/graph/mermaidflow/mermaid_html.py:54` — `render_mermaid_html_async` — `async def render_mermaid_html_async(mermaid_code: str, *, title: str='Pipelex Graph', theme: str='dark') -> str`-- `pipelex/graph/mermaidflow/mermaid_html.py:86` — `render_mermaid_html_with_data_async` — `async def render_mermaid_html_with_data_async(mermaid_code: str, *, stuff_data: dict[str, str | dict[str, object] | list[str] | list[dict[str, object]] | None] | None=None, stuff_data_text: dict[str, str] | None=None, stuff_data_html: dict[str, str] | None=None, stuff_metadata: dict[str, dict[str, str]] | None=None, stuff_content_type: dict[str, str] | None=None, title: str='Pipelex Graph', theme: str='dark') -> str`-- `pipelex/graph/mermaidflow/mermaidflow_factory.py:53` — `MermaidflowFactory.make_from_graphspec` — `def make_from_graphspec(cls, graph: GraphSpec, *, graph_config: GraphConfig, direction: FlowchartDirection | None=None, show_stuff_codes: bool=False, include_subgraphs: bool=True) -> Mermaidflow`-- `pipelex/graph/mermaidflow/mermaidflow_factory.py:349` — `MermaidflowFactory._render_node` — `def _render_node(cls, node: NodeSpec, *, mermaid_id: str, indent: str=' ') -> str`-- `pipelex/graph/mermaidflow/mermaidflow_factory.py:389` — `MermaidflowFactory._render_stuff_node` — `def _render_stuff_node(cls, digest: str, *, name: str, concept: str | None, stuff_id_mapping: dict[str, str], show_stuff_codes: bool, indent: str=' ') -> str`-- `pipelex/graph/mermaidflow/mermaidflow_factory.py:427` — `MermaidflowFactory._render_dashed_edges` — `def _render_dashed_edges(cls, edges: list[EdgeSpec], *, lines: list[str], stuff_id_mapping: dict[str, str], all_stuff_info: dict[str, tuple[str, str | None]], show_stuff_codes: bool) -> None`-- `pipelex/graph/mermaidflow/mermaidflow_factory.py:487` — `MermaidflowFactory._render_subgraph_recursive` — `def _render_subgraph_recursive(cls, node_id: str, *, nodes_by_id: dict[str, NodeSpec], id_mapping: dict[str, str], children_map: dict[str, list[str]], stuff_registry: dict[str, tuple[str, str | None]], stuff_producers: dict[str, str], stuff_consumers: dict[str, list[str]], stuff_id_mapping: dict[str, str], subgraph_depths: dict[str, int], show_stuff_codes: bool, rendered_orphan_stuffs: set[str], controller_output_stuffs: dict[str, dict[str, tuple[str, str | None]]], indent_level: int=1, depth: int=0) -> list[str]`-- `pipelex/graph/mermaidflow/stuff_collector.py:18` — `_collect_stuff_field` — `def _collect_stuff_field(graph: GraphSpec, *, extractor: Callable[[IOSpec], T | None]) -> dict[str, T]`-- `pipelex/graph/reactflow/reactflow_html.py:34` — `_build_templating_context` — `def _build_templating_context(graphspec: GraphSpec, *, config: ReactFlowRenderingConfig, title: str | None) -> dict[str, object]`-- `pipelex/graph/reactflow/reactflow_html.py:51` — `generate_reactflow_html` — `def generate_reactflow_html(graphspec: GraphSpec, *, config: ReactFlowRenderingConfig, title: str | None=None) -> str`-- `pipelex/graph/reactflow/reactflow_html.py:75` — `generate_reactflow_html_async` — `async def generate_reactflow_html_async(graphspec: GraphSpec, *, config: ReactFlowRenderingConfig, title: str | None=None) -> str`-- `pipelex/graph/trace_context.py:66` — `TraceContext.copy_for_child` — `def copy_for_child(self, child_node_id: str, *, next_sequence: int) -> 'TraceContext'`--### `hub.py` (1)--- `pipelex/hub.py:121` — `PipelexHub.setup_config` — `def setup_config(self, config_cls: type[ConfigRoot], *, config_overrides: dict[str, Any] | None=None, config_dir: Path | None=None)`--### `kit` (8)--- `pipelex/kit/cursor_rules.py:34` — `_front_matter_for` — `def _front_matter_for(name: str, *, kit_index: KitIndex) -> dict[str, Any]`-- `pipelex/kit/cursor_rules.py:77` — `update_cursor_rules` — `def update_cursor_rules(repo_root: Path, *, kit_index: KitIndex, agent_set: str) -> None`-- `pipelex/kit/cursor_rules.py:113` — `remove_cursor_rules` — `def remove_cursor_rules(repo_root: Path, *, kit_index: KitIndex | None=None) -> None`-- `pipelex/kit/single_file_agent_rules.py:12` — `_read_agent_file` — `def _read_agent_file(agents_dir: Traversable, *, name: str) -> str`-- `pipelex/kit/single_file_agent_rules.py:25` — `_demote_headings` — `def _demote_headings(md_content: str, *, levels: int) -> str`-- `pipelex/kit/single_file_agent_rules.py:49` — `build_merged_rules` — `def build_merged_rules(kit_index: KitIndex, *, agent_set: str | None=None, file_list: list[str] | None=None) -> str`-- `pipelex/kit/single_file_agent_rules.py:107` — `update_single_file_agent_rules` — `def update_single_file_agent_rules(repo_root: Path, *, kit_index: KitIndex, agent_set: str, targets: dict[str, Target]) -> None`-- `pipelex/kit/single_file_agent_rules.py:145` — `remove_from_targets` — `def remove_from_targets(repo_root: Path, *, targets: dict[str, Target]) -> None`--### `language` (5)--- `pipelex/language/mthds_factory.py:52` — `MthdsFactory.convert_dicts_to_inline_tables` — `def convert_dicts_to_inline_tables(cls, value: Any, *, field_ordering: list[str] | None=None) -> Any`-- `pipelex/language/mthds_factory.py:91` — `MthdsFactory.convert_mapping_to_table` — `def convert_mapping_to_table(cls, mapping: Mapping[str, Any], *, field_ordering: list[str] | None=None) -> Any`-- `pipelex/language/mthds_schema_generator.py:89` — `_remove_from_required` — `def _remove_from_required(schema_obj: dict[str, Any], *, field_names: set[str]) -> None`-- `pipelex/language/mthds_schema_generator.py:310` — `_walk_schema` — `def _walk_schema(node: dict[str, Any] | list[Any] | Any, *, visitor: Callable[[dict[str, Any]], None]) -> None`-- `pipelex/language/toml_string_utils.py:14` — `format_toml_string` — `def format_toml_string(text: str, *, force_multiline: bool=False, length_limit_to_multiline: int=100, ensure_trailing_newline: bool=True, ensure_leading_blank_line: bool=True, prefer_literal: bool=False) -> Any`--### `libraries` (28)--- `pipelex/libraries/concept/concept_library.py:191` — `ConceptLibrary.add_dependency_concept` — `def add_dependency_concept(self, alias: str, *, concept: Concept) -> None`-- `pipelex/libraries/concept/concept_library_abstract.py:33` — `ConceptLibraryAbstract.is_compatible` — `def is_compatible(self, tested_concept: Concept, *, wanted_concept: Concept, strict: bool=False) -> bool`-- `pipelex/libraries/concept/concept_library_abstract.py:53` — `ConceptLibraryAbstract.get_required_concept_from_concept_ref_or_code` — `def get_required_concept_from_concept_ref_or_code(self, concept_ref_or_code: str, *, search_domain_codes: list[str] | None=None) -> Concept`-- `pipelex/libraries/concept_reference_validation.py:8` — `validate_concept_references_in_blueprints` — `def validate_concept_references_in_blueprints(blueprints: list[PipelexBundleBlueprint], *, already_loaded_concept_refs: set[str] | None=None) -> None`-- `pipelex/libraries/contract_match.py:24` — `_canonical_concept_spec` — `def _canonical_concept_spec(spec: str, *, domain_code: str) -> str`-- `pipelex/libraries/library_crate.py:39` — `LibraryCrate.compute_fingerprint_from_content` — `def compute_fingerprint_from_content(concepts: dict[str, 'ConceptBlueprint | str'], *, pipes: dict[str, PipeBlueprintUnion]) -> str`-- `pipelex/libraries/library_crate_factory.py:150` — `LibraryCrateFactory._reconcile_pipe_collision` — `def _reconcile_pipe_collision(cls, pipe_ref: str, *, existing: PipeDeclaration, incoming: PipeDeclaration, domain_code: str) -> PipeDeclaration`-- `pipelex/libraries/library_crate_factory.py:204` — `LibraryCrateFactory._contract_mismatch_msg` — `def _contract_mismatch_msg(pipe_ref: str, *, existing: PipeDeclaration, incoming: PipeDeclaration) -> str`-- `pipelex/libraries/library_manager.py:764` — `LibraryManager._load_mthds_files_into_library` — `def _load_mthds_files_into_library(self, library_id: str, *, valid_mthds_paths: list[Path]) -> list[PipeAbstract]`-- `pipelex/libraries/library_manager.py:839` — `LibraryManager._warn_if_mthds_version_unsatisfied` — `def _warn_if_mthds_version_unsatisfied(self, mthds_version_constraint: str, *, package_address: str) -> None`-- `pipelex/libraries/library_manager.py:860` — `LibraryManager._check_package_visibility` — `def _check_package_visibility(self, blueprints: list[PipelexBundleBlueprint], *, mthds_paths: list[Path]) -> MethodsManifest | None`-- `pipelex/libraries/library_manager.py:945` — `LibraryManager._load_dependency_packages` — `def _load_dependency_packages(self, library_id: str, *, manifest: MethodsManifest, package_root: Path) -> None`-- `pipelex/libraries/library_manager.py:981` — `LibraryManager._load_single_dependency` — `def _load_single_dependency(self, library: Library, *, resolved_dep: ResolvedDependency) -> None`-- `pipelex/libraries/library_manager.py:1142` — `LibraryManager._load_address_based_dependencies` — `def _load_address_based_dependencies(self, library_id: str, *, blueprints: list[PipelexBundleBlueprint]) -> None`-- `pipelex/libraries/library_manager.py:1193` — `LibraryManager._load_address_based_dependency` — `def _load_address_based_dependency(self, library: 'Library', *, full_address: str, extra_search_dirs: list[Path] | None=None) -> bool`-- `pipelex/libraries/library_manager.py:1357` — `LibraryManager._detect_concept_cycles.check_for_cycle` — `def check_for_cycle(concept_ref: str, *, visiting: set[str], path: list[str]) -> None`-- `pipelex/libraries/library_manager_abstract.py:84` — `LibraryManagerAbstract.load_from_crate` — `def load_from_crate(self, library_id: str, *, crate: LibraryCrate) -> list[PipeAbstract]`-- `pipelex/libraries/library_manager_abstract.py:96` — `LibraryManagerAbstract.load_from_blueprints` — `def load_from_blueprints(self, library_id: str, *, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]`-- `pipelex/libraries/library_manager_abstract.py:100` — `LibraryManagerAbstract.load_concepts_only_from_blueprints` — `def load_concepts_only_from_blueprints(self, library_id: str, *, blueprints: list[PipelexBundleBlueprint]) -> list['Concept']`-- `pipelex/libraries/library_manager_abstract.py:116` — `LibraryManagerAbstract._remove_from_blueprint` — `def _remove_from_blueprint(self, library_id: str, *, blueprint: PipelexBundleBlueprint) -> None`-- `pipelex/libraries/library_manager_abstract.py:120` — `LibraryManagerAbstract._remove_from_blueprints` — `def _remove_from_blueprints(self, library_id: str, *, blueprints: list[PipelexBundleBlueprint]) -> None`-- `pipelex/libraries/library_manager_abstract.py:124` — `LibraryManagerAbstract.load_libraries` — `def load_libraries(self, library_id: str, *, library_dirs: list[Path] | None=None, library_file_paths: list[Path] | None=None) -> list[PipeAbstract]`-- `pipelex/libraries/library_manager_abstract.py:134` — `LibraryManagerAbstract.load_libraries_concepts_only` — `def load_libraries_concepts_only(self, library_id: str, *, library_dirs: list[Path] | None=None, library_file_paths: list[Path] | None=None) -> list['Concept']`-- `pipelex/libraries/library_utils.py:24` — `get_pipelex_mthds_files_from_package._find_mthds_in_traversable` — `def _find_mthds_in_traversable(traversable: Traversable, *, collected: list[Path]) -> None`-- `pipelex/libraries/pipe/pipe_library.py:97` — `PipeLibrary.add_dependency_pipe` — `def add_dependency_pipe(self, alias: str, *, pipe: PipeAbstract) -> None`-- `pipelex/libraries/pipe/pipe_library.py:137` — `PipeLibrary.pretty_list_pipes._format_concept_code` — `def _format_concept_code(concept_code: str | None, *, current_domain: str) -> str`-- `pipelex/libraries/visibility_utils.py:35` — `make_visibility_checker` — `def make_visibility_checker(manifest: MethodsManifest | None, *, blueprints: list[PipelexBundleBlueprint]) -> PackageVisibilityChecker`-- `pipelex/libraries/visibility_utils.py:53` — `check_visibility_for_blueprints` — `def check_visibility_for_blueprints(manifest: MethodsManifest | None, *, blueprints: list[PipelexBundleBlueprint]) -> list[VisibilityError]`--### `observer` (1)--- `pipelex/observer/local_observer.py:22` — `LocalObserver._write_to_jsonl` — `def _write_to_jsonl(self, event_type: str, *, payload: PayloadType) -> None`--### `pipe_controllers` (6)--- `pipelex/pipe_controllers/batch/pipe_batch.py:152` — `PipeBatch._live_run_controller_pipe._run_branch` — `async def _run_branch(item_input_stuff: 'Stuff', *, branch_output_item_code: str) -> PipeOutput`-- `pipelex/pipe_controllers/parallel/pipe_parallel.py:326` — `PipeParallel._register_branch_outputs_with_graph_tracer` — `def _register_branch_outputs_with_graph_tracer(self, job_metadata: JobMetadata, *, output_stuffs: dict[str, 'Stuff']) -> None`-- `pipelex/pipe_controllers/parallel/pipe_parallel.py:369` — `PipeParallel._register_parallel_combine_with_graph_tracer` — `def _register_parallel_combine_with_graph_tracer(self, job_metadata: JobMetadata, *, combined_stuff: 'Stuff', branch_stuffs: dict[str, 'Stuff']) -> None`-- `pipelex/pipe_controllers/pipe_controller.py:71` — `PipeController._live_run_controller_pipe` — `async def _live_run_controller_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`-- `pipelex/pipe_controllers/pipe_controller.py:83` — `PipeController._dry_run_controller_pipe` — `async def _dry_run_controller_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`-- `pipelex/pipe_controllers/sub_pipe.py:34` — `SubPipe.run_pipe` — `async def run_pipe(self, calling_pipe_code: str, *, working_memory: WorkingMemory, job_metadata: JobMetadata, sub_pipe_run_params: PipeRunParams, library_crate: 'LibraryCrate | None'=None) -> PipeOutput`--### `pipe_operators` (40)--- `pipelex/pipe_operators/compose/pipe_compose.py:165` — `PipeCompose._run_template_mode` — `async def _run_template_mode(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None) -> PipeComposeOutput`-- `pipelex/pipe_operators/compose/pipe_compose.py:224` — `PipeCompose._run_construct_mode` — `async def _run_construct_mode(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None) -> PipeComposeOutput`-- `pipelex/pipe_operators/compose/pipe_compose.py:318` — `PipeCompose._make_mock_construct_output` — `def _make_mock_construct_output(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, output_name: str | None) -> PipeComposeOutput`-- `pipelex/pipe_operators/compose/pipe_compose_factory.py:52` — `PipeComposeFactory._make_template_mode` — `def _make_template_mode(cls, pipe_code: str, *, domain_code: str, description: str, inputs: InputStuffSpecs, output: StuffSpec, blueprint: PipeComposeBlueprint) -> PipeCompose`-- `pipelex/pipe_operators/compose/structured_content_composer.py:119` — `StructuredContentComposer._resolve_field` — `async def _resolve_field(self, field_blueprint: ConstructFieldBlueprint, *, field_name: str) -> Any`-- `pipelex/pipe_operators/compose/structured_content_composer.py:145` — `StructuredContentComposer._resolve_from_var` — `def _resolve_from_var(self, field_blueprint: ConstructFieldBlueprint, *, field_name: str) -> Any`-- `pipelex/pipe_operators/compose/structured_content_composer.py:187` — `StructuredContentComposer._convert_list_to_dict_keyed_by` — `def _convert_list_to_dict_keyed_by(self, value: Any, *, key_attr: str) -> dict[str, Any]`-- `pipelex/pipe_operators/compose/structured_content_composer.py:243` — `StructuredContentComposer._resolve_dotted_path` — `def _resolve_dotted_path(self, path: str, *, expected_type: type[Any] | None) -> Any`-- `pipelex/pipe_operators/compose/structured_content_composer.py:283` — `StructuredContentComposer._resolve_from_stuff_name` — `def _resolve_from_stuff_name(self, name: str, *, expected_type: type[Any] | None) -> StuffContent | list[dict[str, Any]] | str`-- `pipelex/pipe_operators/compose/structured_content_composer.py:298` — `StructuredContentComposer._convert_for_target_type` — `def _convert_for_target_type(self, stuff_content: StuffContent, *, expected_type: type[Any] | None) -> StuffContent | list[dict[str, Any]] | str`-- `pipelex/pipe_operators/compose/structured_content_composer.py:324` — `StructuredContentComposer._convert_text_content` — `def _convert_text_content(self, text_content: TextContent, *, expected_type: Any) -> TextContent | str`-- `pipelex/pipe_operators/compose/structured_content_composer.py:348` — `StructuredContentComposer._convert_list_content` — `def _convert_list_content(self, list_content: ListContent[StuffContent], *, expected_type: type[Any] | None) -> ListContent[StuffContent] | list[dict[str, Any]]`-- `pipelex/pipe_operators/compose/structured_content_composer.py:392` — `StructuredContentComposer._expects_type` — `def _expects_type(self, expected_type: type[Any], *, target_type: type) -> bool`-- `pipelex/pipe_operators/compose/structured_content_composer.py:410` — `StructuredContentComposer._convert_content_for_field` — `def _convert_content_for_field(self, stuff_content: StuffContent, *, expected_type: type[StuffContent]) -> StuffContent`-- `pipelex/pipe_operators/compose/structured_content_composer.py:501` — `StructuredContentComposer._convert_list_items_as_dicts` — `def _convert_list_items_as_dicts(self, items: list[StuffContent], *, expected_item_type: type[Any] | None) -> list[dict[str, Any]]`-- `pipelex/pipe_operators/compose/structured_content_composer.py:528` — `StructuredContentComposer._convert_list_items_as_objects` — `def _convert_list_items_as_objects(self, items: list[StuffContent], *, expected_item_type: type[Any] | None) -> list[StuffContent]`-- `pipelex/pipe_operators/compose/structured_content_composer.py:555` — `StructuredContentComposer._validate_item_compatibility` — `def _validate_item_compatibility(self, item: StuffContent, *, expected_type: type[Any], idx: int) -> None`-- `pipelex/pipe_operators/compose/structured_content_composer.py:599` — `StructuredContentComposer._convert_single_item_as_object` — `def _convert_single_item_as_object(self, item: StuffContent, *, expected_type: type[Any], idx: int) -> StuffContent`-- `pipelex/pipe_operators/compose/structured_content_composer.py:662` — `StructuredContentComposer._resolve_nested` — `async def _resolve_nested(self, field_blueprint: ConstructFieldBlueprint, *, field_name: str) -> StuffContent`-- `pipelex/pipe_operators/img_gen/img_gen_prompt_blueprint.py:65` — `ImgGenPromptBlueprint.make_img_gen_prompt` — `async def make_img_gen_prompt(self, context_provider: ContextProviderAbstract, *, extra_params: dict[str, Any] | None=None, max_prompt_images: int | None=None) -> ImgGenPrompt`-- `pipelex/pipe_operators/img_gen/img_gen_prompt_blueprint.py:198` — `ImgGenPromptBlueprint._extract_direct_image` — `def _extract_direct_image(self, image_ref: ImageReference, *, context_provider: ContextProviderAbstract, image_registry: ImageRegistry, image_registry_indices: dict[str, int]) -> None`-- `pipelex/pipe_operators/img_gen/img_gen_prompt_blueprint.py:224` — `ImgGenPromptBlueprint._extract_direct_list_images` — `def _extract_direct_list_images(self, image_ref: ImageReference, *, context_provider: ContextProviderAbstract, image_registry: ImageRegistry, image_registry_indices: dict[str, int]) -> None`-- `pipelex/pipe_operators/img_gen/img_gen_prompt_blueprint.py:269` — `ImgGenPromptBlueprint._render_text` — `async def _render_text(self, context_provider: ContextProviderAbstract, *, template_blueprint: TemplateBlueprint, extra_params: dict[str, Any] | None=None, image_registry: ImageRegistry | None=None) -> str`-- `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:61` — `LLMPromptBlueprint.make_llm_prompt` — `async def make_llm_prompt(self, output_concept_ref: str, *, context_provider: ContextProviderAbstract, output_structure_prompt: str | None=None, extra_params: dict[str, Any] | None=None) -> LLMPrompt`-- `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:279` — `LLMPromptBlueprint._extract_direct_image` — `def _extract_direct_image(self, image_ref: ImageReference, *, context_provider: ContextProviderAbstract, image_registry: ImageRegistry, image_registry_indices: dict[str, int]) -> None`-- `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:305` — `LLMPromptBlueprint._extract_direct_list_images` — `def _extract_direct_list_images(self, image_ref: ImageReference, *, context_provider: ContextProviderAbstract, image_registry: ImageRegistry, image_registry_indices: dict[str, int]) -> None`-- `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:350` — `LLMPromptBlueprint._unravel_text` — `async def _unravel_text(self, context_provider: ContextProviderAbstract, *, jinja2_blueprint: TemplateBlueprint, extra_params: dict[str, Any] | None=None, image_registry: ImageRegistry | None=None) -> str`-- `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:382` — `LLMPromptBlueprint._extract_direct_document` — `def _extract_direct_document(self, doc_ref: DocumentReference, *, context_provider: ContextProviderAbstract, prompt_user_documents: dict[str, PromptDocument]) -> None`-- `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:410` — `LLMPromptBlueprint._extract_direct_list_documents` — `def _extract_direct_list_documents(self, doc_ref: DocumentReference, *, context_provider: ContextProviderAbstract, prompt_user_documents: dict[str, PromptDocument]) -> None`-- `pipelex/pipe_operators/llm/pipe_llm.py:330` — `PipeLLM._llm_gen_object_stuff_content` — `async def _llm_gen_object_stuff_content(self, job_metadata: JobMetadata, *, pipe_run_params: PipeRunParams, is_multiple_output: bool, fixed_nb_output: int | None, output_class_name: str, llm_setting_for_object: LLMSetting, llm_prompt_for_object: LLMPrompt, content_generator: ContentGeneratorProtocol) -> StuffContent`-- `pipelex/pipe_operators/llm/pipe_llm.py:386` — `PipeLLM._format_llm_error` — `def _format_llm_error(self, exc: LLMCompletionError, *, settings: list[LLMSetting]) -> str`-- `pipelex/pipe_operators/llm/template_document_analyzer.py:31` — `TemplateDocumentAnalyzer.analyze_template_for_documents` — `def analyze_template_for_documents(cls, template_source: str, *, input_specs: dict[str, str], domain_code: str) -> list[DocumentReference]`-- `pipelex/pipe_operators/llm/template_document_analyzer.py:107` — `TemplateDocumentAnalyzer._resolve_concept` — `def _resolve_concept(cls, concept_ref_or_code: str, *, domain_code: str) -> Concept`-- `pipelex/pipe_operators/llm/template_document_analyzer.py:128` — `TemplateDocumentAnalyzer._resolve_variable_type` — `def _resolve_variable_type(cls, var_path: str, *, root_var: str, root_concept: Concept) -> tuple[bool, bool] | None`-- `pipelex/pipe_operators/pipe_operator.py:101` — `PipeOperator._live_run_operator_pipe` — `async def _live_run_operator_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None) -> PipeOperatorOutputType`-- `pipelex/pipe_operators/pipe_operator.py:111` — `PipeOperator._dry_run_operator_pipe` — `async def _dry_run_operator_pipe(self, job_metadata: JobMetadata, *, working_memory: WorkingMemory, pipe_run_params: PipeRunParams, output_name: str | None=None) -> PipeOperatorOutputType`-- `pipelex/pipe_operators/shared/template_image_analyzer.py:34` — `TemplateImageAnalyzer.analyze_template_for_images` — `def analyze_template_for_images(cls, template_source: str, *, input_specs: dict[str, str], domain_code: str, template_category: TemplateCategory=TemplateCategory.LLM_PROMPT) -> list[ImageReference]`-- `pipelex/pipe_operators/shared/template_image_analyzer.py:137` — `TemplateImageAnalyzer.validate_unused_inputs` — `def validate_unused_inputs(cls, template_sources: list[str], *, input_specs: dict[str, str], template_category: TemplateCategory=TemplateCategory.LLM_PROMPT) -> None`-- `pipelex/pipe_operators/shared/template_image_analyzer.py:174` — `TemplateImageAnalyzer._resolve_concept` — `def _resolve_concept(cls, concept_ref_or_code: str, *, domain_code: str) -> Concept`-- `pipelex/pipe_operators/shared/template_image_analyzer.py:195` — `TemplateImageAnalyzer._resolve_variable_type` — `def _resolve_variable_type(cls, var_path: str, *, root_var: str, root_concept: Concept) -> tuple[bool, bool, bool, list[str] | None] | None`--### `pipe_run` (18)--- `pipelex/pipe_run/delivery_executor.py:40` — `DeliveryExecutor.execute` — `async def execute(self, pipe_output: PipeOutput | None, *, user_id: str, pipeline_run_id: str, delivery_assignment: DeliveryAssignment, status: DeliveryStatus, error_report: ErrorReport | None=None, request_id: str | None=None) -> None`-- `pipelex/pipe_run/delivery_executor.py:163` — `DeliveryExecutor._generate_main_stuff_files_from_raw` — `def _generate_main_stuff_files_from_raw(cls, raw_main_stuff: dict[str, Any], *, files: dict[str, ResultFile]) -> None`-- `pipelex/pipe_run/delivery_executor.py:178` — `DeliveryExecutor._generate_main_stuff_files` — `async def _generate_main_stuff_files(self, main_stuff: Stuff, *, files: dict[str, ResultFile]) -> None`-- `pipelex/pipe_run/delivery_executor.py:185` — `DeliveryExecutor._generate_graph_files` — `async def _generate_graph_files(self, graph_spec: Any, *, files: dict[str, ResultFile]) -> None`-- `pipelex/pipe_run/delivery_executor.py:201` — `DeliveryExecutor._try_add_rendered_file` — `async def _try_add_rendered_file(cls, files: dict[str, ResultFile], *, filename: str, render: Awaitable[str], content_type: str) -> None`-- `pipelex/pipe_run/delivery_executor.py:219` — `DeliveryExecutor._add_optional_text_file` — `def _add_optional_text_file(cls, files: dict[str, ResultFile], *, filename: str, text: str | None, content_type: str) -> None`-- `pipelex/pipe_run/delivery_executor.py:226` — `DeliveryExecutor._store_results` — `async def _store_results(self, pipe_output: PipeOutput, *, user_id: str, pipeline_run_id: str, storage: StorageTarget, request_id: str | None=None) -> str`-- `pipelex/pipe_run/dry_run_in_process.py:33` — `best_effort_graph_spec` — `async def best_effort_graph_spec(pipe_ref: str | None, *, library_id: str | None, log_context: str) -> GraphSpec | None`-- `pipelex/pipe_run/dry_run_in_process.py:69` — `dry_run_pipe_in_process` — `async def dry_run_pipe_in_process(pipe: PipeAbstract, *, library_id: str) -> GraphSpec`-- `pipelex/pipe_run/pipe_job_factory.py:13` — `PipeJobFactory.make_pipe_job` — `def make_pipe_job(cls, pipe: PipeAbstract, *, job_metadata: JobMetadata, pipe_run_params: PipeRunParams | None=None, working_memory: WorkingMemory | None=None, output_name: str | None=None, library_crate: LibraryCrate | None=None) -> PipeJob`-- `pipelex/pipe_run/pipe_router_protocol.py:24` — `PipeRouterProtocol._after_successful_run` — `async def _after_successful_run(self, pipe_job: PipeJob, *, pipe_output: PipeOutput) -> None`-- `pipelex/pipe_run/pipe_router_protocol.py:37` — `PipeRouterProtocol._after_failing_run` — `async def _after_failing_run(self, pipe_job: PipeJob, *, error: Exception) -> None`-- `pipelex/pipe_run/pipe_run_params.py:28` — `output_multiplicity_to_apply` — `def output_multiplicity_to_apply(base_multiplicity: VariableMultiplicity | None, *, override_multiplicity: VariableMultiplicity | None) -> VariableMultiplicityResolution`-- `pipelex/pipe_run/pipe_run_params.py:121` — `BatchParams.make_batch_params` — `def make_batch_params(cls, input_list_name: str, *, input_item_name: str) -> BatchParams`-- `pipelex/pipe_run/pipe_run_params.py:206` — `PipeRunParams.copy_by_injecting_multiplicity` — `def copy_by_injecting_multiplicity(cls, pipe_run_params: Self, *, applied_output_multiplicity: VariableMultiplicity | None) -> Self`-- `pipelex/pipe_run/pipe_run_protocol.py:20` — `PipeRunProtocol.run` — `async def run(self, pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None=None) -> PipeOutput`-- `pipelex/pipe_run/tracing_assembly.py:51` — `assemble_tracing` — `def assemble_tracing(pipeline_run_id: str, *, assemble_graph: bool, assemble_usage: bool, domain_code: str | None=None, main_pipe_code: str | None=None) -> TracingAssembly`-- `pipelex/pipe_run/tracing_assembly.py:142` — `assemble_tracing_on_output` — `def assemble_tracing_on_output(pipe_output: PipeOutput, *, pipeline_run_id: str, assemble_graph: bool, assemble_usage: bool, domain_code: str | None=None, main_pipe_code: str | None=None) -> None`--### `pipe_signature` (2)--- `pipelex/pipe_signature/signature_walk.py:14` — `collect_signature_refs` — `def collect_signature_refs(pipe: PipeAbstract, *, visited: set[str] | None=None) -> set[str]`-- `pipelex/pipe_signature/signature_walk.py:40` — `collect_signature_paths` — `def collect_signature_paths(pipe: PipeAbstract, *, current_path: list[str] | None=None, paths: dict[str, list[str]] | None=None) -> dict[str, list[str]]`--### `pipelex.py` (3)--- `pipelex/pipelex.py:145` — `Pipelex._get_validation_error_msg` — `def _get_validation_error_msg(component_name: str, *, validation_exc: Exception) -> str`-- `pipelex/pipelex.py:165` — `Pipelex.setup` — `def setup(self, integration_mode: IntegrationMode, *, needs_inference: bool=True, temporal_enabled: bool | None=None, needs_model_specs: bool | None=None, class_registry: ClassRegistryAbstract | None=None, secrets_provider: SecretsProviderAbstract | None=None, storage_provider: StorageProviderAbstract | None=None, models_manager: ModelManagerAbstract | None=None, inference_manager: InferenceManager | None=None, content_generator: ContentGeneratorProtocol | None=None, pipeline_manager: PipelineManagerAbstract | None=None, pipe_router: PipeRouterProtocol | None=None, reporting_delegate: ReportingProtocol | None=None, telemetry_config: TelemetryConfig | None=None, telemetry_manager: TelemetryManagerAbstract | None=None, observers: dict[str, ObserverProtocol] | None=None, library_manager: LibraryManagerAbstract | None=None, library_dirs: list[str] | list[Path] | None=None, **kwargs: Any)`-- `pipelex/pipelex.py:512` — `Pipelex.make` — `def make(cls, integration_mode: IntegrationMode=IntegrationMode.PYTHON, *, needs_inference: bool=True, temporal_enabled: bool | None=None, needs_model_specs: bool | None=None, class_registry: ClassRegistryAbstract | None=None, secrets_provider: SecretsProviderAbstract | None=None, storage_provider: StorageProviderAbstract | None=None, models_manager: ModelManagerAbstract | None=None, inference_manager: InferenceManager | None=None, content_generator: ContentGeneratorProtocol | None=None, pipeline_manager: PipelineManager | None=None, pipe_router: PipeRouterProtocol | None=None, reporting_delegate: ReportingProtocol | None=None, telemetry_config: TelemetryConfig | None=None, telemetry_manager: TelemetryManagerAbstract | None=None, observers: dict[str, ObserverProtocol] | None=None, library_dirs: list[str] | list[Path] | None=None, config_overrides: dict[str, Any] | None=None, **kwargs: Any) -> Self`--### `pipeline` (18)--- `pipelex/pipeline/bundle_validator.py:171` — `BundleValidator.validate_pipes` — `async def validate_pipes(self, pipes: list[PipeAbstract], *, library_id: str, allow_signatures: bool=False) -> dict[str, DryRunOutput]`-- `pipelex/pipeline/dry_run_pipeline.py:26` — `dry_run_pipeline` — `async def dry_run_pipeline(mthds_contents: list[str] | None=None, *, bundle_uris: list[str] | None=None, library_dirs: list[str] | None=None) -> tuple[GraphSpec, str]`-- `pipelex/pipeline/execution_seams.py:55` — `acquire_library` — `def acquire_library(library_id: str, *, library_dirs: list[str] | None=None, mthds_contents: list[str] | None=None, bundle_uris: list[str] | None=None) -> tuple[str, str | None]`-- `pipelex/pipeline/execution_seams.py:154` — `prepare_pipe_job` — `async def prepare_pipe_job(pipe: PipeAbstract, *, library_id: str, execution_config: PipelineExecutionConfig, pipe_run_mode: PipeRunMode, pipeline_run_id: str, user_id: str, inputs: PipelineInputs | WorkingMemory | None=None, search_domain_codes: list[str] | None=None, trace_context: 'TraceContext | None'=None, otel_context: OtelContext | None=None, output_name: str | None=None, output_multiplicity: VariableMultiplicity | None=None, dynamic_output_concept_ref: str | None=None, request_id: str | None=None, is_mock_usage: bool=False) -> PipeJob`-- `pipelex/pipeline/input_normalizer.py:61` — `_normalize_value` — `async def _normalize_value(value: Any, *, storage: StorageProviderAbstract) -> tuple[Any, bool]`-- `pipelex/pipeline/input_normalizer.py:96` — `_normalize_structured_content` — `async def _normalize_structured_content(structured_content: StructuredContent, *, storage: StorageProviderAbstract) -> tuple[StructuredContent, bool]`-- `pipelex/pipeline/input_normalizer.py:127` — `_normalize_list_content` — `async def _normalize_list_content(list_content: ListContent[Any], *, storage: StorageProviderAbstract) -> tuple[ListContent[Any], bool]`-- `pipelex/pipeline/input_normalizer.py:161` — `_normalize_list` — `async def _normalize_list(items: list[Any], *, storage: StorageProviderAbstract) -> tuple[list[Any], bool]`-- `pipelex/pipeline/input_normalizer.py:187` — `_normalize_url_content` — `async def _normalize_url_content(content: NormalizableContent, *, storage: StorageProviderAbstract) -> NormalizableContent`-- `pipelex/pipeline/job_metadata.py:89` — `JobMetadata.copy_with_update` — `def copy_with_update(self, otel_context: OtelContext | None, *, trace_context: TraceContext | None=None, **updates: Any) -> 'JobMetadata'`-- `pipelex/pipeline/pipeline_manager.py:35` — `PipelineManager._set_pipeline` — `def _set_pipeline(self, pipeline_run_id: str, *, pipeline: Pipeline) -> Pipeline`-- `pipelex/pipeline/pipeline_manager_abstract.py:24` — `PipelineManagerAbstract.add_new_pipeline` — `def add_new_pipeline(self, pipe_code: str | None, *, pipeline_run_id: str | None=None) -> Pipeline`-- `pipelex/pipeline/pipeline_response.py:36` — `PipelexRunResultExecute.from_pipe_output` — `def from_pipe_output(cls, pipe_output: PipeOutput, *, pipeline_run_id: str='', created_at: str='', state: RunState=RunState.COMPLETED, finished_at: str | None=None) -> PipelexRunResultExecute`-- `pipelex/pipeline/pipeline_run_setup.py:43` — `pipeline_run_setup` — `async def pipeline_run_setup(execution_config: PipelineExecutionConfig, *, library_id: str | None=None, library_dirs: list[str] | None=None, pipe_code: str | None=None, mthds_contents: list[str] | None=None, bundle_uris: list[str] | None=None, inputs: PipelineInputs | WorkingMemory | None=None, output_name: str | None=None, output_multiplicity: VariableMultiplicity | None=None, dynamic_output_concept_ref: str | None=None, pipe_run_mode: PipeRunMode | None=None, is_mock_usage: bool=False, search_domain_codes: list[str] | None=None, user_id: str | None=None, pipeline_run_id: str | None=None, request_id: str | None=None) -> tuple[PipeJob, str, str]`-- `pipelex/pipeline/validate_bundle.py:204` — `_pipes_to_dry_run` — `def _pipes_to_dry_run(loaded_pipes: list[PipeAbstract], *, dry_run_pipe_codes: list[str] | None) -> list[PipeAbstract]`-- `pipelex/pipeline/validate_bundle.py:230` — `validate_bundle` — `async def validate_bundle(mthds_file_path: Path | None=None, *, mthds_contents: list[str] | None=None, library_dirs: Sequence[Path] | None=None, allow_signatures: bool=False, dry_run_pipe_codes: list[str] | None=None) -> ValidateBundleResult`-- `pipelex/pipeline/validate_bundle.py:335` — `validate_bundles_from_directory` — `async def validate_bundles_from_directory(directory: Path, *, allow_signatures: bool=False) -> ValidateBundleResult`-- `pipelex/pipeline/validate_bundle.py:378` — `load_concepts_only` — `def load_concepts_only(mthds_file_path: Path | None=None, *, mthds_contents: list[str] | None=None, library_dirs: Sequence[Path] | None=None) -> LoadConceptsOnlyResult`--### `plugins` (49)--- `pipelex/plugins/anthropic/anthropic_factory.py:38` — `AnthropicFactory.make_anthropic_client` — `def make_anthropic_client(plugin: Plugin, *, backend: InferenceBackend) -> AsyncAnthropic | AsyncAnthropicBedrock`-- `pipelex/plugins/anthropic/anthropic_factory.py:105` — `AnthropicFactory._make_document_block_param` — `def _make_document_block_param(prepped_document: PreparedFile, *, title: str | None=None) -> DocumentBlockParam`-- `pipelex/plugins/anthropic/anthropic_factory.py:169` — `AnthropicFactory.openai_typed_user_message` — `def openai_typed_user_message(cls, user_content_txt: str, *, prepped_user_images: list[PreparedFile] | None=None, prepped_user_documents: list[tuple[PreparedFile, str | None]] | None=None) -> 'ChatCompletionMessageParam'`-- `pipelex/plugins/anthropic/anthropic_factory.py:244` — `AnthropicFactory.make_nb_tokens_by_category_from_nb` — `def make_nb_tokens_by_category_from_nb(nb_input: int, *, nb_output: int) -> NbTokensByCategoryDict`-- `pipelex/plugins/anthropic/anthropic_list.py:20` — `list_anthropic_models` — `async def list_anthropic_models(sdk: str, *, backend_name: str, backend: InferenceBackend, flat: bool, any_listed: bool) -> None`-- `pipelex/plugins/anthropic/anthropic_list.py:72` — `_display_anthropic_models_flat` — `def _display_anthropic_models_flat(models: list[ModelInfo], *, sdk: str, backend_name: str, any_listed: bool) -> None`-- `pipelex/plugins/anthropic/anthropic_list.py:89` — `_display_anthropic_models_table` — `def _display_anthropic_models_table(models: list[ModelInfo], *, sdk: str, backend_name: str) -> None`-- `pipelex/plugins/anthropic/anthropic_llm_worker.py:115` — `AnthropicLLMWorker._build_thinking_params` — `def _build_thinking_params(self, job_params: LLMJobParams, *, max_tokens: int) -> _ThinkingParams`-- `pipelex/plugins/anthropic/anthropic_llm_worker.py:145` — `AnthropicLLMWorker._build_thinking_params_for_effort` — `def _build_thinking_params_for_effort(self, thinking_mode: ThinkingMode, *, effort: ReasoningEffort, max_tokens: int) -> _ThinkingParams`-- `pipelex/plugins/anthropic/anthropic_llm_worker.py:200` — `AnthropicLLMWorker._build_thinking_params_for_budget` — `def _build_thinking_params_for_budget(self, thinking_mode: ThinkingMode, *, budget: int, max_tokens: int) -> _ThinkingParams`-- `pipelex/plugins/anthropic/anthropic_llms.py:10` — `anthropic_list_available_models` — `async def anthropic_list_available_models(plugin: Plugin, *, backend: InferenceBackend) -> list[ModelInfo]`-- `pipelex/plugins/bedrock/bedrock_client_protocol.py:9` — `BedrockClientProtocol.chat` — `async def chat(self, messages: BedrockMessageDictList, *, system_text: str | None, model: str, temperature: float, max_tokens: int | None=None) -> tuple[str, NbTokensByCategoryDict]`-- `pipelex/plugins/bedrock/bedrock_factory.py:27` — `BedrockFactory.make_bedrock_client` — `def make_bedrock_client(cls, plugin: Plugin, *, backend: InferenceBackend) -> BedrockClientProtocol`-- `pipelex/plugins/bedrock/bedrock_list.py:20` — `list_bedrock_models` — `def list_bedrock_models(sdk: str, *, backend_name: str, backend: InferenceBackend, flat: bool, any_listed: bool) -> None`-- `pipelex/plugins/bedrock/bedrock_list.py:80` — `_display_bedrock_models_flat` — `def _display_bedrock_models_flat(models: list[dict[str, Any]], *, sdk: str, backend_name: str, aws_region: str, any_listed: bool) -> None`-- `pipelex/plugins/bedrock/bedrock_list.py:99` — `_display_bedrock_models_table` — `def _display_bedrock_models_table(models: list[dict[str, Any]], *, sdk: str, aws_region: str) -> None`-- `pipelex/plugins/bedrock/bedrock_llms.py:10` — `bedrock_list_available_models` — `def bedrock_list_available_models(plugin: Plugin, *, backend: InferenceBackend) -> list[dict[str, Any]]`-- `pipelex/plugins/fal/fal_img_gen_worker.py:37` — `FalImgGenWorker._submit_and_get_result` — `async def _submit_and_get_result(self, img_gen_job: ImgGenJob, *, nb_images: int) -> Any`-- `pipelex/plugins/gateway/gateway_completions_factory.py:104` — `GatewayCompletionsFactory.make_portkey_openai_client_for_completions` — `def make_portkey_openai_client_for_completions(cls, plugin: Plugin, *, backend: InferenceBackend) -> openai.AsyncOpenAI`-- `pipelex/plugins/gateway/gateway_completions_factory.py:138` — `GatewayCompletionsFactory.make_extract_output_from_response` — `def make_extract_output_from_response(cls, inference_model: InferenceModelSpec, *, response: GenericResponse) -> ExtractOutput`-- `pipelex/plugins/gateway/gateway_extract_worker.py:191` — `GatewayExtractWorker._extract_base64_url` — `async def _extract_base64_url(self, extract_job: ExtractJob, *, base64_url: str, should_include_images: bool=False) -> ExtractOutput`-- `pipelex/plugins/gateway/gateway_factory.py:64` — `GatewayFactory.make_extras` — `def make_extras(cls, inference_model: InferenceModelSpec, *, inference_job: InferenceJobAbstract, output_desc: str) -> tuple[dict[str, str], dict[str, Any]]`-- `pipelex/plugins/gateway/gateway_responses_factory.py:26` — `GatewayResponsesFactory.make_portkey_openai_client_for_responses` — `def make_portkey_openai_client_for_responses(cls, plugin: Plugin, *, backend: InferenceBackend) -> openai.AsyncOpenAI`-- `pipelex/plugins/gateway/gateway_search_worker.py:128` — `GatewaySearchWorker._extract_usage` — `def _extract_usage(self, response: GenericResponse, *, search_job: SearchJob) -> None`-- `pipelex/plugins/gateway/gateway_search_worker.py:140` — `GatewaySearchWorker._call_relay` — `async def _call_relay(self, model: str, *, content: str) -> GenericResponse`-- `pipelex/plugins/google/google_factory.py:79` — `GoogleFactory.extract_text_from_response` — `def extract_text_from_response(cls, response: genai_types.GenerateContentResponse, *, model_desc: str) -> str`-- `pipelex/plugins/google/google_img_gen_factory.py:106` — `GoogleImgGenFactory.dimensions_for_aspect_ratio_and_size` — `def dimensions_for_aspect_ratio_and_size(cls, model: str, *, aspect_ratio: AspectRatio, size: GoogleImageSize) -> tuple[int, int]`-- `pipelex/plugins/google/google_list.py:16` — `list_google_models` — `async def list_google_models(sdk: str, *, backend_name: str, backend: InferenceBackend, flat: bool, any_listed: bool) -> None`-- `pipelex/plugins/google/google_list.py:57` — `_display_google_models_flat` — `def _display_google_models_flat(models: list[Any], *, sdk: str, backend_name: str, any_listed: bool) -> None`-- `pipelex/plugins/google/google_list.py:75` — `_display_google_models_table` — `def _display_google_models_table(models: list[Any], *, sdk: str, backend_name: str) -> None`-- `pipelex/plugins/google/google_llm_worker.py:111` — `GoogleLLMWorker._build_thinking_config` — `def _build_thinking_config(self, job_params: LLMJobParams, *, max_tokens: int | None) -> genai_types.ThinkingConfig | None`-- `pipelex/plugins/google/google_llm_worker.py:135` — `GoogleLLMWorker._build_thinking_config_for_effort` — `def _build_thinking_config_for_effort(self, thinking_mode: ThinkingMode, *, effort: ReasoningEffort, max_tokens: int | None) -> genai_types.ThinkingConfig`-- `pipelex/plugins/google/google_llm_worker.py:172` — `GoogleLLMWorker._build_thinking_config_for_budget` — `def _build_thinking_config_for_budget(self, thinking_mode: ThinkingMode, *, budget: int, max_tokens: int | None) -> genai_types.ThinkingConfig`-- `pipelex/plugins/mistral/mistral_extract_worker.py:86` — `MistralExtractWorker._extract_pages_from_document` — `async def _extract_pages_from_document(self, document_uri: str, *, extract_job_params: ExtractJobParams) -> ExtractOutput`-- `pipelex/plugins/mistral/mistral_factory.py:365` — `MistralFactory.make_mistral_document_url_chunk_from_uri` — `async def make_mistral_document_url_chunk_from_uri(cls, mistral_client: Mistral, *, uri: str) -> DocumentURLChunkTypedDict`-- `pipelex/plugins/mistral/mistral_list.py:13` — `list_mistral_models` — `def list_mistral_models(sdk: str, *, backend_name: str, flat: bool, any_listed: bool) -> None`-- `pipelex/plugins/mistral/mistral_list.py:54` — `_display_mistral_models_flat` — `def _display_mistral_models_flat(models: list[Any], *, sdk: str, backend_name: str, any_listed: bool) -> None`-- `pipelex/plugins/mistral/mistral_list.py:70` — `_display_mistral_models_table` — `def _display_mistral_models_table(models: list[Any], *, sdk: str, backend_name: str) -> None`-- `pipelex/plugins/openai/openai_client_factory.py:27` — `OpenAIClientFactory.make_openai_client` — `def make_openai_client(cls, plugin: Plugin, *, backend: InferenceBackend) -> openai.AsyncClient`-- `pipelex/plugins/openai/openai_list.py:19` — `list_openai_models` — `async def list_openai_models(sdk: str, *, backend_name: str, backend: InferenceBackend, flat: bool, any_listed: bool) -> None`-- `pipelex/plugins/openai/openai_list.py:49` — `_display_openai_models_flat` — `def _display_openai_models_flat(models: list[Model], *, sdk: str, backend_name: str, any_listed: bool) -> None`-- `pipelex/plugins/openai/openai_list.py:70` — `_display_openai_models_table` — `def _display_openai_models_table(models: list[Model], *, sdk: str, backend_name: str) -> None`-- `pipelex/plugins/openai/openai_llms.py:8` — `openai_list_available_models` — `async def openai_list_available_models(plugin: Plugin, *, backend: InferenceBackend) -> list[Model]`-- `pipelex/plugins/openai/vertexai_factory.py:47` — `VertexAIFactory._make_endpoint` — `def _make_endpoint(cls, gcp_project_id: str, *, gcp_location: str) -> str`-- `pipelex/plugins/plugin_factory_abstract.py:10` — `PluginFactoryAbstract.make_extras` — `def make_extras(self, inference_model: InferenceModelSpec, *, inference_job: InferenceJobAbstract, output_desc: str) -> tuple[dict[str, str], dict[str, Any]]`-- `pipelex/plugins/plugin_sdk_registry.py:22` — `PluginSdkRegistry.set_sdk_instance` — `def set_sdk_instance(self, plugin: Plugin, *, sdk_instance: Any) -> Any`-- `pipelex/plugins/portkey/portkey_completions_factory.py:95` — `PortkeyCompletionsFactory.make_portkey_openai_client_for_completions` — `def make_portkey_openai_client_for_completions(cls, plugin: Plugin, *, backend: InferenceBackend) -> openai.AsyncOpenAI`-- `pipelex/plugins/portkey/portkey_factory.py:40` — `PortkeyFactory.make_extras` — `def make_extras(cls, inference_model: InferenceModelSpec, *, inference_job: InferenceJobAbstract, output_desc: str) -> tuple[dict[str, str], dict[str, Any]]`-- `pipelex/plugins/portkey/portkey_responses_factory.py:26` — `PortkeyResponsesFactory.make_portkey_openai_client_for_responses` — `def make_portkey_openai_client_for_responses(cls, plugin: Plugin, *, backend: InferenceBackend) -> openai.AsyncOpenAI`--### `reporting` (5)--- `pipelex/reporting/reporting_manager.py:179` — `ReportingManager._emit_usage_event` — `def _emit_usage_event(self, inference_job: InferenceJobAbstract, *, tokens_usage: AnyTokensUsage) -> None`-- `pipelex/reporting/reporting_manager.py:229` — `ReportingManager._emit_via_registered_context` — `def _emit_via_registered_context(context: _EventLogContext, *, trace_context: TraceContext, tokens_usage: AnyTokensUsage) -> None`-- `pipelex/reporting/reporting_manager.py:251` — `ReportingManager._emit_best_effort` — `def _emit_best_effort(event_log: EventLogProtocol, *, event: UsageReportEvent) -> None`-- `pipelex/reporting/reporting_manager.py:265` — `ReportingManager._emit_usage_event_runner_fallback` — `def _emit_usage_event_runner_fallback(self, inference_job: InferenceJobAbstract, *, tokens_usage: AnyTokensUsage, trace_context: TraceContext) -> None`-- `pipelex/reporting/reporting_protocol.py:16` — `ReportingProtocol.set_event_log` — `def set_event_log(self, context_key: str, *, event_log: EventLogProtocol, workflow_id: str, pipeline_run_id: str) -> None`--### `runtime_bridge` (12)--- `pipelex/runtime_bridge/bridge.py:106` — `run_pipe_via_bridge` — `async def run_pipe_via_bridge(input_payload: PipelexPipeRunInput, *, trace_context: TraceContext | None=None) -> PipelexPipeRunOutput`-- `pipelex/runtime_bridge/bridge.py:160` — `build_pipe_job_from_input` — `def build_pipe_job_from_input(input_payload: PipelexPipeRunInput, *, library_crate: LibraryCrate | None, trace_context: TraceContext | None=None) -> PipeJob`-- `pipelex/runtime_bridge/bridge.py:240` — `_validate_input` — `def _validate_input(input_payload: PipelexPipeRunInput, *, delivery_assignment: DeliveryAssignment | None) -> None`-- `pipelex/runtime_bridge/bridge.py:270` — `_run_direct` — `async def _run_direct(pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput`-- `pipelex/runtime_bridge/bridge.py:297` — `_run_temporal_blocking` — `async def _run_temporal_blocking(pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput`-- `pipelex/runtime_bridge/bridge.py:330` — `_run_temporal_fire_and_forget` — `async def _run_temporal_fire_and_forget(pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput`-- `pipelex/runtime_bridge/bridge.py:358` — `_serialize_completed_output` — `def _serialize_completed_output(pipe_output: PipeOutput, *, workflow_id: str | None) -> PipelexPipeRunOutput`-- `pipelex/runtime_bridge/bridge.py:408` — `_run_mistral_native` — `async def _run_mistral_native(pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None) -> PipelexPipeRunOutput`-- `pipelex/runtime_bridge/primitives/hydration.py:17` — `_validate_as_known_class` — `def _validate_as_known_class(item_class: type[StuffContent], *, raw_item: StuffContent | dict[str, Any]) -> StuffContent`-- `pipelex/runtime_bridge/primitives/hydration.py:75` — `hydrate_content` — `def hydrate_content(raw_content: list[Any] | dict[str, Any] | str, *, concept: Concept) -> StuffContent`-- `pipelex/runtime_bridge/primitives/scoped_library.py:16` — `scoped_library_for_crate` — `def scoped_library_for_crate(library_crate: LibraryCrate | None, *, library_id_prefix: str) -> Generator[str | None, None, None]`-- `pipelex/runtime_bridge/primitives/submitter_hydration.py:30` — `rehydrate_pipe_output_with_crate` — `def rehydrate_pipe_output_with_crate(pipe_output: PipeOutput, *, library_crate: 'LibraryCrate | None') -> PipeOutput`--### `system` (38)--- `pipelex/system/configuration/config_loader.py:90` — `ConfigLoader.resolve_config_file` — `def resolve_config_file(self, relative_path: str, *, config_dir: Path | None=None) -> Path`-- `pipelex/system/configuration/config_loader.py:171` — `ConfigLoader._override_files_for_dir` — `def _override_files_for_dir(cls, config_dir: Path, *, include_run_mode: bool) -> list[Path]`-- `pipelex/system/configuration/config_model.py:15` — `ConfigModel.transform_dict_str_to_enum` — `def transform_dict_str_to_enum(input_dict: dict[str, str], *, key_enum_cls: type[StrEnumType] | None=None, value_enum_cls: type[StrEnumType] | None=None) -> dict[str, StrEnumType] | dict[StrEnumType, str] | dict[StrEnumType, StrEnumType]`-- `pipelex/system/configuration/config_model.py:43` — `ConfigModel.transform_dict_of_floats_str_to_enum` — `def transform_dict_of_floats_str_to_enum(input_dict: dict[str, float], *, key_enum_cls: type[StrEnumType]) -> dict[StrEnumType, float]`-- `pipelex/system/configuration/config_model.py:61` — `ConfigModel.transform_dict_keys_str_to_enum` — `def transform_dict_keys_str_to_enum(input_dict: dict[str, Any], *, key_enum_cls: type[StrEnumType]) -> dict[StrEnumType, Any]`-- `pipelex/system/configuration/config_model.py:79` — `ConfigModel.transform_list_of_str_to_enum` — `def transform_list_of_str_to_enum(input_list: list[str], *, enum_cls: type[StrEnumType]) -> list[StrEnumType]`-- `pipelex/system/configuration/configs.py:234` — `MigrationConfig.text_in_renaming_keys` — `def text_in_renaming_keys(self, text: str, *, category: str) -> list[tuple[str, str]]`-- `pipelex/system/configuration/configs.py:240` — `MigrationConfig.text_in_renaming_values` — `def text_in_renaming_values(self, text: str, *, category: str) -> list[tuple[str, str]]`-- `pipelex/system/pipelex_service/gateway_config_merger.py:21` — `GatewayConfigMerger.merge` — `def merge(cls, gateway_model_specs: BackendModelSpecs, *, local_overrides: BackendModelSpecs) -> dict[str, Any]`-- `pipelex/system/pipelex_service/gateway_config_merger.py:69` — `GatewayConfigMerger._apply_overrides_to_model` — `def _apply_overrides_to_model(cls, model_name: str, *, gateway_model_specs: BackendModelSpecs, local_model_config: BackendModelSpecs) -> None`-- `pipelex/system/pipelex_service/pipelex_service_agreement.py:29` — `update_service_terms_acceptance` — `def update_service_terms_acceptance(accepted: bool, *, config_dir: Path | None=None) -> None`-- `pipelex/system/pipelex_service/pipelex_service_agreement.py:53` — `update_inference_setup_completed` — `def update_inference_setup_completed(completed: bool, *, config_dir: Path | None=None) -> None`-- `pipelex/system/pipelex_service/remote_config_fetcher.py:167` — `RemoteConfigFetcher._build_unavailable_error` — `def _build_unavailable_error(cls, fetch_error: RemoteConfigFetchError, *, cache_refused: bool=False) -> RemoteConfigUnavailableError`-- `pipelex/system/registries/class_registry_utils.py:21` — `ClassRegistryUtils.register_classes_in_file` — `def register_classes_in_file(cls, file_path: Path, *, base_class: type[Any] | None, is_include_imported: bool) -> None`-- `pipelex/system/registries/class_registry_utils.py:41` — `ClassRegistryUtils.register_classes_in_folder` — `def register_classes_in_folder(cls, folder_path: Path, *, base_class: type[Any] | None=None, is_recursive: bool=True, is_include_imported: bool=False, force_exclude_dirs: list[Path] | None=None) -> None`-- `pipelex/system/registries/class_registry_utils.py:77` — `ClassRegistryUtils.import_modules_in_folder` — `def import_modules_in_folder(cls, folder_path: Path, *, force_include_dirs: list[Path] | None=None, is_recursive: bool=True, base_class_names: list[str] | None=None) -> None`-- `pipelex/system/registries/func_registry.py:99` — `FuncRegistry.register_function` — `def register_function(self, func: Callable[..., Any], *, name: str | None=None) -> None`-- `pipelex/system/registries/func_registry.py:200` — `FuncRegistry.is_eligible_function` — `def is_eligible_function(self, func: Any, *, require_decorator: bool=False) -> bool`-- `pipelex/system/registries/func_registry.py:347` — `FuncRegistry.register_ineligible_function` — `def register_ineligible_function(self, func: Callable[..., Any], *, reason: str, source_file: str | None=None) -> None`-- `pipelex/system/registries/func_registry_utils.py:18` — `FuncRegistryUtils.register_pipe_funcs_from_package` — `def register_pipe_funcs_from_package(cls, package_name: str, *, package: Any) -> int`-- `pipelex/system/registries/func_registry_utils.py:75` — `FuncRegistryUtils.register_funcs_in_folder` — `def register_funcs_in_folder(cls, folder_path: Path, *, force_include_dirs: list[Path] | None=None, is_recursive: bool=True) -> None`-- `pipelex/system/telemetry/otel_factory.py:32` — `OtelFactory.make_truncated_content` — `def make_truncated_content(cls, content: str, *, max_length: int | None) -> str`-- `pipelex/system/telemetry/otel_factory.py:61` — `OtelFactory.make_inputs_json` — `def make_inputs_json(cls, working_memory: WorkingMemory, *, needed_input_names: set[str], max_length: int | None) -> str`-- `pipelex/system/telemetry/otel_factory.py:90` — `OtelFactory.make_output_json` — `def make_output_json(cls, pipe_output: PipeOutput, *, max_length: int | None) -> str`-- `pipelex/system/telemetry/otel_factory.py:133` — `OtelFactory.make_trace_names` — `def make_trace_names(cls, pipeline_run_id: str, *, pipe_code: str) -> tuple[str, str]`-- `pipelex/system/telemetry/otel_factory.py:198` — `OtelFactory.make_ai_tracer` — `def make_ai_tracer(cls, user_id: str | None, *, custom_posthog_client: 'Posthog | None', custom_redaction_config: TelemetryRedactionConfig, pipelex_posthog_client: 'Posthog | None', pipelex_gateway_redaction_config: TelemetryRedactionConfig, pipelex_distinct_id: str | None, otlp_exporters: list[OtlpExporterConfig] | None, langfuse_config: LangfuseConfig | None) -> 'tuple[OTelTracer, OTelTracerProvider]'`-- `pipelex/system/telemetry/posthog_span_exporter.py:43` — `PostHogSpanExporter._capture_event` — `def _capture_event(self, event: PostHogEvent, *, properties: dict[str, Any]) -> None`-- `pipelex/system/telemetry/posthog_span_exporter.py:99` — `PostHogSpanExporter._build_redacted_pipe_span_name` — `def _build_redacted_pipe_span_name(self, original_span_name: str, *, pipe_type: str | None) -> str`-- `pipelex/system/telemetry/posthog_span_exporter.py:127` — `PostHogSpanExporter._build_redacted_generation_span_name` — `def _build_redacted_generation_span_name(self, original_span_name: str, *, pipe_code: str | None, unit_job_id: str | None, model_name: str | None, output_class_name: str | None) -> str`-- `pipelex/system/telemetry/posthog_span_exporter.py:198` — `PostHogSpanExporter._get_base_properties` — `def _get_base_properties(self, span: ReadableSpan, *, attributes: Mapping[str, AttributeValue]) -> dict[str, Any]`-- `pipelex/system/telemetry/posthog_span_exporter.py:222` — `PostHogSpanExporter._export_generation_span` — `def _export_generation_span(self, span: ReadableSpan, *, attributes: Mapping[str, AttributeValue]) -> None`-- `pipelex/system/telemetry/posthog_span_exporter.py:288` — `PostHogSpanExporter._export_pipe_span` — `def _export_pipe_span(self, span: ReadableSpan, *, attributes: Mapping[str, AttributeValue]) -> None`-- `pipelex/system/telemetry/telemetry_factory.py:23` — `TelemetryFactory.make_telemetry_manager` — `def make_telemetry_manager(cls, secrets_provider: SecretsProviderAbstract, *, integration_mode: IntegrationMode, remote_config: RemoteConfig | None, is_pipelex_telemetry_enabled: bool=False, telemetry_config: TelemetryConfig | None=None, injected_telemetry_manager: TelemetryManagerAbstract | None=None) -> TelemetryManagerAbstract`-- `pipelex/system/telemetry/telemetry_manager.py:269` — `TelemetryManager._track_anonymous_event` — `def _track_anonymous_event(self, event_name: str, *, properties: dict[str, Any])`-- `pipelex/system/telemetry/telemetry_manager.py:277` — `TelemetryManager._track_identified_event` — `def _track_identified_event(self, event_name: str, *, properties: dict[str, Any], user_id: str)`-- `pipelex/system/telemetry/telemetry_manager.py:284` — `TelemetryManager._track_to_pipelex` — `def _track_to_pipelex(self, event_name: str, *, properties: dict[str, Any])`-- `pipelex/system/telemetry/telemetry_manager_abstract.py:97` — `TelemetryManagerAbstract.track_event` — `def track_event(self, event_name: EventName, *, properties: dict[EventProperty, Any] | None=None)`-- `pipelex/system/telemetry/telemetry_manager_abstract.py:156` — `TelemetryManagerAbstract.handle_trace_start` — `def handle_trace_start(self, trace_name: str, *, trace_name_redacted: str, trace_id: int) -> None`--### `temporal` (39)--- `pipelex/temporal/codec/codec_server.py:31` — `_set_cors_headers` — `def _set_cors_headers(response: web.Response, *, request_origin: str, cors_origins: list[str]) -> None`-- `pipelex/temporal/codec/codec_server.py:50` — `_error_response` — `def _error_response(request: web.Request, *, cors_origins: list[str], status: int, text: str) -> web.Response`-- `pipelex/temporal/codec/codec_server.py:89` — `build_codec_server` — `def build_codec_server(codec: StoragePayloadCodec, *, cors_origins: list[str]) -> web.Application`-- `pipelex/temporal/codec/storage_payload_codec.py:79` — `StoragePayloadCodec._build_storage_key` — `def _build_storage_key(self, payload: Payload, *, hash_hex: str) -> str`-- `pipelex/temporal/config_temporal.py:469` — `WorkerConfig.resolve_queue` — `def resolve_queue(self, activity_name: str, *, routing_key: str | None=None) -> str | None`-- `pipelex/temporal/config_temporal.py:508` — `WorkerConfig.resolve_dispatch` — `def resolve_dispatch(self, activity_name: str, *, routing_key: str | None=None, queue_options_by_queue: dict[str, 'QueueOptions'] | None=None, is_traced: bool=False) -> DispatchOptions`-- `pipelex/temporal/task_manager.py:15` — `TaskManager.complement_catalog` — `def complement_catalog(self, extra_catalog: dict[str, TaskPack], *, extra_workflows: list[WorkflowType], extra_activities: list[ActivityType])`-- `pipelex/temporal/task_manager.py:23` — `TaskManager.make_worker` — `def make_worker(self, temporal_client: TemporalClient, *, task_queue: str, is_not_sandboxed: bool=False, scope: WorkerScope | None=None, runtime_profile: WorkerRuntimeProfile | None=None, substitute_activities: dict[ActivityType, ActivityType] | None=None, test_workflows: WorkflowList | None=None, test_activities: ActivityList | None=None) -> Worker`-- `pipelex/temporal/task_manager.py:36` — `TaskManager.run_worker` — `async def run_worker(self, is_not_sandboxed: bool, *, is_unit_testing: bool, task_queue: str | None=None, scope_name: str | None=None, profile_name: str | None=None)`-- `pipelex/temporal/task_manager.py:48` — `TaskManager.workflows_and_activities` — `def workflows_and_activities(self, scope: WorkerScope | None=None, *, test_workflows: WorkflowList | None=None, test_activities: ActivityList | None=None, substitute_activities: dict[ActivityType, ActivityType] | None=None) -> tuple[list[WorkflowType], list[ActivityType]]`-- `pipelex/temporal/temporal_connect.py:18` — `connect_to_temporal_server` — `async def connect_to_temporal_server(server_config: TemporalServerConfig, *, name: str | None=None) -> TemporalClient`-- `pipelex/temporal/temporal_data_converter.py:64` — `BaseModelPayloadConverter._kajson_to_payload` — `def _kajson_to_payload(self, value: object, *, source_type_holder: object) -> Payload`-- `pipelex/temporal/temporal_data_converter.py:97` — `BaseModelPayloadConverter._restore_class_source` — `def _restore_class_source(self, value: BaseModel, *, class_source_code: str) -> None`-- `pipelex/temporal/temporal_manager.py:47` — `TemporalManager.connect_temporal` — `async def connect_temporal(self, temporal_client: TemporalClient | None=None, *, temporal_server_config: TemporalServerConfig | None=None, temporal_selected_server: str | None=None) -> TemporalClient`-- `pipelex/temporal/temporal_tasks.py:19` — `TemporalTasks.complement_catalog` — `def complement_catalog(self, extra_catalog: dict[str, TaskPack], *, extra_workflows: list[WorkflowType], extra_activities: list[ActivityType])`-- `pipelex/temporal/temporal_tasks.py:38` — `TemporalTasks.replace_catalog` — `def replace_catalog(self, new_catalog: dict[str, TaskPack], *, new_workflows: list[WorkflowType], new_activities: list[ActivityType])`-- `pipelex/temporal/temporal_tasks.py:68` — `TemporalTasks._register_workflow` — `def _register_workflow(cls, all_workflows: dict[str, WorkflowType], *, workflow: WorkflowType, source: str) -> None`-- `pipelex/temporal/temporal_tasks.py:79` — `TemporalTasks._register_activity` — `def _register_activity(cls, all_activities: dict[str, ActivityType], *, activity: ActivityType, source: str) -> None`-- `pipelex/temporal/temporal_tasks.py:89` — `TemporalTasks.workflows_and_activities` — `def workflows_and_activities(self, scope: WorkerScope | None=None, *, test_workflows: WorkflowList | None=None, test_activities: ActivityList | None=None, substitute_activities: dict[ActivityType, ActivityType] | None=None) -> tuple[list[WorkflowType], list[ActivityType]]`-- `pipelex/temporal/temporal_tasks.py:122` — `TemporalTasks._resolve_scope` — `def _resolve_scope(self, scope: WorkerScope, *, all_workflows: dict[str, WorkflowType], all_activities: dict[str, ActivityType]) -> tuple[set[WorkflowType], set[ActivityType]]`-- `pipelex/temporal/tprl/namespace_check.py:71` — `format_temporal_cli_command` — `def format_temporal_cli_command(missing: Sequence[str], *, namespace: str) -> str`-- `pipelex/temporal/tprl/namespace_check.py:84` — `format_tcld_cli_command` — `def format_tcld_cli_command(missing: Sequence[str], *, namespace: str) -> str`-- `pipelex/temporal/tprl/namespace_check.py:94` — `check_required_search_attributes` — `async def check_required_search_attributes(temporal_client: TemporalClient, *, namespace: str, configured_attributes: Sequence[str]) -> None`-- `pipelex/temporal/tprl/namespace_check.py:155` — `ensure_required_search_attributes_registered` — `async def ensure_required_search_attributes_registered(temporal_client: TemporalClient, *, namespace: str, configured_attributes: Sequence[str]) -> RegistrationFailure | tuple[str, ...]`-- `pipelex/temporal/tprl/observability.py:38` — `_truncate_utf8` — `def _truncate_utf8(text: str, *, max_bytes: int) -> str`-- `pipelex/temporal/tprl/observability.py:163` — `build_activity_summary` — `def build_activity_summary(method_label: str, *, job_metadata: JobMetadata, extras: Mapping[str, str] | None=None) -> str`-- `pipelex/temporal/tprl/temporal_error.py:232` — `TemporalError.from_message_exception` — `def from_message_exception(cls, exc: PipelexError, *, force_non_retryable: bool=False) -> Self`-- `pipelex/temporal/tprl/temporal_error.py:270` — `TemporalError._is_non_retryable` — `def _is_non_retryable(cls, exc: PipelexError, *, error_type: str) -> bool`-- `pipelex/temporal/tprl/workflow_caller.py:91` — `WorkflowExecutor.execute_workflow` — `async def execute_workflow(self, workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]], *, workflow_arg: WorkflowInput, workflow_id: str, search_attributes: TypedSearchAttributes | None=None, static_summary: str | None=None, static_details: str | None=None, memo: Mapping[str, Any] | None=None) -> WorkflowOutput`-- `pipelex/temporal/tprl/workflow_caller.py:140` — `WorkflowExecutor.start_workflow` — `async def start_workflow(self, workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]], *, workflow_arg: WorkflowInput, workflow_id: str, callbacks: Sequence[Callback] | None=None, search_attributes: TypedSearchAttributes | None=None, static_summary: str | None=None, static_details: str | None=None, memo: Mapping[str, Any] | None=None) -> WorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]`-- `pipelex/temporal/tprl/workflow_caller.py:180` — `WorkflowExecutor.execute_child_workflow` — `async def execute_child_workflow(self, workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]], *, workflow_arg: WorkflowInput, workflow_id: str, child_task_queue: str | None=None, search_attributes: TypedSearchAttributes | None=None, static_summary: str | None=None, static_details: str | None=None, memo: Mapping[str, Any] | None=None) -> WorkflowOutput`-- `pipelex/temporal/tprl/workflow_caller.py:246` — `WorkflowExecutor.start_child_workflow` — `async def start_child_workflow(self, workflow_class: type[WorkflowClass[WorkflowInput, WorkflowOutput]], *, workflow_arg: WorkflowInput, workflow_id: str, child_task_queue: str | None=None, search_attributes: TypedSearchAttributes | None=None, static_summary: str | None=None, static_details: str | None=None, memo: Mapping[str, Any] | None=None) -> ChildWorkflowHandle[WorkflowClass[WorkflowInput, WorkflowOutput], WorkflowOutput]`-- `pipelex/temporal/tprl/workflow_caller.py:298` — `WorkflowExecutorFactory.create_executor` — `def create_executor(cls, task_queue: str | None=None, *, workflow_execution_timeout: timedelta | None=None, retry_policy: RetryPolicy | None=None, run_timeout: timedelta | None=None, task_timeout: timedelta | None=None, start_delay: timedelta | None=None, rpc_timeout: timedelta | None=None, temporal_client: TemporalClient | None=None, should_auto_connect_temporal: bool=False, worker_environment: TemporalWorkerEnvironment=TemporalWorkerEnvironment.EXTERNAL) -> WorkflowExecutor[WorkflowInput, WorkflowOutput]`-- `pipelex/temporal/tprl_content_generation/content_generator_in_workflow.py:51` — `_revalidate_against_object_class` — `def _revalidate_against_object_class(raw_obj: BaseModel, *, object_class: type[BaseModelTypeVar], is_mock_built: bool) -> BaseModelTypeVar`-- `pipelex/temporal/tprl_pipe/dry_validate_dispatch.py:22` — `dispatch_dry_validate` — `async def dispatch_dry_validate(arg: DryValidateArg, *, task_queue: str | None=None, should_auto_connect_temporal: bool=True) -> DryValidateResult`-- `pipelex/temporal/tprl_pipe/temporal_pipe_router.py:127` — `make_temporal_pipe_router` — `def make_temporal_pipe_router(task_queue: str | None=None, *, workflow_execution_timeout: timedelta | None=None, retry_policy: RetryPolicy | None=None, should_auto_connect_temporal: bool=True, worker_environment: TemporalWorkerEnvironment=TemporalWorkerEnvironment.EXTERNAL) -> TemporalPipeRouter`-- `pipelex/temporal/tprl_pipe/temporal_pipe_run.py:92` — `TemporalPipeRun.start` — `async def start(self, pipe_job: PipeJob, *, delivery_assignment: DeliveryAssignment | None=None) -> tuple[str, WorkflowHandle[WorkflowClass[PipeRunArg, PipeOutput], PipeOutput]]`-- `pipelex/temporal/tprl_pipe/temporal_pipe_run.py:131` — `make_temporal_pipe_run` — `def make_temporal_pipe_run(task_queue: str | None=None, *, workflow_execution_timeout: timedelta | None=None, retry_policy: RetryPolicy | None=None, should_auto_connect_temporal: bool=True, worker_environment: TemporalWorkerEnvironment=TemporalWorkerEnvironment.EXTERNAL) -> TemporalPipeRun`-- `pipelex/temporal/worker_cli.py:25` — `run_worker` — `async def run_worker(project: str | None=None, *, is_not_sandboxed: bool=False, is_unit_testing: bool=False, task_queue: str | None=None, scope_name: str | None=None, profile_name: str | None=None)`--### `tools` (128)--- `pipelex/tools/jinja2/image_renderable.py:36` — `ImageRenderable.render_with_images` — `def render_with_images(self, registry: ImageRegistry, *, text_format: TextFormat) -> str`-- `pipelex/tools/jinja2/jinja2_environment.py:9` — `make_jinja2_env_from_loader` — `def make_jinja2_env_from_loader(template_category: TemplateCategory, *, loader: BaseLoader, enable_async: bool=True) -> Environment`-- `pipelex/tools/jinja2/jinja2_environment.py:57` — `_register_filters` — `def _register_filters(jinja2_env: Environment, *, template_category: TemplateCategory, enable_async: bool) -> None`-- `pipelex/tools/jinja2/jinja2_environment.py:81` — `make_jinja2_env_without_loader` — `def make_jinja2_env_without_loader(template_category: TemplateCategory, *, enable_async: bool=True) -> Environment`-- `pipelex/tools/jinja2/jinja2_environment.py:97` — `make_jinja2_env_from_registry` — `def make_jinja2_env_from_registry(template_category: TemplateCategory, *, enable_async: bool=True) -> Environment`-- `pipelex/tools/jinja2/jinja2_filters.py:110` — `apply_tag_style` — `def apply_tag_style(context: Context, *, value: str, tag_name: str | None=None) -> str`-- `pipelex/tools/jinja2/jinja2_parsing.py:8` — `check_jinja2_parsing` — `def check_jinja2_parsing(template_source: str, *, template_category: TemplateCategory=TemplateCategory.LLM_PROMPT)`-- `pipelex/tools/jinja2/jinja2_rendering.py:24` — `_add_to_templating_context` — `def _add_to_templating_context(templating_context: dict[str, Any], *, jinja2_context_key: Jinja2ContextKey, value: Any) -> None`-- `pipelex/tools/jinja2/jinja2_rendering.py:37` — `_compile_jinja2_template` — `def _compile_jinja2_template(template_source: str, *, template_category: TemplateCategory, use_registry: bool=False, enable_async: bool=True, finalize: Callable[[Any], Any] | None=None) -> _Jinja2Template`-- `pipelex/tools/jinja2/jinja2_rendering.py:66` — `_prepare_templating_context` — `def _prepare_templating_context(templating_context: dict[str, Any], *, templating_style: TemplatingStyle | None) -> dict[str, Any]`-- `pipelex/tools/jinja2/jinja2_rendering.py:87` — `_make_type_error_msg` — `def _make_type_error_msg(template_source: str, *, templating_context: dict[str, Any], type_error: TypeError) -> str`-- `pipelex/tools/jinja2/jinja2_rendering.py:97` — `_make_non_type_error_msg` — `def _make_non_type_error_msg(template_source: str, *, error_label: str, error: Jinja2StuffError | TemplateSyntaxError | UndefinedError | Jinja2ContextError) -> str`-- `pipelex/tools/jinja2/jinja2_rendering.py:106` — `_render_template_sync` — `def _render_template_sync(template_source: str, *, template: _Jinja2Template, templating_context: dict[str, Any]) -> str`-- `pipelex/tools/jinja2/jinja2_rendering.py:127` — `_render_template_async` — `async def _render_template_async(template_source: str, *, template: _Jinja2Template, templating_context: dict[str, Any]) -> str`-- `pipelex/tools/jinja2/jinja2_rendering.py:148` — `render_jinja2_sync` — `def render_jinja2_sync(template_source: str, *, template_category: TemplateCategory, templating_context: dict[str, Any], templating_style: TemplatingStyle | None=None, use_registry: bool=False) -> str`-- `pipelex/tools/jinja2/jinja2_rendering.py:173` — `render_jinja2_async` — `async def render_jinja2_async(template_source: str, *, template_category: TemplateCategory, templating_context: dict[str, Any], templating_style: TemplatingStyle | None=None, use_registry: bool=False, finalize: Callable[[Any], Any] | None=None) -> str`-- `pipelex/tools/jinja2/jinja2_required_variables.py:66` — `_collect_full_variable_paths` — `def _collect_full_variable_paths(node: nodes.Node, *, paths: set[str], declared_names: set[str]) -> None`-- `pipelex/tools/jinja2/jinja2_required_variables.py:121` — `detect_jinja2_required_variables` — `def detect_jinja2_required_variables(template_category: TemplateCategory, *, template_source: str) -> set[str]`-- `pipelex/tools/jinja2/jinja2_required_variables.py:185` — `_collect_variable_references` — `def _collect_variable_references(node: nodes.Node, *, references: dict[str, VariableReference], declared_names: set[str]) -> None`-- `pipelex/tools/jinja2/jinja2_required_variables.py:255` — `detect_jinja2_variable_references` — `def detect_jinja2_variable_references(template_category: TemplateCategory, *, template_source: str) -> list[VariableReference]`-- `pipelex/tools/jinja2/jinja2_with_images_filter.py:78` — `_render_sequence_with_images` — `def _render_sequence_with_images(sequence: list[Any] | tuple[Any, ...], *, registry: ImageRegistry, text_format: TextFormat) -> str`-- `pipelex/tools/log/log.py:203` — `Log.set_level_for_package` — `def set_level_for_package(self, package_name: str, *, level: LogLevel)`-- `pipelex/tools/log/log.py:224` — `Log.verbose` — `def verbose(self, content: str | Any, *, title: str | None=None, inline: str | None=None)`-- `pipelex/tools/log/log.py:243` — `Log.debug` — `def debug(self, content: str | Any, *, title: str | None=None, inline: str | None=None)`-- `pipelex/tools/log/log.py:262` — `Log.dev` — `def dev(self, content: str | Any, *, title: str | None=None, inline: str | None=None)`-- `pipelex/tools/log/log.py:281` — `Log.info` — `def info(self, content: str | Any, *, title: str | None=None, inline: str | None=None)`-- `pipelex/tools/log/log.py:300` — `Log.warning` — `def warning(self, content: str | Any, *, title: str | None=None, inline: str | None=None, problem_id: str | None=None)`-- `pipelex/tools/log/log.py:323` — `Log.error` — `def error(self, content: str | Any, *, title: str | None=None, inline: str | None=None, include_exception: bool=False, problem_id: str | None=None)`-- `pipelex/tools/log/log.py:354` — `Log.critical` — `def critical(self, content: str | Any, *, title: str | None=None, inline: str | None=None, include_exception: bool=False, problem_id: str | None=None)`-- `pipelex/tools/log/log_dispatch.py:66` — `LogDispatch.dispatch` — `def dispatch(self, content: str | Any, *, severity: int, title: str | None=None, inline: str | None=None, include_exception: bool=False)`-- `pipelex/tools/log/log_dispatch.py:125` — `LogDispatch._log_message` — `def _log_message(self, message: str, *, severity: int, caller_info_str: str | None, title: str | None=None, inline: str | None=None, include_exception: bool=False)`-- `pipelex/tools/log/log_dispatch.py:160` — `LogDispatch._log_data` — `def _log_data(self, data: Any, *, severity: int, caller_info_str: str | None, title: str | None=None, include_exception: bool=False)`-- `pipelex/tools/log/log_dispatch.py:234` — `LogDispatch._log_to_console` — `def _log_to_console(self, message: str, *, severity: int)`-- `pipelex/tools/mermaid/mermaid_utils.py:64` — `print_mermaid_url` — `def print_mermaid_url(url: str, *, title: str) -> None`-- `pipelex/tools/mermaid/mermaid_utils.py:142` — `render_mermaid_html_generic` — `def render_mermaid_html_generic(mermaid_code: str, *, title: str='Mermaid Diagram', theme: str='default') -> str`-- `pipelex/tools/mermaid/mermaid_utils.py:173` — `render_mermaid_html_generic_async` — `async def render_mermaid_html_generic_async(mermaid_code: str, *, title: str='Mermaid Diagram', theme: str='default') -> str`-- `pipelex/tools/misc/async_utils.py:19` — `gather_bounded` — `async def gather_bounded(task_factories: Sequence[Callable[[], Awaitable[_T]]], *, max_concurrency: int | None) -> list[_T]`-- `pipelex/tools/misc/attribute_utils.py:12` — `AttributePolisher._truncate_string` — `def _truncate_string(cls, value: str, *, max_length: int) -> str`-- `pipelex/tools/misc/attribute_utils.py:19` — `AttributePolisher._truncate_bytes` — `def _truncate_bytes(cls, value: bytes, *, max_length: int) -> bytes`-- `pipelex/tools/misc/attribute_utils.py:65` — `AttributePolisher.apply_truncation_recursive` — `def apply_truncation_recursive(cls, obj: Any, *, name: str | None=None) -> Any`-- `pipelex/tools/misc/base64_utils.py:38` — `make_base64_url` — `def make_base64_url(base64_data: str, *, file_type: FileType) -> str`-- `pipelex/tools/misc/context_provider_abstract.py:11` — `ContextProviderAbstract.get_typed_object_or_attribute` — `def get_typed_object_or_attribute(self, name: str, *, wanted_type: type[Any] | None=None, accept_list: bool=False) -> Any`-- `pipelex/tools/misc/dict_utils.py:14` — `insert_before` — `def insert_before(dictionary: dict[K, V], *, target_key: K, new_key: K, new_value: V) -> dict[K, V]`-- `pipelex/tools/misc/dict_utils.py:51` — `apply_to_strings_recursive` — `def apply_to_strings_recursive(data: Any, *, transform_func: Callable[[str], str]) -> dict[str, Any]`-- `pipelex/tools/misc/dict_utils.py:86` — `apply_to_strings_in_list` — `def apply_to_strings_in_list(data: list[Any], *, transform_func: Callable[[str], str]) -> list[Any]`-- `pipelex/tools/misc/dict_utils.py:102` — `substitute_nested_in_context` — `def substitute_nested_in_context(context: dict[str, Any], *, extra_params: dict[str, Any] | None=None) -> dict[str, Any]`-- `pipelex/tools/misc/diff.py:97` — `_generate_diff_summary` — `def _generate_diff_summary(diff_content: str, *, left_is_newer: bool) -> str | None`-- `pipelex/tools/misc/diff.py:241` — `make_diff_dirs_pretty._collect_diffs` — `def _collect_diffs(dir_comparison: filecmp.dircmp[str], *, relative_path: str='') -> None`-- `pipelex/tools/misc/file_fetch_utils.py:7` — `fetch_file_from_url_httpx` — `async def fetch_file_from_url_httpx(url: str, *, request_timeout: int | None=None) -> bytes`-- `pipelex/tools/misc/file_utils.py:13` — `reject_bare_str_or_path` — `def reject_bare_str_or_path(value: object, *, param_name: str) -> None`-- `pipelex/tools/misc/file_utils.py:38` — `save_bytes_to_binary_file` — `def save_bytes_to_binary_file(byte_data: bytes, *, file_path: Path, create_directory: bool=False) -> Path`-- `pipelex/tools/misc/file_utils.py:59` — `save_text_to_path` — `def save_text_to_path(text: str, *, path: Path, create_directory: bool=False)`-- `pipelex/tools/misc/file_utils.py:151` — `copy_file_from_package` — `def copy_file_from_package(package_name: str, *, file_path_in_package: str, target_path: Path, overwrite: bool=True) -> None`-- `pipelex/tools/misc/file_utils.py:167` — `copy_folder_from_package` — `def copy_folder_from_package(package_name: str, *, folder_path_in_package: str, target_dir: Path, overwrite: bool=True, non_overwrite_files: list[str] | None=None) -> None`-- `pipelex/tools/misc/file_utils.py:460` — `get_incremental_directory_path` — `def get_incremental_directory_path(base_path: Path, *, base_name: str, start_at: int=1) -> Path`-- `pipelex/tools/misc/file_utils.py:486` — `get_incremental_file_path` — `def get_incremental_file_path(base_path: Path, *, base_name: str, extension: str, start_at: int=1, avoid_suffix_if_possible: bool=False) -> Path`-- `pipelex/tools/misc/file_utils.py:532` — `find_files_in_dir` — `def find_files_in_dir(dir_path: Path, *, pattern: str, is_recursive: bool=True, excluded_dirs: list[str] | None=None, force_include_dirs: list[str] | None=None) -> list[Path]`-- `pipelex/tools/misc/hash_utils.py:4` — `hash_sha256` — `def hash_sha256(data: str | bytes, *, length: int | None=None) -> str`-- `pipelex/tools/misc/image_utils.py:95` — `pil_image_to_bytes` — `def pil_image_to_bytes(pil_image: Image.Image, *, image_format: ImageFormat | None) -> bytes`-- `pipelex/tools/misc/json_utils.py:71` — `clean_json_dumps` — `def clean_json_dumps(data: Any, *, indent: int | None=None) -> str`-- `pipelex/tools/misc/json_utils.py:92` — `json_str` — `def json_str(some_object: Any, *, title: str | None=None, is_spaced: bool=False) -> str`-- `pipelex/tools/misc/json_utils.py:128` — `save_as_json_to_path` — `def save_as_json_to_path(object_to_save: Any, *, path: Path, indent: int | None=4, is_warning_enabled: bool=True, create_directory: bool=False)`-- `pipelex/tools/misc/json_utils.py:227` — `deep_update` — `def deep_update(target_dict: dict[str, Any], *, updates: Mapping[str, Any])`-- `pipelex/tools/misc/json_utils.py:315` — `purify_json` — `def purify_json(data: Any, *, indent: int | None=None, is_truncate_bytes_enabled: bool=False, is_warning_enabled: bool=True) -> tuple[dict[Any, Any] | list[Any], str]`-- `pipelex/tools/misc/json_utils.py:395` — `purify_json_list` — `def purify_json_list(data: list[Any], *, indent: int | None=None, is_truncate_bytes_enabled: bool=False) -> tuple[list[Any], str]`-- `pipelex/tools/misc/json_utils.py:453` — `purify_json_dict` — `def purify_json_dict(data: Any, *, indent: int | None=None, is_warning_enabled: bool=True) -> tuple[dict[str, Any], str]`-- `pipelex/tools/misc/json_utils.py:508` — `pure_json_str` — `def pure_json_str(data: Any, *, indent: int | None=None, is_warning_enabled: bool=True) -> str`-- `pipelex/tools/misc/markdown_utils.py:7` — `convert_to_markdown` — `def convert_to_markdown(data: Any, *, level: int=1, is_pretty: bool=False) -> str`-- `pipelex/tools/misc/pretty.py:99` — `pretty_print` — `def pretty_print(content: str | Any, *, title: TextType | None=None, subtitle: TextType | None=None, inner_title: str | None=None, border_style: StyleType | None=None, width: int | None=None, console_width: int | None=None)`-- `pipelex/tools/misc/pretty.py:120` — `pretty_print_md` — `def pretty_print_md(content: str, *, title: TextType | None=None, subtitle: TextType | None=None, inner_title: str | None=None, border_style: StyleType | None=None, width: int | None=None, console_width: int | None=None)`-- `pipelex/tools/misc/pretty.py:143` — `pretty_print_url` — `def pretty_print_url(url: str, *, title: TextType | None=None, subtitle: TextType | None=None, inner_title: str | None=None, border_style: StyleType | None=None, width: int | None=None, console_width: int | None=None)`-- `pipelex/tools/misc/pretty.py:170` — `PrettyPrinter.pretty_print` — `def pretty_print(cls, content: str | Any, *, title: TextType | None=None, subtitle: TextType | None=None, inner_title: str | None=None, border_style: StyleType | None=None, width: int | None=None, console_width: int | None=None)`-- `pipelex/tools/misc/pretty.py:198` — `PrettyPrinter.pretty_print_using_rich` — `def pretty_print_using_rich(cls, content: str | Any, *, title: TextType | None=None, subtitle: TextType | None=None, inner_title: str | None=None, border_style: StyleType | None=None, width: int | None=None, console_width: int | None=None)`-- `pipelex/tools/misc/pretty.py:222` — `PrettyPrinter.pretty_width` — `def pretty_width(cls, width: int | None=None, *, depth: int | None=None) -> int`-- `pipelex/tools/misc/pretty.py:234` — `PrettyPrinter.make_pretty_panel` — `def make_pretty_panel(cls, content: str | Any, *, title: TextType | None=None, subtitle: TextType | None=None, inner_title: str | None=None, border_style: StyleType | None=None, width: int | None=None, console_width: int | None=None) -> Panel`-- `pipelex/tools/misc/pretty.py:259` — `PrettyPrinter.wrap_in_panel` — `def wrap_in_panel(cls, pretty: PrettyPrintable, *, title: TextType | None=None, subtitle: TextType | None=None, border_style: StyleType | None=None, width: int | None=None) -> Panel`-- `pipelex/tools/misc/pretty.py:282` — `PrettyPrinter.pretty_text` — `def pretty_text(cls, pretty: PrettyPrintable, *, width: int=PRETTY_WIDTH_FOR_EXPORT) -> str`-- `pipelex/tools/misc/pretty.py:303` — `PrettyPrinter.pretty_html` — `def pretty_html(cls, pretty: PrettyPrintable, *, width: int=PRETTY_WIDTH_FOR_EXPORT) -> str`-- `pipelex/tools/misc/pretty.py:315` — `PrettyPrinter.pretty_svg` — `def pretty_svg(cls, pretty: PrettyPrintable, *, width: int=PRETTY_WIDTH_FOR_EXPORT) -> str`-- `pipelex/tools/misc/pretty.py:322` — `PrettyPrinter.make_pretty` — `def make_pretty(cls, value: Any, *, inner_title: str | None=None, depth: int=0) -> PrettyPrintable`-- `pipelex/tools/misc/pretty.py:381` — `PrettyPrinter.pretty_print_without_rich` — `def pretty_print_without_rich(cls, content: str | Any, *, title: TextType | None=None, subtitle: TextType | None=None, inner_title: str | None=None, width: int | None=None, console_width: int | None=None)`-- `pipelex/tools/misc/pretty.py:438` — `PrettyPrinter.pretty_print_url_without_rich` — `def pretty_print_url_without_rich(cls, content: str | Any, *, title: TextType | None=None, subtitle: TextType | None=None)`-- `pipelex/tools/misc/semver.py:59` — `version_satisfies` — `def version_satisfies(version: Version, *, constraint: SimpleSpec) -> bool`-- `pipelex/tools/misc/semver.py:73` — `select_minimum_version` — `def select_minimum_version(available_versions: list[Version], *, constraint: SimpleSpec) -> Version | None`-- `pipelex/tools/misc/semver.py:96` — `select_minimum_version_for_multiple_constraints` — `def select_minimum_version_for_multiple_constraints(available_versions: list[Version], *, constraints: list[SimpleSpec]) -> Version | None`-- `pipelex/tools/misc/string_utils.py:335` — `matches_wildcard_pattern` — `def matches_wildcard_pattern(text: str, *, pattern: str) -> bool`-- `pipelex/tools/misc/toml_sync.py:42` — `get_nested_value` — `def get_nested_value(doc: TOMLDocument | Table | dict[str, Any], *, key_path: str) -> tuple[bool, Any]`-- `pipelex/tools/misc/toml_sync.py:69` — `set_nested_value` — `def set_nested_value(doc: TOMLDocument | Table | dict[str, Any], *, key_path: str, value: Any) -> bool`-- `pipelex/tools/misc/toml_sync.py:120` — `collect_leaf_key_paths` — `def collect_leaf_key_paths(doc: TOMLDocument | Table | dict[str, Any], *, prefix: str='') -> list[str]`-- `pipelex/tools/misc/toml_utils.py:67` — `save_toml_to_path` — `def save_toml_to_path(data: dict[str, Any] | tomlkit.TOMLDocument, *, path: str | Path) -> None`-- `pipelex/tools/network/ssrf_guard.py:37` — `resolve_to_allowed_ips` — `async def resolve_to_allowed_ips(host: str, *, port: int, timeout: float | None=None) -> list[str]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:68` — `_extract_image_from_pdf_object` — `def _extract_image_from_pdf_object(image_obj: PdfImage, *, output_format: ImageFormat | None) -> ExtractedImageFromPage`-- `pipelex/tools/pdf/pypdfium2_renderer.py:159` — `PyPdfium2Renderer._render_pdf_pages_sync` — `def _render_pdf_pages_sync(pdf_input: PdfInput, *, scale: float) -> list[Image.Image]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:190` — `PyPdfium2Renderer._extract_embedded_images_from_page_sync` — `def _extract_embedded_images_from_page_sync(pdf_input: PdfInput, *, page_index: int, max_depth: int=DEFAULT_IMAGE_EXTRACTION_MAX_DEPTH, output_format: ImageFormat | None=None) -> list[ExtractedImageFromPage]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:227` — `PyPdfium2Renderer._extract_embedded_images_from_pdf_sync` — `def _extract_embedded_images_from_pdf_sync(pdf_input: PdfInput, *, max_depth: int=DEFAULT_IMAGE_EXTRACTION_MAX_DEPTH, output_format: ImageFormat | None=None) -> dict[int, list[ExtractedImageFromPage]]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:267` — `PyPdfium2Renderer.render_pdf_pages` — `async def render_pdf_pages(self, pdf_input: PdfInput, *, dpi: int) -> list[Image.Image]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:278` — `PyPdfium2Renderer.render_pdf_pages_from_uri` — `async def render_pdf_pages_from_uri(self, pdf_uri: str, *, dpi: int) -> list[Image.Image]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:287` — `PyPdfium2Renderer.extract_embedded_images_from_page` — `async def extract_embedded_images_from_page(self, pdf_input: PdfInput, *, page_index: int, max_depth: int=DEFAULT_IMAGE_EXTRACTION_MAX_DEPTH, output_format: ImageFormat | None=None) -> list[ExtractedImageFromPage]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:316` — `PyPdfium2Renderer.extract_embedded_images_from_pdf` — `async def extract_embedded_images_from_pdf(self, pdf_input: PdfInput, *, max_depth: int=DEFAULT_IMAGE_EXTRACTION_MAX_DEPTH, output_format: ImageFormat | None=None) -> dict[int, list[ExtractedImageFromPage]]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:342` — `PyPdfium2Renderer.extract_embedded_images_from_pdf_uri` — `async def extract_embedded_images_from_pdf_uri(self, pdf_uri: str, *, max_depth: int=DEFAULT_IMAGE_EXTRACTION_MAX_DEPTH, output_format: ImageFormat | None=None) -> dict[int, list[ExtractedImageFromPage]]`-- `pipelex/tools/secrets/secrets_provider_abstract.py:14` — `SecretsProviderAbstract.get_required_secret_specific_version` — `def get_required_secret_specific_version(self, secret_id: str, *, version_id: str) -> str`-- `pipelex/tools/secrets/secrets_provider_abstract.py:17` — `SecretsProviderAbstract.get_optional_secret_specific_version` — `def get_optional_secret_specific_version(self, secret_id: str, *, version_id: str) -> str | None`-- `pipelex/tools/secrets/secrets_provider_abstract.py:20` — `SecretsProviderAbstract.set_secret_as_env_var` — `def set_secret_as_env_var(self, secret_id: str, *, version_id: str=LATEST_SECRET_VERSION_NAME)`-- `pipelex/tools/secrets/secrets_utils.py:22` — `substitute_vars` — `def substitute_vars(content: str, *, secrets_provider: SecretsProviderAbstract, raise_on_missing_var: bool=True) -> str`-- `pipelex/tools/secrets/secrets_utils.py:92` — `_handle_fallback_pattern` — `def _handle_fallback_pattern(var_spec: str, *, secrets_provider: SecretsProviderAbstract) -> str`-- `pipelex/tools/secrets/secrets_utils.py:139` — `_get_secret` — `def _get_secret(secret_name: str, *, secrets_provider: SecretsProviderAbstract) -> str`-- `pipelex/tools/storage/gcp_storage_provider.py:134` — `GcpStorageProvider._store_sync` — `def _store_sync(self, data: bytes, *, key: str, content_type: str | None) -> None`-- `pipelex/tools/storage/storage_provider_abstract.py:97` — `StorageProviderAbstract.store` — `async def store(self, data: bytes, *, key: str, content_type: str | None=None) -> str`-- `pipelex/tools/storage/storage_provider_abstract.py:113` — `StorageProviderAbstract._store` — `async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None`-- `pipelex/tools/tabular/csv_codec.py:141` — `_validate_header` — `def _validate_header(header: list[str], *, path: Path) -> None`-- `pipelex/tools/tabular/csv_codec.py:160` — `_read_table` — `def _read_table(path: Path, *, delimiter: str, encoding: str) -> tuple[list[str], list[tuple[int, list[str]]]]`-- `pipelex/tools/tabular/csv_codec.py:219` — `_row_to_dict` — `def _row_to_dict(data_row: list[str], *, header: list[str]) -> dict[str, str]`-- `pipelex/tools/tabular/csv_codec.py:224` — `read_rows` — `def read_rows(path: Path, *, delimiter: str=DEFAULT_DELIMITER, encoding: str=DEFAULT_READ_ENCODING) -> list[dict[str, str]]`-- `pipelex/tools/tabular/csv_codec.py:312` — `list_content_from_csv` — `def list_content_from_csv(path: Path, *, row_model: type[StuffContentType], delimiter: str=DEFAULT_DELIMITER, encoding: str=DEFAULT_READ_ENCODING) -> ListContent[StuffContentType]`-- `pipelex/tools/tabular/csv_codec.py:379` — `csv_from_list_content` — `def csv_from_list_content(list_content: ListContent[StuffContentType], *, row_model: type[StuffContentType], path: Path, delimiter: str=DEFAULT_DELIMITER, encoding: str=DEFAULT_ENCODING) -> None`-- `pipelex/tools/typing/class_utils.py:122` — `has_compatible_field` — `def has_compatible_field(model_cls: type[Any], *, target_type: type[Any]) -> bool`-- `pipelex/tools/typing/module_inspector.py:100` — `find_class_names_in_file` — `def find_class_names_in_file(file_path: Path, *, base_class_names: list[str] | None=None) -> list[str]`-- `pipelex/tools/typing/module_inspector.py:161` — `find_decorated_function_names_in_file` — `def find_decorated_function_names_in_file(file_path: Path, *, decorator_names: list[str]) -> list[str]`-- `pipelex/tools/typing/module_inspector.py:226` — `import_module_from_file_if_has_decorated_functions` — `def import_module_from_file_if_has_decorated_functions(file_path: Path, *, decorator_names: list[str]) -> Any | None`-- `pipelex/tools/typing/module_inspector.py:259` — `import_module_from_file_if_has_classes` — `def import_module_from_file_if_has_classes(file_path: Path, *, base_class_names: list[str] | None=None) -> Any | None`-- `pipelex/tools/typing/module_inspector.py:294` — `find_classes_in_module` — `def find_classes_in_module(module: Any, *, base_class: type[Any] | None, include_imported: bool) -> list[type[Any]]`-- `pipelex/tools/typing/pydantic_utils.py:31` — `empty_dict_factory_of` — `def empty_dict_factory_of(_key: type[K], *, _val: type[V] | None=None) -> Callable[[], dict[K, Any]]`-- `pipelex/tools/typing/pydantic_utils.py:317` — `serialize_model` — `def serialize_model(obj: Any, *, field_visibility: FieldVisibility=FieldVisibility.NO_HIDDEN_FIELDS, is_stringify_enums: bool=True) -> dict[str, Any] | list[Any] | Any`-- `pipelex/tools/typing/structure_printer.py:69` — `StructurePrinter.get_type_structure` — `def get_type_structure(self, tp: type[Any], *, seen_types: set[str] | None=None, collected_types: dict[str, type[Any]] | None=None, collected_enums: dict[str, type[Enum]] | None=None, base_class: type[Any]=BaseModel) -> list[str]`-- `pipelex/tools/typing/validation_utils.py:4` — `has_exactly_one_among_attributes_from_list` — `def has_exactly_one_among_attributes_from_list(obj: Any, *, attributes_list: list[str]) -> bool`-- `pipelex/tools/typing/validation_utils.py:23` — `has_more_than_one_among_attributes_from_list` — `def has_more_than_one_among_attributes_from_list(obj: Any, *, attributes_list: list[str]) -> bool`-- `pipelex/tools/typing/validation_utils.py:42` — `has_more_than_one_among_attributes_from_lists` — `def has_more_than_one_among_attributes_from_lists(obj: Any, *, attributes_lists: list[list[str]]) -> list[str] | None`-- `pipelex/tools/uri/uri_resolver.py:160` — `make_base64_url_from_any_uri` — `async def make_base64_url_from_any_uri(uri: str, *, storage_provider: StorageProviderAbstract | None=None) -> str`--### `tracing` (5)--- `pipelex/tracing/activity_event_log.py:80` — `ActivityEventLogCache.log_once_runner_fallback_engaged` — `def log_once_runner_fallback_engaged(cls, workflow_id: str, *, writer_id: str) -> None`-- `pipelex/tracing/dynamodb_event_log.py:99` — `DynamoDBEventLog._make_sk` — `def _make_sk(workflow_id: str, *, writer_id: str, sequence: int) -> str`-- `pipelex/tracing/event_log_factory.py:10` — `make_event_log` — `def make_event_log(tracing_config: TracingConfig, *, writer_id: str='primary') -> EventLogProtocol`-- `pipelex/tracing/graphspec_assembler.py:116` — `GraphSpecAssembler.assemble` — `def assemble(events: Sequence[TraceEvent], *, graph_id: str, pipeline_ref: PipelineRef | None=None) -> GraphSpec`-- `pipelex/tracing/ndjson_event_log.py:73` — `NdjsonEventLog._file_name_for` — `def _file_name_for(workflow_id: str, *, writer_id: str) -> str`--## B. `LONE_SUBJECT` — single positional param (low signal)--_A single positional parameter and nothing else after it (`def f(subject)`, optionally `**kwargs`). No "choice" of subject was possible — only one arg exists. Lowest signal; included for completeness. A bare bool/int/str subject here is the only thing arguably worth a second look (`do_thing(True)` reads opaquely)._--Count: 1003--### `base_exceptions.py` (8)--- `pipelex/base_exceptions.py:12` — `iter_cause_chain` — `def iter_cause_chain(exc: BaseException) -> Iterator[BaseException]`-- `pipelex/base_exceptions.py:66` — `_redact_provider_metadata_for_strict` — `def _redact_provider_metadata_for_strict(metadata_payload: dict[str, Any]) -> dict[str, Any] | None`-- `pipelex/base_exceptions.py:178` — `error_domain_to_http_status` — `def error_domain_to_http_status(error_domain: ErrorDomain | str | None) -> int`-- `pipelex/base_exceptions.py:209` — `error_domain_is_input` — `def error_domain_is_input(error_domain: ErrorDomain | str | None) -> bool`-- `pipelex/base_exceptions.py:268` — `ErrorReport.to_dict` — `def to_dict(self, disclosure_mode: DisclosureMode=DisclosureMode.VERBOSE) -> dict[str, Any]`-- `pipelex/base_exceptions.py:360` — `ErrorReport.from_dict` — `def from_dict(cls, data: dict[str, Any]) -> 'ErrorReport'`-- `pipelex/base_exceptions.py:397` — `_humanize_class_name` — `def _humanize_class_name(class_name: str) -> str`-- `pipelex/base_exceptions.py:510` — `PipelexError._enrich_error_report_from_cause` — `def _enrich_error_report_from_cause(self, report: ErrorReport) -> ErrorReport`--### `builder` (16)--- `pipelex/builder/concept/concept_spec.py:339` — `ConceptSpec._format_type_display` — `def _format_type_display(self, field_spec: ConceptStructureSpec) -> str`-- `pipelex/builder/operations/concept_ops.py:14` — `parse_concept_spec` — `def parse_concept_spec(spec_data: Any) -> ConceptSpec`-- `pipelex/builder/operations/concept_ops.py:95` — `structure_field_to_dict` — `def structure_field_to_dict(field_spec: ConceptStructureSpec) -> dict[str, Any]`-- `pipelex/builder/operations/concept_ops.py:127` — `concept_spec_to_toml` — `def concept_spec_to_toml(concept_spec: ConceptSpec) -> str`-- `pipelex/builder/operations/models_ops.py:240` — `format_models_markdown` — `def format_models_markdown(result: dict[str, Any]) -> str`-- `pipelex/builder/operations/pipe_ops.py:39` — `_normalize_sub_pipe_dict` — `def _normalize_sub_pipe_dict(data: dict[str, Any]) -> None`-- `pipelex/builder/operations/pipe_ops.py:69` — `_normalize_pipe_code_aliases` — `def _normalize_pipe_code_aliases(data: dict[str, Any]) -> None`-- `pipelex/builder/operations/pipe_ops.py:79` — `_normalize_prompt_aliases` — `def _normalize_prompt_aliases(data: dict[str, Any]) -> None`-- `pipelex/builder/operations/pipe_ops.py:303` — `pipe_spec_to_toml` — `def pipe_spec_to_toml(pipe_spec: PipeSpec) -> str`-- `pipelex/builder/operations/validate_ops.py:22` — `validate_all` — `async def validate_all(library_dirs: list[Path] | None=None) -> dict[str, Any]`-- `pipelex/builder/operations/validate_ops.py:82` — `validate_bundle_content` — `async def validate_bundle_content(mthds_contents: list[str]) -> dict[str, Any]`-- `pipelex/builder/pipe/pipe_spec.py:132` — `PipeSpec.validate_pipe_code_syntax` — `def validate_pipe_code_syntax(cls, pipe_code: str) -> str`-- `pipelex/builder/runner_code.py:48` — `_is_multiple` — `def _is_multiple(multiplicity: VariableMultiplicity | None) -> bool`-- `pipelex/builder/runner_code.py:86` — `_get_structure_class_import` — `def _get_structure_class_import(class_name: str) -> str | None`-- `pipelex/builder/runner_code.py:104` — `_collect_concept_info` — `def _collect_concept_info(concept: Concept) -> CustomClassInfo | None`-- `pipelex/builder/runner_code.py:125` — `_collect_imports_for_inputs` — `def _collect_imports_for_inputs(inputs: InputStuffSpecs) -> tuple[set[str], dict[str, CustomClassInfo]]`--### `cli` (98)--- `pipelex/cli/_cli.py:98` — `version_callback` — `def version_callback(value: bool) -> None`-- `pipelex/cli/agent_cli/_agent_cli.py:73` — `version_callback` — `def version_callback(value: bool) -> None`-- `pipelex/cli/agent_cli/commands/agent_output.py:41` — `record_setup_warning` — `def record_setup_warning(warning_payload: dict[str, Any]) -> None`-- `pipelex/cli/agent_cli/commands/agent_output.py:76` — `set_agent_cli_error_format` — `def set_agent_cli_error_format(error_format: CliOutputFormat) -> None`-- `pipelex/cli/agent_cli/commands/agent_output.py:187` — `_build_error_source` — `def _build_error_source(exc: BaseException) -> list[str]`-- `pipelex/cli/agent_cli/commands/agent_output.py:290` — `_render_error_markdown` — `def _render_error_markdown(payload: dict[str, Any]) -> str`-- `pipelex/cli/agent_cli/commands/agent_output.py:361` — `agent_success` — `def agent_success(result: dict[str, Any]) -> None`-- `pipelex/cli/agent_cli/commands/agent_output.py:410` — `extract_validation_errors` — `def extract_validation_errors(exc: ValidateBundleError) -> list[dict[str, Any]]`-- `pipelex/cli/agent_cli/commands/check_model_cmd.py:21` — `_format_check_markdown` — `def _format_check_markdown(result: dict[str, Any]) -> str`-- `pipelex/cli/agent_cli/commands/concept_cmd.py:19` — `_concept_spec_to_toml` — `def _concept_spec_to_toml(concept_spec: ConceptSpec) -> str`-- `pipelex/cli/agent_cli/commands/doctor_cmd.py:27` — `_status_icon` — `def _status_icon(healthy: bool) -> str`-- `pipelex/cli/agent_cli/commands/doctor_cmd.py:32` — `_format_doctor_markdown` — `def _format_doctor_markdown(result: dict[str, Any]) -> str`-- `pipelex/cli/agent_cli/commands/init_cmd.py:34` — `_parse_config_arg` — `def _parse_config_arg(config_arg: str | None) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/init_cmd.py:67` — `_format_init_markdown` — `def _format_init_markdown(result: dict[str, Any]) -> str`-- `pipelex/cli/agent_cli/commands/init_cmd.py:88` — `_resolve_target_dir` — `def _resolve_target_dir(global_: bool) -> Path`-- `pipelex/cli/agent_cli/commands/init_cmd.py:110` — `_copy_inference_templates` — `def _copy_inference_templates(target_dir: Path) -> None`-- `pipelex/cli/agent_cli/commands/pipe_cmd.py:36` — `_pipe_spec_to_toml` — `def _pipe_spec_to_toml(pipe_spec: PipeSpec) -> str`-- `pipelex/cli/agent_cli/commands/run/_output_helpers.py:52` — `_render_json_payload_lines` — `def _render_json_payload_lines(result_payload: Any) -> list[str]`-- `pipelex/cli/agent_cli/commands/run/stdin_resolver.py:19` — `_extract_concept_code` — `def _extract_concept_code(concept_data: Any) -> str`-- `pipelex/cli/agent_cli/commands/run/stdin_resolver.py:37` — `_extract_stuff_entry` — `def _extract_stuff_entry(stuff_data: dict[str, Any]) -> dict[str, Any] | None`-- `pipelex/cli/agent_cli/commands/run/stdin_resolver.py:57` — `resolve_stdin_inputs` — `def resolve_stdin_inputs(stdin_data: dict[str, Any]) -> dict[str, Any]`-- `pipelex/cli/agent_cli/commands/run/stdin_resolver.py:155` — `_parse_inputs_arg` — `def _parse_inputs_arg(inputs_arg: str) -> dict[str, Any] | None`-- `pipelex/cli/agent_cli/commands/validate/_output_helpers.py:8` — `format_validate_markdown` — `def format_validate_markdown(result: dict[str, Any]) -> str`-- `pipelex/cli/commands/build/structures_cmd.py:34` — `_compute_relative_path_from_output_dir` — `def _compute_relative_path_from_output_dir(output_directory: Path) -> Path | None`-- `pipelex/cli/commands/doctor_cmd.py:109` — `check_config_files` — `def check_config_files(config_dir: Path | None=None) -> tuple[bool, int, str]`-- `pipelex/cli/commands/doctor_cmd.py:162` — `check_telemetry_config` — `def check_telemetry_config(config_dir: Path | None=None) -> tuple[bool, str]`-- `pipelex/cli/commands/doctor_cmd.py:191` — `check_backend_credentials` — `def check_backend_credentials(config_dir: Path | None=None) -> tuple[bool, dict[str, BackendCredentialsReport], str]`-- `pipelex/cli/commands/doctor_cmd.py:268` — `check_kit_template_exists` — `def check_kit_template_exists(backend_name: str) -> bool`-- `pipelex/cli/commands/doctor_cmd.py:333` — `check_backend_files` — `def check_backend_files(config_dir: Path | None=None) -> tuple[bool, dict[str, BackendFileReport], str]`-- `pipelex/cli/commands/doctor_cmd.py:706` — `check_deck_sync` — `def check_deck_sync(config_dir: Path | None=None) -> tuple[bool, DeckSyncReport, str]`-- `pipelex/cli/commands/doctor_cmd.py:796` — `check_models` — `def check_models(config_dir: Path | None=None) -> tuple[bool, str, dict[str, BackendFileReport]]`-- `pipelex/cli/commands/doctor_cmd.py:894` — `doctor_cmd` — `def doctor_cmd(fix: bool=False) -> None`-- `pipelex/cli/commands/doctor_cmd.py:918` — `do_doctor_cmd` — `def do_doctor_cmd(fix: bool=False) -> None`-- `pipelex/cli/commands/init/backends.py:49` — `get_selected_backend_keys` — `def get_selected_backend_keys(backends_toml_path: Path) -> list[str]`-- `pipelex/cli/commands/init/backends.py:76` — `disable_gateway_backend` — `def disable_gateway_backend(backends_toml_path: Path) -> None`-- `pipelex/cli/commands/init/command.py:61` — `attempt_prime_remote_config_cache` — `def attempt_prime_remote_config_cache(target_config_dir: Path | None=None) -> CachePrimingResult`-- `pipelex/cli/commands/init/command.py:427` — `_init_agreement` — `def _init_agreement(console: Console) -> None`-- `pipelex/cli/commands/init/credentials.py:23` — `read_env_file` — `def read_env_file(env_path: Path) -> dict[str, str]`-- `pipelex/cli/commands/init/credentials.py:72` — `get_required_vars_for_enabled_backends` — `def get_required_vars_for_enabled_backends(backends_toml_path: Path) -> dict[str, list[str]]`-- `pipelex/cli/commands/init/ide_extension.py:26` — `_is_extension_installed` — `def _is_extension_installed(cmd: str) -> bool`-- `pipelex/cli/commands/init/ide_extension.py:89` — `suggest_extension_install_if_needed` — `def suggest_extension_install_if_needed(console: Console) -> None`-- `pipelex/cli/commands/init/ui/gateway_ui.py:94` — `prompt_gateway_acceptance` — `def prompt_gateway_acceptance(console: Console) -> bool`-- `pipelex/cli/commands/init/ui/gateway_ui.py:114` — `display_gateway_declined_message` — `def display_gateway_declined_message(console: Console) -> None`-- `pipelex/cli/commands/init/ui/gateway_ui.py:136` — `display_gateway_accepted_message` — `def display_gateway_accepted_message(console: Console) -> None`-- `pipelex/cli/commands/login/command.py:115` — `save_api_key` — `def save_api_key(api_key: str) -> None`-- `pipelex/cli/commands/run/_inputs_path_resolver.py:19` — `is_relative_local_path` — `def is_relative_local_path(uri: str) -> bool`-- `pipelex/cli/commands/show_cmd.py:65` — `do_show_pipe` — `def do_show_pipe(pipe_code: str) -> None`-- `pipelex/cli/commands/show_cmd.py:72` — `do_show_backends` — `def do_show_backends(show_all: bool=False) -> None`-- `pipelex/cli/commands/update_cmd.py:91` — `_resolve_deck_dir` — `def _resolve_deck_dir(local: bool) -> Path`-- `pipelex/cli/commands/update_cmd.py:124` — `_action_description` — `def _action_description(status: DeckFileStatus) -> str`-- `pipelex/cli/commands/validate/_validate_core.py:47` — `_format_signatures_summary_suffix` — `def _format_signatures_summary_suffix(signature_count: int) -> str`-- `pipelex/cli/dev_cli/commands/check_keyword_only_cmd.py:29` — `_print_report` — `def _print_report(violations: list[Violation]) -> None`-- `pipelex/cli/dev_cli/commands/check_keyword_only_cmd.py:136` — `_print_violation_lines` — `def _print_violation_lines(violations: list[Violation]) -> None`-- `pipelex/cli/dev_cli/commands/gateway_models_generator.py:30` — `normalize_for_comparison` — `def normalize_for_comparison(content: str) -> str`-- `pipelex/cli/dev_cli/commands/gateway_models_generator.py:54` — `extract_reference_data` — `def extract_reference_data(model_specs: BackendModelSpecs) -> dict[str, list[dict[str, Any]]]`-- `pipelex/cli/dev_cli/commands/gateway_models_generator.py:107` — `generate_pure_markdown_list` — `def generate_pure_markdown_list(models: list[dict[str, Any]]) -> str`-- `pipelex/cli/dev_cli/commands/gateway_models_generator.py:145` — `generate_markdown_table` — `def generate_markdown_table(models: list[dict[str, Any]]) -> str`-- `pipelex/cli/dev_cli/commands/gateway_models_generator.py:222` — `generate_reference_markdown` — `def generate_reference_markdown(model_specs: BackendModelSpecs) -> str`-- `pipelex/cli/dev_cli/commands/gateway_models_generator.py:274` — `generate_reference_pure_markdown` — `def generate_reference_pure_markdown(model_specs: BackendModelSpecs) -> str`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:124` — `_attribute_tail` — `def _attribute_tail(node: ast.expr) -> tuple[str, ...]`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:137` — `_decorator_matches_carveout` — `def _decorator_matches_carveout(decorator: ast.expr) -> bool`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:165` — `_has_carveout_decorator` — `def _has_carveout_decorator(node: ast.FunctionDef | ast.AsyncFunctionDef) -> bool`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:170` — `_iter_subscript_metadata` — `def _iter_subscript_metadata(annotation: ast.expr) -> Iterator[ast.expr]`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:179` — `_callee_name` — `def _callee_name(func: ast.expr) -> str | None`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:188` — `_annotation_has_typer_metadata` — `def _annotation_has_typer_metadata(annotation: ast.expr | None) -> bool`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:203` — `_has_typer_param_annotation` — `def _has_typer_param_annotation(node: ast.FunctionDef | ast.AsyncFunctionDef) -> bool`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:227` — `_is_dunder` — `def _is_dunder(name: str) -> bool`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:298` — `_Collector._visit_function` — `def _visit_function(self, node: ast.FunctionDef | ast.AsyncFunctionDef) -> None`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:349` — `_module_qname_for` — `def _module_qname_for(path: Path) -> str`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:357` — `iter_source_files` — `def iter_source_files(root: Path) -> Iterator[Path]`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:365` — `collect_all_violations` — `def collect_all_violations(root: Path) -> list[Violation]`-- `pipelex/cli/dev_cli/commands/keyword_only_guard.py:443` — `main` — `def main(argv: list[str]) -> int`-- `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:42` — `_extract_models_from_backend_toml` — `def _extract_models_from_backend_toml(backend_path: Path) -> dict[str, list[str]]`-- `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:222` — `_load_test_profile` — `def _load_test_profile(profile_name: str) -> dict[str, Any]`-- `pipelex/cli/dev_cli/commands/refresh_graph_ui_sri_cmd.py:93` — `_validate_sri` — `def _validate_sri(value: str) -> str`-- `pipelex/cli/dev_cli/commands/refresh_graph_ui_sri_cmd.py:116` — `_sha384_sri` — `def _sha384_sri(payload: bytes) -> str`-- `pipelex/cli/dev_cli/commands/refresh_graph_ui_sri_cmd.py:122` — `_python_string_literal` — `def _python_string_literal(value: str) -> str`-- `pipelex/cli/dev_cli/commands/sync_main_config_cmd.py:30` — `_format_value` — `def _format_value(value: object) -> str`-- `pipelex/cli/dev_cli/commands/update_gateway_models_cmd.py:65` — `update_gateway_models_cmd` — `def update_gateway_models_cmd(quiet: bool=False) -> None`-- `pipelex/cli/error_handlers.py:57` — `set_traceback_requested` — `def set_traceback_requested(value: bool) -> None`-- `pipelex/cli/error_handlers.py:80` — `print_traceback_if_requested` — `def print_traceback_if_requested(console: Console) -> None`-- `pipelex/cli/error_handlers.py:370` — `handle_inference_setup_required_error` — `def handle_inference_setup_required_error(exc: InferenceSetupRequiredError) -> NoReturn`-- `pipelex/cli/error_handlers.py:396` — `handle_telemetry_config_validation_error` — `def handle_telemetry_config_validation_error(exc: TelemetryConfigValidationError) -> NoReturn`-- `pipelex/cli/error_handlers.py:426` — `handle_gateway_terms_not_accepted_error` — `def handle_gateway_terms_not_accepted_error(exc: GatewayTermsNotAcceptedError) -> NoReturn`-- `pipelex/cli/error_handlers.py:453` — `handle_gateway_api_key_missing_error` — `def handle_gateway_api_key_missing_error(exc: GatewayApiKeyMissingError) -> NoReturn`-- `pipelex/cli/error_handlers.py:483` — `handle_gateway_do_not_track_conflict_error` — `def handle_gateway_do_not_track_conflict_error(exc: GatewayDoNotTrackConflictError) -> NoReturn`-- `pipelex/cli/error_handlers.py:512` — `handle_remote_config_validation_error` — `def handle_remote_config_validation_error(exc: RemoteConfigValidationError) -> NoReturn`-- `pipelex/cli/error_handlers.py:549` — `handle_remote_config_unavailable_error` — `def handle_remote_config_unavailable_error(exc: RemoteConfigUnavailableError) -> NoReturn`-- `pipelex/cli/error_handlers.py:582` — `handle_gateway_unknown_model_error` — `def handle_gateway_unknown_model_error(exc: GatewayUnknownModelError) -> NoReturn`-- `pipelex/cli/installed_methods.py:142` — `discover_methods_from_library_dirs` — `def discover_methods_from_library_dirs(library_dirs: list[str]) -> list[InstalledMethod]`-- `pipelex/cli/method_resolver.py:22` — `_get_all_exported_pipes` — `def _get_all_exported_pipes(method: InstalledMethod) -> set[str]`-- `pipelex/cli/method_resolver.py:30` — `_find_method_by_exported_pipe` — `def _find_method_by_exported_pipe(pipe_code: str) -> InstalledMethod`-- `pipelex/cli/method_resolver.py:63` — `is_github_url` — `def is_github_url(target: str) -> bool`-- `pipelex/cli/method_resolver.py:68` — `parse_github_url` — `def parse_github_url(url: str) -> tuple[str, str | None]`-- `pipelex/cli/method_resolver.py:127` — `is_local_path` — `def is_local_path(target: str) -> bool`-- `pipelex/cli/method_resolver.py:134` — `resolve_method_from_path` — `def resolve_method_from_path(method_path: str) -> InstalledMethod`-- `pipelex/cli/method_resolver.py:168` — `resolve_method_from_url` — `def resolve_method_from_url(url: str) -> InstalledMethod`-- `pipelex/cli/method_resolver.py:294` — `resolve_pipe_from_exports` — `def resolve_pipe_from_exports(pipe_code: str) -> list[str] | None`--### `cogt` (172)--- `pipelex/cogt/config_cogt.py:89` — `LLMConfig.get_template` — `def get_template(self, template_name: str) -> str`-- `pipelex/cogt/content_generation/assignment_models.py:43` — `LLMAssignment.clone_with_new_prompt` — `def clone_with_new_prompt(self, new_prompt: LLMPrompt) -> 'LLMAssignment'`-- `pipelex/cogt/content_generation/content_generator_protocol.py:26` — `update_job_metadata` — `def update_job_metadata(func: Callable[P, Coroutine[Any, Any, R]]) -> Callable[P, Coroutine[Any, Any, R]]`-- `pipelex/cogt/content_generation/content_generator_protocol.py:142` — `ContentGeneratorProtocol.make_search_sourced_answer` — `def make_search_sourced_answer(self, search_assignment: SearchAssignment) -> Coroutine[Any, Any, SearchResultContent]`-- `pipelex/cogt/content_generation/dry_mock.py:167` — `build_mock_object` — `def build_mock_object(model_class: type[BaseModelTypeVar], **field_values: Any) -> BaseModelTypeVar`-- `pipelex/cogt/content_generation/dry_mock.py:200` — `_reconstruct_object_class` — `def _reconstruct_object_class(object_assignment: ObjectAssignment) -> type[BaseModel]`-- `pipelex/cogt/content_generation/dry_mock.py:214` — `stamp_mock_main_coordination` — `def stamp_mock_main_coordination(items: Sequence[Any]) -> None`-- `pipelex/cogt/content_generation/dry_mock.py:229` — `_nb_list_items` — `def _nb_list_items(object_assignment: ObjectAssignment) -> int`-- `pipelex/cogt/content_generation/dry_mock.py:278` — `_dry_report_func` — `def _dry_report_func(cogt_run_params: CogtRunParams) -> _ReportLLMJobFunc`-- `pipelex/cogt/content_generation/dry_mock.py:293` — `dry_llm_gen_text` — `def dry_llm_gen_text(llm_assignment: LLMAssignment) -> str`-- `pipelex/cogt/content_generation/dry_mock.py:303` — `dry_llm_gen_object` — `def dry_llm_gen_object(object_assignment: ObjectAssignment) -> BaseModel`-- `pipelex/cogt/content_generation/dry_mock.py:309` — `dry_llm_gen_object_list` — `def dry_llm_gen_object_list(object_assignment: ObjectAssignment) -> list[BaseModel]`-- `pipelex/cogt/content_generation/dry_mock.py:323` — `dry_templating_gen_text` — `def dry_templating_gen_text(templating_assignment: TemplatingAssignment) -> str`-- `pipelex/cogt/content_generation/dry_mock.py:356` — `dry_img_gen_image_contents` — `def dry_img_gen_image_contents(img_gen_assignment: ImgGenAssignment) -> list[ImageContent]`-- `pipelex/cogt/content_generation/dry_mock.py:370` — `dry_extract_page_contents` — `def dry_extract_page_contents(extract_assignment: ExtractAssignment) -> list[PageContent]`-- `pipelex/cogt/content_generation/dry_mock.py:394` — `dry_render_page_views` — `def dry_render_page_views(render_assignment: RenderPageViewsAssignment) -> list[ImageContent]`-- `pipelex/cogt/content_generation/dry_mock.py:406` — `dry_search_gen_sourced_answer` — `def dry_search_gen_sourced_answer(search_assignment: SearchAssignment) -> SearchResultContent`-- `pipelex/cogt/content_generation/dry_mock.py:414` — `dry_search_gen_structured` — `def dry_search_gen_structured(search_object_assignment: SearchObjectAssignment) -> dict[str, Any]`-- `pipelex/cogt/content_generation/dry_run_factory.py:64` — `DryRunFactory.generate_dict_snake_key_pascal_value` — `def generate_dict_snake_key_pascal_value(cls, num_items: int=2) -> dict[str, str]`-- `pipelex/cogt/content_generation/dry_run_factory.py:97` — `DryRunFactory._get_examples_from_field` — `def _get_examples_from_field(cls, field_info: FieldInfo) -> list[Any] | None`-- `pipelex/cogt/content_generation/dry_run_factory.py:107` — `DryRunFactory._get_mock_format_from_field` — `def _get_mock_format_from_field(cls, field_info: FieldInfo) -> MockFormat | None`-- `pipelex/cogt/content_generation/dry_run_factory.py:133` — `DryRunFactory._detect_examples_constraints` — `def _detect_examples_constraints(cls, object_class: type[BaseModelTypeVar]) -> dict[str, list[Any]]`-- `pipelex/cogt/content_generation/dry_run_factory.py:143` — `DryRunFactory._get_literal_values_from_annotation` — `def _get_literal_values_from_annotation(cls, annotation: Any) -> tuple[Any, ...] | None`-- `pipelex/cogt/content_generation/dry_run_factory.py:168` — `DryRunFactory._detect_literal_fields` — `def _detect_literal_fields(cls, object_class: type[BaseModel]) -> dict[str, tuple[Any, ...]]`-- `pipelex/cogt/content_generation/dry_run_factory.py:188` — `DryRunFactory._make_example_picker` — `def _make_example_picker(examples: list[Any]) -> Callable[[], Any]`-- `pipelex/cogt/content_generation/dry_run_factory.py:238` — `DryRunFactory._extract_types_from_annotation` — `def _extract_types_from_annotation(cls, annotation: Any) -> list[Any]`-- `pipelex/cogt/content_generation/dry_run_factory.py:258` — `DryRunFactory._create_nested_factory` — `def _create_nested_factory(cls, nested_class: type[BaseModel]) -> type[ModelFactory[Any]]`-- `pipelex/cogt/content_generation/dry_run_factory.py:318` — `DryRunFactory._detect_format_constraints` — `def _detect_format_constraints(cls, object_class: type[BaseModelTypeVar]) -> dict[MockFormat, set[str]]`-- `pipelex/cogt/content_generation/exceptions.py:35` — `DryRunObjectFidelityError.for_object_class` — `def for_object_class(cls, object_class_name: str) -> 'DryRunObjectFidelityError'`-- `pipelex/cogt/content_generation/exceptions.py:63` — `DryRunMockBuildError.for_object_class` — `def for_object_class(cls, object_class_name: str) -> 'DryRunMockBuildError'`-- `pipelex/cogt/content_generation/extract_generate.py:10` — `extract_gen_pages` — `async def extract_gen_pages(extract_assignment: ExtractAssignment) -> ExtractOutput`-- `pipelex/cogt/content_generation/generated_content_factory.py:65` — `GeneratedContentFactory._fetch_remote_content` — `async def _fetch_remote_content(self, url: str) -> bytes`-- `pipelex/cogt/content_generation/img_gen_generate.py:11` — `img_gen_single_image` — `async def img_gen_single_image(img_gen_assignment: ImgGenAssignment) -> GeneratedImageRawDetails`-- `pipelex/cogt/content_generation/img_gen_generate.py:24` — `img_gen_image_list` — `async def img_gen_image_list(img_gen_assignment: ImgGenAssignment) -> list[GeneratedImageRawDetails]`-- `pipelex/cogt/content_generation/llm_generate.py:17` — `llm_gen_text` — `async def llm_gen_text(llm_assignment: LLMAssignment) -> str`-- `pipelex/cogt/content_generation/llm_generate.py:31` — `llm_gen_object` — `async def llm_gen_object(object_assignment: ObjectAssignment) -> BaseModel`-- `pipelex/cogt/content_generation/llm_generate.py:52` — `llm_gen_object_list` — `async def llm_gen_object_list(object_assignment: ObjectAssignment) -> list[BaseModel]`-- `pipelex/cogt/content_generation/schema_to_model_factory.py:115` — `SchemaToModelFactory._normalize_class_name` — `def _normalize_class_name(cls, title: str) -> str`-- `pipelex/cogt/content_generation/schema_to_model_factory.py:146` — `SchemaToModelFactory._reject_unsafe_schema_extensions` — `def _reject_unsafe_schema_extensions(cls, schema: dict[str, Any]) -> None`-- `pipelex/cogt/content_generation/schema_to_model_factory.py:161` — `SchemaToModelFactory._generate_source_from_schema` — `def _generate_source_from_schema(cls, schema: dict[str, Any]) -> str`-- `pipelex/cogt/content_generation/schema_to_model_factory.py:241` — `SchemaToModelFactory._exec_source_to_types` — `def _exec_source_to_types(cls, source_code: str) -> dict[str, type[Any]]`-- `pipelex/cogt/content_generation/schema_to_model_factory.py:289` — `SchemaToModelFactory.make_types_from_source` — `def make_types_from_source(cls, source_code: str) -> dict[str, type[Any]]`-- `pipelex/cogt/content_generation/search_generate.py:25` — `_make_search_worker` — `def _make_search_worker(search_assignment: SearchAssignment) -> SearchWorkerAbstract`-- `pipelex/cogt/content_generation/search_generate.py:34` — `_make_search_job` — `def _make_search_job(search_assignment: SearchAssignment) -> SearchJob`-- `pipelex/cogt/content_generation/search_generate.py:46` — `search_gen_sourced_answer` — `async def search_gen_sourced_answer(search_assignment: SearchAssignment) -> SearchResultContent`-- `pipelex/cogt/content_generation/search_generate.py:54` — `search_gen_structured` — `async def search_gen_structured(search_object_assignment: SearchObjectAssignment) -> dict[str, Any]`-- `pipelex/cogt/content_generation/templating_generate.py:6` — `templating_gen_text` — `async def templating_gen_text(templating_assignment: TemplatingAssignment) -> str`-- `pipelex/cogt/document/prompt_document.py:71` — `PromptDocumentUri.get_content_hash` — `def get_content_hash(self, length: int | None=None) -> str`-- `pipelex/cogt/document/prompt_document.py:112` — `PromptDocumentBase64.get_content_hash` — `def get_content_hash(self, length: int | None=None) -> str`-- `pipelex/cogt/document/prompt_document.py:145` — `PromptDocumentBinary.get_content_hash` — `def get_content_hash(self, length: int | None=None) -> str`-- `pipelex/cogt/document/prompt_document_utils.py:114` — `prepare_prompt_document_as_base64` — `async def prepare_prompt_document_as_base64(prompt_document: PromptDocument) -> PreparedFileBase64`-- `pipelex/cogt/exceptions.py:112` — `find_inference_error_category_in_chain` — `def find_inference_error_category_in_chain(exc: BaseException) -> InferenceErrorCategory | None`-- `pipelex/cogt/extract/extract_job.py:22` — `ExtractJob.extract_job_before_start` — `def extract_job_before_start(self, inference_model: InferenceModelSpec)`-- `pipelex/cogt/extract/extract_report.py:25` — `ExtractTokenCostReportField.report_field_for_nb_tokens_by_category` — `def report_field_for_nb_tokens_by_category(token_category: TokenCategory) -> str`-- `pipelex/cogt/extract/extract_report.py:29` — `ExtractTokenCostReportField.report_field_for_cost_by_category` — `def report_field_for_cost_by_category(token_category: CostCategory) -> str`-- `pipelex/cogt/extract/extract_worker_abstract.py:53` — `ExtractWorkerAbstract._check_can_perform_job` — `def _check_can_perform_job(self, extract_job: ExtractJob)`-- `pipelex/cogt/extract/extract_worker_abstract.py:69` — `ExtractWorkerAbstract.extract_pages` — `async def extract_pages(self, extract_job: ExtractJob) -> ExtractOutput`-- `pipelex/cogt/extract/extract_worker_abstract.py:111` — `ExtractWorkerAbstract._extract_pages` — `async def _extract_pages(self, extract_job: ExtractJob) -> ExtractOutput`-- `pipelex/cogt/image/prompt_image_utils.py:115` — `prepare_prompt_image_as_base64` — `async def prepare_prompt_image_as_base64(prompt_image: PromptImage) -> PreparedFileBase64`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:231` — `ImgGenArgsFactory.make_args_from_specific` — `def make_args_from_specific(cls, specific_taxonomy: SpecificTaxonomy) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_args_factory.py:509` — `ImgGenArgsFactory.make_args_from_output_compression` — `def make_args_from_output_compression(cls, output_compression_taxonomy: OutputCompressionTaxonomy) -> dict[str, Any]`-- `pipelex/cogt/img_gen/img_gen_job.py:22` — `ImgGenJob.img_gen_job_before_start` — `def img_gen_job_before_start(self, inference_model: InferenceModelSpec)`-- `pipelex/cogt/img_gen/img_gen_report.py:25` — `ImgGenTokenCostReportField.report_field_for_nb_tokens_by_category` — `def report_field_for_nb_tokens_by_category(token_category: TokenCategory) -> str`-- `pipelex/cogt/img_gen/img_gen_report.py:29` — `ImgGenTokenCostReportField.report_field_for_cost_by_category` — `def report_field_for_cost_by_category(token_category: CostCategory) -> str`-- `pipelex/cogt/img_gen/img_gen_worker_abstract.py:38` — `ImgGenWorkerAbstract._check_can_perform_job` — `def _check_can_perform_job(self, img_gen_job: ImgGenJob)`-- `pipelex/cogt/img_gen/img_gen_worker_abstract.py:42` — `ImgGenWorkerAbstract.gen_image` — `async def gen_image(self, img_gen_job: ImgGenJob) -> GeneratedImageRawDetails`-- `pipelex/cogt/img_gen/img_gen_worker_abstract.py:75` — `ImgGenWorkerAbstract._gen_image` — `async def _gen_image(self, img_gen_job: ImgGenJob) -> GeneratedImageRawDetails`-- `pipelex/cogt/inference/error_classification.py:231` — `_is_quota_exhaustion_openai` — `def _is_quota_exhaustion_openai(error_message: str) -> bool`-- `pipelex/cogt/inference/error_classification.py:237` — `_is_quota_exhaustion_anthropic` — `def _is_quota_exhaustion_anthropic(error_message: str) -> bool`-- `pipelex/cogt/inference/error_classification.py:243` — `_is_quota_exhaustion_google` — `def _is_quota_exhaustion_google(error_message: str) -> bool`-- `pipelex/cogt/inference/error_classification.py:288` — `_is_content_policy_violation` — `def _is_content_policy_violation(error_message: str) -> bool`-- `pipelex/cogt/inference/error_classification.py:294` — `_stringify_for_scan` — `def _stringify_for_scan(body: Any) -> str`-- `pipelex/cogt/inference/error_classification.py:308` — `extract_underlying_sdk_exception` — `def extract_underlying_sdk_exception(instructor_exc: Any) -> BaseException | None`-- `pipelex/cogt/inference/error_classification.py:345` — `_parse_retry_after_seconds` — `def _parse_retry_after_seconds(value: Any) -> float | None`-- `pipelex/cogt/inference/error_classification.py:370` — `_provider_error_code_from_body` — `def _provider_error_code_from_body(body: Any) -> str | None`-- `pipelex/cogt/inference/error_classification.py:383` — `extract_openai_metadata` — `def extract_openai_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:430` — `extract_anthropic_metadata` — `def extract_anthropic_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:465` — `_provider_error_code_from_flat_body` — `def _provider_error_code_from_flat_body(body: Any) -> str | None`-- `pipelex/cogt/inference/error_classification.py:481` — `_parse_response_text_body` — `def _parse_response_text_body(response: Any) -> tuple[Any | None, str | None]`-- `pipelex/cogt/inference/error_classification.py:510` — `_google_provider_error_code_from_details` — `def _google_provider_error_code_from_details(details: Any) -> str | None`-- `pipelex/cogt/inference/error_classification.py:533` — `extract_google_metadata` — `def extract_google_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:570` — `extract_azure_metadata` — `def extract_azure_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:623` — `extract_fal_metadata` — `def extract_fal_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:662` — `extract_huggingface_metadata` — `def extract_huggingface_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:695` — `extract_gateway_metadata` — `def extract_gateway_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:729` — `extract_mistral_metadata` — `def extract_mistral_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:796` — `extract_bedrock_metadata` — `def extract_bedrock_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classification.py:843` — `extract_linkup_metadata` — `def extract_linkup_metadata(exc: BaseException) -> ProviderErrorMetadata`-- `pipelex/cogt/inference/error_classify.py:73` — `_classify_statusless` — `def _classify_statusless(metadata: SDKErrorEnvelope) -> ClassificationResult`-- `pipelex/cogt/inference/error_classify.py:97` — `classify_inference_error` — `def classify_inference_error(metadata: SDKErrorEnvelope) -> ClassificationResult`-- `pipelex/cogt/inference/inference_constants.py:11` — `InferenceOutputType.is_text` — `def is_text(cls, output_desc: str) -> bool`-- `pipelex/cogt/inference/inference_manager.py:53` — `InferenceManager._setup_one_internal_llm_worker` — `def _setup_one_internal_llm_worker(self, llm_handle: str) -> LLMWorkerInternalAbstract`-- `pipelex/cogt/inference/inference_manager.py:88` — `InferenceManager._setup_one_img_gen_worker` — `def _setup_one_img_gen_worker(self, img_gen_handle: str) -> ImgGenWorkerAbstract`-- `pipelex/cogt/inference/inference_manager.py:109` — `InferenceManager._setup_one_extract_worker` — `def _setup_one_extract_worker(self, extract_handle: str) -> ExtractWorkerAbstract`-- `pipelex/cogt/inference/inference_manager_protocol.py:19` — `InferenceManagerProtocol.get_llm_worker` — `def get_llm_worker(self, llm_handle: str) -> LLMWorkerAbstract`-- `pipelex/cogt/inference/inference_manager_protocol.py:33` — `InferenceManagerProtocol.get_img_gen_worker` — `def get_img_gen_worker(self, img_gen_handle: str) -> ImgGenWorkerAbstract`-- `pipelex/cogt/inference/inference_manager_protocol.py:39` — `InferenceManagerProtocol.get_extract_worker` — `def get_extract_worker(self, extract_handle: str) -> ExtractWorkerAbstract`-- `pipelex/cogt/inference/transport_retry.py:49` — `_parse_retry_after` — `def _parse_retry_after(raw_value: str | None) -> float | None`-- `pipelex/cogt/inference/transport_retry.py:69` — `_transport_retry_wait` — `def _transport_retry_wait(retry_state: RetryCallState) -> float`-- `pipelex/cogt/inference/transport_retry.py:81` — `_make_retry_predicate` — `def _make_retry_predicate(retry_on_ambiguous_failure: bool) -> Callable[[BaseException], bool]`-- `pipelex/cogt/inference/transport_retry.py:91` — `_make_retry_predicate.should_retry` — `def should_retry(exc: BaseException) -> bool`-- `pipelex/cogt/llm/instructor_retry.py:7` — `make_instructor_schema_retrying` — `def make_instructor_schema_retrying(max_attempts: int) -> AsyncRetrying`-- `pipelex/cogt/llm/llm_job.py:28` — `LLMJob.llm_job_before_start` — `def llm_job_before_start(self, inference_model: InferenceModelSpec)`-- `pipelex/cogt/llm/llm_prompt.py:54` — `LLMPrompt.desc` — `def desc(self, truncate_text_length: int | None=None) -> str`-- `pipelex/cogt/llm/llm_prompt_template_inputs.py:50` — `LLMPromptTemplateInputs.complemented_by` — `def complemented_by(self, additional_template_inputs: LLMPromptTemplateInputs | None) -> LLMPromptTemplateInputs`-- `pipelex/cogt/llm/llm_prompt_template_inputs.py:56` — `LLMPromptTemplateInputs.complemented_by_dict` — `def complemented_by_dict(self, additional_inputs_dict: LLMPromptTemplateInputsDict | None) -> LLMPromptTemplateInputs`-- `pipelex/cogt/llm/llm_report.py:25` — `LLMTokenCostReportField.report_field_for_nb_tokens_by_category` — `def report_field_for_nb_tokens_by_category(token_category: TokenCategory) -> str`-- `pipelex/cogt/llm/llm_report.py:29` — `LLMTokenCostReportField.report_field_for_cost_by_category` — `def report_field_for_cost_by_category(token_category: CostCategory) -> str`-- `pipelex/cogt/llm/llm_utils.py:7` — `dump_prompt` — `def dump_prompt(llm_prompt: LLMPrompt) -> None`-- `pipelex/cogt/llm/llm_utils.py:22` — `dump_response_from_text_gen` — `def dump_response_from_text_gen(response: Any) -> None`-- `pipelex/cogt/llm/llm_utils.py:30` — `dump_response_from_structured_gen` — `def dump_response_from_structured_gen(response: Any) -> None`-- `pipelex/cogt/llm/llm_utils.py:34` — `dump_error` — `def dump_error(error: Exception) -> None`-- `pipelex/cogt/llm/llm_worker_abstract.py:317` — `LLMWorkerAbstract._before_job` — `async def _before_job(self, llm_job: LLMJob)`-- `pipelex/cogt/llm/llm_worker_abstract.py:357` — `LLMWorkerAbstract._check_can_perform_job` — `def _check_can_perform_job(self, llm_job: LLMJob)`-- `pipelex/cogt/llm/llm_worker_abstract.py:361` — `LLMWorkerAbstract.gen_text` — `async def gen_text(self, llm_job: LLMJob) -> str`-- `pipelex/cogt/llm/llm_worker_abstract.py:392` — `LLMWorkerAbstract._gen_text` — `async def _gen_text(self, llm_job: LLMJob) -> str`-- `pipelex/cogt/llm/llm_worker_internal_abstract.py:88` — `LLMWorkerInternalAbstract._apply_constraints` — `def _apply_constraints(self, llm_job: LLMJob) -> LLMJobParams | None`-- `pipelex/cogt/llm/llm_worker_internal_abstract.py:152` — `LLMWorkerInternalAbstract._validate_no_reasoning_for_structured_gen` — `def _validate_no_reasoning_for_structured_gen(self, job_params: LLMJobParams)`-- `pipelex/cogt/llm/llm_worker_internal_abstract.py:157` — `LLMWorkerInternalAbstract._check_vision_support` — `def _check_vision_support(self, llm_job: LLMJob)`-- `pipelex/cogt/llm/llm_worker_internal_abstract.py:169` — `LLMWorkerInternalAbstract._check_document_support` — `def _check_document_support(self, llm_job: LLMJob)`-- `pipelex/cogt/model_backends/backend.py:27` — `PipelexBackend.is_gateway_backend` — `def is_gateway_backend(cls, backend_name: str) -> bool`-- `pipelex/cogt/model_backends/backend.py:54` — `InferenceBackend.get_model_spec` — `def get_model_spec(self, model_name: str) -> InferenceModelSpec | None`-- `pipelex/cogt/model_backends/backend.py:58` — `InferenceBackend.get_extra_config` — `def get_extra_config(self, key: str) -> Any | None`-- `pipelex/cogt/model_backends/backend_library.py:392` — `InferenceBackendLibrary.get_inference_backend` — `def get_inference_backend(self, backend_name: str) -> InferenceBackend | None`-- `pipelex/cogt/models/deck_manifest.py:77` — `DeckSyncReport.files_with_status` — `def files_with_status(self, status: DeckFileStatus) -> list[str]`-- `pipelex/cogt/models/deck_manifest.py:81` — `compute_sha256` — `def compute_sha256(data: bytes) -> str`-- `pipelex/cogt/models/deck_manifest.py:85` — `compute_file_sha256` — `def compute_file_sha256(path: Path) -> str`-- `pipelex/cogt/models/deck_manifest.py:98` — `_is_managed_deck_filename` — `def _is_managed_deck_filename(filename: str) -> bool`-- `pipelex/cogt/models/deck_manifest.py:119` — `list_managed_installed_files` — `def list_managed_installed_files(deck_dir: Path) -> dict[str, str]`-- `pipelex/cogt/models/deck_manifest.py:133` — `manifest_path` — `def manifest_path(deck_dir: Path) -> Path`-- `pipelex/cogt/models/deck_manifest.py:137` — `read_manifest` — `def read_manifest(deck_dir: Path) -> DeckManifest | None`-- `pipelex/cogt/models/deck_manifest.py:163` — `is_deck_stale_fast` — `def is_deck_stale_fast(deck_dir: Path) -> bool`-- `pipelex/cogt/models/deck_manifest.py:193` — `_try_parse_version` — `def _try_parse_version(version: str) -> tuple[int, ...] | None`-- `pipelex/cogt/models/deck_manifest.py:206` — `status_rich_label` — `def status_rich_label(status: DeckFileStatus) -> str`-- `pipelex/cogt/models/deck_manifest.py:221` — `suggest_x_custom_filename` — `def suggest_x_custom_filename(numbered_filename: str) -> str`-- `pipelex/cogt/models/deck_manifest.py:233` — `compute_deck_sync_report` — `def compute_deck_sync_report(deck_dir: Path) -> DeckSyncReport`-- `pipelex/cogt/models/model_deck.py:123` — `ModelDeck.get_aliases_and_waterfalls_for_type` — `def get_aliases_and_waterfalls_for_type(self, model_type: ModelType) -> tuple[dict[str, str], dict[str, list[str]]]`-- `pipelex/cogt/models/model_deck.py:163` — `ModelDeck._warn_if_ambiguous_llm` — `def _warn_if_ambiguous_llm(self, name: str) -> None`-- `pipelex/cogt/models/model_deck.py:177` — `ModelDeck._warn_if_ambiguous_extract` — `def _warn_if_ambiguous_extract(self, name: str) -> None`-- `pipelex/cogt/models/model_deck.py:191` — `ModelDeck._warn_if_ambiguous_img_gen` — `def _warn_if_ambiguous_img_gen(self, name: str) -> None`-- `pipelex/cogt/models/model_deck.py:205` — `ModelDeck._warn_if_ambiguous_search` — `def _warn_if_ambiguous_search(self, name: str) -> None`-- `pipelex/cogt/models/model_deck.py:309` — `ModelDeck.get_llm_setting` — `def get_llm_setting(self, llm_choice: LLMModelChoice) -> LLMSetting`-- `pipelex/cogt/models/model_deck.py:362` — `ModelDeck.get_extract_setting` — `def get_extract_setting(self, extract_choice: ExtractModelChoice) -> ExtractSetting`-- `pipelex/cogt/models/model_deck.py:411` — `ModelDeck.get_search_setting` — `def get_search_setting(self, search_choice: SearchModelChoice) -> SearchSetting`-- `pipelex/cogt/models/model_deck.py:460` — `ModelDeck.get_img_gen_setting` — `def get_img_gen_setting(self, img_gen_choice: ImgGenModelChoice) -> ImgGenSetting`-- `pipelex/cogt/models/model_deck.py:510` — `ModelDeck.final_validate` — `def final_validate(cls, deck: Self)`-- `pipelex/cogt/models/model_deck_check.py:41` — `check_llm_choice_with_deck` — `def check_llm_choice_with_deck(llm_choice: LLMModelChoice) -> None`-- `pipelex/cogt/models/model_deck_check.py:103` — `check_extract_choice_with_deck` — `def check_extract_choice_with_deck(extract_choice: ExtractModelChoice) -> None`-- `pipelex/cogt/models/model_deck_check.py:165` — `check_search_choice_with_deck` — `def check_search_choice_with_deck(search_choice: SearchModelChoice) -> None`-- `pipelex/cogt/models/model_deck_check.py:227` — `check_img_gen_choice_with_deck` — `def check_img_gen_choice_with_deck(img_gen_choice: ImgGenModelChoice) -> None`-- `pipelex/cogt/models/model_deck_loader.py:12` — `load_model_deck_blueprint` — `def load_model_deck_blueprint(model_deck_paths: list[str]) -> ModelDeckBlueprint`-- `pipelex/cogt/models/model_manager.py:44` — `ModelManager.get_model_deck_paths` — `def get_model_deck_paths(cls, deck_dir_path: str) -> list[str]`-- `pipelex/cogt/models/model_manager.py:143` — `ModelManager._collect_deck_referenced_handles` — `def _collect_deck_referenced_handles(cls, deck: ModelDeck) -> list[tuple[str, ModelType]]`-- `pipelex/cogt/models/model_manager.py:178` — `ModelManager._extract_choice_handle` — `def _extract_choice_handle(cls, choice: LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | ModelReference | str | None) -> str | None`-- `pipelex/cogt/models/model_manager_abstract.py:41` — `ModelManagerAbstract.get_required_inference_backend` — `def get_required_inference_backend(self, backend_name: str) -> InferenceBackend`-- `pipelex/cogt/models/model_reference.py:63` — `ModelReference.parse` — `def parse(cls, value: str) -> 'ModelReference'`-- `pipelex/cogt/models/model_reference.py:165` — `ensure_model_reference` — `def ensure_model_reference(value: str | ModelReference) -> ModelReference`-- `pipelex/cogt/models/model_reference.py:175` — `parse_model_reference` — `def parse_model_reference(value: Any) -> ModelReference | Any`-- `pipelex/cogt/search/search_job.py:33` — `SearchJob.search_job_before_start` — `def search_job_before_start(self, inference_model: InferenceModelSpec)`-- `pipelex/cogt/search/search_report.py:25` — `SearchTokenCostReportField.report_field_for_nb_tokens_by_category` — `def report_field_for_nb_tokens_by_category(token_category: TokenCategory) -> str`-- `pipelex/cogt/search/search_report.py:29` — `SearchTokenCostReportField.report_field_for_cost_by_category` — `def report_field_for_cost_by_category(token_category: CostCategory) -> str`-- `pipelex/cogt/search/search_worker_abstract.py:31` — `SearchWorkerAbstract.search_sourced_answer` — `async def search_sourced_answer(self, search_job: SearchJob) -> SearchResultContent`-- `pipelex/cogt/search/search_worker_abstract.py:78` — `SearchWorkerAbstract._search_sourced_answer` — `async def _search_sourced_answer(self, search_job: SearchJob) -> SearchResultContent`-- `pipelex/cogt/search/search_worker_factory.py:9` — `SearchWorkerFactory.make_search_worker` — `def make_search_worker(cls, inference_model: InferenceModelSpec) -> SearchWorkerAbstract`-- `pipelex/cogt/templating/template_preprocessor.py:73` — `_replace_at_sigil` — `def _replace_at_sigil(match: Match[str]) -> str`-- `pipelex/cogt/templating/template_preprocessor.py:86` — `_replace_dollar_sigil` — `def _replace_dollar_sigil(match: Match[str]) -> str`-- `pipelex/cogt/templating/template_preprocessor.py:139` — `_normalize_and_escape` — `def _normalize_and_escape(template: str) -> str`-- `pipelex/cogt/templating/template_preprocessor.py:158` — `rewrite_template_sigils` — `def rewrite_template_sigils(template: str) -> str`-- `pipelex/cogt/usage/cost_registry.py:218` — `CostRegistry.aggregate_costs` — `def aggregate_costs(cls, tokens_usages: Sequence[TokensUsage]) -> AggregatedCosts`-- `pipelex/cogt/usage/cost_registry.py:283` — `CostRegistry.build_cost_summary` — `def build_cost_summary(cls, tokens_usages: Sequence[TokensUsage]) -> dict[str, Any] | None`-- `pipelex/cogt/usage/cost_registry.py:314` — `CostRegistry.compute_cost_report` — `def compute_cost_report(cls, tokens_usage: TokensUsage) -> TokenCostReport`-- `pipelex/cogt/usage/cost_registry.py:360` — `CostRegistry.complete_cost_report` — `def complete_cost_report(cls, tokens_usage: TokensUsage) -> TokenCostReport`--### `core` (118)--- `pipelex/core/bundles/pipe_sorter.py:7` — `sort_pipes_by_dependencies` — `def sort_pipes_by_dependencies(pipes: dict[str, PipeBlueprintUnion]) -> list[tuple[str, PipeBlueprintUnion]]`-- `pipelex/core/bundles/pipe_sorter.py:48` — `sort_pipes_by_dependencies.visit` — `def visit(pipe_code: str) -> None`-- `pipelex/core/bundles/pipelex_bundle_blueprint.py:143` — `PipelexBundleBlueprint.get_elaboration_for` — `def get_elaboration_for(self, pipe_code: str) -> ElaborationMetadata | None`-- `pipelex/core/concepts/concept.py:86` — `Concept.sentence_from_concept` — `def sentence_from_concept(cls, concept: 'Concept') -> str`-- `pipelex/core/concepts/concept.py:90` — `Concept.is_native_concept` — `def is_native_concept(cls, concept: 'Concept') -> bool`-- `pipelex/core/concepts/concept.py:167` — `Concept.is_valid_structure_class` — `def is_valid_structure_class(cls, structure_class_name: str) -> bool`-- `pipelex/core/concepts/concept.py:227` — `Concept._render_schema_representation` — `def _render_schema_representation(self, is_multiple: bool=False) -> tuple[dict[str, Any], set[str]]`-- `pipelex/core/concepts/concept_factory.py:98` — `ConceptFactory.make_native_concept` — `def make_native_concept(cls, native_concept_code: NativeConceptCode) -> Concept`-- `pipelex/core/concepts/concept_representation_generator.py:183` — `ConceptRepresentationGenerator._unwrap_optional` — `def _unwrap_optional(self, field_type: Any) -> Any`-- `pipelex/core/concepts/concept_representation_generator.py:227` — `ConceptRepresentationGenerator._generate_dict_value` — `def _generate_dict_value(self, field_name: str) -> dict[str, str]`-- `pipelex/core/concepts/concept_representation_generator.py:266` — `ConceptRepresentationGenerator._generate_basemodel_representation` — `def _generate_basemodel_representation(self, model_class: type[BaseModel]) -> dict[str, Any] | str`-- `pipelex/core/concepts/concept_representation_generator.py:347` — `ConceptRepresentationGenerator._format_python_value` — `def _format_python_value(self, value: Any) -> str`-- `pipelex/core/concepts/helpers.py:52` — `strip_multiplicity_from_concept_ref_or_code` — `def strip_multiplicity_from_concept_ref_or_code(concept_ref_or_code: str) -> str`-- `pipelex/core/concepts/helpers.py:66` — `normalize_structure_blueprint` — `def normalize_structure_blueprint(structure_dict: dict[str, str | ConceptStructureBlueprint]) -> dict[str, ConceptStructureBlueprint]`-- `pipelex/core/concepts/helpers.py:111` — `extract_concept_code_from_concept_ref_or_code` — `def extract_concept_code_from_concept_ref_or_code(concept_ref_or_code: str) -> str`-- `pipelex/core/concepts/native/concept_native.py:77` — `NativeConceptCode.is_native_structure_class` — `def is_native_structure_class(cls, class_name: str) -> bool`-- `pipelex/core/concepts/native/concept_native.py:89` — `NativeConceptCode.get_native_structure_class` — `def get_native_structure_class(cls, class_name: str) -> type | None`-- `pipelex/core/concepts/native/concept_native.py:107` — `NativeConceptCode.is_text_concept` — `def is_text_concept(cls, concept_code: str) -> bool`-- `pipelex/core/concepts/native/concept_native.py:132` — `NativeConceptCode.is_dynamic_concept` — `def is_dynamic_concept(cls, concept_code: str) -> bool`-- `pipelex/core/concepts/native/concept_native.py:165` — `NativeConceptCode.is_native_concept_ref_or_code` — `def is_native_concept_ref_or_code(cls, concept_ref_or_code: str) -> bool`-- `pipelex/core/concepts/native/concept_native.py:176` — `NativeConceptCode.is_valid_native_concept_ref` — `def is_valid_native_concept_ref(cls, concept_ref: str) -> bool`-- `pipelex/core/concepts/native/concept_native.py:195` — `NativeConceptCode.validate_native_concept_ref_or_code` — `def validate_native_concept_ref_or_code(cls, concept_ref_or_code: str) -> None`-- `pipelex/core/concepts/native/concept_native.py:201` — `NativeConceptCode.get_validated_native_concept_ref` — `def get_validated_native_concept_ref(cls, concept_ref_or_code: str) -> str`-- `pipelex/core/concepts/structure_generation/exceptions.py:16` — `SyntaxErrorData.from_syntax_error` — `def from_syntax_error(cls, syntax_error: SyntaxError) -> 'SyntaxErrorData'`-- `pipelex/core/concepts/structure_generation/generator.py:173` — `StructureGenerator._escape_string_for_python` — `def _escape_string_for_python(self, value: str) -> str`-- `pipelex/core/concepts/structure_generation/generator.py:205` — `StructureGenerator._format_default_value` — `def _format_default_value(self, value: Any) -> str`-- `pipelex/core/concepts/structure_generation/generator.py:227` — `StructureGenerator._format_field_description` — `def _format_field_description(self, description: str) -> str`-- `pipelex/core/concepts/structure_generation/generator.py:351` — `StructureGenerator._get_python_type_from_blueprint` — `def _get_python_type_from_blueprint(self, field_blueprint: ConceptStructureBlueprint) -> str`-- `pipelex/core/concepts/structure_generation/generator.py:424` — `StructureGenerator._resolve_concept_ref_to_type` — `def _resolve_concept_ref_to_type(self, concept_ref: str | None) -> str`-- `pipelex/core/concepts/validation.py:6` — `is_concept_code_valid` — `def is_concept_code_valid(concept_code: str) -> bool`-- `pipelex/core/concepts/validation.py:10` — `validate_concept_code` — `def validate_concept_code(concept_code: str) -> None`-- `pipelex/core/concepts/validation.py:16` — `is_concept_ref_valid` — `def is_concept_ref_valid(concept_ref: str) -> bool`-- `pipelex/core/concepts/validation.py:32` — `validate_concept_ref` — `def validate_concept_ref(concept_ref: str) -> None`-- `pipelex/core/concepts/validation.py:42` — `is_concept_ref_or_code_valid` — `def is_concept_ref_or_code_valid(concept_ref_or_code: str) -> bool`-- `pipelex/core/concepts/validation.py:59` — `validate_concept_ref_or_code` — `def validate_concept_ref_or_code(concept_ref_or_code: str) -> None`-- `pipelex/core/domains/domain.py:12` — `SpecialDomain.is_native` — `def is_native(cls, domain_code: str) -> bool`-- `pipelex/core/domains/domain_factory.py:11` — `DomainFactory.make_from_blueprint` — `def make_from_blueprint(cls, blueprint: DomainBlueprint) -> Domain`-- `pipelex/core/domains/validation.py:8` — `is_domain_code_valid` — `def is_domain_code_valid(code: Any) -> bool`-- `pipelex/core/domains/validation.py:26` — `validate_domain_code` — `def validate_domain_code(code: str) -> None`-- `pipelex/core/interpreter/bundle_elaborator.py:26` — `_is_preliminary_text_pipe` — `def _is_preliminary_text_pipe(pipe_blueprint: PipeBlueprintUnion) -> TypeGuard[PipeLLMBlueprint]`-- `pipelex/core/interpreter/bundle_elaborator.py:48` — `BundleElaborator.elaborate` — `def elaborate(cls, bundle: PipelexBundleBlueprint) -> PipelexBundleBlueprint`-- `pipelex/core/interpreter/helpers.py:8` — `is_pipelex_file` — `def is_pipelex_file(file_path: Path) -> bool`-- `pipelex/core/interpreter/helpers.py:28` — `ValidationErrorScope.is_pipe_scope` — `def is_pipe_scope(cls, scope: str) -> bool`-- `pipelex/core/interpreter/helpers.py:46` — `get_error_scope` — `def get_error_scope(loc: tuple[int | str, ...]) -> ValidationErrorScope`-- `pipelex/core/interpreter/validation_error_categorizer.py:17` — `_extract_variable_names_from_message` — `def _extract_variable_names_from_message(message: str) -> list[str] | None`-- `pipelex/core/interpreter/validation_error_categorizer.py:113` — `_is_pydantic_internal_loc_element` — `def _is_pydantic_internal_loc_element(element: str) -> bool`-- `pipelex/core/memory/working_memory.py:67` — `WorkingMemory.get_optional_stuff` — `def get_optional_stuff(self, name: str) -> Stuff | None`-- `pipelex/core/memory/working_memory.py:80` — `WorkingMemory.get_stuff` — `def get_stuff(self, name: str) -> Stuff`-- `pipelex/core/memory/working_memory.py:96` — `WorkingMemory.get_stuffs` — `def get_stuffs(self, names: set[str]) -> list[Stuff]`-- `pipelex/core/memory/working_memory.py:102` — `WorkingMemory.get_existing_stuffs` — `def get_existing_stuffs(self, names: set[str]) -> list[Stuff]`-- `pipelex/core/memory/working_memory.py:109` — `WorkingMemory.is_stuff_code_used` — `def is_stuff_code_used(self, stuff_code: str) -> bool`-- `pipelex/core/memory/working_memory.py:112` — `WorkingMemory.remove_stuff` — `def remove_stuff(self, name: str)`-- `pipelex/core/memory/working_memory.py:122` — `WorkingMemory.is_stuff_exists` — `def is_stuff_exists(self, name: str) -> bool`-- `pipelex/core/memory/working_memory.py:174` — `WorkingMemory.remove_alias` — `def remove_alias(self, alias: str) -> None`-- `pipelex/core/memory/working_memory.py:183` — `WorkingMemory.get_aliases_for` — `def get_aliases_for(self, target: str) -> list[str]`-- `pipelex/core/memory/working_memory.py:384` — `WorkingMemory.get_stuff_as_text` — `def get_stuff_as_text(self, name: str) -> TextContent`-- `pipelex/core/memory/working_memory.py:388` — `WorkingMemory.get_stuff_as_str` — `def get_stuff_as_str(self, name: str) -> str`-- `pipelex/core/memory/working_memory.py:392` — `WorkingMemory.get_stuff_as_image` — `def get_stuff_as_image(self, name: str) -> ImageContent`-- `pipelex/core/memory/working_memory.py:396` — `WorkingMemory.get_stuff_as_text_and_image` — `def get_stuff_as_text_and_image(self, name: str) -> TextAndImagesContent`-- `pipelex/core/memory/working_memory.py:400` — `WorkingMemory.get_stuff_as_document` — `def get_stuff_as_document(self, name: str) -> DocumentContent`-- `pipelex/core/memory/working_memory.py:404` — `WorkingMemory.get_stuff_as_number` — `def get_stuff_as_number(self, name: str) -> NumberContent`-- `pipelex/core/memory/working_memory.py:408` — `WorkingMemory.get_stuff_as_html` — `def get_stuff_as_html(self, name: str) -> HtmlContent`-- `pipelex/core/memory/working_memory.py:412` — `WorkingMemory.get_stuff_as_mermaid` — `def get_stuff_as_mermaid(self, name: str) -> MermaidContent`-- `pipelex/core/memory/working_memory.py:420` — `WorkingMemory.main_stuff_as` — `def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType`-- `pipelex/core/memory/working_memory.py:424` — `WorkingMemory.main_stuff_as_list` — `def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]`-- `pipelex/core/memory/working_memory.py:430` — `WorkingMemory.main_list_stuff_first_item_as` — `def main_list_stuff_first_item_as(self, item_type: type[StuffContentType]) -> StuffContentType`-- `pipelex/core/memory/working_memory_factory.py:31` — `WorkingMemoryFactory.make_from_single_stuff` — `def make_from_single_stuff(cls, stuff: Stuff) -> WorkingMemory`-- `pipelex/core/memory/working_memory_factory.py:96` — `WorkingMemoryFactory.convert_to_working_memory_format` — `def convert_to_working_memory_format(cls, needed_inputs_spec: InputStuffSpecs) -> list[TypedNamedStuffSpec]`-- `pipelex/core/memory/working_memory_factory.py:174` — `WorkingMemoryFactory.make_mock_content` — `def make_mock_content(cls, typed_named_stuff_spec: TypedNamedStuffSpec) -> StuffContent`-- `pipelex/core/memory/working_memory_factory.py:196` — `WorkingMemoryFactory._get_mockable_class` — `def _get_mockable_class(cls, structure_class: type[StuffContent]) -> type[StuffContent]`-- `pipelex/core/memory/working_memory_factory.py:216` — `WorkingMemoryFactory.make_mock_stuff` — `def make_mock_stuff(cls, typed_named_stuff_spec: TypedNamedStuffSpec) -> Stuff`-- `pipelex/core/memory/working_memory_factory.py:251` — `WorkingMemoryFactory.make_mock_inputs` — `def make_mock_inputs(cls, needed_inputs: list[TypedNamedStuffSpec]) -> 'WorkingMemory'`-- `pipelex/core/pipes/handle_pipe_errors.py:18` — `_extract_wrapped_pipe_validation_error` — `def _extract_wrapped_pipe_validation_error(error: ErrorDetails) -> PipeValidationError | None`-- `pipelex/core/pipes/handle_pipe_errors.py:46` — `categorize_pipe_validation_error` — `def categorize_pipe_validation_error(validation_error: ValidationError) -> list[PipesAndConceptValidationErrorData]`-- `pipelex/core/pipes/handle_pipe_errors.py:173` — `categorize_pipe_validation_with_libraries_error` — `def categorize_pipe_validation_with_libraries_error(pipe_error: PipeValidationError) -> PipesAndConceptValidationErrorData`-- `pipelex/core/pipes/handle_pipe_errors.py:202` — `categorize_pipe_factory_error` — `def categorize_pipe_factory_error(factory_error: PipeFactoryError) -> PipeFactoryErrorData`-- `pipelex/core/pipes/inputs/input_stuff_specs.py:67` — `InputStuffSpecs.set_default_domain` — `def set_default_domain(self, domain_code: str)`-- `pipelex/core/pipes/inputs/input_stuff_specs.py:74` — `InputStuffSpecs.get_required_stuff_spec` — `def get_required_stuff_spec(self, variable_name: str) -> StuffSpec`-- `pipelex/core/pipes/inputs/input_stuff_specs.py:81` — `InputStuffSpecs.is_variable_existing` — `def is_variable_existing(self, variable_name: str) -> bool`-- `pipelex/core/pipes/inputs/input_stuff_specs.py:136` — `InputStuffSpecs.format_for_display` — `def format_for_display(self, indent: int=6) -> str`-- `pipelex/core/pipes/inputs/input_stuff_specs.py:153` — `InputStuffSpecs.render_inputs` — `def render_inputs(self, indent: int=2) -> str`-- `pipelex/core/pipes/output/output_renderer.py:191` — `_render_python_output` — `def _render_python_output(the_pipe: PipeAbstract) -> str`-- `pipelex/core/pipes/pipe_abstract.py:121` — `PipeAbstract._make_single_concept_data_for_registry` — `def _make_single_concept_data_for_registry(self, concept: Concept) -> dict[str, Any]`-- `pipelex/core/pipes/pipe_abstract.py:415` — `PipeAbstract.needed_inputs` — `def needed_inputs(self, visited_pipes: set[str] | None=None) -> InputStuffSpecs`-- `pipelex/core/pipes/pipe_abstract.py:427` — `PipeAbstract._format_pipe_run_info` — `def _format_pipe_run_info(self, pipe_run_params: PipeRunParams) -> str`-- `pipelex/core/pipes/pipe_blueprint.py:36` — `PipeCategory.is_controller_by_str` — `def is_controller_by_str(cls, category_str: str) -> bool`-- `pipelex/core/pipes/pipe_output.py:35` — `PipeOutput.prepare_for_temporal` — `def prepare_for_temporal(self, library_crate: LibraryCrate | None=None) -> 'PipeOutput'`-- `pipelex/core/pipes/pipe_output.py:66` — `PipeOutput.main_stuff_as_list` — `def main_stuff_as_list(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]`-- `pipelex/core/pipes/pipe_output.py:72` — `PipeOutput.main_stuff_as_items` — `def main_stuff_as_items(self, item_type: type[StuffContentType]) -> list[StuffContentType]`-- `pipelex/core/pipes/pipe_output.py:78` — `PipeOutput.main_stuff_as` — `def main_stuff_as(self, content_type: type[StuffContentType]) -> StuffContentType`-- `pipelex/core/pipes/stuff_spec/stuff_spec.py:28` — `StuffSpec.render_stuff_spec` — `def render_stuff_spec(self, output_format: ConceptRepresentationFormat) -> dict[str, Any]`-- `pipelex/core/pipes/validation.py:56` — `is_valid_input_name` — `def is_valid_input_name(input_name: str) -> bool`-- `pipelex/core/pipes/validation.py:106` — `validate_input_name` — `def validate_input_name(input_name: str) -> None`-- `pipelex/core/pipes/validation.py:126` — `is_pipe_code_valid` — `def is_pipe_code_valid(pipe_code: str) -> bool`-- `pipelex/core/pipes/variable_multiplicity.py:58` — `parse_concept_with_multiplicity` — `def parse_concept_with_multiplicity(concept_ref_or_code: str) -> MultiplicityParseResult`-- `pipelex/core/qualified_ref.py:41` — `QualifiedRef.parse` — `def parse(cls, raw: str) -> 'QualifiedRef'`-- `pipelex/core/qualified_ref.py:71` — `QualifiedRef.parse_stripping_cross_package` — `def parse_stripping_cross_package(cls, raw: str) -> 'QualifiedRef'`-- `pipelex/core/qualified_ref.py:82` — `QualifiedRef.parse_concept_ref` — `def parse_concept_ref(cls, raw: str) -> 'QualifiedRef'`-- `pipelex/core/qualified_ref.py:109` — `QualifiedRef.parse_pipe_ref` — `def parse_pipe_ref(cls, raw: str) -> 'QualifiedRef'`-- `pipelex/core/qualified_ref.py:148` — `QualifiedRef.is_local_to` — `def is_local_to(self, domain: str) -> bool`-- `pipelex/core/qualified_ref.py:161` — `QualifiedRef.is_external_to` — `def is_external_to(self, domain: str) -> bool`-- `pipelex/core/qualified_ref.py:175` — `QualifiedRef.is_address_based_alias` — `def is_address_based_alias(alias: str) -> bool`-- `pipelex/core/qualified_ref.py:190` — `QualifiedRef.has_cross_package_prefix` — `def has_cross_package_prefix(raw: str) -> bool`-- `pipelex/core/qualified_ref.py:204` — `QualifiedRef.split_cross_package_ref` — `def split_cross_package_ref(raw: str) -> CrossPackageRef`-- `pipelex/core/stuffs/image_field_search.py:130` — `check_generic_container_for_images` — `def check_generic_container_for_images(container_type: Any) -> bool`-- `pipelex/core/stuffs/list_content.py:21` — `ListContent.get_items` — `def get_items(self, item_type: type[StuffContent]) -> list[StuffContent]`-- `pipelex/core/stuffs/structured_content.py:44` — `StructuredContent._render_value_html` — `def _render_value_html(self, value: Any) -> str`-- `pipelex/core/stuffs/stuff.py:38` — `Stuff.make_stuff_name` — `def make_stuff_name(cls, concept: Concept) -> str`-- `pipelex/core/stuffs/stuff.py:95` — `Stuff.content_as` — `def content_as(self, content_type: type[StuffContentType]) -> StuffContentType`-- `pipelex/core/stuffs/stuff.py:170` — `Stuff.as_list_of_fixed_content_type` — `def as_list_of_fixed_content_type(self, item_type: type[StuffContentType]) -> ListContent[StuffContentType]`-- `pipelex/core/stuffs/stuff.py:250` — `Stuff.pretty_print_stuff` — `def pretty_print_stuff(self, title: str | None=None) -> None`-- `pipelex/core/stuffs/stuff_artefact.py:334` — `StuffArtefact.rendered_for_template_async` — `async def rendered_for_template_async(self, text_format: TextFormat) -> str`-- `pipelex/core/stuffs/stuff_content.py:79` — `StuffContent.rendered_for_prompt` — `def rendered_for_prompt(self, text_format: TextFormat=TextFormat.PLAIN) -> str`-- `pipelex/core/stuffs/stuff_content.py:95` — `StuffContent.rendered_for_template_async` — `async def rendered_for_template_async(self, text_format: TextFormat=TextFormat.PLAIN) -> str`-- `pipelex/core/stuffs/stuff_content.py:135` — `StuffContent.pretty_print_content` — `def pretty_print_content(self, title: str | None=None) -> None`-- `pipelex/core/stuffs/stuff_factory.py:47` — `StuffFactory.make_stuff_name` — `def make_stuff_name(cls, concept: Concept) -> str`-- `pipelex/core/stuffs/stuff_factory.py:91` — `StuffFactory.make_from_blueprint` — `def make_from_blueprint(cls, blueprint: StuffBlueprint) -> 'Stuff'`-- `pipelex/core/stuffs/stuff_viewer.py:118` — `_get_html_tab_label` — `def _get_html_tab_label(content_type: str | None) -> str`--### `errors` (12)--- `pipelex/errors/error_pages_generator.py:106` — `_is_production_subclass` — `def _is_production_subclass(cls: type[PipelexError]) -> bool`-- `pipelex/errors/error_pages_generator.py:195` — `page_slug` — `def page_slug(cls: type[PipelexError]) -> str`-- `pipelex/errors/error_pages_generator.py:200` — `render_error_page` — `def render_error_page(cls: type[PipelexError]) -> str`-- `pipelex/errors/error_pages_generator.py:423` — `_subsystem_key` — `def _subsystem_key(cls: type[PipelexError]) -> str`-- `pipelex/errors/error_pages_generator.py:435` — `_humanize_subsystem` — `def _humanize_subsystem(key: str) -> str`-- `pipelex/errors/error_pages_generator.py:440` — `_group_by_subsystem` — `def _group_by_subsystem(classes: Iterable[type[PipelexError]]) -> dict[str, list[type[PipelexError]]]`-- `pipelex/errors/error_pages_generator.py:505` — `render_index_page` — `def render_index_page(by_subsystem: dict[str, list[type[PipelexError]]]) -> str`-- `pipelex/errors/error_pages_generator.py:550` — `has_authored_marker` — `def has_authored_marker(content: str) -> bool`-- `pipelex/errors/error_pages_generator.py:560` — `_resolve_class_level_domain` — `def _resolve_class_level_domain(cls: type[PipelexError]) -> str`-- `pipelex/errors/error_pages_generator.py:570` — `_resolve_class_level_user_action` — `def _resolve_class_level_user_action(cls: type[PipelexError]) -> str | None`-- `pipelex/errors/error_pages_generator.py:581` — `_parent_link` — `def _parent_link(cls: type[PipelexError]) -> str`-- `pipelex/errors/error_pages_generator.py:589` — `_short_docstring` — `def _short_docstring(cls: type[PipelexError]) -> str | None`--### `graph` (27)--- `pipelex/graph/graph_analysis.py:87` — `GraphAnalysis.from_graphspec` — `def from_graphspec(cls, graph: GraphSpec) -> 'GraphAnalysis'`-- `pipelex/graph/graph_analysis.py:169` — `GraphAnalysis.get_children` — `def get_children(self, node_id: str) -> list[str]`-- `pipelex/graph/graph_analysis.py:180` — `GraphAnalysis.is_controller` — `def is_controller(self, node_id: str) -> bool`-- `pipelex/graph/graph_analysis.py:191` — `GraphAnalysis.is_root` — `def is_root(self, node_id: str) -> bool`-- `pipelex/graph/graph_analysis.py:202` — `GraphAnalysis.get_stuff_info` — `def get_stuff_info(self, digest: str) -> StuffInfo | None`-- `pipelex/graph/graph_analysis.py:213` — `GraphAnalysis.get_producer` — `def get_producer(self, digest: str) -> str | None`-- `pipelex/graph/graph_analysis.py:224` — `GraphAnalysis.get_consumers` — `def get_consumers(self, digest: str) -> list[str]`-- `pipelex/graph/graph_rendering.py:19` — `_sanitize_graph_name` — `def _sanitize_graph_name(graph_name: str) -> str`-- `pipelex/graph/graph_tracer.py:164` — `GraphTracer._emit_event` — `def _emit_event(self, event: TraceEvent) -> None`-- `pipelex/graph/graph_tracer_manager.py:58` — `GraphTracerManager.get_instance_tracer` — `def get_instance_tracer(cls, lookup_key: str) -> GraphTracerProtocol | None`-- `pipelex/graph/graph_tracer_manager.py:79` — `GraphTracerManager._get_tracer` — `def _get_tracer(self, graph_id: str) -> GraphTracer | None`-- `pipelex/graph/graph_tracer_manager.py:174` — `GraphTracerManager.close_tracer` — `def close_tracer(self, tracer_key: str) -> GraphSpec | None`-- `pipelex/graph/graph_tracer_manager.py:188` — `GraphTracerManager.get_tracer` — `def get_tracer(self, graph_id: str) -> GraphTracer | None`-- `pipelex/graph/mermaidflow/mermaidflow_factory.py:333` — `MermaidflowFactory._get_node_label` — `def _get_node_label(cls, node: NodeSpec) -> str`-- `pipelex/graph/mermaidflow/mermaidflow_utils.py:9` — `make_stuff_id` — `def make_stuff_id(digest: str) -> str`-- `pipelex/graph/mermaidflow/stuff_collector.py:63` — `collect_stuff_data` — `def collect_stuff_data(graph: GraphSpec) -> dict[str, Any]`-- `pipelex/graph/mermaidflow/stuff_collector.py:76` — `collect_stuff_data_text` — `def collect_stuff_data_text(graph: GraphSpec) -> dict[str, str]`-- `pipelex/graph/mermaidflow/stuff_collector.py:89` — `collect_stuff_data_html` — `def collect_stuff_data_html(graph: GraphSpec) -> dict[str, str]`-- `pipelex/graph/mermaidflow/stuff_collector.py:102` — `collect_stuff_content_type` — `def collect_stuff_content_type(graph: GraphSpec) -> dict[str, str]`-- `pipelex/graph/mermaidflow/stuff_collector.py:117` — `collect_stuff_metadata` — `def collect_stuff_metadata(graph: GraphSpec) -> dict[str, dict[str, str]]`-- `pipelex/graph/mermaidflow/stuff_collector.py:127` — `collect_stuff_metadata.extract_metadata` — `def extract_metadata(io_spec: IOSpec) -> dict[str, str]`-- `pipelex/graph/reactflow/reactflow_html.py:20` — `_build_viewer_config` — `def _build_viewer_config(config: ReactFlowRenderingConfig) -> dict[str, object]`-- `pipelex/graph/validation.py:10` — `validate_graphspec` — `def validate_graphspec(graph: GraphSpec) -> None`-- `pipelex/graph/validation.py:32` — `_validate_unique_node_ids` — `def _validate_unique_node_ids(graph: GraphSpec) -> None`-- `pipelex/graph/validation.py:42` — `_validate_unique_edge_ids` — `def _validate_unique_edge_ids(graph: GraphSpec) -> None`-- `pipelex/graph/validation.py:52` — `_validate_edge_references` — `def _validate_edge_references(graph: GraphSpec) -> None`-- `pipelex/graph/validation.py:65` — `_validate_failed_nodes_have_errors` — `def _validate_failed_nodes_have_errors(graph: GraphSpec) -> None`--### `hub.py` (44)--- `pipelex/hub.py:112` — `PipelexHub.set_instance` — `def set_instance(cls, pipelex_hub: 'PipelexHub') -> None`-- `pipelex/hub.py:137` — `PipelexHub.set_config` — `def set_config(self, config: ConfigRoot)`-- `pipelex/hub.py:148` — `PipelexHub.set_console_print_target` — `def set_console_print_target(self, target: ConsoleTarget)`-- `pipelex/hub.py:158` — `PipelexHub.set_console` — `def set_console(self, console: Console)`-- `pipelex/hub.py:161` — `PipelexHub.set_secrets_provider` — `def set_secrets_provider(self, secrets_provider: SecretsProviderAbstract)`-- `pipelex/hub.py:164` — `PipelexHub.set_storage_provider` — `def set_storage_provider(self, storage_provider: StorageProviderAbstract | None)`-- `pipelex/hub.py:167` — `PipelexHub.set_class_registry` — `def set_class_registry(self, class_registry: ClassRegistryAbstract)`-- `pipelex/hub.py:170` — `PipelexHub.set_telemetry_manager` — `def set_telemetry_manager(self, telemetry_manager: TelemetryManagerAbstract)`-- `pipelex/hub.py:175` — `PipelexHub.set_models_manager` — `def set_models_manager(self, models_manager: ModelManagerAbstract)`-- `pipelex/hub.py:178` — `PipelexHub.set_plugin_manager` — `def set_plugin_manager(self, plugin_manager: PluginManager)`-- `pipelex/hub.py:181` — `PipelexHub.set_inference_manager` — `def set_inference_manager(self, inference_manager: InferenceManagerProtocol)`-- `pipelex/hub.py:184` — `PipelexHub.set_report_delegate` — `def set_report_delegate(self, reporting_delegate: ReportingProtocol)`-- `pipelex/hub.py:187` — `PipelexHub.set_content_generator` — `def set_content_generator(self, content_generator: ContentGeneratorProtocol)`-- `pipelex/hub.py:190` — `PipelexHub.set_dry_run_forced` — `def set_dry_run_forced(self, is_forced: bool) -> None`-- `pipelex/hub.py:198` — `PipelexHub.set_domain_library` — `def set_domain_library(self, domain_library: DomainLibraryAbstract)`-- `pipelex/hub.py:201` — `PipelexHub.set_concept_library` — `def set_concept_library(self, concept_library: ConceptLibraryAbstract)`-- `pipelex/hub.py:204` — `PipelexHub.set_pipe_library` — `def set_pipe_library(self, pipe_library: PipeLibraryAbstract)`-- `pipelex/hub.py:207` — `PipelexHub.set_pipe_router` — `def set_pipe_router(self, pipe_router: 'PipeRouterProtocol')`-- `pipelex/hub.py:210` — `PipelexHub.set_pipe_run` — `def set_pipe_run(self, pipe_run: 'PipeRunProtocol') -> None`-- `pipelex/hub.py:213` — `PipelexHub.set_pipeline_manager` — `def set_pipeline_manager(self, pipeline_manager: PipelineManagerAbstract)`-- `pipelex/hub.py:216` — `PipelexHub.set_observer` — `def set_observer(self, observer: ObserverProtocol)`-- `pipelex/hub.py:357` — `PipelexHub.set_library_manager` — `def set_library_manager(self, library_manager: LibraryManagerAbstract)`-- `pipelex/hub.py:360` — `PipelexHub.set_default_library_dirs` — `def set_default_library_dirs(self, library_dirs: list[Path] | None) -> None`-- `pipelex/hub.py:378` — `PipelexHub.set_func_registry` — `def set_func_registry(self, func_registry: FuncRegistry)`-- `pipelex/hub.py:389` — `set_pipelex_hub` — `def set_pipelex_hub(pipelex_hub: PipelexHub)`-- `pipelex/hub.py:455` — `get_llm_worker` — `def get_llm_worker(llm_handle: str) -> LLMWorkerAbstract`-- `pipelex/hub.py:461` — `get_img_gen_worker` — `def get_img_gen_worker(img_gen_handle: str) -> ImgGenWorkerAbstract`-- `pipelex/hub.py:467` — `get_extract_worker` — `def get_extract_worker(extract_handle: str) -> ExtractWorkerAbstract`-- `pipelex/hub.py:481` — `scoped_content_generator` — `def scoped_content_generator(content_generator: ContentGeneratorProtocol) -> Generator[None, None, None]`-- `pipelex/hub.py:515` — `get_secret` — `def get_secret(secret_id: str) -> str`-- `pipelex/hub.py:525` — `set_current_library` — `def set_current_library(library_id: str) -> None`-- `pipelex/hub.py:561` — `scoped_current_library` — `def scoped_current_library(library_id: str) -> Generator[None, None, None]`-- `pipelex/hub.py:578` — `resolve_library_dirs` — `def resolve_library_dirs(library_dirs: Sequence[str | Path] | None=None) -> tuple[list[Path], str]`-- `pipelex/hub.py:612` — `get_required_domain` — `def get_required_domain(domain_code: str) -> Domain`-- `pipelex/hub.py:616` — `get_optional_domain` — `def get_optional_domain(domain_code: str) -> Domain | None`-- `pipelex/hub.py:628` — `get_required_pipe` — `def get_required_pipe(pipe_code: str) -> PipeAbstract`-- `pipelex/hub.py:632` — `get_optional_pipe` — `def get_optional_pipe(pipe_code: str) -> PipeAbstract | None`-- `pipelex/hub.py:636` — `get_pipe_source` — `def get_pipe_source(pipe_code: str) -> Path | None`-- `pipelex/hub.py:652` — `get_required_concept` — `def get_required_concept(concept_ref: str) -> Concept`-- `pipelex/hub.py:659` — `set_pipe_router` — `def set_pipe_router(pipe_router: 'PipeRouterProtocol') -> None`-- `pipelex/hub.py:678` — `scoped_pipe_router` — `def scoped_pipe_router(pipe_router: 'PipeRouterProtocol') -> Generator[None, None, None]`-- `pipelex/hub.py:712` — `scoped_event_log` — `def scoped_event_log(event_log: 'EventLogProtocol') -> Generator[None, None, None]`-- `pipelex/hub.py:751` — `get_pipeline` — `def get_pipeline(pipeline_run_id: str) -> Pipeline`-- `pipelex/hub.py:763` — `get_native_concept` — `def get_native_concept(native_concept: NativeConceptCode) -> Concept`--### `kit` (3)--- `pipelex/kit/cursor_rules.py:20` — `_iter_agent_files` — `def _iter_agent_files(agents_dir: Traversable) -> Iterable[tuple[str, str]]`-- `pipelex/kit/cursor_rules.py:55` — `_is_pipelex_managed` — `def _is_pipelex_managed(mdc_path: Path) -> bool`-- `pipelex/kit/single_file_agent_rules.py:39` — `_demote_headings.demote_match` — `def demote_match(match: re.Match[str]) -> str`--### `language` (16)--- `pipelex/language/mthds_factory.py:36` — `MthdsFactory.format_tomlkit_string` — `def format_tomlkit_string(cls, text: str) -> Any`-- `pipelex/language/mthds_factory.py:163` — `MthdsFactory.add_spaces_to_inline_tables` — `def add_spaces_to_inline_tables(cls, toml_string: str) -> str`-- `pipelex/language/mthds_factory.py:171` — `MthdsFactory.add_spaces_to_inline_tables.find_and_replace_inline_tables` — `def find_and_replace_inline_tables(text: str) -> str`-- `pipelex/language/mthds_factory.py:228` — `MthdsFactory.make_template_table` — `def make_template_table(cls, template_value: Mapping[str, Any]) -> Any`-- `pipelex/language/mthds_factory.py:238` — `MthdsFactory.make_construct_table` — `def make_construct_table(cls, construct_value: Mapping[str, Any]) -> Any`-- `pipelex/language/mthds_factory.py:250` — `MthdsFactory.make_table_obj_for_pipe` — `def make_table_obj_for_pipe(cls, section_value: Mapping[str, Any]) -> Any`-- `pipelex/language/mthds_factory.py:267` — `MthdsFactory.make_table_obj_for_concept` — `def make_table_obj_for_concept(cls, section_value: Mapping[str, Any]) -> Any`-- `pipelex/language/mthds_factory.py:324` — `MthdsFactory.dict_to_mthds_styled_toml` — `def dict_to_mthds_styled_toml(cls, data: Mapping[str, Any]) -> str`-- `pipelex/language/mthds_factory.py:358` — `MthdsFactory.make_mthds_content` — `def make_mthds_content(cls, blueprint: PipelexBundleBlueprint) -> str`-- `pipelex/language/mthds_schema_generator.py:57` — `_remove_internal_fields` — `def _remove_internal_fields(schema: dict[str, Any]) -> dict[str, Any]`-- `pipelex/language/mthds_schema_generator.py:98` — `_promote_schema_required_fields` — `def _promote_schema_required_fields(schema: dict[str, Any]) -> dict[str, Any]`-- `pipelex/language/mthds_schema_generator.py:125` — `_require_type_on_pipe_definitions` — `def _require_type_on_pipe_definitions(schema: dict[str, Any]) -> dict[str, Any]`-- `pipelex/language/mthds_schema_generator.py:152` — `_convert_to_draft4` — `def _convert_to_draft4(schema: dict[str, Any]) -> dict[str, Any]`-- `pipelex/language/mthds_schema_generator.py:173` — `_draft4_visitor` — `def _draft4_visitor(node: dict[str, Any]) -> None`-- `pipelex/language/mthds_schema_generator.py:198` — `_patch_construct_schema` — `def _patch_construct_schema(schema: dict[str, Any]) -> dict[str, Any]`-- `pipelex/language/mthds_schema_generator.py:287` — `_add_taplo_metadata` — `def _add_taplo_metadata(schema: dict[str, Any]) -> dict[str, Any]`--### `libraries` (46)--- `pipelex/libraries/concept/concept_library.py:27` — `ConceptLibrary.set_concept_resolver` — `def set_concept_resolver(self, resolver: Callable[[str], Concept | None]) -> None`-- `pipelex/libraries/concept/concept_library.py:107` — `ConceptLibrary.get_optional_concept` — `def get_optional_concept(self, concept_ref: str) -> Concept | None`-- `pipelex/libraries/concept/concept_library.py:204` — `ConceptLibrary.is_concept_exists` — `def is_concept_exists(self, concept_ref: str) -> bool`-- `pipelex/libraries/concept/concept_library_abstract.py:9` — `ConceptLibraryAbstract.add_new_concept` — `def add_new_concept(self, concept: Concept) -> None`-- `pipelex/libraries/concept/concept_library_abstract.py:13` — `ConceptLibraryAbstract.add_concepts` — `def add_concepts(self, concepts: list[Concept]) -> None`-- `pipelex/libraries/concept/concept_library_abstract.py:17` — `ConceptLibraryAbstract.remove_concepts_by_concept_refs` — `def remove_concepts_by_concept_refs(self, concept_refs: list[str]) -> None`-- `pipelex/libraries/concept/concept_library_abstract.py:21` — `ConceptLibraryAbstract.list_concepts_by_domain` — `def list_concepts_by_domain(self, domain_code: str) -> list[Concept]`-- `pipelex/libraries/concept/concept_library_abstract.py:29` — `ConceptLibraryAbstract.get_required_concept` — `def get_required_concept(self, concept_ref: str) -> Concept`-- `pipelex/libraries/concept/concept_library_abstract.py:49` — `ConceptLibraryAbstract.get_native_concept` — `def get_native_concept(self, native_concept: NativeConceptCode) -> Concept`-- `pipelex/libraries/domain/domain_library.py:33` — `DomainLibrary.add_domain` — `def add_domain(self, domain: Domain)`-- `pipelex/libraries/domain/domain_library.py:59` — `DomainLibrary.add_domains` — `def add_domains(self, domains: list[Domain])`-- `pipelex/libraries/domain/domain_library.py:63` — `DomainLibrary.remove_domain_by_code` — `def remove_domain_by_code(self, domain_code: str) -> None`-- `pipelex/libraries/domain/domain_library_abstract.py:8` — `DomainLibraryAbstract.get_domain` — `def get_domain(self, domain_code: str) -> Domain | None`-- `pipelex/libraries/domain/domain_library_abstract.py:12` — `DomainLibraryAbstract.get_required_domain` — `def get_required_domain(self, domain_code: str) -> Domain`-- `pipelex/libraries/library.py:44` — `Library.set_class_registry` — `def set_class_registry(self, class_registry: ClassRegistry) -> None`-- `pipelex/libraries/library.py:56` — `Library.get_dependency_library` — `def get_dependency_library(self, alias: str) -> 'Library | None'`-- `pipelex/libraries/library.py:67` — `Library.resolve_concept` — `def resolve_concept(self, concept_ref: str) -> 'Concept | None'`-- `pipelex/libraries/library.py:148` — `Library._has_unresolved_cross_package_deps` — `def _has_unresolved_cross_package_deps(self, pipe: PipeController) -> bool`-- `pipelex/libraries/library_crate_factory.py:33` — `LibraryCrateFactory.make_from_blueprints` — `def make_from_blueprints(cls, blueprints: list[PipelexBundleBlueprint]) -> LibraryCrate`-- `pipelex/libraries/library_crate_factory.py:199` — `LibraryCrateFactory._declaration_sort_key` — `def _declaration_sort_key(declaration: PipeDeclaration) -> tuple[str, str]`-- `pipelex/libraries/library_manager.py:62` — `_find_methods_dirs_from_blueprints` — `def _find_methods_dirs_from_blueprints(blueprints: list[PipelexBundleBlueprint]) -> list[Path]`-- `pipelex/libraries/library_manager.py:116` — `LibraryManager._pop_and_teardown_library` — `def _pop_and_teardown_library(self, library_id: str) -> bool`-- `pipelex/libraries/library_manager.py:621` — `LibraryManager._load_concepts_from_blueprints` — `def _load_concepts_from_blueprints(self, blueprints: list[PipelexBundleBlueprint]) -> list['Concept']`-- `pipelex/libraries/library_manager.py:670` — `LibraryManager._load_concepts_from_crate` — `def _load_concepts_from_crate(self, concepts: dict[str, ConceptBlueprint | str]) -> list['Concept']`-- `pipelex/libraries/library_manager.py:700` — `LibraryManager._topological_load_concepts` — `def _topological_load_concepts(self, ref_to_entry: 'Mapping[str, tuple[str, str, ConceptBlueprint | str]]') -> list['Concept']`-- `pipelex/libraries/library_manager.py:916` — `LibraryManager._find_package_root` — `def _find_package_root(self, mthds_paths: list[Path]) -> Path | None`-- `pipelex/libraries/library_manager.py:1237` — `LibraryManager._remove_pipes_from_blueprint` — `def _remove_pipes_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> None`-- `pipelex/libraries/library_manager.py:1245` — `LibraryManager._remove_concepts_from_blueprint` — `def _remove_concepts_from_blueprint(self, blueprint: PipelexBundleBlueprint) -> None`-- `pipelex/libraries/library_manager.py:1264` — `LibraryManager._rebuild_models_with_forward_refs` — `def _rebuild_models_with_forward_refs(self, concepts: list['Concept']) -> None`-- `pipelex/libraries/library_manager.py:1295` — `LibraryManager._detect_concept_cycles` — `def _detect_concept_cycles(self, concepts: list['Concept']) -> None`-- `pipelex/libraries/library_manager.py:1316` — `LibraryManager._detect_concept_cycles.get_referenced_concepts` — `def get_referenced_concepts(concept: 'Concept') -> list[str]`-- `pipelex/libraries/library_manager.py:1323` — `LibraryManager._detect_concept_cycles.get_referenced_concepts.extract_type_names` — `def extract_type_names(annotation: type) -> list[str]`-- `pipelex/libraries/library_manager_abstract.py:22` — `LibraryManagerAbstract.teardown` — `def teardown(self, library_id: str | None=None) -> None`-- `pipelex/libraries/library_manager_abstract.py:30` — `LibraryManagerAbstract.open_library` — `def open_library(self, library_id: str | None=None) -> tuple[str, 'Library']`-- `pipelex/libraries/library_manager_abstract.py:34` — `LibraryManagerAbstract.open_fresh_library` — `def open_fresh_library(self, library_id: str) -> 'Library'`-- `pipelex/libraries/library_manager_abstract.py:45` — `LibraryManagerAbstract.get_library` — `def get_library(self, library_id: str) -> 'Library'`-- `pipelex/libraries/library_manager_abstract.py:52` — `LibraryManagerAbstract.get_library_class_registry` — `def get_library_class_registry(self, library_id: str) -> ClassRegistry | None`-- `pipelex/libraries/library_manager_abstract.py:59` — `LibraryManagerAbstract.get_pipe_source` — `def get_pipe_source(self, pipe_code: str) -> Path | None`-- `pipelex/libraries/library_manager_abstract.py:71` — `LibraryManagerAbstract.get_crate` — `def get_crate(self, library_id: str) -> LibraryCrate | None`-- `pipelex/libraries/library_utils.py:65` — `get_pipelex_mthds_files_from_dirs` — `def get_pipelex_mthds_files_from_dirs(dirs: set[Path]) -> list[Path]`-- `pipelex/libraries/pipe/pipe_library_abstract.py:21` — `PipeLibraryAbstract.get_required_pipe` — `def get_required_pipe(self, pipe_code: str) -> PipeAbstract`-- `pipelex/libraries/pipe/pipe_library_abstract.py:25` — `PipeLibraryAbstract.get_optional_pipe` — `def get_optional_pipe(self, pipe_code: str) -> PipeAbstract | None`-- `pipelex/libraries/pipe/pipe_library_abstract.py:36` — `PipeLibraryAbstract.remove_pipes_by_refs` — `def remove_pipes_by_refs(self, pipe_refs: list[str]) -> None`-- `pipelex/libraries/pipe/pipe_library_abstract.py:44` — `PipeLibraryAbstract.add_new_pipe` — `def add_new_pipe(self, pipe: PipeAbstract) -> None`-- `pipelex/libraries/pipe/pipe_library_abstract.py:48` — `PipeLibraryAbstract.add_pipes` — `def add_pipes(self, pipes: list[PipeAbstract]) -> None`-- `pipelex/libraries/visibility_utils.py:14` — `_blueprint_to_metadata` — `def _blueprint_to_metadata(blueprint: PipelexBundleBlueprint) -> BundleMetadata`--### `observer` (4)--- `pipelex/observer/multi_observer.py:14` — `MultiObserver.remove_observer` — `def remove_observer(self, name: str) -> None`-- `pipelex/observer/observer_protocol.py:18` — `ObserverProtocol.observe_before_run` — `async def observe_before_run(self, payload: PayloadType) -> None`-- `pipelex/observer/observer_protocol.py:25` — `ObserverProtocol.observe_after_successful_run` — `async def observe_after_successful_run(self, payload: PayloadType) -> None`-- `pipelex/observer/observer_protocol.py:32` — `ObserverProtocol.observe_after_failing_run` — `async def observe_after_failing_run(self, payload: PayloadType) -> None`--### `pipe_controllers` (5)--- `pipelex/pipe_controllers/batch/pipe_batch.py:35` — `resolve_batch_max_concurrency` — `def resolve_batch_max_concurrency(max_concurrency_setting: int | str) -> int | None`-- `pipelex/pipe_controllers/condition/pipe_condition.py:197` — `PipeCondition._evaluate_expression` — `async def _evaluate_expression(self, working_memory: WorkingMemory) -> str`-- `pipelex/pipe_controllers/condition/special_outcome.py:15` — `SpecialOutcome.is_continue` — `def is_continue(cls, outcome: str) -> bool`-- `pipelex/pipe_controllers/condition/special_outcome.py:30` — `SpecialOutcome.is_fail` — `def is_fail(cls, outcome: str) -> bool`-- `pipelex/pipe_controllers/sub_pipe_factory.py:13` — `SubPipeFactory.make_from_blueprint` — `def make_from_blueprint(cls, blueprint: SubPipeBlueprint) -> SubPipe`--### `pipe_operators` (13)--- `pipelex/pipe_operators/compose/construct_blueprint.py:108` — `ConstructFieldBlueprint.make_from_raw` — `def make_from_raw(cls, raw: Any) -> ConstructFieldBlueprint`-- `pipelex/pipe_operators/compose/construct_blueprint.py:294` — `ConstructBlueprint.make_from_raw` — `def make_from_raw(cls, raw: dict[str, Any]) -> ConstructBlueprint`-- `pipelex/pipe_operators/compose/structured_content_composer.py:78` — `StructuredContentComposer._build_field_type_summary` — `def _build_field_type_summary(self, field_values: dict[str, Any]) -> str`-- `pipelex/pipe_operators/compose/structured_content_composer.py:377` — `StructuredContentComposer._get_field_expected_type` — `def _get_field_expected_type(self, field_name: str) -> type[Any] | None`-- `pipelex/pipe_operators/compose/structured_content_composer.py:451` — `StructuredContentComposer._expects_list_content_type` — `def _expects_list_content_type(self, expected_type: type[Any]) -> bool`-- `pipelex/pipe_operators/compose/structured_content_composer.py:474` — `StructuredContentComposer._expects_list_type` — `def _expects_list_type(self, expected_type: type[Any]) -> bool`-- `pipelex/pipe_operators/compose/structured_content_composer.py:486` — `StructuredContentComposer._get_list_item_type` — `def _get_list_item_type(self, expected_type: type[Any]) -> type[Any] | None`-- `pipelex/pipe_operators/compose/structured_content_composer.py:624` — `StructuredContentComposer._resolve_template` — `async def _resolve_template(self, field_blueprint: ConstructFieldBlueprint) -> str`-- `pipelex/pipe_operators/compose/structured_content_composer.py:691` — `StructuredContentComposer._get_nested_field_class` — `def _get_nested_field_class(self, field_name: str) -> type[StuffContent]`-- `pipelex/pipe_operators/llm/helpers.py:12` — `get_output_structure_prompt` — `async def get_output_structure_prompt(concept_ref: str) -> str | None`-- `pipelex/pipe_operators/llm/pipe_llm.py:141` — `PipeLLM.resolve_dynamic_output_stuff_spec` — `def resolve_dynamic_output_stuff_spec(self, pipe_run_params: PipeRunParams) -> StuffSpec`-- `pipelex/pipe_operators/llm/pipe_llm.py:383` — `PipeLLM._format_error_location` — `def _format_error_location(self, pipe_run_params: PipeRunParams) -> str`-- `pipelex/pipe_operators/structure/pipe_structure.py:204` — `PipeStructure._format_error_location` — `def _format_error_location(self, pipe_run_params: PipeRunParams) -> str`--### `pipe_run` (9)--- `pipelex/pipe_run/delivery_executor.py:77` — `DeliveryExecutor.generate_result_files` — `async def generate_result_files(self, pipe_output: PipeOutput) -> dict[str, ResultFile]`-- `pipelex/pipe_run/delivery_executor.py:121` — `DeliveryExecutor._get_raw_main_stuff_dict` — `def _get_raw_main_stuff_dict(cls, working_memory_raw: dict[str, Any]) -> dict[str, Any] | None`-- `pipelex/pipe_run/delivery_executor.py:132` — `DeliveryExecutor.try_local_hydrate_stuff` — `def try_local_hydrate_stuff(cls, stuff_raw: dict[str, Any]) -> Stuff | None`-- `pipelex/pipe_run/pipe_router_protocol.py:14` — `PipeRouterProtocol._before_run` — `async def _before_run(self, pipe_job: PipeJob) -> None`-- `pipelex/pipe_run/pipe_router_protocol.py:50` — `PipeRouterProtocol.run` — `async def run(self, pipe_job: PipeJob) -> PipeOutput`-- `pipelex/pipe_run/pipe_router_protocol.py:81` — `PipeRouterProtocol._run_pipe_job` — `async def _run_pipe_job(self, pipe_job: PipeJob) -> PipeOutput`-- `pipelex/pipe_run/pipe_run_params.py:229` — `PipeRunParams.push_pipe_to_stack` — `def push_pipe_to_stack(self, pipe_code: str) -> None`-- `pipelex/pipe_run/pipe_run_params.py:239` — `PipeRunParams.pop_pipe_from_stack` — `def pop_pipe_from_stack(self, pipe_code: str) -> None`-- `pipelex/pipe_run/pipe_run_params.py:249` — `PipeRunParams.push_pipe_layer` — `def push_pipe_layer(self, pipe_code: str) -> None`--### `pipelex.py` (1)--- `pipelex/pipelex.py:140` — `Pipelex._get_config_file_not_found_error_msg` — `def _get_config_file_not_found_error_msg(component_name: str) -> str`--### `pipeline` (13)--- `pipelex/pipeline/blueprint_selection.py:26` — `select_primary_blueprint` — `def select_primary_blueprint(blueprints: Sequence[PipelexBundleBlueprint]) -> PrimaryBlueprintSelection`-- `pipelex/pipeline/bundle_validator.py:270` — `BundleValidator._signature_pre_pass` — `def _signature_pre_pass(cls, pipes: list[PipeAbstract]) -> None`-- `pipelex/pipeline/execution_seams.py:132` — `load_libraries_and_activate` — `def load_libraries_and_activate(library_dirs: Sequence[str | Path] | None=None) -> str`-- `pipelex/pipeline/input_normalizer.py:32` — `normalize_data_urls_to_storage` — `async def normalize_data_urls_to_storage(working_memory: WorkingMemory) -> WorkingMemory`-- `pipelex/pipeline/pipe_io_contracts.py:63` — `build_pipe_io_contracts` — `def build_pipe_io_contracts(pipes: Sequence[PipeAbstract]) -> dict[str, PipeIOContract]`-- `pipelex/pipeline/pipeline_factory.py:9` — `PipelineFactory.make_pipeline` — `def make_pipeline(cls, pipeline_run_id: str | None=None) -> Pipeline`-- `pipelex/pipeline/pipeline_manager_abstract.py:16` — `PipelineManagerAbstract.get_optional_pipeline` — `def get_optional_pipeline(self, pipeline_run_id: str) -> Pipeline | None`-- `pipelex/pipeline/pipeline_manager_abstract.py:20` — `PipelineManagerAbstract.get_pipeline` — `def get_pipeline(self, pipeline_run_id: str) -> Pipeline`-- `pipelex/pipeline/pipeline_manager_abstract.py:28` — `PipelineManagerAbstract.remove_pipeline` — `def remove_pipeline(self, pipeline_run_id: str) -> None`-- `pipelex/pipeline/validate_bundle.py:68` — `build_validated_pipes` — `def build_validated_pipes(dry_run_result: dict[str, DryRunOutput]) -> list[ValidatedPipeEntry]`-- `pipelex/pipeline/validate_bundle.py:79` — `build_pending_signatures` — `def build_pending_signatures(pipes_by_ref: dict[str, PipeAbstract]) -> list[str]`-- `pipelex/pipeline/validate_bundle.py:95` — `_translate_to_validate_bundle_error` — `def _translate_to_validate_bundle_error(category: Literal['pipe', 'concept']) -> Generator[None, None, None]`-- `pipelex/pipeline/validate_bundle.py:463` — `load_concepts_only_from_directory` — `def load_concepts_only_from_directory(directory: Path) -> LoadConceptsOnlyResult`--### `plugins` (83)--- `pipelex/plugins/anthropic/anthropic_config.py:26` — `AnthropicConfig.get_reasoning_level` — `def get_reasoning_level(self, effort: ReasoningEffort) -> AnthropicEffortLevel | None`-- `pipelex/plugins/anthropic/anthropic_factory.py:78` — `AnthropicFactory._make_image_block_param` — `def _make_image_block_param(prepped_image: PreparedFile) -> ImageBlockParam`-- `pipelex/plugins/anthropic/anthropic_factory.py:137` — `AnthropicFactory.make_user_message` — `async def make_user_message(cls, llm_job: LLMJob) -> MessageParam`-- `pipelex/plugins/anthropic/anthropic_factory.py:199` — `AnthropicFactory.make_simple_messages` — `async def make_simple_messages(cls, llm_job: LLMJob) -> 'list["ChatCompletionMessageParam"]'`-- `pipelex/plugins/anthropic/anthropic_factory.py:236` — `AnthropicFactory.make_nb_tokens_by_category` — `def make_nb_tokens_by_category(usage: Usage) -> NbTokensByCategoryDict`-- `pipelex/plugins/anthropic/anthropic_factory.py:252` — `AnthropicFactory.calculate_safe_max_tokens_for_timeout` — `def calculate_safe_max_tokens_for_timeout(timeout_seconds: int) -> int`-- `pipelex/plugins/azure_rest/azure_img_gen_worker.py:65` — `AzureImgGenWorker._raise_categorized_azure_status_error` — `def _raise_categorized_azure_status_error(self, exc: httpx.HTTPStatusError) -> None`-- `pipelex/plugins/bedrock/bedrock_factory.py:62` — `BedrockFactory.make_simple_message` — `def make_simple_message(cls, llm_job: LLMJob) -> BedrockMessage`-- `pipelex/plugins/bedrock/bedrock_llm_worker.py:46` — `BedrockLLMWorker._validate_no_reasoning_params` — `def _validate_no_reasoning_params(self, job_params: LLMJobParams) -> None`-- `pipelex/plugins/docling/docling_extract_worker.py:63` — `DoclingExtractWorker._extract_from_source` — `async def _extract_from_source(self, source_uri: str) -> ExtractOutput`-- `pipelex/plugins/docling/docling_factory.py:17` — `DoclingFactory.make_extract_output_from_docling_document` — `def make_extract_output_from_docling_document(cls, doc: 'DoclingDocument') -> ExtractOutput`-- `pipelex/plugins/fal/fal_factory.py:13` — `FalFactory.make_generated_image` — `def make_generated_image(cls, fal_result: dict[str, Any]) -> GeneratedImageRawDetails`-- `pipelex/plugins/fal/fal_factory.py:29` — `FalFactory.make_generated_image_list` — `def make_generated_image_list(cls, fal_result: dict[str, Any]) -> list[GeneratedImageRawDetails]`-- `pipelex/plugins/fal/fal_poller.py:19` — `FalPoller._is_transient_http` — `def _is_transient_http(self, exc: BaseException) -> bool`-- `pipelex/plugins/fal/fal_poller.py:25` — `FalPoller.poll_queue_until_complete` — `async def poll_queue_until_complete(self, response_dict: dict[str, Any]) -> dict[str, Any]`-- `pipelex/plugins/gateway/gateway_completions_factory.py:156` — `GatewayCompletionsFactory._extract_pages_from_choices_content` — `def _extract_pages_from_choices_content(cls, response: GenericResponse) -> list[dict[str, Any]] | None`-- `pipelex/plugins/gateway/gateway_completions_factory.py:187` — `GatewayCompletionsFactory._make_extract_output_from_response_azure` — `def _make_extract_output_from_response_azure(cls, response: GenericResponse) -> ExtractOutput`-- `pipelex/plugins/gateway/gateway_completions_factory.py:226` — `GatewayCompletionsFactory._make_extract_output_from_response_mistral` — `def _make_extract_output_from_response_mistral(cls, response: GenericResponse) -> ExtractOutput`-- `pipelex/plugins/gateway/gateway_completions_factory.py:276` — `GatewayCompletionsFactory._make_extract_output_from_response_deepseek` — `def _make_extract_output_from_response_deepseek(cls, response: GenericResponse) -> ExtractOutput`-- `pipelex/plugins/gateway/gateway_completions_factory.py:311` — `GatewayCompletionsFactory._make_extract_output_from_response_linkup_fetch` — `def _make_extract_output_from_response_linkup_fetch(cls, response: GenericResponse) -> ExtractOutput`-- `pipelex/plugins/gateway/gateway_completions_factory.py:345` — `GatewayCompletionsFactory._extract_content_string_from_response` — `def _extract_content_string_from_response(cls, response: GenericResponse) -> str | None`-- `pipelex/plugins/gateway/gateway_constants.py:9` — `GatewayOpenAISdkVariant.is_completions` — `def is_completions(cls, sdk: str) -> bool`-- `pipelex/plugins/gateway/gateway_constants.py:21` — `GatewayOpenAISdkVariant.is_responses` — `def is_responses(cls, sdk: str) -> bool`-- `pipelex/plugins/gateway/gateway_deck.py:9` — `GatewayDeck.get_config_id` — `def get_config_id(cls, headers: dict[str, str]) -> str`-- `pipelex/plugins/gateway/gateway_extract_worker.py:94` — `GatewayExtractWorker._extract_document` — `async def _extract_document(self, extract_job: ExtractJob) -> ExtractOutput`-- `pipelex/plugins/gateway/gateway_extract_worker.py:128` — `GatewayExtractWorker._extract_web_fetch` — `async def _extract_web_fetch(self, extract_job: ExtractJob) -> ExtractOutput`-- `pipelex/plugins/gateway/gateway_factory.py:32` — `GatewayFactory.is_debug_enabled` — `def is_debug_enabled(cls, backend: InferenceBackend) -> bool`-- `pipelex/plugins/gateway/gateway_factory.py:37` — `GatewayFactory.get_endpoint` — `def get_endpoint(cls, backend: InferenceBackend) -> str`-- `pipelex/plugins/gateway/gateway_factory.py:41` — `GatewayFactory.get_api_key` — `def get_api_key(cls, backend: InferenceBackend) -> str`-- `pipelex/plugins/gateway/gateway_factory.py:48` — `GatewayFactory.make_portkey_client` — `def make_portkey_client(cls, backend: InferenceBackend) -> AsyncPortkey`-- `pipelex/plugins/gateway/gateway_protocols.py:13` — `GatewayExtractProtocol.make_from_model_handle` — `def make_from_model_handle(cls, model_handle: str) -> GatewayExtractProtocol`-- `pipelex/plugins/gateway/gateway_search_worker.py:178` — `GatewaySearchWorker._extract_content` — `def _extract_content(self, response: GenericResponse) -> str`-- `pipelex/plugins/google/google_config.py:32` — `GoogleConfig.get_reasoning_level` — `def get_reasoning_level(self, effort: ReasoningEffort) -> genai_types.ThinkingLevel | None`-- `pipelex/plugins/google/google_factory.py:21` — `GoogleFactory.make_google_client` — `def make_google_client(cls, backend: InferenceBackend) -> GoogleGenAiClient`-- `pipelex/plugins/google/google_factory.py:31` — `GoogleFactory.prepare_image_part` — `async def prepare_image_part(cls, prompt_image: PromptImage) -> genai_types.Part`-- `pipelex/plugins/google/google_factory.py:42` — `GoogleFactory.prepare_document_part` — `async def prepare_document_part(cls, prompt_document: PromptDocument) -> genai_types.Part`-- `pipelex/plugins/google/google_factory.py:53` — `GoogleFactory.prepare_user_contents` — `async def prepare_user_contents(cls, llm_prompt: LLMPrompt) -> genai_types.ContentListUnion`-- `pipelex/plugins/google/google_factory.py:139` — `GoogleFactory.extract_token_usage` — `def extract_token_usage(cls, usage_metadata: genai_types.GenerateContentResponseUsageMetadata | None) -> NbTokensByCategoryDict`-- `pipelex/plugins/google/google_img_gen_factory.py:82` — `GoogleImgGenFactory.aspect_ratio_literal` — `def aspect_ratio_literal(cls, aspect_ratio: AspectRatio) -> GoogleAspectRatioType`-- `pipelex/plugins/huggingface/huggingface_factory.py:6` — `HuggingFaceFactory.make_huggingface_inference_provider` — `def make_huggingface_inference_provider(cls, provider_str: str) -> PROVIDER_OR_POLICY_T`-- `pipelex/plugins/huggingface/huggingface_img_gen_worker.py:37` — `HuggingFaceImgGenWorker._generate_single_image` — `async def _generate_single_image(self, img_gen_job: ImgGenJob) -> Image.Image`-- `pipelex/plugins/linkup/linkup_search_worker.py:42` — `LinkupSearchWorker._parse_date` — `def _parse_date(self, date_str: str | None) -> date | None`-- `pipelex/plugins/mistral/mistral_config.py:29` — `MistralConfig.get_reasoning_level` — `def get_reasoning_level(self, effort: ReasoningEffort) -> MistralPromptMode | None`-- `pipelex/plugins/mistral/mistral_extract_worker.py:62` — `MistralExtractWorker._extract_page_from_image` — `async def _extract_page_from_image(self, image_uri: str) -> ExtractOutput`-- `pipelex/plugins/mistral/mistral_factory.py:50` — `MistralFactory.make_mistral_client` — `def make_mistral_client(cls, backend: InferenceBackend) -> Mistral`-- `pipelex/plugins/mistral/mistral_factory.py:73` — `MistralFactory.make_simple_messages` — `async def make_simple_messages(self, llm_job: LLMJob) -> list[Messages]`-- `pipelex/plugins/mistral/mistral_factory.py:93` — `MistralFactory.make_mistral_image_url` — `async def make_mistral_image_url(self, prompt_image: PromptImage) -> ImageURLChunk`-- `pipelex/plugins/mistral/mistral_factory.py:114` — `MistralFactory.make_mistral_document_url` — `async def make_mistral_document_url(self, prompt_document: PromptDocument) -> DocumentURLChunk`-- `pipelex/plugins/mistral/mistral_factory.py:136` — `MistralFactory.make_simple_messages_openai_typed` — `async def make_simple_messages_openai_typed(self, llm_job: LLMJob) -> 'list["ChatCompletionMessageParam"]'`-- `pipelex/plugins/mistral/mistral_factory.py:189` — `MistralFactory.make_nb_tokens_by_category` — `def make_nb_tokens_by_category(self, usage: UsageInfo) -> NbTokensByCategoryDict`-- `pipelex/plugins/mistral/mistral_factory.py:197` — `MistralFactory.make_extract_output_from_mistral_response` — `async def make_extract_output_from_mistral_response(cls, mistral_extract_response: mistralai.OCRResponse) -> ExtractOutput`-- `pipelex/plugins/mistral/mistral_factory.py:222` — `MistralFactory._clean_mistral_image_base64` — `def _clean_mistral_image_base64(cls, base64_str: str) -> str`-- `pipelex/plugins/mistral/mistral_factory.py:278` — `MistralFactory.make_extracted_image_from_page_from_mistral_ocr_image_obj` — `def make_extracted_image_from_page_from_mistral_ocr_image_obj(cls, mistral_ocr_image_obj: mistralai.OCRImageObject) -> ExtractedImageFromPage`-- `pipelex/plugins/mistral/mistral_factory.py:326` — `MistralFactory.make_mistral_image_url_chunk_from_uri` — `async def make_mistral_image_url_chunk_from_uri(cls, uri: str) -> ImageURLChunkTypedDict`-- `pipelex/plugins/mistral/mistral_llm_worker.py:68` — `MistralLLMWorker._resolve_prompt_mode` — `def _resolve_prompt_mode(self, job_params: LLMJobParams) -> 'OptionalNullable[MistralPromptMode]'`-- `pipelex/plugins/openai/openai_completions_factory.py:27` — `OpenAICompletionsFactory.make_simple_messages` — `async def make_simple_messages(self, llm_job: LLMJob) -> 'list["ChatCompletionMessageParam"]'`-- `pipelex/plugins/openai/openai_completions_factory.py:94` — `OpenAICompletionsFactory._get_document_filename` — `def _get_document_filename(prompt_document: PromptDocument) -> str`-- `pipelex/plugins/openai/openai_completions_factory.py:99` — `OpenAICompletionsFactory.make_nb_tokens_by_category` — `def make_nb_tokens_by_category(self, usage: 'CompletionUsage') -> NbTokensByCategoryDict`-- `pipelex/plugins/openai/openai_completions_img_gen_worker.py:177` — `OpenAICompletionsImgGenWorker._build_messages_with_images` — `async def _build_messages_with_images(self, img_gen_job: ImgGenJob) -> 'list[ChatCompletionMessageParam]'`-- `pipelex/plugins/openai/openai_completions_llm_worker.py:87` — `OpenAICompletionsLLMWorker._resolve_reasoning_effort` — `def _resolve_reasoning_effort(self, job_params: LLMJobParams) -> ChatCompletionReasoningEffort | None`-- `pipelex/plugins/openai/openai_config.py:32` — `OpenAIConfig.get_reasoning_level` — `def get_reasoning_level(self, effort: ReasoningEffort) -> 'ChatCompletionReasoningEffort | None'`-- `pipelex/plugins/openai/openai_func.py:12` — `create_pydantic_model_from_function` — `def create_pydantic_model_from_function(func: Callable[..., Any]) -> type[BaseModel]`-- `pipelex/plugins/openai/openai_func.py:33` — `create_openai_schema_from_function` — `def create_openai_schema_from_function(func: Callable[..., Any]) -> dict[str, Any]`-- `pipelex/plugins/openai/openai_func.py:63` — `list_openai_tools` — `def list_openai_tools(openai_message: ChatCompletionMessage) -> list[str]`-- `pipelex/plugins/openai/openai_img_gen_factory.py:121` — `OpenAIImgGenFactory.moderation_for_openai_image` — `def moderation_for_openai_image(cls, is_moderated: bool | None) -> OpenAIImageModerationType`-- `pipelex/plugins/openai/openai_img_gen_factory.py:130` — `OpenAIImgGenFactory.input_fidelity_for_openai_image` — `def input_fidelity_for_openai_image(cls, input_fidelity: InputFidelity) -> OpenAIImageInputFidelityType`-- `pipelex/plugins/openai/openai_img_gen_factory.py:138` — `OpenAIImgGenFactory._size_to_string` — `def _size_to_string(size: ImageSize) -> str`-- `pipelex/plugins/openai/openai_img_gen_worker.py:186` — `OpenAIImgGenWorker._convert_image_data_urls_for_openai_sdk` — `def _convert_image_data_urls_for_openai_sdk(image_arg: Any) -> list[tuple[str, bytes, str]]`-- `pipelex/plugins/openai/openai_img_gen_worker.py:209` — `OpenAIImgGenWorker._get_requested_size` — `def _get_requested_size(args_dict: dict[str, Any]) -> str | None`-- `pipelex/plugins/openai/openai_responses_factory.py:36` — `OpenAIResponsesFactory.make_input_items` — `async def make_input_items(self, llm_job: LLMJob) -> list[ResponseInputItemParam]`-- `pipelex/plugins/openai/openai_responses_factory.py:93` — `OpenAIResponsesFactory._get_document_filename` — `def _get_document_filename(prompt_document: PromptDocument) -> str`-- `pipelex/plugins/openai/openai_responses_factory.py:98` — `OpenAIResponsesFactory.make_nb_tokens_by_category` — `def make_nb_tokens_by_category(self, usage: ResponseUsage) -> NbTokensByCategoryDict`-- `pipelex/plugins/openai/openai_responses_llm_worker.py:90` — `OpenAIResponsesLLMWorker._resolve_reasoning` — `def _resolve_reasoning(self, job_params: LLMJobParams) -> Reasoning | None`-- `pipelex/plugins/openai/vertexai_factory.py:20` — `VertexAIFactory.make_endpoint_and_api_key` — `def make_endpoint_and_api_key(cls, extra_config: dict[str, Any]) -> tuple[str, str]`-- `pipelex/plugins/openai/vertexai_factory.py:51` — `VertexAIFactory._make_api_key` — `def _make_api_key(cls, gcp_credentials_file_path: str) -> str`-- `pipelex/plugins/plugin.py:19` — `Plugin.make_for_inference_model` — `def make_for_inference_model(cls, inference_model: InferenceModelSpec) -> 'Plugin'`-- `pipelex/plugins/plugin_sdk_registry.py:19` — `PluginSdkRegistry.get_sdk_instance` — `def get_sdk_instance(self, plugin: Plugin) -> Any | None`-- `pipelex/plugins/portkey/portkey_constants.py:17` — `PortkeyOpenAISdkVariant.is_completions` — `def is_completions(cls, sdk: str) -> bool`-- `pipelex/plugins/portkey/portkey_constants.py:29` — `PortkeyOpenAISdkVariant.is_responses` — `def is_responses(cls, sdk: str) -> bool`-- `pipelex/plugins/portkey/portkey_factory.py:24` — `PortkeyFactory.is_debug_enabled` — `def is_debug_enabled(cls, backend: InferenceBackend) -> bool`-- `pipelex/plugins/portkey/portkey_factory.py:29` — `PortkeyFactory.get_endpoint` — `def get_endpoint(cls, backend: InferenceBackend) -> str`-- `pipelex/plugins/portkey/portkey_factory.py:33` — `PortkeyFactory.get_api_key` — `def get_api_key(cls, backend: InferenceBackend) -> str`-- `pipelex/plugins/pypdfium2/pypdfium2_worker.py:38` — `Pypdfium2Worker._resolve_pdf_uri` — `async def _resolve_pdf_uri(self, pdf_uri: str) -> PdfInput`--### `reporting` (7)--- `pipelex/reporting/cost_report_renderer.py:86` — `render_cost_report_for_output` — `def render_cost_report_for_output(pipe_output: PipeOutput) -> None`-- `pipelex/reporting/reporting_manager.py:143` — `ReportingManager._report_llm_job` — `def _report_llm_job(self, llm_job: LLMJob)`-- `pipelex/reporting/reporting_manager.py:152` — `ReportingManager._report_img_gen_job` — `def _report_img_gen_job(self, img_gen_job: ImgGenJob)`-- `pipelex/reporting/reporting_manager.py:161` — `ReportingManager._report_extract_job` — `def _report_extract_job(self, extract_job: ExtractJob)`-- `pipelex/reporting/reporting_manager.py:170` — `ReportingManager._report_search_job` — `def _report_search_job(self, search_job: SearchJob)`-- `pipelex/reporting/reporting_protocol.py:10` — `ReportingProtocol.report_inference_job` — `def report_inference_job(self, inference_job: InferenceJobAbstract)`-- `pipelex/reporting/reporting_protocol.py:25` — `ReportingProtocol.clear_event_log` — `def clear_event_log(self, context_key: str) -> None`--### `runtime_bridge` (10)--- `pipelex/runtime_bridge/bootstrap.py:24` — `ensure_pipelex_booted` — `def ensure_pipelex_booted(config_overrides: dict[str, Any] | None=None) -> None`-- `pipelex/runtime_bridge/bridge.py:225` — `serialize_pipe_output` — `def serialize_pipe_output(pipe_output: PipeOutput) -> dict[str, Any]`-- `pipelex/runtime_bridge/bridge.py:250` — `_decode_library_crate` — `def _decode_library_crate(library_crate_dump: dict[str, Any] | None) -> LibraryCrate | None`-- `pipelex/runtime_bridge/bridge.py:260` — `_decode_delivery_assignment` — `def _decode_delivery_assignment(delivery_assignment_dump: dict[str, Any] | None) -> DeliveryAssignment | None`-- `pipelex/runtime_bridge/bridge.py:383` — `_resolve_main_stuff_root_key` — `def _resolve_main_stuff_root_key(pipe_output: PipeOutput) -> str | None`-- `pipelex/runtime_bridge/primitives/hydration.py:39` — `_hydrate_list_item` — `def _hydrate_list_item(raw_item: dict[str, Any] | str | StuffContent) -> StuffContent`-- `pipelex/runtime_bridge/primitives/hydration.py:109` — `hydrate_working_memory` — `def hydrate_working_memory(working_memory_raw: dict[str, Any]) -> WorkingMemory`-- `pipelex/runtime_bridge/primitives/pipe_classification.py:17` — `is_controller_pipe` — `def is_controller_pipe(pipe: PipeAbstract) -> bool`-- `pipelex/runtime_bridge/primitives/pipe_classification.py:29` — `is_leaf_pipe` — `def is_leaf_pipe(pipe: PipeAbstract) -> bool`-- `pipelex/runtime_bridge/primitives/trace_flush.py:14` — `flush_trace_events_to_backend` — `async def flush_trace_events_to_backend(events: list[TraceEvent]) -> None`--### `system` (61)--- `pipelex/system/configuration/config_check.py:12` — `check_is_initialized` — `def check_is_initialized(print_warning_if_not: bool=True) -> bool`-- `pipelex/system/configuration/config_loader.py:32` — `ConfigLoader.find_project_root` — `def find_project_root(start_dir: Path) -> Path | None`-- `pipelex/system/configuration/configs.py:83` — `PromptingConfig.get_prompting_style` — `def get_prompting_style(self, prompting_target: PromptingTarget | None=None) -> TemplatingStyle | None`-- `pipelex/system/environment.py:21` — `get_required_env` — `def get_required_env(key: str) -> str`-- `pipelex/system/environment.py:29` — `get_optional_env` — `def get_optional_env(key: str) -> str | None`-- `pipelex/system/environment.py:33` — `is_env_var_set` — `def is_env_var_set(key: str) -> bool`-- `pipelex/system/environment.py:37` — `all_env_vars_are_set` — `def all_env_vars_are_set(keys: list[str]) -> bool`-- `pipelex/system/environment.py:41` — `any_env_var_is_placeholder` — `def any_env_var_is_placeholder(keys: list[str]) -> bool`-- `pipelex/system/environment.py:53` — `is_env_var_truthy` — `def is_env_var_truthy(key: str) -> bool`-- `pipelex/system/pipelex_service/pipelex_details.py:22` — `PipelexDetails.make_distinct_id` — `def make_distinct_id(cls, gateway_api_key: str) -> str`-- `pipelex/system/pipelex_service/pipelex_service_config.py:24` — `load_pipelex_service_config_if_exists` — `def load_pipelex_service_config_if_exists(config_dir: Path) -> PipelexServiceConfig | None`-- `pipelex/system/pipelex_service/pipelex_service_config.py:45` — `is_pipelex_gateway_enabled` — `def is_pipelex_gateway_enabled(backends_file_path: Path | None=None) -> bool`-- `pipelex/system/pipelex_service/remote_config_cache.py:99` — `RemoteConfigCache.store` — `def store(cls, remote_config_payload: dict[str, Any]) -> None`-- `pipelex/system/pipelex_service/remote_config_fetcher.py:79` — `RemoteConfigFetcher._log_retry_attempt` — `def _log_retry_attempt(cls, retry_state: RetryCallState) -> None`-- `pipelex/system/pipelex_service/remote_config_fetcher.py:85` — `RemoteConfigFetcher._fetch_remote_config_with_retry` — `def _fetch_remote_config_with_retry(cls, url: str) -> httpx.Response`-- `pipelex/system/pipelex_service/remote_config_fetcher.py:107` — `RemoteConfigFetcher._fetch_remote_config_with_retry._fetch_with_retry` — `def _fetch_with_retry(url: str) -> httpx.Response`-- `pipelex/system/pipelex_service/remote_config_fetcher.py:133` — `RemoteConfigFetcher._fetch_fresh` — `def _fetch_fresh(cls, url: str) -> tuple[dict[str, Any], RemoteConfig]`-- `pipelex/system/pipelex_service/remote_config_fetcher.py:201` — `RemoteConfigFetcher.fetch_remote_config` — `def fetch_remote_config(cls, require_fresh: bool=False) -> RemoteConfigResult`-- `pipelex/system/registries/class_registry_utils.py:135` — `ClassRegistryUtils.auto_register_all_subclasses` — `def auto_register_all_subclasses(cls, base_class: type[Any]) -> int`-- `pipelex/system/registries/func_registry.py:22` — `pipe_func` — `def pipe_func(name: str | None=None) -> Callable[[T], T]`-- `pipelex/system/registries/func_registry.py:48` — `pipe_func.decorator` — `def decorator(func: T) -> T`-- `pipelex/system/registries/func_registry.py:88` — `FuncRegistry.log` — `def log(self, message: str) -> None`-- `pipelex/system/registries/func_registry.py:91` — `FuncRegistry.set_logger` — `def set_logger(self, logger: logging.Logger) -> None`-- `pipelex/system/registries/func_registry.py:116` — `FuncRegistry.unregister_function` — `def unregister_function(self, func: Callable[..., Any]) -> None`-- `pipelex/system/registries/func_registry.py:125` — `FuncRegistry.unregister_function_by_name` — `def unregister_function_by_name(self, name: str) -> None`-- `pipelex/system/registries/func_registry.py:132` — `FuncRegistry.register_functions_dict` — `def register_functions_dict(self, functions: dict[str, Callable[..., Any]]) -> None`-- `pipelex/system/registries/func_registry.py:137` — `FuncRegistry.register_functions` — `def register_functions(self, functions: list[Callable[..., Any]]) -> None`-- `pipelex/system/registries/func_registry.py:142` — `FuncRegistry.get_function` — `def get_function(self, name: str) -> Callable[..., Any] | None`-- `pipelex/system/registries/func_registry.py:146` — `FuncRegistry.get_required_function` — `def get_required_function(self, name: str) -> Callable[..., Any]`-- `pipelex/system/registries/func_registry.py:167` — `FuncRegistry.get_required_function_with_signature` — `def get_required_function_with_signature(self, name: str) -> Callable[..., object]`-- `pipelex/system/registries/func_registry.py:183` — `FuncRegistry.has_function` — `def has_function(self, name: str) -> bool`-- `pipelex/system/registries/func_registry.py:187` — `FuncRegistry.is_marked_pipe_func` — `def is_marked_pipe_func(self, func: Any) -> bool`-- `pipelex/system/registries/func_registry.py:273` — `FuncRegistry.check_function_eligibility` — `def check_function_eligibility(self, func: Any) -> str | None`-- `pipelex/system/registries/func_registry.py:368` — `FuncRegistry.get_ineligible_function_info` — `def get_ineligible_function_info(self, name: str) -> IneligibleFunctionInfo | None`-- `pipelex/system/registries/func_registry_utils.py:113` — `FuncRegistryUtils._register_funcs_in_file` — `def _register_funcs_in_file(cls, file_path: Path) -> None`-- `pipelex/system/registries/func_registry_utils.py:176` — `FuncRegistryUtils._find_functions_in_module` — `def _find_functions_in_module(cls, module: Any) -> list[Callable[..., Any]]`-- `pipelex/system/registries/func_registry_utils.py:211` — `FuncRegistryUtils._get_function_registration_name` — `def _get_function_registration_name(cls, func: Callable[..., Any]) -> str`-- `pipelex/system/registries/singleton.py:21` — `MetaSingleton.clear_subclass_instances` — `def clear_subclass_instances(cls, base_cls: type[Any]) -> None`-- `pipelex/system/registries/singleton.py:28` — `MetaSingleton.get_subclass_instance` — `def get_subclass_instance(cls, base_cls: type[T]) -> T | None`-- `pipelex/system/runtime.py:100` — `RuntimeManager.set_run_mode` — `def set_run_mode(self, run_mode: RunMode)`-- `pipelex/system/runtime.py:103` — `RuntimeManager.set_worker_mode` — `def set_worker_mode(self, worker_mode: WorkerMode)`-- `pipelex/system/telemetry/exception_capture.py:56` — `DualClientExceptionCapture._thread_exception_handler` — `def _thread_exception_handler(self, args: threading.ExceptHookArgs) -> None`-- `pipelex/system/telemetry/exception_capture.py:62` — `DualClientExceptionCapture._capture_exception` — `def _capture_exception(self, exc_info: tuple[type[BaseException], BaseException | None, TracebackType | None]) -> None`-- `pipelex/system/telemetry/otel_constants.py:93` — `make_otel_gen_ai_output_type` — `def make_otel_gen_ai_output_type(output_type: str) -> otel_gen_ai_attributes.GenAiOutputTypeValues`-- `pipelex/system/telemetry/otel_factory.py:48` — `OtelFactory.stringify_json` — `def stringify_json(cls, json_conent: JsonContent) -> str`-- `pipelex/system/telemetry/otel_factory.py:118` — `OtelFactory.make_trace_id` — `def make_trace_id(cls, pipeline_run_id: str) -> int`-- `pipelex/system/telemetry/otel_factory.py:151` — `OtelFactory._is_unresolved_placeholder` — `def _is_unresolved_placeholder(cls, value: str | None) -> bool`-- `pipelex/system/telemetry/otel_factory.py:156` — `OtelFactory.make_langfuse_exporter` — `def make_langfuse_exporter(cls, langfuse_config: LangfuseConfig) -> 'OTLPSpanExporter'`-- `pipelex/system/telemetry/posthog_span_exporter.py:64` — `PostHogSpanExporter._truncate_content` — `def _truncate_content(self, content: str) -> str`-- `pipelex/system/telemetry/posthog_span_exporter.py:80` — `PostHogSpanExporter._apply_content_redaction` — `def _apply_content_redaction(self, properties: dict[str, Any]) -> None`-- `pipelex/system/telemetry/posthog_span_exporter.py:172` — `PostHogSpanExporter._get_redacted_pipe_code` — `def _get_redacted_pipe_code(self, pipe_code: str | None) -> str | None`-- `pipelex/system/telemetry/posthog_span_exporter.py:185` — `PostHogSpanExporter._get_redacted_output_class_name` — `def _get_redacted_output_class_name(self, output_class_name: str | None) -> str | None`-- `pipelex/system/telemetry/posthog_span_exporter.py:337` — `PostHogSpanExporter._do_export` — `def _do_export(self, spans: Sequence[ReadableSpan]) -> SpanExportResult`-- `pipelex/system/telemetry/telemetry_config.py:182` — `TelemetryConfig.is_custom_telemetry_allowed_for_mode` — `def is_custom_telemetry_allowed_for_mode(self, mode: str) -> bool`-- `pipelex/system/telemetry/telemetry_config.py:214` — `TelemetryRedactionConfig.make_from_posthog_config` — `def make_from_posthog_config(cls, posthog_config: PostHogConfig | None) -> Self`-- `pipelex/system/telemetry/telemetry_config.py:239` — `load_telemetry_config` — `def load_telemetry_config(secrets_provider: SecretsProviderAbstract) -> TelemetryConfig`-- `pipelex/system/telemetry/telemetry_manager.py:155` — `TelemetryManager._wrap_capture_exception` — `def _wrap_capture_exception(self, client: Posthog) -> None`-- `pipelex/system/telemetry/telemetry_manager.py:163` — `TelemetryManager._wrap_capture_exception.sanitized_capture_exception` — `def sanitized_capture_exception(exception: ExceptionArg | None=None, **kwargs: Unpack[OptionalCaptureArgs]) -> Any`-- `pipelex/system/telemetry/telemetry_manager_abstract.py:89` — `TelemetryManagerAbstract.setup` — `def setup(self, integration_mode: IntegrationMode)`-- `pipelex/system/telemetry/telemetry_manager_abstract.py:106` — `TelemetryManagerAbstract.is_custom_portkey_logging_enabled` — `def is_custom_portkey_logging_enabled(self, is_debug_configured: bool) -> bool`-- `pipelex/system/telemetry/telemetry_manager_abstract.py:114` — `TelemetryManagerAbstract.is_pipelex_gateway_portkey_logging_enabled` — `def is_pipelex_gateway_portkey_logging_enabled(self, is_debug_configured: bool) -> bool`--### `temporal` (48)--- `pipelex/temporal/codec/codec_server.py:39` — `_make_cors_handler` — `def _make_cors_handler(cors_origins: list[str]) -> Callable[[web.Request], Awaitable[web.Response]]`-- `pipelex/temporal/codec/codec_server.py:42` — `_make_cors_handler.cors_options` — `async def cors_options(request: web.Request) -> web.Response`-- `pipelex/temporal/codec/storage_payload_codec.py:49` — `StoragePayloadCodec._extract_job_routing` — `def _extract_job_routing(payload: Payload) -> tuple[str, str] | None`-- `pipelex/temporal/codec/storage_payload_codec.py:75` — `StoragePayloadCodec._sanitize_path_segment` — `def _sanitize_path_segment(segment: str) -> str`-- `pipelex/temporal/config_temporal.py:192` — `RetryPolicyConfigBase.make_retry_policy` — `def make_retry_policy(self, merged_non_retryable_types: list[str]) -> 'RetryPolicy'`-- `pipelex/temporal/config_temporal.py:445` — `WorkerConfig.all_non_retryable_error_types` — `def all_non_retryable_error_types(self, queue_options_by_queue: dict[str, 'QueueOptions']) -> set[str]`-- `pipelex/temporal/config_temporal.py:731` — `Temporal.validate_task_queue_known` — `def validate_task_queue_known(self, task_queue: str) -> None`-- `pipelex/temporal/log_temporal.py:64` — `_RequestIdLog.verbose` — `def verbose(self, content: Union[str, Any]) -> None`-- `pipelex/temporal/log_temporal.py:67` — `_RequestIdLog.debug` — `def debug(self, content: Union[str, Any]) -> None`-- `pipelex/temporal/log_temporal.py:70` — `_RequestIdLog.dev` — `def dev(self, content: Union[str, Any]) -> None`-- `pipelex/temporal/log_temporal.py:73` — `_RequestIdLog.info` — `def info(self, content: Union[str, Any]) -> None`-- `pipelex/temporal/log_temporal.py:76` — `_RequestIdLog.warning` — `def warning(self, content: Union[str, Any]) -> None`-- `pipelex/temporal/log_temporal.py:79` — `_RequestIdLog.error` — `def error(self, content: Union[str, Any]) -> None`-- `pipelex/temporal/log_temporal.py:82` — `_RequestIdLog.critical` — `def critical(self, content: Union[str, Any]) -> None`-- `pipelex/temporal/sandbox_manager.py:9` — `SandboxManager.set_sandbox_callable` — `def set_sandbox_callable(self, sandbox_callable: Callable[[], bool])`-- `pipelex/temporal/temporal_connect.py:83` — `connect_to_temporal_selected_server` — `async def connect_to_temporal_selected_server(selected_server_config: str) -> TemporalClient`-- `pipelex/temporal/temporal_data_converter.py:57` — `BaseModelPayloadConverter._is_kajson_wire_value` — `def _is_kajson_wire_value(cls, value: object) -> bool`-- `pipelex/temporal/temporal_data_converter.py:79` — `BaseModelPayloadConverter._first_kajson_list_element` — `def _first_kajson_list_element(cls, value: object) -> object | None`-- `pipelex/temporal/temporal_data_converter.py:101` — `BaseModelPayloadConverter._kajson_deserialize_from_payload` — `def _kajson_deserialize_from_payload(self, payload: Payload) -> Any`-- `pipelex/temporal/temporal_data_converter.py:133` — `BaseModelPayloadConverter._is_kajson_type` — `def _is_kajson_type(cls, type_hint: Any) -> bool`-- `pipelex/temporal/temporal_data_converter.py:145` — `BaseModelPayloadConverter._unwrap_optional_kajson_type` — `def _unwrap_optional_kajson_type(cls, type_hint: type[Any] | None) -> bool`-- `pipelex/temporal/temporal_data_converter.py:200` — `make_data_converter` — `def make_data_converter(payload_codec: PayloadCodec | None=None) -> DataConverter`-- `pipelex/temporal/temporal_hub.py:20` — `TemporalHub.set_task_manager` — `def set_task_manager(self, task_manager: TaskManager)`-- `pipelex/temporal/temporal_manager.py:37` — `TemporalManager.setup` — `def setup(cls, session_id: str) -> None`-- `pipelex/temporal/temporal_manager.py:86` — `TemporalManager.get_temporal_client` — `async def get_temporal_client(self, should_auto_connect: bool) -> TemporalClient`-- `pipelex/temporal/temporal_manager.py:95` — `TemporalManager.make_top_workflow_id` — `def make_top_workflow_id(self, pipeline_run_id: str) -> str`-- `pipelex/temporal/temporal_manager.py:115` — `get_temporal_client` — `async def get_temporal_client(should_auto_connect: bool) -> TemporalClient`-- `pipelex/temporal/temporal_task_manager.py:236` — `TemporalTaskManager._resolve_scope_by_name` — `def _resolve_scope_by_name(scope_name: str | None) -> WorkerScope`-- `pipelex/temporal/temporal_task_manager.py:245` — `TemporalTaskManager._resolve_runtime_profile_by_name` — `def _resolve_runtime_profile_by_name(profile_name: str | None) -> WorkerRuntimeProfile`-- `pipelex/temporal/test_extras/wf_test_structured_output_cross_process.py:65` — `_assert_invoice_round_trip` — `def _assert_invoice_round_trip(invoice: FixtureInvoice) -> None`-- `pipelex/temporal/test_extras/wf_test_structured_output_cross_process.py:101` — `_assert_invoice_list_round_trip` — `def _assert_invoice_list_round_trip(invoices: list[FixtureInvoice]) -> None`-- `pipelex/temporal/test_helpers/temporal_pytest_plugins.py:11` — `pytest_addoption` — `def pytest_addoption(parser: Parser)`-- `pipelex/temporal/tprl/activity_error_boundary.py:25` — `convert_pipelex_errors` — `def convert_pipelex_errors(func: Callable[_ActivityParams, Awaitable[_ActivityReturn]]) -> Callable[_ActivityParams, Awaitable[_ActivityReturn]]`-- `pipelex/temporal/tprl/conditional_worker.py:17` — `with_conditional_worker` — `def with_conditional_worker(execute_workflow: FuncExecuteWorkflow) -> FuncExecuteWorkflow`-- `pipelex/temporal/tprl/observability.py:64` — `stamp_submitter_session_id` — `def stamp_submitter_session_id(pipe_job: PipeJob) -> PipeJob`-- `pipelex/temporal/tprl/observability.py:85` — `build_search_attributes` — `def build_search_attributes(pipe_job: PipeJob) -> TypedSearchAttributes`-- `pipelex/temporal/tprl/observability.py:124` — `build_static_summary` — `def build_static_summary(pipe: PipeAbstract) -> str`-- `pipelex/temporal/tprl/observability.py:137` — `build_static_details` — `def build_static_details(pipe_job: PipeJob) -> str`-- `pipelex/temporal/tprl/temporal_error.py:25` — `error_report_dict_from_details` — `def error_report_dict_from_details(details: Sequence[Any]) -> dict[str, Any] | None`-- `pipelex/temporal/tprl/temporal_error.py:41` — `_find_error_report_dict` — `def _find_error_report_dict(exc: BaseException) -> dict[str, Any] | None`-- `pipelex/temporal/tprl/temporal_error.py:64` — `_message_from_exc` — `def _message_from_exc(exc: BaseException) -> str`-- `pipelex/temporal/tprl/temporal_error.py:85` — `recover_error_report` — `def recover_error_report(exc: BaseException) -> ErrorReport`-- `pipelex/temporal/tprl/temporal_error.py:168` — `TemporalError._log_critical` — `def _log_critical(cls, message: str) -> None`-- `pipelex/temporal/tprl/temporal_error.py:181` — `TemporalError._log_error` — `def _log_error(cls, message: str) -> None`-- `pipelex/temporal/tprl/temporal_error.py:192` — `TemporalError._error_type_in_name_list` — `def _error_type_in_name_list(cls, error_type: str | None) -> bool`-- `pipelex/temporal/tprl/temporal_error.py:205` — `TemporalError.from_app_error` — `def from_app_error(cls, exc: ApplicationError) -> Self`-- `pipelex/temporal/tprl/workflow_caller.py:86` — `WorkflowExecutor.make_workflow_id` — `def make_workflow_id(self, pipeline_run_id: str) -> str`-- `pipelex/temporal/tprl_pipe/wf_pipe_router.py:25` — `_carries_temporal_failure` — `def _carries_temporal_failure(exc: BaseException) -> bool`--### `test_extras` (6)--- `pipelex/test_extras/shared_pytest_plugins.py:69` — `pytest_addoption` — `def pytest_addoption(parser: Parser)`-- `pipelex/test_extras/shared_pytest_plugins.py:93` — `pytest_configure` — `def pytest_configure(config: Config) -> None`-- `pipelex/test_extras/shared_pytest_plugins.py:151` — `_setup_env_var_placeholders` — `def _setup_env_var_placeholders(env_var_keys: list[str]) -> None`-- `pipelex/test_extras/shared_pytest_plugins.py:174` — `_cleanup_placeholder_env_vars` — `def _cleanup_placeholder_env_vars(env_var_keys: list[str]) -> None`-- `pipelex/test_extras/shared_pytest_plugins.py:230` — `needs_inference_in_pipelex` — `def needs_inference_in_pipelex(request: FixtureRequest) -> bool`-- `pipelex/test_extras/shared_pytest_plugins.py:256` — `is_inference_disabled_in_pipelex` — `def is_inference_disabled_in_pipelex(request: FixtureRequest) -> bool`--### `tools` (165)--- `pipelex/tools/aws/aws_config.py:37` — `AwsConfig.get_aws_access_keys_with_method` — `def get_aws_access_keys_with_method(self, api_key_method: AwsKeyMethod) -> tuple[str, str, str]`-- `pipelex/tools/jinja2/image_registry.py:27` — `ImageRegistry.register_image` — `def register_image(self, image: Any) -> int`-- `pipelex/tools/jinja2/image_registry.py:48` — `ImageRegistry.get_image_placeholder` — `def get_image_placeholder(self, image: Any) -> str | None`-- `pipelex/tools/jinja2/image_registry.py:74` — `ImageRegistry.make_finalize.finalize` — `def finalize(value: Any) -> Any`-- `pipelex/tools/jinja2/jinja2_filters.py:139` — `escape_script_tag` — `def escape_script_tag(value: Any) -> Any`-- `pipelex/tools/jinja2/jinja2_required_variables.py:32` — `_build_full_path` — `def _build_full_path(node: nodes.Node) -> str | None`-- `pipelex/tools/jinja2/jinja2_required_variables.py:50` — `_collect_declarations_from_body` — `def _collect_declarations_from_body(body: list[nodes.Node]) -> set[str]`-- `pipelex/tools/jinja2/jinja2_required_variables.py:163` — `_extract_filters_and_variable` — `def _extract_filters_and_variable(node: nodes.Node) -> tuple[list[str], nodes.Node | None]`-- `pipelex/tools/jinja2/jinja2_template_loader.py:117` — `TemplateLoader.load` — `def load(cls, name: str) -> None`-- `pipelex/tools/jinja2/jinja2_template_loader.py:157` — `TemplateLoader.reload` — `def reload(cls, name: str | None=None) -> None`-- `pipelex/tools/jinja2/jinja2_template_loader.py:173` — `TemplateLoader.is_loaded` — `def is_loaded(cls, name: str) -> bool`-- `pipelex/tools/jinja2/jinja2_template_registry.py:47` — `TemplateRegistry.get` — `def get(cls, key: str) -> str`-- `pipelex/tools/jinja2/jinja2_template_registry.py:77` — `TemplateRegistry.is_registered` — `def is_registered(cls, key: str) -> bool`-- `pipelex/tools/jinja2/text_format_renderable.py:32` — `TextFormatRenderable.rendered_for_template_async` — `async def rendered_for_template_async(self, text_format: TextFormat) -> str`-- `pipelex/tools/log/log.py:28` — `Log.set_log_mode` — `def set_log_mode(self, mode: LogMode)`-- `pipelex/tools/log/log.py:73` — `Log.configure_if_unset` — `def configure_if_unset(self, log_config: LogConfig) -> bool`-- `pipelex/tools/log/log.py:92` — `Log.configure` — `def configure(self, log_config: LogConfig)`-- `pipelex/tools/log/log.py:131` — `Log.set_poor_log_formatter` — `def set_poor_log_formatter(self, formatter: logging.Formatter)`-- `pipelex/tools/log/log.py:143` — `Log._should_ignore` — `def _should_ignore(self, problem_id: str | None=None) -> bool`-- `pipelex/tools/log/log.py:170` — `Log.set_level_by_int` — `def set_level_by_int(self, level_int: int)`-- `pipelex/tools/log/log.py:179` — `Log.set_level_by_name` — `def set_level_by_name(self, level_name: str)`-- `pipelex/tools/log/log.py:194` — `Log.set_level` — `def set_level(self, level: LogLevel)`-- `pipelex/tools/log/log.py:214` — `Log.set_levels_for_packages` — `def set_levels_for_packages(self, package_log_levels: dict[str, LogLevel])`-- `pipelex/tools/log/log_config.py:41` — `CallerInfoTemplate.for_template_key` — `def for_template_key(cls, key: CallerInfoTemplate) -> str`-- `pipelex/tools/log/log_config.py:71` — `RichLogConfig.make_rich_handler` — `def make_rich_handler(self, target: ConsoleTarget) -> RichHandler`-- `pipelex/tools/log/log_dispatch.py:23` — `LogDispatch.set_log_mode` — `def set_log_mode(self, mode: LogMode)`-- `pipelex/tools/log/log_dispatch.py:46` — `LogDispatch.configure` — `def configure(self, log_config: LogConfig)`-- `pipelex/tools/log/log_formatter.py:10` — `emoji_for_channel` — `def emoji_for_channel(channel_name: str) -> str | None`-- `pipelex/tools/log/log_levels.py:47` — `LogLevel.from_int` — `def from_int(logging_level: int) -> 'LogLevel'`-- `pipelex/tools/mermaid/mermaid_utils.py:32` — `encode_pako_from_bytes` — `def encode_pako_from_bytes(state_bytes: bytes) -> str`-- `pipelex/tools/mermaid/mermaid_utils.py:39` — `encode_pako_from_string` — `def encode_pako_from_string(state: str) -> str`-- `pipelex/tools/mermaid/mermaid_utils.py:45` — `make_mermaid_url` — `def make_mermaid_url(mermaid_code: str) -> str`-- `pipelex/tools/mermaid/mermaid_utils.py:80` — `clean_str_for_mermaid_node_title` — `def clean_str_for_mermaid_node_title(text: str) -> str`-- `pipelex/tools/mermaid/mermaid_utils.py:97` — `sanitize_mermaid_id` — `def sanitize_mermaid_id(node_id: str) -> str`-- `pipelex/tools/mermaid/mermaid_utils.py:114` — `escape_mermaid_label` — `def escape_mermaid_label(label: str) -> str`-- `pipelex/tools/misc/async_utils.py:8` — `_invoke` — `async def _invoke(factory: Callable[[], Awaitable[_T]]) -> _T`-- `pipelex/tools/misc/attribute_utils.py:26` — `AttributePolisher.should_truncate` — `def should_truncate(cls, value: Any) -> bool`-- `pipelex/tools/misc/attribute_utils.py:39` — `AttributePolisher.get_truncated_value` — `def get_truncated_value(cls, value: Any) -> Any`-- `pipelex/tools/misc/attribute_utils.py:53` — `AttributePolisher._looks_like_base64` — `def _looks_like_base64(cls, value: str) -> bool`-- `pipelex/tools/misc/base64_utils.py:9` — `load_binary_as_base64` — `async def load_binary_as_base64(path: Path) -> str`-- `pipelex/tools/misc/base64_utils.py:15` — `make_base64_url_from_path` — `async def make_base64_url_from_path(path: Path) -> str`-- `pipelex/tools/misc/base64_utils.py:23` — `make_base64_url_from_http_url` — `async def make_base64_url_from_http_url(url: str) -> str`-- `pipelex/tools/misc/base64_utils.py:31` — `make_base64_url_from_bytes` — `def make_base64_url_from_bytes(raw_bytes: bytes) -> str`-- `pipelex/tools/misc/base64_utils.py:47` — `is_prefixed_base64_url` — `def is_prefixed_base64_url(possibly_base64_url: str) -> bool`-- `pipelex/tools/misc/base64_utils.py:51` — `strip_base64_str_if_needed` — `def strip_base64_str_if_needed(base64_str: str) -> str`-- `pipelex/tools/misc/base64_utils.py:59` — `extract_base64_str_from_base64_url_if_possible` — `def extract_base64_str_from_base64_url_if_possible(possibly_base64_url: str) -> tuple[str, str] | None`-- `pipelex/tools/misc/dict_utils.py:173` — `extract_vars_from_strings_recursive` — `def extract_vars_from_strings_recursive(data: Any) -> set[str]`-- `pipelex/tools/misc/dict_utils.py:194` — `extract_vars_from_strings_recursive.extract_from_string` — `def extract_from_string(text: str) -> None`-- `pipelex/tools/misc/dict_utils.py:221` — `extract_vars_from_strings_recursive.traverse` — `def traverse(value: Any) -> None`-- `pipelex/tools/misc/diff.py:44` — `has_diff_dirs._filter_excluded_files` — `def _filter_excluded_files(file_list: list[str]) -> list[str]`-- `pipelex/tools/misc/diff.py:47` — `has_diff_dirs._has_diff` — `def _has_diff(dir_comparison: filecmp.dircmp[str]) -> bool`-- `pipelex/tools/misc/diff.py:238` — `make_diff_dirs_pretty._filter_excluded_files` — `def _filter_excluded_files(file_list: list[str]) -> list[str]`-- `pipelex/tools/misc/document_utils.py:59` — `DocumentFormat.is_supported_mime_type` — `def is_supported_mime_type(cls, mime_type: str) -> bool`-- `pipelex/tools/misc/document_utils.py:64` — `DocumentFormat.raise_if_unsupported_mime_type` — `def raise_if_unsupported_mime_type(cls, mime_type: str) -> None`-- `pipelex/tools/misc/document_utils.py:79` — `DocumentFormat.from_mime_type` — `def from_mime_type(cls, mime_type: str) -> 'DocumentFormat'`-- `pipelex/tools/misc/exceptions.py:36` — `TomlError.from_tomli_error` — `def from_tomli_error(cls, exc: tomli.TOMLDecodeError) -> Self`-- `pipelex/tools/misc/file_utils.py:82` — `load_text_from_path` — `def load_text_from_path(path: Path) -> str`-- `pipelex/tools/misc/file_utils.py:101` — `failable_load_text_from_path` — `def failable_load_text_from_path(path: Path) -> str | None`-- `pipelex/tools/misc/file_utils.py:119` — `load_binary` — `def load_binary(path: Path) -> bytes`-- `pipelex/tools/misc/file_utils.py:123` — `load_binary_async` — `async def load_binary_async(path: Path) -> bytes`-- `pipelex/tools/misc/file_utils.py:217` — `remove_file` — `def remove_file(file_path: Path)`-- `pipelex/tools/misc/file_utils.py:234` — `remove_folder` — `def remove_folder(folder_path: Path) -> None`-- `pipelex/tools/misc/file_utils.py:273` — `_reraise_walk_error` — `def _reraise_walk_error(walk_error: OSError) -> None`-- `pipelex/tools/misc/file_utils.py:406` — `ensure_directory_exists` — `def ensure_directory_exists(directory_path: Path) -> None`-- `pipelex/tools/misc/file_utils.py:416` — `ensure_path` — `def ensure_path(path: Path) -> bool`-- `pipelex/tools/misc/file_utils.py:435` — `ensure_directory_for_file_path` — `def ensure_directory_for_file_path(file_path: Path) -> None`-- `pipelex/tools/misc/file_utils.py:444` — `path_exists` — `def path_exists(path_str: str | Path) -> bool`-- `pipelex/tools/misc/filetype_utils.py:38` — `detect_file_type_from_path` — `def detect_file_type_from_path(path: Path) -> FileType`-- `pipelex/tools/misc/filetype_utils.py:60` — `detect_file_type_from_bytes` — `def detect_file_type_from_bytes(raw_bytes: bytes) -> FileType`-- `pipelex/tools/misc/filetype_utils.py:82` — `detect_file_type_from_base64` — `def detect_file_type_from_base64(base64_data: str | bytes) -> FileType`-- `pipelex/tools/misc/filetype_utils.py:115` — `mime_type_to_extension` — `def mime_type_to_extension(mime_type: str) -> str`-- `pipelex/tools/misc/hash_utils.py:24` — `hash_md5_to_int` — `def hash_md5_to_int(string: str) -> int`-- `pipelex/tools/misc/http_utils.py:24` — `validate_url_resource_exists` — `def validate_url_resource_exists(url: str) -> None`-- `pipelex/tools/misc/http_utils.py:50` — `_validate_http_url` — `def _validate_http_url(url: str) -> None`-- `pipelex/tools/misc/http_utils.py:80` — `_validate_local_path` — `def _validate_local_path(url: str) -> None`-- `pipelex/tools/misc/image_utils.py:58` — `ImageFormat.from_mime_type` — `def from_mime_type(cls, mime_type: str) -> 'ImageFormat'`-- `pipelex/tools/misc/image_utils.py:72` — `ImageFormat.is_supported_mime_type` — `def is_supported_mime_type(cls, mime_type: str) -> bool`-- `pipelex/tools/misc/image_utils.py:77` — `ImageFormat.raise_if_unsupported_mime_type` — `def raise_if_unsupported_mime_type(cls, mime_type: str) -> None`-- `pipelex/tools/misc/json_utils.py:23` — `clean_json_content` — `def clean_json_content(content: Any) -> Any`-- `pipelex/tools/misc/json_utils.py:156` — `load_json_from_path` — `def load_json_from_path(path: Path) -> JsonContent`-- `pipelex/tools/misc/json_utils.py:177` — `load_json_dict_from_path` — `def load_json_dict_from_path(path: Path) -> dict[str, Any]`-- `pipelex/tools/misc/json_utils.py:202` — `load_json_list_from_path` — `def load_json_list_from_path(path: Path) -> list[Any]`-- `pipelex/tools/misc/json_utils.py:262` — `remove_none_values` — `def remove_none_values(json_content: JsonContent | Any) -> JsonContent | Any`-- `pipelex/tools/misc/json_utils.py:307` — `remove_none_values_from_dict` — `def remove_none_values_from_dict(data: Mapping[str, Any]) -> dict[str, Any]`-- `pipelex/tools/misc/placeholder.py:4` — `make_placeholder_value` — `def make_placeholder_value(key: str) -> str`-- `pipelex/tools/misc/placeholder.py:17` — `value_is_placeholder` — `def value_is_placeholder(value: str | None) -> bool`-- `pipelex/tools/misc/semver.py:18` — `parse_version` — `def parse_version(version_str: str) -> Version`-- `pipelex/tools/misc/semver.py:40` — `parse_constraint` — `def parse_constraint(constraint_str: str) -> SimpleSpec`-- `pipelex/tools/misc/semver.py:119` — `parse_version_tag` — `def parse_version_tag(tag: str) -> Version | None`-- `pipelex/tools/misc/string_utils.py:6` — `has_text` — `def has_text(text: str) -> bool`-- `pipelex/tools/misc/string_utils.py:22` — `is_none_or_has_text` — `def is_none_or_has_text(text: str | None) -> bool`-- `pipelex/tools/misc/string_utils.py:39` — `can_inject_text` — `def can_inject_text(value: Any | None) -> bool`-- `pipelex/tools/misc/string_utils.py:63` — `is_not_none_and_has_text` — `def is_not_none_and_has_text(text: str | None) -> bool`-- `pipelex/tools/misc/string_utils.py:80` — `camel_to_snake_case` — `def camel_to_snake_case(name: str) -> str`-- `pipelex/tools/misc/string_utils.py:101` — `pascal_case_to_snake_case` — `def pascal_case_to_snake_case(name: str) -> str`-- `pipelex/tools/misc/string_utils.py:121` — `pascal_case_to_kebab` — `def pascal_case_to_kebab(name: str) -> str`-- `pipelex/tools/misc/string_utils.py:158` — `pascal_case_to_sentence` — `def pascal_case_to_sentence(name: str) -> str`-- `pipelex/tools/misc/string_utils.py:199` — `snake_to_pascal_case` — `def snake_to_pascal_case(snake_str: str) -> str`-- `pipelex/tools/misc/string_utils.py:220` — `snake_to_capitalize_first_letter` — `def snake_to_capitalize_first_letter(snake_str: str) -> str`-- `pipelex/tools/misc/string_utils.py:242` — `snake_to_title_case` — `def snake_to_title_case(snake_str: str) -> str`-- `pipelex/tools/misc/string_utils.py:262` — `is_snake_case` — `def is_snake_case(word: str) -> bool`-- `pipelex/tools/misc/string_utils.py:266` — `is_pascal_case` — `def is_pascal_case(word: str) -> bool`-- `pipelex/tools/misc/string_utils.py:270` — `normalize_to_ascii` — `def normalize_to_ascii(text: str) -> str`-- `pipelex/tools/misc/string_utils.py:309` — `get_root_from_dotted_path` — `def get_root_from_dotted_path(dotted_path: str) -> str`-- `pipelex/tools/misc/tenacity_utils.py:6` — `log_retry` — `def log_retry(retry_state: RetryCallState) -> None`-- `pipelex/tools/misc/toml_utils.py:17` — `load_toml_from_content` — `def load_toml_from_content(content: str) -> dict[str, Any]`-- `pipelex/tools/misc/toml_utils.py:25` — `load_toml_from_path` — `def load_toml_from_path(path: str | Path) -> dict[str, Any]`-- `pipelex/tools/misc/toml_utils.py:46` — `load_toml_from_path_if_exists` — `def load_toml_from_path_if_exists(path: str | Path) -> dict[str, Any] | None`-- `pipelex/tools/misc/toml_utils.py:53` — `load_toml_with_tomlkit` — `def load_toml_with_tomlkit(path: str | Path) -> tomlkit.TOMLDocument`-- `pipelex/tools/misc/toml_utils.py:79` — `load_toml_from_path_and_merge_with_overrides` — `def load_toml_from_path_and_merge_with_overrides(paths: Sequence[str | Path]) -> dict[str, Any]`-- `pipelex/tools/network/host_rules.py:19` — `is_disallowed_ip` — `def is_disallowed_ip(host: str) -> bool`-- `pipelex/tools/network/host_rules.py:43` — `is_disallowed_host` — `def is_disallowed_host(host: str) -> bool`-- `pipelex/tools/pdf/pypdfium2_renderer.py:41` — `_resolve_pdf_uri_to_input` — `async def _resolve_pdf_uri_to_input(pdf_uri: str) -> PdfInput`-- `pipelex/tools/pdf/pypdfium2_renderer.py:178` — `PyPdfium2Renderer._extract_text_from_pdf_pages_sync` — `def _extract_text_from_pdf_pages_sync(pdf_input: PdfInput) -> list[str]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:273` — `PyPdfium2Renderer.extract_text_from_pdf_pages` — `async def extract_text_from_pdf_pages(self, pdf_input: PdfInput) -> list[str]`-- `pipelex/tools/pdf/pypdfium2_renderer.py:282` — `PyPdfium2Renderer.extract_text_from_pdf_pages_from_uri` — `async def extract_text_from_pdf_pages_from_uri(self, pdf_uri: str) -> list[str]`-- `pipelex/tools/secrets/secrets_provider_abstract.py:8` — `SecretsProviderAbstract.get_required_secret` — `def get_required_secret(self, secret_id: str) -> str`-- `pipelex/tools/secrets/secrets_provider_abstract.py:11` — `SecretsProviderAbstract.get_optional_secret` — `def get_optional_secret(self, secret_id: str) -> str | None`-- `pipelex/tools/secrets/secrets_provider_abstract.py:22` — `SecretsProviderAbstract.get_secret` — `def get_secret(self, secret_id: str) -> str`-- `pipelex/tools/secrets/secrets_utils.py:51` — `substitute_vars.replace_var` — `def replace_var(match: re.Match[str]) -> str`-- `pipelex/tools/secrets/secrets_utils.py:130` — `_get_env_var` — `def _get_env_var(var_name: str) -> str`-- `pipelex/tools/storage/gcp_storage_provider.py:84` — `GcpStorageProvider._load_with_metadata_sync` — `def _load_with_metadata_sync(self, key: str) -> StoredData`-- `pipelex/tools/storage/gcp_storage_provider.py:170` — `GcpStorageProvider._make_public_url` — `def _make_public_url(self, key: str) -> str`-- `pipelex/tools/storage/gcp_storage_provider.py:181` — `GcpStorageProvider._generate_signed_url_sync` — `def _generate_signed_url_sync(self, key: str) -> str | None`-- `pipelex/tools/storage/local_storage_provider.py:26` — `LocalStorageProvider._validate_key` — `def _validate_key(self, key: str) -> Path`-- `pipelex/tools/storage/s3_storage_provider.py:176` — `S3StorageProvider._make_public_url` — `def _make_public_url(self, key: str) -> str`-- `pipelex/tools/storage/storage_provider_abstract.py:28` — `StorageProviderAbstract._strip_scheme` — `def _strip_scheme(self, uri: str) -> str`-- `pipelex/tools/storage/storage_provider_abstract.py:45` — `StorageProviderAbstract._add_scheme` — `def _add_scheme(self, key: str) -> str`-- `pipelex/tools/storage/storage_provider_abstract.py:62` — `StorageProviderAbstract.load` — `async def load(self, uri: str) -> bytes`-- `pipelex/tools/storage/storage_provider_abstract.py:74` — `StorageProviderAbstract.load_with_metadata` — `async def load_with_metadata(self, uri: str) -> StoredData`-- `pipelex/tools/storage/storage_provider_abstract.py:87` — `StorageProviderAbstract._load_with_metadata` — `async def _load_with_metadata(self, key: str) -> StoredData`-- `pipelex/tools/storage/storage_provider_abstract.py:123` — `StorageProviderAbstract.public_url` — `async def public_url(self, uri: str) -> str | None`-- `pipelex/tools/storage/storage_provider_factory.py:14` — `make_storage_provider_from_config` — `def make_storage_provider_from_config(storage_provider_config: StorageProviderConfig) -> StorageProviderAbstract`-- `pipelex/tools/tabular/csv_codec.py:60` — `_is_flat_annotation` — `def _is_flat_annotation(annotation: Any) -> bool`-- `pipelex/tools/tabular/csv_codec.py:86` — `_annotation_allows_none` — `def _annotation_allows_none(annotation: Any) -> bool`-- `pipelex/tools/tabular/csv_codec.py:93` — `flat_field_names` — `def flat_field_names(row_model: type[StuffContent]) -> list[str]`-- `pipelex/tools/tabular/csv_codec.py:120` — `_assert_single_char_delimiter` — `def _assert_single_char_delimiter(delimiter: str) -> None`-- `pipelex/tools/tabular/csv_codec.py:275` — `is_tabular_path` — `def is_tabular_path(path: Path) -> bool`-- `pipelex/tools/tabular/csv_codec.py:287` — `assert_supported_table_suffix` — `def assert_supported_table_suffix(path: Path) -> None`-- `pipelex/tools/tabular/csv_codec.py:407` — `_validation_error_label` — `def _validation_error_label(error: 'ErrorDetails') -> str`-- `pipelex/tools/tabular/csv_codec.py:419` — `_to_cell` — `def _to_cell(value: Any) -> str`-- `pipelex/tools/typing/class_utils.py:14` — `normalize_property_for_comparison` — `def normalize_property_for_comparison(prop: dict[str, Any]) -> dict[str, Any]`-- `pipelex/tools/typing/class_utils.py:46` — `normalize_properties_for_comparison` — `def normalize_properties_for_comparison(properties: dict[str, Any]) -> dict[str, Any]`-- `pipelex/tools/typing/class_utils.py:129` — `has_compatible_field._is_compatible` — `def _is_compatible(type_param: Any) -> bool`-- `pipelex/tools/typing/module_inspector.py:11` — `import_module_from_file` — `def import_module_from_file(file_path: Path) -> Any`-- `pipelex/tools/typing/module_inspector.py:58` — `convert_file_path_to_module_path` — `def convert_file_path_to_module_path(file_path: Path) -> str`-- `pipelex/tools/typing/pydantic_utils.py:24` — `empty_list_factory_of` — `def empty_list_factory_of(_: type[T]) -> Callable[[], list[T]]`-- `pipelex/tools/typing/pydantic_utils.py:62` — `analyze_pydantic_validation_error` — `def analyze_pydantic_validation_error(exc: ValidationError) -> PydanticValidationErrorAnalysis`-- `pipelex/tools/typing/pydantic_utils.py:145` — `format_pydantic_validation_error` — `def format_pydantic_validation_error(exc: ValidationError) -> str`-- `pipelex/tools/typing/pydantic_utils.py:158` — `_serialize_input_value` — `def _serialize_input_value(value: Any) -> Any`-- `pipelex/tools/typing/pydantic_utils.py:180` — `_serialize_context` — `def _serialize_context(ctx: dict[str, Any]) -> dict[str, Any]`-- `pipelex/tools/typing/pydantic_utils.py:196` — `format_pydantic_validation_error_for_agent` — `def format_pydantic_validation_error_for_agent(exc: ValidationError) -> tuple[str, dict[str, Any]]`-- `pipelex/tools/typing/pydantic_utils.py:277` — `convert_strenum_to_str` — `def convert_strenum_to_str(obj: dict[str, Any] | list[Any] | StrEnum | Any) -> dict[str, Any] | list[Any] | str | Any`-- `pipelex/tools/typing/pydantic_utils.py:305` — `clean_model_to_dict` — `def clean_model_to_dict(obj: BaseModel) -> dict[str, Any]`-- `pipelex/tools/typing/pydantic_utils.py:409` — `_truncated_rich_repr_from_items` — `def _truncated_rich_repr_from_items(rich_repr_items: RichReprResult) -> RichReprResult`-- `pipelex/tools/typing/pydantic_utils.py:431` — `make_truncated_wrapper` — `def make_truncated_wrapper(model: BaseModel) -> Any`-- `pipelex/tools/typing/pydantic_utils.py:440` — `make_truncated_wrapper.rich_repr_method` — `def rich_repr_method(_self: Any) -> RichReprResult`-- `pipelex/tools/typing/structure_printer.py:32` — `StructurePrinter.pretty_type` — `def pretty_type(self, tp: object) -> str`-- `pipelex/tools/typing/structure_printer.py:95` — `StructurePrinter.get_type_structure.format_type` — `def format_type(tp: Any) -> str`-- `pipelex/tools/typing/structure_printer.py:137` — `StructurePrinter.get_type_structure.collect_types` — `def collect_types(tp: type[Any]) -> None`-- `pipelex/tools/uri/uri_resolver.py:23` — `extract_filename_from_uri` — `def extract_filename_from_uri(uri: str) -> str | None`-- `pipelex/tools/uri/uri_resolver.py:45` — `describe_uri` — `def describe_uri(uri: str) -> str`-- `pipelex/tools/uri/uri_resolver.py:58` — `resolve_uri` — `def resolve_uri(uri: str) -> ResolvedUri`-- `pipelex/tools/uri/uri_resolver.py:118` — `_resolve_file_uri` — `def _resolve_file_uri(uri: str) -> ResolvedLocalPath`-- `pipelex/tools/uri/uri_resolver.py:135` — `_resolve_base64_data_url` — `def _resolve_base64_data_url(uri: str) -> ResolvedBase64DataUrl`--### `tracing` (18)--- `pipelex/tracing/activity_event_log.py:44` — `ActivityEventLogCache.get_or_create` — `def get_or_create(cls, tracing_config: TracingConfig) -> EventLogProtocol | None`-- `pipelex/tracing/dynamodb_event_log.py:95` — `DynamoDBEventLog._make_pk` — `def _make_pk(pipeline_run_id: str) -> str`-- `pipelex/tracing/dynamodb_event_log.py:102` — `DynamoDBEventLog._key_condition` — `def _key_condition(self, pipeline_run_id: str) -> Any`-- `pipelex/tracing/event_log_protocol.py:54` — `EventLogProtocol.emit` — `def emit(self, event: TraceEvent) -> None`-- `pipelex/tracing/event_log_protocol.py:63` — `EventLogProtocol.read_events` — `def read_events(self, pipeline_run_id: str) -> list[TraceEvent]`-- `pipelex/tracing/event_log_protocol.py:79` — `EventLogProtocol.cleanup` — `def cleanup(self, pipeline_run_id: str) -> None`-- `pipelex/tracing/graphspec_assembler.py:163` — `_AssemblerState.pass_one` — `def pass_one(self, events: Sequence[TraceEvent]) -> None`-- `pipelex/tracing/graphspec_assembler.py:220` — `_AssemblerState._track_earliest_timestamp` — `def _track_earliest_timestamp(self, event: TraceEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:224` — `_AssemblerState._handle_pipe_start` — `def _handle_pipe_start(self, event: PipeStartEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:248` — `_AssemblerState._handle_pipe_end_success` — `def _handle_pipe_end_success(self, event: PipeEndSuccessEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:276` — `_AssemblerState._handle_pipe_end_error` — `def _handle_pipe_end_error(self, event: PipeEndErrorEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:286` — `_AssemblerState._handle_edge_event` — `def _handle_edge_event(self, event: EdgeEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:304` — `_AssemblerState._handle_controller_output` — `def _handle_controller_output(self, event: ControllerOutputEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:314` — `_AssemblerState._handle_execution_data` — `def _handle_execution_data(self, event: ExecutionDataEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:321` — `_AssemblerState._handle_batch_item` — `def _handle_batch_item(self, event: BatchItemEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:327` — `_AssemblerState._handle_batch_aggregate` — `def _handle_batch_aggregate(self, event: BatchAggregateEvent) -> None`-- `pipelex/tracing/graphspec_assembler.py:333` — `_AssemblerState._handle_parallel_combine` — `def _handle_parallel_combine(self, event: ParallelCombineEvent) -> None`-- `pipelex/tracing/usage_aggregator.py:13` — `UsageAggregator.aggregate` — `def aggregate(events: Sequence[TraceEvent]) -> list[AnyTokensUsage]`--## C. `SYMMETRIC_ALLOWLIST` — Exception 2--_Exception 2 — the curated symmetric-tuple allowlist (multiple positionals that read better ordered, e.g. `set_env(key, value)`)._--Count: 5--- `pipelex/kit/single_file_agent_rules.py:86` — `unified_diff` — `def unified_diff(before: str, after: str, path: str) -> str`-- `pipelex/system/environment.py:49` — `set_env` — `def set_env(key: str, value: str) -> None`-- `pipelex/tools/misc/diff.py:79` — `diff_files` — `def diff_files(path1: str | Path, path2: str | Path) -> str`-- `pipelex/tools/misc/diff.py:356` — `diff_dirs` — `def diff_dirs(dir1: str | Path, dir2: str | Path) -> None`-- `pipelex/tools/typing/class_utils.py:69` — `are_classes_equivalent` — `def are_classes_equivalent(class_1: type[Any], class_2: type[Any]) -> bool`--## D. `MULTI_POSITIONAL` — variadic wrapper--_Survives the guard with 2+ positionals or a `*args` — i.e. a variadic wrapper/closure, not a deliberate subject choice._--Count: 1--- `pipelex/temporal/tprl/conditional_worker.py:36` — `with_conditional_worker.wrapper` — `async def wrapper(self: WorkflowExecutor[WorkflowInput, WorkflowOutput], *args: Any, **kwargs: Any) -> Any`-diff --git a/wip/history/keyword-only-arguments/audit/review-prompt.md b/wip/history/keyword-only-arguments/audit/review-prompt.mddeleted file mode 100644index 7f4e3cdc0..000000000--- a/wip/history/keyword-only-arguments/audit/review-prompt.md+++ /dev/null@@ -1,31 +0,0 @@-# Cold-start prompt — audit the positional-subject decisions--Paste the block below into a fresh session to run the review. It points the session at the audit file, fans out Sonnet 4.6 sub-agents per package, and produces a consolidated suspect list without touching any source.-------We just finished a keyword-only-arguments refactor of the pipelex/ runtime. Exception 1 of the convention lets the first true parameter (the "subject") stay positional. I want to audit whether that exception was ever ABUSED — i.e. a first arg made positional just to satisfy the rule, when it isn't really the semantic object of the function and would read better keyworded.--The full population is enumerated in:- wip/keyword-only-subject-positional-audit.md--Start by reading that file's "## Review strategy" section — it's the playbook. The headline points, so you don't re-derive them:--- This is a naming/API-taste call, not a reasoning task. Run the per-line judgments with SONNET 4.6 sub-agents (model: sonnet), fanned out one per package. Don't use Opus for the sweep.-- The bottleneck is CONTEXT, not model grade. Each sub-agent MUST have Read + Grep and is expected to open the function body (and a call site or two) whenever the signature alone is ambiguous. Do NOT just hand it the text line.-- Calibration matters: instruct each sub-agent to DEFAULT TO "fine" and emit ONLY the suspects, each as: file:line — function — current signature — why the positional reads wrong — suggested fix (usually "move the * before this arg too"). Cite the rubric in docs/contribute/keyword-only-arguments.md.--Triage (don't review all 1806 lines):--1. Section B (LONE_SUBJECT, ~1003) is mostly noise — one positional param means there was no subject choice to abuse. From it, keep ONLY primitive-typed lone subjects (bool/int/str/float, e.g. do_thing(True)); filter the rest mechanically. No LLM needed for the filter.-2. Fan out Sonnet 4.6 over Section A (SUBJECT_THEN_KEYWORDS, 797) + the primitive lone-subjects, one agent per package (the audit groups entries by package with counts).-3. Each sub-agent WRITES its package's shortlist to its own file:- wip/keyword-only-audit-findings/<package>.md- (disjoint paths, so parallel writes are safe; this preserves exact file:line citations that a returned chat message might truncate). Each entry:- file:line — function — current signature — why the positional reads wrong — suggested fix- An agent that finds nothing writes a file stating "no suspects".-4. After the fan-out, YOU (orchestrator) merge those into one consolidated report:- wip/keyword-only-subject-positional-suspects.md- grouped by package, ordered by confidence, with a top summary count. Then stop and show me — I'll adjudicate (or we point Opus at just that shortlist).--Scope of "don't change code": do NOT edit any pipelex/ source or tests in this pass — this is review only. Writing the findings/ files and the consolidated suspects report IS the deliverable, not a code change.diff --git a/wip/history/keyword-only-arguments/autofix-review-followups.md b/wip/history/keyword-only-arguments/autofix-review-followups.mddeleted file mode 100644index 19ac4ba13..000000000--- a/wip/history/keyword-only-arguments/autofix-review-followups.md+++ /dev/null@@ -1,67 +0,0 @@-# Keyword-only auto-fix — review follow-ups--Follow-up items from the code review of the **`--fix` auto-fix** addition to the keyword-only guard (branch `feature/Tweaks-for-validation-api`, 2026-06-17). This is a follow-on to the landed keyword-only refactor tracked in this folder's [README](README.md).--## What the auto-fix change added--- `pipelex-dev check-keyword-only --fix` — inserts a bare `*` as far left as possible (right after `self`/`cls`, and after any `/`) so **every** non-`self`/`cls` parameter becomes keyword-only, for every *mechanically-fixable* violation, and reports the rest for a manual fix. Core lives in the stdlib-only `keyword_only_guard.py` (`fix_source` / `fix_all_violations` / `_fix_insertion_point`); reporting in `check_keyword_only_cmd.py` (`_run_fix`).-- `make fix-keyword-only` / `fko` Makefile target.-- `make agent-check` now runs `fix-keyword-only` early (after `fix-unused-imports`, before `format`), replacing the old check-only `check-keyword-only` step that ran last. `make check` / `c` / `cc` keep the **check-only** variant (read-only, CI-equivalent).-- Tests in `tests/unit/pipelex/cli/dev/test_keyword_only_guard_fix.py`; docs "Auto-fix" section in `docs/contribute/keyword-only-arguments.md`.--The review (3 independent finder angles + verification, conventions/markdown clean) surfaced the items below. Ranked; each tagged **bug** (fix) or **tradeoff** (decide).--## Placement change (2026-06-17)--Per user direction, the auto-fixer places the bare `*` **as far left as possible** so all args become keyword-only — `def f(a, b)` → `def f(*, a, b)`, `def m(self, a, b)` → `def m(self, *, a, b)`, `def make(cls, source, target)` → `def make(cls, *, source, target)` — rather than the original after-subject form (`def f(a, *, b)`). The subject-stays-positional exception is a *permission*, not a requirement, and the convention explicitly calls the all-keyword-only form "always allowed and often preferable", so the fixer now takes it. `_fix_insertion_point` targets the first positional-or-keyword param that is not `self`/`cls`; a single positional-only subject is still fixable (the `*` goes after the `/`), and only two-or-more positional-only params remain unfixable. The unfixable shapes (`*args`, an existing keyword-only section) are unchanged. Tests in `test_keyword_only_guard_fix.py` and docs Auto-fix section updated to match.--## Findings--### F1 — `fix_source` can silently corrupt a file: `str.splitlines()` ≠ `ast.lineno` on exotic whitespace — **bug** — ✅ FIXED (2026-06-17)--`keyword_only_guard.py:424` — `lines = source.splitlines(keepends=True)`, indexed by `lineno - 1`.--`ast`/CPython count line breaks as only `\n`, `\r`, `\r\n`. `str.splitlines()` *also* splits on form-feed `\x0c`, ` `/` `, NEL `\x85`, VT `\x0b`, `\x1c–\x1e`. When any such character appears **before** a violation in the same file — most realistically a form-feed page-break inside a docstring — `lines[lineno-1]` points at the **wrong physical line** and the `b"*, "` insert lands there.--Verified end-to-end: when the wrong line is inside a string literal the result **still re-parses**, so the re-parse guard at `keyword_only_guard.py:431-434` does NOT catch it → corrupted file is **written to disk**, and the real violation is reported as "fixed" (it isn't). When the misplaced insert instead breaks syntax, a genuinely-fixable violation is wrongly reported "unfixable" (non-destructive but wrong).--Latent today: `grep -rlP '[\x0b\x0c\x85\x{2028}\x{2029}\x1c\x1d\x1e]' pipelex/ --include='*.py'` finds nothing. High impact when triggered. Landmine for any future file with such a char.--### F2 — pre-existing: same divergence breaks `# kw-only: ignore` detection — **bug** — ✅ FIXED (2026-06-17)--`keyword_only_guard.py` — `_def_line_has_escape_hatch` reads `source_lines[node.lineno-1]`, where `source_lines = source.splitlines()` (built in `find_violations_in_source` at `:387` and `_find_fix_records_in_source` at `:400`). Same root cause: with an exotic-whitespace char earlier in the file, the escape-hatch marker on a def line is read off the wrong line → **false-positive violation**, and the `PostToolUse` hook would wrongly block the edit. Not introduced by the `--fix` PR, but same module/mechanism, and one fix covers both (`CLAUDE.md`: "Flag and fix existing bugs").--**Proposed fix for F1+F2 (single change):** stop using `str.splitlines()` for `ast`-line indexing. Add a shared helper that splits on the tokenizer's newline set only — e.g. `re.split(r'(\r\n|\r|\n)', source)` (capturing group), take the content elements (even indices) for `source_lines`, and in `fix_source` edit the content element in place and reconstruct with `"".join(...)`. `re` is already imported in the module. Add a regression test: a form-feed-in-docstring file followed by a violation, asserting the violation (not the docstring) is what gets the `*` and the docstring is byte-for-byte intact.--**Resolution (2026-06-17):** Implemented as proposed. Added `_LINE_BREAK_RE = re.compile(r"(\r\n|\r|\n)")` and a shared `_split_source_lines(source)` helper (`_LINE_BREAK_RE.split(source)[::2]` — content lines are the even indices). The two detection paths (`find_violations_in_source`, `_find_fix_records_in_source`) now build `source_lines` via the helper, and `fix_source` splits with the capturing group (`parts = _LINE_BREAK_RE.split(source)`), edits the content element at `(lineno - 1) * 2` in place, and rejoins — reconstructing the file byte-for-byte. Two regression tests added (both verified red before the fix): `test_form_feed_in_string_neither_corrupts_nor_misreports` (F1 — exercises the *silent-corruption* path where the misplaced `*` lands inside a triple-quoted string and still re-parses) and `test_escape_hatch_survives_form_feed_earlier_in_file` (F2 — false-positive on a `# kw-only: ignore`d def). `make agent-check` green; full guard suite green.--### F3 — standalone `make fko` / `--fix` leaves the tree NOT `ruff format`-clean — **tradeoff (lean fix)** — ✅ FIXED (2026-06-17)--`Makefile:328` (`fix-keyword-only`) runs `check-keyword-only --fix --quiet` with no following `format`. The insert is correct but un-normalized: `def f(a,b)` → `def f(a,*, b)`; a multiline signature yields `*, b` on one line. Inside `agent-check` it's masked because `format` runs next — but the docs now tell a developer to run `make fko` on its own ("auto-insert the bare `*`"), and committing that output fails CI's `ruff-format --check`.--**Options:** (a) chain `format` (or `ruff-format`) into the `fix-keyword-only` target so the standalone path is self-consistent; (b) leave the target raw and add a one-line note in the docs that `fko` should be followed by `make format` (or run inside `agent-check`). (a) is the more solid fix.--**Resolution (2026-06-17):** Took option (a). The `fix-keyword-only` target (`Makefile`) now runs `$(VENV_RUFF) format . --config pyproject.toml` immediately after `check-keyword-only --fix --quiet`, so `make fix-keyword-only` / `make fko` leaves a `ruff format`-clean tree. `ruff format` (not `format`) is sufficient — the fixer only edits `.py` files, never the TOML/MTHDS that `plxt fmt` owns. Inside `agent-check` the later `format` step re-runs `ruff format`, but that is a near-instant no-op on the already-formatted tree. Docs Auto-fix section updated to match. Verified with a throwaway `pipelex/` violation: raw insert `def probe(a,*, b):` → after `make fko` → `def probe(a, *, b):`, `ruff format --check` exit 0.--### F4 — an unfixable violation now halts `agent-check` before pyright/mypy, possibly mid-mutation — **tradeoff** — ✅ FIXED (2026-06-17, Option B)--`check_keyword_only_cmd.py:127` (`_run_fix` `sys.exit(1)` on `unfixable`) + `Makefile:1168` (`agent-check: fix-unused-imports fix-keyword-only format lint pyright mypy`, fail-fast make).--Previously `check-keyword-only` ran **last** in `agent-check`, so pyright/mypy always ran. Now a single non-mechanically-fixable violation (`*args` present, an existing keyword-only section, a positional-only-only def) short-circuits the whole type-check pass. And if `fixed` was non-empty alongside `unfixable`, files were already written but `format` is skipped → partially-mutated, un-formatted tree (compounds F3).--This is partly by design (the goal was to surface keyword-only issues early). The decision: accept the fail-fast tradeoff, or run the keyword-only autofix in a way that doesn't gate the type-check phase (e.g. keep the fixer early but defer the non-zero exit on *unfixable* until after the rest, or split "fix" from "gate").--**Resolution (2026-06-17) — split fix from gate (Option B).** The `--fix` path is now **non-gating**: `_run_fix` reports both fixed and unfixable violations but no longer `sys.exit(1)`s on the unfixable ones (the `if not fixed:` "nothing to fix" branch was tightened to `if not fixed and not unfixable:` so an unfixable run never prints the empty-happy-path message). `agent-check` gained the read-only `check-keyword-only` as its **last** prerequisite (`... pyright mypy check-keyword-only`), which is what now gates. Net effect: the fixer runs early (fix → `ruff format`, both always run since `--fix` exits 0), so `format`/`lint`/`pyright`/`mypy` always run and the tree is never left half-mutated-and-unformatted (also fully closes the F3-compounded wart); the read-only check enforces last. `make check` / `c` / CI were already read-only-gating and are unchanged. Standalone `make fko` is now fix+report (exit 0); `make cko` / `make check-keyword-only` is the gate. Verified end-to-end with a fixable+unfixable probe: `fko` fixed + formatted + reported + exited 0 (`ruff format --check` clean), `cko` FAILED on the remaining unfixable, full `agent-check` runs pyright/mypy then gates last. F5 tests updated to the non-gating contract (unfixable → exit 0 + report). Docs Auto-fix section + gate list updated.--### F5 — `_run_fix` exit-code / reporting logic is untested — **tradeoff (add tests)** — ✅ FIXED (2026-06-17)--`test_keyword_only_guard_fix.py` covers `fix_source` / `fix_all_violations` well, but the command-layer control flow in `check_keyword_only_cmd.py:98-134` (`_run_fix`) has no coverage: fix-only → exit 0; unfixable → exit 1; the fixed-and-unfixable path (files written *and* exit 1); `fix` precedence over `report`; quiet vs verbose "nothing to fix" branch. `test_check_keyword_only_cmd.py` was not updated.--**Resolution (2026-06-17):** Added `tests/unit/pipelex/cli/dev/test_check_keyword_only_cmd_fix.py` (`TestCheckKeywordOnlyCmdFix`), driving the public `check_keyword_only_cmd(fix=...)` so the real `_run_fix` control flow runs. `fix_all_violations` / `collect_all_violations` are mocked (pytest-mock) to drive each branch; `get_console` is routed to a `StringIO`-backed `Console` and `SOURCE_ROOT` to a real temp dir, so assertions are deterministic and touch no real files. Pins the enumerated cases: fix-only → exit 0 + "Auto-fixed"; quiet "nothing to fix" one-liner vs verbose success panel; and `fix` precedence (`collect_all_violations` / `_print_report` asserted not called). A new module (not the existing `test_check_keyword_only_cmd.py`, which holds the guard-core `TestCheckKeywordOnly` class) — one TestClass per module. `make agent-check` green; CLI unit suite green. (Note: the unfixable cases were updated by F4 below — they now assert non-gating exit 0 + a printed manual-fix report, not `SystemExit(1)`.)--## Recommended order--1. ~~**F1 + F2** — real silent-correctness bugs; fix together with the tokenizer-accurate line split + regression test.~~ ✅ Done 2026-06-17.-2. ~~**F3** — small, worthwhile; chain `format` into `fko` (option a).~~ ✅ Done 2026-06-17.-3. ~~**F4** — decide consciously; current behavior is defensible but is a regression in feedback completeness vs the old ordering.~~ ✅ Done 2026-06-17 (Option B: split fix from gate).-4. ~~**F5** — add command-layer tests if the exit-code contract is worth pinning (it gates CI/`agent-check`, so likely yes).~~ ✅ Done 2026-06-17.diff --git a/wip/history/keyword-only-arguments/downstream-consumer-breakage.md b/wip/history/keyword-only-arguments/downstream-consumer-breakage.mddeleted file mode 100644index 240618dc8..000000000--- a/wip/history/keyword-only-arguments/downstream-consumer-breakage.md+++ /dev/null@@ -1,76 +0,0 @@-# Downstream consumer breakage — keyword-only refactor follow-up--> **Status (2026-06-15):** lockstep PRs opened against each consumer's `dev`, each pinning `pipelex` to the keyword-only branch git rev (`529b9082`, branch `refactor/Function-calling-4`) and carrying the call-site fixes. All four pass local `make agent-check` + `make agent-test`:->-> - **pipelex-api** → [PR #19](https://github.com/Pipelex/pipelex-api/pull/19) — keyword-only delta only (it already tracked pipelex `dev`): one `parse_pipe_spec(pipe_type, spec_data=...)` fix.-> - **pipelex-starter-python** → [PR #55](https://github.com/Pipelex/pipelex-starter-python/pull/55)-> - **cocode** → [PR #75](https://github.com/Pipelex/cocode/pull/75)-> - **pipelex-cookbook** → [PR #153](https://github.com/Pipelex/pipelex-cookbook/pull/153)->-> **Key finding:** cookbook/cocode/starter-python were pinned to **PyPI `pipelex` 0.32.x**, so pinning them to the branch (which tracks pipelex `dev`) surfaced the **whole accumulated dev delta**, dominated by a *non*-keyword-only change — the **`PipelexRunner` → `PipelexMTHDSProtocol`** runner rename (MTHDS Protocol surface alignment: `execute_pipeline()` → `execute()`, `response.pipe_output` unchanged). Those three PRs therefore carry that runtime-runner migration **in addition to** the keyword-only fixes. Keyword-only-specific fixes were tiny: cookbook's `save_text_to_path` + its `get_stuff_as`/`get_stuff_as_list` (`content_type=`/`item_type=`) calls; nothing extra in cocode/starter. The git-rev pins are **temporary** — swap each back to a released `pipelex` version once the refactor lands on PyPI.->-> **Follow-up (2026-06-15b):** `parse_pipe_spec` arg order was **reversed** in pipelex — `parse_pipe_spec(pipe_type, *, spec_data)` → `parse_pipe_spec(spec_data, *, pipe_type)` — so the subject is now the thing being parsed (the data), matching its sibling `parse_concept_spec(spec_data)`. This re-breaks the **single** call site PR #19 had already fixed: `pipelex-api/api/routes/pipelex/agent/pipe_spec.py:48` must change from `parse_pipe_spec(request_data.pipe_type, spec_data=request_data.spec)` to `parse_pipe_spec(request_data.spec, pipe_type=request_data.pipe_type)`. **Sequence (don't desync the pin):** (1) commit the pipelex reversal → new SHA; (2) bump pipelex-api's `[tool.uv.sources]` rev from `529b908255…` to the new SHA on PR #19's branch `chore/keyword-only-pipelex-rev`; (3) apply the call-site fix; (4) `make agent-check` + `make agent-test` in pipelex-api. No other consumer calls `parse_pipe_spec`, so #55 / #75 / #153 are unaffected by the reversal (only a rev bump if we want them on the newest SHA).->-> **Follow-up (2026-06-15c) — DONE (push pending):** pipelex HEAD `755b82117fd4c26ba090455cfd571099cf05708b` is pushed to `refactor/Function-calling-4`. That commit turned out **broader** than just the `parse_pipe_spec` reversal — it also flips the subject/keyword order of several other signatures (`save_bytes_to_binary_file`, `hydrate_content`, `load_template`, `load_config`, `build_deck`, `write_manifest`, `text_in_renaming_keys`/`text_in_renaming_values`, `_row_to_dict`, plus the `init` CLI helpers `customize_backends_config`/`setup_telemetry`/`display_selected_backends`/`prompt_primary_backend`). Re-audited every consumer for those names: **none** call them positionally (each site already passes by keyword), so `parse_pipe_spec` in pipelex-api stayed the *only* real call-site break. All four consumer PR branches (`chore/keyword-only-pipelex-rev`) had their pin bumped `529b9082…` → `755b8211…` in **both** `pyproject.toml` and `uv.lock`, re-synced, and verified green on `make agent-check` + `make agent-test`; pipelex-api additionally carries the `pipe_spec.py` call-site fix + CHANGELOG entry. Each branch has a fresh commit on top — **not yet pushed** (push was left to the user / blocked by auto-mode). When pushing, the open PRs #19 / #55 / #75 / #153 update in place.->-> **Follow-up (2026-06-15d) — recheck after the positional-subject consolidation:** pipelex HEAD moved again to `0e32c8c02` ("Consolidate keyword-only refactor for positional-subject abuse suspects", pushed to `refactor/Function-calling-4`). That commit turned all 59 positional-subject suspects **fully** keyword-only by relocating the bare `*` to immediately after `self`/`cls` — **no reordering, no renames** (see [`positional-subject-suspects.md`](positional-subject-suspects.md)). So the only way it can break a consumer is a now-keyword-only *subject* still passed **positionally**. Re-audited every consumer (`pipelex-api`, `cocode`, `pipelex-cookbook`, `pipelex-starter-python`, `pipelex-mistralai-workflows`, `pipelex-worker`) for call sites of all 59 changed functions. **Exactly one new break:** `pipelex-api/tests/unit/conftest.py:12` calls `Pipelex.make(IntegrationMode.PYTEST, needs_inference=..., temporal_enabled=False)` — `integration_mode` positional. This was *compliant* under `755b82117` (Wave 5 left the subject positional-allowed) but `0e32c8c02` made `Pipelex.make` fully keyword-only (`make(cls, *, integration_mode=..., ...)`), so it now raises `TypeError`. Fix: `Pipelex.make(integration_mode=IntegrationMode.PYTEST, needs_inference=..., temporal_enabled=False)`. Every other consumer is clean: `cocode` passes `integration_mode=IntegrationMode.CI` by keyword and `make_pipelex_for_cli(context=...)` by keyword; cookbook/starter-python omit `integration_mode`; the `pipe_job.pipe.run_pipe(job_metadata=..., ...)` site in `pipelex-mistralai-workflows` is all-keyword; `make_temporal_pipe_run()`/`make_temporal_pipe_router()`/`check_is_initialized()` are no-arg. `mistralai-workflows` is on `feature/Mistral-native` (mistralai-2x HOLD pin, not the keyword-only branch) and `pipelex-worker` is on `dev` (PyPI version pin) — both out of scope. **Pin staleness:** the four `chore/keyword-only-pipelex-rev` branches (pipelex-api / cocode / cookbook / starter-python) are still pinned to `755b82117`, now one commit behind HEAD `0e32c8c02`. To restore true lockstep, bump `755b82117…` → `0e32c8c02…` in `pyproject.toml` **and** `uv.lock` on each; only **pipelex-api** additionally needs the conftest call-site fix above. **DONE (push pending):** all four `chore/keyword-only-pipelex-rev` branches re-pinned `755b82117…` → `0e32c8c02600e17bbe146331f9b701ccc03a1ca1` (pyproject + uv.lock), reinstalled via `uv sync --all-extras`, and verified green on `make agent-check` + `make agent-test` — pipelex-api (commit `24c91c4`, also carries the `tests/unit/conftest.py` `integration_mode=` fix + CHANGELOG line), cocode (`b82d26c`), pipelex-starter-python (`fbf69be`), pipelex-cookbook (`7d48804`). **Pushed** to the existing PR branches (#19 / #75 / #55 / #153), which update in place. The only remaining open item is the at-release pin swap (see checklist).->-> **Follow-up (2026-06-15e) — pipelex-worker brought into scope:** previously marked out of scope only because it pinned `pipelex` via a PyPI version (`pipelex[dynamodb,s3,temporal]==0.33.0`), not a git rev. Now aligned with the others on a new `chore/keyword-only-pipelex-rev` branch off `dev`: the PyPI pin is overridden with the `0e32c8c0` git rev via `[tool.uv.sources]` and relocked (`uv lock`: registry → git source). The worker has **no `pipelex` imports in its own source** (it just runs the installed `pipelex worker` process), so there is **no keyword-only call-site break**. Its `pipelex doctor` config-health smoke tests then failed on a *pre-existing* brittleness, not a config regression: `pipelex doctor` renders through rich and emits ANSI style/reset codes even when piped (`NO_COLOR` only drops colors, not the bold/reset), and a `\x1b[0m` between `Configuration Files` and the `✓` defeated the section regex — fixed by stripping ANSI before matching. The merged config is valid against 0.33.0 (no `ConfigValidationError`); the only residual is the expected soft `Model Deck installed for 0.32.1, current 0.33.0` warning, which the test ignores by design and does not break the worker. All three env smoke tests green. Commit `af8572a`, pushed, **[PR #23](https://github.com/Pipelex/pipelex-worker/pull/23)** opened against `dev`. `pipelex-mistralai-workflows` remains genuinely out of scope — its own mistralai-2x hold track, editable-pinned to the `../_workflows` pipelex worktree, gated until instructor #2298.--> Original release-time checklist below (the pipelex-internal keyword-only refactor is **complete** — all waves landed, guard hard-blocks; see [`../../../TODOS.md`](../../../TODOS.md)). Companion to the convention itself at [`../../../docs/contribute/keyword-only-arguments.md`](../../../docs/contribute/keyword-only-arguments.md).--The keyword-only refactor intentionally turns multi-arg public helpers keyword-only (e.g. `save_text_to_path(text, *, path, create_directory=False)`, and on the public surface `Pipelex.make()` / `Pipelex.setup()` / `PipelexHub.setup_config()` — every argument after the first is now keyword-only). This is a deliberate breaking change, documented in the runtime `CHANGELOG.md`. Breaking changes are allowed in this workspace, but consumers must be updated in lockstep — otherwise the official examples ship broken.--## Confirmed breakage (Wave 1 — `tools/`)--- **`pipelex-cookbook/utils/results_utils.py:42`** — `save_text_to_path(content, result_file_path)` passes `path` positionally, but `path` is now keyword-only. Fix: `save_text_to_path(text=content, path=result_file_path)`. (The third param `create_directory` defaults to `False` and is unchanged.)--That is the only positional break found across the consumer repos for the Wave 1 public surface. The later waves (`core/` … the public `pipelex.py` / `hub.py` surface) have **not** yet been re-audited across the consumer repos — do that at release (recipe below).--## Repos audited and clean (Wave 1)--- **`cocode`** — already calls `save_text_to_path` in keyword form (`text=…, path=…`) at every site (`repox/repox_cmd.py`, `swe/swe_utils.py`, `swe/swe_cmd.py`). Its other pipelex-tools imports (`load_text_from_path`, `failable_load_text_from_path`, `ensure_path`, `path_exists`, `load_json_list_from_path`, `pascal_case_to_kebab`, `pascal_case_to_sentence`, `empty_list_factory_of`) are all single-arg or already keyword — no break.-- **`pipelex-starter-python`** — only uses `pretty_print` (single positional `content`), no break.-- **`pipelex-api`, `pipelex-worker`, `pipelex-mistralai-workflows`** — no positional calls to keyword-only Wave 1 helpers. (Re-check `Pipelex.make` / `setup` call sites — the public-surface change landed in Wave 5.)-- **`pretty_print` / `pretty_print_md` / `pretty_print_url`** (the only `pipelex/__init__.py` `__all__` exports) — now keyword-only on everything after `content`; no consumer passes `title` (or any later arg) positionally, so nothing breaks today. Worth keeping an eye on since these are the headline public API.--## Public surface converted in Wave 5 (re-audit before release)--The final wave made the top-level entry points keyword-only after their first parameter. A consumer passing a second-or-later argument positionally breaks:--- `Pipelex.make(integration_mode, *, ...)` — e.g. `Pipelex.make(IntegrationMode.PYTHON, False)` → `Pipelex.make(IntegrationMode.PYTHON, needs_inference=False)`.-- `Pipelex.setup(...)` and `PipelexHub.setup_config(config_cls, *, ...)` — same shape.--The common no-arg / all-keyword calls (`Pipelex.make()`, `Pipelex.make(needs_inference=False)`) are unaffected. Consumers to check: `pipelex-api`, `pipelex-worker`, `n8n-nodes-pipelex`, `pipelex-cookbook`, `pipelex-starter-python`, `pipelex-mistralai-workflows`.--## How to re-run the audit--From the workspace root (`/Users/lchoquel/repos/Pipelex`):--```bash-# 1. positional second-arg calls to the file/bytes writers-rg -n -g '*.py' -e 'save_text_to_path\(' -e 'save_bytes_to_path\(' \- cocode pipelex-cookbook pipelex-starter-python pipelex-mistralai-workflows pipelex-api pipelex-worker \- | grep -vE 'text=|path=|content='--# 2. pretty_print* called with a positional non-content arg-rg -n -g '*.py' 'pretty_print(_md|_url)?\([^)]*,[^=)]' \- cocode pipelex-cookbook pipelex-starter-python | grep -vE 'title=|subtitle='--# 3. public bootstrap entry points with a positional second arg-rg -n -g '*.py' -e 'Pipelex\.make\(' -e 'Pipelex\.setup\(' -e 'setup_config\(' \- cocode pipelex-cookbook pipelex-starter-python pipelex-mistralai-workflows pipelex-api pipelex-worker n8n-nodes-pipelex \- | grep -E '\([^)]+,[^=)]'-```--Generalise the helper list per wave: take the functions a wave converted (the `### Changed` keyword-only entries in `CHANGELOG.md` name the packages and the headline signatures), keep only those imported by a consumer repo and exposing a second positional arg, then grep for positional call sites.--## Checklist--- [x] `pipelex-cookbook/utils/results_utils.py:42` → keyword form (PR #153, with its unit-test assertion updated to keyword form).-- [x] Re-run the audit greps for the full converted surface (Waves 1–5). Only positional keyword-only break found across consumers: cookbook's `save_text_to_path` + `get_stuff_as*`, and pipelex-api's `parse_pipe_spec`. All `Pipelex.make` call sites already pass the subject positionally + the rest by keyword (compliant).-- [x] Open lockstep PRs pinning each consumer to the branch git rev, all green on local `make agent-check` + `make agent-test` (PRs #19 / #55 / #75 / #153).-- [x] **`parse_pipe_spec` reversal (2026-06-15b) — pipelex-api lockstep:** done. pipelex reversal pushed (SHA `755b82117`); PR #19's pin bumped (pyproject + uv.lock), `pipe_spec.py:48` changed to `parse_pipe_spec(request_data.spec, pipe_type=request_data.pipe_type)`, CHANGELOG updated. `make agent-check` + `make agent-test` green. Committed on `chore/keyword-only-pipelex-rev`; push pending. The 2026-06-15c note widens this to all four consumers (the same SHA also reordered other signatures, none of which any consumer calls positionally).-- [x] **Positional-subject consolidation (2026-06-15d) — re-audit + fix DONE (push pending):** pipelex HEAD `0e32c8c02` made all 59 suspects fully keyword-only (no reorder). Re-audited all consumers → only new break was `pipelex-api/tests/unit/conftest.py:12` (`Pipelex.make(IntegrationMode.PYTEST, …)` positional → now `integration_mode=...`). Bumped the four `chore/keyword-only-pipelex-rev` pins `755b82117…` → `0e32c8c02…` (pyproject + uv.lock), applied the conftest fix to pipelex-api, reinstalled, and verified `make agent-check` + `make agent-test` green on all four. Commits: pipelex-api `24c91c4`, cocode `b82d26c`, starter-python `fbf69be`, cookbook `7d48804` — **pushed** to the PR branches (#19 / #75 / #55 / #153), which update in place.-- [x] **pipelex-worker (2026-06-15e) — brought into scope:** PyPI `==0.33.0` pin → `0e32c8c0` git rev (`[tool.uv.sources]` + `uv lock` relock, registry→git). No call-site break (no `pipelex` imports). Fixed a brittle ANSI assertion in the `pipelex doctor` smoke test (strip ANSI before matching); 3/3 env tests green. Commit `af8572a`, [PR #23](https://github.com/Pipelex/pipelex-worker/pull/23) vs `dev`. (`pipelex-mistralai-workflows` stays out of scope — own mistralai-2x hold track.)-- [ ] **At pipelex release:** swap each consumer's temporary `[tool.uv.sources]` git pin back to the released `pipelex` version (`==`/`>=`), and confirm CI stays green. Remember the release also ships the `PipelexRunner → PipelexMTHDSProtocol` rename — already handled in these PRs.diff --git a/wip/history/keyword-only-arguments/keyword-only-arguments.html b/wip/history/keyword-only-arguments/keyword-only-arguments.htmldeleted file mode 100644index 62be01a66..000000000--- a/wip/history/keyword-only-arguments/keyword-only-arguments.html+++ /dev/null@@ -1,392 +0,0 @@-<!doctype html>-<html lang="en">-<head>-<meta charset="utf-8" />-<meta name="viewport" content="width=device-width, initial-scale=1" />-<title>Keyword-only arguments — what we built</title>-<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>-<style>- :root {- --bg: #0f1117; --panel: #171a23; --panel2: #1d212c; --ink: #e6e9ef; --muted: #9aa3b2;- --line: #2a2f3a; --accent: #6ea8fe; --good: #4ec98a; --bad: #f0768b; --warn: #e7b964;- --add-bg: #11271b; --add-ink: #7fe0a8; --del-bg: #2a1620; --del-ink: #f3899d; --code: #cdd6e6;- --mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;- }- * { box-sizing: border-box; }- body {- margin: 0; background: var(--bg); color: var(--ink);- font: 16px/1.65 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;- }- .wrap { max-width: 980px; margin: 0 auto; padding: 40px 24px 96px; }- header.hero { border-bottom: 1px solid var(--line); padding-bottom: 22px; margin-bottom: 28px; }- .eyebrow { color: var(--accent); font-weight: 700; letter-spacing: .14em; text-transform: uppercase; font-size: 12px; }- h1 { font-size: 33px; line-height: 1.15; margin: 8px 0 6px; }- h2 { font-size: 23px; margin: 44px 0 12px; padding-top: 10px; border-top: 1px solid var(--line); }- h3 { font-size: 17px; margin: 26px 0 8px; color: #cfd6e4; }- p, li { color: #d3d8e2; }- a { color: var(--accent); }- code { font-family: var(--mono); background: #11141c; border: 1px solid var(--line); border-radius: 5px; padding: .08em .36em; font-size: .9em; color: var(--code); }- .sub { color: var(--muted); font-size: 15px; }- .tldr {- background: linear-gradient(180deg, #1a2030, #161b28);- border: 1px solid #2c3650; border-left: 4px solid var(--accent);- border-radius: 12px; padding: 18px 22px; margin: 4px 0 8px;- }- .tldr h2 { border: 0; margin: 0 0 8px; padding: 0; font-size: 18px; color: var(--accent); }- .tldr ul { margin: 8px 0 0; padding-left: 20px; }- .tldr li { margin: 5px 0; }- .grid { display: grid; grid-template-columns: 1fr 1fr; gap: 14px; margin: 16px 0; }- @media (max-width: 720px){ .grid { grid-template-columns: 1fr; } }- .card { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 14px 16px; }- .card h3 { margin-top: 0; }- .pill { display: inline-block; font-size: 12px; font-weight: 600; padding: 2px 9px; border-radius: 999px; border: 1px solid var(--line); color: var(--muted); }- .pill.good { color: var(--good); border-color: #275; } .pill.warn { color: var(--warn); border-color: #654; }- figure { margin: 18px 0; background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 14px; }- figure figcaption { color: var(--muted); font-size: 13px; margin-top: 8px; text-align: center; }- .mermaid { background: transparent; text-align: center; }- details { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 8px 16px; margin: 12px 0; }- details[open] { padding-bottom: 14px; }- summary { cursor: pointer; font-weight: 650; color: #d8def0; padding: 6px 0; }- summary::marker { color: var(--accent); }- table { width: 100%; border-collapse: collapse; margin: 14px 0; font-size: 14.5px; }- th, td { border: 1px solid var(--line); padding: 8px 11px; text-align: left; vertical-align: top; }- th { background: var(--panel2); color: #cfd6e4; }- td code { font-size: .86em; }- /* tabs */- .tabs { margin: 16px 0; border: 1px solid var(--line); border-radius: 12px; overflow: hidden; background: var(--panel); }- .tabbar { display: flex; flex-wrap: wrap; gap: 0; background: var(--panel2); border-bottom: 1px solid var(--line); }- .tabbar button {- appearance: none; background: transparent; border: 0; border-right: 1px solid var(--line);- color: var(--muted); padding: 10px 15px; cursor: pointer; font: 600 13.5px var(--mono); letter-spacing: .02em;- }- .tabbar button:hover { color: var(--ink); }- .tabbar button.active { color: var(--ink); background: var(--panel); box-shadow: inset 0 -2px 0 var(--accent); }- .tabpanel { display: none; padding: 0; }- .tabpanel.active { display: block; }- pre { margin: 0; padding: 16px 18px; overflow-x: auto; font-family: var(--mono); font-size: 13px; line-height: 1.55; color: var(--code); background: #0c0e14; }- pre.plain { border: 1px solid var(--line); border-radius: 10px; }- .diff .ln { display: block; white-space: pre; }- .diff .add { background: var(--add-bg); color: var(--add-ink); }- .diff .del { background: var(--del-bg); color: var(--del-ink); }- .diff .ctx { color: #8b93a4; }- .diff .hdr { color: #6f7891; }- .filechip { font: 600 12px var(--mono); color: var(--muted); padding: 8px 18px; background: var(--panel2); border-bottom: 1px solid var(--line); }- .note { font-size: 14px; color: var(--muted); }- .kbd { font-family: var(--mono); background:#11141c; border:1px solid var(--line); border-bottom-width:2px; border-radius:5px; padding:.03em .4em; }- .legend { display:flex; gap:16px; flex-wrap:wrap; font-size:12.5px; color:var(--muted); margin: 6px 2px 0;}- .legend span { display:inline-flex; align-items:center; gap:6px;}- .sw { width:14px; height:14px; border-radius:3px; display:inline-block; border:1px solid var(--line);}- footer { margin-top: 50px; border-top: 1px solid var(--line); padding-top: 18px; color: var(--muted); font-size: 13.5px; }-</style>-</head>-<body>-<div class="wrap">--<header class="hero">- <div class="eyebrow">Pipelex runtime · refactor/Function-calling-4</div>- <h1>Keyword-only arguments, everywhere — and a guard that keeps it that way</h1>- <p class="sub">A codebase-wide refactor + a mechanical enforcement guard, plus the lockstep update of every downstream consumer. Written for the people who'll write the next function.</p>-</header>--<div class="tldr">- <h2>TL;DR</h2>- <ul>- <li><b>What changed:</b> across all of <code>pipelex/</code>, every <i>non-subject</i> function parameter is now <b>keyword-only</b>. A function may keep its first real argument (the "subject") positional; everything after it must be passed by name. So <code>do_thing(retries=3, timeout=30)</code> — never the opaque <code>do_thing(3, 30)</code>.</li>- <li><b>Why:</b> call sites become self-documenting, and you can reorder / insert parameters without silently rebinding a caller's positional args.</li>- <li><b>How it stays true:</b> a stdlib-only AST guard (<code>check-keyword-only</code>) runs in <code>make agent-check</code>, in the <code>make check</code> aggregate, and in CI — it <b>hard-blocks on any violation</b>. A <code>PostToolUse</code> hook runs the same guard on each edited file in ~40 ms, so agents get blocked in-loop, not at CI.</li>- <li><b>The sweep:</b> ~1,800 functions audited (one Sonnet sub-agent per package); <b>59</b> "kept-positional-out-of-habit" suspects were found and all made <i>fully</i> keyword-only — by relocating the bare <code>*</code> only, <b>no reordering, no renames</b>.</li>- <li><b>Breaking & downstream:</b> the public surface (<code>Pipelex.make()</code> / <code>setup()</code> / <code>PipelexHub.setup_config()</code>) is now keyword-only after the subject. All four consumer repos (pipelex-api, cocode, cookbook, starter-python) were re-pinned to the branch and fixed in lockstep — exactly <b>one</b> real call-site break across all of them. Green on <code>agent-check</code> + <code>agent-test</code>; PRs pushed.</li>- <li><b>One open item:</b> at pipelex release, swap each consumer's temporary git pin back to a released version.</li>- </ul>-</div>--<h2>The rule, in one picture</h2>-<p>The compliant shape places a bare <code>*</code> right after the subject. The subject — the first non-<code>self</code>/<code>cls</code> parameter — <i>may</i> stay positional; making it keyword-only too is always allowed and often nicer.</p>--<figure>-<svg viewBox="0 0 900 220" width="100%" role="img" aria-label="Anatomy of a compliant signature">- <style>- .mono{font-family:var(--mono);font-size:18px;fill:#e6e9ef}- .lbl{font-family:var(--mono);font-size:12.5px}- .box{fill:none;stroke-width:2;rx:8}- </style>- <rect x="0" y="0" width="900" height="220" fill="#12151d" rx="14"/>- <text x="40" y="74" class="mono">def <tspan fill="#6ea8fe">handle</tspan>(<tspan fill="#9aa3b2">self</tspan>, <tspan fill="#4ec98a">request</tspan>, <tspan fill="#e7b964">*</tspan>, <tspan fill="#cdd6e6">retries</tspan>=3, <tspan fill="#cdd6e6">timeout</tspan>=30):</text>-- <!-- subject brace -->- <rect x="148" y="48" width="86" height="34" class="box" stroke="#4ec98a"/>- <text x="191" y="120" text-anchor="middle" class="lbl" fill="#4ec98a">subject</text>- <text x="191" y="138" text-anchor="middle" class="lbl" fill="#7f9">may stay positional</text>-- <!-- star -->- <rect x="245" y="48" width="22" height="34" class="box" stroke="#e7b964"/>- <text x="256" y="120" text-anchor="middle" class="lbl" fill="#e7b964">bare *</text>- <text x="256" y="138" text-anchor="middle" class="lbl" fill="#caa">the boundary</text>-- <!-- keywords -->- <rect x="278" y="48" width="300" height="34" class="box" stroke="#6ea8fe"/>- <text x="428" y="120" text-anchor="middle" class="lbl" fill="#6ea8fe">keyword-only</text>- <text x="428" y="138" text-anchor="middle" class="lbl" fill="#9bf">must be passed by name</text>-- <text x="40" y="190" class="lbl" fill="#9aa3b2">Call site → <tspan fill="#4ec98a">handle(request)</tspan> or <tspan fill="#4ec98a">handle(request=req, retries=5, timeout=10)</tspan> — never <tspan fill="#f0768b">handle(req, 5, 10)</tspan></text>-</svg>-<figcaption>A lone subject (<code>def render(node)</code>) is compliant. A second bare positional (<code>def f(a, b)</code>, <code>def truncate(text, max_length=80)</code>) is a violation.</figcaption>-</figure>--<div class="tabs" id="rule-tabs">- <div class="tabbar">- <button class="active" data-tab="ok">✓ compliant</button>- <button data-tab="bad">✗ violations</button>- <button data-tab="hatch"># kw-only: ignore</button>- </div>- <div class="tabpanel active" data-panel="ok">-<pre class="plain">def render(node): ... # lone subject — fine-def f(subject, *, opt1, opt2): ... # subject positional, rest keyword-def f(*, a, b): ... # fully keyword-only — always allowed-def make(cls, *, integration_mode=...): ... # classmethod, fully keyword-only</pre>- </div>- <div class="tabpanel" data-panel="bad">-<pre class="plain">def f(a, b): ... # second bare positional-def truncate(text, max_length=80): ... # max_length must be keyword-only-def make(cls, integration_mode, *, ...): .. # subject kept positional "by habit"- # when every call site already- # passes integration_mode=...</pre>- </div>- <div class="tabpanel" data-panel="hatch">-<pre class="plain"># A genuinely justified one-off — placed right after the open paren so-# `ruff format` keeps it on the def header line:-def legacy_cb(a, b): # kw-only: ignore- ...--# Carve-outs are automatic (no marker needed): dunders, pydantic-# validators/serializers, Typer/Temporal/pytest/Jinja2 entrypoints,-# @override impls, and a curated symmetric-tuple allowlist.</pre>- </div>-</div>--<h2>How it stays true: three layers of enforcement</h2>-<p>One rule, one AST core, three places it fires. The guard is <b>stdlib-only</b> (<code>ast</code>/<code>re</code>/<code>pathlib</code>) so the single-file path can run without importing the <code>pipelex</code> package chain — that's what keeps the editor hook at a few tens of milliseconds.</p>--<figure>-<div class="mermaid">-flowchart LR- subgraph DEV["In-loop (per edit)"]- E["Edit/Write a *.py file"] --> H{{"PostToolUse hook<br/>check-keyword-only.sh"}}- H -->|"by file path,<br/>no -m import"| G1["guard.main()<br/>single file, ~40ms"]- G1 -->|violation| B1["exit 2 → blocks the agent<br/>prints offending signatures"]- G1 -->|clean| P1["exit 0"]- end- subgraph CI["Gate (pre-merge)"]- C["make agent-check / make check / CI"] --> G2["pipelex-dev check-keyword-only<br/>full pipelex/ tree"]- G2 -->|any violation| B2["non-zero → hard-block"]- G2 -->|clean| P2["pass"]- end- G1 -. "same AST core<br/>(find_violations_in_source)" .- G2-</div>-<figcaption>The single-file hook result is an exact subset of the full-tree scan — <code>relative_path</code>/<code>module_qname</code> are computed identically, so the carve-out allowlist matches either way.</figcaption>-</figure>--<details>- <summary>The carve-outs (skipped automatically) and the escape hatch</summary>- <ul>- <li><b>Dunders</b> — <code>__init__</code>, <code>__call__</code>, … (interpreter calls them positionally).</li>- <li><b>pydantic</b> — <code>@field_validator</code>, <code>@model_validator</code>, <code>@field_serializer</code>, <code>@model_serializer</code>, <code>@validator</code>, <code>@root_validator</code>.</li>- <li><b>Framework entrypoints invoked positionally</b> — Typer <code>@app.command</code>/<code>@app.callback</code> (and the decorator-less <code>Argument(...)</code>/<code>Option(...)</code> annotation form), Temporal <code>@activity.defn</code> / <code>@workflow.run|signal|query|update</code>, <code>@pytest.fixture</code>, and Jinja2 <code>@pass_context</code>/<code>@pass_environment</code>/<code>@pass_eval_context</code>.</li>- <li><b><code>@override</code> impls</b> — they track the base signature.</li>- <li><b>Curated symmetric-tuple allowlist</b> — a handful of <code>(qualified_name, path)</code> pairs where both operands are genuinely peers (e.g. <code>diff_files(a, b)</code>). No pattern guessing; both name and path must match exactly.</li>- <li><b>Escape hatch</b> — an inline <code># kw-only: ignore</code> on the <code>def</code> line, for a justified one-off.</li>- </ul>- <p class="note">The type checker can't see framework-positional callbacks (route handlers, <code>__import__</code> hooks), so <code>make agent-test</code> is the safety net — and it caught real cases during the rollout.</p>-</details>--<h2>Rolling it out: audit → consolidation → enforce</h2>-<p>The tree was made compliant wave by wave (execution path, then the framework-sensitive & public API last). Then a dedicated pass hunted the subtler problem: signatures that satisfied the rule on a technicality — they kept <i>a</i> subject positional — but where that "subject" was really a lookup table, a mode flag, or a trace-context carrier, while the actual operand sat in the keyword args.</p>--<figure>-<div class="mermaid">-flowchart TD- A["Waves 1–5: make pipelex/ compliant<br/>(bare * after the subject)"] --> B["Guard built + CI-gated<br/>(hard-block, baseline scaffolding removed)"]- B --> C["Positional-subject audit<br/>~1,800 functions · 1 Sonnet sub-agent per package"]- C --> D{"Is the positional<br/>arg really the subject?"}- D -->|"yes — fine"| E["leave it"]- D -->|"no — abuse"| F["59 suspects<br/>(lookup tables, mode flags,<br/>trace carriers, symmetric pairs)"]- F --> G["Consolidate: all 59 made FULLY keyword-only<br/>move * before the subject · no reorder · no rename"]- G --> H["agent-check + agent-test green"]- H --> I["Downstream consumers re-pinned + fixed in lockstep"]-</div>-<figcaption>"No reordering, no renames" is the load-bearing property: it means the only way the consolidation can break a caller is if that caller passed a now-keyword-only subject <i>positionally</i>.</figcaption>-</figure>--<details>- <summary>The recurring shapes the 59 suspects collapsed into</summary>- <ul>- <li><b>Lookup-table / registry carried positional</b> while the real query is keyworded — e.g. <code>model_cost_per_token(costs, *, cost_category)</code>, <code>get_reasoning_level_str(effort_to_level_map, *, effort)</code>.</li>- <li><b><code>*_for_pipe</code> / <code>*_from_response</code> / <code>*_from_uri</code></b> — the name advertises the operand, but a <i>different</i> arg was positional.</li>- <li><b>Mode flag / bare bool positional</b> — <code>f(True)</code> reveals nothing, and every call site already keyworded it.</li>- <li><b>Trace/provenance carrier positional</b> (<code>job_metadata</code>, <code>lookup_key</code>, <code>calling_pipe_code</code>) while the function acts on the keyword args.</li>- <li><b>Accumulator/sink dict positional</b> while the content being added is keyworded.</li>- <li><b>Symmetric pair split across the <code>*</code></b> — neither operand is "the subject", so the split was just noise.</li>- <li><b>Public-API surface</b> — <code>Pipelex.make</code>/<code>setup</code>, a deliberate keyword-only break.</li>- </ul>- <p class="note">A near-universal tell: the existing call sites already passed the "positional" arg by keyword — the positional permission was buying nothing.</p>-</details>--<h2>Downstream: one branch SHA, five consumers, exactly one real break</h2>-<p>The consumers were pinned to the pipelex branch git rev and walked forward as the branch moved. The headache wasn't the keyword-only changes — it was that bumping off PyPI 0.32.x surfaced the whole accumulated <code>dev</code> delta (notably the unrelated <code>PipelexRunner → PipelexMTHDSProtocol</code> runner rename). The keyword-only-specific call-site fixes were tiny.</p>--<table>- <thead><tr><th>Consumer</th><th>Branch</th><th>Keyword-only break?</th><th>Result</th></tr></thead>- <tbody>- <tr><td>pipelex-api</td><td><code>chore/keyword-only-pipelex-rev</code> · <a href="https://github.com/Pipelex/pipelex-api/pull/19">PR #19</a></td><td><b>Yes (1)</b> — <code>tests/unit/conftest.py</code></td><td class="pill good">green · pushed</td></tr>- <tr><td>cocode</td><td><code>chore/keyword-only-pipelex-rev</code> · <a href="https://github.com/Pipelex/cocode/pull/75">PR #75</a></td><td>No (all keyword already)</td><td class="pill good">green · pushed</td></tr>- <tr><td>pipelex-cookbook</td><td><code>chore/keyword-only-pipelex-rev</code> · <a href="https://github.com/Pipelex/pipelex-cookbook/pull/153">PR #153</a></td><td>No (earlier waves only)</td><td class="pill good">green · pushed</td></tr>- <tr><td>pipelex-starter-python</td><td><code>chore/keyword-only-pipelex-rev</code> · <a href="https://github.com/Pipelex/pipelex-starter-python/pull/55">PR #55</a></td><td>No</td><td class="pill good">green · pushed</td></tr>- <tr><td>pipelex-worker</td><td><code>chore/keyword-only-pipelex-rev</code> · <a href="https://github.com/Pipelex/pipelex-worker/pull/23">PR #23</a></td><td>No — no <code>pipelex</code> imports (thin worker); fixed a brittle ANSI assertion in the doctor smoke test</td><td class="pill good">green · pushed</td></tr>- <tr><td>pipelex-mistralai-workflows</td><td><code>feature/Mistral-native</code> (mistralai-2x hold pin)</td><td>Out of scope (own hold track)</td><td class="pill">n/a</td></tr>- </tbody>-</table>--<figure>-<div class="mermaid">-sequenceDiagram- participant PX as pipelex branch- participant API as pipelex-api- participant REST as cocode, cookbook, starter- Note over PX: 529b9082 -- initial keyword-only waves- PX->>API: pin to 529b9082, one parse_pipe_spec fix- PX->>REST: pin to 529b9082- Note over PX: 755b8211 -- parse_pipe_spec arg order reversed- PX->>API: re-pin to 755b8211, pipe_spec.py call-site fix- PX->>REST: re-pin to 755b8211, no positional call sites- Note over PX: 0e32c8c0 -- positional-subject consolidation- PX->>API: re-pin to 0e32c8c0, conftest integration_mode fix- PX->>REST: re-pin to 0e32c8c0, no-op bump, call sites already keyword-</div>-<figcaption>Each re-pin touched <code>pyproject.toml</code> <b>and</b> <code>uv.lock</code>, then <code>uv sync --all-extras</code> + <code>make agent-check</code> + <code>make agent-test</code> before committing.</figcaption>-</figure>--<h2>Minimal diffs for the critical changes</h2>-<div class="legend">- <span><i class="sw" style="background:var(--add-bg)"></i> added</span>- <span><i class="sw" style="background:var(--del-bg)"></i> removed</span>- <span><i class="sw" style="background:#0c0e14"></i> context / new file</span>-</div>--<div class="tabs" id="diff-tabs">- <div class="tabbar">- <button class="active" data-tab="d-make">Pipelex.make</button>- <button data-tab="d-flips">representative flips</button>- <button data-tab="d-guard">the guard (core)</button>- <button data-tab="d-hook">PostToolUse hook</button>- <button data-tab="d-down">downstream fix</button>- </div>-- <div class="tabpanel active" data-panel="d-make">- <div class="filechip">pipelex/pipelex.py — the public-API change that drove the one consumer break</div>- <pre class="diff"><span class="ln ctx"> def make(</span><span class="ln ctx"> cls,</span><span class="ln del">- integration_mode: IntegrationMode = IntegrationMode.PYTHON,</span><span class="ln ctx"> *,</span><span class="ln add">+ integration_mode: IntegrationMode = IntegrationMode.PYTHON,</span><span class="ln ctx"> needs_inference: bool = True,</span><span class="ln ctx"> temporal_enabled: bool | None = None,</span><span class="ln ctx"> ...</span></pre>- <p class="note">The <code>*</code> moved up one line — that's the whole change. <code>Pipelex.make(IntegrationMode.PYTHON)</code> no longer type-checks; <code>Pipelex.make(integration_mode=...)</code> does. Same shape applied to <code>Pipelex.setup()</code> and <code>PipelexHub.setup_config()</code>.</p>- </div>-- <div class="tabpanel" data-panel="d-flips">- <div class="filechip">representative one-line flips — move <code>*</code> before the subject, nothing else</div>- <pre class="diff"><span class="ln hdr"># pipelex/cogt/usage/costs_per_token.py — lookup table was positional</span><span class="ln del">-def model_cost_per_token(costs: CostsByCategoryDict, *, cost_category: CostCategory) -> float:</span><span class="ln add">+def model_cost_per_token(*, costs: CostsByCategoryDict, cost_category: CostCategory) -> float:</span><span class="ln ctx"> </span><span class="ln hdr"># pipelex/core/stuffs/stuff_content.py — both args are render options, self is the subject</span><span class="ln del">- def rendered_markdown(self, level: int = 1, *, is_pretty: bool = False) -> str:</span><span class="ln add">+ def rendered_markdown(self, *, level: int = 1, is_pretty: bool = False) -> str:</span><span class="ln ctx"> </span><span class="ln hdr"># pipelex/libraries/concept/concept_library.py — alias is a namespacing key, concept is what's added</span><span class="ln del">- def add_dependency_concept(self, alias: str, *, concept: Concept) -> None:</span><span class="ln add">+ def add_dependency_concept(self, *, alias: str, concept: Concept) -> None:</span></pre>- <p class="note">All 59 suspects were resolved exactly like this — relocate the bare <code>*</code>, no reorder, no rename. Internal call sites that passed the subject positionally were converted to keyword in the same commit.</p>- </div>-- <div class="tabpanel" data-panel="d-guard">- <div class="filechip">pipelex/cli/dev_cli/commands/keyword_only_guard.py (new · stdlib-only) — the rule</div>- <pre class="plain">def _positional_or_keyword_count(node, *, is_method: bool) -> int:- """Count positional-or-keyword params after dropping self/cls and the allowed subject."""- params = [*node.args.posonlyargs, *node.args.args]- if is_method and params and params[0].arg in {"self", "cls"}:- params = params[1:]- params = params[1:] # drop the single allowed subject (Exception 1)- return len(params) # *args/**kwargs and kw-only params are not counted--def _evaluate_def(node, *, module_qname, relative_path, class_stack,- func_stack, is_method, source_lines) -> Violation | None:- # carve-outs first — any match => skip entirely- if _is_dunder(node.name): return None- if _def_line_has_escape_hatch(node, source_lines=source_lines): return None- if _has_carveout_decorator(node): return None # pydantic / Typer / Temporal / pytest / Jinja2 / @override- if _has_typer_param_annotation(node): return None # decorator-less Typer entrypoints- if (dotted_qname, relative_path) in SYMMETRIC_ALLOWLIST: return None-- # the rule: a bare * is required once any non-subject positional-or-keyword param remains- if _positional_or_keyword_count(node, is_method=is_method) >= 1:- return Violation(relative_path=relative_path, qualified_name=local_qname, lineno=node.lineno)- return None</pre>- <p class="note">A <code>_Collector(ast.NodeVisitor)</code> walks the module tracking the class/function scope; <code>find_violations_in_source()</code> is the shared core both entrypoints call. A mid-edit <code>SyntaxError</code> is skipped, never reported — the hook must not block on a transient parse error.</p>- </div>-- <div class="tabpanel" data-panel="d-hook">- <div class="filechip">.claude/hooks/check-keyword-only.sh (new) — the in-loop gate</div>- <pre class="plain"># PostToolUse hook: run the stdlib-only AST guard on the just-edited file (~40ms).-PROJECT_DIR="${CLAUDE_PROJECT_DIR:-$PWD}"-GUARD="$PROJECT_DIR/pipelex/cli/dev_cli/commands/keyword_only_guard.py"-PYTHON="$PROJECT_DIR/.venv/bin/python"--input=$(cat) # tool-call JSON on stdin (pure bash, no jq — house style)-file_path=$(echo "$input" | grep -o '"file_path"[[:space:]]*:[[:space:]]*"[^"]*"' \- | head -1 | sed 's/.*"file_path"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/' || true)--[ -z "$file_path" ] && exit 0 # fast, silent pass-through ...-case "$file_path" in *.py) ;; *) exit 0 ;; esac # ... non-.py rejected in bash, no Python launch-[ -x "$PYTHON" ] || exit 0-[ -f "$GUARD" ] || exit 0--cd "$PROJECT_DIR" # invoked by FILE PATH (not -m) → skips the pipelex import chain-exec "$PYTHON" "$GUARD" "$file_path" # exit 2 + stderr → Claude Code surfaces it as blocking feedback</pre>- <p class="note">Running by file path instead of <code>python -m</code> is deliberate: <code>-m</code> would import <code>pipelex/__init__.py → rich</code> and blow the cold-start budget.</p>- </div>-- <div class="tabpanel" data-panel="d-down">- <div class="filechip">pipelex-api — the only real downstream call-site break + the lockstep pin bump</div>- <pre class="diff"><span class="ln hdr"># tests/unit/conftest.py — integration_mode was passed positionally</span><span class="ln ctx"> pipelex_instance = Pipelex.make(</span><span class="ln del">- IntegrationMode.PYTEST,</span><span class="ln add">+ integration_mode=IntegrationMode.PYTEST,</span><span class="ln ctx"> needs_inference=needs_inference_in_pipelex(request),</span><span class="ln ctx"> temporal_enabled=False,</span><span class="ln ctx"> )</span><span class="ln ctx"> </span><span class="ln hdr"># pyproject.toml (mirrored in uv.lock) — re-pin to the consolidation SHA</span><span class="ln ctx"> [tool.uv.sources]</span><span class="ln del">-pipelex = { git = "...pipelex.git", rev = "755b82117fd4c26ba090455cfd571099cf05708b" }</span><span class="ln add">+pipelex = { git = "...pipelex.git", rev = "0e32c8c02600e17bbe146331f9b701ccc03a1ca1" }</span></pre>- <p class="note">This call was <i>compliant</i> under the earlier SHA (the subject was positional-allowed) and only broke once the consolidation made <code>Pipelex.make</code> fully keyword-only — a textbook example of why the rule pays off: insert/reorder safety. cocode / cookbook / starter-python needed only the rev bump.</p>- </div>-</div>--<h2>Status & the one open thread</h2>-<div class="grid">- <div class="card">- <h3>Done <span class="pill good">landed on branch</span></h3>- <p>Tree fully compliant; guard hard-blocks in <code>agent-check</code> + CI; convention promoted to the contributor docs (in the mkdocs nav); a standing rule lives in the kit source that regenerates <code>CLAUDE.md</code>/<code>AGENTS.md</code>. All four consumer PRs pushed and green.</p>- </div>- <div class="card">- <h3>Open <span class="pill warn">at release</span></h3>- <p>When the refactor lands on PyPI, swap each consumer's temporary <code>[tool.uv.sources]</code> git pin back to the released <code>pipelex</code> version (<code>==</code>/<code>>=</code>) and confirm CI stays green. Tracked in the downstream-consumer-breakage record's checklist.</p>- </div>-</div>--<footer>- Branch <code>refactor/Function-calling-4</code>. Consumer PRs: pipelex-api #19, cocode #75, pipelex-cookbook #153, pipelex-starter-python #55, pipelex-worker #23. The full convention spec, the resolved suspects list, and the raw audit live alongside this file in the <code>keyword-only-arguments/</code> track.-</footer>--</div>--<script>- mermaid.initialize({- startOnLoad: true,- theme: "dark",- themeVariables: {- background: "#171a23", primaryColor: "#1d212c", primaryBorderColor: "#2c3650",- primaryTextColor: "#e6e9ef", lineColor: "#6ea8fe", fontSize: "14px"- },- flowchart: { curve: "basis" }- });- // lightweight tabs- document.querySelectorAll(".tabs").forEach(function (tabs) {- var buttons = tabs.querySelectorAll(".tabbar button");- var panels = tabs.querySelectorAll(".tabpanel");- buttons.forEach(function (btn) {- btn.addEventListener("click", function () {- var key = btn.getAttribute("data-tab");- buttons.forEach(function (b) { b.classList.toggle("active", b === btn); });- panels.forEach(function (p) { p.classList.toggle("active", p.getAttribute("data-panel") === key); });- });- });- });-</script>-</body>-</html>diff --git a/wip/history/keyword-only-arguments/positional-subject-suspects.md b/wip/history/keyword-only-arguments/positional-subject-suspects.mddeleted file mode 100644index ffd5d87f0..000000000--- a/wip/history/keyword-only-arguments/positional-subject-suspects.md+++ /dev/null@@ -1,280 +0,0 @@-# Keyword-only refactor — positional-subject ABUSE suspects (consolidated)--> **RESOLVED (2026-06-15).** All 59 suspects — high-confidence, med/low, the public-API pair (`Pipelex.make`/`setup`), and the "decide as a family" clusters (`pipe_controllers` `_*_controller_pipe` family, the eight `graph` `GraphTracerManager` `lookup_key` dispatch methods) — were applied as **fully keyword-only** by relocating the bare `*` to immediately after `self`/`cls` (no reordering). Sibling/override sets were flipped consistently (e.g. `StuffContent.rendered_markdown` base + all subclass overrides; `PipeController._*_controller_pipe` base + overrides). Two convention boundaries were deliberately **left untouched** as out-of-scope: the broader `PipeAbstract._live_run_pipe`/`_dry_run_pipe` `job_metadata`-first convention (not among the suspects), and `GraphTracerProtocol`/`GraphTracer` (`node_id`-first, a different class). Positional call sites — production (`plugins/*_config.py` → `get_reasoning_level_str`, `check_rules_sync_cmd.py` → `build_merged_rules`) and tests — were converted to keyword; temporal `test_worker_cli.py` mock assertions were rewritten from `.call_args.args[0]` to `.call_args.kwargs[...]`. `make agent-check` (ruff, pyright 0 errors, mypy clean, `check-keyword-only` PASSED) and `make agent-test` both green. Public-API break (`Pipelex.make`/`setup` now keyword-only after the subject) is folded into the already-breaking refactor branch; downstream impact tracked in [`downstream-consumer-breakage.md`](downstream-consumer-breakage.md).--Consolidated shortlist from the positional-subject audit ([`audit/positional-subject-audit.md`](audit/positional-subject-audit.md)). The question for every entry: *was the first arg genuinely the semantic object of the call, or was it kept positional merely to satisfy Exception 1 ("the subject may stay positional"), when the call would actually read better with that arg keyworded too?*--## How this was produced--- **Population reviewed:** 1140 of the 1806 audited functions — the 797 `SUBJECT_THEN_KEYWORDS` entries (`def f(subject, *, …)`, where the subject choice was actually made) plus 343 **primitive-typed** lone subjects (`def f(x: bool/int/str/float)`, which read opaquely at the call site). The remaining ~660 non-primitive lone subjects and the symmetric-allowlist/variadic entries were filtered out mechanically (no LLM) — a single positional param with a domain type offers no subject to abuse.-- **Method:** one **Sonnet 4.6** sub-agent per package (25 agents), each with Read+Grep, instructed to open the function body and a call site or two whenever the signature alone was ambiguous, to **default to "fine" and emit only suspects**, and to cite the rubric in `docs/contribute/keyword-only-arguments.md`. Per-package shortlists live in [`audit/findings/<package>.md`](audit/findings/).-- **Calibration was healthy:** every package came back with a short list (0–5 suspects); no agent rubber-stamped or over-flagged.--## Summary--- **Total reviewed:** 1140 entries (797 Section A + 343 primitive lone-subjects).-- **Total suspects: 59** — **31 high-confidence**, 28 medium/low.-- **Zero-suspect packages:** `base_exceptions`, `errors`, `hub`, `language`, `pipe_signature`, `tracing`.-- The suggested fix is **almost always the same**: move the `*` before the subject too (make the signature *fully* keyword-only). A handful suggest *reordering* so a different arg becomes the positional subject.--| Package | High | Med/Low | Total |-| --- | ---: | ---: | ---: |-| `temporal` | 4 | 1 | 5 |-| `pipe_controllers` | 3 | 2 | 5 |-| `builder` | 3 | 1 | 4 |-| `cli` | 3 | 1 | 4 |-| `tools` | 3 | 1 | 4 |-| `pipe_operators` | 2 | 2 | 4 |-| `plugins` | 1 | 3 | 4 |-| `system` | 1 | 3 | 4 |-| `libraries` | 1 | 3 | 4 |-| `cogt` | 2 | 1 | 3 |-| `core` | 2 | 1 | 3 |-| `pipe_run` | 2 | 1 | 3 |-| `pipelex` | 1 | 2 | 3 |-| `observer` | 2 | 0 | 2 |-| `kit` | 1 | 1 | 2 |-| `graph` | 0 | 1 | 1 |-| `pipeline` | 0 | 1 | 1 |-| `reporting` | 0 | 1 | 1 |-| `runtime_bridge` | 0 | 1 | 1 |-| **Total** | **31** | **28** | **59** |--## Cross-cutting patterns (for adjudication)--The 59 suspects collapse into a small number of recurring shapes. A near-universal tell: **the existing call sites already pass the "positional" arg by keyword**, which means the positional permission is buying nothing today — exactly the signal that Exception 1 was leaned on mechanically.--1. **Lookup-table / registry / config carried positional while the real query is keyworded.** The strongest cluster. The positional arg is a deck/map/dict you read *out of*, and the keyword arg is what you're looking up.- - `builder` `_resolve_preset_backend(model_deck, *, model_handle, …)` · `cogt` `model_cost_per_token(costs, *, cost_category)` · `cogt` `get_reasoning_level_str(effort_to_level_map, *, effort)` · `tools` `apply_tag_style(context, *, value, …)` · `kit` `_read_agent_file(agents_dir, *, name)`.--2. **`*_for_pipe` / `*_from_response` / `*_from_uri` — the name advertises the operand, but a *different* arg is positional.** The function name itself says which arg is the object, and it isn't the positional one.- - `builder` `build_output_for_pipe(mthds_contents, *, pipe_code, …)` · `builder` `build_runner_code_for_pipe(mthds_contents, *, pipe_code)` · `plugins` `make_extract_output_from_response(inference_model, *, response)` · `plugins` `make_mistral_document_url_chunk_from_uri(mistral_client, *, uri)` · `tools` `detect_jinja2_required_variables(template_category, *, template_source)` (+ `detect_jinja2_variable_references`).--3. **Mode flag / bare bool carried positional** — `f(True)` reveals nothing, and every call site already keywords it.- - `cli` `build_run_output(with_memory, *, …)` · `cli` `determine_needs(reset, *, …)` · `temporal` `run_worker(is_not_sandboxed, *, …)` · `system` `update_service_terms_acceptance(accepted, …)` · `system` `update_inference_setup_completed(completed, …)` · `system` `check_is_initialized(print_warning_if_not=…)` · `core` `rendered_markdown(level=1, *, is_pretty=…)` (+ async twin).--4. **`task_queue` positional on Temporal factories** — a defaulted config option, not "the thing being made". Three parallel sites.- - `temporal` `create_executor` · `make_temporal_pipe_router` · `make_temporal_pipe_run`.--5. **Trace/provenance context carrier positional** (`job_metadata`, `lookup_key`, `calling_pipe_code`) while the function acts on the stuffs/outputs in the keyword args.- - `pipe_controllers` `_register_branch_outputs_with_graph_tracer`, `_register_parallel_combine_with_graph_tracer`, `sub_pipe.run_pipe(calling_pipe_code, …)` · `graph` `on_pipe_end_success(lookup_key, …)` and 7 sibling dispatch methods.--6. **Accumulator/sink dict carried positional** while the content being added is keyworded.- - `pipe_run` `_try_add_rendered_file(files, *, filename, …)` · `_add_optional_text_file(files, *, …)`.--7. **Symmetric / directional pair arbitrarily split across the `*`** — neither operand is "the subject", so a positional+keyword split is just noise.- - `pipe_operators` `_expects_type(expected_type, *, target_type)` · `plugins` `make_nb_tokens_by_category_from_nb(nb_input, *, nb_output)` · `libraries` `compute_fingerprint_from_content(concepts, *, pipes)` · `pipe_run` `output_multiplicity_to_apply(base, *, override)` · `pipeline` `copy_with_update(otel_context, *, trace_context, …)`.--8. **Public-API surface** — already known to be keyword-breaking from the refactor; worth a deliberate call.- - `pipelex` `Pipelex.make(integration_mode=…, *, …)` and `Pipelex.setup(integration_mode, *, …)`.--**Note on conventions:** several med/low entries are deliberate codebase-wide conventions (e.g. `job_metadata` as the first param on the whole run-pipe family in `pipe_controllers`; the `lookup_key` dispatch discriminant across the eight `GraphTracerManager` methods). Those should be decided *as a family* — fix the base/convention or leave it, don't pick off one site.-------## Suspects by package (ordered by confidence)--### `temporal` (4 high · 1 med/low)--**High**--- `pipelex/temporal/task_manager.py:36` — `TaskManager.run_worker` — `async def run_worker(self, is_not_sandboxed: bool, *, is_unit_testing: bool, ...)` — `is_not_sandboxed` is a boolean flag that reads opaquely positionally: `run_worker(True)` reveals nothing. It's not the semantic object the method acts on; the worker it launches is. Both real call sites already pass it as `is_not_sandboxed=...`. The `temporal_task_manager.py` override (which carries `@override` and is thus skip-exempt) would track automatically once the Protocol base is fixed. Suggested fix: make the signature fully keyword-only — `async def run_worker(self, *, is_not_sandboxed: bool, is_unit_testing: bool, ...)`.-- `pipelex/temporal/tprl/workflow_caller.py:298` — `WorkflowExecutorFactory.create_executor` — `def create_executor(cls, task_queue: str | None = None, *, workflow_execution_timeout, retry_policy, ...)` — `task_queue` is one of many configuration options for the executor being created; `create_executor` doesn't designate it as the subject (no "for this queue" semantic). All four existing call sites pass it as `task_queue=...`. Suggested fix: move `task_queue` after `*`.-- `pipelex/temporal/tprl_pipe/temporal_pipe_router.py:127` — `make_temporal_pipe_router` — `def make_temporal_pipe_router(task_queue: str | None = None, *, workflow_execution_timeout, retry_policy, ...)` — same pattern as `create_executor`: `task_queue` is a config option defaulting to `None`, not the semantic object of a "make router" call. Suggested fix: `def make_temporal_pipe_router(*, task_queue: str | None = None, ...)`.-- `pipelex/temporal/tprl_pipe/temporal_pipe_run.py:131` — `make_temporal_pipe_run` — `def make_temporal_pipe_run(task_queue: str | None = None, *, workflow_execution_timeout, retry_policy, ...)` — same pattern as `make_temporal_pipe_router`. Suggested fix: `def make_temporal_pipe_run(*, task_queue: str | None = None, ...)`.--**Medium / low**--- `pipelex/temporal/worker_cli.py:25` — `run_worker` — `async def run_worker(project: str | None = None, *, is_not_sandboxed: bool, is_unit_testing: bool, ...)` — `project` defaults to `None` and is looked up from pyproject.toml when absent; its role is closer to a context/config override than the semantic subject of "run a worker". The one real call site passes it positionally from the Typer command, but a fully keyword-only form would be equally natural. Medium confidence — `project` does describe which project's worker to run, so the positional reading is defensible.--### `pipe_controllers` (3 high · 2 med/low)--**High**--- `pipelex/pipe_controllers/parallel/pipe_parallel.py:326` — `PipeParallel._register_branch_outputs_with_graph_tracer` — `def _register_branch_outputs_with_graph_tracer(self, job_metadata: JobMetadata, *, output_stuffs: dict[str, 'Stuff']) -> None` — `job_metadata` is a trace-context carrier / lookup key, not the semantic object; the function acts on `output_stuffs`. Both call sites (210, 309) already pass `job_metadata=job_metadata`. Suggested fix: make fully keyword-only.-- `pipelex/pipe_controllers/parallel/pipe_parallel.py:369` — `PipeParallel._register_parallel_combine_with_graph_tracer` — `def _register_parallel_combine_with_graph_tracer(self, job_metadata: JobMetadata, *, combined_stuff: 'Stuff', branch_stuffs: dict[str, 'Stuff']) -> None` — same pattern: `job_metadata` is a context/trace carrier; the function acts on `combined_stuff` and `branch_stuffs`. Both call sites (203, 302) use `job_metadata=job_metadata`. Suggested fix: move `*` before `job_metadata`.-- `pipelex/pipe_controllers/sub_pipe.py:34` — `SubPipe.run_pipe` — `async def run_pipe(self, calling_pipe_code: str, *, working_memory: WorkingMemory, job_metadata: JobMetadata, sub_pipe_run_params: PipeRunParams, library_crate: 'LibraryCrate | None'=None) -> PipeOutput` — `calling_pipe_code` is caller provenance/context, not the semantic object; `self` (the SubPipe) is the object being run. Call sites (pipe_parallel.py:152, pipe_sequence.py:195, pipe_condition.py:328) already pass `calling_pipe_code=self.code`. Suggested fix: make fully keyword-only.--**Medium / low**--- `pipelex/pipe_controllers/pipe_controller.py:71` — `PipeController._live_run_controller_pipe` — `async def _live_run_controller_pipe(self, job_metadata: JobMetadata, *, ...)` — `job_metadata` positional follows the codebase-wide `_live_run_pipe(self, job_metadata, *, ...)` convention on `PipeAbstract`. The single call site uses keyword form. Medium confidence only — changing the controller without changing the base class would create inconsistency. **Decide as a family** with the base run-pipe signatures.-- `pipelex/pipe_controllers/pipe_controller.py:83` — `PipeController._dry_run_controller_pipe` — same reasoning as `_live_run_controller_pipe`; treat as a pair.--### `builder` (3 high · 1 med/low)--**High**--- `pipelex/builder/operations/models_ops.py:36` — `_resolve_preset_backend` — `def _resolve_preset_backend(model_deck: ModelDeck, *, model_handle: str, model_type: ModelType) -> InferenceModelSpec | None` — Docstring: "Resolve a preset's model handle to an InferenceModelSpec" — the thing resolved is `model_handle`; `model_deck` is a lookup registry. Call sites confirm: `_resolve_preset_backend(model_deck, model_handle=setting.model, model_type=model_type)` — the deck reads as an unlabeled blob while the real target is buried in keyword args. Suggested fix: make fully keyword-only.-- `pipelex/builder/operations/output_ops.py:14` — `build_output_for_pipe` — `async def build_output_for_pipe(mthds_contents: list[str], *, pipe_code: str, output_format: ...) -> dict[str, Any]` — the name says "for_pipe", so the pipe is the subject; `mthds_contents` is raw bundle source material (context/input). `build_output_for_pipe(some_list, pipe_code="my.pipe")` makes the caller think twice about the first list. Suggested fix: make fully keyword-only, or reorder so `pipe_code` is the positional subject.-- `pipelex/builder/operations/runner_code_ops.py:17` — `build_runner_code_for_pipe` — `async def build_runner_code_for_pipe(mthds_contents: list[str], *, pipe_code: str) -> str` — same pattern as `build_output_for_pipe`: name says "for_pipe", the pipe is the object, but `mthds_contents` was made positional. Suggested fix: make fully keyword-only, or promote `pipe_code`.--**Medium / low**--- `pipelex/builder/operations/models_ops.py:101` — `_build_presets_for_category` (+ parallels `_build_aliases_for_category` :135, `_build_waterfalls_for_category` :160) — `def _build_presets_for_category(model_deck: ModelDeck, *, category: ModelCategory, backend: str | None)` — name says "for_category"; `category` drives behavior, `model_deck` is the data source. At call sites the deck reads as an unnamed blob. Lower confidence than `_resolve_preset_backend` because the deck is a richer object with a plausible claim to being the queried subject. Suggested fix: make fully keyword-only (all three).--### `cli` (3 high · 1 med/low)--**High**--- `pipelex/cli/cli_factory.py:31` — `make_pipelex_for_cli` — `def make_pipelex_for_cli(context: ErrorContext, *, library_dirs: ..., needs_inference: bool, ...) -> Pipelex` — `context` is error-reporting metadata (an `ErrorContext` enum that shapes error messages), not the object being created. The function builds a `Pipelex`; `context` is a side-channel tag. Every call site uses `context=...`. Suggested fix: make fully keyword-only.-- `pipelex/cli/agent_cli/commands/run/_output_helpers.py:14` — `build_run_output` — `def build_run_output(with_memory: bool, *, main_stuff_json: dict, working_memory_dump: dict, compact_result: ..., extra_metadata: ...) -> dict` — `with_memory: bool` is a mode flag, not the object being assembled; `main_stuff_json` is the more central data. Both call sites pass `with_memory=with_memory`. Suggested fix: move `*` before `with_memory`.-- `pipelex/cli/commands/init/command.py:182` — `determine_needs` — `def determine_needs(reset: bool, *, check_config: bool, check_inference: bool, ...) -> tuple[bool, bool, bool, bool]` — `reset: bool` is one mode flag among several others; not more "the subject" than `check_config`/`check_inference`. The single call site passes `reset=reset`. Suggested fix: move `*` before `reset` (consistent with the other flags).--**Medium / low**--- `pipelex/cli/agent_cli/commands/init_cmd.py:173` — `_configure_backends` — `def _configure_backends(config: dict[str, Any], *, backends_toml_path: Path, template_backends_path: Path) -> list[str]` — the primary action modifies `backends_toml_path`; `config: dict` is read-only context. The call site passes `config` positionally, so there's no immediate caller-side benefit. Private helper, contained impact, and `config` as "data being acted upon" is defensible. Low confidence.--### `tools` (3 high · 1 med/low)--**High**--- `pipelex/tools/jinja2/jinja2_required_variables.py:121` — `detect_jinja2_required_variables` — `def detect_jinja2_required_variables(template_category: TemplateCategory, *, template_source: str) -> set[str]` — `template_source` is the object analyzed; `template_category` is a configuration/mode parameter. All call sites pass `template_category=` as keyword, so the positional exemption buries the real subject keyword-only. Suggested fix: swap order — `detect_jinja2_required_variables(template_source: str, *, template_category: TemplateCategory)`.-- `pipelex/tools/jinja2/jinja2_required_variables.py:255` — `detect_jinja2_variable_references` — `def detect_jinja2_variable_references(template_category: TemplateCategory, *, template_source: str) -> list[VariableReference]` — same pattern: source is the object, category is config; all call sites keyword the category. Suggested fix: swap order so `template_source` is the positional subject.-- `pipelex/tools/jinja2/jinja2_filters.py:110` — `apply_tag_style` — `def apply_tag_style(context: Context, *, value: str, tag_name: str | None=None) -> str` — `context` is a Jinja2 `Context` used only to retrieve `TAG_STYLE`; `value` is the string being wrapped. Single call site passes `context` positionally (framework-driven), but conceptually the positional arg is a lookup registry, not the acted-on object. Suggested fix: fully keyword-only. *(Caveat: confirm the Jinja2 invocation path — if a `@pass_context` filter calls it positionally, the framework owns the convention.)*--**Medium / low**--- `pipelex/tools/jinja2/jinja2_environment.py:9` — `make_jinja2_env_from_loader` — `def make_jinja2_env_from_loader(template_category: TemplateCategory, *, loader: BaseLoader, enable_async: bool=True) -> Environment` — the name says "from loader", pointing to `loader` as the operative arg; `template_category` is config. Minor — `template_category` plausibly drives env construction. Suggested fix: reorder to `loader` positional, or fully keyword-only.--### `pipe_operators` (2 high · 2 med/low)--**High**--- `pipelex/pipe_operators/img_gen/img_gen_prompt_blueprint.py:269` — `ImgGenPromptBlueprint._render_text` — `async def _render_text(self, context_provider: ContextProviderAbstract, *, template_blueprint: TemplateBlueprint, ...)` — `context_provider` is a lookup environment/registry, not the thing rendered; `template_blueprint` is the real subject. Call sites always pass `context_provider=`. Suggested fix: make fully keyword-only.-- `pipelex/pipe_operators/llm/llm_prompt_blueprint.py:350` — `LLMPromptBlueprint._unravel_text` — `async def _unravel_text(self, context_provider: ContextProviderAbstract, *, jinja2_blueprint: TemplateBlueprint, ...)` — identical to `_render_text`: `context_provider` is the resolution environment, `jinja2_blueprint` is what gets unraveled. Call sites use `context_provider=`. Suggested fix: make fully keyword-only.--**Medium / low**--- `pipelex/pipe_operators/compose/structured_content_composer.py:392` — `StructuredContentComposer._expects_type` — `def _expects_type(self, expected_type: type[Any], *, target_type: type)` — symmetric type-comparison pair (`issubclass(expected_type, target_type)`); neither arg is more "subject". Call sites pass `expected_type=` as keyword. Suggested fix: make fully keyword-only.-- `pipelex/pipe_operators/img_gen/img_gen_prompt_blueprint.py:65` — `ImgGenPromptBlueprint.make_img_gen_prompt` — `async def make_img_gen_prompt(self, context_provider: ContextProviderAbstract, *, extra_params, max_prompt_images)` — `context_provider` is a resolution environment, not what's generated (the prompt is built from `self`). Lower confidence because "context to use for building" is a defensible subject role. Suggested fix: make fully keyword-only.--### `plugins` (1 high · 3 med/low)--**High**--- `pipelex/plugins/gateway/gateway_completions_factory.py:138` — `GatewayCompletionsFactory.make_extract_output_from_response` — `def make_extract_output_from_response(cls, inference_model: InferenceModelSpec, *, response: GenericResponse) -> ExtractOutput` — name is "from_response", so `response` is the primary operand; `inference_model` is only a lookup key selecting the extraction sub-method (`GatewayExtractProtocol.make_from_model_handle`). Suggested fix: make fully keyword-only (or reorder so `response` is the subject).--**Medium / low**--- `pipelex/plugins/anthropic/anthropic_factory.py:244` — `AnthropicFactory.make_nb_tokens_by_category_from_nb` — `def make_nb_tokens_by_category_from_nb(nb_input: int, *, nb_output: int) -> NbTokensByCategoryDict` — symmetric directional pair (`nb_input`/`nb_output`); splitting across `*` is arbitrary. No external call sites found, so risk is low. Suggested fix: make fully keyword-only.-- `pipelex/plugins/mistral/mistral_factory.py:365` — `MistralFactory.make_mistral_document_url_chunk_from_uri` — `async def make_mistral_document_url_chunk_from_uri(cls, mistral_client: Mistral, *, uri: str) -> DocumentURLChunkTypedDict` — name ends "from_uri", so `uri` is the operand; `mistral_client` is a dependency (the docstring calls it "required for local file uploads"). Suggested fix: make fully keyword-only.-- `pipelex/plugins/plugin_sdk_registry.py:22` — `PluginSdkRegistry.set_sdk_instance` — `def set_sdk_instance(self, plugin: Plugin, *, sdk_instance: Any) -> Any` — registry keyed by `plugin.sdk_handle`; `plugin` acts as an index, `sdk_instance` is what's stored. "Set X instance" implies `sdk_instance` is the subject. Call sites already keyword both. Suggested fix: make fully keyword-only.--### `system` (1 high · 3 med/low)--**High**--- `pipelex/system/pipelex_service/gateway_config_merger.py:69` — `GatewayConfigMerger._apply_overrides_to_model` — `def _apply_overrides_to_model(cls, model_name: str, *, gateway_model_specs: BackendModelSpecs, local_model_config: BackendModelSpecs) -> None` — `model_name` is a string used only in log/warning messages; the real operands are the two keyword specs. The sole call site passes `model_name=model_name`. Suggested fix: make fully keyword-only.--**Medium / low**--- `pipelex/system/pipelex_service/pipelex_service_agreement.py:29` — `update_service_terms_acceptance` — `def update_service_terms_acceptance(accepted: bool, *, config_dir: Path | None=None) -> None` — bare `bool` positional; `update_service_terms_acceptance(True)` is opaque. All call sites use `accepted=...`. Suggested fix: make fully keyword-only.-- `pipelex/system/pipelex_service/pipelex_service_agreement.py:53` — `update_inference_setup_completed` — `def update_inference_setup_completed(completed: bool, *, config_dir: Path | None=None) -> None` — same pattern; always called as `completed=True`. Suggested fix: make fully keyword-only.-- `pipelex/system/configuration/config_check.py:12` — `check_is_initialized` — `def check_is_initialized(print_warning_if_not: bool=True) -> bool` — (primitive lone-subject) single `bool` with a descriptive name; `check_is_initialized(False)` would be opaque. Callers always keyword it or use the default. Low priority. Suggested fix: `def check_is_initialized(*, print_warning_if_not: bool=True)`.--### `libraries` (1 high · 3 med/low)--**High**--- `pipelex/libraries/libraries/library_crate.py:39` — `LibraryCrate.compute_fingerprint_from_content` — `def compute_fingerprint_from_content(concepts: dict[str, 'ConceptBlueprint | str'], *, pipes: dict[str, PipeBlueprintUnion]) -> str` — `concepts` and `pipes` are co-equal operands (both dict inputs to a hash); neither is the object. Call sites keyword both. Suggested fix: make fully keyword-only.--**Medium / low**--- `pipelex/libraries/concept/concept_library.py:191` — `ConceptLibrary.add_dependency_concept` — `def add_dependency_concept(self, alias: str, *, concept: Concept) -> None` — `alias` is a namespace/prefix used to build a composite key; `concept` is what's actually added. Call sites use `alias=alias`. Suggested fix: make `alias` keyword-only too, or reorder to `(self, concept, *, alias)`.-- `pipelex/libraries/pipe/pipe_library.py:97` — `PipeLibrary.add_dependency_pipe` — `def add_dependency_pipe(self, alias: str, *, pipe: PipeAbstract) -> None` — identical pattern to `add_dependency_concept`: `alias` is the namespacing key, `pipe` is the subject. Suggested fix: same.-- `pipelex/libraries/visibility_utils.py:35` — `make_visibility_checker` (+ `check_visibility_for_blueprints` :53) — `def make_visibility_checker(manifest: MethodsManifest | None, *, blueprints: list[PipelexBundleBlueprint]) -> PackageVisibilityChecker` — `manifest` is nullable config/filter (often `None`); `blueprints` is the primary data inspected. Call sites pass `manifest=manifest`. Low confidence — `manifest` could be argued as the filter subject. Suggested fix: make both keyword-only (both functions).--### `cogt` (2 high · 1 med/low)--**High**--- `pipelex/cogt/usage/costs_per_token.py:4` — `model_cost_per_token` — `def model_cost_per_token(costs: CostsByCategoryDict, *, cost_category: CostCategory) -> float` — `costs` is a dict used purely as a lookup table; the function dispatches on `cost_category` (`costs.get(cost_category) / 1_000_000`). The real subject is `cost_category`. All three call sites use `costs=...`. Suggested fix: make fully keyword-only, or reorder with `cost_category` first.-- `pipelex/cogt/llm/reasoning_config_base.py:40` — `get_reasoning_level_str` — `def get_reasoning_level_str(effort_to_level_map: EffortToLevelMap, *, effort: ReasoningEffort) -> str | None` — body is `effort_to_level_map.get(effort)`; the map is a lookup table, `effort` is the query. All four call sites pass the map positionally as `self.effort_to_level_map`, reading opaquely. Suggested fix: make fully keyword-only.--**Medium / low**--- `pipelex/cogt/model_backends/backend_credentials.py:26` — `BackendCredentialsErrorMsgFactory.make_one_variable_missing_error_msg` — `def make_one_variable_missing_error_msg(cls, secrets_provider: SecretsProviderAbstract, *, backend_name: str | None, var_name: str) -> str` — `secrets_provider` is a dispatch context (`isinstance`-branches to pick wording); `backend_name`/`var_name` are the real domain objects. Sole call site uses `secrets_provider=`. Lower confidence because the provider does control branching. Low urgency.--### `core` (2 high · 1 med/low)--**High**--- `pipelex/core/stuffs/stuff_content.py:64` — `StuffContent.rendered_markdown` — `def rendered_markdown(self, level: int=1, *, is_pretty: bool=False) -> str` — `level` and `is_pretty` are both optional rendering options; neither is the subject (`self` is). All call sites use keyword form or the no-arg default. `rendered_markdown(2)` is opaque. Suggested fix: move `*` before `level`.-- `pipelex/core/stuffs/stuff_content.py:113` — `StuffContent.rendered_markdown_async` — `async def rendered_markdown_async(self, level: int=1, *, is_pretty: bool=False) -> str` — same issue as `rendered_markdown`; parallel async signature. Suggested fix: move `*` before `level`.--**Medium / low**--- `pipelex/core/concepts/concept.py:194` — `Concept.render_concept_representation` — `def render_concept_representation(self, output_format: ConceptRepresentationFormat, *, is_multiple: bool=False) -> tuple[dict[str, Any], set[str]]` — `self` (Concept) is the real subject; `output_format` and `is_multiple` are equal-standing rendering options. All call sites keyword both. Suggested fix: move `*` before `output_format`.--### `pipe_run` (2 high · 1 med/low)--**High**--- `pipelex/pipe_run/delivery_executor.py:201` — `DeliveryExecutor._try_add_rendered_file` — `async def _try_add_rendered_file(cls, files: dict[str, ResultFile], *, filename: str, render: Awaitable[str], content_type: str) -> None` — `files` is a mutable accumulator/sink, not the object; the real subjects of the "add" are `filename`/`render`/`content_type`. Call site: `self._try_add_rendered_file(files, filename="main_stuff.json", ...)`. Suggested fix: make fully keyword-only.-- `pipelex/pipe_run/delivery_executor.py:219` — `DeliveryExecutor._add_optional_text_file` — `def _add_optional_text_file(cls, files: dict[str, ResultFile], *, filename: str, text: str | None, content_type: str) -> None` — same pattern: `files` is the accumulator carried positionally, `filename`/`text`/`content_type` are the content added. Suggested fix: make fully keyword-only.--**Medium / low**--- `pipelex/pipe_run/pipe_run_params.py:28` — `output_multiplicity_to_apply` — `def output_multiplicity_to_apply(base_multiplicity: VariableMultiplicity | None, *, override_multiplicity: VariableMultiplicity | None) -> VariableMultiplicityResolution` — two-operand priority merge; neither input is clearly "the object". Docstring examples (`output_multiplicity_to_apply(None, None)`) show opaque positional calling. Suggested fix: make fully keyword-only.--### `pipelex` (1 high · 2 med/low)--**High**--- `pipelex/pipelex.py:512` — `Pipelex.make` — `def make(cls, integration_mode: IntegrationMode = IntegrationMode.PYTHON, *, ...)` — `integration_mode` is a mode/config option, not the subject of the factory (the subject is the instance created). All call sites omit it or pass `integration_mode=...`. `Pipelex.make(IntegrationMode.CLI)` reads opaquely. **Primary public API entry point** — keyword-only is the right default. Suggested fix: `def make(cls, *, integration_mode: IntegrationMode = IntegrationMode.PYTHON, ...)`.--**Medium / low**--- `pipelex/pipelex.py:165` — `Pipelex.setup` — `def setup(self, integration_mode: IntegrationMode, *, ...)` — same logic as `make`: `integration_mode` is a mode flag, not the subject (`self` is). The sole internal call site uses `integration_mode=integration_mode`. Public API surface (though `make` is the primary entry). Suggested fix: `def setup(self, *, integration_mode: IntegrationMode, ...)`.-- `pipelex/pipelex.py:145` — `Pipelex._get_validation_error_msg` — `def _get_validation_error_msg(component_name: str, *, validation_exc: Exception) -> str` — `component_name` is a descriptive label ("routing profile library", …), behaving like context alongside `validation_exc`. Call sites: `self._get_validation_error_msg("routing profile library", validation_exc=exc)` — the bare positional string reads opaquely. Private, low call-count, but inconsistent (`validation_exc` already keyword-only). Suggested fix: make fully keyword-only.--### `observer` (2 high · 0 med/low)--**High**--- `pipelex/observer/multi_observer.py:14` — `MultiObserver.remove_observer` — `def remove_observer(self, name: str) -> None` — asymmetric with `add_observer(*, name, observer)` which is fully keyword-only; `remove_observer("some_name")` passes a bare string with no call-site context. Convention consistency + readability → `def remove_observer(self, *, name: str)`. *(Primitive lone-subject — flagged for the asymmetry, not opacity alone.)*-- `pipelex/observer/local_observer.py:22` — `LocalObserver._write_to_jsonl` — `def _write_to_jsonl(self, event_type: str, *, payload: PayloadType) -> None` — `event_type` is a routing/selector string (picks the file + injects a key), not the object written; the payload is the real subject. Call sites pass a named constant so readability is acceptable, but the positional/keyword split is inconsistent (both are equally descriptive). Suggested fix: `def _write_to_jsonl(self, *, event_type, payload)`.--### `kit` (1 high · 1 med/low)--**High**--- `pipelex/kit/single_file_agent_rules.py:12` — `_read_agent_file` — `def _read_agent_file(agents_dir: Traversable, *, name: str) -> str` — `agents_dir` is a lookup-context directory; `name` is the file read — the real object. "read [name] from [agents_dir]" has the subject keyworded and the context positional. Call site (:79): `_read_agent_file(agents_dir, name=name)`. Suggested fix: make fully keyword-only.--**Medium / low**--- `pipelex/kit/single_file_agent_rules.py:49` — `build_merged_rules` — `def build_merged_rules(kit_index: KitIndex, *, agent_set: str | None=None, file_list: list[str] | None=None) -> str` — `kit_index` is a config/registry data source + lookup table, reading more like context than subject. Lower confidence — a `KitIndex` could reasonably be "the thing being merged from". Suggested fix: make fully keyword-only.--### `graph` (0 high · 1 med/low)--**Medium / low**--- `pipelex/graph/graph_tracer_manager.py:266` — `GraphTracerManager.on_pipe_end_success` (+ siblings `on_pipe_end_error` :325, `register_execution_data` :304, `add_edge` :360, `register_controller_output` :389, `register_batch_item_extraction` :411, `register_batch_aggregation` :440, `register_parallel_combine` :469) — `lookup_key` is a registry dispatch key routing to the right `GraphTracer` via `self._get_tracer(lookup_key)`; the semantic subject is `node_id` (in the keyword args). All call sites already pass `lookup_key=` as keyword, so no opacity exists today. The case for keyword-only: `lookup_key` is a dispatch concern, not the object, and the naming asymmetry with `GraphTracerProtocol` (which takes `node_id` positionally) is mildly confusing. Low confidence — `lookup_key` as first-positional is a coherent "which tracer" discriminant. **Decide as a family** across all eight manager dispatch methods.--### `pipeline` (0 high · 1 med/low)--**Medium / low**--- `pipelex/pipeline/job_metadata.py:89` — `JobMetadata.copy_with_update` — `def copy_with_update(self, otel_context: OtelContext | None, *, trace_context: TraceContext | None = None, **updates: Any)` — `otel_context` is not the subject (`self` is); it's one of two context fields being updated, asymmetrically split from `trace_context` across `*`. Call sites pass `otel_context=...`, and the docstring justifies the asymmetry semantically (different inheritance semantics), but from a signature-readability standpoint both context args are peers. Suggested fix: make fully keyword-only.--### `reporting` (0 high · 1 med/low)--**Medium / low**--- `pipelex/reporting/reporting_manager.py:251` — `ReportingManager._emit_best_effort` — `def _emit_best_effort(event_log: EventLogProtocol, *, event: UsageReportEvent) -> None` — `event` is the object emitted; `event_log` is the target/sink (`event_log.emit(event)`). Both call sites pass `event_log=`. "emit best effort" names the action, not the receiver, so neither arg has a clear subject claim. Suggested fix: make fully keyword-only.--### `runtime_bridge` (0 high · 1 med/low)--**Medium / low**--- `pipelex/runtime_bridge/primitives/hydration.py:17` — `_validate_as_known_class` — `def _validate_as_known_class(item_class: type[StuffContent], *, raw_item: StuffContent | dict[str, Any]) -> StuffContent` — `item_class` is a type/schema used to validate, not the data acted on; `raw_item` is the actual object. "validate raw_item into item_class" → `raw_item` is arguably the subject. Call sites: `_validate_as_known_class(item_class_or_none, raw_item=raw_item)` — a type as positional subject is unusual. Suggested fix: make fully keyword-only, or reorder to `(raw_item, *, item_class)`.-------## Zero-suspect packages--`base_exceptions` (1 reviewed), `errors` (7), `hub` (15), `language` (8), `pipe_signature` (2), `tracing` (9) — reviewed, no positional-subject abuse found.--## Suggested next step--Reserve adjudication for the **31 high-confidence** entries first — they cluster tightly into patterns 1–6 above and most already have call sites that keyword the "positional" arg, so the fix is low-risk and mechanical (`move *` / reorder). The `pipelex.make`/`setup` pair (pattern 8) is a deliberate public-API call. The med/low entries flagged "decide as a family" (`pipe_controllers` run-pipe convention, `graph` dispatch methods) should be settled at the base/convention level, not site-by-site. Point Opus at this shortlist, or hand-pick — happy to drive either.diff --git a/wip/history/plugin-system-overview.html b/wip/history/plugin-system-overview.htmldeleted file mode 100644index 6761ed0d4..000000000--- a/wip/history/plugin-system-overview.html+++ /dev/null@@ -1,477 +0,0 @@-<!DOCTYPE html>-<html lang="en">-<head>-<meta charset="utf-8">-<meta name="viewport" content="width=device-width, initial-scale=1">-<title>Pipelex Plugin System, Orchestrators, and Deployment Flavors — current state</title>-<style>- :root{- --bg:#0d1117; --ink:#e6edf3; --muted:#9aa7b4; --line:#283039;- --brand:#7c9bff; --brand-soft:#16203a; --good:#3fb950; --good-soft:#102a17;- --bad:#ff7b72; --bad-soft:#2d1416; --warn:#e3b341; --warn-soft:#2a2412;- --code-bg:#010409; --code-ink:#e6edf3; --chip:#1c232c; --panel:#161b22;- }- *{box-sizing:border-box}- body{margin:0;background:var(--bg);color:var(--ink);- font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;- -webkit-font-smoothing:antialiased}- .wrap{max-width:980px;margin:0 auto;padding:40px 22px 90px}- h1{font-size:30px;line-height:1.2;margin:0 0 6px}- h2{font-size:22px;margin:46px 0 14px;padding-bottom:8px;border-bottom:2px solid var(--line)}- h3{font-size:17px;margin:26px 0 8px}- p{margin:12px 0}- a{color:var(--brand);text-decoration:none}- a:hover{text-decoration:underline}- .sub{color:var(--muted);font-size:15px;margin:0 0 22px}- code{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:13.5px;- background:var(--chip);padding:1px 6px;border-radius:5px}- pre{margin:0;overflow:auto}- pre code{background:none;padding:0}- .card{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:18px 20px;margin:16px 0;- box-shadow:0 1px 2px rgba(0,0,0,.3)}- .tldr{background:linear-gradient(180deg,#172244,#11161f);border:1px solid #2b3a5e;border-left:5px solid var(--brand);- border-radius:12px;padding:20px 22px;margin:18px 0 8px}- .tldr h2{border:0;margin:0 0 8px;font-size:19px;color:var(--brand)}- .tldr ul{margin:8px 0 0;padding-left:20px}- .tldr li{margin:5px 0}- .callout{border-radius:10px;padding:12px 16px;margin:16px 0;font-size:14.5px}- .callout.good{background:var(--good-soft);border:1px solid #1f6f33}- .callout.warn{background:var(--warn-soft);border:1px solid #5a4d1a}- .callout.info{background:var(--brand-soft);border:1px solid #2b3a5e}- .chips{display:flex;flex-wrap:wrap;gap:8px;margin:14px 0}- .chip{background:var(--chip);border:1px solid var(--line);border-radius:999px;padding:4px 12px;font-size:13px;color:var(--muted)}- .chip b{color:var(--ink)}- table{border-collapse:collapse;width:100%;margin:14px 0;font-size:14px}- th,td{border:1px solid var(--line);padding:8px 11px;text-align:left;vertical-align:top}- th{background:#1c232c;font-weight:600}- td code{font-size:12.5px}- .seam-table th:first-child,.seam-table td:first-child{width:38%;min-width:290px}- .status-ok{color:var(--good);font-weight:600}- .status-wait{color:var(--warn);font-weight:600}- /* diff blocks */- .code{background:var(--code-bg);color:var(--code-ink);border-radius:10px;padding:14px 16px;- font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:13px;line-height:1.55;overflow:auto}- .code .del{display:block;color:#ffb4ab;background:rgba(207,34,46,.16);margin:0 -16px;padding:0 16px}- .code .add{display:block;color:#7ee2a8;background:rgba(26,127,55,.18);margin:0 -16px;padding:0 16px}- .code .ctx{display:block;color:#9fb0c0}- .code .cmt{color:#7d8c9c;font-style:italic}- .code .kw{color:#ff9b72}- .code .fn{color:#7cc7ff}- /* tabs */- .tabs{margin:16px 0}- .tab-bar{display:flex;flex-wrap:wrap;gap:4px;border-bottom:1px solid var(--line);margin-bottom:0}- .tab-btn{background:none;border:0;border-bottom:2px solid transparent;padding:9px 14px;font:inherit;font-size:14px;- color:var(--muted);cursor:pointer}- .tab-btn:hover{color:var(--ink)}- .tab-btn.active{color:var(--brand);border-bottom-color:var(--brand);font-weight:600}- .tab-panel{display:none;padding-top:14px}- .tab-panel.active{display:block}- details{background:var(--panel);border:1px solid var(--line);border-radius:10px;padding:2px 16px;margin:12px 0}- details[open]{padding-bottom:14px}- summary{cursor:pointer;font-weight:600;padding:12px 0;color:var(--ink)}- summary::marker{color:var(--brand)}- .mermaid{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:18px;margin:16px 0;text-align:center}- figcaption{color:var(--muted);font-size:13px;text-align:center;margin-top:6px}- .small{font-size:13px;color:var(--muted)}- hr{border:0;border-top:1px solid var(--line);margin:40px 0}- .legend{font-size:12.5px;color:var(--muted);margin-top:6px}- .kbd{font-family:ui-monospace,monospace;font-size:12px;background:var(--chip);border:1px solid var(--line);- border-bottom-width:2px;border-radius:5px;padding:1px 6px}-</style>-</head>-<body>-<div class="wrap">--<h1>Pipelex Plugin System, Orchestrators, and Deployment Flavors</h1>-<p class="sub">What we built, what changed since the first overview, and where it stands · branch <code>refactor/Plugins-4</code></p>--<div class="tldr">- <h2>TL;DR</h2>- <p><strong>Pipelex core no longer dispatches by vendor branches.</strong></p>- <p>Backends, orchestrators, validators, model listers, boot-runtime claims, and HTTP error mappers now register themselves through one plugin contract. Core installs registries and looks up capabilities by data.</p>- <ul>- <li><strong>Mechanism:</strong> <code>PipelexPlugin</code> + <code>PluginRegistrar</code> + <code>build_registrar()</code>. Built-ins and <code>pipelex.plugins</code> entry points use the same path.</li>- <li><strong>New since the first overview:</strong> plugin API v2, lazy HTTP error mappers, env-aware plugin configs, <code>plugins.boot_orchestrator</code>, <code>orchestration_mode</code> + <code>DeliveryMode</code>, and <code>BundleValidatorRegistry</code>.</li>- <li><strong>Status:</strong> core is current on <code>refactor/Plugins-4</code>. Temporal is a private <code>pipelex-temporal</code> git+ssh dist. The public base is orchestrator-agnostic; deployment-flavor wiring and release pin flips remain.</li>- </ul>-</div>--<div class="chips">- <span class="chip"><b>core</b> pipelex @ <code>refactor/Plugins-4</code> · pushed ✓</span>- <span class="chip"><b>plugin API</b> <code>2</code> · HTTP mappers + validators</span>- <span class="chip">all gates <b class="status-ok">green</b></span>- <span class="chip">deployment flavors <b class="status-wait">pending</b></span>- <span class="chip"><b>pipelex-temporal</b> = private (git+ssh, not PyPI)</span>-</div>--<!-- ============================================================ -->-<h2>1 · The problem we set out to kill</h2>-<p>Pipelex core used to <em>hard-code</em> every optional integration behind a <code>match</code> statement. Adding a backend meant editing core; an optional SDK could be imported at the wrong time; and Temporal — a heavy, optional dependency — was woven through the core runtime. The coupling lived in a handful of <strong>seams</strong>:</p>--<figure>-<svg viewBox="0 0 900 330" width="100%" role="img" aria-label="Before and after architecture">- <defs>- <marker id="arr" markerWidth="9" markerHeight="9" refX="7" refY="3" orient="auto">- <path d="M0,0 L7,3 L0,6 Z" fill="#8a98a8"/>- </marker>- <marker id="arrG" markerWidth="9" markerHeight="9" refX="7" refY="3" orient="auto">- <path d="M0,0 L7,3 L0,6 Z" fill="#3fb950"/>- </marker>- </defs>- <!-- BEFORE -->- <text x="210" y="26" text-anchor="middle" font-size="14" font-weight="700" fill="#ff7b72">BEFORE — core hard-codes each integration</text>- <rect x="150" y="48" width="120" height="46" rx="8" fill="#2d1416" stroke="#ff7b72" stroke-width="1.5"/>- <text x="210" y="69" text-anchor="middle" font-size="13" font-weight="700" fill="#e6edf3">Pipelex core</text>- <text x="210" y="85" text-anchor="middle" font-size="11" fill="#ff7b72">match sdk / mode</text>- <g font-size="11.5" fill="#cdd6df">- <rect x="40" y="150" width="90" height="34" rx="6" fill="#161b22" stroke="#5a4048"/><text x="85" y="171" text-anchor="middle">anthropic</text>- <rect x="145" y="150" width="90" height="34" rx="6" fill="#161b22" stroke="#5a4048"/><text x="190" y="171" text-anchor="middle">openai</text>- <rect x="250" y="150" width="90" height="34" rx="6" fill="#161b22" stroke="#5a4048"/><text x="295" y="171" text-anchor="middle">mistral</text>- <rect x="120" y="210" width="110" height="34" rx="6" fill="#161b22" stroke="#5a4048"/><text x="175" y="231" text-anchor="middle">temporalio</text>- </g>- <g stroke="#8a98a8" stroke-width="1.3" marker-end="url(#arr)" fill="none">- <path d="M180,94 L95,150"/><path d="M205,94 L195,150"/><path d="M235,94 L300,150"/><path d="M205,94 L185,210"/>- </g>- <!-- divider -->- <line x1="450" y1="40" x2="450" y2="300" stroke="#283039" stroke-width="1.5" stroke-dasharray="4 4"/>- <!-- AFTER -->- <text x="690" y="26" text-anchor="middle" font-size="14" font-weight="700" fill="#3fb950">AFTER — plugins register; core names none</text>- <rect x="630" y="48" width="120" height="46" rx="8" fill="#102a17" stroke="#3fb950" stroke-width="1.5"/>- <text x="690" y="69" text-anchor="middle" font-size="13" font-weight="700" fill="#e6edf3">Pipelex core</text>- <text x="690" y="85" text-anchor="middle" font-size="11" fill="#3fb950">lookup()</text>- <rect x="600" y="135" width="180" height="40" rx="8" fill="#16203a" stroke="#7c9bff" stroke-width="1.5"/>- <text x="690" y="160" text-anchor="middle" font-size="12.5" font-weight="700" fill="#7c9bff">Hub registries</text>- <path d="M690,94 L690,135" stroke="#3fb950" stroke-width="1.4" marker-end="url(#arrG)" fill="none"/>- <g font-size="11.5" fill="#cdd6df">- <rect x="520" y="232" width="96" height="34" rx="6" fill="#161b22" stroke="#2f5a3a"/><text x="568" y="253" text-anchor="middle">anthropic</text>- <rect x="626" y="232" width="96" height="34" rx="6" fill="#161b22" stroke="#2f5a3a"/><text x="674" y="253" text-anchor="middle">openai</text>- <rect x="732" y="232" width="130" height="34" rx="6" fill="#161b22" stroke="#2f5a3a"/><text x="797" y="253" text-anchor="middle">pipelex-temporal</text>- </g>- <g stroke="#3fb950" stroke-width="1.3" marker-end="url(#arrG)" fill="none">- <path d="M568,232 L660,175"/><path d="M674,232 L688,175"/><path d="M797,232 L720,175"/>- </g>- <text x="690" y="296" text-anchor="middle" font-size="11" fill="#9aa7b4">register(registrar) — at discovery</text>-</svg>-<figcaption>The inversion in one picture: the <code>match</code> fan-out becomes a registry that plugins populate.</figcaption>-</figure>--<table class="seam-table">- <thead><tr><th>Seam (where core named an integration)</th><th>File(s)</th><th>How it's resolved now</th></tr></thead>- <tbody>- <tr><td>Inference dispatch — 4 families (LLM, image-gen, extract, search)</td><td><code>cogt/{llm,img_gen,extract,search}/*_worker_factory.py</code></td><td><code>InferenceBackendRegistry.lookup(family, sdk)</code></td></tr>- <tr><td>Model listing (the <code>show models</code> path)</td><td><code>cogt/model_backends/model_lists.py</code></td><td><code>ModelListerRegistry.get_optional(sdk)</code></td></tr>- <tr><td>Orchestrator dispatch (Direct / Temporal / Mistral)</td><td><code>runtime_bridge/bridge.py</code></td><td><code>OrchestratorRegistry.get_optional(mode)</code> + <code>DeliveryMode</code></td></tr>- <tr><td>Bundle validation dispatch (<code>/validate</code>)</td><td><code>pipeline/*</code>, <code>runtime_bridge/*</code></td><td><code>BundleValidatorRegistry.get_optional(mode)</code></td></tr>- <tr><td>HTTP transport-fault rendering</td><td>host runtimes such as <code>pipelex-api</code></td><td>plugin-contributed lazy <code>ErrorReport</code> mappers</td></tr>- <tr><td>Boot / teardown hub-swap (Temporal runtime)</td><td><code>pipelex.py</code></td><td>hub-slot claims + LIFO teardown callbacks</td></tr>- <tr><td>Config schema + plugin config files</td><td><code>configs.py</code>, <code>config_loader.py</code></td><td>Temporal schema relocated to core; plugin TOMLs load through env-aware layering</td></tr>- <tr><td>Operational CLI commands</td><td>external plugin packages</td><td>ops commands ship as plugin-owned console scripts, not harvested onto <code>pipelex</code></td></tr>- </tbody>-</table>-<p class="legend">Every row above used to contain a vendor or <code>temporalio</code> branch. After the refactor, dispatch and boot paths use registries; built-in vendors are named only by their plugin modules/list, typed config, and prose.</p>--<!-- ============================================================ -->-<h2>2 · The design — one contract, one registrar, one discovery pass</h2>-<p>A plugin is a tiny object with a <code>register</code> method. Discovery hands it a <strong>registrar</strong> (a menu of "I provide X" methods); the registrar accumulates contributions into registries that get installed on the hub at boot. That's the whole machinery.</p>--<h3>The contract</h3>-<div class="code"><pre><code><span class="cmt"># pipelex/plugins/contract.py</span>-PLUGIN_API_VERSION: int = 2--@runtime_checkable-<span class="kw">class</span> <span class="fn">PipelexPlugin</span>(Protocol):- name: str- targets_api: int- <span class="kw">def</span> <span class="fn">register</span>(self, registrar: "PluginRegistrar") -> None: ...- <span class="cmt"># INVARIANT: register is side-effect-free — it may ONLY call registrar</span>- <span class="cmt"># menu methods. No hub access, no I/O, no SDK/client construction.</span>- <span class="cmt"># Heavy work happens lazily inside worker closures, slot thunks,</span>- <span class="cmt"># plugin config loaders, and HTTP-error type providers.</span></code></pre></div>--<h3>The discovery → boot flow</h3>-<pre class="mermaid">-sequenceDiagram- autonumber- participant Boot as Pipelex.setup()- participant Disc as build_registrar(config)- participant Built as BUILTIN_PLUGINS- participant EP as entry_points("pipelex.plugins")- participant Reg as PluginRegistrar- participant Hub as Hub- Boot->>Disc: build_registrar(config)- Disc->>Built: iterate built-in plugins- Built->>Reg: plugin.register(registrar)- Disc->>EP: iterate installed dists (e.g. pipelex-temporal)- EP->>Reg: plugin.register(registrar)- Note over Reg: side-effect-free — menu calls only, imports no SDK- Disc-->>Boot: registrar (registries + slot thunks + teardown cbs + mapper providers)- Boot->>Hub: set inference / orchestrator / validator / model-lister registries- Boot->>Hub: resolve hub slots (explicit param > slot thunk > core default)-</pre>-<figcaption>One pure pass builds the registrar; boot installs the result. Because <code>register</code> is side-effect-free, the pass is safely repeatable.</figcaption>--<h3>What the registrar offers</h3>-<table>- <thead><tr><th>Menu method</th><th>Contribution</th></tr></thead>- <tbody>- <tr><td><code>add_inference_backend(family, sdk, make_worker)</code></td><td>a backend driver, keyed by <code>(family, sdk)</code></td></tr>- <tr><td><code>add_orchestrator(mode, orchestrator)</code></td><td>an execution strategy, keyed by open <code>orchestration_mode</code></td></tr>- <tr><td><code>add_bundle_validator(mode, validator)</code></td><td>a <code>/validate</code> strategy, keyed by <code>orchestration_mode</code></td></tr>- <tr><td><code>add_model_lister(sdk, lister)</code></td><td>an optional model-listing capability</td></tr>- <tr><td><code>add_http_error_mapper(exc_type_provider, to_error_report)</code></td><td>a lazy transport-exception → <code>ErrorReport</code> mapping for host runtimes</td></tr>- <tr><td><code>claim_content_generator / pipe_router / pipe_run / task_manager(factory)</code></td><td>a hub-slot claim — a <em>deferred thunk</em>, run only if the slot is taken</td></tr>- <tr><td><code>add_teardown(callback)</code></td><td>a teardown hook, run LIFO at shutdown</td></tr>- </tbody>-</table>--<!-- ============================================================ -->-<h2>3 · The critical changes (minimal diffs)</h2>-<p>The heart of the work is a set of <code>match</code>-to-registry collapses. Pick a tab:</p>--<div class="tabs" data-group="diffs">- <div class="tab-bar">- <button class="tab-btn active" onclick="showTab('diffs',0)">Inference dispatch</button>- <button class="tab-btn" onclick="showTab('diffs',1)">Orchestrator dispatch</button>- <button class="tab-btn" onclick="showTab('diffs',2)">A driver plugin</button>- <button class="tab-btn" onclick="showTab('diffs',3)">Temporal externalized</button>- </div>-- <div class="tab-panel active">- <p class="small"><code>cogt/llm/llm_worker_factory.py</code> — a ~230-line <code>match plugin.sdk</code> with ten vendor arms collapses to a registry lookup.</p>- <div class="code"><pre><code><span class="ctx"> def make_llm_worker(inference_model, *, reporting_delegate=None):</span>-<span class="del">- match plugin.sdk:</span>-<span class="del">- case "gateway_completions": ... # ten vendor arms, each one:</span>-<span class="del">- case "anthropic" | "bedrock_anthropic": # require_sdk(...) + lazy import</span>-<span class="del">- client = AnthropicFactory.make_anthropic_client(...) # + build client</span>-<span class="del">- llm_worker = AnthropicLLMWorker(...) # + build worker</span>-<span class="del">- case "mistral": ... ; case "google": ...</span>-<span class="del">- case _:</span>-<span class="del">- raise NotImplementedError(f"{plugin.sdk} is not supported")</span>-<span class="del">- return llm_worker</span>-<span class="add">+ model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)</span>-<span class="add">+ backend = get_models_manager().get_required_inference_backend(inference_model.backend_name)</span>-<span class="add">+ make_worker = get_inference_backend_registry().lookup(family=InferenceFamily.LLM, sdk=model_handle.sdk)</span>-<span class="add">+ worker = make_worker(inference_model=inference_model, backend=backend,</span>-<span class="add">+ sdk_clients=get_sdk_client_manager().sdk_client_registry,</span>-<span class="add">+ reporting_delegate=reporting_delegate)</span>-<span class="add">+ return cast("LLMWorkerInternalAbstract", worker)</span></code></pre></div>- <p class="small">The same shape was applied to image-gen, extract, and search. A registry miss raises a structured <code>InferenceBackendNotFoundError</code> ("… is its plugin installed and enabled?"), never a <code>None</code> deref.</p>- </div>-- <div class="tab-panel">- <p class="small"><code>runtime_bridge/bridge.py</code> — the old <code>match execution_mode</code> (with hard <code>temporalio</code> / Mistral imports) became a registry lookup. Since then, the flat mode enum was split into an open <code>orchestration_mode</code> token plus a closed <code>DeliveryMode</code>.</p>- <div class="code"><pre><code><span class="del">- match execution_mode:</span>-<span class="del">- case PipelexExecutionMode.DIRECT:</span>-<span class="del">- return await _run_direct(...)</span>-<span class="del">- case PipelexExecutionMode.TEMPORAL_BLOCKING:</span>-<span class="del">- return await _run_temporal_blocking(...) # imported temporalio</span>-<span class="del">- case PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET:</span>-<span class="del">- return await _run_temporal_fire_and_forget(...)</span>-<span class="del">- case PipelexExecutionMode.MISTRAL_NATIVE:</span>-<span class="del">- return await _run_mistral_native(...) # imported mistral</span>-<span class="add">+ orchestrator = get_orchestrator_registry().get_optional(mode=input_payload.orchestration_mode)</span>-<span class="add">+ if orchestrator is None:</span>-<span class="add">+ raise MissingOrchestratorError(mode=input_payload.orchestration_mode)</span>-<span class="add">+ _validate_input(input_payload, orchestrator=orchestrator, delivery_assignment=delivery_assignment)</span>-<span class="add">+ return await orchestrator.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment,</span>-<span class="add">+ delivery=input_payload.delivery)</span></code></pre></div>- <p class="small"><code>MissingOrchestratorError</code> is generic and plugin-decoupled. Fire-and-forget is checked against <code>supports_fire_and_forget</code> before dispatch, so blocking-only modes reject honestly instead of pretending to enqueue.</p>- </div>-- <div class="tab-panel">- <p class="small"><code>plugins/anthropic/anthropic_plugin.py</code> — a whole backend in one small file. <code>register</code> is trivial and import-light; the closure does the heavy lifting lazily.</p>- <div class="code"><pre><code><span class="kw">def</span> <span class="fn">_make_anthropic_worker</span>(*, inference_model, backend, sdk_clients, reporting_delegate):- require_sdk(spec="anthropic", extra="anthropic", msg=_ANTHROPIC_MISSING_MSG)- <span class="kw">from</span> ...anthropic_factory <span class="kw">import</span> AnthropicFactory <span class="cmt"># lazy: no SDK at boot</span>- <span class="kw">from</span> ...anthropic_llm_worker <span class="kw">import</span> AnthropicLLMWorker- model_handle = ModelHandle.make_for_inference_model(inference_model=inference_model)- sdk_instance = sdk_clients.get_or_create(handle=model_handle,- build=<span class="kw">lambda</span>: AnthropicFactory.make_anthropic_client(model_handle=model_handle, backend=backend))- <span class="kw">return</span> AnthropicLLMWorker(sdk_instance=sdk_instance, ...)--<span class="kw">class</span> <span class="fn">AnthropicPlugin</span>:- name = "anthropic"- targets_api = PLUGIN_API_VERSION- <span class="kw">def</span> <span class="fn">register</span>(self, registrar):- registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="anthropic", make_worker=_make_anthropic_worker)- registrar.add_inference_backend(family=InferenceFamily.LLM, sdk="bedrock_anthropic", make_worker=_make_anthropic_worker)- registrar.add_model_lister(sdk="anthropic", lister=_list_anthropic_models)</code></pre></div>- <p class="small">One plugin can serve several <code>(family, sdk)</code> keys and several families at once (Mistral registers into both LLM and Extract; Linkup into both Extract and Search).</p>- </div>-- <div class="tab-panel">- <p class="small">Once orchestrators went through the registry, Temporal no longer needed to live in core. It became its own private distribution, discovered by an entry point.</p>- <div class="code"><pre><code><span class="cmt"># pipelex (core) — pyproject.toml: the temporal extra is gone</span>-<span class="ctx"> [project.optional-dependencies]</span>-<span class="del">-temporal = ["temporalio==1.24.0", "aiohttp>=3.14.0"]</span>-<span class="ctx"> [tool.pytest.ini_options] markers = [</span>-<span class="del">- "temporal: tests that require a Temporal server",</span>--<span class="cmt"># pipelex (core) — plugins/builtins.py: TemporalPlugin removed from the list</span>-<span class="ctx"> BUILTIN_PLUGINS = [ DirectOrchestratorPlugin(), OpenAIPlugin(), ... ]</span>-<span class="del">- TemporalPlugin(),</span>-<span class="add">+# Temporal is NOT here: it ships as the external pipelex-temporal distribution,</span>-<span class="add">+# discovered via its pipelex.plugins entry point. Zero config, no core temporalio import.</span></code></pre></div>- <div class="code" style="margin-top:10px"><pre><code><span class="cmt"># pipelex-temporal (new dist) — pyproject.toml</span>-[project]-name = "pipelex-temporal"-dependencies = ["pipelex>=0.35.0", "temporalio==1.24.0", "aiohttp>=3.14.0"]--[project.scripts]-pipelex-temporal = "pipelex_temporal.temporal_cli:app" <span class="cmt"># worker / setup-namespace</span>--[project.entry-points."pipelex.plugins"]-temporal = "pipelex_temporal.temporal_plugin:TemporalPlugin" <span class="cmt"># <- discovery hook</span>--[tool.uv.sources]-pipelex = { path = "../_plugins", editable = true } <span class="cmt"># dev cross-pin → pipelex==Y at publish</span></code></pre></div>- </div>-</div>--<!-- ============================================================ -->-<h2>4 · What ships in core today</h2>-<p>Core ships a list of built-in plugins. Importing this module imports <strong>no backend SDK</strong> — the SDKs load lazily inside the <code>make_worker</code> closures.</p>-<div class="code"><pre><code><span class="cmt"># pipelex/plugins/builtins.py</span>-BUILTIN_PLUGINS: list[PipelexPlugin] = [- DirectOrchestratorPlugin(), <span class="cmt"># core orchestrator</span>- OpenAIPlugin(), GatewayPlugin(), PortkeyPlugin(), AnthropicPlugin(),- MistralPlugin(), BedrockPlugin(), GooglePlugin(), FalPlugin(),- HuggingFacePlugin(), BlackboxaiPlugin(), OpenRouterPlugin(),- AzureRestPlugin(), DoclingPlugin(), Pypdfium2Plugin(), LinkupPlugin(),-]-<span class="cmt"># Naming one of these in plugins.disabled is a config error, not a no-op:</span>-CORE_UNCONDITIONAL_PLUGIN_NAMES = frozenset({"direct", "openai"})</code></pre></div>-<p class="small">Between them these built-ins register every <code>(family, sdk)</code> inference backend across the four families. <code>direct</code> owns the in-process orchestration mode and bundle validator; <code>openai</code> is the always-on default driver. Observability: <code class="kbd">pipelex plugins list</code> shows what's discovered, what each registered, and the denylist state.</p>--<!-- ============================================================ -->-<h2>5 · Temporal as an external plugin</h2>-<p>Phase 5 lifted <code>pipelex/temporal/</code> into a sibling distribution, <strong>pipelex-temporal</strong> (top-level package <code>pipelex_temporal</code>). It depends <em>on</em> core; core depends on nothing Temporal. Installing the dist contributes the <code>"temporal"</code> orchestration mode, its async delivery capability, its bundle validator, and its optional boot-runtime slot claims — no core change.</p>--<pre class="mermaid">-graph TB- subgraph core["pipelex (core, MIT) — refactor/Plugins-4"]- bridge["runtime_bridge dispatch + validate"]- orr["OrchestratorRegistry (on hub)"]- bridge --> orr- end- subgraph pt["pipelex-temporal (PRIVATE dist) — main"]- tp["TemporalPlugin (entry point: pipelex.plugins)"]- cli["console script: pipelex-temporal worker / setup-namespace"]- end- tp -. "install via git+ssh: entry-point discovery" .-> orr- pt -- "depends on" --> core- subgraph priv["our private infra — installs pipelex-temporal via git+ssh"]- w["pipelex-worker"]- hosted["pipelex-api-hosted"]- end- subgraph pub["public packages (MIT) — drop the temporal coupling"]- api["pipelex-api"]- mw["pipelex-mistralai-workflows"]- end- w -. "git+ssh pin" .-> pt- hosted -. "git+ssh pin (runner enqueues)" .-> pt-</pre>-<figcaption>Core never imports Temporal. The private dist plugs in through the <code>pipelex.plugins</code> entry point. Only <strong>our private infra</strong> installs it (via a <code>git+ssh</code> pin); the <strong>public MIT packages drop temporal</strong> — they can't depend on a private repo.</figcaption>--<details>- <summary>The key prerequisite (Step 0b): why the config schema had to move first</summary>- <p>Core config (<code>configs.py</code>) imported the Temporal config <em>schema</em>, which lived under <code>pipelex.temporal</code>. You cannot "keep config in core, move <code>temporal/</code> out" while core config imports from <code>temporal/</code>. So the schema (<code>config_temporal.py</code>) and the two exceptions it raises were relocated into core (<code>pipelex/system/configuration/{config_temporal,exceptions}.py</code>), re-based on <code>PipelexError</code>. After this, <strong>the last hard <code>pipelex.temporal</code> import in core was gone</strong> — and the move became pure packaging. The <code>RetryPolicy</code> field aliases to <code>Any</code> at runtime so the schema validates without importing <code>temporalio</code>.</p>- <p class="small">A latent bug surfaced and was fixed along the way: a <code>(ValueError, PipelexError)</code> MRO meant <code>ValueError.__init__</code> won and <code>.message</code> was never set. Bases were reordered to <code>(PipelexError, ValueError)</code>, with a regression test sweeping every <code>PipelexError</code> subclass.</p>-</details>--<details>- <summary>"Option A": why operational commands are a console script, not a plugin contribution</summary>- <p>An earlier design harvested plugin-contributed CLI commands by running discovery while building the <code>pipelex</code> CLI. A review found this brittle: one broken or colliding external plugin (or an unreadable user config) could brick <em>every</em> command — including the <code>--help</code>/<code>doctor</code>/<code>plugins</code> recovery commands. <strong>We removed the surface instead of hardening it.</strong> Temporal's <code>worker</code> and <code>setup-namespace</code> now ship as a standalone <code>pipelex-temporal</code> console script (<code>[project.scripts]</code>), so <code>pipelex --help</code> no longer loads config or scans entry points, and a third party can't shadow a core command.</p>-</details>--<!-- ============================================================ -->-<h2>6 · The invariants that hold every phase</h2>-<div class="callout good"><strong>Import-light boot.</strong> Booting with built-ins registered imports no optional SDK (<code>anthropic</code>, <code>mistralai</code>, <code>boto3</code>, <code>fal_client</code>, …) and no <code>temporalio</code>. Enforced by a subprocess test with a <code>sys.meta_path</code> finder that <em>raises</em> on those modules — boot must not trip it.</div>-<div class="callout info"><strong>Side-effect-free <code>register</code>.</strong> A plugin's <code>register</code> may only call registrar menu methods — no hub access, no I/O, no client construction. This is what makes discovery a pure, repeatable function.</div>-<div class="callout warn"><strong>Fail loud.</strong> Duplicate <code>(family, sdk)</code> / duplicate mode / a double-claimed slot / an API-version mismatch / a broken entry point each raise a named, contextual error. A missing backend says "install X" — it never dereferences <code>None</code>.</div>--<h3>Presence vs activation (so you don't add a flag that shouldn't exist)</h3>-<table>- <thead><tr><th>Question</th><th>Answer</th></tr></thead>- <tbody>- <tr><td>Is a plugin <em>present</em>?</td><td>It's in <code>BUILTIN_PLUGINS</code> or an installed dist exposes the <code>pipelex.plugins</code> entry point. Zero config.</td></tr>- <tr><td>When does an inference backend <em>activate</em>?</td><td>By data — a model names its <code>sdk</code>; the registry resolves it.</td></tr>- <tr><td>When does an orchestrator <em>activate</em>?</td><td>Per call — the request's open <code>orchestration_mode</code> selects the plugin; the endpoint-selected <code>DeliveryMode</code> says whether the caller waits.</td></tr>- <tr><td>When does the Temporal <em>runtime</em> boot?</td><td>Only when <code>plugins.boot_orchestrator = "temporal"</code> — a per-process "boot as a Temporal worker" gate, not a generic on/off toggle.</td></tr>- <tr><td>Can I turn a plugin off?</td><td>A narrow <code>plugins.disabled</code> denylist (can't disable a core-unconditional plugin). There is deliberately no redundant allowlist.</td></tr>- </tbody>-</table>--<!-- ============================================================ -->-<h2>7 · Phase history</h2>-<p>The original plugin-system phases landed as checkpoint commits on <code>refactor/Plugins-3</code>. <code>refactor/Plugins-4</code> then added the consumer-facing seams needed for an orchestrator-agnostic API base and deployment flavors.</p>-<table>- <thead><tr><th>Phase</th><th>What it did</th><th>Checkpoint commit</th></tr></thead>- <tbody>- <tr><td><b>0</b> · Rename</td><td>Freed the word "plugin": <code>Plugin→ModelHandle</code>, <code>PluginManager→SdkClientManager</code>, …</td><td><code>7213a90c0</code></td></tr>- <tr><td><b>1</b> · Seam + LLM</td><td>Contract, registrar, discovery, registries; LLM family migrated (contract proven on a real family)</td><td><code>42e9f7797</code></td></tr>- <tr><td><b>2</b> · Inference families</td><td>Image-gen / extract / search migrated; all four families go through the registry</td><td><code>d62628f0b</code>, <code>94fa908df</code></td></tr>- <tr><td><b>3</b> · Orchestrators</td><td>Bridge <code>match</code> + four boot blocks + teardown → registry & hub-slot thunks</td><td><code>5f61db323</code></td></tr>- <tr><td>· Option A</td><td>Dropped the CLI-harvest seam; ops commands → <code>pipelex-temporal</code> console script</td><td><code>989c9beed</code></td></tr>- <tr><td><b>4</b> · Model listing</td><td>The 5th seam: <code>model_lists.py match</code> → <code>ModelListerRegistry</code></td><td><code>6a0796548</code>, <code>e4e021c2b</code></td></tr>- <tr><td><b>5</b> · Step 0b</td><td>Relocated the Temporal config schema into core (the move's prerequisite)</td><td><code>7d64eb6ac</code></td></tr>- <tr><td><b>5</b> · Cut-over</td><td>Externalized <code>pipelex/temporal/</code> → the <code>pipelex-temporal</code> dist</td><td><code>a8382d1a1</code>, <code>27da150a5</code></td></tr>- <tr><td><b>5</b> · Finalize</td><td>Removed dead Temporal dev-tooling from core (skills + Makefile targets → relocated)</td><td><code>e410ae971</code></td></tr>- <tr><td><b>Plugins-4 A</b></td><td>Added env-aware plugin-config loading and the HTTP error mapper seam; bumped <code>PLUGIN_API_VERSION</code> to 2</td><td><code>4b5449f88</code>, <code>75196e2e3</code></td></tr>- <tr><td><b>Boot gate</b></td><td>Moved Temporal runtime activation from <code>temporal.is_enabled</code> to <code>plugins.boot_orchestrator</code></td><td><code>eaa671d10</code></td></tr>- <tr><td><b>Mode split</b></td><td>Split flat <code>PipelexExecutionMode</code> into open <code>orchestration_mode</code> plus closed <code>DeliveryMode</code></td><td><code>bd4e52827</code></td></tr>- <tr><td><b>Validate</b></td><td>Added per-call <code>BundleValidatorRegistry</code> dispatch for orchestrator-aware <code>/validate</code></td><td><code>bbe44c929</code></td></tr>- </tbody>-</table>--<!-- ============================================================ -->-<h2>8 · Where we stand</h2>-<div class="callout good"><strong>Core foundation current.</strong> In this repo, built-in inference plugins, direct orchestration, model listing, HTTP mapper contributions, bundle validators, slot claims, and plugin discovery are wired through the registrar. The public core still imports no Temporal SDK at boot.</div>--<table>- <thead><tr><th>Repo</th><th>Branch · tip</th><th>State</th></tr></thead>- <tbody>- <tr><td>pipelex (core)</td><td><code>refactor/Plugins-4</code> · <code>221a2c611</code></td><td class="status-ok">pushed — plugin API v2, mode/delivery split, bundle validator seam</td></tr>- <tr><td>pipelex-temporal <span class="small">(private)</span></td><td><code>feature/Orchestrator-dispatched-validate</code></td><td class="status-ok">adopts the v2 seams: env-aware config, lazy Temporal error mapper, Temporal validator</td></tr>- <tr><td>pipelex-api <span class="small">(public base)</span></td><td><code>refactor/orchestrator-agnostic-base</code> + follow-ons</td><td class="status-ok">Temporal dependency removed; consumes mapper and dispatch seams; follow-on execute/start work stacked</td></tr>- <tr><td>pipelex-api-hosted <span class="small">(private flavor)</span></td><td>—</td><td class="status-wait">build Temporal flavor image: install <code>pipelex-temporal</code> via git+ssh and bake <code>pipelex_{env}.toml</code> + <code>temporal_{env}.toml</code></td></tr>- <tr><td>pipelex-worker <span class="small">(private)</span></td><td>—</td><td class="status-wait">flip pin from <code>pipelex[temporal]</code> to <code>pipelex</code> + <code>pipelex-temporal</code>; CLI becomes <code>pipelex-temporal worker</code></td></tr>- <tr><td>pipelex-mistralai-workflows <span class="small">(public)</span></td><td>feature branch</td><td class="status-ok">migrated to the open <code>orchestration_mode</code> / <code>DeliveryMode</code> surface in its plugin work</td></tr>- </tbody>-</table>--<h3>What's left — deployment flavors and release pins</h3>-<p class="small"><code>pipelex-temporal</code> is <strong>private / not open-source</strong>, so it is <strong>not published to PyPI</strong>. It travels by a pinned <code>git+ssh</code> reference — exactly how this workspace already pins <code>pipelex-shared</code> (<code>… @ git+ssh://[email protected]/Pipelex/infra-python-tools.git@<ref></code>).</p>-<ol>- <li>Build the private Temporal API flavor: base <code>pipelex-api</code> image + one <code>pipelex-temporal</code> git+ssh install + env config child images.</li>- <li>Move Temporal connection settings into plugin-owned <code>temporal_{env}.toml</code>; the main <code>pipelex_{env}.toml</code> keeps only core/base settings such as <code>[plugins] boot_orchestrator = "temporal"</code>.</li>- <li>Flip <code>pipelex-worker</code> to the external plugin in the same release window as the public core/API pins.</li>- <li>Run consumer suites against the exact public version + private plugin ref pair before deploying hosted Temporal.</li>-</ol>-<p class="callout warn"><strong>Known boundary:</strong> the plugin system now supports both private and public orchestrator plugins, but the release/install channel is still per plugin. Private Temporal rides git+ssh; public Mistral rides PyPI once published. Do not add a central allowlist or a generic "enable plugin" flag to bridge that packaging difference.</p>--<!-- ============================================================ -->-<h2>9 · Links</h2>-<ul>- <li><strong>pipelex (core):</strong> <a href="https://github.com/Pipelex/pipelex">github.com/Pipelex/pipelex</a> · branch <a href="https://github.com/Pipelex/pipelex/tree/refactor/Plugins-4">refactor/Plugins-4</a></li>- <li><strong>pipelex-temporal:</strong> <a href="https://github.com/Pipelex/pipelex-temporal">github.com/Pipelex/pipelex-temporal</a> <span class="small">(private — git+ssh install)</span></li>- <li><strong>Deployment consumers:</strong> <a href="https://github.com/Pipelex/pipelex-worker">pipelex-worker</a> <span class="small">(private)</span> · <a href="https://github.com/Pipelex/pipelex-api-hosted">pipelex-api-hosted</a> <span class="small">(private Temporal flavor)</span> · <a href="https://github.com/Pipelex/pipelex-api">pipelex-api</a> <span class="small">(public base)</span> · <a href="https://github.com/Pipelex/pipelex-mistralai-workflows">pipelex-mistralai-workflows</a> <span class="small">(public plugin)</span></li>-</ul>--<hr>-<p class="small">Updated for co-developer review · core <code>refactor/Plugins-4</code> @ <code>221a2c611</code>. Diagrams render with Mermaid (loaded from CDN when this file is opened in a browser).</p>--</div>--<script>-function showTab(group, idx){- document.querySelectorAll('.tabs[data-group="'+group+'"] .tab-btn').forEach(function(b,i){b.classList.toggle('active', i===idx);});- document.querySelectorAll('.tabs[data-group="'+group+'"] .tab-panel').forEach(function(p,i){p.classList.toggle('active', i===idx);});-}-</script>-<script type="module">- import mermaid from 'https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.esm.min.mjs';- mermaid.initialize({ startOnLoad: true, theme: 'dark', securityLevel: 'loose' });-</script>-</body>-</html>diff --git a/wip/history/pydantic-before-validator-typing.md b/wip/history/pydantic-before-validator-typing.mddeleted file mode 100644index c1ff269db..000000000--- a/wip/history/pydantic-before-validator-typing.md+++ /dev/null@@ -1,57 +0,0 @@-# Typing `mode="before"` validators (Pydantic v2) — a short brief--## The core fact--A `@model_validator(mode="before")` (and `@field_validator(..., mode="before")`) runs **before Pydantic has coerced or validated anything**. It receives the *raw* input as handed to the model. For a model that input is usually a `dict`, but it can be anything the caller passed: a `str`, an `int`, `None`, an already-constructed instance, or — crucially — a value being trial-validated as one arm of a `Union`.--So the input type is genuinely unconstrained. Pydantic's own type stubs annotate the before-validator value as `Any`.--## What's pythonic for the annotation--Two honest options, in order of preference:--1. **`Any` in, `Any` out.** Truthful about the contract (raw, pre-validation input), and decoupled from any particular caller. The validator's job is to *normalize toward the dict shape*, not to assert it.-2. **An explicit `dict[str, Any] | T` union** — only when the model deliberately participates in a known `T`-vs-other union and you want the two expected shapes documented at the signature. This reads as self-documenting but is *circumstantial*: the non-dict arm names a sibling type from one consumer's union, so it leaks that consumer into the model. Avoid widening it past what you actually branch on.--Anti-pattern: annotating the value as `dict[str, Any]` when the validator can in fact receive a non-dict. That's a comfortable lie — it makes `values.get(...)` typecheck while the code crashes at runtime on a `str`/`None`.--## The load-bearing rule: guard on shape, inside the validator--```python-@model_validator(mode="before")-@classmethod-def normalize(cls, data: Any) -> Any:- if not isinstance(data, dict):- return data # let a later arm / the model machinery handle it- # ... dict-only logic here ...- return data-```--The `isinstance` guard — not the annotation — is what makes the validator robust. A before-validator should never assume its input shape.--## Why this matters for unions (the real trap)--Pydantic converts only `ValueError` and `AssertionError` raised inside a validator into a `ValidationError`. **Any other exception** (`AttributeError`, `TypeError`, `KeyError`, …) propagates unchanged.--Inside `Union` validation, Pydantic tries members and catches each member's `ValidationError` to move on to the next. A *non-`ValidationError`* exception is not caught — it aborts the whole union. So an unguarded before-validator that does `data.get(...)` on a `str` raises `AttributeError`, which escapes the union instead of cleanly rejecting that arm. The symptom looks like "this valid input doesn't match the union," but the cause is a validator that wasn't shape-safe.--## Don't fix it by reordering the union--Putting the simpler type first (`str | Model` instead of `Model | str`) can make the symptom vanish — Pydantic's default *smart* mode prefers an exact-type match, so a `str` input may hit the `str` arm and never invoke the model's before-validator. But this is masking at the wrong layer:--- it only helps the one call site whose ordering you changed;-- it doesn't protect `Model.model_validate(<non-dict>)` called directly, or any other union the model appears in;-- it depends on union-resolution behavior (smart vs `left_to_right`) that's easy to perturb later.--Fix the validator (guard on shape); treat union ordering as a readability choice, not a correctness mechanism.--## Smart vs left-to-right, in one line--Default `union_mode="smart"` picks the best-matching member regardless of declaration order (exact type matches win); `union_mode="left_to_right"` tries members top-to-bottom and takes the first that validates. Neither rescues a before-validator that raises a non-`ValidationError` on the arm it *does* try.--## Takeaways--- A before-validator receives `Any`; type it `Any` (or a deliberately-narrow union you actually branch on) — never a shape you merely hope for.-- Guard with `isinstance(...)` and return non-matching input untouched.-- Raise `ValueError`/`AssertionError` for real validation failures so Pydantic wraps them; let unexpected types fall through rather than crash.-- Robustness belongs on the model's validator, not on the order of a consumer's `Union`.diff --git a/wip/history/recursivity/domain-metadata-merge.md b/wip/history/recursivity/domain-metadata-merge.mddeleted file mode 100644index 7d0cc56ce..000000000--- a/wip/history/recursivity/domain-metadata-merge.md+++ /dev/null@@ -1,72 +0,0 @@-# Handoff — domain metadata merge under the additive multi-file model (pipelex)--> **Status: DONE.** Implemented on `feature/Support-recursive-design`. Surfaced during the final `mthds-plugins/wip/recursive/design.md` review. Small, self-contained pipelex change. Builds on the additive multi-file model already shipped on this branch (see `../../TODOS.md` Parts A+B and [`recursive-followups.md`](recursive-followups.md) Tasks 1–2). Does **not** block the additive model — it removes friction the model introduces.->-> Verified: `make agent-test` (green) + `make agent-check` (clean).->-> **As built.** Added the order-independent, omission-quiet merge as a shared helper `pipelex/libraries/domain/domain_metadata_merge.py::merge_domain_metadata_field(...)`, wired from both `LibraryCrateFactory.make_from_blueprints` (crate-blueprint layer) and `DomainLibrary.add_domain` (runtime-`Domain` layer). The resolved conflict-resolution TODO at `domain_library.py` is gone (the separate system_prompt-inlining TODO is kept). `main_pipe` left untouched (inert, per below). Tests in `tests/unit/pipelex/libraries/test_library_crate.py` + `test_domain_library.py` (both-orders membership, same-value no-warn, conflict-warns, across description + system_prompt) with warnings asserted via `caplog`. CHANGELOG `[Unreleased]` sub-bullet folded into the "Additive multi-file library construction" entry.--## Why--The additive model authors a same-domain library as many `.mthds` files: one **root** (`bundle.mthds`) that carries the full domain header (`domain`, `description`, `main_pipe`, optional `system_prompt`) and N **non-root definition files** that carry only `domain = "<same_domain>"` to declare membership (see `design.md` §2.7 and the §3 worked example — `build_research_brief.mthds` sets only `domain = "research_brief"`).--The domain-metadata merge was written for the single-file world (first-write-wins + warn on any difference). Under the additive model it misbehaves in two ways, both rooted in "an omitted field is treated as a conflicting empty value":--1. **Spurious warnings on every validation.** A non-root file omits `description`, so the merge compares the root's real description against `""` and logs `Domain '<d>' declared with different descriptions: '<real>' vs ''. Keeping the first.` This fires once per non-root file, at **two** layers (see Where), and the PostToolUse hook validates the whole library (`-L <dir>`) on **every per-file save** — so a build of K files emits O(K) warnings on every save. Pure noise that will train authors/agents to ignore warnings.--2. **Load-order-dependent metadata loss (the real bug).** First-write-wins means the *first* domain blueprint seen wins. If a non-root file (empty description) is merged before the root, the assembled domain keeps `description = ""` and the root's real description is **discarded** (with the warning above). The assembled domain's `description` / `system_prompt` therefore depend on file load order — which for a `-L` directory is filesystem/sort order, not authoring intent. (Example: `build_research_brief.mthds` sorts before `bundle.mthds`, so the empty-description sibling can win.)--`description` / `system_prompt` are display/doc/PipeLLM-fallback metadata, not execution-critical, so this is not a correctness-of-results bug — but it is wrong, order-dependent, and noisy.--**Not affected — `main_pipe` (verified, do not "fix").** The runtime `Domain` model (`pipelex/core/domains/domain.py`) carries only `code`, `description`, `system_prompt` — **no `main_pipe`** — and `DomainFactory.make_from_blueprint` copies only description + system_prompt. The `DomainBlueprint.main_pipe` set during the crate merge is dropped when the runtime `Domain` is built, so the first-write-wins on `main_pipe` is inert. Main-pipe existence is validated **per file** on the bundle blueprint that declares it (`PipelexBundleBlueprint`, the `main_pipe in self.pipe` check), and the root always carries both `main_pipe` and the matching header (Layer 0), so that check passes. No order-fragility there. Leave it alone.--## Where (verified file:line)--Two sites apply the same first-write-wins + warn-on-difference logic. Both need the same relaxation:--- `pipelex/libraries/library_crate_factory.py` — `make_from_blueprints`, the domain block (currently ~l.63–83). Builds the per-domain `DomainBlueprint`; `description=blueprint.description or ""`; the `else` arm warns when `existing.description != (blueprint.description or "")` and when `existing.system_prompt != blueprint.system_prompt`.-- `pipelex/libraries/domain/domain_library.py` — `add_domain` (~l.33–50). Same warn-on-difference at runtime-`Domain` registration. **Already carries a TODO** at l.34: `# TODO: resolve domain metadata conflicts properly — currently first-write-wins with a warning.` This handoff is that TODO.--Supporting facts (verified):-- `PipelexBundleBlueprint.description: str | None = None` — omitting it is legal (`pipelex/core/bundles/pipelex_bundle_blueprint.py`).-- `DomainBlueprint.description: str` (required) — the factory coerces an omitted bundle description to `""` (`pipelex/core/domains/domain_blueprint.py`).-- `Domain.description: str | None = None`, `Domain.system_prompt: str | None = None` (`pipelex/core/domains/domain.py`).--## The fix--Make domain-metadata merge **order-independent** and **quiet for omissions**: an omitted (empty/`None`) `description` or `system_prompt` contributes *no opinion* — it neither overrides an established non-empty value nor triggers a warning. Warn **only** when two *non-empty* values genuinely differ (a real conflict the author should resolve).--Concretely, per domain field (`description`, `system_prompt`):-- established empty/None + incoming non-empty → take incoming, no warning (root can arrive after a sibling).-- established non-empty + incoming empty/None → keep established, no warning (sibling defers to root).-- both non-empty and equal → keep, no warning.-- both non-empty and different → keep first, **warn** (genuine conflict — unchanged behavior).--This means the root's `description`/`system_prompt` always wins over membership-only siblings regardless of load order, and the additive flow is warning-free, while genuine double-declarations still warn.--**Apply at both sites**, or — preferred — extract one shared helper (e.g. `pipelex/libraries/domain/domain_metadata_merge.py::merge_domain_metadata(...)` or a `DomainBlueprint`/`Domain` "fold" method) and call it from both `make_from_blueprints` and `add_domain`, so the two layers can't drift. Keep the existing class/`type_uri`/message slugs; this is behavior-narrowing on the no-conflict path only.--### Authoring-side alternative (rejected as primary)--We could instead require every non-root file to repeat the full domain header (`description`, `system_prompt`). Rejected: it forces prose duplication across every definition file, which is its own drift source and defeats the point of membership-only sibling files. The runtime fix is the right lever; if anything, the cheat sheet should explicitly say non-root files carry **only** `domain = "..."`.--## Tasks--- [x] Add the order-independent, omission-quiet merge (shared helper) and wire both `make_from_blueprints` and `DomainLibrary.add_domain` to it. Removed the now-resolved conflict-resolution TODO at `domain_library.py` (kept the separate system_prompt-inlining TODO).-- [x] `make agent-check` clean (ruff, plxt, pyright, mypy).-- [x] Tests below green; then full `make agent-test`.-- [x] CHANGELOG `[Unreleased]` entry (folded into the existing "Additive multi-file library construction" entry).--## Tests--- Root (real description) + sibling (`domain` only) merged **both orders** → assembled domain keeps the root's description/system_prompt; **no warning** emitted. Assert at both layers (crate factory result and `DomainLibrary` after `add_domains`).-- Two files each declaring a **different non-empty** description → first kept, **warning** still emitted (unchanged).-- Two files with the **same non-empty** description → kept, no warning.-- Cover `system_prompt` with the same matrix.-- Regression: existing single-file domain-metadata tests still pass.--(Capture warnings with `caplog`/the project's log-capture fixture; assert presence/absence explicitly — the whole point is the no-warning path.)--## Tracked from--`mthds-plugins/wip/recursive/design.md` §2.7 (the additive multi-file layout: root carries the domain header, siblings carry only `domain =`) and §3 (the worked example whose sibling files omit `description`). Also indexed in `mthds-plugins/wip/deferred-issues.md`.diff --git a/wip/history/recursivity/protocol-validate-pending-signatures.md b/wip/history/recursivity/protocol-validate-pending-signatures.mddeleted file mode 100644index e61adcd92..000000000--- a/wip/history/recursivity/protocol-validate-pending-signatures.md+++ /dev/null@@ -1,32 +0,0 @@-# Follow-up — surface `pending_signatures` / `is_runnable` on the protocol-level `validate`--> **Status: ✅ DONE — implemented 2026-06-12.** `pending_signatures` + `is_runnable` added to `PipelexValidationReport` and populated in `PipelexMTHDSProtocol.validate`; first test coverage for the protocol `validate` landed in `tests/integration/pipelex/pipeline/test_protocol_validate.py` (verdict on both paths, strict-mode raise, library-lifecycle contract); changelog updated under `[Unreleased]`.--## Context (cold start)--v0.33.0 (PR #985) turned the runner into a local MTHDS Protocol implementation: `PipelexMTHDSProtocol` in `pipelex/pipeline/runner.py`. Its `validate(mthds_contents, allow_signatures=False)` method wraps our `validate_bundle` and maps the result onto `PipelexValidationReport`, currently returning only `blueprint`, `graph_spec=None`, and `pipe_structures`.--Meanwhile this branch's recursive-design work made `validate_bundle` compute the runnability verdict: `ValidateBundleResult.pending_signatures` (`pipelex/pipeline/validate_bundle.py` — `build_pending_signatures`, library-wide, cross-package refs excluded). It is surfaced as `pending_signatures: string[]` + `is_runnable: bool` on the agent-CLI / builder validate envelopes (`pipelex/cli/agent_cli/commands/validate/_validate_core.py`, `pipelex/builder/operations/validate_ops.py` — `is_runnable = not pending_signatures` everywhere), but **not** on the protocol `validate` — so a top-down build driving the runtime through the MTHDS Protocol (instead of the agent CLI) can't see what remains to implement.--This is protocol-legal without any spec change: the protocol base `mthds.protocol.models.ValidationReport` deliberately declares **no body fields** and sets `extra="allow"` — "implementations may include their own artifacts". `PipelexValidationReport` already follows that pattern with `blueprint` / `graph_spec` / `pipe_structures` as Pipelex extension fields.--## Plan--1. **Add the two extension fields to `PipelexValidationReport`** (`pipelex/pipeline/runner.py`): `pending_signatures: list[str] = Field(default_factory=list)` and `is_runnable: bool = True`.-2. **Populate them in `PipelexMTHDSProtocol.validate`** from the `validate_bundle` result, mirroring the CLI convention: `pending_signatures=result.pending_signatures, is_runnable=not result.pending_signatures`. The wrapper already holds `result` in hand where it builds the report — a two-line change.-3. **Tests — note there is currently NO test coverage for `PipelexMTHDSProtocol.validate` at all** (grep `\.validate(` under `tests/` finds nothing against the protocol wrapper). So this needs a first integration test, e.g. `tests/integration/pipelex/pipeline/test_protocol_validate.py`:- - a complete bundle → `pending_signatures == []`, `is_runnable is True`;- - a bundle with an unsatisfied `PipeSignature`, `allow_signatures=True` → the signature's qualified ref in `pending_signatures`, `is_runnable is False`;- - reuse the `.mthds` fixtures under `tests/e2e/pipelex/pipes/additive_multi_file_library/signature_only/` (header + concepts, no definition) rather than authoring new ones.- - While there, the test can also pin the wrapper's library-lifecycle contract (restores the caller's current library, tears the validation library down) — it's documented in a comment in `validate` but untested.-4. **Changelog** — entry under `[Unreleased]` (Changed, alongside the existing additive-multi-file entry that already documents `pending_signatures` on the CLI surfaces).--## Verify--`make agent-check` clean + `make agent-test` green (per repo standard).--## Watch-outs--- **Strict-mode interplay:** with `allow_signatures=False` (the protocol default), an unsatisfied signature makes `validate_bundle` *raise* (`ValidateBundleError` from the strict refusal) — the report fields only matter on the lenient path. Don't try to populate them on the failure path; failures are RFC 7807 problems per the protocol docstring.-- **Downstream — CORRECTED 2026-06-12:** the original assumption here ("pipelex-api serializes whatever `validate` returns, fields appear automatically") is **wrong**. `pipelex-api`'s `/validate` route does NOT call `PipelexMTHDSProtocol.validate` — it builds its own strict `ValidateResponse` pydantic envelope (`api/routes/pipelex/validate.py`), calling `validate_bundle` directly (direct mode) or `act_dry_validate` (Temporal mode), and neither that envelope nor `DryValidateResult` carries `pending_signatures`. Surfacing the verdict on the hosted `/validate` therefore needs explicit changes — now part of a broader protocol-surface alignment (the full consistency pass also found `/models` divergence, `pipe_structures` shape conflicts, and a duplicated protocol-version constant). Live plan: workspace root `wip/mthds-protocol-surface-alignment.md` (supersedes `../pipelex-api/wip/validate-pending-signatures-envelope.md`).-- A related deliberate omission: `validate all` / `validate pipe` intentionally omit `pending_signatures` (it's a per-bundle, top-down-build nudge — rationale in the comments at `pipelex/builder/operations/validate_ops.py` `validate_all` and `pipelex/cli/agent_cli/commands/validate/_validate_core.py`). The protocol `validate` is bundle-shaped, so it takes the field; don't propagate it to the all/pipe surfaces. The original feature record is Task 1 in [`recursive-followups.md`](recursive-followups.md).diff --git a/wip/history/recursivity/recursive-followups.md b/wip/history/recursivity/recursive-followups.mddeleted file mode 100644index ac7bd11a8..000000000--- a/wip/history/recursivity/recursive-followups.md+++ /dev/null@@ -1,58 +0,0 @@-# Handoff — recursive-building runtime follow-ups (pipelex)--> **Status: ✅ BOTH DONE** — shipped on `feature/Support-recursive-design`. Task 1 (`pending_signatures` on validate) and Task 2 (normalized contract conformance) are implemented and verified. The detail below is kept as the record of what was built.--Two follow-ups on the pipelex side, surfaced during the `mthds-plugins` design review. Both build on the additive multi-file model already shipped on this branch (`feature/Support-recursive-design`; see `../../TODOS.md` for Parts A + B). Neither blocks the additive model; both make it robust/complete. Independent of each other — land in either order.--Verify after each: `make agent-test` (green) + `make agent-check` (clean).-------## Task 1 — Expose the unsatisfied-signature set on a successful lenient validate ✅ DONE--> Shipped: `validate bundle` now emits `pending_signatures: string[]` (library-wide namespaced refs still typed `PipeSignature`; empty when complete) in the JSON envelope and as a "Pending signatures" markdown section, on agent-CLI + builder surfaces.--**Why.** The `mthds-plugins` PostToolUse hook wants a non-blocking "these pipes are still unimplemented" nudge on a *successful* `--allow-signatures` run (design.md §5.2). Today the reachable-signature list is surfaced only on *strict* failure (`SignaturesNotAllowedError`); a lenient success emits no machine-readable list of what remains.--**Key simplification (from Part A).** Once a concrete reconciles with a signature, the signature is *replaced* in the merged library. So "unsatisfied signatures" = exactly the pipes still typed `PipeSignature` in the assembled library. No `{signatures} − {concretes}` arithmetic needed — just enumerate `is_signature` pipes.--**Where.**-- `pipelex/pipeline/validate_bundle.py` — `ValidateBundleResult` (l.39) already carries `pipes: list[PipeAbstract]`. Add a projection next to `build_validated_pipes` (l.57):-- ```python- def build_pending_signatures(pipes: list[PipeAbstract]) -> list[str]:- """Qualified refs of pipes still declared as PipeSignature (unsatisfied headers)."""- return sorted(pipe.pipe_ref for pipe in pipes if pipe.is_signature)- ```-- (`PipeAbstract.is_signature` and `.pipe_ref` already exist.)-- Surface it on the agent-CLI validate JSON envelope, next to `validated_pipes` — find the surface that calls `build_validated_pipes` (the `pipelex-agent validate bundle` command under `pipelex/cli/agent_cli/.../validate/`). Add a `pending_signatures` array. Mirror on the plain `pipelex validate bundle` command if it emits structured output. A one-line human note on success ("N pipe(s) still PipeSignature: …") is optional.--**Scope nuance — compute from the library, not `result.pipes`.** For the file-path path, when the file is already loaded `result.pipes` is only *that file's* pipes (l.262-265). The hook validates one file at a time but wants the *whole-library* pending set. Compute pending from the assembled library (`library.pipe_library`), not from `result.pipes`, so the answer is library-wide regardless of which file triggered the run.--**Tests.** Lenient validate of a bundle with unsatisfied headers → `pending_signatures` lists them; after every header has a concrete definition → empty. Fold into the additive integration test (`tests/integration/pipelex/libraries/test_additive_multi_file_library.py`).-------## Task 2 — Proper (normalized) contract conformance, replacing raw-string `contract_equals` ✅ DONE--> Shipped: header↔definition contracts reconcile by concept identity (bare ≡ qualified, `Text` ≡ `native.Text`, structural multiplicity); genuinely different concepts or differing multiplicity still error; refinement substitutability stays deferred.--**Why.** `PipeBlueprint.contract_equals` (pipe_blueprint.py:111-120) compares raw strings. But Part B resolves concept refs flexibly (bare ↔ domain-qualified), so a header `output = "Brief"` and a definition `output = "thisdomain.Brief"` denote the *same* concept yet fail `contract_equals` → a false mismatch. Raw-string is not a proper check.--**Scope: normalize for _identity_** — bare↔qualified, whitespace, multiplicity parsed structurally. This is **not** refinement substitutability (covariant output / contravariant inputs); that stays deferred (`mthds-plugins/wip/deferred-issues.md` → "Substitutable contract conformance…").--**Normalization (reuse existing helpers).** For each concept spec:-- Parse multiplicity with `parse_concept_with_multiplicity` (already used in `PipeBlueprint.generic_validate_output`) → `(concept_ref_or_code, multiplicity)`.-- Canonicalize the ref: native (`NativeConceptCode`) → keep the native code; cross-package (`QualifiedRef.has_cross_package_prefix`) → keep; external-domain qualified → keep; bare / same-domain → `ConceptFactory.make_concept_ref_with_domain(domain_code, code)`.-- Compare canonical `(qualified_ref, multiplicity)` tuples: `inputs` as a `name → tuple` dict (`None` ≡ `{}`), `output` as a tuple.--**Domain context.** `contract_equals` lives on the blueprint, which doesn't carry its domain. The caller does: `_reconcile_pipe_collision` (library_crate_factory.py:136) runs inside the merge loop that has `domain_code` (l.59), and both declarations share the domain (same `pipe_ref`). So pass `domain_code` into `_reconcile_pipe_collision` and into the comparison. Suggest a focused free function `pipelex/libraries/contract_match.py::contracts_match(a, b, *, domain_code) -> bool` (mirroring `collision_messages.py` / `concept_reference_validation.py`), and replace the `existing.blueprint.contract_equals(incoming.blueprint)` call at l.166. Retire `contract_equals` or keep it as a thin raw fallback — your call.--**Tests.** Header `Brief` + def `thisdomain.Brief` reconcile; whitespace / multiplicity-spelling variants reconcile; native refs reconcile; genuinely different concepts or differing multiplicity still mismatch. Update any existing exact-match tests that assumed raw-string equality.--**Boundary.** Refinement-aware substitutability stays deferred — see `mthds-plugins/wip/deferred-issues.md`.-------These two follow-ups are tracked from the design doc: Task 1 backs design.md §5.2/§5.4 (the leftover-signature nudge), Task 2 backs §2.7/§4.5 (the exact-match contract rule, which this upgrades from raw-string to normalized).diff --git a/wip/history/recursivity/signature-taxonomy-refactor.md b/wip/history/recursivity/signature-taxonomy-refactor.mddeleted file mode 100644index a2ad537d8..000000000--- a/wip/history/recursivity/signature-taxonomy-refactor.md+++ /dev/null@@ -1,215 +0,0 @@-# Refactor: `PipeSignature` is a blueprint-layer contract, not an executable pipe type--> **Status:** ✅ implemented. Raised by review of the additive-multi-file PR: *"I disagree that it's a new type of pipe. Pipe signatures are a substrate of the `PipeAbstract`. Each pipe has a pipe signature."* This plan implemented the agreed reconciliation. All gates green (`make tb`, `make agent-check`, `make agent-test`); MTHDS schema regeneration produced **no diff** (the eviction is invisible to the generated schema — `pipe_category` is stripped from pipe defs and `type` is unchanged). **Step 1b (class-based `is_controller`) was deliberately skipped** — see [As-built notes](#as-built-notes).--## The decision--Keep `PipeSignature(PipeAbstract)` — it stays a `PipeAbstract` subclass, because the dry-run engine dispatches over `PipeAbstract` and a signature must be dry-runnable (verify inputs, mock output). But **evict it from the executable-kind taxonomy**: remove `PipeSignature` from the `PipeType` and `PipeCategory` enums, and identify "is this a signature?" by class, not by an enum field.--`signature_for: PipeType | None` **stays as a field** for now (it is consumed by `../mthds-plugins` and we don't want to break that). Its two `reject_signature_for_pipe_signature` guard validators are **deleted** — see [the forced interaction](#the-signature_for-interaction) below.--This is **not** the "Option A" collapse (a `signature = true` flag on a real operator type, optional bodies on `PipeLLM`, etc.). That was explicitly rejected: a signature is a contract that stands on its own, **not** a pending/incomplete version of a concrete pipe.--## The diagnosis (why it ended up looking like a pipe type)--`PipeType` and `PipeCategory` each silently do **two jobs**:--1. A **parse-time allowlist** — "is this `type` string legal under `[pipe.X]`?" (`PipeBlueprint.validate_pipe_type`, `PipeAbstract.validate_pipe_type`, `PipeSpec.validate_pipe_type`, all checking `value in PipeType.value_list()`).-2. The **executable-kind / orchestration-role taxonomy** — `PipeType.category` → operator vs controller; `PipeCategory.is_controller`.--A signature legitimately belongs in job (1): `type = "PipeSignature"` is a real thing you write in a bundle. But the only way to clear that allowlist was to also become a `PipeType` member — which **forced** it to claim a `.category`, which dragged it into job (2) as a sibling of `PipeLLM`/`PipeSequence`. `PipeCategory.PIPE_SIGNATURE` (a peer of `PIPE_OPERATOR`/`PIPE_CONTROLLER`) is the visible scar. The fix separates the two jobs: the allowlist becomes "executable kinds ∪ the signature tag"; the taxonomy enums shrink back to executable kinds only.--## The principle--- **`PipeAbstract` = the substrate** (contract-bearing graph node: `inputs` / `output` / `description`, dry-runnable). A `PipeSignature` is that substrate with nothing added. It stays here. *(This is exactly the co-dev's "substrate of `PipeAbstract`".)*-- **`PipeType` / `PipeCategory` = the executable taxonomy** (how a pipe runs and whether it orchestrates). A signature does not run, so it is in **neither**.-- **Identity by class, not by enum field.** `is_signature` becomes a class fact (base returns `False`; `PipeSignature` / `PipeSignatureBlueprint` override to `True`), so it no longer reads `pipe_category`. Every existing `is_signature` call site keeps working unchanged — that property is the keystone.-- **The runtime `PipeSignature` is the dry-run shim of a blueprint-layer contract.** Fully determined by the contract; carries no body. Live-run always errors; dry-run mocks, and only under `--allow-signatures`.--## Scope decision: the spec layer stays put--`../mthds-plugins` authors against the **spec** layer (`PipeSignatureSpec`, including `signature_for`). To keep that external surface stable, the spec layer keeps its `type = "PipeSignature"` and `pipe_category = "PipeSignature"` string literals — they are display/authoring tags, they are **not** typed as `PipeCategory`, and `PipeSignatureSpec.to_blueprint()` does not propagate `pipe_category` into the blueprint anyway. The only required spec-layer edits are: widen `PipeSpec.validate_pipe_type` to admit the signature tag, and delete the now-broken `signature_for` guard. The taxonomy correction lands at the **blueprint + runtime** layers, where it belongs.-------## Implementation steps--### Step 1 — Keystone: `is_signature` by class--`pipelex/core/pipes/pipe_abstract.py` — replace the enum-based property (currently `return PipeCategory(self.pipe_category) is PipeCategory.PIPE_SIGNATURE`, lines 71-73) with a base that returns `False`:--```python-@property-def is_signature(self) -> bool:- return False-```--`pipelex/pipe_signature/pipe_signature.py` — add the override:--```python-@property-@override-def is_signature(self) -> bool:- return True-```--`pipelex/core/pipes/pipe_blueprint.py` — same on `PipeBlueprint` (lines 107-109 → `return False`).-`pipelex/pipe_signature/pipe_signature_blueprint.py` — add the `is_signature → True` override.--**Step 1b (recommended, same principle, optional):** convert `PipeAbstract.is_controller` (lines 67-69) the same way — base `return False`, `PipeController` overrides `return True` — so `is_controller` also stops reading `pipe_category`. Without this, `is_controller` still works for a signature (it calls `PipeCategory.is_controller_by_str(None)`, which hits `except ValueError → False`), so 1b is cleanliness, not correctness. Do it unless we want the smallest possible diff.--### Step 2 — Shrink the enums--`pipelex/core/pipes/pipe_blueprint.py`:--- `PipeCategory`: delete `PIPE_SIGNATURE = "PipeSignature"` (line 16). In `is_controller` (the match), drop `PipeCategory.PIPE_SIGNATURE` from the `case PipeCategory.PIPE_OPERATOR | PipeCategory.PIPE_SIGNATURE:` arm → `case PipeCategory.PIPE_OPERATOR:`. Still exhaustive over `{OPERATOR, CONTROLLER}`.-- `PipeType`: delete `PIPE_SIGNATURE = "PipeSignature"` (line 54). In `.category`, delete the `case PipeType.PIPE_SIGNATURE: return PipeCategory.PIPE_SIGNATURE` arm (lines 85-86). `.category` is now a **total** function over executable kinds — which is the tidiness win that makes eviction strictly better than keeping it with an `Optional` return.--### Step 3 — Tolerate the signature in the shared validators--The signature's `type` ("PipeSignature") is no longer in `PipeType.value_list()`, and its `pipe_category` becomes `None` (Step 5). The base validators on **both** `PipeBlueprint` and `PipeAbstract` need three small allowances:--- `validate_pipe_type` (field, after): admit the signature tag.-- ```python- if value not in PipeType.value_list() and value != PipeSignatureBlueprint type tag "PipeSignature":- raise ...- ```-- Express the allowlist honestly — define a module-level `PIPE_SIGNATURE_TYPE_TAG = "PipeSignature"` in `pipe_blueprint.py` and check `value not in PipeType.value_list() and value != PIPE_SIGNATURE_TYPE_TAG`. (Avoid importing `PipeSignatureBlueprint` into the base module — that's a cycle; a bare string constant is the right tool.)--- `validate_pipe_category` (field, after): admit `None`.-- ```python- if value is not None and value not in PipeCategory.value_list():- raise ...- ```-- Only signatures carry `None` (executable subclasses pin `Literal["PipeOperator"|"PipeController"]`), so this is safe.--- `validate_pipe_category_based_on_type` (model, after): early-return for signatures, before the `PipeType(self.type)` coercion that would now raise.-- ```python- def validate_pipe_category_based_on_type(self) -> Self:- if self.is_signature:- return self- ...- ```-- `is_signature` is class-based after Step 1, so this is reliable at model-validation time.--### Step 4 — Factory: branch around the `PipeType` coercion--`pipelex/core/pipes/pipe_factory.py` (lines 105-106) currently does `pipe_type = PipeType(blueprint.type)` then `pipe_category = pipe_type.category` — both raise for a signature. Branch on `is_signature` and resolve the factory by name from the string (no `PipeSignatureFactory` import → no cycle; the class-registry lookup already in place resolves it):--```python-if blueprint.is_signature:- pipe_category = None- factory_class_name = f"{blueprint.type}Factory" # "PipeSignatureFactory"-else:- pipe_type = PipeType(blueprint.type)- pipe_category = pipe_type.category- factory_class_name = f"{pipe_type.value}Factory"-```--The existing registry lookup + `pipe_factory.make(pipe_category=pipe_category, ...)` call (lines 110-131) stays. `PipeSignatureFactory.make` already accepts `pipe_category: Any`; it ignores it and builds the shim.--### Step 5 — Signature runtime + blueprint: `pipe_category` → `None`--- `pipelex/pipe_signature/pipe_signature.py` (line 28): `pipe_category: Literal["PipeSignature"] = "PipeSignature"` → `pipe_category: None = None`. (`type` stays `Literal["PipeSignature"]` — it's the blueprint tag, still the discriminator string.)-- `pipelex/pipe_signature/pipe_signature_blueprint.py` (line 18): → `pipe_category: None = Field(default=None, exclude=True)`. **Keep `exclude=True`** — overriding the field type drops the base's `Field(exclude=True)` unless re-specified, and we still don't want `pipe_category` serialized into round-tripped `.mthds`.--The discriminated union (`PipeBlueprintUnion`) discriminates on `type`, not `pipe_category`, so `None` here is invisible to parsing.--### Step 6 — `output_renderer` guard--`pipelex/core/pipes/output/output_renderer.py`: line 32 `PipeType(the_pipe.type)` raises for a signature, and the match arm (lines 99-110) lists `PipeType.PIPE_SIGNATURE`. Add an early return and drop the arm:--```python-if the_pipe.is_signature:- return []-pipe_type = PipeType(the_pipe.type)-...-case (PipeType.PIPE_FUNC | ... | PipeType.PIPE_PARALLEL): # PIPE_SIGNATURE removed- return []-```--Still exhaustive over the shrunken `PipeType`.--### Step 7 — Delete the `signature_for` guards (keep the field) {#the-signature_for-interaction}--`pipelex/pipe_signature/pipe_signature_blueprint.py` (lines 24-30) and `pipelex/builder/pipe/pipe_signature_spec.py` (lines 53-59): **delete** both `reject_signature_for_pipe_signature` validators. They reference `PipeType.PIPE_SIGNATURE`, which no longer exists → they would raise `AttributeError` whenever a signature is parsed. Their intent (forbid `signature_for=PipeSignature`) is now enforced structurally: `signature_for: PipeType | None` can no longer coerce the string `"PipeSignature"`, so Pydantic rejects it with a clean validation error.--`signature_for` itself, `PipeSignatureFactory`'s `signature_for=blueprint.signature_for` pass-through (line 32), and `PipeSignatureSpec.to_blueprint`'s `signature_for=self.signature_for` (line 67) all **stay**. Soften the docstrings that say "cannot itself be `PipeSignature`" (`pipe_signature_spec.py` lines 42-43) since the enum now enforces it.--### Step 8 — Spec layer: widen `validate_pipe_type`--`pipelex/builder/pipe/pipe_spec.py` (lines 79-85): same widening as Step 3's `validate_pipe_type` — admit the signature tag so `PipeSignatureSpec` still validates. No other spec-layer change (see [scope decision](#scope-decision-the-spec-layer-stays-put)); `pipe_spec_map.py:27` (`"PipeSignature": PipeSignatureSpec`) is a string-keyed map and is unaffected.-------## ⛳ Checkpoint — run the core gate before docs--After Steps 1-8 (all code), this is the risky core. Run and get green before touching prose:--- `make tb` — boot/config load (catches enum + Pydantic field breakage fast).-- `.venv/bin/pipelex-dev generate-mthds-schema` — regenerate `derived/mthds_schema.json` (the `PipeSignatureBlueprint` definition still exists in the union; `_PIPE_INTERNAL_FIELDS = {"pipe_category"}` still strips the field). Commit the regenerated schema.-- `make agent-check` — ruff / pyright / mypy / plxt.-- `make agent-test` — full suite. Expect edits in the signature/taxonomy unit tests that asserted `PipeCategory.PIPE_SIGNATURE` or `signature_for` rejection messages; update them to the new model.--### Step 9 — Docs / CHANGELOG / TODOS reframe--- `docs/building-methods/pipes/signature-pipes.md` — stop framing it as "a new type of pipe." Frame: every pipe *has* a signature (its contract); a `PipeSignature` is the contract alone — a blueprint-layer declaration whose runtime form is a dry-run-only mock shim, deliberately outside the executable `PipeType`/`PipeCategory` taxonomy.-- `TODOS.md` change #1 ("Pipe signature ↔ concrete reconciliation") — reword from "replace pipe-type `PipeSignature` with pipe-type `PipeLLM`" to "**a contract fulfilled by an implementation**" (the reconciliation table is unchanged; `contracts_match` is unchanged).-- `CHANGELOG.md` `[Unreleased]` — note: `PipeSignature` moved out of the `PipeType`/`PipeCategory` taxonomy; `PipeCategory.PIPE_SIGNATURE` removed (breaking for anything matching on it); `signature_for` retained.-------## A welcome side effect--The headline feature reads truer. `library_crate_factory._reconcile_pipe_collision` keys off `blueprint.is_signature` (lines 169-190) — after this it means **"a contract is fulfilled by an implementation,"** not "swap pipe-type `PipeSignature` for pipe-type `PipeLLM`." `contract_match.contracts_match()` is untouched.--## Out of scope / future--- **Eventually retire `signature_for`.** Kept now only for `../mthds-plugins`. It is an advisory implementor hint pointing at an *implementable* kind — it is **not** the signature's underlying type (a signature has no underlying type). Revisit once mthds-plugins no longer depends on it; likely fold any hint into `description` rather than a typed field.-- **`pipe_category` is itself redundant** — it must equal `PipeType(type).category` for executable pipes and is now `None` for signatures. It could later be derived rather than stored. Not in this change.-- **`is_controller` class-conversion** (Step 1b) can be deferred if minimizing diff.--## Touch-point index--| File | What |-| --- | --- |-| `pipelex/core/pipes/pipe_blueprint.py` | shrink both enums; `PipeBlueprint.is_signature` → `False`; widen `validate_pipe_type` / `validate_pipe_category`; `validate_pipe_category_based_on_type` early-return; add `PIPE_SIGNATURE_TYPE_TAG` |-| `pipelex/core/pipes/pipe_abstract.py` | `is_signature` → `False` base; same validator widening; (1b) `is_controller` → `False` base |-| `pipelex/pipe_signature/pipe_signature.py` | `pipe_category: None = None`; `is_signature` override → `True` |-| `pipelex/pipe_signature/pipe_signature_blueprint.py` | `pipe_category: None = Field(default=None, exclude=True)`; `is_signature` override → `True`; delete guard validator |-| `pipelex/pipe_signature/pipe_signature_factory.py` | no change (still passes `signature_for`) |-| `pipelex/pipe_controllers/pipe_controller.py` | (1b only) `is_controller` override → `True` |-| `pipelex/core/pipes/pipe_factory.py` | branch on `is_signature` around the `PipeType` coercion |-| `pipelex/core/pipes/output/output_renderer.py` | early `is_signature` return; drop `PIPE_SIGNATURE` match arm |-| `pipelex/builder/pipe/pipe_spec.py` | widen `validate_pipe_type` |-| `pipelex/builder/pipe/pipe_signature_spec.py` | delete guard validator; soften docstring; keep tags + `signature_for` |-| `derived/mthds_schema.json` | regenerate |-| tests + docs + CHANGELOG | per Steps 8-9 |--`registry_models.py` (`PIPE_SIGNATURES` / `PIPE_SIGNATURES_FACTORY` ClassVars) **stays** — keeping signatures as their own registry group reinforces "not an operator/controller."--## Checklist--- [x] Step 1 — `is_signature` class-based (abstract + blueprint + overrides); (1b) `is_controller` — **1b skipped on purpose** (see below)-- [x] Step 2 — `PIPE_SIGNATURE` out of `PipeType` + `PipeCategory` (+ match arms)-- [x] Step 3 — base validators tolerate the signature tag + `None` category + early-return-- [x] Step 4 — factory branch-- [x] Step 5 — signature `pipe_category` → `None` (keep blueprint `exclude=True`)-- [x] Step 6 — `output_renderer` guard-- [x] Step 7 — delete `signature_for` guards, keep the field-- [x] Step 8 — spec `validate_pipe_type` widened-- [x] ⛳ `make tb` + schema regen (no diff) + `make agent-check` + `make agent-test` green-- [x] Step 9 — docs / CHANGELOG / TODOS reframed--## As-built notes--- **Step 1b (class-based `is_controller`) skipped — deliberate.** The plan flagged 1b as optional cleanliness, not correctness. Doing it would turn `PipeCategory.is_controller` and `PipeCategory.is_controller_by_str` into dead code (their only remaining caller is `PipeAbstract.is_controller`), and removing that dead code would diverge from Step 2's "edit the `is_controller` match arm" instruction. Keeping the base `PipeAbstract.is_controller` reading `PipeCategory.is_controller_by_str(self.pipe_category)` is already correct for a signature: `pipe_category` is now `None`, `PipeCategory(None)` raises `ValueError`, and the `except ValueError → False` path returns `False`. So `is_controller` stays correct, the two `PipeCategory` helpers stay legitimately used, and there is no dead code. The keystone (`is_signature` by class) is fully implemented. Revisit 1b only if `is_controller` ever needs to stop touching `pipe_category` for another reason.-- **Allowlist expressed honestly.** A module-level `PIPE_SIGNATURE_TYPE_TAG = "PipeSignature"` in `pipe_blueprint.py` is the single source of the signature tag; `validate_pipe_type` on `PipeBlueprint`, `PipeAbstract`, and `PipeSpec` each check `value not in PipeType.value_list() and value != PIPE_SIGNATURE_TYPE_TAG`, and their error messages list `[*PipeType.value_list(), PIPE_SIGNATURE_TYPE_TAG]` so the signature tag shows up as a legal value.-- **`signature_for` rejection is now structural.** Both `reject_signature_for_pipe_signature` guards (blueprint + spec) were deleted; the field is typed `PipeType | None` and `PipeSignature` is no longer a `PipeType`, so Pydantic rejects `signature_for = "PipeSignature"` on its own. Tests were updated to pass the raw string and assert `ValidationError`.-- **Tests updated to the new model:** `test_pipe_blueprint_signature_enums.py` now locks in the eviction (members removed, tag not in either enum, coercion raises); `test_pipe_abstract_signature_surface.py` asserts the base descriptor returns `False` even for a stub whose `pipe_category == "PipeSignature"` (identity is class-based); `test_pipe_signature.py` asserts `type == "PipeSignature"` and `pipe_category is None`.diff --git a/wip/history/registry/README.md b/wip/history/registry/README.mddeleted file mode 100644index a64ab8cb7..000000000--- a/wip/history/registry/README.md+++ /dev/null@@ -1,28 +0,0 @@-# `UsageRegistry` leak fix + distributed cost reporting--Cost reporting was reworked from a leaky per-run, in-process `UsageRegistry` buffer into an **event-sourced artifact that rides back on `PipeOutput`**. The success-path registry leak is gone by removal; a multi-worker (Temporal) run now aggregates usage from every worker into a single end-of-run cost report. A default-on `--costs` switch decouples cost collection from `--graph` over a shared event-log transport, and a cheap deterministic mock validates the distributed path (originally `--mock-inference`; since the unified dry run it is the internal `--dry-run --mock-usage` trigger — see `../../dry-run-refactor/plan-unified-dry-run.md`).--Shipped in **PR #967** on `fix/For-API-update`, with the companion rename **#968** (`GraphContext → TraceContext`, `is_generate_costs → is_generate_usage`).--> **Scope — NDJSON only.** The cost-reporting event log is implemented and validated through **NDJSON**. The end-of-run read-back is backend-agnostic, so the code path covers DynamoDB for free — but making distributed cost reporting actually work on DynamoDB is **deferred** and is not a validation target of this work.--## What we built--- [`cost-reporting-overview.html`](cost-reporting-overview.html) — **the illustrated overview. Start here.** TL;DR, the two-bugs-one-lifecycle framing, before/after architecture, the shared-transport `--costs` switch, the cross-worker flow, the critical diffs, and the deferred items.--## Shipped follow-ups--- [`cost-report-submitter-helper-handoff.md`](cost-report-submitter-helper-handoff.md) — ✅ the one-arg `render_cost_report_for_output(pipe_output)` submitter helper (the embedder-ergonomics facet): submitters render the standard cost report from a finished output in one call, with the `--costs` gate read off the output (`tokens_usages is None`) instead of re-read from config. Landed on `feature/Cost-report-helper`; the downstream cocode delegate collapse is still pending. Composes with — does not resolve — decisions #3/#6 below.--## Open follow-ups--- [`deferred-followups.md`](deferred-followups.md) — the deliberate non-goals: the costs-only in-memory tracer skip (E1 deep half), cost-per-node correlation, the A1 `TraceContext.from_execution_config` factory, T5, T3 request-scoped tracing state, the costs-only event-log explicit-close cleanup (PR #977 re-review), `--mock-inference` coverage, and the leftover `graph_id → run_id` rename + shelved `TraceContext` split.-- [`cost-report-deferred-decisions.md`](cost-report-deferred-decisions.md) — two deferred **design decisions**: (#3) cost reporting is coupled to `tracing_config.is_enabled` (costs-on + tracing-off ⇒ no report); (#6) the agent CLI gates costs on the raw `costs` param while the main CLI gates on the resolved `is_generate_usage`. Both want a deliberate call.-- [`usage-reporting-without-cost.md`](usage-reporting-without-cost.md) — capability idea (low priority): make usage (token counts) a separately-requestable output, with cost as one view layered on top.--## Companion track--The cost-reporting half mirrors graph-spec assembly, documented in the distributed-execution feature:--- [`../../distributed-execution/tracing-cost-reporting.md`](../../distributed-execution/tracing-cost-reporting.md) — as-built tracing & cost reporting (T2 cross-worker cost-report assembly = **FIXED** by this work).-- [`../../distributed-execution/README.md`](../../distributed-execution/README.md) — the priority plan (P0.1 `--mock-inference` and P1 cost-report assembly both **shipped** here).diff --git a/wip/history/registry/cost-report-deferred-decisions.md b/wip/history/registry/cost-report-deferred-decisions.mddeleted file mode 100644index 9f9b8b477..000000000--- a/wip/history/registry/cost-report-deferred-decisions.md+++ /dev/null@@ -1,64 +0,0 @@-# Cost reporting — deferred design decisions (Phase 3 review)--Two findings from the Phase 3 code review (event-sourced cost-report rendering) are **design decisions, not bugs to reflexively patch**. They are deferred here so they are not lost. The review's correctness/cleanup findings (#1 free-model suppression, #2 try-scope, #4/#7 aggregation consolidation) were fixed in the Phase 3 change itself; these two need a deliberate call.--> **Naming (post-#968):** the internal gate was renamed `is_generate_costs` → `is_generate_usage` (config field on `PipelineExecutionConfig`) and the override param `generate_costs` → `generate_usage` (`with_execution_overrides`); the symbol names below are current. Unchanged: the public `--costs` / `--no-costs` flag and the renderer's `render_run_cost_report(is_generate_costs=…)` param (the cost domain keeps its cost name).--Both touch the same question the registry track is converging on: *what is the single, intentional trigger for "collect and report costs"?* Right now that intent is spread across three switches — `--costs` / `is_generate_usage`, `tracing_config.is_enabled`, and the agent CLI's own `costs` param default — and they don't fully agree.-------## #3 — Cost reporting is coupled to `tracing_config.is_enabled`--**What.** After the Phase 3 cutover, the *only* cost renderer reads `pipe_output.tokens_usages`, which is assembled from the trace-event stream. That stream only exists when tracing is enabled. So with `[pipelex.tracing_config] is_enabled = false` but `--costs` on (the default), **no cost report is produced at all** — silently.--**Evidence.**--- `pipelex/pipeline/pipeline_run_setup.py:193` — the event log is created only `if tracing_config.is_enabled:`; otherwise `event_log` stays `None`.-- `pipelex/pipeline/pipeline_run_setup.py:231` — usage events are emitted only `if is_generate_usage and event_log is not None:` → no event log ⇒ no usage events.-- `pipelex/pipe_run/tracing_assembly.py:80` — `assemble_tracing` early-returns `if not tracing_config.is_enabled:`, leaving `tokens_usages = None`.-- `pipelex/reporting/cost_report_renderer.py` — `render_run_cost_report` no-ops on `not tokens_usages`.--**Why it's a regression.** Before Phase 3, the cost report was rendered from the live `UsageRegistry`, which `_report_*_job` populated directly — independent of tracing. So `pipelex run --costs` printed a table even with tracing disabled. Now it doesn't. Default config has `is_enabled = true`, so the default experience is fine; the gap is for embedders who turn tracing off (e.g. to avoid NDJSON files) and reasonably expect `--costs` to keep working.--**The design question.** Is `--costs` *meant* to require the event-log transport?--- **Option A — accept the coupling, make it honest.** Treat tracing as the cost transport (which is the Phase 3/4 architecture). Then: when `is_generate_usage` is on but `tracing_config.is_enabled` is off, emit a one-time `log.warning` ("costs requested but tracing is disabled; no cost report will be produced — enable tracing or disable --costs"), and document the dependency in `reporting-config.md` / the `--costs` help text. Cheapest, keeps one transport.-- **Option B — decouple at setup.** When `is_generate_usage` is on, force the event log into existence even if `tracing_config.is_enabled` is off (a costs-only, in-memory event log that never writes NDJSON), and let `assemble_tracing` run for the usage concern regardless of `is_enabled`. Restores the pre-Phase-3 "costs work without tracing" guarantee at the cost of a second reason the event log can exist.-- **Option C — separate enable flags.** Split "emit trace events to a backend (NDJSON/DDB)" from "assemble in-memory usage for reporting." `is_enabled` governs the former; `is_generate_usage` governs the latter. Cleanest conceptually, most config surface.--**Lean:** A for now (it matches the locked Option-B-events-as-single-source direction and is a few lines), with C as the principled end state if embedders hit it. Decide before Phase 4 removes the registry — after that there is no fallback renderer, so the coupling becomes the only behavior.-------## #6 — Agent CLI gates costs on the raw `costs` param; main CLI gates on the resolved config--**What.** The two CLIs resolve "should we collect/report costs" differently:--- **Main CLI** (`pipelex run`): the renderer is gated on `execution_config.is_generate_usage` — the value *resolved* from the `--costs/--no-costs` tri-state over the config default. A user who sets `is_generate_usage = false` in config and passes no flag gets no costs.-- **Agent CLI** (`pipelex-agent run`): `costs` is a hard-coded `bool = True` parameter (see `run_pipeline_core` / the `pipe_cmd`/`bundle_cmd`/`method_cmd` defaults). It is forced into the config via `with_execution_overrides(generate_usage=costs)` and also gates `build_cost_summary` directly (`pipelex/cli/agent_cli/commands/run/_run_core.py:111`). It **never consults the config default** `is_generate_usage`.--**Consequence.** An embedder who disables cost collection globally (`[pipeline_execution_config] is_generate_usage = false`, e.g. to cut event-log overhead) gets it honored by `pipelex run` but **not** by `pipelex-agent run` — the agent keeps emitting usage events and attaching a `cost_report` to every `--with-memory` envelope / on-disk JSON unless `--no-costs` is passed explicitly.--**History.** This predates Phase 3 (the agent `--costs` default-True override landed in Phase 1). Phase 3 only made it *observable*, because it surfaces a `cost_report` object in the agent output where before nothing was shown.--**The design question.** Should the agent CLI honor the config `is_generate_usage` default like the main CLI, or is "agent always collects unless told not to" intentional?--- **Option A — make the agent tri-state like the main CLI.** Change the agent `costs` param to `bool | None = None` and resolve it through `with_execution_overrides` (None ⇒ use config default). One-line-ish per command + the `_run_core` gate reads the resolved `execution_config.is_generate_usage`. Makes the two surfaces consistent.-- **Option B — keep agent default-on by design, document it.** If the agent is meant to be maximally informative for machine consumers regardless of an embedder's human-CLI preference, leave it but document the divergence in `agent_cli/CLAUDE.md` so it's a choice, not an accident.--**Lean:** A — consistency across the two surfaces is worth more than the agent being unconditionally chatty, and `--no-costs` still lets a caller force it off. But it's a behavior change for agent consumers, so it needs a deliberate sign-off rather than being folded silently into the Phase 3 fix.--### #6b — Agent `--costs` is a silent no-op in API mode (same root)--**What.** The agent `run` subcommands declare `--costs/--no-costs` unconditionally, but the agent has two runner paths: `run_pipeline_core` (local) and `run_pipeline_core_api` (remote API runner). Only the local path builds an `execution_config` and renders/attaches a cost summary. In **API mode** the flag is accepted and then **silently ignored** — execution happens remotely and the local `costs` value does nothing. Flagged by cubic (P2, `pipe_cmd.py:51`) and Greptile's summary ("the agent API runner accepts cost flags but does not apply or report them", `_run_core_api.py`) on PR #967.--**Why it's a design decision, not a reflexive patch.** "Rejecting `--costs` as unsupported in API mode" needs to distinguish an *explicitly-passed* flag from the `True` default — but the agent uses `bool = True`, so a user passing `--costs` is indistinguishable from the default (the exact same ambiguity #6 Option A would resolve by going tri-state `bool | None`). And whether API-mode cost reporting should instead surface costs *from the API response* is a product call, not a lint fix. So #6b is a strict consequence of #6: making the agent `costs` tri-state (#6 Option A) is the prerequisite for any honest API-mode handling (warn / reject / surface-from-response). Resolve them together; do not bolt a one-off rejection onto the `True`-default param.--**Broader framing (found at PR #967 final review): API-mode flag handling is inconsistent across the board.** The `RunnerType.API` branch in the agent `run` subcommands *rejects* `--dry-run` and `--mock-inputs` (`agent_error("... not supported with --runner api")`) but *silently ignores* `--graph` AND `--costs` (neither is threaded into `run_pipeline_core_api`). So a one-off `--no-costs` rejection would deepen the inconsistency (why reject `--no-costs` but not `--graph`?). The right fix is one holistic decision about how the agent CLI handles local-runner-only flags under `--runner api` — reject them all uniformly, or document them all as no-ops — covering `--graph` and `--costs` together. That is the deliberate call #6/#6b is waiting on; a partial patch on just `--costs` is explicitly the wrong move.-------## Cross-cutting note--Both findings are facets of the same gap: there is no *single owner* of "is cost collection/reporting on for this run?" The value is currently derived in three places (config default, the two CLIs' flag resolution, and the tracing toggle) and they disagree at the edges. Whoever takes the Phase 4 registry removal should consider collapsing this into one resolved predicate on `PipelineExecutionConfig` that both CLIs and the assembly path read — at which point #3 and #6 both fall out as consequences of one decision.diff --git a/wip/history/registry/cost-report-submitter-helper-handoff.md b/wip/history/registry/cost-report-submitter-helper-handoff.mddeleted file mode 100644index 2a894f5c2..000000000--- a/wip/history/registry/cost-report-submitter-helper-handoff.md+++ /dev/null@@ -1,86 +0,0 @@-# Handoff — one-arg `render_cost_report_for_output(pipe_output)` submitter helper--**Status: ✅ Shipped** (`feature/Cost-report-helper`). `render_cost_report_for_output` is in `pipelex/reporting/cost_report_renderer.py`, the `pipelex run` CLI (`_run_core.py`) uses it, `render_run_cost_report` is unchanged as the primitive, unit tests pin the gate-from-output derivation (`tests/unit/pipelex/reporting/test_render_cost_report_for_output.py`), and the as-built overview + `docs/features/cost-tracking.md` document the surface. Downstream cocode collapse is still pending (re-pin `pipelex`, then delete cocode's local `_render_cost_report` gate). The rationale below is kept because it composes with the still-open decisions #3/#6 in `cost-report-deferred-decisions.md`.--**Task:** add a thin, self-contained cost-report rendering helper to `pipelex/reporting/cost_report_renderer.py` that takes only a finished `PipeOutput`, so every "submitter" (the `pipelex run` CLI, and external embedders like cocode) stops re-deriving the same three arguments — and stops re-reading global config to reconstruct a decision the run already recorded on the output.--This is the **submitter-ergonomics facet** of the same theme as the deferred decisions in this registry (`cost-report-deferred-decisions.md` #3/#6, and its cross-cutting note: "there is no single owner of *is cost reporting on for this run?*"). It does not resolve #3 or #6 — it is a smaller, additive convenience that should land on its own, and it composes cleanly with whatever those decisions settle.--## Where this came from--Bumping the `pipelex` dependency in the sibling `../cocode/` repo broke its build: the new `ReportingProtocol` (post the event-sourced cost-report rework, PR #967) no longer has `generate_report()`. cocode migrated its call sites to `render_run_cost_report(...)`, and to do so it had to write this glue (`../cocode/cocode/swe/swe_cmd.py`, `_render_cost_report`):--```python-render_run_cost_report(- pipeline_run_id=pipe_output.pipeline_run_id,- tokens_usages=pipe_output.tokens_usages,- is_generate_costs=get_config().pipelex.pipeline_execution_config.is_generate_usage,-)-```--Two smells fall out of that:--1. **Duplication.** All three arguments are derivable from a finished run. The `pipelex run` CLI writes the same unpacking by hand (`pipelex/cli/commands/run/_run_core.py:320`). Every embedder that wants the standard cost report has to reproduce it.-2. **Wrong source of truth for the gate.** cocode re-reads global config (`is_generate_usage`) to reconstruct whether costs were on — but the runner already made that decision at run time, with all `--costs/--no-costs` override resolution applied, and **recorded it on the output**: `PipeOutput.tokens_usages` is `None` precisely when cost reporting was off. See the field docstring at `pipelex/core/pipes/pipe_output.py:28-32`:-- > the submitter renders the cost report from this field. None when cost reporting was off or the run emitted no trace events at all; an empty list when on with events present but no inference happened.-- Reading config afterwards re-derives a fact the output already carries — correct only when config wasn't overridden and wasn't mutated between run and render. The output is authoritative; config is not.--## Proposed change--Add a free function next to `render_run_cost_report` (do **not** add a method on `PipeOutput` — that would couple the core model to the renderer's deps: config, `CostRegistry`, Rich; reporting may depend on core, not the reverse):--```python-def render_cost_report_for_output(pipe_output: PipeOutput) -> None:- """Render the end-of-run cost report from a finished output (self-contained).-- The gate is read off the output, not config: pipe_output.tokens_usages is None- exactly when cost reporting was off for this run (see PipeOutput.tokens_usages).- """- render_run_cost_report(- pipeline_run_id=pipe_output.pipeline_run_id,- tokens_usages=pipe_output.tokens_usages,- is_generate_costs=pipe_output.tokens_usages is not None,- )-```--Keep `render_run_cost_report(*, pipeline_run_id, tokens_usages, is_generate_costs)` exactly as-is — it stays the **low-level primitive** for the cases where the three values genuinely come from different places: the agent CLI's machine-readable envelope path (`build_cost_summary`, `pipelex/cli/agent_cli/commands/run/_run_core.py`) and any distributed/Temporal reassembly where the output isn't the single carrier.--Then collapse the `pipelex run` call site (`_run_core.py:320`) to `render_cost_report_for_output(pipe_output)`.--## Why deriving the gate from the output is behaviorally identical to the current CLI gate--`render_run_cost_report` already ANDs both guards (`if not is_generate_costs or not tokens_usages: return`, `cost_report_renderer.py:50`), so for every run produced by the standard runner the *rendered outcome* is unchanged when the gate is read from `tokens_usages` instead of from `execution_config.is_generate_usage`:--- usage on, inference happened → `tokens_usages` non-empty → render (both agree).-- usage on, dry run (no inference) → `tokens_usages == []` → `is not None` True, then `has_reportable_usage` suppresses (both agree).-- usage on but tracing disabled (decision **#3**) → `tokens_usages is None` → no render. The current CLI gate is `True` here but the renderer no-ops on `not tokens_usages` anyway → same outcome. **The helper inherits #3, it does not worsen it.**-- usage off → `tokens_usages is None` → no render (both agree).--So this is a pure de-duplication + correct-source-of-truth change, not a behavior change.--## Scope / non-goals--- **Do not** fold in decisions #3 (cost reporting coupled to `tracing_config.is_enabled`) or #6 (agent-CLI vs main-CLI gate divergence). Those are separate deliberate calls tracked in `cost-report-deferred-decisions.md`. This helper is render-surface only (console/CSV) and is orthogonal to the agent CLI's `build_cost_summary` JSON path.-- **Do not** change the signature of `render_run_cost_report`.-- **Do not** add a `PipeOutput` method (layering — see above).-- This is complementary to the cross-cutting note's "one resolved predicate on `PipelineExecutionConfig`": even after that lands, the *render-time* consumer should read the resolved decision off the output, which already records it.--## Read first--- `cost-reporting-overview.html` — as-built architecture (start here for the event-sourced model).-- `cost-report-deferred-decisions.md` — #3 and #6, the adjacent gate-source decisions; this handoff is the third, smaller facet.-- `deferred-followups.md` — T3 request-scoped tracing state (the broader "single owner" refactor this rolls toward).--## Acceptance criteria--- `render_cost_report_for_output(pipe_output)` added to `pipelex/reporting/cost_report_renderer.py`; `render_run_cost_report` unchanged.-- `pipelex run` (`_run_core.py`) uses the new helper; no behavior change to its console/CSV output (verify with a real run under `--costs`, `--no-costs`, `--dry-run`, and a free/zero-price model).-- No new import cycle (`reporting` → `core.pipe_output` is fine; `core.pipe_output` must not import the renderer).-- Docs updated in `pipelex/docs/` where the cost-report rendering surface is described.-- `make check` / `make test` green.--## Downstream (sibling repo, not part of this change)--Once shipped and pipelex is re-pinned in `../cocode/`, cocode's `_render_cost_report` collapses to a one-line delegate to `render_cost_report_for_output(...)` and then deletes — its current config-reading gate goes away. No action needed in pipelex; noting it so the API shape stays embedder-friendly (the one-arg form is the whole point).diff --git a/wip/history/registry/cost-reporting-overview.html b/wip/history/registry/cost-reporting-overview.htmldeleted file mode 100644index 729e03dee..000000000--- a/wip/history/registry/cost-reporting-overview.html+++ /dev/null@@ -1,507 +0,0 @@-<!DOCTYPE html>-<html lang="en">-<head>-<meta charset="utf-8">-<meta name="viewport" content="width=device-width, initial-scale=1.0">-<title>Cost Reporting Rework — UsageRegistry leak fix + distributed cost reporting</title>-<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>-<style>- :root{- --bg:#0f1220; --panel:#171a2b; --panel-2:#1d2136; --ink:#e7e9f3; --muted:#9aa0bd;- --line:#2a2f4a; --accent:#6c7bff; --accent-2:#3ad1c4; --warn:#ffb454; --danger:#ff6b6b;- --good:#56d364; --add-bg:#10301c; --add-ink:#9be9a8; --del-bg:#3a1620; --del-ink:#ff9aa6;- --code:#c9d1ff;- --maxw:1080px;- }- *{box-sizing:border-box}- html{scroll-behavior:smooth}- body{- margin:0; background:linear-gradient(180deg,#0c0e1a,#0f1220 240px); color:var(--ink);- font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;- -webkit-font-smoothing:antialiased;- }- a{color:var(--accent-2);text-decoration:none}- a:hover{text-decoration:underline}- .wrap{max-width:var(--maxw);margin:0 auto;padding:0 22px 96px}- header.hero{- max-width:var(--maxw);margin:0 auto;padding:54px 22px 26px;- }- .eyebrow{letter-spacing:.16em;text-transform:uppercase;font-size:12px;color:var(--accent);font-weight:700}- h1{font-size:34px;line-height:1.15;margin:.35em 0 .25em;letter-spacing:-.01em}- h1 .sub{display:block;font-size:18px;color:var(--muted);font-weight:500;margin-top:.5em}- .badges{display:flex;flex-wrap:wrap;gap:8px;margin-top:18px}- .badge{font-size:12.5px;font-weight:600;padding:5px 11px;border-radius:999px;border:1px solid var(--line);background:var(--panel);color:var(--ink)}- .badge.green{border-color:#1f5132;background:#10271a;color:var(--good)}- .badge.blue{border-color:#28346b;background:#141a36;color:#aeb8ff}- .badge.amber{border-color:#5a4420;background:#2a2012;color:var(--warn)}-- h2{font-size:24px;margin:54px 0 14px;padding-top:10px;letter-spacing:-.01em;border-top:1px solid var(--line)}- h2 .n{color:var(--accent);font-variant-numeric:tabular-nums;margin-right:.5ch}- h3{font-size:18px;margin:28px 0 10px}- p{margin:.7em 0}- ul,ol{margin:.6em 0;padding-left:1.3em}- li{margin:.3em 0}- code{font-family:"SF Mono",SFMono-Regular,Menlo,Consolas,monospace;font-size:.88em;background:#0c1024;border:1px solid var(--line);border-radius:5px;padding:1px 5px;color:var(--code)}- strong{color:#fff}- .lead{font-size:17.5px;color:#cfd3ee}-- .card{background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:20px 22px;margin:18px 0}- .card.tldr{background:linear-gradient(160deg,#191d33,#15182a);border-color:#33407e}- .tldr h2{border:0;margin:0 0 8px;padding:0;font-size:21px}- .tldr ul{margin:.4em 0}- .kpis{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:12px;margin-top:16px}- .kpi{background:var(--panel-2);border:1px solid var(--line);border-radius:10px;padding:13px 14px}- .kpi b{display:block;font-size:21px;color:#fff;line-height:1.1}- .kpi span{font-size:12.5px;color:var(--muted)}-- .callout{border-radius:12px;padding:16px 18px 16px 18px;margin:20px 0;border:1px solid;position:relative}- .callout .tag{font-size:11.5px;font-weight:800;letter-spacing:.08em;text-transform:uppercase;display:inline-block;margin-bottom:6px}- .callout.scope{background:linear-gradient(120deg,#2a2012,#241b10);border-color:#6a5223}- .callout.scope .tag{color:var(--warn)}- .callout.info{background:#10182f;border-color:#274073}- .callout.info .tag{color:#7fb0ff}- .callout.good{background:#0f2519;border-color:#1f5132}- .callout.good .tag{color:var(--good)}- .callout p:first-of-type{margin-top:0}- .callout p:last-child{margin-bottom:0}-- .mermaid{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:18px;margin:18px 0;overflow-x:auto;text-align:center}-- /* tabs */- .tabs{margin:18px 0}- .tabs .tabbar{display:flex;gap:6px;flex-wrap:wrap;border-bottom:1px solid var(--line);margin-bottom:0}- .tabs .tabbtn{appearance:none;border:1px solid var(--line);border-bottom:none;background:var(--panel);color:var(--muted);- font:inherit;font-size:13.5px;font-weight:600;padding:8px 14px;border-radius:9px 9px 0 0;cursor:pointer;margin-bottom:-1px}- .tabs .tabbtn[aria-selected="true"]{background:var(--panel-2);color:#fff;border-color:var(--accent)}- .tabs .tabpane{display:none;background:var(--panel-2);border:1px solid var(--line);border-radius:0 10px 10px 10px;padding:4px 0 0}- .tabs .tabpane.active{display:block}- .tabpane .mermaid{border:0;background:transparent;margin:6px 0}-- /* code / diff */- pre{margin:0;background:#0b0e1f;border:1px solid var(--line);border-radius:10px;overflow-x:auto}- pre.code{padding:14px 16px}- pre code{background:none;border:none;padding:0;color:#cdd3f5;font-size:13px;line-height:1.7;display:block}- .filename{font-family:"SF Mono",monospace;font-size:12px;color:var(--muted);background:#0c1024;border:1px solid var(--line);border-bottom:none;border-radius:9px 9px 0 0;padding:7px 13px;display:inline-block;margin-bottom:-1px}- .filename + pre{border-radius:0 10px 10px 10px}- .diff{padding:12px 0;font-size:13px;line-height:1.65}- .diff .ln{display:block;padding:0 16px;white-space:pre;font-family:"SF Mono",monospace;color:#aeb4d6}- .diff .add{background:var(--add-bg);color:var(--add-ink)}- .diff .del{background:var(--del-bg);color:var(--del-ink)}- .diff .ctx{color:#8b91b4}- .diff .cmt{color:#6f76a0;font-style:italic}-- details{background:var(--panel);border:1px solid var(--line);border-radius:11px;margin:14px 0;overflow:hidden}- details[open]{background:var(--panel)}- summary{cursor:pointer;padding:14px 18px;font-weight:600;list-style:none;display:flex;align-items:center;gap:10px}- summary::-webkit-details-marker{display:none}- summary::before{content:"▸";color:var(--accent);font-size:13px;transition:transform .15s}- details[open] summary::before{transform:rotate(90deg)}- summary:hover{background:var(--panel-2)}- .details-body{padding:2px 18px 18px}-- table{border-collapse:collapse;width:100%;margin:16px 0;font-size:14.5px;overflow:hidden;border-radius:10px}- th,td{border:1px solid var(--line);padding:9px 12px;text-align:left;vertical-align:top}- th{background:var(--panel-2);color:#fff;font-weight:600}- td code{font-size:.85em}- tr:nth-child(even) td{background:#13162600}- .pill{font-size:11px;font-weight:700;padding:2px 8px;border-radius:999px;white-space:nowrap}- .pill.add{background:#10271a;color:var(--good);border:1px solid #1f5132}- .pill.rm{background:#2c1620;color:#ff9aa6;border:1px solid #5a2233}- .pill.chg{background:#241c12;color:var(--warn);border:1px solid #5a4420}-- .two{display:grid;grid-template-columns:1fr 1fr;gap:16px}- @media(max-width:760px){.two{grid-template-columns:1fr}}- .mini{font-size:13.5px;color:var(--muted)}- hr.soft{border:0;border-top:1px solid var(--line);margin:30px 0}- .footer{margin-top:60px;padding-top:18px;border-top:1px solid var(--line);color:var(--muted);font-size:13px}- .anchor-note{color:var(--muted);font-size:13.5px}- kbd{font-family:"SF Mono",monospace;font-size:.82em;background:#0c1024;border:1px solid var(--line);border-bottom-width:2px;border-radius:5px;padding:1px 6px;color:#dfe3ff}-</style>-</head>-<body>--<header class="hero">- <div class="eyebrow">Pipelex · cost-reporting rework</div>- <h1>From a leaky in-process registry to event-sourced cost on the run result- <span class="sub">The <code>UsageRegistry</code> success-path leak fix & distributed cost reporting — one piece of work</span>- </h1>- <div class="badges">- <span class="badge green">Status: shipped on <code>fix/For-API-update</code></span>- <span class="badge blue">PR #967 (core) · #968 (rename)</span>- <span class="badge amber">Validated backend: NDJSON only — DynamoDB deferred</span>- </div>-</header>--<div class="wrap">-- <!-- ============ TL;DR ============ -->- <div class="card tldr">- <h2>TL;DR</h2>- <p class="lead">Cost reporting used to ride on a per-run, in-process <code>UsageRegistry</code> buffer. That buffer <strong>leaked on every successful run</strong> (it was only closed on the failure path) and it <strong>could not see usage from other worker processes</strong>, so distributed (Temporal) runs never reported their true cost. We removed the registry entirely and made cost an <strong>event-sourced artifact that rides back on <code>PipeOutput</code></strong>, exactly like <code>graph_spec</code> already does.</p>- <ul>- <li><strong>The leak is gone by removal.</strong> No submitter-side registry is opened anymore, so there is nothing to leak — the bug is structurally impossible, not just patched.</li>- <li><strong>Cost now rides on the run result.</strong> New <code>PipeOutput.tokens_usages</code> is assembled from the trace-event stream at end-of-run, in <em>both</em> direct and Temporal modes. A multi-worker run aggregates usage from every worker into a <strong>single end-of-run cost report</strong>.</li>- <li><strong>New <code>--costs / --no-costs</code> switch (default on)</strong> decouples cost collection from <code>--graph</code> over a shared event-log transport. The old <code>--cost-report</code> flag is removed (no back-compat).</li>- <li><strong><code>--mock-inference</code></strong> fakes the LLM leaf with reportable synthetic usage so the cross-worker path can be validated cheaply and deterministically — no tokens billed.</li>- <li><strong>Companion rename (#968):</strong> <code>GraphContext → TraceContext</code> and the internal gate <code>is_generate_costs → is_generate_usage</code> — the object carries two streams (graph + usage), so it's named for what it is.</li>- </ul>-- <div class="callout scope" style="margin-bottom:0">- <span class="tag">⚠ Read this — scope of what shipped</span>- <p style="margin-bottom:0"><strong>We only implemented & validated the cost-reporting event log through NDJSON.</strong> The read-back is backend-agnostic, so the <em>code path</em> covers DynamoDB for free — but <strong>making distributed cost reporting actually work on DynamoDB is deferred</strong> and is <em>not</em> a validation target of this work. Local NDJSON is the one backend we care about and test here.</p>- </div>-- <div class="kpis">- <div class="kpi"><b>0</b><span>submitter registries opened (was 1/run, leaked)</span></div>- <div class="kpi"><b>1</b><span>aggregated cost report per run, even cross-worker</span></div>- <div class="kpi"><b>NDJSON</b><span>validated transport · DynamoDB deferred</span></div>- <div class="kpi"><b>−175</b><span>lines deleted from <code>reporting_manager.py</code></span></div>- </div>- </div>-- <!-- ============ THE PROBLEM ============ -->- <h2><span class="n">1.</span>Two bugs that were really one lifecycle</h2>- <p>The investigation started as a narrow "the registry leaks on success" bug. On closer look it was the <em>same</em> object lifecycle as the long-deferred <strong>distributed cost-report aggregation</strong>. The <code>UsageRegistry</code> is a per-run, in-process cost buffer whose full lifecycle is <strong>open → populate → report → close</strong>. Two different steps were broken:</p>- <ul>- <li><strong>The leak</strong> = the <em>close</em> step never ran on the success path. The registry was opened in <code>pipeline_run_setup</code> but closed only inside <code>if not success:</code>. In a long-lived process (e.g. the Pipelex API) this orphaned a registry on every request, and reusing a <code>pipeline_run_id</code> could even collide on the leftover.</li>- <li><strong>Distributed cost reporting</strong> = the cross-worker <em>replay-populate</em> step was never wired. Usage events from inference activities on separate worker processes landed in the backend but were never read back into a report. Cross-worker runs reported nothing.</li>- </ul>- <p>Because they're the same lifecycle, fixing them separately would have meant baking in registry-placement decisions for the leak that the cost-reporting design then had to unwind. So they were designed and built together.</p>-- <div class="mermaid">-flowchart LR- O([open]):::ok --> P([populate]):::ok --> R([report]):::ok --> C([close]):::bad- P -. "replay-populate from<br/>other workers — never wired" .-> X[/distributed cost<br/>reporting missing/]:::bad- C -. "never runs on success" .-> L[/registry leak/]:::bad- classDef ok fill:#10271a,stroke:#1f5132,color:#9be9a8;- classDef bad fill:#3a1620,stroke:#5a2233,color:#ff9aa6;- </div>- <p class="anchor-note">Same buffer, same lifecycle, two missing steps — the unifying insight behind treating the leak fix and distributed cost reporting as one piece of work.</p>-- <!-- ============ THE FIX ============ -->- <h2><span class="n">2.</span>The fix — mirror graph-spec assembly (Option B)</h2>- <p>Graph tracing had already solved the exact cross-worker problem: "analyze the event stream at the end of the run and ride the artifact back on <code>PipeOutput</code>." Cost reporting now does the literal same thing. We <strong>removed the submitter-side registry</strong> and instead assemble usage from the trace-event stream at the same hook that assembles the graph, attaching it to <code>PipeOutput.tokens_usages</code>. The submitter renders the cost report from that field.</p>-- <div class="callout good">- <span class="tag">✓ Why removal beats "remember to close"</span>- <p style="margin-bottom:0">With nothing populating a submitter-side registry, there is no registry to leak — on every entry point and both backends. The live <code>_try_add_to_registry</code> path and the dead <code>inject_tokens_usages</code> retired with it. The leak fix is a <strong>structural guarantee</strong>, not a discipline you can forget.</p>- </div>-- <h3>Before vs after</h3>- <div class="tabs" id="arch">- <div class="tabbar" role="tablist">- <button class="tabbtn" role="tab" aria-selected="true" data-tab="arch-before">Before — side buffer (leaks)</button>- <button class="tabbtn" role="tab" aria-selected="false" data-tab="arch-after">After — rides on PipeOutput</button>- </div>- <div class="tabpane active" id="arch-before" role="tabpanel">- <div class="mermaid">-flowchart TB- subgraph SUB["submitter process"]- RS["pipeline_run_setup<br/>open_registry()"] --> REG[("UsageRegistry<br/>per-run buffer")]- INF["in-process inference<br/>_try_add_to_registry"] --> REG- REND["generate_report()"] --> REG- RS -. "close_registry()<br/><b>only if not success</b>" .-> REG- end- REG ==> LEAK["💧 success path:<br/>never closed = LEAK"]:::bad- WK["usage from other<br/>worker processes"] -. "emitted, never read back" .-> NOPE["✗ no cross-worker report"]:::bad- classDef bad fill:#3a1620,stroke:#5a2233,color:#ff9aa6;- </div>- </div>- <div class="tabpane" id="arch-after" role="tabpanel">- <div class="mermaid">-flowchart TB- INF["inference anywhere<br/>(DIRECT or any worker)"] -->|"UsageReportEvent"| EL[("event log<br/>NDJSON partition")]- EL -->|"read once at end of run"| AS["assemble_tracing"]- AS -->|"UsageAggregator"| TU["tokens_usages"]- AS -->|"GraphSpecAssembler"| GS["graph_spec"]- TU --> PO[["PipeOutput"]]- GS --> PO- PO -->|"flows back to submitter"| RR["render_run_cost_report()"]:::ok- classDef ok fill:#10271a,stroke:#1f5132,color:#9be9a8;- </div>- </div>- </div>- <p class="mini">Direct mode assembles in the submitter process (<code>PipeRun.run</code> <code>finally</code>); Temporal mode assembles in the <code>act_assemble_tracing</code> activity (renamed from <code>act_assemble_graph</code>) and the workflow sets both artifacts on <code>PipeOutput</code> before returning. Same hook, same flow, one mental model.</p>-- <!-- ============ SHARED TRANSPORT ============ -->- <h2><span class="n">3.</span>The load-bearing decoupling — <code>--costs</code> over a shared transport</h2>- <p>This is the one piece that's <strong>not cosmetic</strong>. Previously usage events were only emitted when <code>--graph</code> + tracing were on (the event log was created <em>inside</em> <code>if is_generate_graph</code>). Once the live registry is gone, <code>--no-graph</code> would lose cost entirely. So we added a dedicated, default-on <strong><code>--costs</code></strong> switch that mirrors <code>--graph</code>: the event log is the <strong>shared transport</strong>; each switch gates its own event type and its own assembler; each rides its artifact back on <code>PipeOutput</code>.</p>-- <div class="mermaid">-flowchart TB- G["--graph<br/>(emit_graph_events)"]:::g -->|gates| GE["graph node / edge events"]:::g- C["--costs<br/>(emit_usage_events)"]:::c -->|gates| UE["UsageReportEvent"]:::c- GE --> EL[("shared event log<br/>NDJSON")]- UE --> EL- EL --> GA["GraphSpecAssembler"]:::g --> GS["graph_spec"]:::g- EL --> UA["UsageAggregator"]:::c --> TU["tokens_usages"]:::c- GS --> PO[["PipeOutput"]]- TU --> PO- classDef g fill:#141a36,stroke:#28346b,color:#aeb8ff;- classDef c fill:#0f2519,stroke:#1f5132,color:#9be9a8;- </div>-- <table>- <thead><tr><th>Invocation</th><th>Graph built?</th><th>Cost reported?</th></tr></thead>- <tbody>- <tr><td><code>--graph --costs</code> (defaults)</td><td>yes</td><td>yes</td></tr>- <tr><td><code>--no-graph --costs</code></td><td>no</td><td><strong>yes</strong> — this is what the switch buys</td></tr>- <tr><td><code>--graph --no-costs</code></td><td>yes</td><td>no</td></tr>- <tr><td><code>--no-graph --no-costs</code></td><td>no</td><td>no</td></tr>- </tbody>- </table>- <p class="mini"><strong>D5 detail:</strong> in costs-only mode the graph tracer is still opened with <code>event_log=None</code> — it keeps minting node ids (so each <code>UsageReportEvent</code> can attribute to its <code>node_id</code>) but emits no graph events. The node tree is the shared substrate both streams hang off; that's the deeper reason the rename to <code>TraceContext</code> is the right framing.</p>-- <!-- ============ DISTRIBUTED ============ -->- <h2><span class="n">4.</span>Distributed cost reporting — cross-worker → one report</h2>- <p>In Temporal mode a single logical run can span several processes: a parent workflow on one worker, a child on another, the inference activity on a runner. Every <code>UsageReportEvent</code> lands in <strong>one partition</strong> with a <code>writer_id</code> to disambiguate the producer. At end-of-run, a single event read feeds both assemblers, and the aggregated usage rides home on <code>PipeOutput</code>.</p>-- <div class="mermaid">-sequenceDiagram- autonumber- participant S as Submitter- participant WA as Worker A<br/>(parent workflow)- participant WC as Runner<br/>(inference activity)- participant L as NDJSON log<br/>(one partition)- participant AT as act_assemble_tracing-- S->>WA: dispatch WfPipeRun (--costs on)- WA->>WC: inference activity- WC-->>L: UsageReportEvent (writer_id = act_*)- WA-->>L: UsageReportEvent (parent steps)- Note over L: every worker's usage<br/>lands in one partition- WA->>AT: Step 2 — assemble (single read)- L-->>AT: read_events()- AT-->>WA: TracingAssembly{ graph_spec, tokens_usages }- WA-->>S: PipeOutput.tokens_usages- S->>S: render_run_cost_report(tokens_usages=…)- </div>-- <div class="callout info">- <span class="tag">ℹ Single source of truth = events</span>- <p style="margin-bottom:0">In direct mode, inference no longer <em>also</em> adds to a live registry — that path was removed. Cost always comes from the aggregated event stream, so there is <strong>no double-counting</strong> to guard against. One run, one source.</p>- </div>-- <!-- ============ MOCK INFERENCE ============ -->- <h2><span class="n">5.</span><code>--mock-inference</code> — validate the distributed path for free</h2>- <p>Proving cross-worker cost reporting used to need real LLM spend in a real multi-worker topology. <code>--mock-inference</code> is a <strong>LIVE</strong> run (operators dispatch real activities — the true distributed path is exercised) whose AI call is faked <em>at the inference leaf</em> with <strong>reportable, non-zero synthetic usage</strong>. So a cost report actually renders from cross-worker usage, deterministically, with nothing billed.</p>- <ul>- <li>It is <strong>not</strong> <code>--dry-run</code>: dry-run swaps the generator <em>before</em> dispatch, so it never reaches the activity and emits zero-token (suppressed) usage. Mock-inference keeps <code>run_mode = LIVE</code> and mocks only the leaf — the one thing dry-run can't validate cheaply is a <em>rendered</em> cost report.</li>- <li><strong>LLM leaf only.</strong> Image-gen / extract / search under <code>--mock-inference</code> <strong>fail loud</strong> with <code>MockInferenceUnsupportedError</code> (pointing at <code>--dry-run</code>) rather than silently calling the real provider — their output is stored <em>above</em> the leaf, so a leaf-level mock would push synthetic data through the storage path.</li>- <li>Mutually exclusive with <code>--dry-run</code>; main <code>run</code> subcommands only (agent CLI deferred).</li>- </ul>-- <!-- ============ CRITICAL CODE ============ -->- <h2><span class="n">6.</span>The critical changes, in code</h2>- <p class="mini">Minimal, representative diffs for the load-bearing edits. Symbol names are the durable anchors; line numbers drift.</p>-- <details open>- <summary>① The leak fix — stop opening a registry at all <span class="pill rm" style="margin-left:auto">removal</span></summary>- <div class="details-body">- <p>The asymmetry that caused the leak: <code>open_registry()</code> ran unconditionally during setup, but <code>close_registry()</code> sat behind <code>if not success:</code>. Both are simply gone now.</p>- <span class="filename">pipelex/pipeline/pipeline_run_setup.py</span>- <pre class="diff">-<span class="ln del">- registry_opened = False</span>-<span class="ln ctx"> ...</span>-<span class="ln del">- if execution_config.is_generate_graph:</span>-<span class="ln add">+ is_generate_graph = execution_config.is_generate_graph</span>-<span class="ln add">+ is_generate_usage = execution_config.is_generate_usage</span>-<span class="ln add">+ if is_generate_graph or is_generate_usage:</span>-<span class="ln ctx"> ...</span>-<span class="ln cmt"> # D5: costs-only (--no-graph --costs) → tracer still mints node ids,</span>-<span class="ln cmt"> # but emits no graph events; usage rides via set_event_log below.</span>-<span class="ln del">- event_log=event_log,</span>-<span class="ln add">+ event_log=event_log if is_generate_graph else None,</span>-<span class="ln add">+ emit_graph_events=is_generate_graph,</span>-<span class="ln add">+ emit_usage_events=is_generate_usage,</span>-<span class="ln ctx"> ...</span>-<span class="ln del">- get_report_delegate().open_registry(pipeline_run_id=pipeline_run_id)</span>-<span class="ln del">- registry_opened = True</span>-<span class="ln del">- if event_log is not None:</span>-<span class="ln add">+ if is_generate_usage and event_log is not None:</span>-<span class="ln ctx"> get_report_delegate().set_event_log(...) # the surviving usage transport</span>-<span class="ln ctx"> ...</span>-<span class="ln ctx"> finally:</span>-<span class="ln del">- if registry_opened:</span>-<span class="ln del">- get_report_delegate().close_registry(pipeline_run_id=pipeline_run_id)</span>- </pre>- </div>- </details>-- <details>- <summary>② Usage rides on the run result — new fields on <code>PipeOutput</code> <span class="pill add" style="margin-left:auto">added</span></summary>- <div class="details-body">- <p>A literal mirror of <code>graph_spec</code> / <code>graph_assembly_error</code>. <code>AnyTokensUsage</code> is already a discriminated union of pydantic models, already serialized through the event-log backend — so it crosses the Temporal boundary safely.</p>- <span class="filename">pipelex/core/pipes/pipe_output.py</span>- <pre class="diff">-<span class="ln ctx"> graph_spec: GraphSpec | None = None</span>-<span class="ln ctx"> graph_assembly_error: str | None = None</span>-<span class="ln add">+ # Token usage assembled from the trace-event stream at end of run (mirrors graph_spec):</span>-<span class="ln add">+ # the submitter renders the cost report from this field. None when cost reporting was off</span>-<span class="ln add">+ # or no trace events emitted; an empty list when on but no inference happened.</span>-<span class="ln add">+ tokens_usages: list[AnyTokensUsage] | None = None</span>-<span class="ln add">+ usage_assembly_error: str | None = None</span>- </pre>- <p class="mini">Because <code>PipelexPipelineExecuteResponse</code> wraps <code>pipe_output</code>, the field is auto-exposed on the Pipelex API response — no extra plumbing.</p>- </div>- </details>-- <details>- <summary>③ The reporting surface shrinks — registry methods leave the protocol <span class="pill rm" style="margin-left:auto">removal</span></summary>- <div class="details-body">- <p><code>ReportingProtocol</code> (and <code>ReportingNoOp</code>) no longer expose registry lifecycle. The <code>UsageRegistry</code> model is deleted outright. Embedders that rendered cost via <code>get_report_delegate().generate_report()</code> now render from the returned <code>pipe_output.tokens_usages</code> instead.</p>- <span class="filename">pipelex/reporting/reporting_protocol.py</span>- <pre class="diff">-<span class="ln ctx"> class ReportingProtocol(Protocol):</span>-<span class="ln del">- def open_registry(self, pipeline_run_id: str): ...</span>-<span class="ln ctx"> def report_inference_job(self, inference_job: InferenceJobAbstract): ...</span>-<span class="ln del">- def generate_report(self, pipeline_run_id: str | None = None, print_to_console: bool = True): ...</span>-<span class="ln del">- def close_registry(self, pipeline_run_id: str): ...</span>-<span class="ln ctx"> def setup(self): ...</span>-<span class="ln ctx"> def teardown(self): ...</span>-<span class="ln ctx"> def set_event_log(self, ...): ... # kept — the usage transport</span>- </pre>- <p class="mini"><code>reporting_manager.py</code> lost <strong>175 lines</strong>: <code>_usage_registries</code>, <code>open_registry</code>, <code>close_registry</code>, the <code>_get_registry_strict</code> / <code>_get_or_create_registry</code> / <code>_try_add_to_registry</code> trio, <code>inject_tokens_usages</code>, <code>generate_report</code>, and the <code>UsageRegistry</code> / <code>UsageRegistryRoot</code> models. The emit core (<code>_emit_usage_event</code> + fast path + runner fallback) is unchanged — that's the surviving transport.</p>- </div>- </details>-- <details>- <summary>④ The only cost renderer is now event-sourced <span class="pill add" style="margin-left:auto">new module</span></summary>- <div class="details-body">- <p>The main CLI renders from <code>pipe_output.tokens_usages</code>. One aggregation pass feeds both the suppression decision and the render. Note the gate: <strong>dry runs are suppressed, but real runs on free/zero-price models still report</strong> their token usage (with a zero total) — gating on cost alone would wrongly hide them.</p>- <span class="filename">pipelex/reporting/cost_report_renderer.py</span>- <pre class="code"><code>def render_run_cost_report(- *, pipeline_run_id: str,- tokens_usages: Sequence[AnyTokensUsage] | None,- is_generate_costs: bool, # public flag name kept; cost domain stays cost-named-) -> None:- if not is_generate_costs or not tokens_usages:- return- reporting_config = get_config().pipelex.reporting_config- print_to_console = reporting_config.is_log_costs_to_console # default now TRUE- is_csv_enabled = reporting_config.is_generate_cost_report_file_enabled- if not (print_to_console or is_csv_enabled):- return- try:- aggregated = CostRegistry.aggregate_costs(tokens_usages) # single pass- if not aggregated.has_reportable_usage: # any tokens OR any cost- return # → suppresses dry runs only- ...- CostRegistry.render_report(aggregated, pipeline_run_id=pipeline_run_id, ...)- except (OSError, UnicodeEncodeError, PipelexError) as e:- log.warning(f"Cost report generation failed (run succeeded): {e}")</code></pre>- <p class="mini">A cost-report failure never fails an otherwise-successful run — aggregation, CSV setup, and render all sit inside one run-succeeded <code>try</code>. The agent CLI takes a different path: no Rich table, a machine-readable <code>cost_report</code> object on its <code>--with-memory</code> JSON envelope instead.</p>- <p>Submitters don't call <code>render_run_cost_report</code> directly. They call the one-arg convenience <code>render_cost_report_for_output(pipe_output)</code>, which unpacks the three primitive arguments from a finished <code>PipeOutput</code> — and, crucially, <strong>derives the <code>--costs</code> gate from the output itself</strong>: <code>pipe_output.tokens_usages is None</code> is exactly the signal that cost reporting was off for this run. The runner already resolved that decision at run time (with all <code>--costs/--no-costs</code> overrides applied) and recorded it on the output, so re-reading global config afterwards would only re-derive — incorrectly, if config was overridden or mutated between run and render — a fact the output already carries authoritatively.</p>- <span class="filename">pipelex/reporting/cost_report_renderer.py</span>- <pre class="code"><code>def render_cost_report_for_output(pipe_output: PipeOutput) -> None:- render_run_cost_report(- pipeline_run_id=pipe_output.pipeline_run_id,- tokens_usages=pipe_output.tokens_usages,- is_generate_costs=pipe_output.tokens_usages is not None, # gate read off the output, not config- )</code></pre>- <p class="mini">This is a pure de-dup + correct-source-of-truth change, not a behavior change: the primitive ANDs both guards (<code>not is_generate_costs or not tokens_usages</code>), so the one case where output and config disagree (costs on but tracing off ⇒ <code>tokens_usages is None</code> — decision #3) no-ops either way. <code>render_run_cost_report</code> stays the low-level primitive for paths where the three values genuinely come from different places: the agent CLI's <code>build_cost_summary</code> JSON envelope, and any distributed reassembly where the output isn't the single carrier. The <code>pipelex run</code> CLI and external embedders (e.g. cocode) collapse to the one-arg form.</p>- </div>- </details>-- <!-- ============ PUBLIC SURFACE ============ -->- <h2><span class="n">7.</span>What changed on the public surface</h2>- <table>- <thead><tr><th>Change</th><th>Kind</th><th>Notes</th></tr></thead>- <tbody>- <tr><td><code>--costs / --no-costs</code> on <code>run pipe|method|bundle</code></td><td><span class="pill add">added</span></td><td>Default on. Reports cost independently of <code>--graph</code>.</td></tr>- <tr><td><code>--cost-report / --no-cost-report</code></td><td><span class="pill rm">removed</span></td><td><strong>Breaking.</strong> Folded into <code>--costs</code>. No deprecation period.</td></tr>- <tr><td><code>PipeOutput.tokens_usages</code> + <code>usage_assembly_error</code></td><td><span class="pill add">added</span></td><td>Cost is part of the run result; exposed on the API response automatically.</td></tr>- <tr><td><code>--mock-inference</code></td><td><span class="pill add">added</span></td><td>LIVE run, faked LLM leaf, reportable synthetic usage. Excl. with <code>--dry-run</code>.</td></tr>- <tr><td>agent CLI <code>cost_report</code> in JSON (<code>--with-memory</code>)</td><td><span class="pill add">added</span></td><td>Best-effort <code>{total_cost, by_model}</code>, real USD. JSON-only, no stderr table.</td></tr>- <tr><td><code>is_log_costs_to_console</code> default <code>false → true</code></td><td><span class="pill chg">changed</span></td><td>CLI prints the cost table by default; per-inference-job console logging removed.</td></tr>- <tr><td><code>UsageRegistry</code> + registry methods on <code>ReportingProtocol</code></td><td><span class="pill rm">removed</span></td><td>Render from <code>pipe_output.tokens_usages</code> instead.</td></tr>- <tr><td><code>act_assemble_graph → act_assemble_tracing</code></td><td><span class="pill chg">changed</span></td><td>Now assembles graph <em>and</em> usage from one event read.</td></tr>- <tr><td>Dry runs never report; free-model runs still do</td><td><span class="pill chg">changed</span></td><td>Gate keys on "did real work" (any tokens or any cost), not total cost.</td></tr>- </tbody>- </table>-- <!-- ============ RENAME ============ -->- <h2><span class="n">8.</span>Companion rename (#968) — <code>GraphContext → TraceContext</code></h2>- <p>Once usage rides on the same per-run object as the graph, the name <code>GraphContext</code> under-described it — a cost concern was riding on something called "Graph." But usage isn't a free-rider: each <code>UsageReportEvent.node_id</code> is stamped from the context's <code>parent_node_id</code>, so <strong>graph events and cost events are two projections of one substrate</strong> — the per-execution node tree. The rename names the object for what it is.</p>- <div class="two">- <div class="card" style="margin:0">- <h3 style="margin-top:0">Renamed</h3>- <ul class="mini" style="margin-bottom:0">- <li><code>GraphContext</code> → <code>TraceContext</code> (file, type, the <code>graph_context</code> attribute incl. <code>JobMetadata.trace_context</code>)</li>- <li><code>is_generate_costs</code> → <code>is_generate_usage</code> (config field; the gate feeds <code>emit_usage_events</code>, so "usage gates usage")</li>- <li><code>with_execution_overrides(generate_costs=…)</code> → <code>generate_usage=…</code></li>- </ul>- </div>- <div class="card" style="margin:0">- <h3 style="margin-top:0">Deliberately kept</h3>- <ul class="mini" style="margin-bottom:0">- <li>The public <code>--costs / --no-costs</code> flag — "costs" is the deliverable the user asked for.</li>- <li>The whole cost domain: <code>CostRegistry</code>, <code>cost_report_renderer.py</code>, <code>render_run_cost_report(is_generate_costs=…)</code> — they genuinely compute money.</li>- <li>The <code>graph_id → run_id</code> nit — deferred to a focused follow-up.</li>- </ul>- </div>- </div>-- <!-- ============ DEFERRED ============ -->- <h2><span class="n">9.</span>Explicitly deferred / non-goals</h2>-- <div class="callout scope">- <span class="tag">⚠ The headline deferral — DynamoDB</span>- <p style="margin-bottom:0"><strong>Distributed cost reporting is validated on NDJSON only.</strong> The end-of-run read-back is backend-agnostic, so DynamoDB rides along in the <em>code path</em> for free — but the DynamoDB-specific concerns (<code>BatchWriteItem</code> batching, the shared-filesystem invariant, per-thread <code>writer_id</code>) are <strong>not</strong> exercised or guaranteed by this work. Treat DynamoDB-backed distributed cost reporting as unproven until it's a deliberate validation target.</p>- </div>-- <details>- <summary>Other deliberate non-goals (none block the shipped work)</summary>- <div class="details-body">- <ul>- <li><strong>Cost-per-node breakdown.</strong> Each <code>UsageReportEvent</code> already carries the producing pipe's <code>node_id</code>, but the report aggregates by model only. A per-node view is a future projection over the same stream — data's already there.</li>- <li><strong>Costs-only in-memory tracer skip (deep half of E1).</strong> Under <code>--no-graph --costs</code> the tracer still accumulates the in-memory graph (cheap; never assembled). The cheap half — gating per-pipe serialization on <code>emit_graph_events</code> — landed; fully skipping the tracer is deferred (must still mint node ids for usage attribution).</li>- <li><strong><code>--mock-inference</code> coverage.</strong> LLM leaf only; img-gen / extract / search fail loud. Agent CLI doesn't expose it yet. Full per-operator synthetic-output coverage comes via the dry-run-refactor follow-up (<code>run_mode=DRY</code>).</li>- <li><strong>Two design decisions (see <code>cost-report-deferred-decisions.md</code>).</strong> #3 — cost reporting is coupled to <code>tracing_config.is_enabled</code> (costs-on + tracing-off ⇒ no report). #6 — the agent CLI gates on the raw <code>costs</code> param while the main CLI gates on the resolved <code>is_generate_usage</code> config. Both want a deliberate call, not a reflexive patch.</li>- <li><strong>T3 — request-scoped tracing state</strong> instead of the process-singleton <code>ReportingManager</code>. Removing the registry shrank the singleton's surface but the per-context dict remains. Open follow-up.</li>- <li><strong>Pre-existing bugs flagged, not fixed here:</strong> the cost report ignores reasoning/audio/prediction token costs; DIRECT vs Temporal object re-validation use different dump modes. Both predate this work.</li>- </ul>- </div>- </details>-- <hr class="soft">- <p class="mini">Forward-looking records live alongside this file in <code>wip/registry/</code> — the deferred non-goals in <code>deferred-followups.md</code>, the two deferred design decisions in <code>cost-report-deferred-decisions.md</code>, and the usage-without-cost capability idea in <code>usage-reporting-without-cost.md</code>. The full build/review history is in PR #967 and git.</p>-- <div class="footer">- Pipelex · cost-reporting rework · branch <code>fix/For-API-update</code> (PR #967) + rename PR #968. Generated for co-developer onboarding — diagrams render via mermaid (CDN).- </div>--</div>--<script>- mermaid.initialize({- startOnLoad:true,- theme:'dark',- themeVariables:{- fontFamily:'-apple-system,Segoe UI,Roboto,sans-serif',- primaryColor:'#1d2136', primaryBorderColor:'#39406e', primaryTextColor:'#e7e9f3',- lineColor:'#6c7bff', secondaryColor:'#141a36', tertiaryColor:'#10182f',- actorBkg:'#1d2136', actorBorder:'#39406e', actorTextColor:'#e7e9f3',- noteBkgColor:'#241b10', noteBorderColor:'#5a4420', noteTextColor:'#ffd9a0',- sequenceNumberColor:'#0f1220'- },- flowchart:{curve:'basis',useMaxWidth:true},- sequence:{useMaxWidth:true,actorMargin:38}- });-- // tabs- document.querySelectorAll('.tabs').forEach(function(group){- var btns = group.querySelectorAll('.tabbtn');- btns.forEach(function(btn){- btn.addEventListener('click', function(){- var id = btn.getAttribute('data-tab');- btns.forEach(function(b){ b.setAttribute('aria-selected', b===btn ? 'true':'false'); });- group.querySelectorAll('.tabpane').forEach(function(p){- p.classList.toggle('active', p.id===id);- });- });- });- });-</script>-</body>-</html>diff --git a/wip/history/registry/deferred-followups.md b/wip/history/registry/deferred-followups.mddeleted file mode 100644index 1e8eb45ac..000000000--- a/wip/history/registry/deferred-followups.md+++ /dev/null@@ -1,43 +0,0 @@-# Registry / cost-reporting — deferred follow-ups--Durable home for the follow-ups this feature deliberately left out of scope. The leak fix + distributed cost reporting shipped in **PR #967** (+ companion rename **#968**); see [`cost-reporting-overview.html`](cost-reporting-overview.html) for the as-built overview. The items below are the known, intentional non-goals — none blocks the shipped work.--Two further deferred items — both *design decisions* rather than follow-up work — live in their own doc: [`cost-report-deferred-decisions.md`](cost-report-deferred-decisions.md) (#3 cost reporting coupled to `tracing_config.is_enabled`; #6 agent-CLI vs main-CLI "is cost reporting on?" gating divergence). Those want a deliberate call; the items here are scoped work to pick up when signals warrant.--## Optimizations--- **D5 — costs-only in-memory tracer skip (deep half of review item E1).** In `--no-graph --costs` mode the graph tracer still runs in memory (minting node ids, accumulating the in-memory graph) even though no `GraphSpec` is ever assembled or output. Phase 2 landed the *cheap* half: per-pipe graph serialization (`pipe_abstract.py` IOSpec dumps, pipe/concept registry, `pipe_parallel.py` controller-output registration, `graph_tracer.teardown` spec build) is gated on `emit_graph_events`, so costs-only no longer pays for serialization. The *deep* half — skipping the in-memory tracer entirely in costs-only mode — is deferred. Watch-item: `on_pipe_start` still mints node ids so usage-event `node_id` correlation is unaffected; any skip must preserve that.--- **Cost-per-node correlation via `UsageReportEvent.node_id`.** Each `UsageReportEvent` already carries the `node_id` of the pipe that produced it, but the assembled cost report aggregates by model only. A future report could break cost down per graph node (which pipe/step cost what), correlating usage events to graph nodes via `node_id`. Not built; the data is already in the event stream.--## Altitude / architecture--- **Review item A1 — `TraceContext.from_execution_config(...)` factory.** The `(graph, costs) → (event_log routing, set_event_log, flags)` decision is currently made at each tracing arm (DIRECT `pipeline_run_setup`, TEMPORAL `wf_pipe_router`). A2 captured most of the value by threading the emit flags into `open_tracer`/`setup` (so the returned `TraceContext` is born correct and the post-hoc `model_copy` footguns are gone). A single factory that owns the whole decision for both arms is the remaining polish — deferred as low-value-now.--- **`graph_id → run_id` rename + the shelved `TraceContext` split (left over from the merged #968 rename).** #968 renamed `GraphContext → TraceContext` and the gate `is_generate_costs → is_generate_usage`, but deliberately parked two follow-ons: (1) the `graph_id → run_id` field rename — it really is the run id, but the rename ripples into the graph-named `GraphTracerManager.open_tracer(graph_id=…)` plumbing and the `make_node_id` format, so it wants a focused pass; (2) a *type* split — a `TraceContext` base with a `GraphContext(TraceContext)` subclass (Option 2), or extracting the `emit_graph_events` / `emit_usage_events` toggles into an `EmitPolicy` value object (Option 3). Both splits are low-value now: the node tree is **always** built (even under `--no-graph --costs`, per D5), so the runtime object is always the full graph-capable context — a split would only narrow what the reporting seam is allowed to *see*, never drop machinery. Revisit Option 2 only if reporting grows graph-independent consumers; Option 3 only if a third trace stream appears. Do **not** touch `UsageReportEvent.node_id` — cost-per-node attribution to the tree is a deliberate feature.--- **Review item T5 — `wf_pipe_router.py` setup-failure `except` doesn't restore `job_metadata.trace_context`.** On a tracer-setup failure the pipe can run with a `tracer_key` pointing at a closed tracer. Pre-existing latent issue (not introduced by this work; Temporal is unshipped — see [[project_temporal_not_shipped]]). Flag-and-fix candidate, parked.--- **T3 — request-scoped tracing state instead of the process-singleton `ReportingManager`.** The deeper refactor in the distributed-execution track ([`../../distributed-execution/tracing-cost-reporting.md`](../../distributed-execution/tracing-cost-reporting.md) T3). Removing the submitter-side `UsageRegistry` shrank the singleton's surface, but the per-context `_event_log_contexts` dict on a process-global manager remains. Open follow-up.-- **Facet flagged on PR #967 (codex, P1) — activity usage writes can mutate a workflow-registered buffer.** In a *single-process* Temporal deployment (one worker hosting both `WfPipeRouter` workflows and the inference activities), the workflow registers its `BufferingEventLog` in the process-global `ReportingManager._event_log_contexts`. An inference activity in the same process then hits the fast path (`_emit_via_registered_context`) and appends to the workflow's in-memory buffer — an activity side effect that is **not** part of Temporal's deterministic replay (the activity *result* replays from history, but the buffer append does not), so on replay the later drain/flush can diverge or drop usage. **Why this is deferred, not patched on this PR:** (1) it is *pre-existing* — the fast path + per-context registration predate this work (the registry track kept the emit core unchanged; it only removed the submitter-side registry), so PR #967 neither introduced nor worsened it; (2) the **recommended split router+runner topology does not hit it** — activities run on the runner process, which never registered the workflow's buffer, so they correctly take the per-process activity-log fallback; (3) the clean fix is request-scoped tracing state (this T3 refactor) or making activities always bypass workflow-registered buffers — bolting Temporal-awareness (`activity.in_activity()`) into the generic, non-Temporal `ReportingManager` would be a layering violation; (4) Temporal is not shipped to prod. Pick this up with the T3 refactor.--- **Costs-only event-log handle relies on `__del__`, not an explicit close (codex P2, PR #977 v0.32.0 release re-review).** In `--no-graph --costs` mode, `pipeline_run_setup` creates the `event_log` and registers it on the report delegate (`set_event_log`) but passes `event_log=None` to the tracer (D5: costs-only emits no graph events). So the tracer never owns it, and `GraphTracer.teardown()` — the only path that *explicitly* closes an event log — has nothing to close. On run end, `runner.py` (and the setup error-path) call `ReportingManager.clear_event_log`, which only `pop`s the `_event_log_contexts` entry; it does **not** close the log. The NDJSON file handle is therefore closed only by `NdjsonEventLog.__del__`. **This is *not* the unbounded FD leak the finding described:** `make_event_log` returns a *fresh* `NdjsonEventLog` per run, so once `clear_event_log` drops the delegate's reference (and the setup-local falls out of scope) the instance is refcount-collected and `__del__` closes the handle promptly under CPython — no descriptor growth across runs. The valid residual is that the costs-only path has no *explicit* close and leans on GC finalization. **Clean fix:** have `ReportingManager.clear_event_log` close the popped context's `event_log` before discarding it (`NdjsonEventLog.close()` is idempotent — in `--graph --costs` mode the tracer's `teardown` close and this close are both safe, the second being a no-op). Deferred off the v0.32.0 release branch (docs-only pass); land via normal `dev` flow. Closely related to **T3** above — the request-scoped `_event_log_contexts` refactor would subsume this.--## Pre-existing gaps surfaced during the PR #967 final review (flag, not fix-here)--These were found by the gstack `/review` adversarial pass (Codex + Claude). They are **pre-existing** (not introduced by the registry/cost work) and out of scope for this PR's purpose, so they are flagged here rather than expanding the PR. Per the repo "flag and fix existing bugs" rule, each is a candidate follow-up.--- **Cost report ignores reasoning / audio / prediction token costs.** `CostRegistry.complete_cost_report` only computes cost for `INPUT` (→ joined/cached/non-cached) and `OUTPUT`; `aggregate_costs` / `compute_total_cost` only sum those. The `TokenCategory` enum also has `OUTPUT_REASONING`, `OUTPUT_AUDIO`, `OUTPUT_ACCEPTED_PREDICTION`, `OUTPUT_REJECTED_PREDICTION`, `INPUT_AUDIO` — providers (e.g. the OpenAI Responses factory) populate some of these, but their cost is never computed or summed, so a run with reasoning/audio tokens **under-reports total cost**. Pre-existing (this PR only refactored `compute_total_cost`; it did not change the category set). Fixing it properly means extending `compute_cost_report` / `complete_cost_report` / `aggregate_costs` / the render columns / the agent summary to handle the extra categories, with per-provider unit-cost wiring — a cost-model change, separate from the leak/distributed work.--- **DIRECT vs TEMPORAL object re-validation use different dump modes.** `ContentGenerator._revalidate_against_object_class` (DIRECT) dumps with `model_dump(serialize_as_any=True)` (python mode) while `ContentGeneratorInWorkflow` (Temporal) uses `model_dump(mode="json", serialize_as_any=True)`. Same logical re-validation, two serialization modes — so a `--mock-inference` synthetic object (or any object) can round-trip cleanly in one execution mode and raise `MockInferenceObjectFidelityError` in the other. The divergence predates this PR (each arm kept its existing behavior), but the shared-helper framing makes it load-bearing. Aligning the two on one canonical mode is the durable fix; which mode is canonical is the judgment call.--## `--mock-inference` coverage — RESOLVED (2026-06-10, unified dry run)--> **All items below are closed by the unified dry run** (`wip/dry-run-refactor/design-unified-dry-run.md` + `plan-unified-dry-run.md`): `--mock-inference` is removed as a mode; the reportable-usage capability is the internal `CogtRunParams.is_mock_usage` sub-flag of DRY; `MockInferenceUnsupportedError` and the per-leaf guards are deleted (dry run covers every leaf). Kept as the historical record.--- **LLM leaf only.** `--mock-inference` mocks the LLM inference leaf (`llm_gen_text`/`object`/`object_list`). Image-gen / extract / search **fail loud** with `MockInferenceUnsupportedError` (their output is stored *above* the leaf by `generated_content_factory`, so a leaf-level mock would push synthetic data through the storage path — whereas `ContentGeneratorDry` short-circuits storage). Full per-operator synthetic-output coverage comes via [`../../dry-run-refactor/followup-leaf-run-mode-mock.md`](../../dry-run-refactor/followup-leaf-run-mode-mock.md) B2, which replaces each hard guard with a real synthetic-output branch.--- **Main `run` subcommands only.** The flag is on `pipelex run pipe|method|bundle`, not the agent CLI. The mechanism (`JobMetadata.is_mock_inference` + leaf branch + `PipelexRunner`) is surface-agnostic, so the agent CLI can adopt it trivially when needed.--- **`is_mock_inference` fate after the dry-run-refactor follow-up — SETTLED 2026-06-10: fully retired as a mode, capability kept as the internal `is_mock_usage` DRY sub-flag (supersedes the interim D8 keep).** Original framing — decided 2026-06-06: build the interim flag now (Phase 5 is the follow-up's B1 core — a reusable leaf-level dry helper), settle retirement at B2. Open sub-decision: *fully retire* `--mock-inference` (DRY-on-Temporal covers cross-worker dispatch + `tokens_usages` assembly once it dispatches) vs *keep a thin reportable-mock* (DRY emits zero-token usage → suppressed report, so the cheap *rendered* cost-report validation — Tier 8b — is unique to a non-zero mock). Decide at B2 time. See [`../../dry-run-refactor/followup-leaf-run-mode-mock.md`](../../dry-run-refactor/followup-leaf-run-mode-mock.md).diff --git a/wip/history/registry/usage-reporting-without-cost.md b/wip/history/registry/usage-reporting-without-cost.mddeleted file mode 100644index 979a5d8a8..000000000--- a/wip/history/registry/usage-reporting-without-cost.md+++ /dev/null@@ -1,31 +0,0 @@-# Capability: usage reporting independent of cost--**Status: IDEA, LOW PRIORITY.** Not decided, not scheduled. Captured because it is interesting in its own right; revisit when a concrete signal appears (a user wanting token counts without dollars, or a model/backend with no cost data).--## The capability gap--Today usage-emission and cost-rendering are **bundled under one switch**. The `--costs` flag (→ internal gate `is_generate_usage`, renamed from `is_generate_costs` in #968) turns on usage-event emission *and* the cost report is the only deliverable. There is **no mode that gives you token counts without dollar figures.**--But usage (token counts) is valuable on its own, not just as an input to a cost number:--- **Consumption monitoring / profiling** — how many tokens a run burns, where, regardless of price. Useful for prompt-size regressions and context-budget work even when you don't care about (or can't price) the spend.-- **Models / backends without cost data.** A handle with no per-token unit cost configured produces a meaningless or zero cost report, but its token counts are still real and worth surfacing.-- **Budget-agnostic dashboards.** Token usage is a stable signal; prices drift. Reporting usage separately lets a downstream consumer apply its own pricing.--The data is already there — `UsageReportEvent` carries `TokensUsage` (`nb_tokens_by_category`); cost is computed *from* it by `CostRegistry`. So "usage without cost" is dropping a derivation step, not capturing new data.--## Why this is the deeper question behind the naming cleanup--The companion rename merged in #968 (`is_generate_costs → is_generate_usage`, `GraphContext → TraceContext`) handled *consistency* — it aligned the internal gate name to the usage stream it feeds, and deliberately left the public `--costs` flag and the cost domain alone. It did **not** add any capability.--This doc is the *capability* question: should usage be a first-class, separately-requestable output, with cost as one **view** layered on top? If yes, the public `--costs`-vs-`--usage` naming falls out of it naturally (usage = the measured data, cost = a priced projection). If no, keeping "cost" as the single user-facing word is correct and the naming work stops at the internal gate. **Decide the capability first; the public naming is downstream of it.**--## Rough shape if pursued (not designed)--- Split the single gate into two intents: *emit usage* and *render cost*. Cost-render becomes gated on usage-present (it already is, post-companion-rename) plus a separate "show costs" choice.-- Surface a **usage-only report**: token counts by category / by model, no dollar columns — a sibling renderer (or a mode of `cost_report_renderer.py`) that omits the priced columns.-- Public surface: likely a `--usage` / `--costs` pair (or `--report usage|cost|both`), where `--costs` implies usage. Exact ergonomics TBD.--## Related, already-captured--- **Cost-per-node breakdown** ([`deferred-followups.md`](deferred-followups.md), "Cost-per-node correlation") — also a *view* over the same `UsageReportEvent` stream (aggregate by `node_id` instead of by model). A usage-only-per-node report is the intersection of that item and this one; if both get built, they share the same "projection over the usage event stream" machinery.diff --git a/wip/history/runtime-bridge.html b/wip/history/runtime-bridge.htmldeleted file mode 100644index 455159f37..000000000--- a/wip/history/runtime-bridge.html+++ /dev/null@@ -1,287 +0,0 @@-<!doctype html>-<html lang="en">-<head>-<meta charset="utf-8" />-<meta name="viewport" content="width=device-width, initial-scale=1" />-<title>Pipelex runtime bridge — extracting runtime-agnostic tools from Temporal</title>-<style>- :root {- --bg: #0e1116;- --panel: #161b22;- --panel-2: #1c2330;- --ink: #e6edf3;- --ink-dim: #9aa6b2;- --accent: #79c0ff;- --accent-2: #d2a8ff;- --good: #7ee787;- --warn: #f5b352;- --bad: #ff7b72;- --line: #30363d;- --kbd: #21262d;- }- html { color-scheme: dark; }- body {- margin: 0; padding: 0;- font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", system-ui, sans-serif;- background: var(--bg); color: var(--ink); line-height: 1.55;- }- .wrap { max-width: 1040px; margin: 0 auto; padding: 32px 28px 96px; }- header.title { border-bottom: 1px solid var(--line); margin-bottom: 22px; padding-bottom: 16px; }- header.title h1 { margin: 0 0 6px; font-size: 26px; line-height: 1.25; }- header.title p.lede { margin: 0; color: var(--ink-dim); font-size: 15px; }- .meta { display: flex; flex-wrap: wrap; gap: 6px 10px; margin-top: 14px; font-size: 13px; color: var(--ink-dim); }- .pill { display: inline-block; background: var(--panel); border: 1px solid var(--line); border-radius: 999px; padding: 2px 10px; color: var(--ink); }- .pill.added { color: var(--good); border-color: rgba(126,231,135,.4); }- .pill.changed { color: var(--accent); border-color: rgba(121,192,255,.4); }- .pill.moved { color: var(--accent-2); border-color: rgba(210,168,255,.4); }- .pill.future { color: var(--warn); border-color: rgba(245,179,82,.4); }- h2 { margin: 38px 0 12px; font-size: 21px; border-bottom: 1px solid var(--line); padding-bottom: 6px; }- h3 { margin: 24px 0 8px; font-size: 17px; color: var(--accent); }- h4 { margin: 16px 0 6px; font-size: 12px; color: var(--ink-dim); text-transform: uppercase; letter-spacing: .04em; }- p { margin: 8px 0 12px; }- ul, ol { margin: 8px 0 14px; padding-left: 22px; }- li { margin: 5px 0; }- a { color: var(--accent); }- code, pre, kbd { font-family: ui-monospace, "SF Mono", Menlo, Consolas, monospace; font-size: 13px; }- code { background: var(--kbd); border: 1px solid var(--line); padding: 1px 5px; border-radius: 4px; }- pre { background: var(--panel); border: 1px solid var(--line); border-radius: 8px; padding: 14px 16px; overflow: auto; margin: 10px 0 16px; line-height: 1.45; }- pre code { background: none; border: none; padding: 0; font-size: 12.5px; }- .tldr { background: linear-gradient(180deg, var(--panel-2), var(--panel)); border: 1px solid var(--line); border-radius: 12px; padding: 18px 22px; margin: 18px 0 8px; }- .tldr h2 { border: none; margin: 0 0 8px; padding: 0; font-size: 16px; letter-spacing: .04em; text-transform: uppercase; color: var(--accent-2); }- .tldr p:last-child { margin-bottom: 0; }- .tldr ul { margin: 10px 0 4px; }- .grid-2 { display: grid; grid-template-columns: 1fr 1fr; gap: 14px; }- @media (max-width: 820px) { .grid-2 { grid-template-columns: 1fr; } }- .card { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 14px 16px; }- .card h4 { margin-top: 0; color: var(--accent); }- .card p { margin: 6px 0 0; }- table { width: 100%; border-collapse: collapse; margin: 12px 0 18px; font-size: 13px; }- th, td { text-align: left; padding: 8px 10px; border-bottom: 1px solid var(--line); vertical-align: top; }- th { color: var(--ink-dim); font-weight: 600; font-size: 12px; text-transform: uppercase; letter-spacing: .03em; }- td code { white-space: nowrap; }- .figure { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 18px; margin: 16px 0; }- .figure .cap { color: var(--ink-dim); font-size: 12.5px; margin-top: 10px; text-align: center; }- .figure .mermaid { display: flex; justify-content: center; }- pre.diff { line-height: 1.5; }- pre.diff .add { color: var(--good); display: block; background: rgba(126,231,135,.08); }- pre.diff .del { color: var(--bad); display: block; background: rgba(255,123,114,.08); }- pre.diff .ctx { color: var(--ink-dim); display: block; }- pre.diff .hd { color: var(--accent-2); display: block; }- pre.diff .cmt { color: #6e7681; display: block; }- .note { border-left: 3px solid var(--accent); background: rgba(121,192,255,.06); padding: 10px 14px; border-radius: 0 8px 8px 0; margin: 14px 0; }- .note.good { border-left-color: var(--good); background: rgba(126,231,135,.06); }- .note.warn { border-left-color: var(--warn); background: rgba(245,179,82,.06); }- .note p:first-child { margin-top: 0; } .note p:last-child { margin-bottom: 0; }- .tag { font-size: 11px; padding: 1px 7px; border-radius: 999px; border: 1px solid var(--line); color: var(--ink-dim); }- .tag.ok { color: var(--good); border-color: rgba(126,231,135,.4); }- .tag.fix { color: var(--accent); border-color: rgba(121,192,255,.4); }- .tag.defer { color: var(--warn); border-color: rgba(245,179,82,.4); }- .tag.now { color: var(--good); border-color: rgba(126,231,135,.4); }- .tag.later { color: var(--warn); border-color: rgba(245,179,82,.4); }- details { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; margin: 10px 0; padding: 0 16px; }- details[open] { padding-bottom: 6px; }- summary { cursor: pointer; padding: 12px 0; font-weight: 600; color: var(--ink); list-style: none; display: flex; align-items: center; gap: 8px; }- summary::-webkit-details-marker { display: none; }- summary::before { content: "▸"; color: var(--accent); transition: transform .15s ease; }- details[open] summary::before { transform: rotate(90deg); }- .tabs { margin: 14px 0; }- .tab-bar { display: flex; gap: 4px; flex-wrap: wrap; border-bottom: 1px solid var(--line); }- .tab-btn { background: transparent; border: 1px solid transparent; border-bottom: none; color: var(--ink-dim); padding: 8px 14px; cursor: pointer; font-size: 13px; border-radius: 8px 8px 0 0; font-family: inherit; }- .tab-btn.active { color: var(--ink); background: var(--panel); border-color: var(--line); }- .tab-panel { display: none; border: 1px solid var(--line); border-top: none; border-radius: 0 0 8px 8px; padding: 4px 16px 8px; background: var(--panel); }- .tab-panel.active { display: block; }- .tab-panel pre { background: var(--bg); }- .small { font-size: 12.5px; color: var(--ink-dim); }- hr { border: none; border-top: 1px solid var(--line); margin: 28px 0; }-</style>-<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>-<script>- mermaid.initialize({- startOnLoad: true,- theme: "dark",- securityLevel: "loose",- themeVariables: {- background: "#161b22", primaryColor: "#1c2330", primaryTextColor: "#e8eaf0",- primaryBorderColor: "#2c3142", lineColor: "#6ea8fe", fontFamily: "ui-monospace, monospace",- fontSize: "13px"- },- flowchart: { curve: "basis", htmlLabels: true }- });- function showTab(group, idx, btn) {- document.querySelectorAll('[data-tabgroup="' + group + '"] .tab-btn').forEach((b, i) => b.classList.toggle('active', i === idx));- document.querySelectorAll('[data-tabgroup="' + group + '"] .tab-panel').forEach((p, i) => p.classList.toggle('active', i === idx));- }-</script>-</head>-<body>-<div class="wrap">--<header class="title">- <h1>Runtime bridge — extracting runtime-agnostic tools out of the Temporal subpackage</h1>- <p class="lede">A lot of what we built for Temporal isn't actually Temporal-specific. This PR pulls that logic into a neutral home so it can be shared across runtimes — used by Temporal today, by the Mistral Workflows plugin when it ships.</p>- <div class="meta">- <span class="pill">PR #969</span>- <span class="pill">branch <code>feature/Runtime-bridge-extraction</code></span>- <span class="pill">base <code>dev</code></span>- <span class="pill added">new package <code>pipelex/runtime_bridge/</code></span>- </div>-</header>--<div class="tldr">- <h2>TL;DR</h2>- <p>We're going to support more than one execution runtime. Today Pipelex runs pipes durably on <strong>Temporal</strong>; next we'll run them on <strong>Mistral Workflows</strong>. While building the Temporal integration, several pieces turned out <strong>not to be Temporal-specific at all</strong>: delivering results (storage + webhook), flushing trace events, hydrating working memory, rehydrating a library crate, classifying a pipe as controller-vs-leaf.</p>- <p>This PR <strong>lifts that runtime-agnostic logic out of the <code>pipelex.temporal</code> subpackage into a new neutral home — <code>pipelex/runtime_bridge/primitives/</code></strong> — and turns the Temporal activities into thin wrappers over it. Nothing about behaviour changes; only the <em>home</em> of the logic does.</p>- <ul>- <li><strong>Used by Temporal today</strong> <span class="tag now">now</span> — the <code>act_*.py</code> activities are now thin <code>@activity.defn</code> wrappers that delegate to the shared primitives; even core <code>delivery_executor</code> consumes one. One copy, instead of logic that would get re-implemented per runtime.</li>- <li><strong>Used by Mistral Workflows next</strong> <span class="tag later">later</span> — the same primitives are what the <code>pipelex-mistralai-workflows</code> plugin will consume when we ship it (currently preview).</li>- <li><strong>Also lands a forward-looking dispatch surface</strong> — <code>run_pipe_via_bridge()</code> + a <code>PipelexExecutionMode</code> enum + JSON-safe boundary types — designed and tested now as the door the Mistral plugin will enter through. <em>It has no in-tree caller yet</em>; Temporal keeps its existing entry points and just consumes the lifted primitives.</li>- </ul>- <p class="small" style="margin-top:10px">The diff looks large, but the code core is small and concentrated in <code>pipelex/runtime_bridge/</code>. Most of the line count is a vendored skill + triage notes that are <strong>not</strong> review material (see <em>What's not part of this work</em>).</p>-</div>--<div class="note">- <p style="margin:0 0 6px"><strong>A note on two words used throughout this doc.</strong> The relocation came in two flavours — the distinction matters when reading the diff:</p>- <ul style="margin:0">- <li><span class="pill moved">moved</span> — the module already stood on its own; it just changed directory, byte-for-byte. Git records it as a <em>rename</em> (high similarity), so the diff shows no real content change — only a new path. Example: <code>hydration.py</code>, <code>pipe_run_arg.py</code>, <code>submitter_hydration.py</code>.</li>- <li><span class="pill added">lifted</span> — the logic was <em>welded inside a Temporal activity's function body</em>. It was extracted into its own standalone, runtime-agnostic function, leaving a thin <code>@activity.defn</code> wrapper behind that just calls it. This is an <em>extraction / refactor</em>, so the diff shows code leaving one file and a new file appearing — not a clean rename. Example: <code>delivery.py</code>, <code>trace_flush.py</code>.</li>- </ul>- <p class="small" style="margin:8px 0 0">In short: <strong>moved = same code, new home</strong>; <strong>lifted = code pulled out of a Temporal wrapper into a reusable home, wrapper kept thin</strong>. Both are behaviour-neutral.</p>-</div>--<h2>The move at a glance</h2>--<div class="figure">- <div class="mermaid">-flowchart LR- subgraph temporal["pipelex.temporal — Temporal-specific glue"]- direction TB- ACT["act_*.py<br/><i>thin @activity.defn wrappers</i>"]- WF["wf_pipe_* · temporal_pipe_*"]- end- subgraph core["pipelex core"]- DEX["pipe_run.delivery_executor"]- end- subgraph bridge["pipelex.runtime_bridge — runtime-agnostic"]- direction TB- PRIM["primitives/<br/>delivery · trace_flush · hydration<br/>submitter_hydration · pipe_run_arg · pipe_classification"]- DISP["dispatch surface (forward-looking)<br/>run_pipe_via_bridge · execution_mode<br/>boundary types · bootstrap"]- end- MISTRAL["pipelex-mistralai-workflows<br/><i>ships later (preview)</i>"]- ACT -->|consumes today| PRIM- WF -->|consumes today| PRIM- DEX -->|consumes today| PRIM- MISTRAL -.->|will consume| PRIM- MISTRAL -.->|will enter through| DISP- classDef p fill:#16241a,stroke:#5fd38f,color:#cfeede;- classDef f fill:#2a2418,stroke:#e0b463,color:#f1e2c4;- class PRIM p;- class DISP f;- </div>- <div class="cap">Solid arrows = consumed in-tree today. Dotted = the Mistral plugin will consume it once shipped. The <span style="color:var(--good)">primitives</span> are the landed, in-use payload of this PR; the <span style="color:var(--warn)">dispatch surface</span> is built ahead of its consumer.</div>-</div>--<h2>What got generalized (the lift)</h2>--<p>Each of these was a body that lived inside a Temporal activity or module. None of it needs Temporal. It moved to <code>runtime_bridge/primitives/</code> with <strong>no behaviour change</strong>, and the Temporal side now imports it back.</p>--<table>- <thead><tr><th>Primitive</th><th>What it does</th><th>Consumed today by</th><th></th></tr></thead>- <tbody>- <tr><td><code>delivery.py</code></td><td>Execute the full delivery (storage + webhook) of a pipe result</td><td><code>act_deliver.py</code></td><td><span class="pill added">lifted</span></td></tr>- <tr><td><code>trace_flush.py</code></td><td>Flush a workflow's buffered trace events to the configured backend</td><td><code>act_flush_trace_events.py</code></td><td><span class="pill added">lifted</span></td></tr>- <tr><td><code>hydration.py</code></td><td>Rebuild typed content / working memory from a dump</td><td><code>wf_pipe_router.py</code> <strong>and core <code>delivery_executor.py</code></strong></td><td><span class="pill moved">moved</span></td></tr>- <tr><td><code>submitter_hydration.py</code></td><td>Rehydrate a <code>PipeOutput</code> with its <code>LibraryCrate</code></td><td><code>temporal_pipe_run.py</code> · <code>temporal_pipe_router.py</code></td><td><span class="pill moved">moved</span></td></tr>- <tr><td><code>pipe_run_arg.py</code></td><td>The <code>PipeRunArg</code> transit shape</td><td><code>temporal_pipe_run.py</code> · <code>wf_pipe_run.py</code></td><td><span class="pill moved">moved</span></td></tr>- <tr><td><code>pipe_classification.py</code></td><td>Controller-vs-leaf decision for per-step decomposition</td><td><span class="small">no in-tree caller yet — for Mistral</span></td><td><span class="pill future">new</span></td></tr>- </tbody>-</table>--<p class="small">That <code>hydration.py</code> is consumed by core <code>delivery_executor</code> — not just Temporal — is the tell that this logic was never Temporal-specific to begin with.</p>--<h3>What a Temporal activity looks like now</h3>--<p>The activity keeps only its Temporal-specific shell — the <code>@activity.defn</code> decorator and arg decode — and delegates the body to the shared primitive:</p>--<pre class="diff"><span class="cmt"># act_deliver.py — Temporal shell only; the body now lives in the runtime-agnostic primitive.</span>-<span class="ctx">from pipelex.runtime_bridge.primitives.delivery import execute_delivery</span>--<span class="ctx">@activity.defn(name="act_deliver")</span>-<span class="ctx">@convert_pipelex_errors</span>-<span class="ctx">async def act_deliver(arg: DeliveryActivityArg) -> None:</span>-<span class="ctx"> await execute_delivery(...) # the lifted, host-agnostic body</span></pre>--<p class="small"><strong>Two robustness fixes also rode along on live code:</strong> tracing now degrades gracefully when the event-log <em>backend</em> can't even be constructed (new <code>EventLogSetupError</code>), not only when a read fails; and <code>hub.py</code> gains <code>scoped_pipe_router</code>, a contextvar router override that restores the prior value on exit.</p>--<h2>The forward-looking dispatch surface</h2>--<div class="note warn">- <p><strong>Designed, not yet wired in.</strong> <code>run_pipe_via_bridge()</code> + a <code>PipelexExecutionMode</code> enum + JSON-safe (<code>extra="forbid"</code>) boundary types are the door the Mistral plugin will enter through. They're built, unit-tested, and hardened — but have <strong>no in-tree caller yet</strong>. Temporal keeps its own entry points and just consumes the lifted primitives. Skim to know it's there; no need to review as live code.</p>-</div>--<details>- <summary>The four execution modes (reference)</summary>- <table>- <thead><tr><th>Mode</th><th>What it does</th><th>Needs</th></tr></thead>- <tbody>- <tr><td><code>DIRECT</code> <span class="small">(default)</span></td><td>In-process, no Temporal. Forces local execution even inside a Temporal-enabled worker.</td><td>nothing</td></tr>- <tr><td><code>TEMPORAL_BLOCKING</code></td><td>Dispatch as a Pipelex Temporal workflow and await it.</td><td><code>pipelex[temporal]</code></td></tr>- <tr><td><code>TEMPORAL_FIRE_AND_FORGET</code></td><td>Dispatch and return the <code>workflow_id</code>; completion arrives via the <code>DeliveryAssignment</code> (required here).</td><td><code>pipelex[temporal]</code> + delivery target</td></tr>- <tr><td><code>MISTRAL_NATIVE</code></td><td>Decompose into native Mistral Workflows primitives (controllers → child workflows, leaves → activities).</td><td><code>pipelex-mistralai-workflows</code> <span class="small">(preview)</span></td></tr>- </tbody>- </table>- <p class="small">Optional deps stay optional — <code>temporalio</code> / the Mistral plugin are lazy-imported only inside the matching branch; a missing extra raises a precise install-hint error. The surface is already covered by unit tests (dispatch, validation, idempotent boot-race, DIRECT router scoping, trace-context isolation), so it's ready the day the consumer lands.</p>-</details>--<h2>Why this is the right shape</h2>--<div class="grid-2">- <div class="card">- <h4>The lift earns its keep now</h4>- <p class="small">It's not abstraction-for-later: the primitives are consumed in-tree today by Temporal (and core <code>delivery_executor</code>). Moving them just stops the logic from being duplicated when Mistral arrives.</p>- </div>- <div class="card">- <h4>Right home, no host coupling</h4>- <p class="small"><code>runtime_bridge/primitives/</code> imports only Pipelex core types — no <code>temporalio</code>, no <code>mistralai</code>. "Temporal SDK glue" stays in <code>pipelex.temporal</code>; the reusable bodies don't.</p>- </div>- <div class="card">- <h4>Not over-engineered</h4>- <p class="small">The dispatch surface is a flat enum, not a plugin registry; <code>__init__.py</code> is empty (no re-export indirection, per project convention). No speculative interfaces beyond the one real upcoming consumer.</p>- </div>-</div>--<h2>What's <em>not</em> part of this work</h2>--<div class="note warn">- <p>The diff looks large, but a vendored <code>/workflows</code> skill under <code>.claude/skills/</code> accounts for most of the line count and is <strong>not part of this PR</strong> — it was adapted from Mistral's own Workflows skill and rode along on this branch for convenience. Internal triage notes under <code>wip/</code> are likewise not review material. A couple of unrelated review-driven fixes also rode along (e.g. honoring <code>transport_max_retries</code> on the gateway LLM client factories).</p>- <p style="margin-top:8px"><strong>Review the code core:</strong> <code>pipelex/runtime_bridge/</code>, <code>pipelex/temporal/tprl_pipe/act_*.py</code>, <code>pipelex/hub.py</code>, <code>pipelex/tracing/exceptions.py</code>, and <code>tests/unit/pipelex/runtime_bridge/</code>.</p>-</div>--<h2>File map</h2>--<table>- <thead><tr><th>Path</th><th>Role</th><th></th></tr></thead>- <tbody>- <tr><td colspan="3" class="small" style="color:var(--good); text-transform:uppercase; letter-spacing:.04em;">Consumed today (the landed payload)</td></tr>- <tr><td><code>runtime_bridge/primitives/delivery.py · trace_flush.py</code></td><td>Activity bodies lifted out of Temporal; behaviour-neutral</td><td><span class="pill added">lifted</span></td></tr>- <tr><td><code>runtime_bridge/primitives/hydration.py · submitter_hydration.py · pipe_run_arg.py</code></td><td>Moved from <code>temporal/tprl_pipe/</code> (git rename)</td><td><span class="pill moved">moved</span></td></tr>- <tr><td><code>temporal/tprl_pipe/act_*.py</code></td><td>Now thin <code>@activity.defn</code> wrappers over the primitives</td><td><span class="pill changed">slimmed</span></td></tr>- <tr><td><code>hub.py</code></td><td><code>scoped_pipe_router</code> + contextvar override in <code>get_pipe_router</code></td><td><span class="pill changed">changed</span></td></tr>- <tr><td><code>tracing/exceptions.py</code> · <code>tracing_assembly.py</code></td><td><code>EventLogError</code> family + best-effort degradation on backend setup failure</td><td><span class="pill changed">new / changed</span></td></tr>- <tr><td colspan="3" class="small" style="color:var(--warn); text-transform:uppercase; letter-spacing:.04em;">Forward-looking (no in-tree caller yet — for Mistral)</td></tr>- <tr><td><code>runtime_bridge/bridge.py</code></td><td><code>run_pipe_via_bridge</code> + boundary types + the four mode runners</td><td><span class="pill future">ready</span></td></tr>- <tr><td><code>runtime_bridge/execution_mode.py · bootstrap.py · exceptions.py</code></td><td>Mode enum · idempotent boot · bridge error family</td><td><span class="pill future">ready</span></td></tr>- <tr><td><code>runtime_bridge/primitives/pipe_classification.py</code></td><td>Controller/leaf split for Mistral's per-step decomposition</td><td><span class="pill future">ready</span></td></tr>- </tbody>-</table>--<h2>Verification</h2>--<div class="note good">- <p><code>make agent-check</code> (pyright 0/0/0, mypy) and <code>make agent-test</code> (full suite) both green. The lifted primitives are covered by the existing Temporal suite (behaviour unchanged); the forward-looking dispatcher ships with its own unit tests.</p>-</div>--<p class="small" style="margin-top:28px; color:#6e7681;">Co-developer overview of the runtime-bridge work (PR #969, branch <code>feature/Runtime-bridge-extraction</code>). Diagrams render via mermaid from CDN.</p>--</div>-</body>-</html>diff --git a/wip/history/signature-based-validation/signature-based-validation-pr-story.html b/wip/history/signature-based-validation/signature-based-validation-pr-story.htmldeleted file mode 100644index 53bb5ed8c..000000000--- a/wip/history/signature-based-validation/signature-based-validation-pr-story.html+++ /dev/null@@ -1,472 +0,0 @@-<!DOCTYPE html>-<html lang="en">-<head>-<meta charset="UTF-8">-<meta name="viewport" content="width=device-width, initial-scale=1.0">-<title>PR Story — Signature-Based Validation</title>-<style>- :root {- --bg: #0f1117;- --panel: #181b24;- --panel-2: #1f2330;- --ink: #e8eaf0;- --muted: #9aa3b6;- --line: #2c3142;- --accent: #6ea8fe;- --green: #5fd38f;- --amber: #f0c060;- --red: #f08a8a;- --violet: #b89cf0;- --mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;- }- * { box-sizing: border-box; }- body {- margin: 0;- background: var(--bg);- color: var(--ink);- font: 16px/1.65 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;- -webkit-font-smoothing: antialiased;- }- .wrap { max-width: 920px; margin: 0 auto; padding: 56px 28px 120px; }- header.hero { border-bottom: 1px solid var(--line); padding-bottom: 32px; margin-bottom: 40px; }- .eyebrow { font: 600 12px/1 var(--mono); letter-spacing: .14em; text-transform: uppercase; color: var(--accent); }- h1 { font-size: 34px; line-height: 1.2; margin: 14px 0 10px; letter-spacing: -.01em; }- .sub { color: var(--muted); font-size: 17px; max-width: 700px; }- .meta { margin-top: 18px; display: flex; flex-wrap: wrap; gap: 8px; }- .chip {- font: 600 12px/1 var(--mono); padding: 6px 10px; border-radius: 999px;- background: var(--panel-2); border: 1px solid var(--line); color: var(--muted);- }- h2 {- font-size: 22px; margin: 52px 0 16px; letter-spacing: -.01em;- padding-left: 12px; border-left: 3px solid var(--accent);- }- h3 { font-size: 16px; margin: 28px 0 10px; color: var(--ink); }- p { margin: 12px 0; }- .muted { color: var(--muted); }- code {- font: 13.5px/1.5 var(--mono); background: var(--panel-2);- padding: 1.5px 6px; border-radius: 5px; border: 1px solid var(--line);- }- .panel {- background: var(--panel); border: 1px solid var(--line);- border-radius: 12px; padding: 20px 22px; margin: 18px 0;- }- .callout {- border-left: 3px solid var(--amber); background: var(--panel);- border-radius: 0 10px 10px 0; padding: 14px 18px; margin: 18px 0;- }- .callout.bug { border-left-color: var(--red); }- .callout.note { border-left-color: var(--accent); }- .callout .lbl { font: 600 11px/1 var(--mono); letter-spacing: .1em; text-transform: uppercase; color: var(--amber); }- .callout.bug .lbl { color: var(--red); }- .callout.note .lbl { color: var(--accent); }-- .cards { display: grid; grid-template-columns: repeat(3, 1fr); gap: 14px; margin: 18px 0; }- .cards.two { grid-template-columns: repeat(2, 1fr); }- .card { background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 18px; }- .card .n { font: 700 13px/1 var(--mono); color: var(--accent); }- .card h4 { margin: 8px 0 6px; font-size: 15px; }- .card p { margin: 0; font-size: 14px; color: var(--muted); }-- .figure { margin: 22px 0; }- .figure svg { display: block; width: 100%; height: auto; }- .figcap { color: var(--muted); font-size: 13px; margin-top: 8px; text-align: center; }-- table { width: 100%; border-collapse: collapse; margin: 16px 0; font-size: 13.5px; }- th, td { text-align: left; padding: 9px 11px; border-bottom: 1px solid var(--line); vertical-align: top; }- th { font: 600 11px/1.3 var(--mono); letter-spacing: .06em; text-transform: uppercase; color: var(--muted); }- td .dot { font-weight: 700; }- .dot.g { color: var(--green); }- .dot.a { color: var(--amber); }- .dot.r { color: var(--red); }- tbody tr:hover { background: var(--panel); }-- .diff {- background: #11141c; border: 1px solid var(--line); border-radius: 10px;- overflow-x: auto; margin: 14px 0; font: 12.5px/1.7 var(--mono);- }- .diff .fname {- padding: 8px 14px; border-bottom: 1px solid var(--line);- color: var(--muted); background: var(--panel-2); font-weight: 600;- }- .diff pre { margin: 0; padding: 12px 14px; }- .diff .add { color: var(--green); display: block; }- .diff .del { color: var(--red); display: block; }- .diff .ctx { color: #8b93a7; display: block; }- .diff .hdr { color: var(--accent); display: block; }-- ul.tight { margin: 10px 0; padding-left: 22px; }- ul.tight li { margin: 5px 0; }- ol.phases { margin: 10px 0; padding-left: 22px; }- ol.phases li { margin: 6px 0; }- .files { columns: 2; column-gap: 26px; font: 12.5px/1.9 var(--mono); color: var(--muted); }- .files .new { color: var(--green); }- footer { margin-top: 70px; padding-top: 24px; border-top: 1px solid var(--line); color: var(--muted); font-size: 13px; }- @media (max-width: 680px) {- .cards, .cards.two { grid-template-columns: 1fr; }- .files { columns: 1; }- }-</style>-</head>-<body>-<div class="wrap">--<header class="hero">- <div class="eyebrow">Pull Request Story</div>- <h1>Signature-Based Validation</h1>- <p class="sub">A new <code>PipeSignature</code> pipe type lets authors and AI agents sketch a whole pipeline <strong>top-down</strong> — declaring each pipe's contract before writing its implementation — and still validate it at every step.</p>- <div class="meta">- <span class="chip">branch · feature/Validate-with-signatures</span>- <span class="chip">7 phases · strict TDD</span>- <span class="chip">new pipe type across 3 layers</span>- <span class="chip">~1,800 LOC of tests</span>- </div>-</header>--<div class="callout note">- <span class="lbl">Update · post-#956</span>- <p style="margin:8px 0 0">This story tells how the feature originally landed. A later refactor — <strong>#956, "consolidate validation into <code>BundleValidator</code> + execution seams"</strong> — removed the standalone <code>pipelex/pipe_run/dry_run.py</code> and its <code>dry_run_pipe</code> / <code>dry_run_pipes</code> functions. The strict signature pre-check shown in the diffs below now lives in <code>BundleValidator._signature_pre_pass</code> (run from <code>validate_pipes</code>) in <code>pipelex/pipeline/bundle_validator.py</code>, and library/job setup moved into <code>pipelex/pipeline/execution_seams.py</code>. The <em>behaviour</em> — strict default, one aggregated <code>SignaturesNotAllowedError</code>, lenient mock-mint, live-run guard — is unchanged; only the home moved. Diffs below are kept as the original PR's footprint.</p>-</div>--<h2>Why we did this</h2>-<p>Validating a bundle <strong>dry-runs every pipe</strong> with mock inputs. Dry-run requires that every sub-pipe of every controller already exists as a fully-defined pipe. There was simply <em>no way</em> to validate a half-built pipeline.</p>--<p>That blocks the most natural way to build a pipeline: sketch the orchestration first, fill in the operators one at a time. It blocks AI agents especially — they want to lay out a <code>PipeSequence</code> of five steps, then implement step 1, validate, implement step 2, validate, and so on. Until now, the bundle wouldn't even load for validation until <em>all five</em> were done.</p>--<div class="callout bug">- <span class="lbl">The gap</span>- <p style="margin:8px 0 0">A controller referencing a not-yet-written pipe fails at dry-run with a generic "pipe not found" — indistinguishable from a typo. There was no first-class way to say <em>"this pipe is a deliberate placeholder; here is its contract; I'll implement it later."</em></p>-</div>--<h2>The idea — a contract-only pipe</h2>-<p>A <code>PipeSignature</code> is a <strong>contract</strong>: <code>inputs</code>, <code>output</code>, <code>description</code> — and no implementation. It is a first-class pipe type that any pipe in a bundle can reference as a placeholder.</p>--<div class="panel" style="padding:0">- <div class="diff" style="margin:0;border:0">- <div class="fname">a PipeSignature in a .mthds file</div>-<pre><span class="ctx">[pipe.summarize_extracted]</span>-<span class="hdr">type = "PipeSignature"</span>-<span class="ctx">description = "Summarize extracted text (contract only)."</span>-<span class="ctx">inputs = { extracted = "Text" }</span>-<span class="ctx">output = "Summary"</span></pre>- </div>-</div>--<p>Three use cases drove the design:</p>-<div class="cards">- <div class="card">- <span class="n">01</span>- <h4>Agents building iteratively</h4>- <p>Sketch the orchestration with signatures, fill implementations one pipe at a time, validate at every step.</p>- </div>- <div class="card">- <span class="n">02</span>- <h4>Contract bundles</h4>- <p>Library authors publish a bundle of signatures for downstream teams to implement against.</p>- </div>- <div class="card">- <span class="n">03</span>- <h4>Safe refactors</h4>- <p>Stub a pipe with its prior contract while the body is rewritten — the rest of the graph still validates.</p>- </div>-</div>--<h2>Two validation modes</h2>-<p>This is the central design decision. Validation has two distinct purposes, so it gets two distinct modes — selected by one flag, <code>--allow-signatures</code>.</p>--<div class="cards two">- <div class="card">- <span class="n" style="color:var(--red)">STRICT · default</span>- <h4>"Ready to ship"</h4>- <p>Fails fast if <em>any</em> pipe reachable from the target is a <code>PipeSignature</code>. CI gates, release checks, and production deploys must reject half-built bundles. Raises <code>SignaturesNotAllowedError</code> listing every signature and the controller chain that reached it.</p>- </div>- <div class="card">- <span class="n" style="color:var(--green)">LENIENT · --allow-signatures</span>- <h4>"Still authoring"</h4>- <p>Accepts signatures. They participate in dry-run by minting a mock output from their declared contract. This is the agent / authoring loop — the agent CLI's <code>validate</code> surfaces the flag for exactly this use.</p>- </div>-</div>--<p>The two modes are complementary with a third line of defense — <strong>live-run enforcement</strong>. Even if strict validation is skipped, executing a signature for real raises <code>PipeSignatureNotExecutableError</code>. A half-built pipeline can never silently run on mock data.</p>--<h2>The design in one picture</h2>-<p>A signature is a real pipe type, mirrored across all three layers of the Pipelex stack — the same spec → blueprint → runtime path every other pipe takes. No discriminator machinery changed; the factory lookup convention (<code>f"{pipe_type}Factory"</code>) just needed one new registration.</p>--<div class="figure">-<svg viewBox="0 0 880 320" xmlns="http://www.w3.org/2000/svg" font-family="ui-monospace, Menlo, Consolas, monospace">- <defs>- <marker id="ah" markerWidth="9" markerHeight="9" refX="7" refY="3.2" orient="auto">- <path d="M0,0 L7,3.2 L0,6.4 Z" fill="#9aa3b6"/>- </marker>- <marker id="ahg" markerWidth="9" markerHeight="9" refX="7" refY="3.2" orient="auto">- <path d="M0,0 L7,3.2 L0,6.4 Z" fill="#5fd38f"/>- </marker>- <marker id="ahr" markerWidth="9" markerHeight="9" refX="7" refY="3.2" orient="auto">- <path d="M0,0 L7,3.2 L0,6.4 Z" fill="#f08a8a"/>- </marker>- </defs>-- <!-- layer labels -->- <text x="20" y="40" fill="#9aa3b6" font-size="11" letter-spacing="1.5">SPEC / BUILDER</text>- <text x="330" y="40" fill="#9aa3b6" font-size="11" letter-spacing="1.5">BLUEPRINT / CORE</text>- <text x="645" y="40" fill="#9aa3b6" font-size="11" letter-spacing="1.5">RUNTIME</text>-- <!-- spec box -->- <rect x="20" y="56" width="250" height="78" rx="10" fill="#1f2330" stroke="#b89cf0"/>- <text x="36" y="84" fill="#e8eaf0" font-size="14" font-weight="700">PipeSignatureSpec</text>- <text x="36" y="106" fill="#9aa3b6" font-size="11.5">authoring format · literal type,</text>- <text x="36" y="122" fill="#9aa3b6" font-size="11.5">multiplicity inputs, no `result`</text>-- <!-- blueprint box -->- <rect x="330" y="56" width="250" height="78" rx="10" fill="#1f2330" stroke="#6ea8fe"/>- <text x="346" y="84" fill="#e8eaf0" font-size="14" font-weight="700">PipeSignatureBlueprint</text>- <text x="346" y="106" fill="#9aa3b6" font-size="11.5">MTHDS language · what a</text>- <text x="346" y="122" fill="#9aa3b6" font-size="11.5">.mthds file parses into</text>-- <!-- runtime box -->- <rect x="640" y="56" width="220" height="78" rx="10" fill="#1f2330" stroke="#5fd38f"/>- <text x="656" y="84" fill="#e8eaf0" font-size="14" font-weight="700">PipeSignature</text>- <text x="656" y="106" fill="#9aa3b6" font-size="11.5">: PipeAbstract</text>- <text x="656" y="122" fill="#9aa3b6" font-size="11.5">executable runtime object</text>-- <!-- arrows between layers -->- <line x1="270" y1="95" x2="326" y2="95" stroke="#9aa3b6" stroke-width="1.6" marker-end="url(#ah)"/>- <text x="272" y="88" fill="#9aa3b6" font-size="10.5">to_blueprint()</text>- <line x1="580" y1="95" x2="636" y2="95" stroke="#9aa3b6" stroke-width="1.6" marker-end="url(#ah)"/>- <text x="566" y="88" fill="#9aa3b6" font-size="10.5">Factory.make()</text>-- <!-- runtime branches -->- <line x1="750" y1="134" x2="660" y2="196" stroke="#5fd38f" stroke-width="1.6" marker-end="url(#ahg)"/>- <line x1="750" y1="134" x2="780" y2="196" stroke="#f08a8a" stroke-width="1.6" marker-end="url(#ahr)"/>-- <rect x="430" y="200" width="270" height="92" rx="10" fill="#181b24" stroke="#2f5d44"/>- <text x="446" y="226" fill="#5fd38f" font-size="12" font-weight="700">_dry_run_pipe()</text>- <text x="446" y="248" fill="#9aa3b6" font-size="11.5">mints a mock Stuff matching the</text>- <text x="446" y="264" fill="#9aa3b6" font-size="11.5">declared output concept +</text>- <text x="446" y="280" fill="#9aa3b6" font-size="11.5">multiplicity → validation passes</text>-- <rect x="715" y="200" width="145" height="92" rx="10" fill="#181b24" stroke="#5d3434"/>- <text x="731" y="226" fill="#f08a8a" font-size="12" font-weight="700">_live_run_pipe()</text>- <text x="731" y="248" fill="#9aa3b6" font-size="11.5">raises PipeSignature-</text>- <text x="731" y="264" fill="#9aa3b6" font-size="11.5">NotExecutableError</text>- <text x="731" y="280" fill="#9aa3b6" font-size="11.5">— fails loudly</text>-- <!-- enum note -->- <rect x="20" y="200" width="385" height="92" rx="10" fill="#181b24" stroke="#2c3142"/>- <text x="36" y="226" fill="#e8eaf0" font-size="12" font-weight="700">New enum values (Phase 1 foundation)</text>- <text x="36" y="248" fill="#9aa3b6" font-size="11.5">PipeType.PIPE_SIGNATURE = "PipeSignature"</text>- <text x="36" y="264" fill="#9aa3b6" font-size="11.5">PipeCategory.PIPE_SIGNATURE — a third category,</text>- <text x="36" y="280" fill="#9aa3b6" font-size="11.5">not operator, not controller. is_controller → False.</text>-</svg>-<div class="figcap">A signature is neither an operator (no inference) nor a controller (no sub-pipe orchestration) — it earns its own <code>PipeCategory</code>.</div>-</div>--<p class="muted">A third category is a feature, not overhead: Pipelex enums are matched exhaustively with no <code>case _</code> default, so the linter forces <em>every</em> match-case in the codebase to acknowledge the new value. Nothing that needs to know about signatures can silently forget them.</p>--<h2>How strict mode finds signatures</h2>-<p>Before any dry-run starts, strict mode walks the dependency graph from the target pipe and collects every reachable signature. The walk is a pair of free functions in <code>pipe_signature/signature_walk.py</code> — <code>collect_signature_refs(pipe)</code> and its companion <code>collect_signature_paths(pipe)</code>, which records the controller chain that reached each signature so the error message is actionable.</p>--<div class="figure">-<svg viewBox="0 0 880 300" xmlns="http://www.w3.org/2000/svg" font-family="ui-monospace, Menlo, Consolas, monospace">- <defs>- <marker id="b1" markerWidth="9" markerHeight="9" refX="7" refY="3.2" orient="auto">- <path d="M0,0 L7,3.2 L0,6.4 Z" fill="#9aa3b6"/>- </marker>- </defs>-- <!-- graph on the left -->- <text x="20" y="34" fill="#9aa3b6" font-size="11" letter-spacing="1.5">DEPENDENCY GRAPH WALK</text>-- <rect x="40" y="56" width="190" height="50" rx="9" fill="#1f2330" stroke="#6ea8fe"/>- <text x="56" y="78" fill="#e8eaf0" font-size="12.5" font-weight="700">process_doc</text>- <text x="56" y="95" fill="#9aa3b6" font-size="10.5">PipeSequence · controller</text>-- <rect x="20" y="160" width="190" height="50" rx="9" fill="#1f2330" stroke="#5fd38f"/>- <text x="36" y="182" fill="#e8eaf0" font-size="12.5" font-weight="700">extract_doc</text>- <text x="36" y="199" fill="#9aa3b6" font-size="10.5">PipeLLM · operator ✓</text>-- <rect x="240" y="160" width="210" height="50" rx="9" fill="#1f2330" stroke="#f0c060"/>- <text x="256" y="182" fill="#e8eaf0" font-size="12.5" font-weight="700">summarize_extracted</text>- <text x="256" y="199" fill="#f0c060" font-size="10.5">PipeSignature · placeholder</text>-- <line x1="110" y1="106" x2="105" y2="156" stroke="#9aa3b6" stroke-width="1.5" marker-end="url(#b1)"/>- <line x1="160" y1="106" x2="320" y2="156" stroke="#9aa3b6" stroke-width="1.5" marker-end="url(#b1)"/>- <text x="118" y="135" fill="#9aa3b6" font-size="10">step 1</text>- <text x="235" y="128" fill="#9aa3b6" font-size="10">step 2</text>-- <text x="20" y="248" fill="#9aa3b6" font-size="11">collect_signature_refs() →</text>- <text x="20" y="266" fill="#5fd38f" font-size="11">{ "signature_mixed.summarize_extracted" }</text>- <text x="20" y="284" fill="#9aa3b6" font-size="11">paths → { sig: ["...process_doc"] }</text>-- <!-- divider -->- <line x1="500" y1="56" x2="500" y2="284" stroke="#2c3142" stroke-width="1"/>-- <!-- two outcomes on the right -->- <text x="530" y="34" fill="#9aa3b6" font-size="11" letter-spacing="1.5">OUTCOME PER MODE</text>-- <rect x="530" y="56" width="330" height="92" rx="10" fill="#181b24" stroke="#5d3434"/>- <text x="548" y="82" fill="#f08a8a" font-size="12.5" font-weight="700">strict (default) — refuses</text>- <text x="548" y="104" fill="#9aa3b6" font-size="11">raise SignaturesNotAllowedError</text>- <text x="548" y="122" fill="#9aa3b6" font-size="11">"process_doc depends on a placeholder:</text>- <text x="548" y="138" fill="#9aa3b6" font-size="11"> summarize_extracted (via process_doc)"</text>-- <rect x="530" y="166" width="330" height="92" rx="10" fill="#181b24" stroke="#2f5d44"/>- <text x="548" y="192" fill="#5fd38f" font-size="12.5" font-weight="700">lenient — accepts</text>- <text x="548" y="214" fill="#9aa3b6" font-size="11">signature dry-runs trivially: mints a</text>- <text x="548" y="232" fill="#9aa3b6" font-size="11">mock `Summary` Stuff. Sequence</text>- <text x="548" y="248" fill="#9aa3b6" font-size="11">validation passes end-to-end.</text>-</svg>-<div class="figcap">The walk is cycle-safe (visited-set keyed by qualified <code>pipe_ref</code>) and deterministic (sorted dependency iteration), so error messages are stable.</div>-</div>--<div class="callout note">- <span class="lbl">A real import-cycle constraint</span>- <p style="margin:8px 0 0">The walk needs to resolve dependency codes to pipe objects via <code>pipelex.hub.get_optional_pipe</code>. That import cannot live in <code>pipe_abstract</code> — the cycle <code>pipe_abstract → hub → library → pipe_library → pipe_abstract</code> is real and breaks at <em>runtime</em>, not just under pyright. So the walk is <strong>not a method on <code>PipeAbstract</code></strong>: it lives as free functions in <code>pipe_signature/signature_walk.py</code>, a module downstream of <code>hub</code>, where importing <code>get_optional_pipe</code> is free of any cycle. <code>PipeAbstract</code> keeps only <code>pipe_dependencies()</code> and <code>is_signature</code> — pure local data the walk reads.</p>-</div>--<h2>The critical code changes</h2>--<p><strong>1 — The enum foundation.</strong> A new pipe type and a new category, threaded through the two exhaustive match statements that classify pipes.</p>-<div class="diff">- <div class="fname">pipelex/core/pipes/pipe_blueprint.py</div>-<pre><span class="ctx">class PipeCategory(StrEnum):</span>-<span class="ctx"> PIPE_OPERATOR = "PipeOperator"</span>-<span class="ctx"> PIPE_CONTROLLER = "PipeController"</span>-<span class="add">+ PIPE_SIGNATURE = "PipeSignature"</span>-<span class="ctx"> ...</span>-<span class="ctx"> match self:</span>-<span class="ctx"> case PipeCategory.PIPE_CONTROLLER:</span>-<span class="ctx"> return True</span>-<span class="del">- case PipeCategory.PIPE_OPERATOR:</span>-<span class="add">+ case PipeCategory.PIPE_OPERATOR | PipeCategory.PIPE_SIGNATURE:</span>-<span class="ctx"> return False</span></pre>-</div>--<p><strong>2 — The strict pre-check.</strong> One guard, originally at the top of <code>dry_run_pipe</code> (post-#956: in <code>BundleValidator._signature_pre_pass</code>). Default <code>allow_signatures=False</code> means every caller is strict unless it opts out — the safe default that guards against a missed code path.</p>-<div class="diff">- <div class="fname">pipelex/pipe_run/dry_run.py</div>-<pre><span class="del">- async def dry_run_pipe(pipe, raise_on_failure=False) -> DryRunOutput:</span>-<span class="add">+ async def dry_run_pipe(pipe, *, allow_signatures=False, raise_on_failure=False) -> DryRunOutput:</span>-<span class="add">+ if not allow_signatures:</span>-<span class="add">+ signature_refs = collect_signature_refs(pipe=pipe)</span>-<span class="add">+ if signature_refs:</span>-<span class="add">+ offending = set() if pipe.is_signature else {pipe.pipe_ref}</span>-<span class="add">+ raise SignaturesNotAllowedError(</span>-<span class="add">+ offending_pipe_refs=offending,</span>-<span class="add">+ signature_refs=signature_refs,</span>-<span class="add">+ dep_paths=collect_signature_paths(pipe=pipe),</span>-<span class="add">+ )</span>-<span class="ctx"> try:</span>-<span class="ctx"> ... # existing dry-run logic unchanged</span></pre>-</div>--<p><strong>3 — The runtime's two paths.</strong> The whole point of the type is in these two methods: dry-run mints a contract-shaped mock; live-run refuses.</p>-<div class="diff">- <div class="fname">pipelex/pipe_signature/pipe_signature.py</div>-<pre><span class="hdr"> async def _live_run_pipe(self, ...) -> PipeOutput:</span>-<span class="add">+ raise PipeSignatureNotExecutableError(pipe_ref=self.pipe_ref)</span>-<span class="ctx"></span>-<span class="hdr"> async def _dry_run_pipe(self, ..., output_name=None, ...) -> PipeOutput:</span>-<span class="add">+ mock_name = output_name or self.code</span>-<span class="add">+ typed_named = convert_stuff_spec_to_typed_named(stuff_spec=self.output, name=mock_name)</span>-<span class="add">+ mock_stuff = WorkingMemoryFactory.make_mock_stuff(typed_named_stuff_spec=typed_named)</span>-<span class="add">+ working_memory.set_new_main_stuff(stuff=mock_stuff, name=mock_name)</span>-<span class="add">+ return PipeOutput(working_memory=working_memory, ...)</span></pre>-</div>-<p class="muted">No content generator, no LLM-mock — just <code>make_mock_stuff</code>, a helper extracted from the existing <code>make_mock_inputs</code> so the per-stuff mock logic is shared rather than duplicated.</p>--<h2>The behaviour matrix we validated</h2>-<table>- <thead><tr><th>Scenario</th><th>Mode</th><th>Outcome</th></tr></thead>- <tbody>- <tr><td>Pipeline with no signatures</td><td>strict</td><td><span class="dot g">●</span> passes — graph walk finds nothing</td></tr>- <tr><td>Signature anywhere in dep graph</td><td>strict</td><td><span class="dot r">●</span> <code>SignaturesNotAllowedError</code> — every signature + dep chain listed</td></tr>- <tr><td>Same pipeline</td><td>lenient</td><td><span class="dot g">●</span> passes — signature mints a mock output</td></tr>- <tr><td><code>validate --all</code>, orphan signature (no caller)</td><td>strict</td><td><span class="dot g">●</span> passes — iteration skips signature pipes</td></tr>- <tr><td><code>validate --all</code>, a real pipe reaches a signature</td><td>strict</td><td><span class="dot r">●</span> fails — the caller is flagged</td></tr>- <tr><td>Signature inside Sequence / Parallel / Condition / Batch</td><td>both</td><td><span class="dot a">●</span> walk descends every controller kind</td></tr>- <tr><td>Live-run a bundle containing a signature</td><td>—</td><td><span class="dot r">●</span> <code>PipeSignatureNotExecutableError</code> — never runs on mock data</td></tr>- <tr><td>Multiplicity inputs (<code>Doc[]</code>, <code>Image[3]</code>)</td><td>lenient</td><td><span class="dot g">●</span> round-trip through interpreter unchanged</td></tr>- </tbody>-</table>--<div class="callout">- <span class="lbl">The subtle one — aggregation order</span>- <p style="margin:8px 0 0">The aggregated pre-pass over the whole batch (originally <code>dry_run_pipes</code>, now <code>BundleValidator._signature_pre_pass</code>) makes the user see <em>every</em> offending signature in one error, not just the first. The first cut aggregated dep-paths "first wins" — but when the signature pipe itself was iterated before its controller, that recorded an empty chain and never replaced it. Fixed: the aggregator keeps the <strong>longest</strong> known dep chain, so the message always shows the most informative path.</p>-</div>--<h2>CLI surface</h2>-<ul class="tight">- <li><code>pipelex validate pipe <code> [--allow-signatures]</code> and <code>validate bundle <path> [--allow-signatures]</code> — strict by default.</li>- <li><code>pipelex validate --all</code> — strict; iterates non-signature pipes only (orphan signatures aren't flagged — they're deliberate placeholders).</li>- <li>The agent CLI's <code>validate</code> subcommands carry the same flag — strict by default, so CI behaviour is consistent across both CLIs.</li>- <li>In lenient mode the summary line gains a suffix in-place: <code>Successfully validated bundle 'X' (2 signatures)</code> — empty when no signature was involved, so finished bundles read naturally.</li>- <li>Strict-mode failures render under an <strong>"Unimplemented Signatures"</strong> heading showing the dep chain, via <code>error_handlers.py</code>.</li>-</ul>--<h2>What landed</h2>-<div class="panel">-<div class="files">-<span class="new">+ pipe_signature/pipe_signature.py</span>-<span class="new">+ pipe_signature/pipe_signature_blueprint.py</span>-<span class="new">+ pipe_signature/pipe_signature_factory.py</span>-<span class="new">+ pipe_signature/exceptions.py</span>-<span class="new">+ pipe_signature/signature_walk.py</span>-<span class="new">+ builder/pipe/pipe_signature_spec.py</span>-core/pipes/pipe_blueprint.py-core/pipes/pipe_abstract.py-core/bundles/pipelex_bundle_blueprint.py-core/memory/working_memory_factory.py-core/registry_models.py-pipeline/bundle_validator.py <span class="muted">(strict pre-check; post-#956, was pipe_run/dry_run.py)</span>-pipeline/validate_bundle.py-pipe_controllers/pipe_controller.py-builder/pipe/pipe_spec_union.py-builder/pipe/pipe_spec_map.py-language/mthds_schema_generator.py-cli/commands/validate/*-cli/agent_cli/commands/validate/*-cli/error_handlers.py-<span class="new">+ docs/.../pipes/signature-pipes.md</span>-<span class="new">+ tests/integration/.../pipe_signature/ (5 suites)</span>-<span class="new">+ tests/e2e/test_signature_validation_*.py</span>-<span class="new">+ tests/e2e/fixtures/signature_bundles/*.mthds</span>-<span class="new">+ tests/unit/.../test_pipe_signature_*.py</span>-</div>-</div>-<p class="muted">Built strict-TDD across 7 phases with stop-checkpoints at phases 3, 5, and 6: type-system foundations → blueprint + runtime + factory → spec layer → strict pre-check → CLI surface → real-<code>.mthds</code> E2E → docs & CHANGELOG. Failing tests written first (red) in every phase. <code>make agent-check</code> and <code>make agent-test</code> green throughout.</p>--<h2 style="border-left-color:var(--green)">Merge gate — closed (Phase 7.4 done)</h2>-<p>This section originally flagged one open item: the bundled <code>plxt</code> JSON Schema (shipped from <code>vscode-pipelex</code>) did not yet know about <code>type = "PipeSignature"</code>, so the branch carried a temporary <code>[[rule]]</code> block in <code>.pipelex/plxt.toml</code> pointing <code>**/*.mthds</code> validation at a locally-regenerated <code>derived/mthds_schema.json</code>. <strong>That gate is now closed</strong> — leaving the history here for context.</p>--<div class="callout note">- <span class="lbl">Resolved</span>- <p style="margin:8px 0 0"><code>pipelex-tools</code> <strong>0.6.0</strong> (and the matching <code>vscode-pipelex</code> extension) now bundle an MTHDS schema that knows <code>type = "PipeSignature"</code> and requires <code>type</code> on every pipe variant. The branch-local <code>[[rule]]</code> block has been <strong>removed</strong> from both <code>.pipelex/plxt.toml</code> and the <code>pipelex/kit/configs/plxt.toml</code> template (the two are byte-identical again), the <code>pipelex-tools</code> floor in <code>pyproject.toml</code> is bumped to <code>>=0.6.0</code>, and <code>plxt lint</code> validates <code>.mthds</code> against the <em>bundled</em> schema directly — green across the repo, including the <code>signature_bundles/*.mthds</code> fixtures.</p>-</div>--<p>What the close-out took (Phase 7.4 in the archived plan), now all done:</p>-<ol class="phases">- <li>✅ In <code>vscode-pipelex</code> / <code>pipelex-tools</code> 0.6.0: regenerated the bundled MTHDS schema so <code>PipeSignature</code> is a valid pipe type; cut the <code>plxt</code> / extension release.</li>- <li>✅ In this repo: removed the <code>[[rule]]</code> block from <code>.pipelex/plxt.toml</code> <em>and</em> the kit template so the bundled schema takes over again, and bumped the <code>pipelex-tools</code> floor to <code>>=0.6.0</code>.</li>- <li>✅ Re-ran <code>make plxt-lint</code> / <code>make agent-check</code> with the bundled <code>plxt</code> 0.6.0 and no local rule — green against the bundled schema, including the signature fixtures. That is the ship-readiness signal.</li>-</ol>--<h2>Explicit non-goals</h2>-<ul class="tight muted">- <li><strong>Signature concepts</strong> — concept stubs analogous to pipe signatures. Bigger design, separate plan.</li>- <li><strong><code>signature_for</code>-driven mocks</strong> — e.g. <code>signature_for = "PipeImgGen"</code> minting a real <code>ImageContent</code> URL. The field is captured today as a tooling hint only; specialization is deferred.</li>- <li><strong>Bundle migration</strong> — existing bundles are untouched. Authors opt in.</li>- <li><strong>Parallel <code>[signature.foo]</code> TOML table</strong> — possible syntactic sugar later, not a structural alternative.</li>-</ul>--<footer>- Full TDD plan with phase-by-phase checkpoints archived at- <code>wip/archive/signature-based-validation-tdd-plan.md</code>.- Design rationale & rejected alternatives in <code>wip/signature-based-validation/signature-based-validation.md</code>.- Questions → ping me on the PR.-</footer>--</div>-</body>-</html>diff --git a/wip/history/signature-based-validation/signature-based-validation.md b/wip/history/signature-based-validation/signature-based-validation.mddeleted file mode 100644index 8825d5d12..000000000--- a/wip/history/signature-based-validation/signature-based-validation.md+++ /dev/null@@ -1,105 +0,0 @@-# Signature-Based Validation for Partially-Defined Pipelines--Status: **Implemented and landed** on `feature/Validate-with-signatures-3` — Phases 1–7.4 complete; `make agent-check`, `make agent-test`, `make ccs` (config-sync) and `make plxt-lint` (against the bundled schema) all green. The cross-repo merge-gate is **closed**: `pipelex-tools` 0.6.0 ships a bundled MTHDS schema that knows `type = "PipeSignature"` (and now requires `type` on every pipe variant), the branch-local schema rule has been removed from both `.pipelex/plxt.toml` and the `pipelex/kit/configs/plxt.toml` template, and the `plxt` floor is bumped to `>=0.6.0`. The branch is ship-ready. The CHANGELOG `[Unreleased]` section is the user-facing record; this doc is the engineering current-state plus a reviewer verification map.--## What it does--`PipeSignature` is a first-class pipe type (`type = "PipeSignature"`) that declares a pipe's `inputs`, `output`, and `description` without an implementation — a contract. Any pipe in a bundle can use one as a placeholder, so an author or agent can sketch a complete pipeline top-down and fill implementations one pipe at a time, validating at every step.--Use cases: AI agents iteratively building pipelines; library authors publishing contract bundles; refactors that stub a pipe with its prior contract while the body is rewritten.--## Two validation modes (as built)--Validation has two distinct purposes, so it has two modes. The mode is the contract — there is no "detect intent" heuristic.--- **Strict (default)** — fails if any pipe reachable from the target *is* or *reaches* a `PipeSignature`. This is the "ready to run in production" semantic. Raises `SignaturesNotAllowedError` listing every reachable signature plus the controller chain (`signature_refs`, `dep_paths`) and the suggested fix.-- **Lenient (`--allow-signatures`)** — accepts signatures; they participate in dry-run by minting a mock `Stuff` from their declared output contract (multiplicity included). For authoring and agent flows.--**Defaults: both the main CLI and the agent CLI are strict by default.** (This corrects an earlier design note that had the agent CLI lenient-by-default — it is not; see `test_agent_validate_defaults_strict.py`.) Lenient is opt-in via `--allow-signatures` everywhere.--Strict and lenient are complementary with live-run safety: `_live_run_pipe` on a signature always raises `PipeSignatureNotExecutableError`, so even if strict validation is skipped or a signature is added between validate and run, live execution fails loudly.--### Whole-bundle vs single-pipe strict semantics--- `validate bundle <file>` (strict) rejects a bundle that *contains* any signature, reached or not — an unimplemented placeholder means the bundle is not fully runnable. (Mechanically: the strict pre-check fires for every signature in the batch because a signature reaches itself.)-- `validate --all` (strict) filters signature pipes out of the iteration, then runs each remaining pipe's strict pre-check — so an orphan signature with no caller does not fail `--all`, but a non-signature pipe that reaches one does.-- To validate just the implemented slice of a partially-stubbed bundle, select it with `--pipe <code>`: only that pipe is dry-run (others are loaded so dependencies resolve). Strict mode is still enforced on the selected pipe.--## Reviewer verification map--Each row names where the behavior lives and the test that pins it.--| Concern | Shipped in | Verified by |-|---|---|---|-| Type system: `PipeType.PIPE_SIGNATURE`, `PipeCategory.PIPE_SIGNATURE`, `is_signature`, default `pipe_dependencies()` on `PipeAbstract` + per-controller override | `pipelex/core/pipes/pipe_abstract.py`, `pipelex/pipe_controllers/pipe_controller.py`, pipe type/category enums | exhaustive-match enum tests; integration suite below |-| Union membership + registry wiring (the pipe type is recognized and `PipeSignatureFactory` resolves) | `pipelex/core/bundles/pipelex_bundle_blueprint.py` (`PipeBlueprintUnion`), `pipelex/core/registry_models.py` (`PIPE_SIGNATURES`, `PIPE_SIGNATURES_FACTORY`) | `tests/integration/pipelex/pipe_signature/` |-| Blueprint + runtime + factory; dry-run mints a mock `Stuff` from the declared output (multiplicity included) | `pipelex/pipe_signature/pipe_signature_blueprint.py`, `pipe_signature.py`, `pipe_signature_factory.py`, `pipelex/core/memory/working_memory_factory.py` (`make_mock_content`) | `tests/integration/pipelex/pipe_signature/` |-| Spec layer: `PipeSignatureSpec` (literal `type`, optional `signature_for`, multiplicity inputs, no `result`) | `pipelex/builder/pipe/pipe_signature_spec.py` | `tests/unit/pipelex/builder/pipe/test_pipe_signature_spec.py` |-| Strict pre-check: `collect_signature_refs` / `collect_signature_paths`, `SignaturesNotAllowedError`, `allow_signatures` threading + cross-batch aggregation, `ValidateBundleError.signature_check_error` | `pipelex/pipe_signature/signature_walk.py`, `pipelex/pipeline/bundle_validator.py` (`_signature_pre_pass`, run from `validate_pipes`), `pipelex/pipeline/validate_bundle.py`, `pipelex/pipe_signature/exceptions.py` (`SignaturesNotAllowedError`), `pipelex/pipeline/exceptions.py` (`ValidateBundleError`) | `tests/integration/pipelex/pipe_signature/test_dry_run_strict_mode.py`, `tests/e2e/test_signature_validation_mthds.py` |-| Main CLI: `validate pipe/bundle [--allow-signatures]`, `--all` filters signatures, lenient summary suffix, friendly `SignaturesNotAllowedError` rendering | `pipelex/cli/commands/validate/_validate_core.py`, `pipelex/cli/error_handlers.py` | `tests/integration/pipelex/cli/test_validate_signatures_cli.py`, `test_validate_signatures_summary.py`, `tests/e2e/test_signature_validation_cli.py` |-| Agent CLI: same flags (strict default), `--pipe` single-slice validation | `pipelex/cli/agent_cli/commands/validate/_validate_core.py`, `bundle_cmd.py` | `tests/integration/pipelex/cli/test_agent_validate_defaults_strict.py`, `test_agent_validate_pipe_in_bundle.py` |-| Live-run guard (`PipeSignatureNotExecutableError`) | `pipelex/pipe_signature/pipe_signature.py` (raise), `pipelex/pipe_signature/exceptions.py` (class) | `tests/e2e/test_signature_validation_mthds.py` (live-run case) |-| Schema generator strips `pipe_category` from `PipeSignatureBlueprint` | schema generator (`pipelex-dev generate-mthds-schema`) | runs clean in `make agent-check` |-| Schema generator requires `type` on every pipe blueprint variant (set derived from `PipeBlueprintUnion`, no hardcoded drift) so a type-less pipe table fails with a clear "missing type" instead of an ambiguous `oneOf` multi-match | `pipelex/language/mthds_schema_generator.py` (`_require_type_on_pipe_definitions`) | `tests/unit/pipelex/language/test_mthds_schema.py` |-| Docs + CHANGELOG | `docs/building-methods/pipes/signature-pipes.md`, `docs/tools/cli/validate.md`, `docs/tools/cli/agent-cli.md`, `mkdocs.yml` (nav), `pipelex/builder/CLAUDE.md`, CHANGELOG `[Unreleased]` | — |--## Review fixes (post-merge, 2026-05-31, commit `4cc60c52`)--After `dev` was merged in, the review bots surfaced reconciliation issues; the confirmed ones landed with regression tests:--- **`--pipe` validates only the requested pipe.** `validate_bundle` gained a `dry_run_pipe_codes` param (with a `_pipes_to_dry_run` helper); `validate_pipe_in_bundle_core` now loads the bundle but dry-runs only the selected pipe, so an unrelated signature or otherwise-broken pipe no longer blocks validating an implemented slice. A typo'd `--pipe` raises `PipeNotFoundError` instead of passing vacuously. — `test_agent_validate_pipe_in_bundle.py`-- **`validate --all` renders `SignaturesNotAllowedError` as a friendly CLI error** (`handle_signatures_not_allowed_error`) instead of an unhandled traceback, matching the bundle/pipe paths. — `test_validate_signatures_cli.py`-- **`handle_signatures_not_allowed_error` honors `--traceback`**, consistent with every other `handle_*`.-- **`PipeSignatureSpec.rendered_pretty` escapes dynamic values** before Rich markup, so concept multiplicity (`Doc[]`, `Img[3]`) and bracketed descriptions render literally instead of being parsed as markup. — `test_pipe_signature_spec.py`-- **Strict whole-bundle docstrings clarified** (on the then-`dry_run_pipes`, now `BundleValidator.validate_pipes` — see the Post-#956 note below — and on `PipeSignatureSpec`): whole-bundle strict rejects any bundle containing a signature; `--pipe` / `--allow-signatures` are the escape hatches.-- A separate `.pipelex/plxt.toml` "schema rule clobbers formatting" report was reviewed and confirmed a **false positive** (taplo merges formatting across matching rules; schema is resolved independently) — no change.--## Review fixes (additive-library completeness, 2026-06-07)--A second review pass found three paths that still did per-file / pre-merge work, breaking the same-domain authoring the merged crate is meant to resolve. The concept side had moved same-domain resolution to the merged level (`validate_concept_references_in_blueprints` in the loader); these three completed the same move for the pipe side and the remaining loaders. All confirmed against the code and fixed with regression tests:--- **Qualified same-domain pipe refs now resolve across sibling files.** The per-file `validate_local_pipe_references` `@model_validator` on `PipelexBundleBlueprint` eagerly rejected a qualified same-domain step ref (`research.find_key_findings`) unless the target pipe was declared in the *same* file — while the bare form was already deferred (the exact asymmetry the concept side had already removed). In an additive library the concrete can live in a sibling file, reconciled by `LibraryCrateFactory`. Every other branch of the validator was already a no-op, so it is removed entirely; a genuinely missing ref is still caught post-merge by `validate_pipe_library_with_libraries` (`get_required_pipe` → `PipeNotFoundError` → `LibraryError`). The shared `collect_pipe_references` helper stays (used by `visibility_utils` and the dependency loader). — `tests/integration/pipelex/libraries/test_additive_multi_file_library.py` (`test_qualified_same_domain_sibling_pipe_ref_resolves`, `test_undeclared_qualified_pipe_ref_surfaces_clean_error`); the now-obsolete `tests/unit/pipelex/core/bundles/test_pipelex_bundle_blueprint_pipe_validation.py` was removed (it only exercised the deleted validator).-- **Concepts-only loading now validates concept references.** `LibraryManager.load_concepts_only_from_blueprints` skipped `validate_concept_references_in_blueprints` (only the full `load_from_blueprints` ran it), so the `pipelex structures` / `load_concepts_only` path accepted a dangling `concept_ref` / `item_concept_ref` and left an unrebuildable schema in the library. It now runs the same merged validator (with `already_loaded_concept_refs` for cross-batch resolution) before adding the batch's concepts. — `tests/integration/pipelex/pipeline/test_load_concepts_only.py` (`test_load_concepts_only_rejects_dangling_concept_ref`, `..._item_concept_ref`).-- **Dependency packages now load through the crate merge.** `LibraryManager._load_single_dependency` built pipes per-blueprint and called `add_new_pipe` directly, so a multi-file dependency (a `PipeSignature` header + concrete sibling) collided on the duplicate pipe ref — swallowed by the per-pipe `except (PipeLibraryError, ValidationError)` as a warning, silently dropping one declaration (load-order dependent: a signature-only survivor is non-executable). It now builds a `LibraryCrate` from the dependency's blueprints first (signature/concrete reconciliation + normalized contract check + duplicate detection) and runs `validate_concept_references_in_blueprints`, then loads the reconciled `crate.pipes` (export filter and synthetic-helper handling preserved). The per-pipe handler now catches only `ValidationError` (a single malformed blueprint stays lenient); a genuine duplicate definition raises from the crate instead of warning — aligning dependencies with the main load path. — `tests/unit/pipelex/libraries/test_dependency_multi_file_reconciliation.py`.--## Architecture (Option A, as built)--Three layers, parallel additions; the discriminator stays `type`:--```-spec/builder PipeSignatureSpec (pipelex/builder/pipe/pipe_signature_spec.py)-blueprint/core PipeSignatureBlueprint (pipelex/pipe_signature/pipe_signature_blueprint.py)-runtime PipeSignature (pipelex/pipe_signature/pipe_signature.py)-```--Signatures are a third pipe *category* (not operator, not controller): no inference, no sub-pipe orchestration. The separate category surfaces them cleanly in tooling and the `is_signature` property, and the exhaustive-match rule for enums forces every match-case to acknowledge the new value (the linter catches every place that needs to know). The factory lookup convention (`f"{pipe_type}Factory"`) needed no dispatch changes — just registering `PipeSignatureFactory`.--The dependency-graph walk lives as free functions in `pipelex/pipe_signature/signature_walk.py`, downstream of `pipelex.hub`, **not** as methods on `PipeAbstract` — `pipe_abstract` importing `hub` forms a real runtime import cycle (`pipe_abstract → hub → libraries.library → libraries.pipe.pipe_library → pipe_abstract`). `PipeAbstract` keeps only `is_signature` and a default `pipe_dependencies()`.--> **Post-#956 note.** The dry-run/validation consolidation (#956, commit `7b2ae132`) removed the standalone `pipelex/pipe_run/dry_run.py` and its `dry_run_pipe` / `dry_run_pipes` functions. The strict signature pre-check now runs inside `BundleValidator.validate_pipes` (`_signature_pre_pass`) in `pipelex/pipeline/bundle_validator.py`, which is what `validate_bundle` (and the `validate --all` standalone path via `acquire_and_validate`) call with `allow_signatures`. The behavior — one aggregated `SignaturesNotAllowedError` over the full pipe set, keeping the longest dep chain per signature — is unchanged; only the home moved. The `_signature_pre_pass` docstring spells out the wiring-before-signature ordering it preserves.--`PipeSignature._dry_run_pipe` mints a `Stuff` of the declared output concept/multiplicity (via `WorkingMemoryFactory.make_mock_content`); `_live_run_pipe` raises `PipeSignatureNotExecutableError`; the static/library validators are no-ops.--## Design rationale & tradeoffs (for the record)--- **MTHDS surface grows** — kept `PipeSignature` deliberately minimal so it reads as obviously-a-contract.-- **Strict gate must be wired at every entry** — default `allow_signatures=False` guards against missed code paths; the test matrix covers the rest.-- **Mock fidelity has limits** — a signature describes structure, not values; downstream pipes whose prompts reference specific content shapes may dry-run but fail live (the general DRY-mode limitation).-- **`required_variables` loses dotted-path info** for signatures (no prompt/template). Acceptable for the target use cases.-- **No signature mode for concepts** — out of scope.--Rejected alternatives: compiling signatures to `PipeFunc` at load (layering violation, silent live success on mock data); a parallel `signature` table in `.mthds` (two resolution paths); implicit "skeletal" pipes such as a prompt-less `PipeLLM` (bug-vs-feature is indistinguishable from field presence).--## Done: Phase 7.4 — cross-repo schema merge-gate (closed)--The merge-gate is closed. `pipelex-tools` 0.6.0 (PyPI) and the matching `vscode-pipelex` extension release bundle an MTHDS schema that knows `type = "PipeSignature"` and requires `type` on every pipe variant. On this branch, in lock-step:--1. ✅ `vscode-pipelex` / `pipelex-tools` 0.6.0 shipped the schema-aware bundled `plxt`.-2. ✅ The branch-local `[[rule]] / [rule.schema]` block was removed from **both** `.pipelex/plxt.toml` and the `pipelex/kit/configs/plxt.toml` template (the two are byte-identical again, with no net change versus `dev`), the `pipelex-tools` floor in `pyproject.toml` is bumped to `>=0.6.0`, and the now-vestigial `generate-mthds-schema-quiet` prerequisite on the `plxt-lint` / `merge-check-plxt-lint` Makefile targets (which existed only to feed the branch-local rule) was reverted — `plxt lint` now validates `.mthds` against the bundled schema directly.-3. ✅ Re-verified with the bundled `plxt` 0.6.0 installed and the branch-local rule gone: `make plxt-lint` is green across the repo — including `tests/e2e/fixtures/signature_bundles/*.mthds`, which use `type = "PipeSignature"` and now validate against the bundled schema — and `make ccs` (config-sync) passes.--The earlier ⚠️ counter-item — the branch-local schema block having been copied into the user-shipped `pipelex/kit/configs/plxt.toml` template — is resolved by step 2: the block is gone from the template, so `pipelex init` no longer points end-user projects at a `derived/mthds_schema.json` path that does not exist for them.--## History--The phase-by-phase TDD implementation log lives at [`../archive/signature-based-validation-tdd-plan.md`](../archive/signature-based-validation-tdd-plan.md) (kept as the build record; this doc is the live current-state reference).diff --git a/wip/history/signature-taxonomy-refactor.html b/wip/history/signature-taxonomy-refactor.htmldeleted file mode 100644index be95fd37f..000000000--- a/wip/history/signature-taxonomy-refactor.html+++ /dev/null@@ -1,705 +0,0 @@-<!DOCTYPE html>-<html lang="en">-<head>-<meta charset="utf-8" />-<meta name="viewport" content="width=device-width, initial-scale=1.0" />-<title>PipeSignature: contract, not pipe type — taxonomy eviction</title>-<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>-<style>- :root {- --bg: #0f1117;- --panel: #171a23;- --panel-2: #1d212c;- --ink: #e6e8ee;- --ink-dim: #a4abbb;- --ink-faint: #717789;- --accent: #7aa2ff;- --accent-2: #5ad1b3;- --warn: #f0b35e;- --bad: #ef6f6f;- --good: #5ad19a;- --border: #2a2f3d;- --code-bg: #0c0e14;- --add: #1f3a2b;- --add-ink: #8be0a8;- --del: #3a2026;- --del-ink: #f08f9c;- --radius: 12px;- --mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;- --sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;- }- * { box-sizing: border-box; }- html { scroll-behavior: smooth; }- body {- margin: 0;- background: var(--bg);- color: var(--ink);- font-family: var(--sans);- line-height: 1.62;- font-size: 16px;- }- .wrap { max-width: 940px; margin: 0 auto; padding: 0 24px 120px; }- header.hero {- padding: 56px 24px 40px;- background:- radial-gradient(1200px 400px at 20% -10%, rgba(122,162,255,0.16), transparent 60%),- radial-gradient(900px 380px at 95% 0%, rgba(90,209,179,0.12), transparent 55%);- border-bottom: 1px solid var(--border);- }- .hero-inner { max-width: 940px; margin: 0 auto; }- .eyebrow {- text-transform: uppercase;- letter-spacing: 0.14em;- font-size: 12px;- font-weight: 700;- color: var(--accent-2);- margin: 0 0 12px;- }- h1 { font-size: 34px; line-height: 1.18; margin: 0 0 14px; letter-spacing: -0.01em; }- h1 code { font-size: 0.82em; }- .sub { color: var(--ink-dim); font-size: 17px; margin: 0; max-width: 760px; }- .badges { margin-top: 22px; display: flex; gap: 10px; flex-wrap: wrap; }- .badge {- font-size: 12.5px; font-weight: 600;- padding: 5px 11px; border-radius: 999px;- border: 1px solid var(--border); background: var(--panel);- color: var(--ink-dim);- display: inline-flex; gap: 7px; align-items: center;- }- .badge.green { color: var(--good); border-color: rgba(90,209,154,0.4); }- .badge .dot { width: 7px; height: 7px; border-radius: 50%; background: currentColor; }-- h2 {- font-size: 24px; margin: 52px 0 16px; letter-spacing: -0.01em;- padding-bottom: 8px; border-bottom: 1px solid var(--border);- }- h3 { font-size: 18.5px; margin: 32px 0 10px; color: var(--ink); }- h2 .num {- color: var(--accent); font-variant-numeric: tabular-nums;- margin-right: 10px; font-weight: 700;- }- p { margin: 12px 0; }- a { color: var(--accent); text-decoration: none; }- a:hover { text-decoration: underline; }- code {- font-family: var(--mono); font-size: 0.88em;- background: var(--code-bg); border: 1px solid var(--border);- padding: 1px 6px; border-radius: 6px; color: #d8dcff;- }- strong { color: #fff; font-weight: 650; }- em { color: var(--ink); }-- /* TL;DR */- .tldr {- margin: 28px 0 8px;- background: linear-gradient(180deg, var(--panel) 0%, var(--panel-2) 100%);- border: 1px solid var(--border);- border-left: 4px solid var(--accent);- border-radius: var(--radius);- padding: 22px 26px;- }- .tldr h2 { border: 0; margin: 0 0 6px; font-size: 20px; }- .tldr .lead { font-size: 17px; color: var(--ink); margin: 0 0 14px; }- .tldr ul { margin: 10px 0 0; padding-left: 20px; }- .tldr li { margin: 7px 0; color: var(--ink-dim); }- .tldr li strong { color: #fff; }-- /* callouts */- .note, .principle {- border: 1px solid var(--border); border-radius: var(--radius);- padding: 16px 20px; margin: 20px 0; background: var(--panel);- }- .note { border-left: 4px solid var(--accent-2); }- .note .label {- font-size: 12px; text-transform: uppercase; letter-spacing: .08em;- font-weight: 700; color: var(--accent-2); margin: 0 0 6px;- }- .principle { border-left: 4px solid var(--warn); }- .principle .label {- font-size: 12px; text-transform: uppercase; letter-spacing: .08em;- font-weight: 700; color: var(--warn); margin: 0 0 6px;- }-- /* TOC */- .toc {- background: var(--panel); border: 1px solid var(--border);- border-radius: var(--radius); padding: 18px 22px; margin: 28px 0;- }- .toc-title { font-size: 12.5px; text-transform: uppercase; letter-spacing: .1em; color: var(--ink-faint); font-weight: 700; margin: 0 0 10px; }- .toc ol { margin: 0; padding-left: 22px; columns: 2; column-gap: 36px; }- .toc li { margin: 5px 0; }- @media (max-width: 640px) { .toc ol { columns: 1; } }-- /* diagrams */- .diagram {- background: var(--panel); border: 1px solid var(--border);- border-radius: var(--radius); padding: 18px; margin: 22px 0;- overflow-x: auto; text-align: center;- }- .diagram .cap {- text-align: center; color: var(--ink-faint); font-size: 13px;- margin-top: 12px; font-style: italic;- }- .mermaid { display: flex; justify-content: center; }-- /* tabs */- .tabs { margin: 22px 0; border: 1px solid var(--border); border-radius: var(--radius); overflow: hidden; }- .tab-bar { display: flex; flex-wrap: wrap; background: var(--panel-2); border-bottom: 1px solid var(--border); }- .tab-btn {- appearance: none; border: 0; background: transparent; cursor: pointer;- color: var(--ink-dim); font-family: var(--sans); font-size: 13.5px; font-weight: 600;- padding: 11px 16px; border-bottom: 2px solid transparent;- }- .tab-btn:hover { color: var(--ink); }- .tab-btn.active { color: #fff; border-bottom-color: var(--accent); background: var(--panel); }- .tab-panel { display: none; }- .tab-panel.active { display: block; }- .tab-panel .filelabel {- font-family: var(--mono); font-size: 12px; color: var(--ink-faint);- padding: 9px 16px 0; background: var(--code-bg);- }-- /* code blocks & diffs */- pre {- margin: 0; background: var(--code-bg);- overflow-x: auto; padding: 16px 18px;- font-family: var(--mono); font-size: 13px; line-height: 1.55;- color: #cdd2e0;- }- pre.diff { padding: 14px 0; }- .diff .ln { display: block; padding: 0 18px; white-space: pre; }- .diff .add { background: var(--add); color: var(--add-ink); }- .diff .del { background: var(--del); color: var(--del-ink); }- .diff .ctx { color: #9aa0b3; }- .diff .hdr { color: var(--ink-faint); }- .diff .meta { color: var(--accent); }- .standalone-code {- border: 1px solid var(--border); border-radius: var(--radius);- overflow: hidden; margin: 20px 0;- }- .standalone-code .filelabel {- font-family: var(--mono); font-size: 12px; color: var(--ink-faint);- padding: 9px 16px; background: var(--panel-2); border-bottom: 1px solid var(--border);- }-- /* details / foldables */- details {- border: 1px solid var(--border); border-radius: var(--radius);- margin: 16px 0; background: var(--panel); overflow: hidden;- }- details summary {- cursor: pointer; padding: 14px 18px; font-weight: 600; color: var(--ink);- list-style: none; display: flex; align-items: center; gap: 10px;- user-select: none;- }- details summary::-webkit-details-marker { display: none; }- details summary::before {- content: "▶"; color: var(--accent); font-size: 11px; transition: transform .15s ease;- }- details[open] summary::before { transform: rotate(90deg); }- details summary:hover { background: var(--panel-2); }- details .body { padding: 4px 20px 18px; border-top: 1px solid var(--border); }- details .body > *:first-child { margin-top: 14px; }-- /* tables */- table { border-collapse: collapse; width: 100%; margin: 20px 0; font-size: 14px; }- th, td { text-align: left; padding: 9px 12px; border-bottom: 1px solid var(--border); vertical-align: top; }- th { color: var(--ink-faint); font-size: 12px; text-transform: uppercase; letter-spacing: .06em; font-weight: 700; }- td code { font-size: 0.85em; }- tbody tr:hover { background: var(--panel); }-- ul.clean { list-style: none; padding-left: 0; }- ul.clean li { padding-left: 26px; position: relative; margin: 9px 0; }- ul.clean li::before { content: "›"; position: absolute; left: 8px; color: var(--accent); font-weight: 700; }-- .pill { font-family: var(--mono); font-size: 12px; padding: 1px 7px; border-radius: 5px; }- .pill.keep { background: rgba(90,209,154,0.13); color: var(--good); border: 1px solid rgba(90,209,154,0.3); }- .pill.gone { background: rgba(239,111,111,0.12); color: var(--bad); border: 1px solid rgba(239,111,111,0.3); }-- footer { margin-top: 60px; padding-top: 20px; border-top: 1px solid var(--border); color: var(--ink-faint); font-size: 13px; }- .kbd { font-family: var(--mono); font-size: 12px; background: var(--panel-2); border: 1px solid var(--border); border-radius: 5px; padding: 1px 6px; }-</style>-</head>-<body>--<header class="hero">- <div class="hero-inner">- <p class="eyebrow">Refactor · pipe taxonomy</p>- <h1><code>PipeSignature</code> is a contract, not an executable pipe type</h1>- <p class="sub">We evicted <code>PipeSignature</code> from the <code>PipeType</code> / <code>PipeCategory</code> taxonomy. A signature stays a <code>PipeAbstract</code> (so the dry-run engine can still mock it), but it is no longer pretending to be a sibling of <code>PipeLLM</code> or <code>PipeSequence</code>.</p>- <div class="badges">- <span class="badge green"><span class="dot"></span>Landed</span>- <span class="badge"><span class="dot"></span><code>make tb</code> · <code>agent-check</code> · <code>agent-test</code> green</span>- <span class="badge"><span class="dot"></span>MTHDS schema: no diff</span>- <span class="badge"><span class="dot"></span>Breaking: <code>PipeCategory.PIPE_SIGNATURE</code> removed</span>- </div>- </div>-</header>--<div class="wrap">-- <!-- TL;DR -->- <div class="tldr">- <h2>TL;DR</h2>- <p class="lead">A pipe <em>has</em> a signature (its contract). A <code>PipeSignature</code> is that contract <strong>alone</strong> — no body. It was a member of <code>PipeType</code>/<code>PipeCategory</code>, which framed it as a way of <em>running</em> next to <code>PipeLLM</code>/<code>PipeSequence</code>. It isn't one. We pulled it out of the executable taxonomy.</p>- <ul>- <li><strong>Removed</strong> <code>PipeType.PIPE_SIGNATURE</code> and <code>PipeCategory.PIPE_SIGNATURE</code>. The two enums now describe <em>only</em> things that execute (operator vs controller).</li>- <li><strong>Identity by class, not enum.</strong> <code>is_signature</code> is now a class fact — base returns <code>False</code>, <code>PipeSignature</code> / <code>PipeSignatureBlueprint</code> override to <code>True</code> — instead of reading an enum field. Every existing call site keeps working unchanged.</li>- <li><strong>It stays a <code>PipeAbstract</code>.</strong> The dry-run engine dispatches over <code>PipeAbstract</code>, and a signature must be dry-runnable (verify inputs, mock output). A signature carries <code>type = "PipeSignature"</code> with <code>pipe_category = None</code>.</li>- <li><strong>One honest allowlist.</strong> A new <code>PIPE_SIGNATURE_TYPE_TAG</code> + <code>valid_pipe_type_tags()</code> expresses "legal <code>type</code> = executable kinds ∪ the signature tag" as the single source of truth for the three validators that gate it.</li>- <li><strong>Breaking but tiny.</strong> Anything matching on <code>PipeCategory.PIPE_SIGNATURE</code> must drop that arm. <code>signature_for</code> is kept (mthds-plugins depends on it); its two guard validators are deleted because the rejection is now <em>structural</em>.</li>- <li><strong>Invisible to the wire.</strong> The blueprint union discriminates on <code>type</code> (unchanged), so the generated MTHDS JSON Schema produced <strong>no diff</strong>.</li>- </ul>- </div>-- <div class="toc">- <p class="toc-title">Contents</p>- <ol>- <li><a href="#diagnosis">The diagnosis</a></li>- <li><a href="#decision">The principle</a></li>- <li><a href="#before-after">Before / after taxonomy</a></li>- <li><a href="#keystone">The keystone: identity by class</a></li>- <li><a href="#diffs">Critical code changes</a></li>- <li><a href="#runtime">Runtime behaviour</a></li>- <li><a href="#scope">Scope: spec layer stays put</a></li>- <li><a href="#kept">What we kept & deferred</a></li>- <li><a href="#touchpoints">Touch-point index</a></li>- </ol>- </div>-- <!-- DIAGNOSIS -->- <h2 id="diagnosis"><span class="num">1</span>The diagnosis</h2>- <p><code>PipeType</code> and <code>PipeCategory</code> were each doing <strong>two jobs</strong>:</p>- <ul class="clean">- <li><strong>Job 1 — parse-time allowlist.</strong> "Is this <code>type</code> legal under <code>[pipe.X]</code>?" (<code>value in PipeType.value_list()</code>, in three validators.)</li>- <li><strong>Job 2 — executable taxonomy.</strong> <code>PipeType.category</code> → operator vs controller; the orchestration role.</li>- </ul>- <p>A signature only needs Job 1 — <code>type = "PipeSignature"</code> is a real thing you write. But the <em>only</em> way to clear the allowlist was to join the enum, which <strong>forced</strong> a <code>.category</code> and dragged it into Job 2 as a peer of the real kinds. <code>PipeCategory.PIPE_SIGNATURE</code> was the scar.</p>-- <div class="diagram">- <div class="mermaid">-flowchart TB- SIG["PipeSignature<br/><span>type = 'PipeSignature' is a real tag</span>"]:::sig- SIG -->|"only needs"| J1- SIG -.->|"forced side effect"| J2-- subgraph OVER["One enum, two jobs (overloaded)"]- direction TB- J1["JOB 1 — parse-time allowlist<br/>is this 'type' legal under [pipe.X]?"]:::ok- J2["JOB 2 — executable taxonomy<br/>operator vs controller · how it runs"]:::bad- end-- J2 --> SCAR["PipeCategory.PIPE_SIGNATURE<br/>peer of OPERATOR / CONTROLLER<br/>— the scar"]:::scar-- classDef sig fill:#1d2740,stroke:#7aa2ff,color:#dbe3ff;- classDef ok fill:#16271f,stroke:#5ad19a,color:#bdeed4;- classDef bad fill:#2a1d24,stroke:#ef6f6f,color:#f3c4c4;- classDef scar fill:#2a1d24,stroke:#ef6f6f,color:#f3c4c4,stroke-dasharray: 4 3;- </div>- <div class="cap">A signature only ever needed Job 1. Joining the enum to clear it dragged it into Job 2.</div>- </div>-- <p><strong>The fix separates the two jobs:</strong> the allowlist becomes "executable kinds ∪ the signature tag"; the enums shrink back to executable kinds only. As a bonus, <code>PipeType.category</code> is a <em>total</em> function again — its <code>match</code> is cleanly exhaustive over <code>{OPERATOR, CONTROLLER}</code>.</p>-- <!-- DECISION -->- <h2 id="decision"><span class="num">2</span>The principle</h2>- <p><code>PipeSignature</code> stays a <code>PipeAbstract</code> subclass (the dry-run engine dispatches over <code>PipeAbstract</code>, and a signature must be dry-runnable) — but it leaves the executable taxonomy.</p>-- <div class="principle">- <p class="label">The four rules</p>- <ul style="margin:6px 0 0; padding-left:18px">- <li><strong><code>PipeAbstract</code> = the substrate.</strong> A contract-bearing graph node: <code>inputs</code> / <code>output</code> / <code>description</code>, dry-runnable. A <code>PipeSignature</code> is that substrate with <em>nothing added</em>. It stays here.</li>- <li><strong><code>PipeType</code> / <code>PipeCategory</code> = the executable taxonomy.</strong> How a pipe runs and whether it orchestrates. A signature does not run → it is in <strong>neither</strong>.</li>- <li><strong>Identity by class, not by enum field.</strong> <code>is_signature</code> is the keystone — it stops reading <code>pipe_category</code> and becomes a class fact.</li>- <li><strong>The runtime <code>PipeSignature</code> is a dry-run shim</strong> of a blueprint-layer contract. Live-run always errors; dry-run mocks the declared output, and only under <code>--allow-signatures</code>.</li>- </ul>- </div>-- <!-- BEFORE/AFTER -->- <h2 id="before-after"><span class="num">3</span>Before / after taxonomy</h2>- <div class="diagram">- <div class="mermaid">-flowchart LR- subgraph B["BEFORE — signature is a peer kind"]- direction TB- BC["PipeCategory"]:::cat- BC --> BO["PIPE_OPERATOR"]:::ok- BC --> BCo["PIPE_CONTROLLER"]:::ok- BC --> BS["PIPE_SIGNATURE"]:::bad- BT["PipeType"]:::cat- BT --> BTo["PipeLLM, PipeExtract, …"]:::ok- BT --> BTc["PipeSequence, PipeBatch, …"]:::ok- BT --> BTs["PipeSignature"]:::bad- end- classDef cat fill:#1d2740,stroke:#7aa2ff,color:#dbe3ff,font-weight:bold;- classDef ok fill:#16271f,stroke:#3f7a5f,color:#bdeed4;- classDef bad fill:#2a1d24,stroke:#ef6f6f,color:#f3c4c4;- </div>- <div class="mermaid">-flowchart LR- subgraph A["AFTER — executable taxonomy (the only members)"]- direction TB- AC["PipeCategory"]:::cat- AC --> AO["PIPE_OPERATOR"]:::ok- AC --> ACo["PIPE_CONTROLLER"]:::ok- AT["PipeType"]:::cat- AT --> ATo["PipeLLM, PipeExtract, …"]:::ok- AT --> ATc["PipeSequence, PipeBatch, …"]:::ok- end- subgraph OUT["Outside · tolerated by the allowlist"]- direction TB- SIG["PipeSignature<br/>type = allowlist tag<br/>pipe_category = None<br/>is_signature = True (by class)"]:::sig- end- classDef cat fill:#1d2740,stroke:#7aa2ff,color:#dbe3ff,font-weight:bold;- classDef ok fill:#16271f,stroke:#3f7a5f,color:#bdeed4;- classDef sig fill:#1d2b2a,stroke:#5ad1b3,color:#bff0e4;- </div>- <div class="cap">The signature stops being a <em>member</em> of the taxonomy and becomes an outsider the allowlist tolerates.</div>- </div>-- <table>- <thead><tr><th>Aspect</th><th>Before</th><th>After</th></tr></thead>- <tbody>- <tr><td><code>PipeType</code> membership</td><td><span class="pill gone">PIPE_SIGNATURE</span> present</td><td>executable kinds only</td></tr>- <tr><td><code>PipeCategory</code> membership</td><td><span class="pill gone">PIPE_SIGNATURE</span> present</td><td><code>PIPE_OPERATOR</code>, <code>PIPE_CONTROLLER</code></td></tr>- <tr><td>signature <code>type</code></td><td><code>"PipeSignature"</code> (a <code>PipeType</code>)</td><td><code>"PipeSignature"</code> (allowlist tag only)</td></tr>- <tr><td>signature <code>pipe_category</code></td><td><code>"PipeSignature"</code></td><td><code>None</code></td></tr>- <tr><td><code>is_signature</code></td><td><code>PipeCategory(self.pipe_category) is …PIPE_SIGNATURE</code></td><td>class fact: base <code>False</code>, override <code>True</code></td></tr>- <tr><td><code>PipeType.category</code></td><td>partial (signature arm → non-exec category)</td><td>total over <code>{OPERATOR, CONTROLLER}</code></td></tr>- <tr><td><code>signature_for</code> guard</td><td>dedicated <code>reject_…</code> validator ×2</td><td>structural (not a valid <code>PipeType</code>)</td></tr>- </tbody>- </table>-- <!-- KEYSTONE -->- <h2 id="keystone"><span class="num">4</span>The keystone: identity by class</h2>- <p>Everything hinges on one property moving from "read an enum field" to "ask the class." Because every existing call site already calls <code>is_signature</code>, swapping its <em>implementation</em> propagates the new model everywhere for free.</p>-- <div class="diagram">- <div class="mermaid">-classDiagram- class PipeAbstract {- +is_signature() bool- }- class PipeBlueprint {- +is_signature() bool- }- class PipeSignature {- +is_signature() bool- +pipe_category None- }- class PipeSignatureBlueprint {- +is_signature() bool- +pipe_category None- }- PipeAbstract <|-- PipeSignature : override True- PipeAbstract <|-- PipeLLM : inherits False- PipeAbstract <|-- PipeSequence : inherits False- PipeBlueprint <|-- PipeSignatureBlueprint : override True- </div>- <div class="cap">The base says "not a signature"; only the signature classes say otherwise. No enum is consulted.</div>- </div>-- <div class="standalone-code">- <div class="filelabel">pipelex/core/pipes/pipe_abstract.py — the keystone diff</div>- <pre class="diff"><span class="ln ctx"> @property</span>-<span class="ln ctx"> def is_signature(self) -> bool:</span>-<span class="ln del">- return PipeCategory(self.pipe_category) is PipeCategory.PIPE_SIGNATURE</span>-<span class="ln add">+ # Identity by class, not by enum field: the base is never a signature;</span>-<span class="ln add">+ # `PipeSignature` overrides this to True. (`pipe_category` no longer encodes signature-ness.)</span>-<span class="ln add">+ return False</span></pre>- </div>- <div class="standalone-code">- <div class="filelabel">pipelex/pipe_signature/pipe_signature.py — the override</div>- <pre class="diff"><span class="ln add">+ @property</span>-<span class="ln add">+ @override</span>-<span class="ln add">+ def is_signature(self) -> bool:</span>-<span class="ln add">+ return True</span></pre>- </div>- <p>Same pair on <code>PipeBlueprint</code> (base <code>False</code>) and <code>PipeSignatureBlueprint</code> (override <code>True</code>).</p>-- <!-- DIFFS -->- <h2 id="diffs"><span class="num">5</span>Critical code changes</h2>-- <div class="tabs" id="difftabs">- <div class="tab-bar">- <button class="tab-btn active" data-tab="t-enums">Shrink the enums</button>- <button class="tab-btn" data-tab="t-allow">Honest allowlist</button>- <button class="tab-btn" data-tab="t-valid">Tolerant validators</button>- <button class="tab-btn" data-tab="t-factory">Factory branch</button>- <button class="tab-btn" data-tab="t-cat">pipe_category → None</button>- <button class="tab-btn" data-tab="t-guards">Delete guards</button>- </div>-- <div class="tab-panel active" id="t-enums">- <div class="filelabel">pipelex/core/pipes/pipe_blueprint.py</div>- <pre class="diff"><span class="ln ctx"> class PipeCategory(StrEnum):</span>-<span class="ln ctx"> PIPE_OPERATOR = "PipeOperator"</span>-<span class="ln ctx"> PIPE_CONTROLLER = "PipeController"</span>-<span class="ln del">- PIPE_SIGNATURE = "PipeSignature"</span>-<span class="ln ctx"> </span>-<span class="ln ctx"> @property</span>-<span class="ln ctx"> def is_controller(self) -> bool:</span>-<span class="ln ctx"> match self:</span>-<span class="ln ctx"> case PipeCategory.PIPE_CONTROLLER:</span>-<span class="ln ctx"> return True</span>-<span class="ln del">- case PipeCategory.PIPE_OPERATOR | PipeCategory.PIPE_SIGNATURE:</span>-<span class="ln add">+ case PipeCategory.PIPE_OPERATOR:</span>-<span class="ln ctx"> return False</span>-<span class="ln ctx"> </span>-<span class="ln ctx"> class PipeType(StrEnum):</span>-<span class="ln ctx"> ...</span>-<span class="ln ctx"> PIPE_SEQUENCE = "PipeSequence"</span>-<span class="ln del">- PIPE_SIGNATURE = "PipeSignature"</span>-<span class="ln ctx"> </span>-<span class="ln ctx"> @property</span>-<span class="ln ctx"> def category(self) -> PipeCategory:</span>-<span class="ln ctx"> match self:</span>-<span class="ln ctx"> ...</span>-<span class="ln ctx"> case PipeType.PIPE_SEQUENCE:</span>-<span class="ln ctx"> return PipeCategory.PIPE_CONTROLLER</span>-<span class="ln del">- case PipeType.PIPE_SIGNATURE:</span>-<span class="ln del">- return PipeCategory.PIPE_SIGNATURE</span></pre>- <p style="padding:0 4px 4px; color:var(--ink-dim); font-size:14px">Both <code>match</code> blocks stay exhaustive — now over the two executable values. <code>.category</code> is once again total.</p>- </div>-- <div class="tab-panel" id="t-allow">- <div class="filelabel">pipelex/core/pipes/pipe_blueprint.py — new single source of truth</div>- <pre class="diff"><span class="ln add">+# A signature is NOT an executable pipe kind: it is deliberately absent from `PipeType` and</span>-<span class="ln add">+# `PipeCategory`. This is the one extra `type` tag the parse-time allowlists admit beyond the</span>-<span class="ln add">+# executable kinds (a signature blueprint carries `type = "PipeSignature"`, `pipe_category = None`).</span>-<span class="ln add">+PIPE_SIGNATURE_TYPE_TAG = "PipeSignature"</span>-<span class="ln add">+</span>-<span class="ln add">+def valid_pipe_type_tags() -> list[str]:</span>-<span class="ln add">+ """Every legal `type` string under `[pipe.X]`: the executable `PipeType` kinds plus the</span>-<span class="ln add">+ signature tag. Single source of truth for the three layers (`PipeBlueprint`,</span>-<span class="ln add">+ `PipeAbstract`, `PipeSpec`) that gate it."""</span>-<span class="ln add">+ return [*PipeType.value_list(), PIPE_SIGNATURE_TYPE_TAG]</span></pre>- <p style="padding:0 4px 4px; color:var(--ink-dim); font-size:14px">A bare module-level string — <em>not</em> an import of <code>PipeSignatureBlueprint</code> into the base module (that would be a cycle). All three validators now read <code>valid_pipe_type_tags()</code> and list the tag in their error messages, so it shows up as a legal value.</p>- </div>-- <div class="tab-panel" id="t-valid">- <div class="filelabel">pipelex/core/pipes/pipe_blueprint.py & pipe_abstract.py — shared validators (same shape on both)</div>- <pre class="diff"><span class="ln ctx"> @field_validator("type", mode="after")</span>-<span class="ln ctx"> def validate_pipe_type(cls, value: Any) -> Any:</span>-<span class="ln del">- if value not in PipeType.value_list():</span>-<span class="ln del">- msg = f"Invalid pipe type '{value}'. Must be one of: {PipeType.value_list()}"</span>-<span class="ln add">+ allowed = valid_pipe_type_tags()</span>-<span class="ln add">+ if value not in allowed:</span>-<span class="ln add">+ msg = f"Invalid pipe type '{value}'. Must be one of: {allowed}"</span>-<span class="ln ctx"> raise ValueError(msg)</span>-<span class="ln ctx"> </span>-<span class="ln ctx"> @field_validator("pipe_category", mode="after")</span>-<span class="ln ctx"> def validate_pipe_category(cls, value: Any) -> Any:</span>-<span class="ln del">- if value not in PipeCategory.value_list():</span>-<span class="ln add">+ # `None` unambiguously means "signature"; every executable pipe pins a Literal.</span>-<span class="ln add">+ if value is not None and value not in PipeCategory.value_list():</span>-<span class="ln ctx"> raise ValueError(...)</span>-<span class="ln ctx"> </span>-<span class="ln ctx"> @model_validator(mode="after")</span>-<span class="ln ctx"> def validate_pipe_category_based_on_type(self) -> Self:</span>-<span class="ln add">+ if self.is_signature:</span>-<span class="ln add">+ # Outside the executable taxonomy: skip the type↔category consistency check</span>-<span class="ln add">+ # that would coerce `PipeType(self.type)` and raise.</span>-<span class="ln add">+ return self</span>-<span class="ln ctx"> pipe_type = PipeType(self.type) # was unconditional</span>-<span class="ln ctx"> ...</span></pre>- <p style="padding:0 4px 4px; color:var(--ink-dim); font-size:14px">Three small allowances: admit the tag, admit <code>None</code> category, early-return for signatures before the coercion. <code>is_signature</code> is class-based, so it's reliable at model-validation time.</p>- </div>-- <div class="tab-panel" id="t-factory">- <div class="filelabel">pipelex/core/pipes/pipe_factory.py</div>- <pre class="diff"><span class="ln del">- pipe_type = PipeType(blueprint.type)</span>-<span class="ln del">- pipe_category = pipe_type.category</span>-<span class="ln del">- factory_class_name = f"{pipe_type.value}Factory"</span>-<span class="ln add">+ # A signature is outside the executable taxonomy: its `type` is not a `PipeType`</span>-<span class="ln add">+ # member and it has no category, so resolve the factory by name and skip the coercion.</span>-<span class="ln add">+ pipe_category: PipeCategory | None</span>-<span class="ln add">+ if blueprint.is_signature:</span>-<span class="ln add">+ pipe_category = None</span>-<span class="ln add">+ factory_class_name = f"{blueprint.type}Factory" # "PipeSignatureFactory"</span>-<span class="ln add">+ else:</span>-<span class="ln add">+ pipe_type = PipeType(blueprint.type)</span>-<span class="ln add">+ pipe_category = pipe_type.category</span>-<span class="ln add">+ factory_class_name = f"{pipe_type.value}Factory"</span></pre>- <p style="padding:0 4px 4px; color:var(--ink-dim); font-size:14px">Resolve the factory by name from the string tag — no <code>PipeSignatureFactory</code> import → no cycle. The existing class-registry lookup downstream is unchanged.</p>- </div>-- <div class="tab-panel" id="t-cat">- <div class="filelabel">pipelex/pipe_signature/pipe_signature.py & pipe_signature_blueprint.py</div>- <pre class="diff"><span class="ln hdr"># runtime</span>-<span class="ln ctx"> type: Literal["PipeSignature"] = "PipeSignature"</span>-<span class="ln del">-pipe_category: Literal["PipeSignature"] = "PipeSignature"</span>-<span class="ln add">+pipe_category: None = None</span>-<span class="ln ctx"></span>-<span class="ln hdr"># blueprint</span>-<span class="ln ctx"> type: Literal["PipeSignature"] = "PipeSignature"</span>-<span class="ln del">-pipe_category: Literal["PipeSignature"] = "PipeSignature"</span>-<span class="ln add">+# Keep exclude=True — overriding the field type drops the base's Field(exclude=True)</span>-<span class="ln add">+# unless re-specified, so it never serializes into `.mthds`.</span>-<span class="ln add">+pipe_category: None = Field(default=None, exclude=True)</span></pre>- <p style="padding:0 4px 4px; color:var(--ink-dim); font-size:14px"><code>type</code> stays the discriminator. The union discriminates on <code>type</code>, not <code>pipe_category</code>, so <code>None</code> is invisible to parsing — and why the MTHDS schema didn't change.</p>- </div>-- <div class="tab-panel" id="t-guards">- <div class="filelabel">pipe_signature_blueprint.py & builder/pipe/pipe_signature_spec.py — both guards deleted</div>- <pre class="diff"><span class="ln del">- @field_validator("signature_for", mode="after")</span>-<span class="ln del">- @classmethod</span>-<span class="ln del">- def reject_signature_for_pipe_signature(cls, value: PipeType | None) -> PipeType | None:</span>-<span class="ln del">- if value is PipeType.PIPE_SIGNATURE: # ← this enum member no longer exists</span>-<span class="ln del">- msg = "A PipeSignature cannot have signature_for=PipeSignature."</span>-<span class="ln del">- raise ValueError(msg)</span>-<span class="ln del">- return value</span></pre>- <p style="padding:0 4px 4px; color:var(--ink-dim); font-size:14px">These referenced <code>PipeType.PIPE_SIGNATURE</code> — they would now <code>AttributeError</code> on every signature parse. Their intent is preserved <strong>structurally</strong>: <code>signature_for: PipeType | None</code> can no longer coerce <code>"PipeSignature"</code>, so Pydantic rejects it on its own with a clean error.</p>- </div>- </div>-- <!-- RUNTIME -->- <h2 id="runtime"><span class="num">6</span>Runtime behaviour</h2>- <p>The runtime <code>PipeSignature</code> is a dry-run shim — fully determined by the contract, no body.</p>- <div class="diagram">- <div class="mermaid">-flowchart TD- START["run_pipe() on a PipeSignature"]:::n --> MODE{run mode?}- MODE -->|LIVE| LIVE["validate_before_run sees is_signature + is_live<br/>→ raise PipeSignatureNotExecutableError"]:::bad- MODE -->|DRY| ALLOW{--allow-signatures?}- ALLOW -->|no| STRICT["strict validation refuses<br/>the pipeline depends on a signature"]:::bad- ALLOW -->|yes| MOCK["mint a mock output from the declared StuffSpec<br/>set_new_main_stuff → PipeOutput"]:::good- classDef n fill:#1d2740,stroke:#7aa2ff,color:#dbe3ff;- classDef bad fill:#2a1d24,stroke:#ef6f6f,color:#f3c4c4;- classDef good fill:#16271f,stroke:#5ad19a,color:#bdeed4;- </div>- <div class="cap">Live execution is always an error; dry-run mocks the declared output, gated on <code>--allow-signatures</code>.</div>- </div>-- <details>- <summary>The <code>output_renderer</code> short-circuit</summary>- <div class="body">- <p><code>_collect_possible_outputs</code> used to coerce <code>PipeType(the_pipe.type)</code> unconditionally and listed <code>PipeType.PIPE_SIGNATURE</code> in its match. Both break once the enum member is gone, so we early-return before the coercion:</p>- <pre class="diff"><span class="ln add">+ if the_pipe.is_signature:</span>-<span class="ln add">+ # Outside the `PipeType` taxonomy; contributes no concrete output options.</span>-<span class="ln add">+ # Short-circuit before the `PipeType(...)` coercion, which would raise.</span>-<span class="ln add">+ return []</span>-<span class="ln ctx"> pipe_type = PipeType(the_pipe.type)</span>-<span class="ln ctx"> ...</span>-<span class="ln ctx"> | PipeType.PIPE_BATCH</span>-<span class="ln ctx"> | PipeType.PIPE_PARALLEL</span>-<span class="ln del">- | PipeType.PIPE_SIGNATURE</span>-<span class="ln ctx"> ):</span>-<span class="ln ctx"> return []</span></pre>- </div>- </details>-- <details>- <summary>A welcome side effect — the headline feature reads truer</summary>- <div class="body">- <p><code>library_crate_factory._reconcile_pipe_collision</code> keys off <code>blueprint.is_signature</code>. After this change that means <strong>"a contract is fulfilled by an implementation,"</strong> not "swap pipe-type <code>PipeSignature</code> for pipe-type <code>PipeLLM</code>." <code>contract_match.contracts_match()</code> is untouched — the reconciliation logic didn't move, only the concept it expresses got honest.</p>- </div>- </details>-- <!-- SCOPE -->- <h2 id="scope"><span class="num">7</span>Scope: the spec layer stays put</h2>- <p>The taxonomy correction lands at the <strong>blueprint + runtime</strong> layers. <code>../mthds-plugins</code> authors against the <strong>spec</strong> layer (<code>PipeSignatureSpec</code>), so that surface is left stable: it keeps its <code>type</code> / <code>pipe_category = "PipeSignature"</code> string literals as display tags (not typed as <code>PipeCategory</code>, and <code>to_blueprint()</code> never propagates them). The only spec edit was widening <code>PipeSpec.validate_pipe_type</code> via the same <code>valid_pipe_type_tags()</code> and dropping the broken <code>signature_for</code> guard.</p>- <div class="note">- <p class="label">Why the spec can keep a tag the blueprint drops</p>- <p style="margin:0">Blueprints are the MTHDS language reference (what <code>.mthds</code> parses into); specs are an authoring format for agents, each with a <code>to_blueprint()</code>. Spec fields may legitimately differ from blueprint fields — so a cosmetic <code>pipe_category</code> string on the spec is fine.</p>- </div>-- <!-- KEPT -->- <h2 id="kept"><span class="num">8</span>What we kept & what we deferred</h2>- <table>- <thead><tr><th>Item</th><th>Status</th><th>Why</th></tr></thead>- <tbody>- <tr><td><code>PipeSignature(PipeAbstract)</code></td><td><span class="pill keep">kept</span></td><td>Dry-run engine dispatches over <code>PipeAbstract</code>; a signature must be dry-runnable.</td></tr>- <tr><td><code>signature_for: PipeType | None</code></td><td><span class="pill keep">kept</span></td><td>Consumed by mthds-plugins. An advisory implementor hint; rejection of <code>"PipeSignature"</code> is now structural.</td></tr>- <tr><td><code>PIPE_SIGNATURES</code> registry group</td><td><span class="pill keep">kept</span></td><td>Its own registry group reinforces "not an operator/controller."</td></tr>- <tr><td><code>reject_signature_for_pipe_signature</code> ×2</td><td><span class="pill gone">deleted</span></td><td>Referenced the removed enum member; intent now enforced by the type.</td></tr>- <tr><td>Class-based <code>is_controller</code> (Step 1b)</td><td><span class="pill keep">deferred</span></td><td>Optional cleanliness, not correctness — see below.</td></tr>- <tr><td>Retire <code>signature_for</code> entirely</td><td><span class="pill keep">future</span></td><td>Once mthds-plugins no longer depends on it; likely fold the hint into <code>description</code>.</td></tr>- <tr><td>Derive <code>pipe_category</code> instead of storing</td><td><span class="pill keep">future</span></td><td>It must equal <code>PipeType(type).category</code> for executables and is <code>None</code> for signatures — redundant, could be computed.</td></tr>- </tbody>- </table>-- <details>- <summary>Why <code>is_controller</code> was <em>not</em> converted to a class fact (deliberate)</summary>- <div class="body">- <p>We left <code>is_controller</code> reading the enum — it's <strong>cleanliness, not correctness</strong>, and converting it would do more harm:</p>- <ul class="clean">- <li>It already returns the right answer for a signature: <code>pipe_category</code> is <code>None</code>, <code>PipeCategory(None)</code> raises <code>ValueError</code>, and <code>is_controller_by_str</code>'s <code>except → False</code> handles it.</li>- <li>A class-based override would orphan <code>PipeCategory.is_controller</code> / <code>is_controller_by_str</code> into dead code (their only caller is <code>PipeAbstract.is_controller</code>).</li>- </ul>- <p>Revisit only if <code>is_controller</code> ever needs to stop touching <code>pipe_category</code> for another reason.</p>- </div>- </details>-- <!-- TOUCHPOINTS -->- <h2 id="touchpoints"><span class="num">9</span>Touch-point index</h2>- <table>- <thead><tr><th>File</th><th>What changed</th></tr></thead>- <tbody>- <tr><td><code>core/pipes/pipe_blueprint.py</code></td><td>shrink both enums; add <code>PIPE_SIGNATURE_TYPE_TAG</code> + <code>valid_pipe_type_tags()</code>; <code>is_signature</code> → <code>False</code>; widen validators; early-return</td></tr>- <tr><td><code>core/pipes/pipe_abstract.py</code></td><td><code>is_signature</code> → <code>False</code> base; same validator widening</td></tr>- <tr><td><code>pipe_signature/pipe_signature.py</code></td><td><code>pipe_category: None = None</code>; <code>is_signature</code> override → <code>True</code></td></tr>- <tr><td><code>pipe_signature/pipe_signature_blueprint.py</code></td><td><code>pipe_category: None = Field(default=None, exclude=True)</code>; override → <code>True</code>; delete guard</td></tr>- <tr><td><code>core/pipes/pipe_factory.py</code></td><td>branch on <code>is_signature</code> around the <code>PipeType</code> coercion</td></tr>- <tr><td><code>core/pipes/output/output_renderer.py</code></td><td>early <code>is_signature</code> return; drop <code>PIPE_SIGNATURE</code> match arm</td></tr>- <tr><td><code>builder/pipe/pipe_spec.py</code></td><td>widen <code>validate_pipe_type</code> via <code>valid_pipe_type_tags()</code></td></tr>- <tr><td><code>builder/pipe/pipe_signature_spec.py</code></td><td>delete guard; soften docstring; keep tags + <code>signature_for</code></td></tr>- <tr><td><code>cli/agent_cli/commands/pipe_cmd.py</code></td><td><code>--type</code> help now lists <code>pipe_type_to_spec_class</code> keys, not <code>PipeType.value_list()</code></td></tr>- <tr><td><code>derived/mthds_schema.json</code></td><td>regenerated — <strong>no diff</strong></td></tr>- <tr><td>tests · docs · CHANGELOG · TODOS</td><td>updated to the new model (enum eviction locked in; surface reframed)</td></tr>- </tbody>- </table>-- <details>- <summary>How the tests were updated</summary>- <div class="body">- <ul class="clean">- <li><code>test_pipe_blueprint_signature_enums.py</code> now locks in the eviction: members removed, tag not in either enum, coercion raises.</li>- <li><code>test_pipe_abstract_signature_surface.py</code> asserts the base descriptor returns <code>False</code> even for a stub whose <code>pipe_category == "PipeSignature"</code> — proving identity is class-based, not field-based.</li>- <li><code>test_pipe_signature.py</code> asserts <code>type == "PipeSignature"</code> and <code>pipe_category is None</code>.</li>- <li>Signature-rejection tests now pass the raw string <code>"PipeSignature"</code> and assert a plain Pydantic <code>ValidationError</code> (structural), not the old custom message.</li>- </ul>- </div>- </details>-- <footer>- <p>Status: landed on <code>refactor/Pipe-signature</code> · gates green (<span class="kbd">make tb</span> · <span class="kbd">agent-check</span> · <span class="kbd">agent-test</span>) · MTHDS schema regeneration produced no diff. Companion to the additive multi-file libraries work.</p>- </footer>--</div>--<script>- // Tabs- document.querySelectorAll('.tabs').forEach(function (tabs) {- var btns = tabs.querySelectorAll('.tab-btn');- var panels = tabs.querySelectorAll('.tab-panel');- btns.forEach(function (btn) {- btn.addEventListener('click', function () {- btns.forEach(function (b) { b.classList.remove('active'); });- panels.forEach(function (p) { p.classList.remove('active'); });- btn.classList.add('active');- var panel = tabs.querySelector('#' + btn.dataset.tab);- if (panel) panel.classList.add('active');- });- });- });-- // Mermaid- mermaid.initialize({- startOnLoad: true,- theme: 'dark',- themeVariables: {- background: '#171a23',- primaryColor: '#1d2740',- primaryTextColor: '#e6e8ee',- primaryBorderColor: '#2a2f3d',- lineColor: '#717789',- fontFamily: 'ui-monospace, SFMono-Regular, Menlo, monospace',- fontSize: '14px'- },- flowchart: { curve: 'basis', htmlLabels: true },- securityLevel: 'loose'- });-</script>-</body>-</html>diff --git a/wip/history/tests/add-tests-branch-overview.html b/wip/history/tests/add-tests-branch-overview.htmldeleted file mode 100644index e3df974b4..000000000--- a/wip/history/tests/add-tests-branch-overview.html+++ /dev/null@@ -1,325 +0,0 @@-<!DOCTYPE html>-<html lang="en">-<head>-<meta charset="utf-8">-<meta name="viewport" content="width=device-width, initial-scale=1">-<title>feature/Add-tests — branch overview</title>-<script src="https://cdn.jsdelivr.net/npm/mermaid@11/dist/mermaid.min.js"></script>-<style>- :root {- --bg: #0f1419; --panel: #1a2027; --panel2: #232b35; --ink: #e6edf3;- --muted: #9aa7b4; --line: #2d3742; --accent: #4cc2ff; --accent2: #7ee787;- --add-bg: #12261a; --add-ink: #7ee787; --del-bg: #2a1416; --del-ink: #ff9492;- --warn: #f0b429; --chip: #2b3744;- }- * { box-sizing: border-box; }- body {- margin: 0; background: var(--bg); color: var(--ink);- font: 16px/1.65 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;- }- .wrap { max-width: 980px; margin: 0 auto; padding: 32px 22px 96px; }- h1 { font-size: 30px; margin: 0 0 4px; letter-spacing: -0.5px; }- h2 { font-size: 22px; margin: 44px 0 14px; padding-bottom: 8px; border-bottom: 1px solid var(--line); }- h3 { font-size: 17px; margin: 26px 0 8px; color: var(--accent); }- p { margin: 10px 0; }- a { color: var(--accent); }- code { background: var(--panel2); padding: 1px 6px; border-radius: 5px; font-size: 13.5px;- font-family: "SF Mono", ui-monospace, Menlo, Consolas, monospace; }- .sub { color: var(--muted); font-size: 14px; margin-top: 0; }- .chips { display: flex; flex-wrap: wrap; gap: 8px; margin: 16px 0 4px; }- .chip { background: var(--chip); color: var(--ink); border: 1px solid var(--line);- padding: 3px 11px; border-radius: 999px; font-size: 12.5px; }- .chip b { color: var(--accent2); }- .tldr { background: linear-gradient(180deg, #16202b, #131a22); border: 1px solid #2c4257;- border-left: 4px solid var(--accent); border-radius: 10px; padding: 18px 20px; margin: 22px 0; }- .tldr h2 { border: 0; margin: 0 0 8px; font-size: 18px; color: var(--accent); }- .tldr ul { margin: 8px 0 0; padding-left: 20px; }- .tldr li { margin: 6px 0; }- .grid { display: grid; grid-template-columns: 1fr 1fr; gap: 12px; margin: 14px 0; }- @media (max-width: 720px) { .grid { grid-template-columns: 1fr; } }- .card { background: var(--panel); border: 1px solid var(--line); border-radius: 10px; padding: 14px 16px; }- .card .big { font-size: 26px; font-weight: 700; color: var(--accent2); }- .card .lbl { color: var(--muted); font-size: 13px; }- pre { background: #0b0f14; border: 1px solid var(--line); border-radius: 9px; padding: 14px 16px;- overflow-x: auto; font-size: 13px; line-height: 1.5;- font-family: "SF Mono", ui-monospace, Menlo, Consolas, monospace; }- pre.diff .add { color: var(--add-ink); background: var(--add-bg); display: block; }- pre.diff .del { color: var(--del-ink); background: var(--del-bg); display: block; }- pre.diff .ctx { color: var(--muted); display: block; }- pre.diff .hd { color: var(--accent); display: block; }- details { background: var(--panel); border: 1px solid var(--line); border-radius: 9px;- padding: 0 16px; margin: 12px 0; }- details[open] { padding-bottom: 14px; }- summary { cursor: pointer; padding: 13px 0; font-weight: 600; list-style: none; }- summary::-webkit-details-marker { display: none; }- summary::before { content: "▸ "; color: var(--accent); }- details[open] summary::before { content: "▾ "; }- .tabs { margin: 16px 0; }- .tabbar { display: flex; flex-wrap: wrap; gap: 6px; border-bottom: 1px solid var(--line); }- .tabbtn { background: transparent; border: 0; color: var(--muted); padding: 9px 13px;- cursor: pointer; font-size: 13.5px; border-bottom: 2px solid transparent; }- .tabbtn:hover { color: var(--ink); }- .tabbtn.active { color: var(--accent); border-bottom-color: var(--accent); font-weight: 600; }- .tabpane { display: none; padding-top: 14px; }- .tabpane.active { display: block; }- .mermaid { background: var(--panel); border: 1px solid var(--line); border-radius: 10px;- padding: 18px; margin: 16px 0; text-align: center; }- table { border-collapse: collapse; width: 100%; margin: 14px 0; font-size: 14px; }- th, td { border: 1px solid var(--line); padding: 8px 11px; text-align: left; vertical-align: top; }- th { background: var(--panel2); }- .ok { color: var(--accent2); } .no { color: var(--del-ink); } .mute { color: var(--muted); }- .note { border-left: 3px solid var(--warn); background: #211c10; padding: 10px 14px;- border-radius: 6px; margin: 14px 0; font-size: 14.5px; }- .foot { color: var(--muted); font-size: 13px; margin-top: 60px; border-top: 1px solid var(--line); padding-top: 16px; }- kbd { background: var(--panel2); border: 1px solid var(--line); border-bottom-width: 2px;- border-radius: 5px; padding: 1px 6px; font-size: 12px; font-family: ui-monospace, monospace; }-</style>-</head>-<body>-<div class="wrap">-- <h1>feature/Add-tests</h1>- <p class="sub">Offline test-coverage grind for pipelex — plus the source bugs writing those tests surfaced. PR #989 → <code>dev</code>.</p>-- <div class="chips">- <span class="chip">coverage <b>~72% → ~78%</b></span>- <span class="chip">source files changed <b>12</b></span>- <span class="chip">net new lines ~<b>10.3k</b> (almost all tests)</span>- <span class="chip">behavior diff <b>196 ins / 85 del</b></span>- </div>-- <div class="tldr">- <h2>TL;DR</h2>- <ul>- <li>This branch is a <b>unit-test grind</b> over the offline-testable layers of pipelex (CLI internals, Temporal entry points, inference plumbing, core runtime, tools/config). The diff is <b>almost entirely new tests</b>.</li>- <li>Writing those tests surfaced a <b>small, deliberate set of real source bugs</b>. Each was pinned by a test first, judged wrong in review, then fixed with the test flipped. <b>That's the only behavior change in the PR.</b></li>- <li>The headline fix is <b>storage <code>uri_format</code> validation</b>: a misconfigured format string used to silently render the same URI for every object (storage-wide overwrites). It now fails fast at boot with an operator-facing message — and the rule is enforced uniformly on <i>every</i> placeholder.</li>- <li>One load-bearing infra fix: pytest's default <code>norecursedirs</code> includes <code>build</code>, so an entire <code>tests/.../build/</code> tree (including a pre-existing regression test) <b>was never being collected</b>. Now fixed in <code>pyproject.toml</code>.</li>- <li><b>How to review:</b> read the source changes below (they're the only behavior diff), then sample the tests with the conventions section in mind. Remaining uncovered lines are deliberate (interface layer is owned by the <code>conformance</code> repo).</li>- </ul>- </div>-- <h2>1 · What's in the branch</h2>- <p>The grind covered five areas. Tests mirror the source tree under <code>tests/unit/pipelex/</code>. None of them touch a live provider, network, or cluster — inference, network, and Temporal SDK boundaries are mocked; real collaborators are used where cheap (real TOML in <code>tmp_path</code>, real PIL images, real pydantic SDK models, a real aiohttp server for the codec).</p>-- <div class="mermaid">-flowchart LR- G["feature/Add-tests<br/>coverage grind"] --> A["CLI internals<br/><i>doctor, run, build codegen,<br/>readiness, show/which, error handlers</i>"]- G --> B["Temporal entry points<br/><i>worker CLI, codec HTTP server,<br/>server connection</i>"]- G --> C["Inference plumbing<br/><i>structured-output modes, creds,<br/>model-deck, img-gen args,<br/>worker factories, gateway, mistral</i>"]- G --> D["Core runtime<br/><i>observer, output renderer,<br/>graph dispatch, bundle-spec,<br/>pipeline runner</i>"]- G --> E["Tools / config<br/><i>TOML config-sync, format enums,<br/>PIL conversion, storage validators</i>"]- </div>-- <div class="grid">- <div class="card"><div class="big">21→92%</div><div class="lbl"><code>doctor_cmd.py</code> — all diagnostic checks, fix mode</div></div>- <div class="card"><div class="big">32→94%</div><div class="lbl"><code>run/_run_core.py</code> — resolution, output/graph/CSV saving</div></div>- <div class="card"><div class="big">51→99%</div><div class="lbl"><code>cli/error_handlers.py</code> — every handler + panels</div></div>- <div class="card"><div class="big">20→97%</div><div class="lbl"><code>cli/readiness.py</code> — venv / dev-install / gate</div></div>- </div>-- <h2>2 · The source bugs (the behavior diff)</h2>- <p>Each tab is one fix. They are independent. Minimal diffs only — the surrounding code and tests are in the branch.</p>-- <div class="tabs" id="bugtabs">- <div class="tabbar">- <button class="tabbtn active" data-tab="t-storage">storage uri_format ★</button>- <button class="tabbtn" data-tab="t-mod">img moderation</button>- <button class="tabbtn" data-tab="t-concept">concept string</button>- <button class="tabbtn" data-tab="t-observer">observer</button>- <button class="tabbtn" data-tab="t-mistral">mistral order</button>- <button class="tabbtn" data-tab="t-render">output renderer</button>- <button class="tabbtn" data-tab="t-pytest">pytest collect ★</button>- </div>-- <div class="tabpane active" id="t-storage">- <h3>Storage <code>uri_format</code> validation — the biggest fix</h3>- <p><b>The bug.</b> GCP's <code>lazy_validate</code> accepted any format string containing the bare substring <code>hash</code> (e.g. <code>assets/myhash/file</code>) — no substitution slot, so every stored object renders the <i>same</i> URI: silent storage-wide overwrites. Local and in-memory configs had <b>no validation at all</b>. The check now parses the format string and requires a real, plain <code>{hash}</code> replacement field.</p>- <p>The per-provider checks moved onto a shared <code>StorageMethodConfig</code> base so the providers can't drift apart:</p>- <div class="mermaid">-classDiagram- ConfigModel <|-- StorageMethodConfig- StorageMethodConfig <|-- StorageLocalConfig- StorageMethodConfig <|-- StorageInMemoryConfig- StorageMethodConfig <|-- StorageBucketConfig- StorageBucketConfig <|-- StorageS3Config- StorageBucketConfig <|-- StorageGcpConfig- class StorageMethodConfig {- +uri_format str- +URI_FORMAT_SUPPORTED_FIELDS- +lazy_validate()- +_collect_uri_format_error_msgs()- }- class StorageBucketConfig {- +bucket_name str- +signed_urls_lifespan_seconds- }- </div>- <p>Validation happens at <b>boot</b>, when the provider is constructed. The flow for one placeholder:</p>- <div class="mermaid">-flowchart TD- S["uri_format"] --> P{"empty?"}- P -- yes --> EM["set a value for uri_format"]- P -- no --> F["string.Formatter().parse()"]- F -- ValueError --> IV["not a valid format string"]- F -- ok --> L["for each placeholder"]- L --> U{"base name in<br/>supported set?"}- U -- no --> UN["'{x}' is not supported"]- U -- yes --> PL{"plain {name}?<br/>no spec / conversion /<br/>attr / index"}- PL -- no --> NP["'{name}' must be plain"]- PL -- yes --> H{"a {hash} present<br/>somewhere?"}- H -- no --> NH["must contain a {hash} placeholder"]- H -- yes --> BK["backstop: render with test values"]- BK --> OK["valid ✓"]- </div>- <p><b>Core of the parser</b> (substring check → parse the format string):</p>-<pre class="diff"><span class="hd">pipelex/tools/storage/storage_config.py</span><span class="del">- elif "{hash}" not in self.uri_format: # GCP: "hash" substring only!</span>-<span class="del">- error_msgs.append("- uri_format must contain a {hash} placeholder")</span>-<span class="add">+ URI_FORMAT_SUPPORTED_FIELDS = frozenset({"primary_id", "secondary_id", "hash", "extension"})</span>-<span class="add">+ parsed = [(name, spec, conv) for _, name, spec, conv in</span>-<span class="add">+ Formatter().parse(self.uri_format) if name is not None]</span>-<span class="add">+ for name, spec, conv in parsed:</span>-<span class="add">+ base = name.split(".")[0].split("[")[0]</span>-<span class="add">+ if base not in URI_FORMAT_SUPPORTED_FIELDS: unsupported.add(name)</span>-<span class="add">+ elif name != base or spec or conv: nonplain.add(name) # see note</span>-<span class="add">+ if base == "hash": hash_found = True</span></pre>- <div class="note"><b>Round-2 finding (Codex adversarial pass).</b> The first version enforced "must be plain" only on <code>{hash}</code>. But a format spec on any other field is just as dangerous: <code>{primary_id:.0}</code> truncates to <code>""</code> (collisions again), and <code>{primary_id:1000000000}</code> would balloon the backstop's test-render into a ~1 GB allocation at boot. The <code>elif name != base or spec or conv</code> above generalizes the rule to <b>every</b> supported field, and rejects it at parse time <i>before</i> any <code>format()</code> call. Pinned by new S3 regression rows.</div>- <p>Also on this fix: bucket configs must set a positive <code>signed_urls_lifespan_seconds</code> (or <code>"disabled"</code>) so signed URLs can't be born dead; the factory now <code>lazy_validate</code>s local/in-memory like it already did S3/GCP; and <code>storage_path</code> dispatches on the configured method instead of blaming a missing local config.</p>- </div>-- <div class="tabpane" id="t-mod">- <h3>OpenAI image moderation mapping was inverted</h3>- <p><code>is_moderated=true</code> sent gpt-image's <i>less</i> restrictive <code>"low"</code>, and <code>false</code> sent <code>"auto"</code> — the exact opposite of "enable moderation".</p>-<pre class="diff"><span class="hd">pipelex/plugins/openai/openai_img_gen_factory.py</span><span class="ctx"> def moderation_for_openai_image(cls, is_moderated: bool | None):</span>-<span class="ctx"> if is_moderated is None:</span>-<span class="ctx"> return omit</span>-<span class="ctx"> if is_moderated:</span>-<span class="del">- return "low" # enabled → LESS restrictive (wrong)</span>-<span class="add">+ return "auto" # enabled → standard filtering</span>-<span class="del">- return "auto"</span>-<span class="add">+ return "low" # disabled → less restrictive</span></pre>-<pre class="diff"><span class="hd">pipelex/cogt/img_gen/img_gen_setting.py</span><span class="del">- is_moderated: bool = False</span>-<span class="add">+ is_moderated: bool | None = None # None → omit param → provider default ("auto")</span></pre>- <p>The default became <code>None</code> so the corrected mapping doesn't silently downgrade default runs: deck alias/handle resolution builds settings <i>without</i> the field, and the old <code>False</code> default would have started sending <code>"low"</code>. With <code>None</code>, workers omit the parameter and the provider's own default applies.</p>- </div>-- <div class="tabpane" id="t-concept">- <h3>String concept values in bundle specs were unconstructible</h3>- <p>Two problems: a <code>mode="before"</code> validator assumed dict input and crashed on the <code>ConceptSpec | str</code> union; and <code>to_blueprint()</code> shoved the string into <code>structure</code>, where the loader only accepts registered structure class names.</p>-<pre class="diff"><span class="hd">pipelex/core/concepts/concept_blueprint.py (and concept_spec.py — same guard)</span><span class="ctx"> @model_validator(mode="before")</span>-<span class="del">- def validate(cls, values: dict[str, Any] | str) -> ...:</span>-<span class="del">- if isinstance(values, dict) and values.get("refines") and values.get("structure"):</span>-<span class="add">+ def validate(cls, values: Any) -> Any:</span>-<span class="add">+ # before-validators get RAW input; a non-dict (str in a union) must fall through</span>-<span class="add">+ if not isinstance(values, dict):</span>-<span class="add">+ return values</span>-<span class="add">+ fields = cast("dict[str, Any]", values)</span>-<span class="add">+ if fields.get("refines") and fields.get("structure"):</span>-<span class="ctx"> raise ValueError(...)</span></pre>-<pre class="diff"><span class="hd">pipelex/builder/bundle_spec.py</span><span class="del">- concept[code] = ConceptBlueprint(description=code, structure=string_value)</span>-<span class="add">+ # a bare string is the concept's description — the blueprint layer accepts it as-is</span>-<span class="add">+ concept[code] = string_value</span></pre>- <p>This matches <code>ConceptFactory.make_from_blueprint</code>, which treats a bare string as <code>ConceptDeclarationType.STRING</code> (a description), not a class reference.</p>- </div>-- <div class="tabpane" id="t-observer">- <h3>LocalObserver let a payload key overwrite the event name</h3>- <p>A payload carrying its own <code>event_type</code> key silently overwrote the lifecycle event name in the JSONL record (which also names the output file). Merge order flipped so the event name wins.</p>-<pre class="diff"><span class="hd">pipelex/observer/local_observer.py</span><span class="ctx"> payload = {</span>-<span class="del">- "event_type": event_type,</span>-<span class="ctx"> **payload,</span>-<span class="add">+ "event_type": event_type, # last, so a payload key cannot clobber it</span>-<span class="ctx"> }</span></pre>- </div>-- <div class="tabpane" id="t-mistral">- <h3>Mistral sent the system message after the user message</h3>- <p><code>make_simple_messages</code> appended the system message <i>after</i> the user message — contradicting its own docstring and the OpenAI-typed sibling. System now comes first.</p>-<pre class="diff"><span class="hd">pipelex/plugins/mistral/mistral_factory.py</span><span class="add">+ if system_text := llm_job.llm_prompt.system_text:</span>-<span class="add">+ messages.append(SystemMessage(content=system_text))</span>-<span class="ctx"> ... build user_content ...</span>-<span class="ctx"> if user_content:</span>-<span class="ctx"> messages.append(UserMessage(content=user_content))</span>-<span class="del">- if system_text := llm_job.llm_prompt.system_text: # was appended LAST</span>-<span class="del">- messages.append(SystemMessage(content=system_text))</span></pre>- </div>-- <div class="tabpane" id="t-render">- <h3>Output-option numbering was nondeterministic</h3>- <p>A <code>PipeCondition</code>'s possible outputs were collected by iterating a dependency <i>set</i>, so the user-facing <code>output_option_N</code> numbering varied between runs with two or more mapped pipes. Now sorted by pipe code.</p>-<pre class="diff"><span class="hd">pipelex/core/pipes/output/output_renderer.py</span><span class="del">- for mapped_pipe_code in mapped_pipe_codes: # set → nondeterministic</span>-<span class="add">+ for mapped_pipe_code in sorted(mapped_pipe_codes): # stable output_option_N</span></pre>- <p>(Also in this group: <code>temporal/worker_cli.py</code> no longer logs <code>Starting worker for current project 'None'</code> when no project is given.)</p>- </div>-- <div class="tabpane" id="t-pytest">- <h3>An entire test tree was silently never collected</h3>- <p>pytest's default <code>norecursedirs</code> includes <code>build</code>. So everything under <code>tests/unit/pipelex/cli/commands/build/</code> (mirroring the source layout) was <b>never collected</b> in full runs — including a pre-existing cross-package refines regression test that had never run. This is load-bearing for the whole branch: without it, a chunk of the new tests wouldn't run in CI.</p>-<pre class="diff"><span class="hd">pyproject.toml — [tool.pytest]</span><span class="add">+ testpaths = ["tests"] # bare `pytest` never scans outside tests/</span>-<span class="add">+ norecursedirs = [ # = pytest defaults MINUS "build"</span>-<span class="add">+ "*.egg", ".*", "_darcs", "CVS", "dist", "node_modules", "venv", "{arch}",</span>-<span class="add">+ ]</span></pre>- <div class="note"><b>Verified empirically.</b> pytest 9.0.3 reads the <code>[tool.pytest]</code> table natively (promoted to canonical in pytest 9). Recursive discovery from the repo root now finds the <code>build/</code> tests (collected via recursion, not by naming the dir), and the <code>addopts</code> marker filter still applies. The default local config clears the new storage validation at boot.</div>- </div>- </div>-- <h2>3 · Test conventions (read before flagging)</h2>- <p>Several patterns that look unusual are deliberate:</p>- <table>- <tr><th>Convention</th><th>Why</th></tr>- <tr><td>One <code>TestClass</code> per module</td><td>House pytest standard — that's why e.g. doctor coverage is split across several <code>test_doctor_*.py</code> files.</td></tr>- <tr><td>pytest-mock (<code>MockerFixture</code>) only</td><td>Never <code>unittest.mock</code> imports. Strong value asserts; parametrization over copy-paste.</td></tr>- <tr><td>CLI <i>interface</i> deliberately untested</td><td>Arg parsing, <code>--help</code>, Typer wrappers, agent JSON shapes are owned by the sibling <code>conformance</code> repo. This branch tests the <code>do_*</code> / <code>_core</code> functions beneath them. Low coverage on <code>*_cmd.py</code> wrapper lines is by design.</td></tr>- <tr><td>Namespace vs source-module patching</td><td>Top-level imports patched at the consuming module; classes imported <i>inside</i> functions (deferred-import in worker factories) patched at their source module. Worker-factory tests use a fresh real <code>PluginSdkRegistry()</code> per test so the booted singleton is never mutated.</td></tr>- <tr><td>Real collaborators where cheap</td><td>Real TOML in <code>tmp_path</code>, real PIL images, real pydantic SDK models, a real aiohttp server for the codec. AsyncMocks only at inference / network / Temporal SDK boundaries.</td></tr>- <tr><td>Remaining uncovered lines are deliberate</td><td><code>TYPE_CHECKING</code> blocks, interface-layer wrappers (conformance-owned), lines pinned by the integration suite, live-call worker paths.</td></tr>- </table>-- <h2>4 · How it was reviewed</h2>- <p>The behavior diff went through CI (tests py3.10–3.14, typecheck, lint, uv-lock), Greptile (multiple rounds, all threads resolved), and a local two-model adversarial pass (Claude + Codex). Findings were verified against the code, deduplicated, and only fixed when a clear win:</p>- <ul>- <li><b>Fixed:</b> the generalized <code>uri_format</code> plain-placeholder rule (Codex High — see the storage tab note).</li>- <li><b>Deferred</b> (recorded in <code>wip/tests/</code>): a pre-existing truthiness-vs-presence edge case in the concept mutual-exclusion check, plus a handful of test-scaffold DRY cleanups. Contrived or pre-existing; not introduced here.</li>- <li><b>No-op:</b> the "bare string is now a description" change <i>is</i> the intended fix, confirmed against the blueprint contract.</li>- </ul>- <p class="mute">Cubic could not run this branch — the workspace hit its free monthly review-line limit (resumes July 1). External constraint, not a code issue.</p>-- <details>- <summary>Deferred items (not in this PR)</summary>- <ul>- <li><b>LocalObserver flat record namespace</b> — the merge-order fix guarantees the event name wins, but lifecycle metadata and payload keys still share one flat dict. Nesting the payload is a breaking JSONL-shape change — decide deliberately.</li>- <li><b>uri_format required on configs that never consume it</b> — the Temporal codec / in-memory paths build their own keys, yet the factory now validates a <code>uri_format</code> they ignore. Possible config-shape split.</li>- <li><b>Concept mutual-exclusion is truthiness-based</b> — <code>refines</code> + empty-but-present <code>structure</code> slips the guard. Pre-existing; switching to <code>is not None</code> changes behavior for legitimately-empty structures.</li>- <li><b><code>img_gen_param_defaults</code> partially dead config</b> — <code>is_moderated</code> / <code>safety_tolerance</code> there are never consumed; whether they should feed setting resolution is an open design call.</li>- <li><b>Test-quality cleanups</b> — a Pipelex-boot opt-out conftest for pure test dirs, hoisting duplicated scaffolds, minor in-file dedup.</li>- </ul>- <p class="mute">Full detail: <code>wip/tests/deferred-source-bugs-pinned-by-tests.md</code>.</p>- </details>-- <h2>5 · Verifying locally</h2>-<pre>make agent-check # lint + typecheck gate-make agent-test # full offline suite (build/ dirs now collect)</pre>-- <p class="foot">- Generated for the <code>feature/Add-tests</code> reviewers. Branch reviewer's guide lives in <code>TODOS.md</code>; user-facing wording in <code>CHANGELOG.md</code> under <code>[Unreleased]</code>.<br>- Tip: this page has <kbd>▸</kbd> foldable sections and tabbed diffs — click around.- </p>--</div>--<script>- // Tabs- document.querySelectorAll('.tabbtn').forEach(function (btn) {- btn.addEventListener('click', function () {- var pane = btn.getAttribute('data-tab');- document.querySelectorAll('#bugtabs .tabbtn').forEach(function (b) { b.classList.remove('active'); });- document.querySelectorAll('#bugtabs .tabpane').forEach(function (p) { p.classList.remove('active'); });- btn.classList.add('active');- document.getElementById(pane).classList.add('active');- });- });- // Mermaid- mermaid.initialize({ startOnLoad: true, theme: 'dark', themeVariables: { fontSize: '14px' } });-</script>-</body>-</html>diff --git a/wip/history/tests/deferred-source-bugs-pinned-by-tests.md b/wip/history/tests/deferred-source-bugs-pinned-by-tests.mddeleted file mode 100644index 115e76831..000000000--- a/wip/history/tests/deferred-source-bugs-pinned-by-tests.md+++ /dev/null@@ -1,29 +0,0 @@-# Source bugs pinned by the coverage-grind tests — RESOLVED (cleanups remaining)--Outcome of the post-grind code review (2026-06-12, branch `feature/Add-tests`). The coverage grind (TODOS.md Phases C/D/E) deliberately wrote tests that pin **current** behavior — including behavior the review then judged to be wrong at the source. The defects were initially deferred here; they have since been **verified and fixed on this same branch** (2026-06-12), with each pinning test flipped alongside its source fix and the fixes recorded in `CHANGELOG.md` under `[Unreleased]`.--Applied fixes (see CHANGELOG for user-facing wording):--1. **OpenAI image moderation mapping inverted** — `moderation_for_openai_image` now maps enabled → `"auto"`, disabled → `"low"`; the OPENAI_MODERATION parametrize rows flipped with it.-2. **GCP storage `lazy_validate` accepted uri_format without a real `{hash}` placeholder** — now checks the literal `"{hash}"` like S3, and the GCP error message gained the `- ` fix-line prefixes and docs URL; the bare-substring acceptance test became a rejection parametrize row. A post-fix review pass then extended this: the per-provider checks and message assembly were consolidated onto a shared `StorageMethodConfig` base (S3/GCP can no longer drift apart), and the local/in-memory configs — which had NO validation at all and were exposed to the same constant-URI overwrite — got `lazy_validate` too, wired into the storage provider factory like S3/GCP and pinned by their own test modules.-3. **Bundle-spec string-concept arm mapped the string into `structure`** — the string now passes through to the blueprint dict, where the loader treats it as the concept's description.-4. **`ConceptSpec.model_validate_spec` crashed on non-dict input** — gained the same `isinstance(values, dict)` guard as `ConceptBlueprint.validate_mutually_exclusive_fields`, so the `ConceptSpec | str` union falls through to `str` normally. All `model_construct` workaround sites in the bundle-spec tests were swept back to normal construction, and a test pins the normal-validation path for string concepts.-5. **Mistral `make_simple_messages` sent the system message AFTER the user message** — system now comes first, matching the docstring and the OpenAI-typed sibling; the ordering test flipped.-6. **`StorageProviderConfig.storage_path` ignored `method` and raised a misleading message** — the property now dispatches on method (mirroring `uri_format`), raising "storage_path is only available when method is local" for non-local methods; both pinning tests flipped.-7. **LocalObserver let a payload's own `event_type` overwrite the event name** — merge order flipped so the lifecycle event name wins; the pinning test flipped with it.--## Deferred design tradeoffs (from the post-fix review pass and the pre-landing /review army)--- **LocalObserver flat record namespace:** the merge-order fix guarantees the lifecycle `event_type` wins, but lifecycle metadata and arbitrary payload keys still share one flat dict in the JSONL record. Any future metadata field (timestamp, run id) re-opens the same collision class with whatever payload key shares its name. The deeper fix is nesting the payload under its own key (e.g. `{"event_type": ..., "payload": {...}}`), which is a breaking change to the JSONL record shape for downstream consumers — decide deliberately, don't slip it into an unrelated change.-- **`uri_format` is required on storage configs that never consume it:** the Temporal payload codec builds its own keys (`storage_prefix + hash_hex` in `storage_payload_codec.py`) and `InMemoryStorageProvider()` takes no config, yet the provider factory now `lazy_validate`s those sections, so a codec storage config must carry a ritual `uri_format = "{hash}"` and the error message tells operators to fix a value that has no effect on that path. Decide whether the uri_format checks belong only to the asset-storage path (e.g. split the codec/in-memory config shape from the asset one).-- **`validate_storage_provider_config` raises `StorageConfigError` from a pydantic `model_validator`:** pre-existing behavior, now pinned by a test. House style says validators raise `ValueError`/`TypeError` so pydantic aggregates them into the config tree's `ValidationError` with proper field locations; the unwrapped custom error aborts whole-config validation instead. Changing it changes the exception contract for any caller catching `StorageConfigError` on construction — decide deliberately.-- **`rendered_pretty` interpolates LLM-authored strings into Rich markup unescaped:** `PipelexBundleSpec.rendered_pretty` (and sibling spec renderers) pass authored strings through `Text.from_markup`, so unbalanced markup like `[/]` raises `rich.errors.MarkupError` and tags restyle output. Console-display-only severity, pre-existing pattern across several fields (the string-concept arm is just newly reachable). Sweep once with `rich.markup.escape` rather than patching one arm.-- **Concept mutual-exclusion check is truthiness-based, not presence-based:** both `ConceptBlueprint.validate_mutually_exclusive_fields` and `ConceptSpec.model_validate_spec` reject "refines and structure both set" with `fields.get("refines") and fields.get("structure")`. That's a *truthiness* test, so a contrived config with `refines = "Base"` and an empty-but-present `structure` (`""`/`{}`) slips past the guard, and `ConceptFactory.make_from_blueprint` then prioritizes `structure is not None` and silently takes the structure path. Pre-existing (this branch only added the non-dict `isinstance` guard, not the truthiness test) and only reachable with a self-inflicted empty structure. Switching to `... is not None` would change behavior for any legitimately-empty-but-present structure value, so decide deliberately rather than slipping it into the test branch. Surfaced by the pre-landing Codex pass.-- **`img_gen_param_defaults` is partially dead config:** `is_moderated` and `safety_tolerance` in `[cogt.img_gen_config.img_gen_param_defaults]` are never consumed — `pipe_img_gen` takes those values from the resolved `ImgGenSetting` with no fallback to the param defaults (unlike `aspect_ratio`/`background`/`is_raw`). The safety consequence for moderation was neutralized by defaulting `ImgGenSetting.is_moderated` to `None` (provider default applies); whether param defaults should feed setting resolution — and what that means for `safety_tolerance = 5` suddenly becoming effective on FAL — is an open design call.--## Test-quality cleanups (no source impact, batch when convenient) — STILL DEFERRED--- **Pipelex-boot opt-out for pure test dirs:** many of the new pure-logic modules (tools/misc, tools/storage, plugins factories, model_deck_check, backend_credentials) pay the autouse module-scoped `Pipelex.make()` from `tests/conftest.py` without using it. Precedent exists: `tests/unit/pipelex/system/pipelex_service/conftest.py` overrides `reset_pipelex_config_fixture` with a no-op. A one-file conftest per pure directory shaves seconds off every suite run.-- **Hoist duplicated scaffolds where a shared home already exists:** the three `graph/test_graph_rendering_*.py` files rebuild the same bundle-write + `dry_run_pipeline` AsyncMock scaffold (and `graph/conftest.py` already exports shared helpers); `make_llm_spec` is copied across the three `builder/test_bundle_spec_*.py` files (builder subpackages use `test_data.py` modules for exactly this); the storage trio re-implements the S3/GCP config builders in `test_storage_provider_config.py`.-- **Minor in-file dedup:** `test_storage_provider_config.py` carries the same 4-row method table twice; `test_local_observer.py`'s str/Path constructor twins could be one parametrized test; `test_model_deck_check.py` rebuilds the full ModelDeck inside every parametrized case (class-scoped fixture would do); `NESTED_TOML_CONTENT` is duplicated across two toml_sync files; `test_inputs_ops.py`'s fixture dict carries an unused `get_library_manager` key; the mistral messages tests use id()-keyed side_effect closures with asserts inside mocks where positional `side_effect=[...]` + post-hoc assertions would fail more readably. (Done already: `test_storage_s3_config.py`'s standalone braced-hash test that duplicated a parametrize row was removed during the fix pass.)-- **More scaffold hoists found by the pre-landing review army:** the three `test_output_renderer_*.py` modules re-declare the same patch-target constant and near-identical mock-pipe builders (a `tests/unit/pipelex/core/pipes/output/conftest.py` would hold both); the `core_mocks`-style fixture stub block is repeated across `test_inputs_core.py`/`test_output_core.py`/`test_runner_core.py` and `builder/operations/test_inputs_ops.py`; `make_backend`/`patch_hub_getters` are duplicated between `test_llm_worker_factory.py` and `test_img_gen_worker_factory.py` (they encode the booted-singleton-protection convention, so drift would silently weaken one factory's isolation); the PNG/PDF `FileType` constants re-declared across the mistral/gateway message tests belong in the existing `test_data.py`.diff --git a/wip/history/tests/missing-tests-menu.md b/wip/history/tests/missing-tests-menu.mddeleted file mode 100644index ea1a94933..000000000--- a/wip/history/tests/missing-tests-menu.md+++ /dev/null@@ -1,64 +0,0 @@-# Missing Tests Menu--Source: full non-inference suite run with `--cov=pipelex` (overall ~78% line coverage after sections A–E were ground down — was ~72%) plus a structural sweep of `pipelex/` vs `tests/`. This is a curated menu of the gaps that matter, not an exhaustive list. Caveat: plugin inference workers' live-call paths are exercised by `inference`-marked tests that don't run in this measurement, so their numbers understate real coverage — only their offline-testable logic (factories, arg builders, error mapping) is listed here.--## A. CLI internal logic — DONE (2026-06-12, see TODOS.md for the as-built notes)--Scope note: the spec'd CLI *interface* (arg parsing, `--help` surfaces, `init` behavior, `validate --all`, agent JSON output shapes) is owned by `../conformance` (paired with `docs/specs/`) — not duplicated here. This section is only the pipelex-internal logic those commands run, which conformance's subprocess-level tests never reach (and which doesn't count toward conformance coverage anyway). All testable offline with Typer's `CliRunner` or direct calls into the `_core` modules.--- [x] `cli/commands/doctor_cmd.py` (21% → **92%**, ~600 stmts) — all diagnostic checks, fix mode, health-report rendering covered; remainder ≈ `setup_doctor_runtime` (once-per-process log.configure side effects, deliberately untested).-- [x] `cli/commands/run/_run_core.py` (32% → **94%**) — happy path, bundle/inputs resolution, output saving (graphs/main_stuff/working-memory/CSV), wrapper error dispatch.-- [x] `cli/commands/show_cmd.py` (27% → **73%**) and `which_cmd.py` (27% → **66%**) — the `do_*` report logic is fully covered; the remainder is the Typer wrappers that boot Pipelex (interface layer, conformance-owned).-- [x] `cli/commands/build/*` internals (`_output_core` 41→**74%**, `_runner_core` 21→**76%**, `_inputs_core` 21→**72%**, `structures_cmd` 41→**69%**) — codegen cores covered; remainder is the Typer command wrappers. NB: these tests live in `tests/unit/pipelex/cli/commands/build/`, which pytest's default `norecursedirs` silently skipped until the pyproject override (defaults minus `build`) — the pre-existing cross-package refines regression test there had never been running.-- [x] `cli/readiness.py` (20% → **97%**) — venv detection, dev-install detection, and the readiness gate.-- [x] `cli/error_handlers.py` (51% → **99%**, ~290 stmts) — every handler incl. gateway/telemetry/signature ones, panel rendering, validation-error detail sections.--## B. Temporal distributed execution — DONE (2026-06-12, see TODOS.md Phase B for the as-built notes)--Never shipped to prod yet, but these are the deploy-critical entry points; bugs here only surface inside a live cluster.--- [x] `temporal/worker_cli.py` (0% → **100%**) — project resolution from pyproject, fast-fail task-queue validation before library load, worker-base library loading, forced `is_enabled`, full Typer arg wiring into the worker loop.-- [x] `temporal/codec/codec_server.py` (0% → **96%**) — encode/decode round-trip over real HTTP, CORS preflight/origin gating, content-type and malformed-payload guards, storage-failure → HTTP status mapping; remainder = the `TYPE_CHECKING` import block. NB: `codec_server_cli.py` stays at 0% — it's the thin arg-parse + `run_app` wrapper around `build_codec_server`, same interface-layer category as the Typer wrappers left out in section A.-- [x] `temporal/temporal_connect.py` (24% → **98%**) — API-key resolution per secret method, TLS/RPC-metadata wiring, payload-codec converter selection, named server-config selection, SDK error wrapping; remainder = the `TYPE_CHECKING` import block.--## C. Inference plumbing (offline-testable parts) — DONE (2026-06-12, see TODOS.md Phase C for the as-built notes)--Not the live calls — the factories, arg-builders, and config logic around them, which can break independently of any provider.--- [x] `cogt/img_gen/img_gen_args_factory.py` (58% → **100%**, ~320 stmts) — all provider taxonomies (Flux, Flux-1.1 Ultra, Qwen, SDXL/Lightning, GPT-image, BFL Flux 2) across aspect-ratio, prompt, inference, safety, output-format, input-images, and input-fidelity arg-building.-- [x] `cogt/llm/llm_worker_factory.py` (28% → **100%**) and `img_gen/img_gen_worker_factory.py` (40% → **99%**) — full SDK routing matrices, client-instance caching, missing-dependency errors; remainder = a `TYPE_CHECKING` import.-- [x] `cogt/llm/structured_output.py` (55% → **99%**) — every `StructureMethod` → instructor mode pinned with a completeness guard; remainder = a `TYPE_CHECKING` import.-- [x] `cogt/model_backends/backend_credentials.py` (30% → **100%**) — env vs generic-provider error messages, missing/placeholder var aggregation, report models.-- [x] `cogt/models/model_deck_check.py` (42% → **100%**) — every check function x every reference kind (found/not-found), fuzzy suggestions, wrong-sigil hints, setting short-circuits.-- [x] `plugins/gateway/gateway_completions_factory.py` (21% → **96%**) and `mistral_factory.py` (27% → **99%**) — client construction, message shaping, and the extract-output parsers (Azure/Mistral/Deepseek/Linkup; OCR responses, image cleanup, document prep/upload). Remainders = `TYPE_CHECKING` blocks, a defensive `except` reachable only if `model_dump` itself throws, and mistral's `make_mistral_client` retry wiring (already pinned by `test_transport_retry_wiring.py`).--## D. Core runtime odds and ends — DONE (2026-06-12, see TODOS.md Phase D for the as-built notes)--- [x] `observer/local_observer.py` (0% → **100%**) — constructor dir resolution, every observe method, JSONL append semantics, event-type collision behavior (originally pinned as payload-wins; the source was then fixed on this same branch so the lifecycle event name wins, and the pinning test flipped with it).-- [x] `core/pipes/output/output_renderer.py` (33% → **98%**) — Anything-output resolution through PipeCondition/PipeSequence (incl. recursion), operator arms, every Anything render format; remainder = the `TYPE_CHECKING` import block.-- [x] `graph/graph_rendering.py` (40% → **100%**) — `_dry_run_bundle` library-dirs matrix, format dispatch, the sanitized-rename branch, `generate_view_for_bundle` direction precedence.-- [x] `builder/bundle_spec.py` (34% → **100%**) and `builder/operations/inputs_ops.py` (20% → **98%**) — spec validation, `to_blueprint()` ordering/error wrapping, pretty rendering; every `build_inputs_for_pipe` branch. Remainder = a `TYPE_CHECKING` import. NB: the test phase surfaced a real source bug here (`ConceptSpec.model_validate_spec` crashed on plain-string concept values); it was fixed later on this same branch with the pinning tests flipped — see `deferred-source-bugs-pinned-by-tests.md` and the CHANGELOG `[Unreleased]` entry.-- [x] `pipeline/runner.py` (65% → **93%**) — `extra` rejection, both `except PipelexError` arms, `except ValidationError`, `start()`, and the protocol surfaces `validate()` (incl. the finally restore matrix), `models()`, `version()`. Remainder = the `TYPE_CHECKING` block plus the happy-path/`PipeRouterError`/tracer-close lines already pinned by the integration suite.--## E. Tools / config — DONE (2026-06-12, see TODOS.md Phase E for the as-built notes)--- [x] `tools/misc/toml_sync.py` (0% → **98%**) and `tools/misc/document_utils.py` (0% → **97%**) — toml_sync's full read/set/sync surface incl. the destroy-config guards (never creates keys, comment/structure preservation, dry-run, no-op write guard, idempotency); document format enum matrix. Remainders = a `TYPE_CHECKING` import, an unreachable defensive `continue`, and the type-checker-appeasement trailing `raise`.-- [x] `tools/storage/storage_config.py` (45% → **100%**) — per-provider `lazy_validate` fault matrices, provider-config validator, `uri_format`/`storage_path` properties. (The originally-pinned S3-vs-GCP `{hash}` asymmetry was then eliminated on this same branch: a shared config base validates every provider, local/in-memory included, each pinned by its own test module.)-- [x] `tools/misc/image_utils.py` (52% → **98%**) — format enum matrix, both unsupported-MIME message branches, PIL conversion round-trips; remainder = the unreachable trailing `raise`.--## Deliberately not on the menu--- CLI interface surfaces owned by `../conformance`: `init` (dedicated modules there), `validate` command wrappers, `--help` smoke tests, agent JSON output shapes. Rule of thumb: cross-repo contract behavior → conformance test paired with a spec section; pipelex-internal logic → here.-- `cli/dev_cli/*` (mostly 0%) — internal dev tooling, not shipped; low value per test.-- `plugins/*_list.py`, `huggingface_factory` (0%) — model-listing scripts hitting live APIs; belongs to inference-marked tests if anything.-- `pipelex.py` / `hub.py` (84%/83%) — already decent; remaining lines are teardown/edge accessors.-- Templating, input normalizer, URI resolver, reporting — explorer flagged them as thin on dedicated tests, but coverage shows they're well exercised (88–100%) via integration; dedicated unit tests would be redundant.--## Suggested grind order--1. ~~**A (CLI)**~~ — DONE 2026-06-12 (branch `feature/Add-tests`; plan + as-built notes in TODOS.md).-2. ~~**B (Temporal)**~~ — DONE 2026-06-12 (user-prioritized ahead of C; same branch, TODOS.md Phase B).-3. ~~**C (inference plumbing)**~~ — DONE 2026-06-12 (same branch, TODOS.md Phase C).-4. ~~**D then E**~~ — DONE 2026-06-12 (same branch, TODOS.md Phases D and E).--All menu sections are complete. Remaining gaps are deliberate (see "Deliberately not on the menu") plus the per-module remainders noted inline above (TYPE_CHECKING blocks, interface-layer wrappers, integration-pinned lines).diff --git a/wip/init-yes-mode-handoff.md b/wip/init-yes-mode-handoff.mddeleted file mode 100644index 747ca4a79..000000000--- a/wip/init-yes-mode-handoff.md+++ /dev/null@@ -1,89 +0,0 @@-# Handoff: non-interactive `pipelex init --yes` mode--Untracked scratch note (in `wip/`, not gitignored — delete when done). Written from a cocode session that hit the downstream symptom. pipelex version in play: 0.31.0.--## Why we want this--Downstream repos need CI to initialize Pipelex **without** interactive prompts and **without** vendoring a full `.pipelex/` config tree into their repo. Today that's impossible: `pipelex init` is interactive end-to-end.--The concrete trigger: cocode CI started failing with `Pipelex must be initialized before running the tests` after cocode de-vendored its `.pipelex/` config (cocode commit `1dd8831`). cocode's `tests/conftest.py` gates the session on Pipelex's `check_is_initialized()`. On CI there's no global `~/.pipelex/` and (after de-vendoring) no project `.pipelex/`, so the gate aborts the run. Locally it passed only because the dev had a global config.--Important: `--disable-inference` does **not** help here. It skips the *gateway terms* check and swaps in a mock content generator — it never bypassed *initialization* (the presence of config files). Pipelex's own CI sidesteps the problem only because pipelex **vendors its full `.pipelex/` tree in-repo**. We don't want every downstream repo to carry (and keep in sync) a vendored config just to run CI.--**Stopgap already applied in cocode** (to be reverted once this lands): re-vendored a minimal `.pipelex/` CI config (pipelex.toml, plxt.toml, `inference/` backends+deck+routing), force-added past the gitignore. See the "Downstream follow-up" section.--## Goal--Add a non-interactive `pipelex init --yes` (`-y`) that materializes a complete, valid config with zero prompts and exit 0 — such that `check_is_initialized()` returns True — suitable for CI and automation.--## What already exists (good news — it's mostly plumbing)--- `init` CLI command: `pipelex/cli/_cli.py:158` (`init_command`) → calls `init_cmd(focus=focus, local=local)`. No `--yes` exposed yet.-- `init_cmd`: `pipelex/cli/commands/init/command.py:466`. Already takes `skip_confirmation: bool` — but that only gates the **final** "Continue with initialization?" confirm (`command.py:545-563`). It does **not** gate the backend/routing/credentials prompts.-- Pattern to mirror: `update_command` already has `--yes/-y` (`_cli.py:209`) threaded into `update_cmd(yes=...)`. Copy that shape.-- `init_cmd(focus=InitFocus.CONFIG, skip_confirmation=True)` is already called non-interactively from doctor (`doctor_cmd.py:1028,1048`).--**Key insight:** every interactive prompt site already carries a sensible `default=`. So `--yes` is well-bounded: short-circuit each prompt to its existing default instead of asking. The defaults already encode the "yes" answer.--## Prompt sites to short-circuit under `--yes`--Each must return its `default` (or computed default) without prompting:--- `command.py:249` — `Confirm.ask("Continue with initialization?", default=True)`. Verify it isn't reached when `assume_yes` (today it's the `skip_confirmation` path).-- `ui/backends_ui.py:188` — `Prompt.ask("Enter your choices", default=default_str)`. First-time default is `"1"` = `pipelex_gateway` (`backends_ui.py:177-184`). Under `--yes`: accept `default_str` (and the computed `default_indices`) directly.-- `ui/routing_ui.py:63` and `:150` — `Prompt.ask(..., default=...)`. Accept defaults.-- `routing.py:74` — `Confirm.ask("Would you like to create it?", default=True)`. Accept True.-- `credentials.py:140` — `Prompt.ask(var, default="", password=True)`. Under `--yes`: **skip the prompt loop entirely** (default `""` already means "skip, set later" — the UI even says so at `credentials.py:128`). Creds come from env in CI.-- `ide_extension.py:109` — `Confirm.ask(... install IDE extension ...)`. Under `--yes`: default to **No** (don't install in CI). Confirm its current default.-- `ui/gateway_ui.py:107` — `Confirm.ask(...)` gateway terms. **Legal decision — do not silently accept ToS.** See below.--## Recommended implementation--1. Add `--yes/-y` typer option on `init_command` (`_cli.py:158`), mirroring `update_command:209`. Pass it as `assume_yes` into `init_cmd`.-2. Thread `assume_yes` into `init_cmd` and down to the UI helpers (`backends_ui`, `routing_ui`, `routing`, `credentials`, `ide_extension`, `gateway_ui`). Centralize with a tiny helper, e.g. `ask_or_default(assume_yes, ask_fn, default)`, so each site reads cleanly.-3. Reconcile with `skip_confirmation`: cleanest is to introduce `assume_yes` and have it **subsume** `skip_confirmation` (assume_yes ⇒ skip the final confirm too). Update the doctor callsites (`doctor_cmd.py:1028,1048`) accordingly.-4. Credentials under `--yes`: skip the loop; don't write blank entries to `.env`.-5. **High-value optional:** auto-enable `assume_yes` when non-interactive — `not sys.stdin.isatty()` or a CI env var (`GITHUB_ACTIONS`/`CI`, already detected via `runtime_manager`/`shared_pytest_plugins.py`). Then CI "just works" even without the flag, and a closed stdin never hangs.--## Gateway terms — handle deliberately--Do **not** auto-accept the Pipelex Gateway ToS just because `--yes` was passed. Options, pick one:-- In CI mode (`runtime_manager.is_ci_testing` / `IntegrationMode.CI`) terms are already skipped for the *test run*; for `init --yes` itself, leave the gateway **enabled but unaccepted** and let the CI test path (which uses mock inference) proceed — it never calls the gateway.-- Or require a separate explicit opt-in for acceptance: `--accept-terms` flag or `PIPELEX_ACCEPT_TERMS=1` env. `--yes` alone should not constitute legal acceptance.--## Resulting config must satisfy `check_is_initialized()`--`pipelex/system/configuration/config_check.py:12-18` requires all of: `pipelex.toml`, `plxt.toml`, the backends file (`config_manager.backends_file_path`), and the routing-profiles file (`config_manager.routing_profiles_file_path`) — resolvable in project or global dir. `--yes` must produce all of them. Default backend = `pipelex_gateway`; default routing profile; telemetry default (decide: accept default vs. disable in non-interactive — check the telemetry default).--## Acceptance criteria--In an environment with **no** global `~/.pipelex/` and **no** project `.pipelex/`:-- `pipelex init --yes --local` → exit 0, **zero prompts**, creates `.pipelex/` such that `python -c "from pipelex.system.configuration.config_check import check_is_initialized; assert check_is_initialized()"` passes.-- `pipelex init --yes` (global) → same for `~/.pipelex/`.-- Runs with stdin closed / non-tty must not hang.-- Gateway ToS is **not** silently accepted (assert).-- The existing interactive `pipelex init` flow is unchanged.--## Tests / conformance--- Unit-test `init_cmd(assume_yes=True)` in a tmp config dir (monkeypatch the config dirs). Patch `Confirm.ask`/`Prompt.ask` to **raise** if called, proving no prompts fire; assert the files exist and `check_is_initialized()` is True.-- The init CLI is a cross-repo surface — there's likely a spec in the workspace `docs/specs/` with a paired test in the `conformance/` repo (see workspace CLAUDE.md). Add the `--yes` flag to that spec section, add a conformance test, and run `conformance/scripts/check-spec-links.py`.--## Downstream follow-up (don't lose this)--Once `init --yes` ships and pipelex is re-released + re-pinned:-- **cocode:** revert the stopgap — remove the re-vendored `.pipelex/` config (currently staged on branch `release/v0.7.0`), and instead either add a CI step `pipelex init --yes --local` before `make gha-tests` in `.github/workflows/tests-check.yml`, **or** rely on auto-init-when-non-tty/CI if you implement option 5 (then cocode's conftest can call an init helper rather than just gating).-- Audit other repos that vendor `.pipelex/` purely for CI (e.g. `methods/`, `test-bed/`, `pipelex-api-deploy/`) — they can adopt the same `init --yes` pattern.--## Files likely to touch--- `pipelex/cli/_cli.py` — `init_command`: add `--yes/-y`, pass through.-- `pipelex/cli/commands/init/command.py` — `init_cmd`: accept `assume_yes`, thread it, reconcile `skip_confirmation`.-- `pipelex/cli/commands/init/ui/backends_ui.py` — short-circuit to default selection.-- `pipelex/cli/commands/init/ui/routing_ui.py` — short-circuit to defaults.-- `pipelex/cli/commands/init/routing.py` — `Confirm` default.-- `pipelex/cli/commands/init/credentials.py` — skip the loop under `assume_yes`.-- `pipelex/cli/commands/init/ide_extension.py` — default No under `assume_yes`.-- `pipelex/cli/commands/init/ui/gateway_ui.py` — terms handling (deliberate).-- `pipelex/cli/commands/doctor_cmd.py` — update callsites if `skip_confirmation` is folded into `assume_yes`.-- Tests + `docs/specs/` + `conformance/`.diff --git a/wip/observer-and-telemetry/identifier-visibility-and-redaction.md b/wip/observer-and-telemetry/identifier-visibility-and-redaction.mddeleted file mode 100644index cf9fca352..000000000--- a/wip/observer-and-telemetry/identifier-visibility-and-redaction.md+++ /dev/null@@ -1,77 +0,0 @@-# Identifier visibility vs. privacy — reflection kickoff--Status: open question, no code touched. This note exists to start a fresh-session reflection. Keep the options open.--## What triggered this--A pipeline run through the API (Temporal path) returned a cost report whose `tokens_usages[]` records carried, per LLM call:--```-job_metadata.pipe_code = "analyze_cv_job_match"-job_metadata.otel_context.trace_name = "analyze_cv_job_match_aced9b60"-job_metadata.otel_context.trace_name_redacted = "aced9b60"-job_metadata.trace_context.tracer_key = "<run_id>/pipe-router"-job_metadata.trace_context.parent_node_id = "<run_id>:<run_id>/pipe-router:node_0"-```--The original question was narrow ("what is `/pipe-router`?"), but it opened a broader one: **which structural identifiers (pipe codes, run ids, user/session ids) are exposed on which surfaces, and is there a single policy that lets us hide them when we need to?**--## ID semantics (so the cold reader isn't guessing)--- **Temporal workflow id is a hierarchical path.** `WfPipeRun` (orchestrator) owns `{run_id}`; it spawns child `WfPipeRouter` with id `{run_id}/pipe-router` (hardcoded suffix, `wf_pipe_run.py:62`). When a controller dispatches a sub-pipe through `TemporalPipeRouter` *inside* a workflow, each sub-pipe becomes its own child workflow with id `{parent}/{pipe_code}-{uuid8}` (`temporal_pipe_router.py:69`).- - `{pipe_code}` = the literal sub-pipe code. `-{uuid8}` = `str(workflow.uuid4())[:8]`, a replay-deterministic collision-breaker for same-code siblings (e.g. batch). Not ordered, not meaningful.- - **Inconsistency:** the top pipe's workflow uses a generic `/pipe-router`; nested pipes use their pipe code. The top pipe also has a code that is currently thrown away here.-- **`node_N` is unrelated to pipe codes.** It's `TraceContext.node_sequence` — a monotonic counter minted per workflow tracer (`graph_tracer.py:198` → `{graph_id}:{workflow_id}:node_{seq}`; direct mode `{graph_id}:node_{seq}`, `trace_context.py:64`). The counter **resets per workflow**, so the `{workflow_id}` component is load-bearing: it's the namespace that keeps `node_0` of one workflow distinct from `node_0` of another.--## Core finding: the cost report is an ungoverned exposure surface--`tokens_usages[]` embeds the **entire `JobMetadata`** (`llm_report.py:65-67`, dumped wholesale at `llm_report.py:44`). Nothing redacts it before it leaves the API. The pipe code alone appears up to three ways in one record: `pipe_code`, `otel_context.trace_name`, and (if the workflow id were renamed) `tracer_key`/`parent_node_id`. `user_id` / `session_id` ride along too (`job_metadata.py:41-74`).--The irony: a **redacted twin already exists** — `OtelFactory.make_trace_names` (`otel_factory.py:131`) computes both `{pipe_code}_{hash}` and the pipe-code-free `{hash}`, and both are shipped side by side. The redaction concept the design intended is present and then ignored on this surface.--Consequence for the `/pipe-router` → `/{pipe_code}` rename idea: it improves OSS observability/consistency but **adds a 4th pipe-code leak to the one surface with no redaction control** — wrong direction for hosted privacy. The "uninformative" `/pipe-router` is actually the *safe* default; the informative naming is what should be opt-in.--## Existing knobs — and what each does NOT cover--Several independent subsystems, each scoped to a different sink:--| Subsystem | Config | Governs | Default | Covers cost-report output? |-|---|---|---|---|---|-| Telemetry export redaction | `[telemetry.custom_posthog.tracing.capture]` → `TelemetryRedactionConfig` (`telemetry_config.py:199`); `is_capture_pipe_codes_enabled()` (`telemetry_manager_abstract.py:44`) | pipe codes / content in PostHog/OTEL/Langfuse spans sent to external sinks | redact by default | No |-| Temporal search attributes | `[temporal.search_attributes].enabled` + allowlist (`config_temporal.py:41`) | what's queryable in the Temporal UI | subset | No |-| Temporal static summary/details | none (`observability.py:124-160`) | pipe code + description + user/session + inputs on every workflow | always on | No |-| Graph data inclusion | `graph_config.data_inclusion` (`graph_config.py:6`) | stuff payload content (json/text/html), stack traces, registry in graph events | per-config | governs payloads, not the ids/codes |-| Error disclosure | `disclosure_mode` STRICT/VERBOSE (`base_exceptions.py`) | error-message redaction in `ErrorReport` | — | separate |--**Two surfaces fall through all of them:**--1. The Temporal **workflow-id naming** (`/pipe-router` vs `/{pipe_code}-{uuid}`).-2. The **`JobMetadata` serialized into `tokens_usages`** (and the runner fallback path that derives `workflow_id`/`node_id` from the same context, `reporting_manager.py:263-264`).--## The tension to reconcile--- OSS / self-host users want pipe codes visible — in the report, logs, Temporal UI, for debugging and querying.-- Hosted runner needs the ability to hide structural detail for privacy.--Both are legitimate; the answer is config-driven, not one-size.--## Open decision (do not foreclose)--The system already has the right primitive — a "may pipe codes be shown" boolean — but it only governs telemetry export. A reflection should weigh making structural-identifier visibility a first-class policy that *also* governs (1) workflow-id naming and (2) `JobMetadata`/cost-report serialization, defaulting visible for OSS and redacted for the hosted runner.--Candidate homes for that policy — left open:--- **(a)** Extend/reuse the telemetry `capture.pipe_codes` flag so one switch covers telemetry *and* run output.-- **(b)** New knob alongside `data_inclusion` on `graph_config` / `tracing_config` (the cost report rides the same trace substrate).-- **(c)** Derive from the existing `data_inclusion` family.--Each has different blast radius and different "single source of truth" implications. Also unresolved: whether redaction should happen at serialization time (strip fields from `JobMetadata` before it reaches `tokens_usages`) or at id-construction time (never bake codes into workflow ids in the first place) — or both, since they protect different sinks.--## Anchor points for a cold start--- Workflow-id naming: `wf_pipe_run.py:62`, `temporal_pipe_router.py:69`-- Node id minting: `graph_tracer.py:198`, `trace_context.py:48,64`-- Cost-report exposure: `llm_report.py:44,65-67`, `job_metadata.py:41-74`-- Redacted-twin generator: `otel_factory.py:131`-- Telemetry redaction policy: `telemetry_config.py:199`, `telemetry_manager_abstract.py:44`-- Other sinks: `config_temporal.py:41` (search attrs), `observability.py:124-160` (static summary/details, ungated), `graph_config.py:6` (data inclusion)diff --git a/wip/observer-and-telemetry/observer-telemetry-posthog.md b/wip/observer-and-telemetry/observer-telemetry-posthog.mddeleted file mode 100644index 76a809d45..000000000--- a/wip/observer-and-telemetry/observer-telemetry-posthog.md+++ /dev/null@@ -1,146 +0,0 @@-# How Observer, Telemetry, and PostHog actually fit together--Written up after a wrong mental model surfaced during the runtime-bridge DIRECT-router review (`wip/runtime-bridge/direct-mode-nested-router-leak.md`). The belief under review was: *"the `multi_observer` has nothing to do with telemetry, and we don't use it at all in practice."* Both halves are wrong in instructive ways. This doc is the corrected map, traced against the code on `feature/Runtime-bridge-extraction`.--## TL;DR — the corrected mental model--1. **The Observer subsystem's *only* job is telemetry.** The default `multi_observer` is `{"noop": ObserverNoOp, "telemetry": ObserverTelemetry}`. The noop does nothing; `ObserverTelemetry` emits PostHog product-analytics events (`PIPE_RUN`, `PIPE_COMPLETE`). There is nothing else in it. "Observer" and "pipe-level PostHog telemetry" are the same thing.--2. **It IS used in practice — but only on the Gateway/hosted path.** By default (BYO-key, direct Python SDK, pytest, CI) the telemetry manager resolves to `TelemetryManagerNoOp`, so the observer fires into a no-op and nothing leaves the process. The moment you run under **Pipelex Gateway**, a real `TelemetryManager` is built with `pipelex_telemetry_enabled=True` and those `PIPE_RUN`/`PIPE_COMPLETE` events flow to Pipelex's PostHog. So "we don't use it" is true for local/OSS, false for the hosted product.--3. **The hub-level observer is dead code.** `PipelexHub.set_observer()` stores `self._observer` and **nothing ever reads it** — there is no `get_observer`. The only live observer is the instance attribute on the `PipeRouter` (`router.observer`), consumed inside `PipeRouterProtocol.run()`. The hub write is a red herring; it provides zero redundant coverage.--## The Observer subsystem--### The protocol--`pipelex/observer/observer_protocol.py` — three async hooks plus a payload dict:--```python-class ObserverProtocol(Protocol):- async def observe_before_run(self, payload: PayloadType) -> None: ...- async def observe_after_successful_run(self, payload: PayloadType) -> None: ...- async def observe_after_failing_run(self, payload: PayloadType) -> None: ...-```--`PayloadType = dict[str, Any]`, keyed by `PayloadKey` (`PIPELINE_RUN_ID`, `PIPE_JOB`, `PIPE_OUTPUT`, `ERROR`).--### Implementations that exist--| Class | File | Does | Wired by default? |-|---|---|---|---|-| `ObserverNoOp` | `observer/observer_protocol.py:40` | nothing | yes (as `"noop"`) |-| `MultiObserver` | `observer/multi_observer.py` | fans out to a named dict of sub-observers | yes (the container) |-| `ObserverTelemetry` | `system/telemetry/observer_telemetry.py` | calls `telemetry_manager.track_event(PIPE_RUN / PIPE_COMPLETE)` | yes (as `"telemetry"`) |-| `LocalObserver` | `observer/local_observer.py` | writes payloads to disk under `results/observer` | **no — defined but never instantiated in production wiring** |--`LocalObserver` is reachable only if a caller passes a custom `observers=` dict into `Pipelex.make()`. Default boot never builds it. (Flag: dead-ish code, or a debugging aid nobody wired.)--### Where the observer fires — the one live path--`PipeRouterProtocol.run()` is the only caller of the observer hooks:--- `pipelex/pipe_run/pipe_router_protocol.py:22` → `observe_before_run`-- `:34` → `observe_after_successful_run`-- `:46` → `observe_after_failing_run`--It calls `self.observer`, i.e. the observer attached to **that router instance** at construction. `PipeRouter.__init__` defaults to `ObserverNoOp` when none is passed (`pipe_run/pipe_router.py:11`).--### The wiring at boot (`pipelex/pipelex.py`)--```python-# pipelex.py ~417-452-if not observers:- no_op_observer = ObserverNoOp()- observer_telemetry = ObserverTelemetry(telemetry_manager=self.telemetry_manager)- observers = {"noop": no_op_observer, "telemetry": observer_telemetry}-multi_observer = MultiObserver(observers=observers)-self.pipelex_hub.set_observer(observer=multi_observer) # <-- DEAD: never read back-...-# non-Temporal:-self.pipelex_hub.set_pipe_router(PipeRouter(observer=multi_observer)) # <-- the ONLY live install-# Temporal:-self.pipelex_hub.set_pipe_router(make_temporal_pipe_router()) # observer = ObserverNoOp-```--So:--- **Non-Temporal hub router** carries `multi_observer` → telemetry fires (into whatever the telemetry manager is — real or no-op).-- **Temporal router** carries `ObserverNoOp` (`temporal/tprl_pipe/temporal_pipe_router.py:52`) → **no** pipe-level telemetry on Temporal at all. Precedent: "no router observer" is already a shipped state.--### The dead hub write — proof--```-hub.py:89 self._observer: ObserverProtocol | None = None # declared-hub.py:206 self._observer = observer # written by set_observer-(no get_observer / get_required_observer anywhere in the repo)-```--Grep for `get_observer` / `_observer` readers returns only the declaration and the setter. The doc-in-review had hoped "the hub also sets the observer separately, so the router-level observer may be redundant." It isn't redundant — the hub copy is simply unused. Cleanup opportunity: either delete `_observer`/`set_observer`, or add a `get_observer` and actually route through it (decide intentionally).--## The Telemetry layer — what the observer feeds into--`ObserverTelemetry` is a thin adapter: it turns each pipe run into a `track_event` call (`observer_telemetry.py`). Everything past that is the **telemetry manager**, which has two independent sinks.--### Sink A — custom PostHog (user's own analytics)--- Config: `telemetry.toml` → `[custom_posthog] mode = ...`, model `TelemetryConfig` in `system/telemetry/telemetry_config.py`.-- Default: `PostHogMode.OFF` (`telemetry_config.py:81`).-- `track_event` (`telemetry_manager.py:245`) matches the mode: `OFF` → logs verbose and emits nothing; `ANONYMOUS`/`IDENTIFIED` → captures to the user's PostHog.--### Sink B — Pipelex Gateway telemetry (our hosted analytics)--- Flag: `pipelex_telemetry_enabled` on the `TelemetryManager`, gated by `is_pipelex_telemetry_enabled` in `TelemetryFactory.make_telemetry_manager` (`telemetry_factory.py`).-- Turned on when running under **Pipelex Gateway** (needs the gateway API key as distinct_id; `GatewayApiKeyMissingError` if absent).-- `track_event` always also fires Sink B when `_pipelex_telemetry_enabled` (`telemetry_manager.py:262`), **independent of `custom_posthog.mode`**. So even with custom PostHog `OFF`, Gateway users still emit `PIPE_RUN`/`PIPE_COMPLETE` to Pipelex's PostHog (`telemetry_factory.py:64-74`).--### When the whole telemetry manager collapses to a no-op--`TelemetryFactory.make_telemetry_manager` returns `TelemetryManagerNoOp` when:--- `DO_NOT_TRACK` env var is truthy (always respected; Gateway+DNT raises `GatewayDoNotTrackConflictError` rather than tracking), OR-- custom PostHog is `OFF` **and** Gateway telemetry is off, OR-- the **integration mode** disallows custom telemetry and Gateway is off. `is_custom_telemetry_allowed_for_mode(integration_mode)` — e.g. `pytest`/`ci`/`python` default to not allowing custom telemetry (see `telemetry.toml` `[telemetry_allowed_modes]` and the model defaults).--When it's `TelemetryManagerNoOp`, `ObserverTelemetry` still runs every pipe but `track_event` does nothing.--### "Is it used in practice?" — the matrix--| Context | custom PostHog | Gateway telemetry | Net effect of the observer |-|---|---|---|---|-| Local dev / direct Python SDK, no config | OFF (default) | off | **no-op** — fires into `TelemetryManagerNoOp` |-| pytest / CI | disallowed by mode | off | **no-op** |-| User explicitly sets `mode=anonymous/identified` | on | off | events → user's PostHog |-| **Pipelex Gateway (hosted runner)** | usually OFF | **on** | **events → Pipelex PostHog** (live, real) |-| Any of the above + `DO_NOT_TRACK=1` | — | forced off | no-op |-| Temporal router (any context) | — | — | **no-op** (router uses `ObserverNoOp`) |--So the honest statement is: *the observer is best-effort PostHog product analytics; it is genuinely inert in OSS/local/test, and genuinely live for Gateway/hosted non-Temporal runs.*--## Don't confuse these three things — they are all "observability" but separate--The word "tracing"/"observability" is overloaded across the codebase. They are independent subsystems:--1. **Observer → PostHog events** (this doc). Coarse product analytics: a `PIPE_RUN` / `PIPE_COMPLETE` event per pipe, with `pipeline_run_id` + `pipe_type`. Driven by `router.observer`. Sinks: user PostHog and/or Pipelex PostHog.--2. **OTel AI tracing** (`custom_posthog.tracing`, `pipelex_telemetry`). OpenTelemetry spans for LLM/inference calls, built in `TelemetryManager` (`telemetry_manager.py:83-106`, `otel_factory.py`). Separate enable flags, separate exporter. Not the observer.--3. **Pipelex trace-event log** (`[pipelex.tracing_config]` in `pipelex.toml`, `flush_trace_events_to_backend`). The distributed run-trace log persisted to NDJSON / DynamoDB for cross-Temporal-worker run reconstruction. This is the subject of `wip/runtime-bridge/trace-flush-blocking-io.md`. Nothing to do with PostHog or the observer.--If someone says "telemetry is off so tracing is off," check *which* of the three they mean — they don't share switches.--## Implications for the bridge DIRECT-router decision--(Full context in `wip/runtime-bridge/direct-mode-nested-router-leak.md`.) Because the router observer's sole effect is best-effort PostHog telemetry, and because:--- the **root** pipe of a DIRECT run **already** carries no observer today (bare `PipeRouter()` → `ObserverNoOp`), so DIRECT runs already emit telemetry on nested pipes but not the root — an inconsistency, not a feature;-- Temporal mode already emits nothing through this path;-- threading the observer into the bridge (Option B) would require a `get_observer` on the hub that **doesn't exist**, i.e. re-coupling the framework-agnostic bridge to internals;--→ **Option A (scope the bare router for the whole DIRECT run)** is safe: it makes root + nested consistently silent on PostHog pipe events, matching Temporal. The only loss is nested-pipe `PIPE_RUN`/`PIPE_COMPLETE` events **for Gateway users running bridge-DIRECT pipes** — a brand-new path no dashboard depends on yet. If complete pipe telemetry on this path is ever wanted, that's a separate "make telemetry whole" task (and would mean giving the root pipe an observer too, which today's code doesn't).--## Flagged cleanups (separate from the bridge work)--- `PipelexHub._observer` / `set_observer` is write-only dead code. Remove it, or wire `get_observer` and route the router through it intentionally.-- `LocalObserver` is defined but never instantiated by default boot. Confirm whether it's a debugging aid worth keeping or removable.-- `multi_observer` always contains a `"noop"` entry that does nothing on every pipe — harmless, but it's pure overhead in the fan-out loop. Likely a placeholder for "there's always at least one observer."diff --git a/wip/pathlib-refactor.workflow.js b/wip/pathlib-refactor.workflow.jsdeleted file mode 100644index 491743dcc..000000000--- a/wip/pathlib-refactor.workflow.js+++ /dev/null@@ -1,278 +0,0 @@-export const meta = {- name: 'pathlib-refactor',- description: 'Refactor filesystem path handling to pathlib.Path, converting to/from str only at boundaries (CLI, API, env)',- whenToUse: 'When you want to modernize path handling across pipelex/: use pathlib.Path internally, str only at boundaries.',- phases: [- { title: 'Inventory', detail: 'parallel read-only scan of each subsystem for filesystem-path handling' },- { title: 'Plan', detail: 'synthesize disjoint, self-contained refactor clusters' },- { title: 'Implement', detail: 'one agent per cluster applies pathlib idioms' },- { title: 'Verify', detail: 'make agent-check + targeted path tests, with repair loop' },- ],-}--// ----------------------------------------------------------------------------// Shared guidance — every agent must respect these rules.-// ----------------------------------------------------------------------------const RULES = `-PROJECT: pipelex (Python runtime). CWD is the repo root (a worktree on branch refactor/Paths).-GOAL: modern, pythonic filesystem-path handling.- - Use \`from pathlib import Path\` and Path objects for ALL filesystem path manipulation internally.- - Replace os.path.join/dirname/basename/exists/isdir/abspath/relpath, os.makedirs, os.walk, os.listdir,- string concatenation of paths, and str-typed FS path params with Path equivalents- (Path / "x", p.parent, p.name, p.exists(), p.is_dir(), p.resolve(), p.relative_to(), p.mkdir(parents=True, exist_ok=True),- Path.rglob/iterdir/walk, etc.).- - Convert to/from str ONLY at genuine boundaries: CLI argument parsing (typer/click), API request/response models,- environment-variable parsing (os.environ, PIPELEXPATH split on os.pathsep), serialization to JSON/TOML,- and third-party library calls that demand str. At those boundaries, str(path) / Path(raw_str) is correct and expected.- - Internal utility functions (e.g. in pipelex/tools/misc/) should take and pass Path. Per project policy there is- NO backward-compat requirement — change signatures directly; do NOT add \`str | Path\` unions just to be safe.- Exception: a function that is a public exported API surface OR a boundary may keep str. Use judgment and note it.--CRITICAL — DO NOT TOUCH non-filesystem "paths". These look like paths but are NOT files on disk:- - dotted/attribute paths: variable_path, dotted_path, var_path, from_path (blueprint refs), key_path, json paths,- "report.pdf"-style template variable references, module import paths (a.b.c), concept/pipe paths.- - URL paths, route paths, API endpoint paths.- - module-path<->file-path conversion helpers where the "module path" side is dotted.- If a *_path symbol is not an actual location on the filesystem, LEAVE IT ALONE and record it as excluded.--Be precise. Prefer correctness over breadth. Read the actual code before judging.-`--const INVENTORY_SCHEMA = {- type: 'object',- additionalProperties: false,- required: ['area', 'files', 'notes'],- properties: {- area: { type: 'string' },- files: {- type: 'array',- items: {- type: 'object',- additionalProperties: false,- required: ['path', 'role', 'fs_path_symbols', 'excluded_logical_paths', 'recommended_changes'],- properties: {- path: { type: 'string', description: 'repo-relative file path' },- role: { type: 'string', enum: ['core_util', 'boundary', 'internal', 'test'] },- fs_path_symbols: {- type: 'array',- description: 'Genuine filesystem-path functions/params/attrs that should move to Path',- items: {- type: 'object',- additionalProperties: false,- required: ['name', 'kind', 'detail'],- properties: {- name: { type: 'string' },- kind: { type: 'string', enum: ['function', 'method', 'param', 'attribute', 'local', 'os_path_call'] },- detail: { type: 'string', description: 'current signature/usage + line refs + what should change' },- },- },- },- excluded_logical_paths: {- type: 'array',- description: 'Symbols that look like paths but are NOT filesystem paths — must not be changed',- items: {- type: 'object',- additionalProperties: false,- required: ['name', 'reason'],- properties: { name: { type: 'string' }, reason: { type: 'string' } },- },- },- recommended_changes: { type: 'string' },- },- },- },- notes: { type: 'string', description: 'cross-file observations, especially callers of functions whose signatures should change' },- },-}--const PLAN_SCHEMA = {- type: 'object',- additionalProperties: false,- required: ['clusters', 'excluded', 'summary'],- properties: {- clusters: {- type: 'array',- items: {- type: 'object',- additionalProperties: false,- required: ['id', 'title', 'rationale', 'files', 'instructions', 'signature_changes'],- properties: {- id: { type: 'string', description: 'short kebab id, e.g. core-file-utils' },- title: { type: 'string' },- rationale: { type: 'string' },- files: { type: 'array', items: { type: 'string' }, description: 'ALL files this cluster edits — MUST be disjoint from every other cluster' },- instructions: { type: 'string', description: 'precise per-cluster refactor instructions' },- signature_changes: { type: 'array', items: { type: 'string' }, description: 'function signatures changing, so the owning agent updates every caller within this cluster' },- },- },- },- excluded: { type: 'array', items: { type: 'string' }, description: 'logical/non-fs paths confirmed left untouched' },- summary: { type: 'string' },- },-}--const CLUSTER_RESULT_SCHEMA = {- type: 'object',- additionalProperties: false,- required: ['cluster_id', 'files_changed', 'summary', 'follow_ups'],- properties: {- cluster_id: { type: 'string' },- files_changed: { type: 'array', items: { type: 'string' } },- summary: { type: 'string' },- follow_ups: { type: 'array', items: { type: 'string' }, description: 'anything left undone or risky for verification to watch' },- },-}--const VERIFY_SCHEMA = {- type: 'object',- additionalProperties: false,- required: ['passed', 'commands_run', 'failures', 'raw_tail'],- properties: {- passed: { type: 'boolean' },- commands_run: { type: 'array', items: { type: 'string' } },- failures: {- type: 'array',- items: {- type: 'object',- additionalProperties: false,- required: ['tool', 'file', 'message'],- properties: {- tool: { type: 'string', description: 'pyright | mypy | ruff | pytest | other' },- file: { type: 'string' },- message: { type: 'string' },- },- },- },- raw_tail: { type: 'string', description: 'last ~60 lines of combined output for context' },- },-}--const REPAIR_SCHEMA = {- type: 'object',- additionalProperties: false,- required: ['files_changed', 'notes'],- properties: {- files_changed: { type: 'array', items: { type: 'string' } },- notes: { type: 'string' },- },-}--// ----------------------------------------------------------------------------// Phase 1 — Inventory (parallel, read-only)-// ----------------------------------------------------------------------------phase('Inventory')--const AREAS = [- { area: 'core-utils', scope: 'pipelex/tools/misc/ (file_utils.py, toml_utils.py, json_utils.py, base64_utils.py, filetype_utils.py, diff.py, toml_sync.py) and pipelex/tools/ broadly. This is the HUB — most FS path helpers live here.' },- { area: 'cli', scope: 'pipelex/cli/ (commands/, agent_cli/, init/). Boundary-heavy: typer/click args are str and convert to Path here.' },- { area: 'tracing-observer', scope: 'pipelex/tracing/ (ndjson_event_log.py) and pipelex/observer/ (local_observer.py).' },- { area: 'system-config', scope: 'pipelex/system/ — especially environment.py (PIPELEXPATH / os.pathsep parsing — a boundary) and configuration/.' },- { area: 'temporal-plugins', scope: 'pipelex/temporal/ (worker_cli.py) and pipelex/plugins/ (mistral_factory.py and others).' },- { area: 'rest-of-src', scope: 'Everything else under pipelex/ NOT covered above: kit/, graph/, builder/, core/, pipe_operators/, pipe_controllers/, etc. Watch hard for logical/dotted paths here (variable_path, dotted_path, from_path) which must be EXCLUDED.' },- { area: 'tests', scope: 'tests/ — find call sites of FS path utilities and any direct FS path handling that will need updating when signatures change. Include tests/helpers/paths.py.' },- { area: 'export-surface', scope: 'Public export surface: pipelex/__init__.py and any re-exports, plus build a caller index. Use grep to list ALL callers across pipelex/ and tests/ of these functions: save_text_to_path, save_bytes_to_binary_file, load_text_from_path, failable_load_text_from_path, load_binary, load_binary_async, copy_file, copy_file_from_package, copy_folder_from_package, remove_file, remove_folder, mirror_dir, ensure_directory_exists, ensure_path, ensure_directory_for_file_path, get_incremental_directory_path, load_toml_from_path, save_toml_to_path. Report which of these are publicly exported (boundary) vs internal, and the full caller list per function.' },-]--const inventories = (await parallel(AREAS.map((a) => () =>- agent(- `${RULES}\n\nYou are inventorying area "${a.area}".\nSCOPE: ${a.scope}\n\n` +- `Search this scope thoroughly (grep for os.path, "import os", os.makedirs/walk/listdir/getcwd, open(, *_path: str, *_dir: str, *_file: str, string path concatenation). ` +- `For each relevant file, READ the actual usage and classify every path-like symbol as a genuine filesystem path (record in fs_path_symbols) or a logical/non-fs path (record in excluded_logical_paths with the reason). ` +- `Note callers of any function whose signature should change. Do NOT edit anything — this is read-only.`,- { agentType: 'Explore', schema: INVENTORY_SCHEMA, label: `inv:${a.area}`, phase: 'Inventory' },- ),-))).filter(Boolean)--log(`Inventory complete: ${inventories.length} areas, ${inventories.reduce((n, i) => n + (i.files?.length || 0), 0)} files flagged`)--// ----------------------------------------------------------------------------// Phase 2 — Plan (single synthesizer; barrier — needs the full inventory)-// ----------------------------------------------------------------------------phase('Plan')--const plan = await agent(- `${RULES}\n\nYou are the refactor architect. Below is the full filesystem-path inventory of the repo as JSON.\n\n` +- `${JSON.stringify(inventories, null, 2)}\n\n` +- `Produce a refactor plan as a set of CLUSTERS with these HARD constraints:\n` +- `1. Clusters must partition the work into DISJOINT file sets — NO file may appear in two clusters. If a file (e.g. a test or CLI module) calls utilities from what would be two clusters, MERGE those clusters so one agent owns all coupled files.\n` +- `2. Each cluster must be SELF-CONTAINED: if it changes a function signature, it must include that function's definition AND every caller of it (across pipelex/ and tests/). This is why disjointness matters — it guarantees parallel agents never touch the same file and signatures stay consistent.\n` +- `3. Group naturally: a "core-file-utils" cluster (file_utils.py + all its callers) will likely be the largest and that is fine — one agent owns it.\n` +- `4. Confirm the excluded logical/non-fs paths so implementers know what to leave alone.\n` +- `5. Verify caller completeness yourself with grep before finalizing — do not rely solely on the inventory notes.\n` +- `Order clusters by importance but assume they are independent (they will run in parallel). Write precise, actionable per-cluster instructions.`,- { schema: PLAN_SCHEMA, label: 'synthesize-plan', phase: 'Plan' },-)--// Safety net: detect any file that leaked into >1 cluster and warn (synthesis is instructed to avoid this).-const seen = {}-const collisions = new Set()-for (const c of plan.clusters) for (const f of (c.files || [])) { if (seen[f]) collisions.add(f); seen[f] = true }-if (collisions.size) log(`WARNING: ${collisions.size} file(s) appear in multiple clusters: ${[...collisions].join(', ')} — running sequentially-safe is NOT guaranteed; review.`)-log(`Plan: ${plan.clusters.length} clusters covering ${Object.keys(seen).length} files. ${plan.summary}`)--// ----------------------------------------------------------------------------// Phase 3 — Implement (parallel, one agent per cluster; disjoint file sets)-// ----------------------------------------------------------------------------phase('Implement')--const results = (await parallel(plan.clusters.map((c) => () =>- agent(- `${RULES}\n\nYou OWN this refactor cluster and ONLY these files — do not edit files outside the list:\n` +- `${JSON.stringify(c.files, null, 2)}\n\n` +- `Title: ${c.title}\nRationale: ${c.rationale}\nSignature changes you must propagate to every caller in your file list: ${JSON.stringify(c.signature_changes)}\n\n` +- `Instructions:\n${c.instructions}\n\n` +- `Apply the edits with the Edit/Write tools. Keep changes idiomatic and minimal — convert genuine FS path handling to pathlib.Path, keep str only at the boundaries described in the rules. ` +- `Do not run linters or tests (a dedicated verify phase does that). When done, report exactly which files you changed.`,- { schema: CLUSTER_RESULT_SCHEMA, label: `impl:${c.id}`, phase: 'Implement' },- ),-))).filter(Boolean)--const changedFiles = [...new Set(results.flatMap((r) => r.files_changed || []))]-log(`Implementation complete: ${results.length} clusters, ${changedFiles.length} files changed.`)--// ----------------------------------------------------------------------------// Phase 4 — Verify + repair loop (barrier; whole-repo lint/type check)-// ----------------------------------------------------------------------------phase('Verify')--const VERIFY_CMDS =- 'Run, from the repo root, IN THIS ORDER and capture output:\n' +- ' 1. `make agent-check` (ruff fix-unused-imports + format + lint, then pyright, then mypy)\n' +- ' 2. `.venv/bin/pytest -q -p no:randomly -m "not (inference or pipelex_api or img_gen or gha_disabled or needs_output or llm or ocr)" -k "test_boot or test_toml_utils or test_paths or inputs_path_resolver or convert_file_path_to_module_path" --tb=short` (targeted: boot/config load + path-related unit tests)\n' +- 'Report passed=true ONLY if make agent-check exits 0 AND the targeted pytest exits 0. ' +- 'Parse failures into the structured list (tool, file, message). Include the last ~60 lines of combined output in raw_tail.'--let verify = await agent(- `You are the verification gate for a pathlib refactor. ${VERIFY_CMDS}`,- { schema: VERIFY_SCHEMA, label: 'verify:1', phase: 'Verify' },-)--let attempt = 0-while (!verify.passed && attempt < 3) {- attempt++- log(`Verify failed (attempt ${attempt}): ${verify.failures.length} issue(s). Repairing...`)- await agent(- `${RULES}\n\nThe pathlib refactor introduced check/test failures. Fix them at the ROOT cause — keep the pathlib best-practice intent; ` +- `do NOT revert to os.path just to silence a checker unless the location is a genuine boundary. ` +- `If a failure reveals a misclassified logical (non-fs) path that should never have been changed, revert that specific change.\n\n` +- `Failures (JSON):\n${JSON.stringify(verify.failures, null, 2)}\n\nRaw output tail:\n${verify.raw_tail}\n\n` +- `Edit the offending files to fix every failure. Do not run the full check yourself; the gate will re-run.`,- { schema: REPAIR_SCHEMA, label: `repair:${attempt}`, phase: 'Verify' },- )- verify = await agent(- `You are the verification gate for a pathlib refactor (re-run after repair attempt ${attempt}). ${VERIFY_CMDS}`,- { schema: VERIFY_SCHEMA, label: `verify:${attempt + 1}`, phase: 'Verify' },- )-}--log(verify.passed ? 'Verification PASSED ✅' : `Verification still FAILING after ${attempt} repair attempt(s) ❌`)--return {- inventory_summary: { areas: inventories.length, files_flagged: inventories.reduce((n, i) => n + (i.files?.length || 0), 0) },- plan_summary: plan.summary,- excluded_logical_paths: plan.excluded,- clusters: plan.clusters.map((c) => ({ id: c.id, title: c.title, files: c.files })),- files_changed: changedFiles,- implementation: results,- verification: verify,-}diff --git a/wip/plugins/README.md b/wip/plugins/README.mddeleted file mode 100644index 87d2a6205..000000000--- a/wip/plugins/README.md+++ /dev/null@@ -1,56 +0,0 @@-# Pipelex plugin architecture--**Status:** assessment / design notes (not started)--> **The decided design lives in [`design.md`](design.md)** — a single unified design covering both inference (driver) plugins and orchestration (strategy) plugins, with decisions and rationale. **The executable [`implementation-plan.md`](implementation-plan.md)** is built from it (phases 0–5, grounded `file:line` refs, per-phase tests, decisions D1–D6, reviewed via `/plan-eng-review` + codex). The four documents here ([`README.md`](README.md), [`inference-backends-as-plugins.md`](inference-backends-as-plugins.md), [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md), [`temporal-as-plugin.md`](temporal-as-plugin.md)) are the background assessment that fed the design. Start with `design.md`, then `implementation-plan.md`; read these for the deeper per-area analysis.--Pipelex has two families of optional integrations that should become real, discoverable **plugins** behind a single shared seam:--1. **[Orchestrators](orchestrators-as-plugins.md)** — *strategy* plugins. They decide *where/how* a pipe runs and replace core orchestration seams. Core ships only the in-process `DIRECT` orchestrator; the distributed ones are optional plugins. Two worked instances:- - **[Temporal](temporal-as-plugin.md)** — `pipelex/temporal/` → `pipelex-temporal`. The heavy case: boot-global hub swap + per-call modes + CLI + config.- - **Mistral Workflows** — the external `pipelex-mistralai-workflows` repo. Per-call only; also a plugin for *its* host (a two-host adapter). Already wired (by a hard-coded core import — the coupling to invert).-2. **[Inference backends](inference-backends-as-plugins.md)** — *driver* plugins. The SDK wrappers under `pipelex/plugins/` (OpenAI, Anthropic, Google, Mistral, Bedrock, Fal, Docling, Linkup, …). Homogeneous, many, selected at runtime by a model's `sdk` handle.--All are already ~80–90% decoupled (lazy imports, optional extras, abstract contracts, hub/registry seams). None is yet a *plugin* in the installable sense — orchestrators are still named by string from core (both Temporal *and* Mistral); inference backends are still dispatched from hardcoded `match` statements. The work is the same shape for all: **invert the last coupling through a registry, discover via entry points, optionally repackage.**--> **Naming caveat:** "plugin" is currently overloaded. `pipelex/plugins/` holds in-tree SDK adapters, `Plugin` is a backend selector, `PluginManager`/`PluginSdkRegistry` is an SDK-client cache — none is a real plugin system. The inference doc proposes renaming those before building the real thing. Read that first if the vocabulary trips you up.-------## Synthesis: best practices for plugins on a codebase like this--Concise, ordered roughly by what to do first.--1. **Invert the dependency, always.** Core must never name a plugin module — not by import, not by string. Plugins depend on core's *published protocols*; core depends on nothing downstream. Every current coupling (Temporal's lazy imports, the inference `match` statements) becomes "ask a registry / call a hook." This is the whole game; the rest is mechanism.--2. **One discovery mechanism: entry points.** Use `importlib.metadata.entry_points(group="pipelex.plugins")`, loaded once at boot. Installing a dist makes it discoverable — zero config, no import-by-string, no scanning. Both strategy and driver plugins ride the same group.--3. **A thin `Plugin` protocol whose hooks are a *menu*, called at known lifecycle points.** Core invokes hooks (`register_hub_implementations`, `register_cli_commands`, `register_orchestrators`, `register_workers`, `setup`, `teardown`); each plugin implements only the subset it needs. Temporal uses nearly all of them (boot-global hub swap + per-call modes + CLI + config); Mistral uses only per-call mode registration. Designing the protocol against *both* keeps it from over-fitting one backend. No magic, no base-class inheritance from core internals — structural typing against a `Protocol`.--4. **Registries replace both `match` statements and hub hard-wiring.** Core owns the registry + the abstract contract (`InferenceWorkerAbstract`, `PipeRouterProtocol`, …); plugins own the entries. Three shapes, one idea: driver registries keyed by data (the `sdk` handle); the execution-mode → orchestrator registry keyed by `PipelexExecutionMode` (per-call dispatch, replacing the bridge `match`); singleton hub setters (boot-global strategy swap). Keep the *taxonomy* enums (`PipelexExecutionMode`) in core so the names exist without the implementations — that's what lets a missing backend produce "install pipelex-temporal" instead of a `None` deref.--5. **Dogfood the plugin API in-tree before externalizing.** Make the built-in backends and Temporal register *through the seam* while still living in the repo. This proves the seam with a green test suite (Phase 1 in both docs) and turns later extraction into a packaging move, not a refactor. Don't externalize on faith.--6. **Lazy SDK imports + a great `MissingDependencyError`.** Never touch a heavy SDK until its plugin is selected. Gate with `importlib.util.find_spec(...)`, and when missing, raise an error naming the lib *and* the extra *and* (where possible) an alternative — the inference factories already do this well; preserve that quality when the guard moves into registration.--7. **Optional extras are a stepping stone, not the destination.** `pipelex[temporal]`, `pipelex[anthropic]` are good today. A real plugin dist packages its own deps, so the extra collapses into the dist's dependency list. Split a dist out only when dependency weight or release cadence justifies it.--8. **Keep config schema importable without the SDK.** The `if TYPE_CHECKING: from sdk import X / else: X = Any` placeholder (see `config_temporal.py`) lets a typed config field survive on installs that skipped the extra. Decide per integration whether the schema stays in core (recommended — keeps static typing) or moves into the plugin (needs a pluggable-config hook).--9. **Detect capability without importing.** Cross-cutting code that must *not* drag an SDK onto the hot path should sniff `sys.modules.get("...")` rather than import (see `reporting_manager.py` detecting a Temporal activity context). Survives extraction unchanged.--10. **Lifecycle symmetry.** `setup()` / `teardown()` hooks, idempotent, with teardown in `finally`/registry cleanup (the `SdkClientRegistry.teardown()` and Temporal teardown already model this). Move the teardown currently inlined in `pipelex.py` behind the plugin so the boot file stops knowing about specific integrations.--11. **Publish the dependency surface as a designed SPI — don't let consumers reach into internals.** Once an out-of-tree plugin depends on you, what it imports *is* a contract. Orchestrators legitimately need more than the runtime bridge (library-crate access, `PipeJob`/`PipeOutput`, the router/run protocols, tracer hooks) — so define an orchestrator SPI rather than an honor-system "this package only" rule (which `pipelex-mistralai-workflows` already breaches). Same theme as the OpenAI *substrate* in the driver doc: a designed surface, not an accidental one. Anything imported outside the SPI is a design bug — promote it or remove the need.--12. **One repo can plug into several hosts.** An adapter is naturally multi-sided (`pipelex-mistralai-workflows` is a Pipelex orchestrator plugin *and* a Mistral Workflows activity library). One wheel, two contracts, two registrations — and they can use different mechanisms (Pipelex: entry-point discovery; Mistral: registration-by-import, since it has no registry). Don't split an adapter whose whole job is bridging two worlds.--13. **Version the core↔plugin contract.** A plugin should declare which core API it targets so an incompatible pairing fails loud at load time, not mysteriously at runtime. "No backward compatibility" is the repo policy — make the contract break *visible*. **Not yet achieved by the implementation** — the shipped `PLUGIN_API_VERSION` gate is documentary only (every plugin imports the constant, so its declared version always equals the installed core and the exact-match gate can never fire; `#1000` already shipped a breaking orchestrator-SPI change without a bump and nothing caught it). The task to build this properly (pin a literal built-against version; semver compat rule) is [`plugin-api-version-gating-semver.md`](plugin-api-version-gating-semver.md).--14. **Tests, markers, and CLI options travel with the plugin.** The `temporal` pytest marker and `--temporal-server` option move with `pipelex-temporal`; backend tests move with their dist. Keep protocol-level conformance tests in core so any plugin can be checked against the contract.--15. **Fail loud on a broken plugin.** If a discovered entry point fails to import or register, surface it with context — don't silently skip (a silently-missing backend looks like a config bug to the user). Distinguish "not installed" (fine, expected) from "installed but broken" (loud).--### The one-line version--> Define a single entry-point–discovered `Plugin` protocol; turn every place core currently *names* an integration (Temporal's lazy imports, the inference `match` statements) into a *registry lookup* or *hook call*; prove it in-tree with a green suite; repackage only what's worth repackaging.diff --git a/wip/plugins/design.md b/wip/plugins/design.mddeleted file mode 100644index 9d74ba736..000000000--- a/wip/plugins/design.md+++ /dev/null@@ -1,482 +0,0 @@-# Pipelex plugin system — design--**Status:** design (proposed, not started)-**Supersedes for design intent:** the four assessment notes in this folder ([`README.md`](README.md), [`inference-backends-as-plugins.md`](inference-backends-as-plugins.md), [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md), [`temporal-as-plugin.md`](temporal-as-plugin.md)) are the research that fed this document. This is the single, decided design the implementation plan is built from.--This document defines one plugin system covering both families of optional integration Pipelex has today: **inference backends** (the SDK adapters selected by a model's `sdk` handle) and **orchestrators** (the strategies that decide where/how a whole pipe runs — Temporal, Mistral Workflows). They are different in shape but share one discovery mechanism, one plugin contract, and one set of design rules.-------## 1. Motivation--Pipelex already has both integration families, and both are already ~80–90% decoupled — lazy imports, optional extras, abstract contracts, hub/registry seams. What's missing in both is the *last* inversion: core still **names** each integration.--- **Inference backends** are dispatched from centralized `match plugin.sdk:` statements inside four worker factories (`cogt/{llm,img_gen,extract,search}/*_worker_factory.py`). A third party cannot add a backend without editing core.-- **Orchestrators** are named by string in `runtime_bridge/bridge.py`'s `match input_payload.execution_mode:` — the `TEMPORAL_*` arms lazy-import `pipelex.temporal`, the `MISTRAL_NATIVE` arm **hard-imports** `pipelex_mistralai_workflows`. Temporal is further wired into core boot/teardown (`pipelex.py`), config (`system/configuration/configs.py`, the one hard import of the package), and the CLI (`cli/_cli.py`, two hard imports).--The job is the same for both: **invert the last coupling through a registry, discover via entry points, and (only where it pays off) repackage.** This is not a rewrite of the integration code — it is a seam in core plus rewiring a small, enumerated set of call sites.--### The naming overload (fix first)--"Plugin" is currently overloaded three ways in `pipelex/plugins/`, and the ambiguity has already leaked into the codebase (the hub raises `"PluginManager2 is not initialized"`). None of the three is a plugin in the installable sense:--- `pipelex/plugins/<backend>/` — directories holding **in-tree SDK adapters**.-- `Plugin` (`plugins/plugin.py`) — a value object (`sdk` + `backend` + `variant`, with an `sdk_handle` like `anthropic@bedrock/claude`). A **backend selector / handle**.-- `PluginManager` / `PluginSdkRegistry` — a **runtime cache of instantiated SDK clients**, keyed by that handle.--The word "plugin" is reserved, from here on, for the new concept: an installable, discoverable unit that registers extensions at boot. See §4 for the rename.-------## 2. Goals and non-goals--### Goals--- **Core names no integration.** No import of an integration module, no string literal naming one, anywhere on the core code path.-- **One discovery mechanism** for both families: the `pipelex.plugins` entry-point group, plus a static built-in list for in-tree plugins.-- **One plugin contract** — a small, menu-style registration surface that a plugin implements only the subset of.-- **Dogfood in-tree first.** Built-in backends and Temporal register *through the seam* while still living in the repo, proven by a green suite, before anything is externalized. Externalization then becomes a packaging move, not a refactor.-- **Designed dependency surfaces (SPIs).** What an out-of-tree plugin imports is a contract; publish it deliberately instead of letting plugins reach into internals.-- **Preserve today's user-facing behavior and error quality** — same model selection by data, same friendly "install the X extra" messages, same execution modes.--### Non-goals--- **Not** splitting every vendor backend into its own dist. In-tree-but-registerable is the default; externalize only where dependency weight or release cadence justifies it (Temporal qualifies; vendor inference dists do not, yet).-- **Not** a third-party-defined execution-mode space. Plugins supply orchestrators for the *existing* `PipelexExecutionMode` values; inventing new modes is an open string-keyed space we defer until something needs it.-- **Not** a generic typed-config-contribution mechanism in v1. First-party plugin config (Temporal's) stays typed in core; a generic per-plugin config namespace is a documented future extension, built when a third party needs it.-- **Not** backward compatibility shims. Per repo policy, breaks are noted in changelogs and made loud, not smoothed over.-------## 3. Concepts and vocabulary--| Term | Meaning |-|---|---|-| **Plugin** | An installable, discoverable unit that registers extensions into core at boot. May ship in-repo (built-in) or as a separate dist (external). Implements the plugin contract (§5). |-| **Driver plugin** | A plugin that *supplies an implementation* for a backend core asked for. Many, homogeneous, selected by data (a model's `sdk` handle). The inference backends. |-| **Orchestrator** | A strategy that decides *where/how a whole pipe job runs* (in-process, on a Temporal fleet, on Mistral Workflows). Distinct from a *pipe controller*, which sequences sub-pipes within one run. |-| **Orchestrator plugin** | A plugin that *replaces core orchestration behavior* for one or more execution modes. Few, heterogeneous. |-| **SPI** (Service Provider Interface) | The published, versioned surface of core that plugins are allowed to import. Two of them: an inference SPI and an orchestrator SPI. |-| **`ModelHandle`** | The renamed value object (was `Plugin`): `sdk` + `backend` + `variant`, with `sdk_handle`. Selects which driver runs a model. |-| **`SdkClientRegistry` / `SdkClientManager`** | The renamed runtime cache (was `PluginSdkRegistry` / `PluginManager`): instantiated SDK clients keyed by `sdk_handle`, with lifecycle teardown. |--"Plugin" and "ships as a separate dist" are **orthogonal axes**. Being a plugin means *registering through the seam* instead of being a hardcoded special case. Shipping separately is a packaging decision. A built-in plugin that lives in this repo is the default, not a contradiction.--| | In-repo | Separate dist |-|---|---|---|-| **Through the seam** | built-in plugins (the default for every backend) | external plugins (`pipelex-temporal`, `pipelex-mistralai-workflows`, future vendor dists) |-| **Hardcoded in core** | the wart we are deleting | n/a |-------## 4. Architecture overview--One discovery pass produces a list of plugin objects. Each plugin is handed a **registrar** and fills whichever of three registry shapes it serves. Core then consults those registries at the points where it used to name an integration. Plugins import only the published **SPIs**.--```-boot- └─ discover plugins ─── built-in static list (in-repo: openai, gateway, anthropic, …, temporal)- └─── pipelex.plugins entry points (external dists)- │- ▼ for each plugin (after API-version check): plugin.register(registrar)- registrar fans contributions into three shapes:- ├─ keyed registries (lookup by data)- │ ├─ InferenceBackendRegistry — keyed by (family, sdk handle) ← driver plugins- │ └─ OrchestratorRegistry — keyed by PipelexExecutionMode ← orchestrator plugins (per-call)- ├─ exclusive hub slots (boot-global swap, at most one claimant)- │ └─ content_generator / pipe_router / pipe_run / task_manager ← orchestrator plugins (boot-global)- └─ additive contributions- ├─ CLI commands (Typer)- └─ teardown callbacks (run LIFO at shutdown)- │- ▼ core consults the registries where it used to name an integration:- worker factories → InferenceBackendRegistry.lookup(family, sdk)- bridge dispatch → OrchestratorRegistry.get(execution_mode)- hub defaults → whatever claimed each exclusive slot, else core default-```--The three registry shapes are deliberate — they match the three ways an integration plugs in:--- **Keyed registries** — looked up by runtime data. Inference backends keyed by `(family, sdk)`; orchestrators keyed by `PipelexExecutionMode`. Many entries, additive.-- **Exclusive hub slots** — a boot-global *swap* of a singleton (the default content generator / pipe router / pipe run / task manager). At most one plugin may claim each slot; two claimants is a loud conflict. Only Temporal uses these today.-- **Additive contributions** — CLI commands and teardown callbacks. Appended, never exclusive.--Two SPIs sit underneath:--- **Inference SPI** — the contracts a driver plugin compiles against (the `*WorkerAbstract` hierarchy, `ModelHandle`, `MissingDependencyError`, the `SdkClientRegistry`, the registration factory contract, and the OpenAI-compat substrate).-- **Orchestrator SPI** — the contracts an orchestrator plugin compiles against (the execution protocols, the boundary/core payload types, library-crate access, tracing hooks). Sized to what the existing orchestrators actually import (§9).-------## 5. Discovery and the plugin contract--### 5.1 The registrar pattern--A plugin is an object exposing identity plus a single `register` method:--```python-@runtime_checkable-class PipelexPlugin(Protocol):- name: str- targets_api: int # the PLUGIN_API_VERSION it was built against- def register(self, registrar: PluginRegistrar) -> None: ...-```--`register` is the *menu*: the plugin calls whichever registrar methods it needs and ignores the rest. This is preferred over a six-method "god protocol" (every plugin stubbing methods it doesn't use) or per-capability `isinstance` dispatch (reflection that ignores signatures): one method, one well-defined call point, no magic.--```python-class PluginRegistrar:- # keyed registries- def add_inference_backend(self, *, family: InferenceFamily, sdk: str, factory: InferenceBackendFactory) -> None: ...- def add_orchestrator(self, *, mode: PipelexExecutionMode, orchestrator: OrchestratorProtocol) -> None: ...- # exclusive boot-global slots (raise on a second claimant)- def claim_content_generator(self, generator: ContentGeneratorProtocol) -> None: ...- def claim_pipe_router(self, router: PipeRouterProtocol) -> None: ...- def claim_pipe_run(self, pipe_run: PipeRunProtocol) -> None: ...- def claim_task_manager(self, task_manager: TaskManagerProtocol) -> None: ...- # additive- def add_cli_command(self, *, name: str, help: str, command: Callable[..., object]) -> None: ...- def add_teardown(self, callback: Callable[[], None]) -> None: ...- # read access to the resolved config, so a plugin can gate its own contributions- @property- def config(self) -> ConfigRoot: ...-```--A plugin gates its own contributions on config. Temporal's `register` claims the exclusive hub slots **only when `config.temporal.is_enabled`**, but registers its per-call orchestrators and CLI **unconditionally** (so the bridge can dispatch a `TEMPORAL_*` run, and the worker CLI exists, regardless of whether *this* process boots as a Temporal-default runtime). This split is the heart of the two activation models — see §8.--### 5.2 Built-ins vs externals (one contract, two discovery paths)--- **Built-in plugins** are registered from a static list in core (`BUILTIN_PLUGINS = [OpenAIDriver(), GatewayDriver(), AnthropicDriver(), …, TemporalOrchestrator(), …]`). Explicit, fast, no self-scanning of dist metadata. One plugin per *externalizable unit* (per vendor for drivers; one for Temporal), so promoting a built-in to an external dist later is "move it out of the static list, add an entry point in its own dist."-- **External plugins** are discovered via `importlib.metadata.entry_points(group="pipelex.plugins")`, loaded once at boot. Installing a dist makes it discoverable — zero config.--Both paths produce `PipelexPlugin` objects fed through the identical `register(registrar)` pipeline. The seam is dogfooded by every built-in; entry-point discovery is exercised by every external (and by Mistral, which is already a separate repo).--Rationale for the hybrid over pure entry-point discovery for built-ins: declaring a dozen self-referential entry points in `pipelex`'s own `pyproject.toml` and having `pipelex` discover *itself* via `importlib.metadata` is needless ceremony. A static list is clearer and faster, and externalization stays a one-line move. (Alternative considered: pure entry points for built-ins too — rejected as over-engineered for the in-tree default.)--### 5.3 Ordering, timing, and safety--- Discovery and `register` run at one well-defined point in boot — after config is loaded (so plugins can gate on it), before the first worker or pipe run is built. Boot-global slot claims (§8) are *collected* at this point and *applied* at the existing ordered setup points in `Pipelex.setup()` (content generator, then task manager, then router, then run), each falling back to the core default when unclaimed.-- **Core defaults are unconditional.** The `DIRECT` orchestrator and every core-dependency inference backend (`openai`, `gateway`, `portkey`, `pypdfium2`, `azure_rest`) register from core's built-in list and can never be dropped by a misconfigured environment.-- **Registration is import-light** (the lazy-load contract — see §6.5). Building `BUILTIN_PLUGINS` and running every plugin's `register` happens at boot for *all* backends; it must not import any backend SDK. Heavy imports stay inside the factory's `make_worker` / the orchestrator's `run`, exactly where the `match` arms lazy-import today. A plugin object and its `register` therefore live in import-light modules; the SDK is touched only when the backend is actually selected.-- **Conflict policy: duplicate keys fail loud.** Registering a second factory for the same `(family, sdk)` or a second orchestrator for the same `execution_mode` is a startup error naming both contributors — never a silent shadow. Same for an exclusive hub slot claimed twice (§8). This is consistent with the repo's "make breaks visible" stance and guards the transition window where an externalized backend could collide with a still-present built-in. *Deliberately overriding* a built-in (a third party replacing core's OpenAI driver) is a separate, explicit feature — deferred until something needs it, never an accident.-- **Teardown** runs registered callbacks LIFO at shutdown, replacing the Temporal teardown currently inlined in `pipelex.py`.--### 5.4 Versioning the contract--Core exposes `PLUGIN_API_VERSION: int`. Each plugin declares `targets_api`. On load, an incompatible pairing fails loud at startup (naming the plugin, its target, and the core version) — never mysteriously at runtime. A single integer, bumped on any breaking change to the contract or either SPI; "no backward compatibility" means we make the break *visible*, not that we tolerate silent skew.--The single integer is deliberately coarse: a change to *either* SPI bumps it, forcing *every* plugin to re-declare even if its own surface was untouched. With the small set of external plugins we control (Temporal, Mistral), that friction is acceptable and the simplicity wins. Per-SPI versions or semver-range matching are deferred until there are external plugins on independent release trains.--### 5.5 Fail-loud, and "not installed" vs "broken"--- **Not installed** is expected and friendly: a requested backend/mode whose plugin isn't present yields a typed error naming the package *and* the extra *and*, where one exists, an alternative (e.g. "use Anthropic via Bedrock instead"). This is preserved verbatim from today's `MissingDependencyError` quality. Today's typed orchestrator errors (`MissingPipelexTemporalExtraError`, `MissingMistralWorkflowsPluginError`) map onto the registry-miss path — the bridge's `MissingOrchestratorError` carries the same "install `pipelex-temporal`" guidance instead of an `ImportError` caught inline.-- **Installed but broken** is loud: a discovered entry point that fails to import or whose `register` raises is surfaced with context — never silently skipped (a silently-missing backend reads to the user as a config bug).-- The **taxonomy enums stay in core** (`PipelexExecutionMode`, the `requires_*` properties). Core knows the *names* of strategies without their *implementations* — that is what lets a missing orchestrator produce "install pipelex-temporal" instead of a `None` dereference, and keeps the gates statically typed.--### 5.6 Developer experience for plugin authors--The plugin author *is* the user of this system, and the SPI + contract *is* the product they touch. Treat that surface with the same care as a public API.--- **Progressive disclosure — learn only your slice.** The registrar menu means a backend author calls only `add_inference_backend` and implements one `InferenceBackendFactory`; an orchestrator author calls only `add_orchestrator` (plus, for the boot-global case, the `claim_*` slots). Neither has to learn the other family, the hub, or the parts of the SPI they don't use. The minimal backend plugin is: a `PipelexPlugin` (name + `targets_api` + `register`), one factory with `make_worker`, and (if external) one entry-point line. That short path is the time-to-first-plugin we optimize for.-- **Author-facing error empathy.** Errors caused by a plugin's *own* mistake must name the offending plugin and the fix, never surface as an opaque core failure: a factory whose `make_worker` returns the wrong `*WorkerAbstract` for its family, a registration under the wrong `(family, sdk)`, a duplicate key (§5.3), or a missing required identity field. Validate at registration where it's cheap (identity present, `targets_api` set), and at first use where it isn't (worker type matches family) — each error names the plugin.-- **Actionable upgrades (fight upgrade fear).** A `targets_api` mismatch fails loud with the *remedy* in the message: "plugin `pipelex-foo` targets plugin API N, this `pipelex` is API M — upgrade `pipelex-foo`, or pin `pipelex<X`." Any `PLUGIN_API_VERSION` bump or SPI change ships with a migration note in the changelog so an external author knows what moved and why.-- **A getting-started guide and a minimal example are part of the deliverable** (§16), not a follow-up — an SPI without an authoring guide has no real time-to-first-plugin. The example is the canonical "hello world" plugin authors copy from.-------## 6. Inference (driver) plugins--### 6.1 The seam today--Four worker factories, each a centralized `match plugin.sdk:` that, per arm, does the same three things: optionally `importlib.util.find_spec(...)` to raise a friendly `MissingDependencyError`, lazily import the backend's factory + worker, then build the worker (reusing a cached SDK client from `SdkClientRegistry`). The lazy import inside each arm is already "don't touch the SDK until selected" — it just needs to become "ask the registry for the selected backend's factory."--| Family | File | Selected by |-|---|---|---|-| LLM | `cogt/llm/llm_worker_factory.py` | `plugin.sdk` (gateway/portkey/openai/anthropic/mistral/bedrock/google variants) |-| ImgGen | `cogt/img_gen/img_gen_worker_factory.py` | `plugin.sdk` (gateway/fal/huggingface/openai/blackboxai/azure_rest/google/openrouter variants) |-| Extract | `cogt/extract/extract_worker_factory.py` | `plugin.sdk` (gateway/mistral/pypdfium2/docling/linkup variants) |-| Search | `cogt/search/search_worker_factory.py` | `plugin.sdk` (linkup/gateway variants) |--### 6.2 The registry and the factory contract--One `InferenceBackendRegistry`, keyed by `(family, sdk)`, mapping to an `InferenceBackendFactory`. Core owns the registry and the abstract contracts; driver plugins own the entries.--```python-class InferenceFamily(StrEnum):- LLM = "llm"- IMG_GEN = "img_gen"- EXTRACT = "extract"- SEARCH = "search"--class InferenceBackendFactory(Protocol):- def make_worker(- self,- *,- inference_model: InferenceModelSpec,- backend: InferenceBackend,- sdk_clients: SdkClientRegistry,- reporting_delegate: ReportingProtocol | None,- ) -> InferenceWorkerAbstract: ...-```--Each worker factory collapses to a lookup:--```python-factory = inference_backend_registry.lookup(family=InferenceFamily.LLM, sdk=model_handle.sdk)-return factory.make_worker(inference_model=…, backend=…, sdk_clients=…, reporting_delegate=…)-```--The `find_spec` guard and its friendly `MissingDependencyError` move **into the factory's `make_worker`**, preserving today's semantics: a model needing an absent extra fails *when used*, with the same helpful message — not at boot. A registry *miss* (backend not registered at all) is a distinct, equally-loud error ("backend `<sdk>` is not registered; is its plugin installed?").--Inconsistent factory shapes today (some have a factory class, some instantiate the worker inline) are normalized onto this one `InferenceBackendFactory` contract so every entry looks the same.--**The refactor is a DRY win, not just a relocation.** Every `match` arm today repeats the same `sdk_clients.get(...) or sdk_clients.set(...)` client-cache dance and the same `find_spec → MissingDependencyError(lib, extra, msg)` shape. Lift both into the shared surface: the client-memoization belongs in a small base (`make_worker` asks `sdk_clients.get_or_create(handle, build_client)`), and the extra-guard becomes one helper (`require_sdk(spec="anthropic", extra="anthropic", msg=…)`). Each backend factory then carries only its genuinely-unique parts: which client to build and which worker to wrap. This collapses the largest block of duplication in the current factories.--### 6.3 Cross-family vendors--`mistral` appears in LLM and Extract; `google` in LLM and ImgGen; `linkup` in Extract and Search; `gateway_*` in all four. A vendor's driver plugin simply calls `add_inference_backend` once per `(family, sdk)` it serves. With `(family, sdk)` as the key this is natural — no special handling, and a future per-vendor dist registers into multiple families from its one `register`.--### 6.4 OpenAI: substrate vs driver--The OpenAI SDK is two concerns with opposite fates:--- **Substrate** — the OpenAI-compatible client (`AsyncOpenAI(base_url=…)`), the shared `ChatCompletion*` types, and the completions/responses plumbing that `gateway`, `portkey`, `openrouter`, `blackboxai`, `azure_rest` reuse (and that `anthropic`/`mistral`/`google` import for shared types). This is **infrastructure other backends import**, not a backend. It belongs in **core as an always-on library** (`openai` is already a core dependency). If it were an optional dist, a dozen backends would diamond-depend on it.-- **Driver** — the `openai` / `azure_openai` / `openai_img_gen` registrations. An **always-on built-in plugin, through the seam** — no extra, registered unconditionally, but with no privileged `match` arm. The seam's whole value is that core names no backend; a surviving special case is a two-tier system the next "important" backend will want to join.--**Decision:** the OpenAI driver is an always-on built-in plugin in v1. Promoting the substrate into a deliberately-named core module (e.g. `pipelex.openai_compat`) with a stable signature is **deferred to the externalization phase** — it only becomes a *contract* problem when an out-of-tree dist imports it, and in-tree backends importing each other's modules is acceptable. Doing it earlier is churn with no in-tree payoff. (Captured as the first step of any future "externalize an OpenAI-compatible backend" work.)--### 6.5 Preserving lazy loading (import-light registration)--The current lazy-load behavior — never touch a backend's SDK until a model selects it — is load-bearing for boot time across the CLI, the API, the workers, and the test suite. The registry must preserve it exactly, and that imposes one hard constraint: **registration is import-light**.--- The built-in plugin objects and their `register` methods live in modules that import no backend SDK. `BUILTIN_PLUGINS = [AnthropicDriver(), …]` must be constructible without importing `anthropic`.-- `add_inference_backend(family, sdk, factory)` receives a factory whose construction is also SDK-free. The backend's worker/client modules are imported *inside* `make_worker`, mirroring today's per-arm lazy import.-- The `find_spec` guard runs inside `make_worker` too, so an absent extra fails at use with the friendly message — never at boot.--A boot-time test asserts the invariant (booting with the built-ins registered imports no optional SDK; see §14.3). This is the one place where a careless refactor would silently regress performance, so it gets an explicit guard.--### 6.6 Packaging granularity--In-tree-but-registerable is the v1 destination for all inference backends: third parties can add a backend in their own dist without a core PR, the `match` statements die, and there is no distribution split to maintain. Per-vendor or bundle dists (`pipelex-anthropic`, `pipelex-aws` = bedrock+s3+dynamodb, …) are **deferred** — split a backend out only when its dependency weight or release cadence actually justifies it. The seam makes that a packaging move.-------## 7. Orchestration (strategy) plugins — the category--An orchestrator answers *"where/how does this pipe run?"*. Pipelex already models the choice as `PipelexExecutionMode`:--- **`DIRECT`** — in-process async. The trivial orchestrator. **Ships in core**, registered through the seam like everything else.-- **`TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`** — distributed via Temporal. Today in `pipelex/temporal/`; destined for `pipelex-temporal`.-- **`MISTRAL_NATIVE`** — distributed via Mistral Workflows. Already in the external `pipelex-mistralai-workflows` repo.--`DIRECT` is the only mode that belongs in core. The other two are **peers** — optional, discovered, neither privileged. Mistral is the proof that "orchestrator" is a category with more than one member, so the seam must be generic and N-ary, not "Temporal plus a special case." Core today fails this test: it names *both* external orchestrators by string.-------## 8. Orchestration — the two activation models--The single most important subtlety: an orchestrator plugs into **two distinct execution entry points**, and a complete plugin serves both.--- **Per-call dispatch (the bridge).** Host runtimes call `run_pipe_via_bridge(...)` and pick an orchestrator by `execution_mode` for each run. This is the `OrchestratorRegistry`, keyed by mode. Both Temporal modes *and* Mistral use this. Registrar hook: `add_orchestrator(mode=…, orchestrator=…)`.-- **Boot-global default (the hub).** A normal (non-bridge) pipe run resolves `get_pipe_run()` / `get_pipe_router()` / `get_content_generator()` from the hub. Inside a Temporal worker these defaults must *be* the Temporal implementations, because the content generator has to dispatch each operation as an activity. This is the **exclusive hub-slot swap**, claimed only when the plugin's config slice is enabled. Registrar hooks: `claim_content_generator/pipe_router/pipe_run/task_manager`.--These are orthogonal and both necessary:--| | Per-call (bridge) | Boot-global (hub default) |-|---|---|---|-| Entry point | `run_pipe_via_bridge`, by `execution_mode` | `get_pipe_run()` / `get_pipe_router()` / `get_content_generator()` |-| Shape | keyed `OrchestratorRegistry` | exclusive hub slots |-| Used by | host runtimes selecting a mode per call | this process running as a Temporal-default runtime |-| Temporal | registers both modes, unconditionally | claims slots only when `temporal.is_enabled` |-| Mistral | registers `MISTRAL_NATIVE`, unconditionally | claims nothing — installs its router *inside* the child workflow at runtime |--Mistral is the counter-example that keeps the contract honest: it uses only `add_orchestrator` and nothing at boot. Temporal is the heaviest instance — it exercises per-call orchestrators, the boot-global swap, CLI commands, lifecycle teardown, and a config slice. Designing the registrar against *both* keeps it from over-fitting Temporal.--### 8.1 The bridge collapse--`bridge.py`'s `match input_payload.execution_mode:` — whose arms hard-code Mistral and lazy-import Temporal — collapses to a registry lookup:--```python-orchestrator = orchestrator_registry.get(input_payload.execution_mode)-if orchestrator is None:- raise MissingOrchestratorError(- mode=input_payload.execution_mode,- hint="install pipelex-temporal / pipelex-mistralai-workflows",- )-return await orchestrator.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)-```--After this, `bridge.py` names no integration. `DIRECT` registers from core and flows through the same lookup. The `requires_pipelex_temporal` / `requires_mistral_workflows_extra` properties on the enum stay as the typed source of truth for *which modes need what*.--**Correctness watch — the DIRECT orchestrator must keep its router scoping.** Today `_run_direct` wraps the run in `scoped_pipe_router(PipeRouter())` so that, inside a Temporal-enabled worker (whose hub-default router is the Temporal one), a `DIRECT` call's nested controller sub-pipes resolve the in-process router instead of leaking to Temporal. The extracted `DirectOrchestrator.run` must preserve this scope verbatim — dropping it is a silent correctness bug that only surfaces for DIRECT-inside-a-Temporal-worker (a real configuration). A regression test pins it (§14.3).--### 8.2 Boot/teardown collapse--The four `if get_config().temporal.is_enabled: from pipelex.temporal…` blocks in `Pipelex.setup()` (content generator, task manager, pipe router, pipe run) and the matching teardown become: core consults the exclusive hub slots at the same ordered points, using whatever a plugin claimed, else the core default. The Temporal plugin's `register` populates those slots (gated on its config). The teardown inlined in `pipelex.py` becomes a registered teardown callback. CLI registration in `_cli.py` (today two hard imports) becomes plugin-contributed commands.-------## 9. The published SPIs--What an out-of-tree plugin imports *is* a contract. We publish two SPIs deliberately rather than letting plugins reach into internals, and we size them to what the orchestrators **actually import today** — measured, not guessed.--### 9.1 Inference SPI--The surface a driver plugin compiles against:--- the worker contracts: `InferenceWorkerAbstract` → `{LLM,ImgGen,Extract,Search}WorkerAbstract`;-- `InferenceModelSpec`, `ModelHandle`, `InferenceBackend` (the backend config record);-- `MissingDependencyError` and the inference error base;-- `SdkClientRegistry` (client memoization + teardown);-- the `InferenceBackendFactory` / `InferenceFamily` registration contract;-- *(deferred)* the `openai_compat` substrate, once extracted (§6.4).--### 9.2 Orchestrator SPI--A measurement of `pipelex-mistralai-workflows`'s actual imports shows the real surface an out-of-process orchestrator needs — far beyond the honor-system "`runtime_bridge.*` only" rule it already breaches:--- **`runtime_bridge.*`** — `bridge`, `bootstrap`, `execution_mode`, `exceptions`, **and `runtime_bridge.primitives.*`** (`delivery`, `hydration`, `pipe_classification`, `submitter_hydration`, `trace_flush`). The primitives live under `runtime_bridge` and are clearly the intended host-runtime helper surface; sanction them explicitly.-- **execution protocols** — `PipeRouterProtocol`, `PipeRunProtocol`, `ContentGeneratorProtocol`, and the task-manager protocol.-- **boundary/core payload types** — `PipeJob`, `PipeOutput`, `DeliveryAssignment`, `WorkingMemory` (+ factory), `JobMetadata`, the `LibraryCrate`.-- **library-crate access + hub scoping** — `set/get_current_library`, `scoped_pipe_router`, `get_class_registry` (per-call library hydration; Mistral already does this via `library_crate_dump`).-- **tracing/graph hooks** — `trace_events`, `graph_tracer_manager`, `tracing_assembly` — an orchestrator must integrate with these to emit per-step trace/usage events across the boundary.--Anything an orchestrator imports *outside* the SPI is a design bug to resolve — promote it into the SPI or remove the need. This is the same theme as the OpenAI substrate becoming a deliberate contract: once an out-of-tree consumer depends on you, the dependency surface must be designed, not accidental.--**Mistral boundary resolution.** `pipelex-mistralai-workflows` documents (its `wip/boundary-violation-mistral-native.md`) that the native tier imports `pipelex.hub`, `pipelex.pipe_run.*`, `pipelex.core.*`, `pipelex.graph.*`, `pipelex.tracing.*` — outside its declared surface. The resolution is **(A) widen the published surface to the SPI above**, then re-point the plugin at it — *not* keep an honor-system rule that's already broken. Until the SPI lands, the plugin's pinned-rev coupling to `pipelex` is the pragmatic interim (already in place).--### 9.3 SPI shape--The SPI is a **documented, versioned set of modules and symbols** — not an `__init__.py` re-export shim (the repo bans re-exports). It is gated by `PLUGIN_API_VERSION` (§5.4). The plan decides whether to additionally provide a thin curated import surface; the contract itself is the documented module/symbol list plus the version marker.-------## 10. Configuration--Temporal is the only integration with a config slice today (`temporal: Temporal` on `PipelexConfig`, the one hard import of the package in `configs.py`). The `Temporal` schema is pure data with **no `temporalio` runtime dependency** (it uses the `if TYPE_CHECKING: from temporalio.common import RetryPolicy / else: RetryPolicy = Any` placeholder), so it is importable on installs that skipped the extra.--**Decision: keep first-party plugin config schemas typed in core.** The `Temporal` schema stays in `pipelex`; only the *implementation* moves to `pipelex-temporal`. This keeps the root config statically typed, keeps boot's config-load test (`make tb`) meaningful, and makes the config seam a near-no-op — the implementation move does not drag config plumbing with it.--A **generic per-plugin config namespace** for *third-party* plugins (which cannot add typed fields to core's model) is a documented future extension, not built in v1 — there are no third-party plugins yet, and building the mechanism now is speculative. When one needs config, add a `plugins: dict[str, dict[str, Any]]` namespace each plugin parses its own slice of. (Alternative considered: move Temporal config out to the plugin via a pluggable section now — rejected: loses static typing on that subtree and adds validation plumbing for no present benefit.)-------## 11. Packaging and distribution--| Integration | v1 home | Externalization |-|---|---|---|-| Core (`DIRECT`, openai/gateway/portkey/pypdfium2/azure_rest drivers, openai-compat substrate) | in `pipelex`, always-on | never leaves core |-| Vendor inference drivers (anthropic, google, mistral, bedrock, fal, docling, linkup, huggingface) | in `pipelex`, through the seam | deferred — split only when dep weight / cadence justifies |-| Temporal | in `pipelex/temporal/`, through the seam (Phase: wire first) | **→ `pipelex-temporal`** (depends on `pipelex` + `temporalio`, declares the entry point). High value: independent cadence, heavy SDK, already ~90% separable, and downstream (`pipelex-worker`, `pipelex-api-hosted`) already pin temporal separately. |-| Mistral Workflows | already `pipelex-mistralai-workflows` (external repo) | flip the **hard import** to **entry-point discovery**; resolve its boundary against the SPI; flip its dev path-pin to a published `pipelex` once the SPI lands |--### 11.1 Tests, markers, and CLI travel with the plugin--When an integration is externalized, its tests move with it. The `temporal` pytest marker and the `--temporal-server` CLI option (in `conftest.py`) relocate to `pipelex-temporal`; backend tests move with their dist. **Protocol-level conformance tests stay in core** so any plugin — built-in or external — can be checked against the contract. Downstream pins flip from `pipelex[temporal]==X` to `pipelex-temporal==Y` (which itself pins a compatible `pipelex`).--### 11.2 One repo, two hosts (the adapter shape)--`pipelex-mistralai-workflows` depends on **both** `pipelex` and `mistralai-workflows`: a Pipelex orchestrator plugin on one side, a Mistral Workflows activity/workflow library on the other. The two directions use **different mechanisms** and must not be conflated:--- **→ Pipelex: entry-point discovery.** `[project.entry-points."pipelex.plugins"]` makes `MISTRAL_NATIVE` available; core stops hard-coding the import.-- **→ Mistral: registration-by-import.** Mistral has no plugin registry; the user (or the package's `register_pipelex_primitives(...)` helper) hands workflow/activity classes to their worker. No entry point. (Temporal is identical on *its* host — you always register workflows/activities with a worker explicitly.)--This is the natural shape for an adapter — do not split it.-------## 12. Worked examples--**A driver plugin (built-in, e.g. Anthropic).** Implements `PipelexPlugin`; its `register` calls `add_inference_backend(family=LLM, sdk="anthropic", factory=AnthropicLLMBackendFactory())` (and `bedrock_anthropic` likewise). The factory's `make_worker` runs the `find_spec("anthropic")` guard → friendly error, builds/caches the client via `sdk_clients`, returns an `AnthropicLLMWorker`. Lives in core's `BUILTIN_PLUGINS`. Externalizing later = move to `pipelex-anthropic` + add an entry point.--**The Temporal orchestrator plugin (heaviest).** Its `register`: always `add_orchestrator(TEMPORAL_BLOCKING, …)` and `add_orchestrator(TEMPORAL_FIRE_AND_FORGET, …)`; always `add_cli_command("worker", …)` and `add_cli_command("setup-temporal-namespace", …)`; **if `config.temporal.is_enabled`**, `claim_content_generator(ContentGeneratorInWorkflow())`, `claim_pipe_router(make_temporal_pipe_router())`, `claim_pipe_run(make_temporal_pipe_run())`, `claim_task_manager(TemporalTaskManager())` and `add_teardown(...)` for the task manager. Config schema stays in core; implementation moves to `pipelex-temporal`.--**The Mistral orchestrator plugin (per-call only).** Its `register`: `add_orchestrator(MISTRAL_NATIVE, MistralWorkflowsOrchestrator())`. Nothing at boot. Installs its own router inside the child workflow at runtime. Discovered via its entry point.--**An out-of-tree third-party backend.** A dist declaring `[project.entry-points."pipelex.plugins"] my-backend = "my_pkg.plugin:MyBackendPlugin"`, depending on `pipelex>=X` and its own SDK. Installing it makes its `sdk` selectable from model TOML — no core change.-------## 13. Key decisions--| # | Decision | Choice | Why / alternative |-|---|---|---|---|-| D1 | Naming | `Plugin`→`ModelHandle`; `PluginManager`/`PluginSdkRegistry`→`SdkClientManager`/`SdkClientRegistry`; "plugin" reserved for the new concept | Kills the three-way overload (and the `PluginManager2` artifact). Pure rename, lands first. Alt name `BackendSelector` noted. |-| D2 | Discovery | Hybrid: static built-in list + `pipelex.plugins` entry points for externals | Avoids self-referential entry points / self-metadata-scan; externalization stays a one-line move. Alt: pure entry points everywhere — over-engineered for the in-tree default. |-| D3 | Plugin contract | One `PipelexPlugin.register(registrar)` method; registrar is the menu | Idiomatic, no god-protocol, no `isinstance`-reflection. Alts: capability protocols / six optional hooks — more magic or more boilerplate. |-| D4 | Registry shapes | Keyed registries (inference, orchestrators) + exclusive hub slots + additive (CLI, teardown) | Matches the three real ways integrations plug in; exclusive slots fail loud on conflict. |-| D5 | Execution-mode space | Closed; enum stays in core; plugins supply orchestrators for existing modes | Enables "install X" errors and typed gates. Open string-keyed modes deferred until needed. |-| D6 | OpenAI | Substrate → core library (always-on); driver → always-on built-in plugin, no special case | Substrate is shared infra (diamond-dep otherwise). Substrate *extraction into a named module* deferred to externalization (no in-tree payoff). |-| D7 | Temporal config | Schema stays typed in core; only implementation moves | Keeps static typing + `make tb`; config seam becomes a near-no-op. Generic third-party config namespace deferred. |-| D8 | SPI | Two published, versioned SPIs sized to measured imports; sanction `runtime_bridge.primitives.*`; widen to cover Mistral's real needs | A designed surface beats an honor rule that's already breached. |-| D9 | Versioning | `PLUGIN_API_VERSION: int`, plugins declare `targets_api`, mismatch fails loud | Make breaks visible (repo policy). Semver ranges deferred. |-| D10 | Packaging | In-tree-but-registerable default; externalize Temporal; flip Mistral to discovery; vendor inference dists deferred | Most value from the seam alone; split only where cadence/weight justifies. |-------## 14. Test strategy--Each rollout phase lands green; tests are written with the change (red-green), not after. Most of the suite already exists (the integrations work today) — the new tests target the *seam* and the *invariants the seam must not break*. Trivial pass-throughs (a factory that just wraps a client) are not worth bespoke tests; the contract and the invariants are.--### 14.1 Contract conformance (core, protocol-level)--Kept in core so any plugin — built-in or external — can be checked against the contract:--- every `BUILTIN_PLUGINS` entry satisfies the `PipelexPlugin` protocol and declares a compatible `targets_api`;-- a plugin declaring an incompatible `targets_api` fails loud at load (named);-- duplicate `(family, sdk)` / duplicate `execution_mode` / double-claimed hub slot each raise, naming both contributors (§5.3 conflict policy);-- a discovered entry point that fails to import or whose `register` raises is surfaced with context, not skipped (§5.5).--### 14.2 Registry round-trip and DIRECT parity--- each worker family: register → `lookup(family, sdk)` → `make_worker` returns the right `*WorkerAbstract`, for a representative built-in;-- the bridge dispatches by `execution_mode` to a fake registered orchestrator (proves the lookup replaced the `match` with no behavior change);-- `DIRECT` routed through the registry produces byte-identical output to the pre-refactor `_run_direct` for a sample pipe.--### 14.3 Correctness invariants (the silent-bug guards)--- **Import-light boot** (§6.5): booting with the built-ins registered imports no optional SDK — assert `anthropic`, `mistralai`, `google.genai`, `boto3`, … are absent from `sys.modules` after boot. This is the guard against a careless refactor regressing lazy-load.-- **DIRECT router scoping** (§8.1): a `DIRECT` run inside a Temporal-enabled hub keeps nested controller sub-pipes in-process (they resolve the scoped `PipeRouter`, not the Temporal default). Pins the one correctness landmine in the orchestrator extraction.-- **Core defaults unconditional** (§5.3): boot in a stripped environment still resolves `DIRECT` and every core-dep backend (extends the existing `make tb` boot test).--### 14.4 Error parity--A representative missing-extra path (e.g. an Anthropic model with the `anthropic` extra absent) raises the *same* friendly `MissingDependencyError` text — lib + extra + alternative — proving the message survived the move into `make_worker` (preserves the security-perimeter habit of asserting the user-facing message, not just the type).--### 14.5 Temporal: green in-tree before extraction--The full Temporal suite (unit + integration, with the `temporal` marker and `--temporal-server`) must pass with Temporal **wired through the seam but still in-repo** before the package is lifted. Extraction then re-runs the relocated suite against the published base. This is the checkpoint that proves the seam before the packaging move.--## 15. Risks and mitigations--- **Registration timing** — anything used before it's registered breaks. Mitigation: single ordered discovery point after config, before first use; core defaults unconditional; a boot test (`make tb`) that asserts `DIRECT` + every core-dep backend resolve.-- **Silently dropping a core backend** — a misconfigured env must not lose openai/gateway/etc. Mitigation: built-in static list (not entry-point-discovered), registered unconditionally; conformance test asserting presence.-- **Over-fitting the contract to Temporal** — Temporal exercises every hook; tempting to model the protocol on it. Mitigation: design the registrar against *both* Temporal (heaviest) and Mistral (per-call only); Mistral is the standing counter-example.-- **SPI under-sizing** — if the orchestrator SPI omits something Mistral genuinely needs, the boundary stays breached. Mitigation: size it to the *measured* import set (§9.2), not a guess; treat any out-of-SPI import as a tracked design bug.-- **Cross-plugin coupling on extraction** — `pipelex-mistralai-workflows`'s `temporal` extra imports `pipelex.temporal.*`; extracting Temporal to `pipelex-temporal` breaks those imports. Mitigation: in the Temporal-extraction phase, repoint Mistral's temporal extra/imports from `pipelex[temporal]` / `pipelex.temporal.*` to `pipelex-temporal`; this is the known downstream-consumer breakage already tracked.-- **Error-quality regression** — moving the `find_spec` guard into factories could dull today's helpful messages. Mitigation: move the messages verbatim; a test asserts the friendly text (lib + extra + alternative) for a representative missing extra.-------## 16. Rollout shape (detail belongs to the plan)--A natural ordering, each step provable by a green suite before the next. The implementation plan expands these into phases with checkpoints.--1. **Rename** (D1) — `Plugin`/`PluginManager`/`PluginSdkRegistry` → new names. Pure rename, unblocks clear vocabulary.-2. **The seam** — `PipelexPlugin` protocol, `PluginRegistrar`, the three registry shapes, discovery (static + entry points), version check, fail-loud. No behavior change.-3. **Inference through the seam** (in-tree) — `InferenceBackendRegistry` + `InferenceBackendFactory`; every `match` arm (OpenAI driver included, no special case) becomes a registration + factory; built-in drivers in the static list. Suite green with all backends in-repo but selected via the registry.-4. **Orchestrators through the seam** (in-tree) — `OrchestratorRegistry` + exclusive hub slots; `DIRECT` registered in core; the bridge `match` collapses to a lookup; Temporal's boot/teardown/CLI/modes move behind its plugin (still in-repo); Mistral's hard import becomes entry-point discovery. Publish the orchestrator SPI; resolve Mistral's boundary against it.-5. **Externalize Temporal** — move `pipelex/temporal/` + tests + marker + CLI option to `pipelex-temporal`; entry point; flip downstream pins; repoint Mistral's temporal extra. Keep config schema in core.--Vendor inference dists and the OpenAI-substrate extraction are out of this rollout — they happen if/when a specific externalization is justified.--**Cross-cutting deliverables.** The published SPI reference (module/symbol list with import examples, §9) lands with the phase that publishes each SPI. A plugin-authoring guide and a minimal example plugin — the "hello world" for plugin authors (§5.6) — land once the seam exists (after the inference phase, since a backend plugin is the simplest example) and are kept current as orchestrators land. These are deliverables, not follow-ups.-------## 17. Open questions for the plan--- **SPI delivery shape** (§9.3): documented module/symbol list only, or also a thin curated import surface? Resolve without violating the no-re-export rule.-- **Registrar config access timing**: confirm the single `register` call point sees a fully-resolved `ConfigRoot` for every plugin's gating needs (Temporal's `is_enabled`, and the `temporal_enabled` boot override in `Pipelex.make`).-- **`InferenceBackendFactory` granularity**: one factory object per `(family, sdk)`, or one per vendor that switches on family internally? (Leaning per-`(family, sdk)` for uniform entries — confirm against the messier arms like huggingface's provider-literal handling.)-- **Where the committed docs land** (§5.6, §16): the plugin-authoring guide + SPI reference are deliverables — the open question is placement (a dedicated user-facing `docs/.../plugins/` section vs `docs/under-the-hood/`), given the authoring guide is user-facing while the seam internals are under-the-hood.diff --git a/wip/plugins/execute-per-request-execution-mode.md b/wip/plugins/execute-per-request-execution-mode.mddeleted file mode 100644index 5f6348be4..000000000--- a/wip/plugins/execute-per-request-execution-mode.md+++ /dev/null@@ -1,128 +0,0 @@-# Plan — `/execute` honors per-request `execution_mode` (extend F1)--> **⚠️ SUPERSEDED (2026-06-22) by [`orchestration-mode-and-delivery-split.md`](orchestration-mode-and-delivery-split.md).** This plan shipped a working surface on top of the **flat** `PipelexExecutionMode` enum. The orchestration-mode/delivery split (now DONE across all repos, incl. Phase 4 — the `pipelex-mistralai-workflows` migration) **removed that flatness**: `PipelexExecutionMode` is gone, replaced by an open `orchestration_mode` string token (the backend) + an endpoint-set `DeliveryMode` (the wait-semantics). The `/execute` change this plan made still stands conceptually (per-call orchestration via the `OrchestratorRegistry`), but its wire field is now `orchestration_mode` and the f&f-rejection workaround it relied on (`FIRE_AND_FORGET_NOT_SUPPORTED`) is deleted — delivery is no longer a requestable mode value. Read the split plan for the current surface; this doc is kept for history.--**Status:** **Phase E0 DONE — PR #27 MERGE-READY.** Implemented in `pipelex-api` on `feature/Execute-per-request-mode` (off the validate tip `72c0efc`); **PR [#27](https://github.com/Pipelex/pipelex-api/pull/27) → `feature/Orchestrator-dispatched-validate`**. CI green (Lint/Tests/doc-check across 3.11–3.13); review bots satisfied — **Greptile 5/5 "safe to merge"**, Codex's findings addressed (the OpenAPI `execution_mode`-advertisement gap fixed; the dangling-`$defs`-ref + error-wrapping items deferred as pre-existing/cross-endpoint), no open threads. gstack pre-landing `/review`: **merge-ready, no blockers.** All gates green: `make agent-check`, `make agent-test`/`make check` (openapi-check + pylint 10/10). PR head `191c900`. As-built in §"As-built (Phase E0)"; deferred/flagged items (incl. review findings #5–#9) in `execute-per-request-mode-deferred.md`. **Not merged** (targets a feature branch; user's call). The git-SHA `pipelex` pin (`c60be1f`) reverts to the PyPI `==<ver>` pin once core ships. **Plan 2 of 2** — its sibling `wip/plugins/orchestrator-dispatched-validate.md` is now **DONE** (V0–V3 implemented + committed across all three repos: core PR **#998** vs `refactor/Plugins-4` with CI green + Greptile happy; `pipelex-api` tip `72c0efc` and `pipelex-temporal` tip `459e04d`, both on `feature/Orchestrator-dispatched-validate`, pushed). This plan **extends locked decision F1** of the orchestrator-agnostic-runner effort (tracker: `_plugins/TODOS.md`; parent plan: `_plugins/wip/plugins/orchestrator-agnostic-runner-and-flavors.md`).--> **START HERE (cold start).** Read this whole doc. This is a **single-repo change in `pipelex-api`** — no core (`pipelex`) and no `pipelex-temporal` change. It converts `POST /execute` from the boot-global pipe-run hub slot to the **per-call `OrchestratorRegistry`** (the same seam `/start` already uses), so `/execute` honors a per-request `execution_mode` (policy-gated), exactly like `/start`. The change reuses the **existing** orchestrator seam — the validate work added a *parallel* `BundleValidatorRegistry` and never touched the `OrchestratorRegistry` — so it is logically independent of the validate plan. **But both edit `pipeline.py`**, so branch off the now-pushed validate tip (`72c0efc`, see §"Cross-repo state") rather than the older Phase-C base `a39841e`, to stack cleanly and avoid a `pipeline.py` merge. The execute change is still its own commit (the plan's "don't bundle" intent holds — stacking ≠ bundling). Begin on the branch named in §"Cross-repo state".--> **⚠️ Anchors below were refreshed against the post-validate `pipeline.py` (tip `72c0efc`).** If you instead branch off the clean `a39841e`, the validate work is absent there: `ApiRunner.validate` (the `@override`) still exists at the old line, `validate_verdict` does not, and the line numbers are the pre-validate ones — re-derive them. Prefer the stacked tip.-------## 1. The ask + locked decisions--**The ask (user):** make `/execute` honor a per-request `execution_mode`, for consistency with `/start`. Today `/start` resolves the mode per request through the `OrchestratorRegistry`, but `/execute` runs through the **boot-global** `get_pipe_run()` hub slot and ignores any requested mode. Unify them.--**Locked decisions:**--- **Convert `/execute` to the per-call `OrchestratorRegistry`** (the `/start` family), so `execution_mode` (not `boot_orchestrator`) is the single source of truth for top-level dispatch backend. `boot_orchestrator` narrows to its real job — the *execution stack* used wherever the pipe actually runs (worker-side, and the in-process scoping inside `DirectOrchestrator`).-- **`/execute` honors a per-request override**, gated by the existing deployment policy (`allow_request_execution_mode_override`), **symmetric with `/start`**. The user chose this explicitly over keeping the asymmetry.-- **Reject fire-and-forget on `/execute`.** `/execute` is synchronous (it returns the full output); `temporal_fire_and_forget` is meaningless for it. Resolve the mode, then refuse a fire-and-forget resolution with a clear 4xx ("`/execute` is synchronous; use `/start` for fire-and-forget"). (Alternative — coerce f&f→blocking — is listed in §"Open sub-decisions"; default is reject.)-- **Keep it a separate change from `/validate`.** `/validate` adds a *new* seam; `/execute` *reuses* the existing one. They share only `resolve_execution_mode` (already shipped). Do not bundle them.--**Why this is the cleaner direction (context for the cold session):** the asymmetry is *incidental*, not principled. Phase C had to move `/start` to the registry (it needs per-request mode, the override policy, fire-and-forget, and a `workflow_id` return). `/execute` was *already* orchestrator-agnostic through the boot slot (`get_pipe_run()` returns whatever's installed; the base imports no `temporalio`), so Phase C had no forcing reason to touch it. This plan finishes that unification. The boot slots do **not** go away — they remain the execution stack on workers and for in-process runs; this only changes how the top-level `/execute` *entry* selects a backend.-------## 2. Current state--All anchors in `pipelex-api/api/routes/pipelex/pipeline.py` unless noted.--- **`/execute` route** (`370-388`): `async def execute(request)` (`370`) calls `_parse_request(request)` and **discards the extras** — line `376`: `run_request, _extras = await _parse_request(request)` — then calls `runner.execute(...)` **without** any `requested_execution_mode`. So the wire **already parses** `PipelineApiExtras.execution_mode`; `/execute` simply throws it away.-- **`ApiRunner.execute` is NOT overridden.** `ApiRunner(PipelexMTHDSProtocol)` (line `67`) overrides `start` (`@override`, `82`) and defines `validate_verdict` (`185`, **not** an `@override`) but **not** `execute` — so `/execute` falls through to the base `PipelexMTHDSProtocol.execute` in core `_plugins/pipelex/pipeline/runner.py:112`. (Note: the validate work **dropped** the old `ApiRunner.validate` `@override` in favor of `validate_verdict`; `ApiRunner` now inherits the base `validate`. That does not affect this plan — `execute` is still un-overridden and still falls through to the boot slot.)-- **The base `execute`** (core `runner.py:112` ff.) does `pipeline_run_setup(...)` → `effective_pipe_run = self._pipe_run or get_pipe_run()` (`205`) → `pipe_output = await effective_pipe_run.run(pipe_job, delivery_assignment=…)` (`206`) → wraps `pipe_output` into `PipelexRunResultExecute`. `get_pipe_run()` is the **boot-global** hub slot (Temporal claims it only when `boot_orchestrator == "temporal"`; else in-process). No `resolve_execution_mode`, no registry.-- **`/start` (the model to mirror)** — `ApiRunner.start` (`82` ff.): `resolve_execution_mode(requested_execution_mode, config=get_api_config())` (`125`) → `pipeline_run_setup(...)` (rich `PipeJob`) → `orchestrator = get_orchestrator_registry().get_optional(mode=execution_mode)` (`173`) → `MissingOrchestratorError` if absent (`175`) → `run_output = await orchestrator.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)` (`176`) → returns `PipelexRunResultStart(..., workflow_id=run_output.workflow_id)`. The `/start` route passes `requested_execution_mode=extras.execution_mode` (`439`). The `/validate` route (now `validate.py`, post-validate work) is a **second** in-file example of threading `execution_mode` through `resolve_execution_mode` — but it uses the typed `ValidateRequest` model, whereas `/execute` mirrors `/start`'s raw-body `_parse_request` + `PipelineApiExtras.execution_mode` path.-- **`resolve_execution_mode`** — `pipelex-api/api/api_config.py:81-99`: `resolve_execution_mode(requested, *, config) -> PipelexExecutionMode`. Deployment default wins unless the caller supplied a *different* mode **and** `config.allow_request_execution_mode_override` is true; a forbidden override raises **403** (`EXECUTION_MODE_OVERRIDE_FORBIDDEN`). `ApiConfig` (`40-53`) + `api.toml` (base: `execution_mode = "direct"`, `allow_request_execution_mode_override = false`).-- **The orchestrators already return the full output.** `_plugins/pipelex/runtime_bridge/direct_orchestrator.py`: `DirectOrchestrator.run` returns `serialize_completed_output(pipe_output=…, workflow_id=None)`. `/start` only reads `.workflow_id`; `/execute` will read the **output** off the same `PipelexPipeRunOutput`. `PipelexExecutionMode`: `DIRECT`, `TEMPORAL_BLOCKING`, `TEMPORAL_FIRE_AND_FORGET`, `MISTRAL_NATIVE` (`_plugins/pipelex/runtime_bridge/execution_mode.py:4-27`).--For full orientation on the two dispatch seams (per-call registry vs boot-global hub slots), see §3 "Background" of the sibling plan `wip/plugins/orchestrator-dispatched-validate.md`.-------## 3. The design (locked)--A **single-repo override in `pipelex-api`**, mirroring `ApiRunner.start`. No core or `pipelex-temporal` change — the `DirectOrchestrator` / `TemporalBlockingOrchestrator` / `MistralNativeOrchestrator` in the registry already do everything needed.--- **Override `ApiRunner.execute`** (`@override`, alongside `start` and `validate`) with a `requested_execution_mode: PipelexExecutionMode | None = None` parameter. Body mirrors `start`:- - `execution_mode = resolve_execution_mode(requested_execution_mode, config=get_api_config())` — **first**, so a forbidden per-request override 403s before any library load / run setup (exactly like `start`).- - **Reject fire-and-forget:** if `execution_mode` resolves to `TEMPORAL_FIRE_AND_FORGET` (use a `match`/`is_*` on the enum — do not `==`-compare per the StrEnum standard), refuse with a 4xx (`/execute` is synchronous). See §"Open sub-decisions" for the coerce-to-blocking alternative.- - `pipe_job, resolved_pipeline_run_id, _ = await pipeline_run_setup(...)` — build the rich `PipeJob` as `start` does.- - `orchestrator = get_orchestrator_registry().get_optional(mode=execution_mode)`; `MissingOrchestratorError(mode=...)` if `None`.- - `run_output = await orchestrator.run(pipe_job=pipe_job, delivery_assignment=None)` — `/execute` is synchronous, no delivery webhooks.- - **Map `run_output` (`PipelexPipeRunOutput`) → `PipelexRunResultExecute`** (the protocol's execute result wrapping the full pipe output). This mapping is the one integration detail to verify (the orchestrator returns a *serialized* `PipelexPipeRunOutput` via `serialize_completed_output`, whereas the base `execute` wraps the raw `pipe_output`); confirm `PipelexPipeRunOutput` carries everything `PipelexRunResultExecute` needs and write the conversion. **This is the "clean vs more-complex-than-expected" hinge** — check it early.-- **Route** (`execute`, `340-373`): stop discarding extras. Parse like `/start` does (`run_request, extras = await _parse_request(request)`), then pass `requested_execution_mode=extras.execution_mode` into `runner.execute(...)`. No new wire field — `/execute` simply starts honoring the `execution_mode` extra it already parses.-- **Base `execute` stays untouched** in core (`runner.py:112`) — the local/CLI runtime legitimately uses the boot slot (it is not mode-dispatched). Only `ApiRunner` overrides, mirroring how it overrides `start`/`validate`.--**Behavior shift to call out (changelog + docs):** `/execute`'s backend is now selected by `execution_mode` (resolved from `api.toml` + optional policy-gated per-request override), **not** by `boot_orchestrator`. For a correctly-configured deployment these already agree (a Temporal deployment that runs `/start` distributed sets `execution_mode = "temporal_blocking"`), so the real-world delta is ~nil — but it removes the two-knobs-can-silently-disagree footgun and makes `execution_mode` authoritative. The base `api.toml` keeps `allow_request_execution_mode_override = false`, so per-request override only takes effect where a deployment opts in.-------## 4. Phase + checkpoint--Single contained phase (one repo, one commit).--### Phase E0 — `pipelex-api` (branch off the validate tip `72c0efc` on `feature/Orchestrator-dispatched-validate`)--- Add the `ApiRunner.execute` override + fire-and-forget rejection + the `PipelexPipeRunOutput → PipelexRunResultExecute` mapping; rewire the `/execute` route to thread `extras.execution_mode`.-- Tests (see §"Testing").-- Docs: changelog entry (`/execute` now honors `execution_mode`, rejects fire-and-forget); document the dual-knob model (`execution_mode` authoritative for top-level dispatch; `boot_orchestrator` = execution stack) where `/execute`/`execution_mode` is described — likely `pipelex-api/docs/` and, if a section exists, the relevant `docs/specs/` doc (see §"Spec/conformance").-- Gates: `make agent-check`, `make agent-test`.--> **Checkpoint E-A** (the gate for `/execute`): `/execute` resolves and dispatches by `execution_mode`, honors the policy-gated override symmetrically with `/start`, rejects fire-and-forget, all gates green, docs + changelog updated. Clean-context `/code-review` on the `pipelex-api` diff. Capture an as-built (final signature, the output-mapping decision, test evidence).-------## As-built (Phase E0)--**Repo / branch:** `pipelex-api` @ `feature/Execute-per-request-mode` (off validate tip `72c0efc`). Single repo; no core / `pipelex-temporal` change.--**Final shape (`api/routes/pipelex/pipeline.py`):**--- `ApiRunner.execute` is now an `@override` taking `requested_execution_mode: PipelexExecutionMode | None = None` (trailing optional param, LSP-compatible with the base). Body: resolve `execution_mode` FIRST (403 on forbidden override), refuse fire-and-forget with a 400 (`is_fire_and_forget`, not `==`), look up the orchestrator (`MissingOrchestratorError` if absent), then **inject it as `self._pipe_run` and delegate to `super().execute(...)`** — so the base owns the entire run lifecycle (library setup/teardown, tracer close, pipeline-manager cleanup, telemetry, error mapping). The `ApiRunner` is per-request, so mutating `_pipe_run` is request-scoped.-- `_OrchestratorPipeRun(PipeRunProtocol)` — the injected adapter: its `run(pipe_job, *, delivery_assignment)` calls `orchestrator.run(...)` and rehydrates the result.-- `_pipe_output_from_run_output(run_output)` — **the output-mapping decision (the §3 hinge):** the orchestrator returns the JSON-safe `PipelexPipeRunOutput`; this rebuilds the rich `PipeOutput` via `hydrate_working_memory(output_dict)` (same routine the Temporal workers use; runs while the run library is still open) + `PipeOutput.model_validate({...}, strict=False)`. `strict=False` is **required** to reverse the orchestrator's `model_dump(mode="json")` of `graph_spec` (its `created_at: datetime` is `strict=True`; str→datetime is rejected under strict). Confirmed empirically. The base `execute` then wraps this into `PipelexRunResultExecute` exactly as before, so `created_at`/`finished_at`/`state`/telemetry are identical to the old path.-- Route `execute()` now threads `requested_execution_mode=extras.execution_mode` (stopped discarding the parsed extras).-- New `ErrorType.FIRE_AND_FORGET_NOT_SUPPORTED` (400, via `raise_bad_request`).--**Why inject-and-delegate (not a full re-implementation):** the base `execute`'s teardown is correct and battle-tested; duplicating it risks drift, and the orchestrator seam differs from the boot slot only in (a) which backend runs and (b) the serialized vs rich output. The adapter bridges exactly those two; everything else is inherited.--**Test evidence:** `tests/unit/test_execute_dispatch.py` (new, mirrors `test_validate_dispatch.py`): direct dispatch returns the rehydrated full output (real `serialize_completed_output` → `hydrate_working_memory` round-trip), per-request override honored when policy allows, forbidden override → 403, fire-and-forget → 400, missing orchestrator → `MissingOrchestratorError`. Existing `test_pipeline_routes.py` / `test_protocol_conformance.py` unaffected (they mock the runner). Full `make agent-test` green.--**Deferred / flagged:** see `execute-per-request-mode-deferred.md` — DIRECT round-trip cost (accepted), `strict=False` rationale, a pre-existing `/start` DIRECT-path resource leak (flagged, not fixed — out of scope), and the absorbed pre-existing `/validate` OpenAPI + config-doc drift.-------## 5. Cross-repo state, pins, gates--- **Branch:** cut a new branch (e.g. `feature/Execute-per-request-mode`) in `pipelex-api` off `feature/Orchestrator-dispatched-validate` @ `72c0efc` (the now-pushed validate tip), **not** off the clean Phase-C base `a39841e`. Rationale: both changes edit `pipeline.py` (the validate work dropped the `validate` override + added `validate_verdict` + adjusted imports; this adds an `execute` override), so stacking avoids a `pipeline.py` merge. The execute change reuses the existing `OrchestratorRegistry` and depends on **none** of the validate seam's logic, so it stays an independent commit (own PR). This is the only repo touched. *(If you deliberately want full independence from validate, branch off `a39841e` instead and accept a small `pipeline.py` merge when both land — and re-derive the §2 anchors, which are written against the post-validate tip.)*-- **Pins:** `pipelex-api/pyproject.toml` pins `pipelex = { path = "../_plugins", editable = true }`, so the core orchestrators/registry are live without a republish. No `pipelex-temporal` dependency is added (the `TemporalBlockingOrchestrator` is exercised by its own repo's tests; the API change is testable with `DirectOrchestrator` + a stub).-- **Gates:** `make agent-check`, `make agent-test` (in `pipelex-api`). If a `docs/specs/` heading is touched, run `make check-spec-links` (in `conformance/`).-------## 6. Spec / conformance--- **No spec pins `/execute`'s backend-selection today.** The `/validate` verdict spec (`docs/specs/pipelex-mthds-protocol.md`) doesn't cover `/execute`; `/execute` dispatch lives in `pipelex-platform-api.md` / `pipelex-hosted-config.md`, where the HTTP arm is Phase-3-deferred and per-request override is not yet detailed. So this is **low conformance risk** and adds **no new wire field** (the `execution_mode` extra already exists; `/execute` just stops ignoring it).-- **Add documentation** of the new behavior: `/execute` honors a policy-gated per-request `execution_mode` and rejects fire-and-forget; `execution_mode` is authoritative for top-level dispatch while `boot_orchestrator` selects the execution stack. Put it where `/execute` / `execution_mode` is described (a `pipelex-api/docs/` page, and the hosted-config spec if you extend it). If you add or edit a spec heading, wire a `> Verified by:` link to a test and a matching `pytest.mark.spec(...)`, then `make check-spec-links`.-- **Conformance:** there is **no** `/execute` conformance module yet (deferred with the HTTP arm). Don't invent the deferred HTTP conformance arm here; cover the behavior with `pipelex-api`'s own tests (§"Testing"). If you later add a conformance surface for `/execute`'s mode-selection, link it bidirectionally.-------## 7. Testing--In `pipelex-api` (no Temporal needed):--- **Direct dispatch:** `/execute` with `execution_mode` unset (or `direct`) runs in-process via `DirectOrchestrator` and returns the full output — equivalent result to today's boot-slot path on the direct base.-- **Per-request override, policy on:** with `allow_request_execution_mode_override = true`, a requested mode is honored; with it `false`, a *different* requested mode 403s (`EXECUTION_MODE_OVERRIDE_FORBIDDEN`) — reuse/mirror the existing `/start` override-policy tests.-- **Fire-and-forget rejection:** a resolved `temporal_fire_and_forget` on `/execute` returns the chosen 4xx, not a hang or a silent coerce (unless the coerce alternative is chosen — then assert blocking semantics).-- **Missing orchestrator:** a mode with no registered orchestrator raises `MissingOrchestratorError` with the install hint.-- **Output mapping:** assert the `PipelexPipeRunOutput → PipelexRunResultExecute` mapping preserves the full output (the integration hinge from §3). Use `DirectOrchestrator` (or a stub orchestrator returning a known `PipelexPipeRunOutput`).-------## 8. Open sub-decisions for the implementing session--- **Fire-and-forget on `/execute`: reject (default) vs coerce-to-blocking.** Reject is more honest (you asked for something incompatible with a synchronous endpoint). Coerce-to-blocking is more lenient (execute always waits anyway, so f&f could just mean `temporal_blocking` here). Plan defaults to **reject** with a clear 4xx; flip only with a reason.-- **Output mapping shape.** Confirm `PipelexPipeRunOutput` (from `serialize_completed_output`) carries everything `PipelexRunResultExecute` needs. If it doesn't, decide whether to enrich the orchestrator output or to keep a thin execute-specific adapter — and note it in the as-built. This determines whether the change stays "clean" or grows.-- **Deployment-config coherence.** Note for Phase D (hosted Temporal flavor, in the parent plan): the hosted runner's `api.toml execution_mode` must be set coherently (it already is for `/start`); after this change it also governs `/execute`. `boot_orchestrator` remains for the execution stack. No code change here — just a deployment note so the two knobs are set with intent. **Related runtime constraint** (surfaced by the validate work, see `validate-dispatch-requires-boot-active.md`): the shared `WorkflowExecutor` hard-requires `boot_orchestrator == "temporal"` (`is_temporal_boot_active()`) to open its client, so a resolved `temporal_blocking` `/execute` on a runner **not** booted under Temporal would fail the same way a temporal-mode `/start` or `/validate` does. That is pre-existing and out of scope here (the agnostic base tests use `DirectOrchestrator`), but it reinforces why `execution_mode = temporal_*` and `boot_orchestrator = "temporal"` must be set together on a Temporal flavor.diff --git a/wip/plugins/execute-per-request-mode-deferred.md b/wip/plugins/execute-per-request-mode-deferred.mddeleted file mode 100644index ce974ed2c..000000000--- a/wip/plugins/execute-per-request-mode-deferred.md+++ /dev/null@@ -1,68 +0,0 @@-# Deferred / flagged items — `/execute` honors per-request `execution_mode`--Companion to `execute-per-request-execution-mode.md` (the plan). Captures findings surfaced while implementing **Phase E0** that were deliberately **not** acted on, plus pre-existing drift that the implementation absorbed. None block the feature.--## 1. DIRECT dispatch does a serialize→rehydrate round-trip (accepted tradeoff, not a bug)--`/execute` now dispatches every mode — including `direct` — through the `OrchestratorRegistry`. The orchestrator answers with the JSON-safe `PipelexPipeRunOutput` (the same shape that crosses the Temporal worker boundary, via `serialize_completed_output`), so the synchronous `/execute` path rehydrates it back into a rich `PipeOutput` (`_pipe_output_from_run_output` in `api/routes/pipelex/pipeline.py`). For `direct` this means the working memory is `dump_for_temporal()`'d and then `hydrate_working_memory()`'d **in-process** — one redundant re-serialization the old boot-slot path didn't pay.--**Why accepted:** it keeps `/execute` on the *same* per-call dispatch seam as `/start` and `/validate` (no per-mode branch), which is the whole point of the locked decision. The cost is bounded (one extra round-trip of a working memory that `/execute` already serializes once for the HTTP response).--**If it ever matters:** the clean fix is at the *contract* layer, not here — e.g. let `OrchestratorProtocol.run` optionally hand back the rich `PipeOutput` for the in-process arm so the DIRECT path skips the round-trip. That's a core (`pipelex`) change to a shared SPI, out of scope for this single-repo plan. Don't special-case DIRECT inside `pipelex-api` (that reintroduces the per-mode branch the seam removed).--## 2. `graph_spec` / `tokens_usages` reconstruction needs `strict=False` (correct, worth knowing)--`serialize_completed_output` dumps `graph_spec` / `tokens_usages` with `model_dump(mode="json")`, and those models are `strict=True` (e.g. `GraphSpec.created_at: datetime`). A strict re-validation of the JSON dump fails (str→datetime is rejected in strict mode — confirmed empirically). `_pipe_output_from_run_output` therefore validates with `strict=False`, which is the correct tool for reversing our *own* trusted JSON dump (a round-trip, not untrusted ingest). Nothing else in the tree rehydrates these dump fields today, so there was no prior precedent to copy — flagged here so a future reader doesn't "tighten" it back to strict.--## 3. Pre-existing `/start` DIRECT-path resource leak (flagged, NOT fixed — out of scope)--While mirroring `start`, noticed that `ApiRunner.start` calls `pipeline_run_setup(...)` then the orchestrator, with **no** teardown on the success path. `pipeline_run_setup` only tears the run library / pipeline-manager entry down on its *error* path; `PipeRun.run` closes the tracer but not the library or the pipeline-manager registration. The base `PipelexMTHDSProtocol.execute` does that teardown in its `finally`. So on the **DIRECT** `/start` success path (reachable on the agnostic base, where `start` runs in-process and blocks), the run library and its pipeline-manager entry appear to leak per request.--`/execute` does **not** have this problem: this change delegates to the base `execute`, inheriting its full teardown. The finding is purely about `/start`.--**Not fixed here** because it touches `/start` (a different surface) and the plan is scoped to `/execute`. Worth a small follow-up: give `ApiRunner.start` the same lifecycle guarantees (or factor the base `execute`'s setup+teardown into a shared context manager both `execute` and `start` use). Verify first that it is a genuine leak and not cleaned up elsewhere before changing `/start`.--## 4. Absorbed pre-existing drift from the `/validate` work (informational)--The committed `docs/openapi/pipelex-api.openapi.yaml` and `docs/configuration.md` at the validate tip (`72c0efc`) were **stale** w.r.t. the `/validate`-by-`execution_mode` work: the OpenAPI artifact lacked the `PipelexExecutionMode` component + the validate request's `execution_mode` field, and `configuration.md` still said `/validate` "always runs in-process and ignores this setting." Both are regenerated/corrected in this change (the OpenAPI artifact is generated, so it must match the current app exactly; the config doc was rewritten holistically for `/execute` + `/start` + `/validate`). Mentioned so the diff's validate-flavored hunks aren't mistaken for execute-scope creep — they are pre-existing drift this change necessarily resolves.--## 5. Reverse helper location — altitude (independent review, deferred)--`_pipe_output_from_run_output` is the exact inverse of `serialize_completed_output` (which lives in `pipelex/runtime_bridge/serialization.py`), but the reverse currently lives as a private function in the `pipelex-api` route module. If `PipelexPipeRunOutput` gains/renames a field, forward (pipelex) and reverse (pipelex-api) could drift independently. **Deferred:** single consumer; moving it crosses a repo boundary; and the reverse must run *inside* the still-open run-library scope (a precondition naturally satisfied at this call site). Confirmed no existing reverse helper is being reinvented (`rehydrate_pipe_output_with_crate` operates on a `PipeOutput.working_memory_raw`, not a `PipelexPipeRunOutput`; the new helper correctly reuses `hydrate_working_memory`). Promote to a `deserialize_completed_output` in `serialization.py` only if `pipelex-temporal`'s blocking path ever needs the same reverse.--## 6. OpenAPI request schema doesn't advertise the `execution_mode` body extra--`/execute`, `/start`, and `/validate` all honor a per-request `execution_mode` override, but the override is parsed from the raw kajson body as a `PipelineApiExtras` extra, so the *generated* OpenAPI request schema only lists it where the route uses a typed body model.--**`/execute` half — RESOLVED (PR #27 review fix).** `/execute`'s `openapi_extra` now documents a new `PipelexApiExecuteRequest(RunRequest)` schema (mirroring the existing `PipelexApiStartRequest` pattern) that declares `execution_mode`, reusing the shared `_EXECUTION_MODE_DESCRIPTION` already on `PipelineApiExtras.execution_mode` (factored to a module constant so the two can't drift). The committed OpenAPI artifact was regenerated; the drift check (`scripts/export_openapi.py --check`) is green. Greptile (P1) / Codex (P2) flagged exactly this gap on `/execute` — the route reads `extras.execution_mode` and honors it, but the published schema (`RunRequest.model_json_schema()`) omitted the field.--**Dangling `#/$defs/` ref for the advertised enum — DEFERRED (pre-existing artifact-generation pattern).** A follow-up Codex pass (P2) noted that `/execute`'s newly-advertised `execution_mode` references `#/$defs/PipelexExecutionMode`, which a root-based OpenAPI resolver can't dereference (the document has no top-level `$defs`; the enum also exists at `#/components/schemas/PipelexExecutionMode`). This is a **pre-existing pattern**, not introduced here: both `/execute` and `/start` already emit dangling `#/$defs/StuffContentAbstract` and `#/$defs/WorkingMemoryAbstract_Any_` refs, because both parse the body through the raw `Request` and supply their schema via `openapi_extra: <Model>.model_json_schema()` — FastAPI hoists nested defs into `components/schemas` only for *typed-body* routes (like `/validate`), not for a manually-supplied schema dict. My `execution_mode` ref is consistent with the existing dangling refs on the same endpoint. Fixing only `execution_mode` would leave it inconsistent with its siblings and still not make the endpoint's schema resolvable; the clean fix is a cross-endpoint OpenAPI-generation improvement (hoist each `openapi_extra` schema's `$defs` into `components/schemas` and rewrite the refs, or move `/execute` + `/start` to typed bodies) — a separate change spanning both endpoints, out of scope for this PR. Tracked here.--**`/start` half — STILL DEFERRED (scope).** `/start`'s `PipelexApiStartRequest` schema *also* omits `execution_mode` (it extends `RunRequest` and only redeclares `callback_urls`; the field lives on `PipelineApiExtras`). That gap is **pre-existing** — `/start`'s `execution_mode` consumption was introduced by the F1 work, not by PR #27, whose diff touches only `/execute`. Not fixed here to honor scope discipline. The clean fix is the same one-liner pattern: have `PipelexApiStartRequest` redeclare `execution_mode = Field(default=None, description=_EXECUTION_MODE_DESCRIPTION)`, then regenerate the artifact. Note: the published `make openapi-export` target does not exist in `pipelex-api`'s Makefile; regenerate by running `.venv/bin/python scripts/export_openapi.py docs/openapi/pipelex-api.openapi.yaml` (write) and `--check` to verify. Never hand-edit the yaml.--## 7. `is_completed` invariant the synchronous path relies on (independent review, rejected-as-fix)--`_pipe_output_from_run_output` unconditionally builds a completed `PipeOutput` without checking `run_output.is_completed`. **Rejected as a fix** (not a bug): `/execute` refuses fire-and-forget upfront with a 400, and every non-f&f orchestrator today (`DirectOrchestrator`, `TemporalBlockingOrchestrator`) returns `is_completed=True`, so a `False` here is unreachable — guarding it is a defensive check for an impossible-today scenario. Recorded only so a future orchestrator author knows the invariant the synchronous path assumes.--## 8. Worker-dispatched (`temporal_blocking`) rehydration is unexercised here (pre-landing review, INVESTIGATE — out of scope)--The adversarial pre-landing review traced the `temporal_blocking` arm: a worker runs the pipe and returns a `PipelexPipeRunOutput` whose `output_dict` carries `__pipelex_class__` markers for any dynamic concept classes registered **on the worker**. `_pipe_output_from_run_output` then calls `hydrate_working_memory` against the **API-side** run library's class registry. This resolves correctly *because* `pipeline_run_setup` already loaded the same bundle API-side (the rich `PipeJob` is built locally before dispatch, registering the same dynamic concept classes) — the same property `/start` and `/validate` already rely on for their worker-dispatched arms. So `/execute` introduces **no new** boundary risk: it reuses the shared `PipelexPipeRunOutput` contract and the same `hydrate_working_memory` round-trip.--**INVESTIGATE, not fixed:** this path cannot be exercised on the orchestrator-agnostic base (no `pipelex-temporal` installed; `test_execute_dispatch.py` covers DIRECT only, with a stub orchestrator that round-trips through the real `serialize_completed_output` → `hydrate_working_memory`). End-to-end coverage of the `temporal_blocking` `/execute` rehydration belongs in `pipelex-temporal`'s own test suite (where the blocking orchestrator and a real worker exist), alongside the analogous `/start` / `/validate` worker-path tests. Out of scope for this single-repo `pipelex-api` PR. Verify there when the hosted-Temporal flavor lands; reinforced by the runtime constraint in `validate-dispatch-requires-boot-active.md` (a `temporal_*` `/execute` on a runner not booted under Temporal fails the same way a temporal-mode `/start`/`/validate` does).--## 9. Failing-pipe `/execute` now yields a double-wrapped error message + unwound `pipe_stack` (pre-landing review, DEFERRED — seam tradeoff, cross-endpoint)--The adversarial pre-landing review traced how a *failing* pipe surfaces differently after the dispatch change. Verified against source:--- **Old `/execute`** ran the pipe through the boot slot (`get_pipe_run()` → `PipeRun.run`), which observe-and-reraises the original `PipeRouterError` (`runner.py` core, `pipe_run.py: raise execution_error`). The base `execute`'s `except PipeRouterError` arm then built `PipelineExecutionError(message=exc.message, pipe_stack=exc.pipe_stack)` — the raw router message and the **failure-site** pipe-stack snapshot.-- **New `/execute`** runs through `DirectOrchestrator.run`, which catches `PIPE_DISPATCH_ERRORS` and re-raises `PipelexBridgeDispatchError("Pipe execution failed in DIRECT mode for pipe 'X': <original>")` (`direct_orchestrator.py`). That no longer matches `except PipeRouterError`, so it falls to the base's `except PipelexError` arm → `PipelineExecutionError(message=exc.message, pipe_stack=pipe_job.pipe_run_params.pipe_stack)`. Net: the operator-facing **message gains the "in DIRECT mode" wrapper** and the **`pipe_stack` is the unwound live stack**, not the failure-site snapshot.--**Not a wire-status / classification regression.** `error_domain` / `http_status` / `error_category` survive because `PipelexBridgeDispatchError.to_error_report()` enriches from the full `__cause__` chain, and `pipe_stack` is not an `ErrorReport` wire field. So a machine consumer sees the same `is_valid`/status/error-domain; only the human-readable `message` wording and log-stack fidelity change.--**DEFERRED, not fixed** — it is a property of the **orchestrator seam, not of `/execute`**: the seam deliberately funnels every mode's pipe failure through the single `PipelexBridgeDispatchError` type so a host catches one error regardless of backend. `/start` (also routed through `DirectOrchestrator`) and the dispatched `/validate` already exhibit the identical wrapping (the agent's symmetric finding: `/start` and `/execute` now render different `message`/`title` for the same underlying pipe failure than the legacy boot-slot path did). Harmonizing the wording — or restoring the failure-site `pipe_stack` for the DIRECT arm — is a change to the **shared seam spanning all three surfaces** (e.g. have `DirectOrchestrator` re-raise the original `PipeRouterError` so the base's precise arm still fires, or teach the base `except` to unwrap a `PipelexBridgeDispatchError` cause), not a `pipelex-api`-local `/execute` edit. Out of scope for this PR; no shipped Temporal flavor means no prod error-shape contract is at risk. Worth a small follow-up if product wants byte-identical error bodies across `/execute` / `/start` and parity with the legacy message.--### Minor (independent review, no action)--- `_validate_extras` reports a malformed `execution_mode` value under `error_type=INVALID_CALLBACK_URLS` (still a correct 422 with the real pydantic detail in the message) — pre-existing label imprecision surfaced only because `execution_mode` joined that model; cosmetic.-- The base `execute`'s `if extra:` rejection is unreachable via the API `execute` override (the override omits `extra`; the route never passes it) — dead-but-harmless, identical to how `start()` already works.-- `ErrorType.FIRE_AND_FORGET_NOT_SUPPORTED` sits just above the `# Request validation` section header in `api/error_types.py` rather than under it (it is a 400 request-shape refusal). Purely cosmetic enum ordering; not worth a diff.diff --git a/wip/plugins/implementation-plan.md b/wip/plugins/implementation-plan.mddeleted file mode 100644index 08e022ba7..000000000--- a/wip/plugins/implementation-plan.md+++ /dev/null@@ -1,295 +0,0 @@-# Pipelex plugin system — implementation plan--**Status:** plan (proposed, not started) — reviewed via `/plan-eng-review` + codex outside voice; decisions D1–D6 locked (see below).-**Built from:** [`design.md`](design.md) — the single decided design. This plan expands design §16's rollout into executable phases with grounded `file:line` references, per-phase red-green tests, and checkpoints, and resolves the §17 open questions. Read `design.md` first; this is the *how*. Where this plan diverges from the design (deferred slot-claim thunks; a fifth seam; config relocation), the divergence is a review finding folded in, called out inline.--## 0. Orientation--**Why now (not speculative generality).** Both families are already ~80–90% decoupled; this inverts the *last* coupling. The justification is present pain, not hypothetical third-party authors: `pipelex-mistralai-workflows` is *already* an external repo whose native tier *already* breaches its declared import boundary (it reaches into `pipelex.hub`, `pipe_run`, `core`, `graph`, `tracing`), held together by a pinned-rev coupling — the orchestrator SPI is the real fix. And Temporal's heavy SDK + independent release cadence + the fact that downstream (`pipelex-worker`, `pipelex-api-hosted`) *already* pin it separately mean externalization pays for itself. The third-party-author story is a bonus the seam enables for free.--**The seams, located.** The design names four inference seams; the review found a fifth (`model_lists.py`). Core still names integrations at each:--| Seam | File | What it does today |-|---|---|---|-| Inference dispatch (×4) | `cogt/{llm,img_gen,extract,search}/*_worker_factory.py` | `match plugin.sdk:` → per-arm lazy import + `find_spec` guard + SDK-client cache + build worker |-| **Model listing (the 5th seam)** | `cogt/model_backends/model_lists.py:58` | `match sdk:` → hardcoded imports of `pipelex.plugins.{openai,anthropic,…}.*_list` (the `list-models` CLI path) — **missed by the design; codex finding** |-| Orchestrator dispatch | `runtime_bridge/bridge.py:147-158` | `match execution_mode:` → hard-imports Mistral (`:414`), lazy-imports Temporal; per-mode install messages at `:399/:417` |-| Boot/teardown hub swap | `pipelex.py:377-479` | four `if get_config().temporal.is_enabled:` blocks + inlined teardown; explicit `content_generator`/`pipe_router` params override at `:376/:450` |-| Config | `system/configuration/configs.py:14,250` | the one hard import of `pipelex.temporal.config_temporal` |-| CLI | `cli/_cli.py:15,20,234,238` | two hard imports registering `worker` + `setup-temporal-namespace`; `PipelexCLI.list_commands` hardcodes order |-| Naming overload | `plugins/{plugin,plugin_manager,plugin_sdk_registry}.py`, `hub.py:289-293` | `Plugin` value object, `PluginManager`/`PluginSdkRegistry` cache, the `PluginManager2 is not initialized` artifact |--| Phase | Scope | PR |-|---|---|---|-| 0 — Rename | `pipelex` only | own PR |-| 1 — Seam + LLM family | `pipelex` only | own PR (merged per D2) |-| 2 — Remaining inference families | `pipelex` only | own PR(s) per family |-| 3 — Orchestrators through the seam | `pipelex` only | own PR |-| 4 — Invert model listing (5th seam) | `pipelex` only | own PR (D6) |-| 5 — Externalize Temporal | cross-repo | separate effort, gated |--## Decisions locked in this review--| # | Decision | Choice |-|---|---|---|-| D1 | Inference backend factory shape | **Typed callable** (`MakeWorkerFn` TypeAlias), not a single-method `Protocol` object — kills ~30 one-method classes |-| D2 | Phase 1 PR boundary | **Merge** the seam machinery with the LLM family so the contract meets a real backend before it locks |-| D3 | Plugin CLI command harvesting | **Run the pure `build_registrar` at CLI-build** (after loading config); same function runs again at boot |-| D4 | `SdkClientManager` wrapper | **Keep as pure rename**; collapse into the registry deferred to a P3 follow-up commit |-| D5 | Hub slot claims | **Deferred thunks** — `claim_*(factory: Callable[[], Impl])`, invoked only at the setup apply-point, so `register` stays import-light (codex finding) |-| D6 | The 5th seam (`model_lists.py`) | **Invert it too**, as a follow-on phase, with a model-listing capability on the inference contract (codex finding) |-| D7 | Plugin control surface | Discovery is the source of truth for *presence*; add `pipelex plugins list` (observability) + a narrow `plugins.disabled` denylist escape hatch; **no** redundant allowlist (model config already gates inference) |--## Plugin enablement model--No central "plugins on/off" list exists, by design. Enablement is two separate things:--- **Presence (is the plugin discovered?) — not config.** A plugin is present iff it is in the static `BUILTIN_PLUGINS` list (core) or is an installed dist declaring the `pipelex.plugins` entry point. Installing enables presence; uninstalling removes it. Zero config. `build_registrar` walks both sources at boot and calls each plugin's `register(registrar)` — that call *is* the plug.-- **Activation (is its contribution used?) — decomposed by registry shape, never a generic flag:**- - *Inference backends* activate by **data**: a model in the backend config names the `sdk`. The plugin registers its `(family, sdk)` entry unconditionally and import-light; it runs only when a model selects it. The model/backend config (`get_models_manager()`) is the de-facto central enable surface for inference — a per-plugin toggle would duplicate it and drift.- - *Orchestrators, per-call* select by `execution_mode` per run.- - *Orchestrators, boot-global* are claimed only when the plugin's own config slice says so. **`temporal.is_enabled` is not a generic "Temporal plugin on" flag** — it means "boot *this process* as a Temporal-default runtime" (swap the hub content-generator/router/run to the Temporal ones). A worker process sets it; a runner does not. The name misleads; treat it as a per-process runtime-mode gate, not plugin enablement. Renaming it is out of scope (touches the live config) but flagged.--**Control surface (D7) — discovery stays the source of truth for presence; on top of it:**--- `pipelex plugins list` — a core (not plugin-contributed) command: prints every discovered plugin (built-in vs external), what each registered (backends / modes / slots / CLI), and whether it's denylisted. The answer to "what's plugged in?"-- `plugins.disabled: list[str]` — a denylist escape hatch in a core-owned `plugins` config section. A discovered plugin whose `name` is listed is skipped in `build_registrar`, **with a log line (never silent)**, without uninstalling it — for testing and quarantining a misbehaving external. A flat list of *names*, **not** the deferred *generic per-plugin typed-config namespace* (design §10), which stays out of scope.-- **The denylist cannot disable a core-unconditional plugin** (the DIRECT orchestrator, or a core-dependency backend: openai/gateway/portkey/pypdfium2/azure_rest). Listing one is a startup error — you cannot boot without DIRECT — preserving design §5.3's "core defaults unconditional" invariant.-------## Phase 0 — Rename (free the word "plugin")--**Goal.** Kill the three-way "plugin" overload so the new vocabulary is unambiguous. Pure rename, no behavior change.--**Renames** (class → file):--- `Plugin` (`plugins/plugin.py`) → `ModelHandle` (`plugins/model_handle.py`). A backend *selector* (`sdk` + `backend` + `variant` + `sdk_handle`), not a plugin. Keep `make_for_inference_model`.-- `PluginManager` (`plugins/plugin_manager.py`) → `SdkClientManager` (`plugins/sdk_client_manager.py`).-- `PluginSdkRegistry` (`plugins/plugin_sdk_registry.py`) → `SdkClientRegistry` (`plugins/sdk_client_registry.py`). Rename `get_sdk_instance`/`set_sdk_instance` → `get`/`set` (the DRY `get_or_create` lands in Phase 1, not here).-- `PluginFactoryAbstract` (`plugins/plugin_factory_abstract.py`, the `make_extras` helper — *unrelated* to the new system) → `BackendExtrasFactory` (`plugins/backend_extras_factory.py`). Name-hygiene; it collides with the new contract vocabulary.--**Blast radius is bigger than "the four factories" (codex C11).** `Plugin` / `plugin_sdk_registry` is imported across the in-tree adapter modules — `openai`, `bedrock`, `anthropic`, `gateway`, `portkey` list+factory modules, plus `search_worker_factory.py:4` (which uses the *correct* `plugins.plugin` path while `llm/img_gen/extract` import from the *wrong* `plugins.plugin_sdk_registry` path). Grep `import Plugin\b` and `plugin_sdk_registry` across `pipelex/` before starting; the rename touches more than the dispatch sites. The wrong-path imports get fixed for free here.--- `hub.py`: `get_plugin_manager`/`set_plugin_manager` → `get_sdk_client_manager`/`set_sdk_client_manager`; **fix the `"PluginManager2 is not initialized"` message** (`hub.py:291`) to `"SdkClientManager is not initialized"`.-- `pipelex.py`: setup/teardown call sites.--**D4 — keep this a pure rename.** The `SdkClientManager` post-rename is still a thin wrapper holding one `SdkClientRegistry`. *Collapsing* it (hub holds the registry directly) is a separate structural change captured as a **P3 TODO** (own commit), not bundled here — rename and restructure don't mix.--**Tests.** No new tests; existing suite + `make tb` (boot/config) stay green.--**Checkpoint 0.** Green suite, clean vocabulary, `PluginManager2` artifact gone, wrong-path `Plugin` imports fixed. Standalone PR.-------## Phase 1 — Seam machinery + LLM family (merged, per D2)--**Goal.** Build the discovery + contract + registry machinery *and* migrate the LLM family through it in one PR, so the contract is proven against a real, messy family before it locks. No behavior change.--**New modules** (all import-light — must not import any backend SDK or `temporalio`):--- `plugins/contract.py` — `PipelexPlugin` (`@runtime_checkable Protocol`: `name: str`, `targets_api: int`, `register(self, registrar) -> None`) and `PLUGIN_API_VERSION: int = 1`. **Contract invariant — `register` is side-effect-free:** it may *only* call registrar methods (no hub touch, no I/O, no client construction). This is what makes `build_registrar` safely repeatable (D3 runs it at CLI-build *and* boot).-- `plugins/registrar.py` — `PluginRegistrar`, the accumulator. Holds the inference registry, orchestrator registry, **deferred** hub-slot-claim thunks, CLI-command list, teardown-callback list, and a read-only `config: ConfigRoot`. Menu methods:- - `add_inference_backend(*, family, sdk, make_worker: MakeWorkerFn)` — D1: a callable, not a factory object.- - `add_orchestrator(*, mode, orchestrator)`.- - `claim_content_generator | pipe_router | pipe_run | task_manager(factory: Callable[[], Impl])` — **D5: a thunk, not an instance.** `register` never constructs the impl, so it never imports `temporalio`.- - `add_cli_command(*, name, help, command)`, `add_teardown(callback)`.-- `plugins/inference_backend_registry.py` — `InferenceFamily` (`StrEnum`: `LLM`/`IMG_GEN`/`EXTRACT`/`SEARCH`), `MakeWorkerFn` (TypeAlias for the callable signature), `InferenceBackendRegistry` keyed by `(family, sdk)`.-- `plugins/orchestrator_registry.py` — `OrchestratorProtocol` (`async def run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput`), `OrchestratorRegistry` keyed by `PipelexExecutionMode`.-- `plugins/discovery.py` — `build_registrar(*, config) -> PluginRegistrar`: a **pure function** iterating `BUILTIN_PLUGINS` + `importlib.metadata.entry_points(group="pipelex.plugins")`, running the version check per plugin, calling `plugin.register(registrar)` (skipping any plugin whose `name` is in `config.plugins.disabled`, logged; raising if a core-unconditional plugin is denylisted — D7), applying fail-loud. Pure ⇒ safely called at CLI-build and at boot.-- `plugins/builtins.py` — `BUILTIN_PLUGINS`. Phase 1 fills it with the LLM drivers.-- `plugins/exceptions.py` — `PluginApiVersionMismatchError`, `DuplicateInferenceBackendError`, `DuplicateOrchestratorError`, `HubSlotAlreadyClaimedError`, `BrokenPluginError`. (`MissingOrchestratorError` lives in `runtime_bridge/exceptions.py` — near its domain — replacing the two old typed errors; see Phase 3.)--The `MakeWorkerFn` signature (D1 — callable, uniform across families):--```python-MakeWorkerFn = Callable[..., InferenceWorkerAbstract]-# called as: make_worker(*, inference_model, backend, sdk_clients, reporting_delegate)-```--**DRY the two repeated blocks** (design §6.2):--- **Client cache** — `SdkClientRegistry.get_or_create(*, handle, build)` replaces the `get_sdk_instance(...) or set_sdk_instance(...)` dance in every arm.-- **Extra guard** — `require_sdk(*, spec, extra, msg)` (raises `MissingDependencyError` if `find_spec(spec) is None`) replaces the repeated guard, moved **into** `make_worker` so a missing extra fails at use, not boot.--**LLM family migration.** `llm_worker_factory.py`'s `match` collapses to `get_inference_backend_registry().lookup(family=LLM, sdk=model_handle.sdk)(...)`. Each LLM driver plugin's `register` calls `add_inference_backend(family=LLM, sdk="anthropic", make_worker=<closure>)`. A registry *miss* raises a distinct "backend `<sdk>` not registered; is its plugin installed?" error.--**Boot wiring.** In `Pipelex.setup()`, after config is fully resolved (incl. the `temporal_enabled` override at `pipelex.py:193-196`) and before the hub setup points, call `build_registrar(config=get_config())`. Store the two keyed registries on the hub (`set_inference_backend_registry` / `set_orchestrator_registry`). Hold the slot-claim thunks, CLI commands, teardown callbacks for their apply-points.--**Conflict & version policy** (fail loud): duplicate `(family, sdk)` / duplicate `execution_mode` / double-claimed slot each raise naming both contributors; `targets_api != PLUGIN_API_VERSION` → `PluginApiVersionMismatchError` with the remedy; a broken entry point → `BrokenPluginError` with context (the one sanctioned `except Exception` site — wraps unbounded third-party plugin code; annotate it).--**Control-surface deliverables (D7).** Add a core-owned `plugins: PluginsConfig` section (`disabled: list[str]`, default `[]` in `pipelex.toml` and `.pipelex/pipelex.toml` — never a class-def default) to `configs.py`; apply the denylist in `build_registrar` (skip + log; error on a core-unconditional name). Add the core `pipelex plugins list` command (lists discovered plugins, their contributions, and denylist state). See the enablement-model section above.--**Tests:**--- Contract conformance (synthetic plugins): protocol satisfaction; `targets_api` mismatch fails loud; duplicate `(family, sdk)` / duplicate mode / double-claimed slot each raise naming both; broken entry point → `BrokenPluginError`; **`build_registrar` idempotent** (two calls, no side effect — pins the side-effect-free invariant).-- Denylist (D7): a discovered non-core plugin named in `plugins.disabled` is skipped and logged; denylisting a core-unconditional plugin (e.g. the OpenAI driver or DIRECT) raises at startup; `pipelex plugins list` shows discovered built-ins + (synthetic) externals with contributions and denylist state.-- **Import-light boot via subprocess + import-blocker (codex C12, replaces the weak `sys.modules` check):** spawn a subprocess with a `sys.meta_path` finder that *raises* on `anthropic`, `mistralai`, `google.genai`, `boto3`, `fal_client`, … then boot with built-ins registered and assert no raise. A `sys.modules`-absence assertion in-process is noisy (prior imports leak); the import-blocker is deterministic.-- Registry round-trip (LLM): register → `lookup(LLM, sdk)` → returns the right `LLMWorkerAbstract`.-- Error parity: an Anthropic model with the `anthropic` extra absent raises the *same* `MissingDependencyError` text (lib + extra + alternative).--**Checkpoint 1.** Seam exists, proven against the real LLM family; import-light enforced by the subprocess guard; conflict/version policy tested. The contract is now locked against a real consumer. Mergeable.-------## Phase 2 — Remaining inference families (ImgGen, Extract, Search)--**Goal.** Migrate the other three families onto the proven contract. One family per sub-PR.--- **ImgGen.** Collapse the `match`. **Fix the pre-existing huggingface bug (codex C10):** `img_gen_worker_factory.py:64` imports `huggingface_hub` with **no `find_spec` guard** — unlike every other arm, a missing extra raises a raw `ImportError` instead of the friendly `MissingDependencyError`. Add the `require_sdk(spec="huggingface_hub", extra="huggingface", …)` guard during migration (flag-and-fix per repo policy). The provider-literal handling (`model_handle.variant` → provider) lives inside the closure. The substrate-reuse workers (`blackboxai`/`gateway_completions`/`openrouter` → `OpenAICompletionsImgGenWorker` with a per-vendor completions factory) become closures capturing that factory.-- **Extract.** Collapse the `match`. Stateless arms (`pypdfium2`, `linkup_fetch`) skip `get_or_create`.-- **Search — normalize the call surface (codex C9).** `search_worker_factory.py` takes **no `reporting_delegate` param** and pulls `get_report_delegate()` from the hub (`:31`, `:41`), and `make_search_worker` itself omits it. The uniform `MakeWorkerFn` passes `reporting_delegate` explicitly. Resolution: normalize `make_search_worker` and its callers to accept `reporting_delegate` (removing search's hidden hub coupling — a strict improvement), so search fits the uniform signature. This is a small call-surface change, not a special case.--Add each migrated driver to `BUILTIN_PLUGINS`. OpenAI is an always-on built-in driver, no privileged arm (design D6-design); the OpenAI-compat *substrate* stays an in-tree library the gateway/portkey/openrouter/blackboxai closures import — substrate *extraction into a named module* is deferred.--**Tests:** registry round-trip per family; the import-light subprocess guard now covers all optional SDKs; **huggingface error-parity** (missing `huggingface` extra now raises `MissingDependencyError`, proving the bug fix); a **cross-family vendor** test (e.g. `mistral` registers into both LLM and Extract from one `register`).--**Cross-cutting deliverable:** the **inference SPI reference** (design §9.1) + the **plugin-authoring guide and minimal example backend plugin** land here (a backend plugin is the simplest example). Placement: authoring guide + SPI reference in a user-facing `docs/.../plugins/` Guide section; seam internals in `docs/under-the-hood/`; the example as runnable in-repo code.--**Checkpoint 2.** All four `match` worker-factory statements gone; every backend via the registry while in-repo; the huggingface guard bug fixed; search's hub coupling removed; SPI + authoring guide shipped. **Natural session handoff** — inference side done, orchestrators open next.-------## Phase 3 — Orchestrators through the seam (in-tree)--**Goal.** Collapse the bridge `match` and the four `temporal.is_enabled` boot blocks into registry/slot lookups; move Temporal's modes/CLI/boot-swap/teardown behind its (still-in-repo) plugin; flip Mistral's hard import to discovery; publish the orchestrator SPI. No behavior change.--**Extract the orchestrators** (verbatim bodies):--- `DirectOrchestrator.run` — from `_run_direct` (`bridge.py:270-294`). **Correctness landmine (design §8.1): keep `with scoped_pipe_router(PipeRouter())` verbatim** — dropping it leaks DIRECT-mode nested sub-pipes to Temporal inside a Temporal worker. Registered in core, always-on.-- `TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` — from `_run_temporal_*` (`:297-355`), keeping the `WorkflowExecutionError` catch and `make_workflow_id` recompute.-- `MistralWorkflowsOrchestrator` — **authored in the `pipelex-mistralai-workflows` repo**, not here.--**The bridge collapses** (design §8.1) to a registry lookup. After it, `bridge.py` names no integration.--**Preserve per-mode error quality (codex C7 — a generic hint is a regression).** Today `bridge.py:399/:417` carry exact, mode-specific install messages ("install `pipelex[temporal]`" vs "install `pipelex-mistralai-workflows`"). The registry-miss path must reproduce *those* messages, not a single generic hint. `MissingOrchestratorError` takes the `mode` and maps it to its exact message via the `PipelexExecutionMode.requires_*` properties (the typed source of truth for which mode needs what). One message per mode, preserved verbatim.--**The Temporal plugin** (`TemporalOrchestrator`). Its `register`:--- **always** `add_orchestrator(TEMPORAL_BLOCKING|TEMPORAL_FIRE_AND_FORGET, …)` and `add_cli_command("worker"|"setup-temporal-namespace", …)`;-- **if `config.temporal.is_enabled`**: `claim_content_generator(factory)`, `claim_pipe_router(factory)`, `claim_pipe_run(factory)`, `claim_task_manager(factory)`, `add_teardown(...)` — **D5: each `claim_*` is handed a thunk** (e.g. `claim_content_generator(lambda: ContentGeneratorInWorkflowFactory.make_content_generator_in_workflow())`), so `register` never imports `temporalio`. The thunk runs at the setup apply-point.--**Boot/teardown collapse with explicit injection precedence (codex C8).** The four `if get_config().temporal.is_enabled:` blocks (`pipelex.py:377-466`) become, at each existing ordered point (content generator → task manager → router → run): **explicit `setup()` param (test/user injection at `:376/:450`) > plugin slot-claim thunk > core default.** The slot claim must *not* silently override an explicit injection — pin this precedence in a test. The inlined teardown (`:471-479`) becomes a registered teardown callback run LIFO.--**CLI collapse (codex C4 + D3).** The two hard imports in `_cli.py:15,20,234,238` become plugin-contributed commands. `PipelexCLI.list_commands` (`:32`, which hardcodes command order) is reworked to merge core commands with `registrar.cli_commands` deterministically (stable order, clean `--help`, unknown-command behavior intact). Per D3, the CLI entry point loads config then runs the pure `build_registrar` once to harvest `registrar.cli_commands`; D5's thunks mean this harvest never constructs a Temporal impl even when `temporal.is_enabled`. **Spike this first in Phase 3** — it's the one genuinely novel integration point.--**Mistral → entry-point discovery** (this repo's side). Remove the `_run_mistral_native` hard import (`bridge.py:414`). Uninstalled (CI default) → `MISTRAL_NATIVE` absent → `MissingOrchestratorError` with the exact Mistral message. The real plugin (entry point + orchestrator re-pointed at the SPI) is a `pipelex-mistralai-workflows` change. pipelex's suite proves dispatch with a *fake* registered orchestrator.--**Publish the orchestrator SPI** (design §9.2, sized to Mistral's *measured* imports): `runtime_bridge.*` incl. `runtime_bridge.primitives.*`; the execution protocols; the boundary/core payload types; library-crate access + hub scoping; the tracing hooks. Documented module/symbol list. Mistral's boundary resolves against it.--**Tests:** bridge dispatch by mode (fake orchestrator); **per-mode error parity** (each missing mode → its exact message — C7); DIRECT router scoping (the landmine); DIRECT parity (byte-identical); **injection-precedence** (explicit param wins over a slot claim — C8); boot-via-slots (stripped env resolves DIRECT + core backends; temporal-enabled resolves the four slots to Temporal impls via thunks); teardown LIFO order; **CLI-command contribution** (`worker` appears in `--help` when the plugin is discoverable, absent otherwise — pins D3); **full Temporal suite green in-tree** (§14.5 — the extraction gate; Phase 5 does not start until this is green).--**Checkpoint 3.** Bridge + boot/teardown + CLI name no integration; Temporal fully behind its plugin in-repo (import-light preserved via thunks); per-mode errors preserved; injection precedence pinned; Mistral via discovery; orchestrator SPI published; Temporal suite green through the seam. **The** checkpoint before externalization.-------## Phase 4 — Invert model listing (the 5th seam, per D6)--**Goal.** Close the goal: `model_lists.py:58`'s `match sdk:` (the `list-models` CLI path) still hardcodes `pipelex.plugins.{openai,anthropic,mistral,google,bedrock}.*_list` imports. Invert it onto the inference contract so core names no integration *anywhere*.--- Add a **model-listing capability** to the inference plugin contract: `add_model_lister(*, sdk, lister: ListModelsFn)` (a callable mirroring `MakeWorkerFn` — import-light, lazy). A backend plugin that lists models registers its lister alongside its worker factory.-- `ModelLister.list_models` (`model_lists.py`) collapses to a registry lookup keyed by `sdk`; a miss → the same friendly "is its plugin installed?" guidance. The `find_spec` guards (already present in this file's arms) move into each lister.-- The contract grows by one optional method — backends that don't support listing simply don't register a lister (progressive disclosure preserved).--**Tests:** round-trip (a registered lister is invoked for its `sdk`); a backend with no lister yields the friendly miss; import-light subprocess guard still green (listers are lazy).--**Checkpoint 4.** `model_lists.py` names no integration; "core names no integration by import or string" holds without an asterisk.-------## Phase 5 — Externalize Temporal → `pipelex-temporal` (cross-repo, gated on Checkpoint 3)--**Goal.** Lift `pipelex/temporal/` into a separate dist. A *packaging* move — but only after a prerequisite the design missed.--**Step 0 — relocate the Temporal config schema to a core-owned module (codex C6 — the design's D7 is self-contradictory as written).** `config_temporal.py` currently *lives at* `pipelex.temporal.config_temporal` and imports `pipelex.temporal.exceptions` (`:10`: `TemporalConfigError`, `WorkerTaskQueueUnknownError`). "Keep config in core, move `pipelex/temporal/` out" is impossible while the schema sits under the `pipelex.temporal` namespace. Relocate the schema **and the specific exceptions it needs** to a core-owned module outside `pipelex.temporal` (e.g. `pipelex/system/configuration/config_temporal.py` + the two exceptions into a core `exceptions.py`). Update `configs.py:14`. The `if TYPE_CHECKING: … RetryPolicy = Any` placeholder keeps it importable without `temporalio`. This step is low-risk and could be pulled into Phase 3; it is the true prerequisite that makes the rest a packaging move. `make tb` stays meaningful.--**Then:**--- Move `pipelex/temporal/` (impl) + `temporal_plugin.py` + tests + the `temporal` marker + the `--temporal-server` conftest option into `pipelex-temporal`; declare `[project.entry-points."pipelex.plugins"]`.-- Protocol-level conformance tests stay in core; Temporal's behavioral suite travels.-- **Flip downstream pins** (cross-repo blast radius): `pipelex-worker`, `pipelex-api-hosted` from `pipelex[temporal]==X` → `pipelex-temporal==Y`.-- **Repoint `pipelex-mistralai-workflows`** (design §15): its `temporal` extra + `pipelex.temporal.*` imports → `pipelex-temporal`.--**Tests.** Re-run the relocated Temporal suite against the published `pipelex`; run downstream consumers' suites against the new pins before publishing.--**Checkpoint 5.** Temporal is an external plugin; core has no `temporal` extra; downstream pins flipped; consumer suites green.--> Vendor inference dists and OpenAI-substrate extraction are **out of this rollout** (design §6.6, §11) — one-line moves if/when justified.-------## Resolved open questions (design §17)--- **SPI delivery shape →** documented module/symbol list only, no curated re-export surface (the repo bans `__init__.py` re-exports). The version marker `PLUGIN_API_VERSION` lives in `plugins/contract.py`.-- **Registrar config-access timing →** `build_registrar` runs after config resolution (incl. the `temporal_enabled` override) and before the hub-slot apply-points; every `register` sees a resolved `ConfigRoot`. The CLI-build second call is safe because D5's thunks make slot-claiming import-free and side-effect-free.-- **Factory granularity →** one **callable** per `(family, sdk)` (D1).-- **Where the committed docs land →** authoring guide + SPI reference in a user-facing `docs/.../plugins/` Guide section; seam internals in `docs/under-the-hood/`; minimal example as runnable in-repo code.--## NOT in scope (considered, deferred)--- Third-party-defined execution modes (open string-keyed space) — design D5; enum stays closed.-- Generic per-plugin typed-config namespace — design D7; Temporal config stays typed in core.-- Deliberately *overriding* a built-in (a third party replacing core's OpenAI driver) — design §5.3.-- Per-SPI versions / semver-range matching — design §5.4; single coarse `int`.-- Vendor inference dists + OpenAI-substrate extraction into a named module — design §6.4/§6.6/§11.-- **Collapsing `SdkClientManager` into `SdkClientRegistry`** — P3 follow-up commit after Phase 0 (D4), not bundled into the rename.-- Distribution/CI for `pipelex-temporal` — part of Phase 5's cross-repo effort, not Phases 0–4.--## What already exists (reused, not rebuilt)--The plan inverts the *last* coupling; it reuses, not rebuilds: the lazy-import pattern in every `match` arm; the optional extras (`pipelex[anthropic|temporal|…]`); the `*WorkerAbstract` hierarchy (`cogt/inference/inference_worker_abstract.py` + the four family abstracts); the `SdkClientRegistry` client cache (renamed); the `PipelexExecutionMode` enum + `requires_*` properties; the hub singletons + `scoped_pipe_router` ContextVar scoping; the `Temporal` config placeholder (`RetryPolicy = Any`). The seam + registries are the only genuinely new machinery; everything downstream of a lookup is existing code.--## Failure modes (new codepaths)--| Codepath | Realistic failure | Test? | Error handling? | User sees |-|---|---|---|---|---|-| `build_registrar` discovery | a broken external entry point | yes (conformance) | `BrokenPluginError` w/ context | clear, named |-| inference lookup | backend not registered (plugin absent) | yes (round-trip miss) | distinct "not registered" error | clear |-| `make_worker` closure | missing extra (incl. the fixed hf arm) | yes (error parity) | `MissingDependencyError` (lib+extra+alt) | clear, friendly |-| `register` at CLI-build | importing `temporalio` at CLI startup | yes (import-light subprocess + CLI-contribution) | D5 thunks prevent it | n/a (prevented) |-| DIRECT inside Temporal worker | nested sub-pipes leak to Temporal | yes (router-scoping) | `scoped_pipe_router` preserved | n/a (prevented) |-| slot claim vs explicit injection | slot silently overrides test/user inject | yes (injection-precedence) | explicit param wins | n/a (prevented) |-| orchestrator miss | wrong/absent mode | yes (per-mode parity) | per-mode `MissingOrchestratorError` | exact install message |--No critical gaps (no failure mode is simultaneously untested, unhandled, and silent).--## Worktree parallelization--| Step | Modules | Depends on |-|---|---|---|-| Phase 0 rename | `plugins/`, `hub.py`, `pipelex.py`, all adapter modules | — |-| Phase 1 seam+LLM | `plugins/`, `cogt/llm/` | Phase 0 |-| Phase 2 ImgGen / Extract / Search | `cogt/{img_gen,extract,search}/`, `plugins/<vendor>/` | Phase 1 (contract) |-| Phase 3 orchestrators | `runtime_bridge/`, `pipelex.py`, `cli/`, `temporal/` | Phase 1 |-| Phase 4 model_lists | `cogt/model_backends/`, `plugins/<vendor>/` | Phase 1 (contract) |--**Lanes after Phase 1 lands:** Phase 2's three families are independent of each other and of Phase 3/4 → `Lane A: ImgGen`, `Lane B: Extract`, `Lane C: Search`, `Lane D: Phase 3 orchestrators`, `Lane E: Phase 4 model_lists` can run in parallel worktrees. **Conflict flag:** Lanes A/B/C and E both touch `plugins/<vendor>/` modules (e.g. a vendor serving multiple families) — coordinate the per-vendor plugin object so two lanes don't both edit it. Phase 0 and Phase 1 are strictly sequential (everything depends on the rename then the contract).--## Execution risks--- **Phase 3 carries most risk** — bridge + boot + teardown + CLI at once, plus the DIRECT-router-scoping landmine and the per-mode-error and injection-precedence subtleties codex surfaced. Mitigation: verbatim orchestrator extraction; pin each subtlety with its own test; land the §14.5 Temporal-green checkpoint before externalizing.-- **CLI timing (D3) + thunks (D5)** are the novel integration point. Mitigation: spike first in Phase 3; the CLI-contribution test proves it.-- **Import-light regression** is invisible until profiled. Mitigation: the subprocess import-blocker guard (C12) lands with Phase 1.-- **Phase 5 is cross-repo** — the only phase that can break downstream deploys; gated, separate, with config relocation as its de-risking step-0.--## Sequencing summary--Phase 0 (rename) → Phase 1 (seam + LLM, merged) → Phase 2 (ImgGen/Extract/Search, parallel sub-PRs) → Phase 3 (orchestrators) → Phase 4 (model_lists) → **[gate: §14.5 green]** → Phase 5 (externalize Temporal, cross-repo). Phases 0–4 are independent PRs in `pipelex`; each lands green. Inference SPI + authoring guide ship with Phase 2; orchestrator SPI with Phase 3.-------## GSTACK REVIEW REPORT--| Review | Trigger | Why | Runs | Status | Findings |-|--------|---------|-----|------|--------|----------|-| CEO Review | `/plan-ceo-review` | Scope & strategy | 0 | — | not run |-| Codex Review | outside voice | Independent 2nd opinion | 1 | issues_found | found 5th seam + import-light hole + Phase-4 config contradiction + hf bug — all folded |-| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 1 | clean | 18 findings, 0 critical gaps, 0 unresolved |-| Design Review | `/plan-design-review` | UI/UX gaps | 0 | — | n/a (backend-only) |-| DX Review | `/plan-devex-review` | Developer experience gaps | 0 | — | not run |--- **CODEX:** outside voice caught what the interactive review missed — the 5th seam (`model_lists.py`), the import-light-at-CLI-build hole (resolved via D5 deferred slot thunks), the Phase-5 config self-contradiction (resolved via core-owned config relocation), a pre-existing huggingface `find_spec` bug, and the per-mode-error / injection-precedence gaps. All folded into the plan.-- **CROSS-MODEL:** no standing tension — codex's findings were additive gaps, not disagreements with the review; all resolved by user decision (D5, D6) or clear fold.-- **VERDICT:** ENG CLEARED — ready to implement.--NO UNRESOLVED DECISIONSdiff --git a/wip/plugins/inference-backends-as-plugins.md b/wip/plugins/inference-backends-as-plugins.mddeleted file mode 100644index 093784031..000000000--- a/wip/plugins/inference-backends-as-plugins.md+++ /dev/null@@ -1,111 +0,0 @@-# Inference backends as Pipelex plugins--**Status:** assessment / proposed plan (not started)-**Scope:** turn the inference-SDK wrappers under `pipelex/plugins/` (OpenAI, Anthropic, Google, Mistral, Bedrock, Fal, Docling, Linkup, …) into *registerable plugins* — selected out of a registry the backends populate, rather than dispatched from hardcoded `match` statements in core. Optionally, package the heavier ones as separate installable dists.--> This is the **driver** half of the plugin story. Inference backends are homogeneous, many, and *selected by data* (a model's `sdk` handle). The **strategy** half — orchestrators (Temporal, Mistral Workflows) that swap core orchestration seams — is in [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md). The cross-cutting best practices — and the shared discovery mechanism both halves use — are in [`README.md`](README.md).--## Verdict--Partially decoupled, and confusingly named. The wrappers already live in one place (`pipelex/plugins/<backend>/`), already share clean abstract base classes (`LLMWorkerAbstract`, `ImgGenWorkerAbstract`, `ExtractWorkerAbstract`, `SearchWorkerAbstract`, all under `InferenceWorkerAbstract`), already gate heavy SDKs behind optional extras + lazy `importlib.util.find_spec(...)` checks, and already cache instantiated SDK clients in a runtime registry (`PluginSdkRegistry`, keyed by an `sdk@backend/variant` handle). **What's missing is the inversion**: core still owns the backend→implementation mapping in centralized `match plugin.sdk:` statements inside the four worker factories. A third party cannot add a backend without editing core.--So the work here is the mirror of Temporal's: define the registry, have backends register into it (via the *same* entry-point discovery), and delete the `match` statements.--## First: resolve the "plugin" naming overload--The word is already overloaded three ways, and that has to be fixed before adding a real plugin system or every conversation will be ambiguous:--- `pipelex/plugins/<backend>/` — the directories. These are **in-tree SDK adapters**, not installable plugins.-- `Plugin` (`pipelex/plugins/plugin.py`) — a value object: `sdk` + `backend` + `variant`, with an `sdk_handle` like `anthropic@bedrock/claude`. This is a **backend selector / handle**, not a plugin.-- `PluginManager` / `PluginSdkRegistry` — a **runtime cache of instantiated SDK clients** keyed by that handle, with `teardown()`. Not a discovery mechanism.--None of these is a plugin in the installable-distribution sense. Proposed vocabulary:--- Rename the existing trio to what it is: `SdkClientRegistry` / `SdkClientManager`, `BackendSelector` (or keep `Plugin` as `ModelHandle`). Pure rename, no behaviour change.-- Reserve **"plugin"** for the new concept shared with Temporal: an installable unit, discovered via the `pipelex.plugins` entry-point group, that registers extensions at boot.--(This rename can land independently and ahead of everything else — it's low-risk and unblocks clear naming for the rest.)--## What kind of plugin these are--**Driver plugins.** Many of them, all the same shape, chosen at runtime by the model spec's `sdk` field. They don't replace core behavior; they *supply an implementation* for a backend the core asked for. The right seam is a **typed registry keyed by `sdk` handle**, one per worker family. A driver plugin's job is to call `registry.register(sdk_handle, factory)` at boot. Contrast Temporal, which swaps singleton seams on the hub — different hook, same discovery.--## The seams today--Four worker factories, each a centralized dispatch on `plugin.sdk`:--| Factory | File | Backends in its `match` |-|---|---|---|-| `LLMWorkerFactory` | `cogt/llm/llm_worker_factory.py` | `gateway_completions`, `portkey_completions`, `openai` / `azure_openai`, `anthropic` / `bedrock_anthropic`, `mistral`, `bedrock_boto3` / `bedrock_aioboto3`, `google` |-| `ImgGenWorkerFactory` | `cogt/img_gen/img_gen_worker_factory.py` | `gateway_img_gen`, `fal`, `huggingface_img_gen`, `openai_img_gen`, `blackboxai_img_gen`, `azure_rest_img_gen`, `google` |-| `ExtractWorkerFactory` | `cogt/extract/extract_worker_factory.py` | `gateway_extract`, `mistral`, `pypdfium2`, `docling_sdk`, `linkup_fetch` |-| `SearchWorkerFactory` | `cogt/search/search_worker_factory.py` | `linkup`, `gateway_search` |--Each `case` does the same three things: optionally `importlib.util.find_spec(...)` to raise a friendly `MissingDependencyError` (lib name + extra name), lazily import the backend's factory + worker, then build the worker (reusing a cached SDK client from `PluginSdkRegistry`). The lazy import inside each `case` is exactly the inversion point — it's already "don't touch the SDK until this backend is selected"; it just needs to become "ask the registry for the selected backend's factory."--Supporting structure already in place:--- **Abstract contracts.** `InferenceWorkerAbstract` → `{LLM,ImgGen,Extract,Search}WorkerAbstract`. A registered factory returns one of these. `PluginFactoryAbstract.make_extras(...)` is a partial start at a common factory contract.-- **Optional extras.** `pyproject.toml`: `anthropic`, `bedrock`, `docling`, `fal`, `google` / `google-genai`, `huggingface`, `linkup`, `mistralai` (plus `dynamodb` / `s3` infra extras). `openai`, `portkey`, `pypdfium2` are **core deps** (always present), so `gateway`/`portkey`/`openai`/`azure_rest` backends never need a guard. (OpenAI is a core dep for a deeper reason than "it's always installed" — see "OpenAI is two things" below.)-- **Runtime client cache.** `PluginSdkRegistry` keyed by `sdk_handle` — keep this as-is (rename aside); it's the right place to memoize clients and already has lifecycle teardown.--## "Plugin" and "in-repo" are orthogonal axes--Before deciding what ships where, kill one confusion: **being a plugin** (registering through the seam) and **shipping as a separate dist** are independent choices. A built-in plugin that lives in this repo is the *default*, not a contradiction.--| | **In-repo** | **Separate dist** |-|---|---|---|-| **Through the seam** (registry / entry point) | built-in plugins — the default for *every* backend (OpenAI driver, gateway, portkey, …) | external plugins (`pipelex-anthropic`, `pipelex-google`, …) |-| **Hardcoded in core** | the wart we're deleting | n/a |--"Plugin" = *registers through the seam* instead of a `match` arm. "In-repo vs separate dist" = a *packaging* decision (the granularity choices below). They don't constrain each other — so "keep it internal" never has to mean "keep it a hardcoded special case."--## OpenAI is two things: substrate vs driver--The OpenAI SDK is not just another backend — it is the lingua franca a large fraction of the others are built on. `gateway`, `portkey`, `openrouter`, `blackboxai` all construct `openai.AsyncOpenAI(base_url=...)` (OpenAI-compatible endpoints), and even `anthropic`, `mistral`, `google` import the SDK for the shared `ChatCompletion*` message types. So "the OpenAI SDK" hides two concerns with different fates:--- **Substrate** — the OpenAI-compatible client (`AsyncOpenAI(base_url=…)`), the shared request/response types, and the completions/responses plumbing that gateway/portkey/openrouter/blackboxai/azure_rest reuse. This is **infrastructure other backends import**, not a backend itself.-- **Driver** — the `openai` / `azure_openai` / `openai_img_gen` registrations: the actual "talk to api.openai.com" worker.--They live in the same `pipelex/plugins/openai/` directory today but want opposite treatment:--- **Substrate → core internal library, always a hard dependency.** Promote it out of `pipelex/plugins/openai/` into a deliberate module (e.g. `pipelex.openai_compat`) with a stable signature, importable by any backend — in-repo or external. The "OpenAI API is a de-facto standard" argument is precisely the argument for keeping the substrate in core: if it were instead an optional dist, gateway/portkey/openrouter/blackboxai/azure would *all* have to depend on it — a diamond where a dozen "plugins" pull `pipelex-openai`. Core is the right home. And once plugins import it, the substrate becomes a **contract** — shape it deliberately rather than letting plugins reach into `pipelex/plugins/openai/` internals (which `anthropic_factory.py` et al. effectively do today).-- **Driver → always-on built-in plugin, through the seam.** No extra, registered unconditionally — but registered like everyone else. Do **not** leave OpenAI as the one surviving hardcoded `match` arm: the seam's whole value is *core names no backend*, and a privileged special case is a two-tier system the next "important" backend will want to join, eroding the abstraction. It ships in-repo and is always present; it just goes through the same door.--Net: **substrate in core (a library, not a plugin); OpenAI driver is an always-on built-in plugin through the seam.** OpenAI never leaves the repo and is never gated — without keeping a privileged backend.--## Recommended approach: a worker-factory registry the backends populate--1. **One registry per worker family** (or one registry, partitioned by family), keyed by `sdk` handle, mapping to a factory callable returning the right `*WorkerAbstract`. Core owns the registry + the abstract contracts; backends own the entries.--2. **Each backend registers itself.** Replace the `match` arms with `registry.lookup(plugin.sdk)`. The friendly `MissingDependencyError` (lib + extra name) moves into the backend's registration/factory so the error stays as helpful as today. Built-in backends register at boot through the same `pipelex.plugins` entry-point mechanism Temporal uses — **dogfood the plugin API in-tree first**, before any backend is externalized.--3. **Keep selection data-driven.** Nothing changes for users: models still declare `sdk` / `backend_name` / `variant` in TOML; `Plugin.make_for_inference_model(...)` (→ `ModelHandle`) still builds the handle; the registry lookup replaces the `match`. The `MISSING extra` failure mode is preserved, just relocated.--4. **Decide packaging granularity** (this is the real product decision, deferrable):- - **In-tree but registerable (lowest churn).** All backends stay in `pipelex`, but register through the seam. Wins: third parties can add a backend in their own dist without a core PR; the `match` statements die; no distribution split to maintain. This alone delivers most of the value.- - **Per-vendor dists** (`pipelex-anthropic`, `pipelex-google`, `pipelex-bedrock`, …): each owns its SDK dep and its entry point; base `pipelex` ships only the always-on backends (openai/portkey/gateway). Wins: smallest base install, independent cadence, clean dependency surface. Costs: many small repos/dists, version-matrix testing, more release overhead.- - **Bundles** (e.g. `pipelex-aws` = bedrock + s3 + dynamodb; `pipelex-google` = genai + vertex): a middle ground grouping by shared SDK/credentials.-- Recommendation: **adopt the registry seam first (in-tree), externalize selectively later** — only split out a backend when its dependency weight or release cadence actually justifies a separate dist. The seam makes that a packaging move, not a refactor.--## Watch-items--- **Granularity of the handle.** `sdk` vs `backend` vs `variant` already encode a lot (`anthropic@bedrock/claude`). The registry key should be the `sdk` handle the factories currently switch on — confirm that's stable and 1:1 with a factory.-- **Inconsistent factory shapes.** Some backends have a factory class (`AnthropicFactory`, `MistralFactory`), some instantiate the worker directly (Fal). Normalize on one registration contract (extend `PluginFactoryAbstract`) so every entry looks the same.-- **Core-dep backends.** openai/portkey/gateway/pypdfium2/azure_rest have no extra and must always be registered — make their registration unconditional so a misconfigured plugin environment can't drop them.-- **OpenAI substrate is a shared contract.** Once the OpenAI-compat substrate is promoted to a core module that other backends import (see "OpenAI is two things"), its signature is part of the surface plugins depend on — version and shape it deliberately, don't let it drift as an accidental internal.-- **Cross-family backends.** `mistral` appears in LLM *and* Extract; `google` in LLM *and* ImgGen; `linkup` in Extract *and* Search; `gateway_*` everywhere. A per-vendor dist must register into multiple family registries — the plugin's `register_*` hook needs access to all of them.-- **Error parity.** The `MissingDependencyError` messages today are genuinely helpful (they suggest alternatives, e.g. "use Anthropic via Bedrock instead"). Preserve that quality when moving the guard into registration.--## Effort & risk--- **Moderate, and mostly mechanical** — but with more surface than Temporal (four factories, ~15 backends, two extras families) and a naming refactor up front. No new hard algorithms; the lazy-import + extras + client-cache machinery already exists.-- The risk concentrates in the **registration timing** (everything must be registered before the first worker is built) and in **not silently dropping a core backend**. Both are covered by keeping built-in registration unconditional and tested.--## Suggested phasing--- **Phase 0 — rename.** `Plugin`/`PluginManager`/`PluginSdkRegistry` → `ModelHandle` (or `BackendSelector`) / `SdkClientManager` / `SdkClientRegistry`. Pure rename. _Checkpoint: suite green, naming unambiguous, "plugin" freed for the real concept._-- **Phase 0.5 — extract the OpenAI substrate.** Promote the OpenAI-compatible client + shared types + completions/responses plumbing out of `pipelex/plugins/openai/` into a deliberate core module (e.g. `pipelex.openai_compat`); repoint gateway/portkey/openrouter/blackboxai/azure_rest (and the type-only imports in anthropic/mistral/google) at it. Pure move + import rewrite, still in core. _Checkpoint: suite green; no backend reaches into `pipelex/plugins/openai/` internals — the substrate is the only OpenAI surface they touch._-- **Phase 1 — registry seam, in-tree.** Add the per-family worker-factory registry; convert each `match` arm (OpenAI driver included — no special case) to a registration + the factory body; built-in backends register through the shared `pipelex.plugins` entry point. No packaging change, no behaviour change. _Checkpoint: full suite green with all backends in-repo but selected via the registry — proves the seam (mirrors Temporal Phase 1)._-- **Phase 2 — externalize selectively (optional).** Split out only the backends whose dependency weight or cadence justifies it, into per-vendor or bundle dists, each declaring its entry point and owning its extra. The OpenAI driver and the `openai_compat` substrate stay in core. _Checkpoint: base `pipelex` ships only always-on backends + the substrate; each external backend's suite green against the published base._diff --git a/wip/plugins/option-a-drop-cli-command-seam.md b/wip/plugins/option-a-drop-cli-command-seam.mddeleted file mode 100644index a4dbb86fa..000000000--- a/wip/plugins/option-a-drop-cli-command-seam.md+++ /dev/null@@ -1,115 +0,0 @@-# Option A — drop the CLI-command seam; Temporal ops as the `pipelex-temporal` console script (as-built)--> ✅ **IMPLEMENTED.** Committed on `refactor/Plugins-2` as `989c9beed` ("Refactor CLI command handling and introduce standalone pipelex-temporal console script"). All gates green: `make agent-check` (pyright 0 / mypy 0 over 2240 files / keyword-only), `make tb`, `make agent-test` ("All tests passed."), `pytest tests/integration/pipelex/temporal/` (156 passed, 4 xpassed = the Phase-3 baseline xdist markers), plus both `--help` smokes. This is the faithful as-built record — the basis Phases 4–5 build on.->-> **What it did:** removed the plugin CLI-command-contribution seam (the former decision **D3**) entirely. The two Temporal operational commands — `worker` and `setup-namespace` — now ship as a standalone `pipelex-temporal` console script (`[project.scripts]` → `pipelex.temporal.temporal_cli:app`), not harvested `pipelex` subcommands. This resolved the whole [critical finding](phase-3-critical-cli-harvest-fragility.md) by **removing the surface**, not hardening it: `pipelex --help` no longer loads config or scans entry points, and no plugin can shadow a core command or brick the recovery commands. The inference-backend / orchestrator / boot-slot seams are **untouched**.->-> Read [TODOS.md cold-start primer](../../TODOS.md#cold-start-primer-read-this-first-if-youre-new-to-the-session) for the plugin-seam vocabulary (registrar, `build_registrar`, `BUILTIN_PLUGINS`, slot-claims, D1–D7).--## Why this, in one screen--The seam treats "CLI command" as just another contribution type alongside "inference backend" and "orchestrator." It isn't — and that mismatch is the whole bug. The other contributions are consumed **lazily, at point-of-use, inside a booted Pipelex** (a backend when a model needs a worker; an orchestrator at execution time; a slot-claim at boot). The seam's **fail-loud** invariant is correct there. CLI commands are consumed **eagerly, at module-import of `pipelex/cli/_cli.py`, before anything boots, on every `pipelex` invocation including `--help`**. On that path fail-loud is a liability: one broken/colliding installed plugin (or an unreadable config) bricks every command — including `doctor`/`plugins`/`init`, the very commands you'd use to disable the bad plugin. The four facets in the critical doc and the "double discovery / import-time cost" item in [`phase-3-review-deferred.md`](phase-3-review-deferred.md) are all symptoms of this one decision.--The consumer that justified the seam is tiny and operational: `add_cli_command` has exactly **one** caller (the Temporal plugin), contributing exactly **two** commands — `worker` (start a long-running worker daemon) and `setup-temporal-namespace` (one-time namespace bootstrap). Neither is how a pipeline *runs* on Temporal — distributed execution goes through the orchestrator registry via `execution_mode`, which is untouched by any of this. Operational daemons are the textbook use case for a **console script** (`[project.scripts]`), which pip materializes into its own executable — no host-CLI harvest, no import-time discovery, no fragility. Pipelex already ships three (`pipelex`, `pipelex-agent`, `pipelex-dev`). And it is exactly where Phase 5 wants to land: when Temporal externalizes to `pipelex-temporal`, that dist owns its own `pipelex-temporal` console script natively, with no custom machinery to move.--**Net effect of this change:** deletes the harvest, all four facets, the double-discovery, and the per-invocation config-load + entry-point-scan on `--help`. Strictly less code, strictly less fragility, strictly better Phase-5 alignment. The only real cost is one user-facing rename (`pipelex worker` → `pipelex-temporal worker`) and a one-line Dockerfile change in the `pipelex-worker` repo (gated on the release — see the cross-repo stage).--## The design (as built)--A single grouped Temporal CLI exposed as a console script:--- New console script in `pyproject.toml`: `pipelex-temporal = "pipelex.temporal.temporal_cli:app"`.-- New module `pipelex/temporal/temporal_cli.py`: a `typer.Typer()` app registering two subcommands — `worker` and `setup-namespace`.-- The two command callables **move** from `pipelex/cli/commands/` to `pipelex/temporal/` (they are Temporal-owned and must travel together in Phase 5; absolute imports keep working from the new location):- - `pipelex/cli/commands/worker_cmd.py` → `pipelex/temporal/worker_cmd.py` (`worker_cmd`)- - `pipelex/cli/commands/setup_temporal_namespace_cmd.py` → `pipelex/temporal/setup_namespace_cmd.py` (`setup_namespace_cmd`)-- Resulting invocations: `pipelex-temporal worker [--no-sandbox --task-queue … --scope … --profile …]` and `pipelex-temporal setup-namespace [--dry-run --server …]`.--Both command callables are already import-light at module top (temporalio is pulled lazily inside their bodies), so importing `temporal_cli.py` when you run `pipelex-temporal` is import-light, and — crucially — `pipelex/cli/_cli.py` no longer references them at all, so `pipelex --help` cannot touch Temporal. The import-light *boot* guard is unaffected and still applies to boot.--**Design choices (as built):**--- *Grouped script vs two flat scripts.* Grouped `pipelex-temporal` (chosen) over flat `pipelex-worker` + `pipelex-setup-temporal-namespace`. Grouped maps 1:1 to the Phase-5 dist and avoids colliding with the `pipelex-worker` *repo/dist* name.-- *Move the command modules vs leave them in `pipelex/cli/commands/`.* Move (chosen) — Phase-5-aligned, "solid over quick." Cost is mechanical: update import + mock paths in one test (`test_setup_temporal_namespace_cmd.py`).-- *Subcommand name `setup-namespace` vs `setup-temporal-namespace`.* `setup-namespace` (chosen) — "temporal" is redundant under a `pipelex-temporal` group, and every embedding string changes anyway.--## Out of scope — do NOT unwind these--The valuable coupling inversion stays exactly as shipped. **Touch none of it:**--- The four inference-family `match` → `InferenceBackendRegistry` lookups (Phases 1–2).-- The bridge `match` → `OrchestratorRegistry.get_optional(mode=…)` dispatch (Phase 3).-- The `temporal.is_enabled` boot-slot claims + LIFO teardown via `_resolve_hub_slot` (Phase 3).-- `build_registrar`, `BUILTIN_PLUGINS`, the `pipelex.plugins` entry-point group for **runtime** contributions, `pipelex plugins list`, the denylist.--This change removes only the **CLI-command** contribution path. Everything else the seam does is consumed at boot/use where fail-loud is correct.--## Implementation steps (all done — retained as the record of what changed)--### Step 1 — Stand up the new Temporal CLI surface--1. Create `pipelex/temporal/temporal_cli.py`: `app = typer.Typer()`; register `worker_cmd` as `worker` and `setup_namespace_cmd` as `setup-namespace` (use `app.command(name="worker")(worker_cmd)` / `app.command(name="setup-namespace")(setup_namespace_cmd)`, or `@app.command` wrappers — match the existing Typer idiom in the repo). Keep it import-light at module top.-2. Move `pipelex/cli/commands/worker_cmd.py` → `pipelex/temporal/worker_cmd.py` and `pipelex/cli/commands/setup_temporal_namespace_cmd.py` → `pipelex/temporal/setup_namespace_cmd.py` (rename the callable `setup_temporal_namespace_cmd` → `setup_namespace_cmd`). Their internal imports (`pipelex.cli.cli_factory`, `pipelex.cli.error_handlers`, `pipelex.config`, `pipelex.pipelex`) are absolute and unaffected.-3. Add to `pyproject.toml` `[project.scripts]`: `pipelex-temporal = "pipelex.temporal.temporal_cli:app"`.--### Step 2 — Remove the seam--4. `pipelex/cli/_cli.py`: delete `_config_for_cli_harvest()`, `_register_discovered_cli_commands()`, and the module-level `_PLUGIN_COMMAND_NAMES = …`. Change `PipelexCLI.list_commands` to `return list(_CORE_COMMAND_ORDER)`. Update the `_CORE_COMMAND_ORDER` comment block (drop the "plugin-contributed commands are appended after" paragraph). Let `make fix-unused-imports` + `make agent-check` strip the now-dead imports (expected: `importlib`, `ValidationError`, `build_registrar`, `config_manager`, `PipelexConfig`, `TomlError`, possibly `log`).-5. `pipelex/plugins/registrar.py`: remove `add_cli_command`, the `CliCommand` NamedTuple, the `self.cli_commands` field (and its mention in the class docstring), and the `"cli command {name}"` contribution line. `plugins_cmd.py` renders `discoveries[].contributions` generically, so no change needed there — those lines simply stop appearing.-6. `pipelex/temporal/temporal_plugin.py`: remove the two `registrar.add_cli_command(...)` calls and the comment above them about declaring commands by `import_path`. Update the module + class docstrings (the bullet that says the plugin "contributes the … `worker` / `setup-temporal-namespace` CLI commands"). The plugin now contributes only orchestrators (always) and slot-claims/teardown (when `is_enabled`).--### Step 3 — Update in-code user-facing strings--7. `pipelex/temporal/tprl/namespace_check.py`: `PIPELEX_SETUP_CLI_COMMAND` → `"pipelex-temporal setup-namespace"`; update the two docstring mentions (~`:19`, `:165`).-8. `pipelex/temporal/exceptions.py` (~`:90`): docstring mention of `pipelex setup-temporal-namespace` → new invocation.-9. The two moved command modules' own docstrings + `Examples:` blocks: `pipelex worker …` → `pipelex-temporal worker …`; `pipelex setup-temporal-namespace …` → `pipelex-temporal setup-namespace …`.--### Step 4 — Tests--10. **Delete** `tests/unit/pipelex/cli/test_plugin_cli_command_harvest.py` — its entire subject (the harvest + `_config_for_cli_harvest`) is gone. Optionally replace with a ~5-line smoke test asserting `pipelex.temporal.temporal_cli.app` exposes `worker` + `setup-namespace` (`[c.name for c in app.registered_commands]`).-11. **Update** `tests/unit/pipelex/cli/test_setup_temporal_namespace_cmd.py`: import path `from pipelex.temporal.setup_namespace_cmd import setup_namespace_cmd` and all `mocker.patch("pipelex.temporal.setup_namespace_cmd.…")` targets; update the callable name if renamed.-12. Confirm the import-light boot guard test (the subprocess that blocks `temporalio`) still passes — it pins boot, not the harvest, so it should be untouched. Grep showed `test_plugin_cli_command_harvest.py` is the *only* harvest-coupled test.--### Step 5 — Docs--13. Mechanical prefix swap in **current** user docs (`pipelex worker` → `pipelex-temporal worker`; `pipelex setup-temporal-namespace` → `pipelex-temporal setup-namespace`; flags `--scope/--profile/--task-queue/--dry-run/--server` unchanged): `docs/distributed-execution/workers.md`, `docs/distributed-execution/temporal/index.md`, `docs/distributed-execution/cluster-setup.md`, `docs/features/distributed-execution.md`, `docs/running-locally-on-temporal.md`. **Do NOT touch** `docs/history/**`, `wip/history/**`, `docs/plans/**` (archival).-14. **Rewrite** `docs/under-the-hood/orchestrator-plugins.md`'s "CLI-by-import-path" section: the orchestrator SPI no longer harvests CLI commands; a plugin that wants an operational command ships its own console script (`[project.scripts]`). **Fold in the deferred-doc fix** here too: the line (~`:116`) claiming the Temporal plugin is "discovered via an entry point pipelex declares on itself" is factually wrong — it is in `BUILTIN_PLUGINS`, and `pyproject.toml` declares no `pipelex.plugins` self-entry-point. State the builtin-list mechanism + the real Phase-5 delta. (`docs/under-the-hood/inference-backend-plugins.md` does **not** mention the CLI seam — leave it.)-15. Sanity-check (almost certainly clean): no `docs/specs/**` or `conformance/**` command-surface entry references these invocations (verified — the grep found none), so no `make check-spec-links` concern. Re-verify after editing.--### Step 6 — Tracker bookkeeping--16. `TODOS.md`: mark **D3 superseded** by this doc (one line in the locked-decisions list + the CLI seam row of the cold-start primer). Note the CLI-command contribution path was removed; runtime contributions (backends/orchestrators) and `plugins list` are unchanged.-17. `phase-3-critical-cli-harvest-fragility.md`: add a top banner — **RESOLVED by removal**, pointing here.-18. `phase-3-review-deferred.md`: mark the "Double discovery + import-time cost" Low item **resolved** (the import-time harvest is gone); mark the "SPI doc factually wrong" Medium item resolved (folded into step 14).--### ✅ CHECKPOINT — verify gate (PASSED)--- ✅ `make agent-check` clean (pyright 0 / mypy 0 over 2240 files / ruff+plxt / keyword-only).-- ✅ `make tb` green — 5 passed (registrar shape changed — boot/config sanity).-- ✅ `make agent-test` green ("All tests passed.", exit 0).-- ✅ `.venv/bin/pytest tests/integration/pipelex/temporal/` — **156 passed, 4 xpassed** (the §14.5 Temporal gate; the 4 xpassed are the pre-existing xdist class-registration flakiness markers, matching the Phase-3 baseline exactly).-- ✅ Manual smoke: `.venv/bin/pipelex --help` lists core commands only — no `worker`/`setup-temporal-namespace`, no config load / entry-point scan; `.venv/bin/pipelex-temporal --help` lists `worker` + `setup-namespace`.-- ✅ Committed on `refactor/Plugins-2` as `989c9beed` (one-commit-per-checkpoint delivery model).--> **Note on the env:** materializing the new console script needs `uv sync --all-extras` (it regenerates `[project.scripts]` wrappers into `.venv/bin/`). That sync also brings the venv up to the already-committed `uv.lock` (it had drifted behind on `cryptography`/`mthds`); `uv.lock` itself is unchanged. Separately, `make cleanderived` deletes the gitignored generated fixture `tests/integration/pipelex/fixtures/_generated_model_sets.py` — regenerate it with `make regenerate-test-models-quiet` (alias `rtm`) before `make agent-check`, or pyright fails on the missing import.--## Cross-repo follow-up — GATED on the release that ships the script--This is the one downstream cost, and it is **release-gated** — do **not** push it ahead of the pipelex release that carries `pipelex-temporal`. Blast radius is exactly one repo (`pipelex-api-hosted`, `sandbox`, `pipelex-api` were checked and do **not** invoke the worker):--- `pipelex-worker/Dockerfile` (~`:25`): `CMD ["pipelex", "worker", "--no-sandbox"]` → `CMD ["pipelex-temporal", "worker", "--no-sandbox"]`.-- `pipelex-worker/Makefile` (~`:125`): `pipelex worker --no-sandbox --task-queue test` → `pipelex-temporal worker …`.--**Rule:** whoever bumps `pipelex-worker`'s `pipelex` pin to the release containing this change must flip these two lines in the **same** commit. Until that bump, the old pinned pipelex still has `pipelex worker`, so nothing breaks. This session is **pipelex-only** — capture the above as a standing follow-up (this section is that record; mirror it into the Phase-5 downstream-pin step in `TODOS.md` so it isn't lost).--## Open cleanups (investigate, note — do NOT expand scope)--- **`ensure_global_if_missing` / `load_base_config_dict`.** ⚠️ **Investigated — confirmed sole caller, FLAGGED for a follow-up (not removed here).** `_config_for_cli_harvest` (now deleted) was the **only** caller of both `config_manager.load_config(ensure_global_if_missing=False)` and `config_manager.load_base_config_dict()` (verified by grep across `pipelex/` + `tests/`). After this change: `load_base_config_dict()` has **zero** callers (dead method), and every `load_config` caller uses the default `ensure_global_if_missing=True` so the `False` branch (`config_loader.py:~260`) is dead. Both are in `pipelex/system/configuration/config_loader.py` (`load_base_config_dict` ~`:195`, the param ~`:205`/`:242`/`:260`). Left in place deliberately — removing the param means reshaping `load_config`'s signature + collapsing the branch, a config-loader edit out of scope for "drop the CLI seam." Neither breaks a gate (ruff/pyright don't flag unused public methods). **Follow-up:** delete `load_base_config_dict()` and simplify `load_config` to always-ensure-global.-- **Legacy `pipelex/temporal/worker_cli.py`.** A second, heavier worker entrypoint (`python -m pipelex.temporal.worker_cli`, the `configure` command) that imports `temporalio` at module top and duplicates worker-start logic. Out of scope here; note it as a unification candidate (it could become a thin alias of, or be folded into, the new `temporal_cli.py`). Don't fix unless trivial.--## Pointers--- Seam removal: `pipelex/cli/_cli.py` (harvest), `pipelex/plugins/registrar.py` (`add_cli_command`/`CliCommand`), `pipelex/temporal/temporal_plugin.py` (the two calls).-- Command callables to move: `pipelex/cli/commands/worker_cmd.py`, `pipelex/cli/commands/setup_temporal_namespace_cmd.py`.-- Strings: `pipelex/temporal/tprl/namespace_check.py` (`PIPELEX_SETUP_CLI_COMMAND`), `pipelex/temporal/exceptions.py`.-- Decision lineage: supersedes **D3** in [TODOS.md](../../TODOS.md); resolves [phase-3-critical-cli-harvest-fragility.md](phase-3-critical-cli-harvest-fragility.md) and two items in [phase-3-review-deferred.md](phase-3-review-deferred.md).diff --git a/wip/plugins/orchestration-mode-and-delivery-split.md b/wip/plugins/orchestration-mode-and-delivery-split.mddeleted file mode 100644index b32c6c319..000000000--- a/wip/plugins/orchestration-mode-and-delivery-split.md+++ /dev/null@@ -1,387 +0,0 @@-# Plan — split `PipelexExecutionMode` into `orchestration_mode` (open token) + `DeliveryMode` (closed enum)--**Status:** **ALL PHASES DONE — Phases 1 (core) + 2 (Temporal) + 3 (`pipelex-api`, THE gate) + 4 (Mistral plugin migration + bookkeeping) complete.** Phase 1: `_plugins` commit `bd4e52827` (+ docs `9c7a7785d`, + this tracker `919c09348`) on `feature/orchestration-mode-delivery-split`. Phase 2: `pipelex-temporal` commit `01d7dc3` on `feature/orchestration-mode-delivery-split`. Phase 3: `pipelex-api` commit `dd84a9a` on `feature/orchestration-mode-delivery-split` (cut off `06cd5e6`). All three with all gates green + clean-context `/code-review` at their checkpoints (Phase 3 review: **no must-fix issues**). The §8 `/start`-honesty conformance risk is **resolved** (honest 400 is safe, landed). **Phase 4 DONE** — the external `pipelex-mistralai-workflows` plugin migrated to the new core surface (commit `0eb3878` on `feature/Mistral-native`): deleted-enum import + registration under the deleted member + own `execution_mode` wire field + old `run` signature → `orchestration_mode`/`delivery`/new protocol, `supports_fire_and_forget=False`. Gates green + clean-context `/code-review` (one must-fix — a stale README — applied). **DEVIATION:** the plugin pins core editable at `../_workflows` (`feature/mistralai-2x-bump`), NOT the orchestration-split worktree, so Phase 4 first merged `feature/orchestration-mode-delivery-split` → `feature/mistralai-2x-bump` (`_workflows` merge `a1448ad8f`, conflict-free, core gates green). `make check-spec-links` was NOT triggered (no `docs/specs/` heading or `conformance/` test asserts mode values). See **"### Phase 4 — as-built"** in §10 for the full record. The one carry-forward from Phase 3 (the `pipelex-api` editable-pin → pushed-git-SHA flip before any PR) is still open and now joined by the `_workflows`/`_plugins` doc-reconciliation note (see §10 Phase 4 as-built). Design **locked** in discussion (2026-06-22). This is **Direction A** — the real two-axis split across all three repos (`pipelex` core → `pipelex-temporal` → `pipelex-api`), not the API-edge-only reshape (Direction B, rejected: it relocates the smell to the boundary instead of removing it). It **supersedes** the dual-mode framing in the two follow-on plans `wip/plugins/execute-per-request-execution-mode.md` and `wip/plugins/orchestrator-dispatched-validate.md` — both shipped a working surface on top of the *flat* `PipelexExecutionMode`; this plan removes the flatness those plans worked around (`_async_start_mode`, the `/execute` fire-and-forget 400, the `ApiConfig` f&f field-validator, the `/validate` double-registration). Tracker: `_plugins/TODOS.md`. Parent effort: `wip/plugins/orchestrator-agnostic-runner-and-flavors.md`.--> **START HERE (cold start).** Read this whole doc. The change is conceptual, not big-mechanical-only: today one enum `PipelexExecutionMode {DIRECT, TEMPORAL_BLOCKING, TEMPORAL_FIRE_AND_FORGET, MISTRAL_NATIVE}` conflates **two orthogonal axes** — *which orchestrator runs the pipe* and *whether the caller waits*. We split them: an **open string token** `orchestration_mode` (core owns only `"direct"`; plugins own `"temporal"`, `"mistralai-workflows"`) and a **closed enum** `DeliveryMode {BLOCKING, FIRE_AND_FORGET}` that endpoints set themselves and never receive from a caller. Delivery becomes a parameter to `OrchestratorProtocol.run`, so the two Temporal orchestrators collapse into one and the `/validate` double-registration disappears. Lands core → temporal → api, one repo / branch / commit per phase, clean-context `/code-review` at each checkpoint — same discipline as the rest of the effort.-------## 1. The ask + locked decisions--**The ask (user):** the API surface for choosing *how* a call to `/execute`, `/start`, `/validate` runs is wrong. `PipelexExecutionMode` carries two things at once — the backend and the wait-semantics — and that conflation forced three separate workarounds (`_async_start_mode`, the `/execute` f&f 400, the `/validate` collapse). Re-cut it into the right enum(s) passed as args.--**Why the conflation is the root cause.** The blocking-vs-fire-and-forget decision is **already expressed by which endpoint you call** — `/execute` and `/validate` are synchronous, `/start` is fire-and-forget. Encoding it a *second* time inside the mode enum is the duplication, and every smell is a workaround for it:--- `_async_start_mode` (`pipeline.py:109-134`) maps `temporal_blocking → temporal_fire_and_forget` because `/start` has to re-derive the delivery axis the enum shouldn't carry.-- `/execute` rejects `temporal_fire_and_forget` with a 400 (`ErrorType.FIRE_AND_FORGET_NOT_SUPPORTED`) — refusing a delivery value the caller should never have been able to express.-- `ApiConfig._reject_fire_and_forget_default` (`api_config.py:54-74`) rejects a *configured* f&f mode — same problem on the config side.-- `TemporalPlugin.register` registers the **same** bundle validator under **both** temporal modes (`temporal_plugin.py:136-138`, decision D of the validate plan) — collapsing the axis because validate doesn't have it either.--Three endpoints, three kludges, one root cause: a wait-semantics dimension the caller/deployment should never receive.--**Locked decisions (do not re-litigate; raise explicitly before deviating):**--- **D-A — Two axes, two types.** Replace `PipelexExecutionMode` with `orchestration_mode` (the backend) + `DeliveryMode` (the wait-semantics). They are orthogonal: orchestration is per-call/config-chosen; delivery is endpoint-intrinsic.-- **D-B — `orchestration_mode` is an OPEN string token, not a closed enum.** Core defines only `"direct"` (its single built-in orchestrator). Each plugin defines and registers its own token *where its orchestrator lives* (`pipelex-temporal` owns `"temporal"`; `pipelex-mistralai-workflows` owns `"mistralai-workflows"`). The registry is keyed by `str`; validation is *"is a mode registered?"* — a lookup miss raises `MissingOrchestratorError` — never *"is it in an enum?"*. **This is the deliberate, justified exception to the repo's StrEnum-everywhere standard:** that standard exists to make the linter scream when a *closed* value is added and a branch forgotten; a plugin-contributed set is genuinely open, exhaustive matching is impossible, and the dict-lookup-or-raise *is* the correct unknown-handling. It is also the only choice faithful to the effort's north star (*the public base names no orchestrator*) — a closed `{DIRECT, TEMPORAL, MISTRAL}` enum would re-introduce the exact core→plugin coupling Phase C removed.-- **D-C — `DeliveryMode {BLOCKING, FIRE_AND_FORGET}` is a closed core StrEnum, endpoint-intrinsic, never on the wire or in config.** Endpoints set it: `/execute` → `BLOCKING`, `/validate` → `BLOCKING`, `/start` → `FIRE_AND_FORGET`. It stays a real StrEnum with exhaustive `match`/`case` (the standard applies — this axis is genuinely closed; a future `STREAMING` would be core's call, never a plugin's).-- **D-D — Delivery is a parameter to dispatch, not a registry key.** `OrchestratorProtocol.run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)`. Each orchestrator honors `delivery` per its nature. This collapses the two Temporal orchestrators into one (`run` branches on `delivery`: `temporal_pipe_run.run(...)` for blocking vs `.start(...)` for f&f) registered **once** under `"temporal"`, and collapses the `/validate` double-registration to a single `"temporal"` validator.-- **D-E — `/start` is HONEST, not degrading.** An orchestrator advertises `supports_fire_and_forget: bool` (`direct`=False, `temporal`=True, `mistralai-workflows`=False). `/start` checks the *resolved* mode's orchestrator and, when it can't do genuine async, returns a 4xx — *"this deployment has no async-capable orchestration; use /execute"* — instead of silently running blocking and acking. (Replaces today's `_async_start_mode` identity-arm degrade.) **Consequence to verify (§8):** on the OSS base (`orchestration_mode = "direct"`) every `/start` now 4xxs — confirm the MTHDS-Protocol/conformance position on `/start` over a non-async backend before committing.-- **D-F — Generic `MissingOrchestratorError` / `MissingBundleValidatorError` message.** *"No orchestrator registered for mode '{mode}'; is its plugin installed?"* — fully decoupled. Drop the per-mode `match`-on-enum install hints (`requires_pipelex_temporal` etc.). Core no longer names `temporal`/`mistral` in its error text.-- **D-G — Rename the Mistral token `"mistral_native"` → `"mistralai-workflows"`** for consistency with the package name (`pipelex-mistralai-workflows`). Mistral is a deferred/secondary plugin; this is just the token string + any doc reference.-- **D-H — Rename the wire field + config key `execution_mode` → `orchestration_mode`** (and `allow_request_execution_mode_override` → `allow_request_orchestration_mode_override`). Breaking wire/config change — fine per the workspace "no backward compatibility" principle; note in changelogs.-------## 2. Current state (anchors verified 2026-06-22)--**Core (`_plugins/pipelex/`):**--- `runtime_bridge/execution_mode.py` — `PipelexExecutionMode(StrEnum)`: the 4 members + three properties to delete: `requires_pipelex_temporal` (`30`), `requires_mistral_workflows_extra` (`38`), `is_fire_and_forget` (`46`).-- `plugins/orchestrator_registry.py` — `OrchestratorProtocol.run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput` (`21`); `OrchestratorRegistry(orchestrators: dict[PipelexExecutionMode, OrchestratorProtocol])` with `get_optional(*, mode)` / `has(*, mode)` / `modes` (`24-42`).-- `plugins/bundle_validator_registry.py` — the `/validate` twin (`BundleValidatorProtocol`, `BundleValidatorRegistry`), same `dict[PipelexExecutionMode, …]` keying (per the validate plan's V0 as-built).-- `plugins/registrar.py` — stores `orchestrators` / `bundle_validators` typed `dict[PipelexExecutionMode, …]` (`111-112`) + the `_*_sources` mirrors (`122-123`); `add_orchestrator(*, mode, …)` (`167`) and `add_bundle_validator(*, mode, …)` (`179`).-- `runtime_bridge/exceptions.py` — `MissingOrchestratorError._build_message` (`27-40`) and `MissingBundleValidatorError._build_message` (`61-74`) both `match` on enum members for per-mode install hints (→ generic, D-F).-- `runtime_bridge/direct_orchestrator.py` — `DirectOrchestrator.run(*, pipe_job, delivery_assignment)` (`29`), returns `serialize_completed_output(pipe_output, workflow_id=None)` (`46-49`). Add `supports_fire_and_forget = False` + the `delivery` param (ignored — in-process always blocks).-- `plugins/direct/direct_plugin.py` — `register` calls `add_orchestrator(mode=DIRECT, …)` + `add_bundle_validator(mode=DIRECT, …)` (`20-22`). `DIRECT` becomes the `"direct"` string constant.-- `runtime_bridge/bridge.py` — `run_pipe_via_bridge` (`51`): `is_direct = input_payload.execution_mode is PipelexExecutionMode.DIRECT` (`85`); dispatch `get_orchestrator_registry().get_optional(mode=input_payload.execution_mode)` → `orchestrator.run(pipe_job, delivery_assignment)` (`92-95`); `_validate_input` checks `execution_mode is TEMPORAL_FIRE_AND_FORGET` for the delivery-target requirement (`164`). This is the **Tier-3 bridge** for *other* host runtimes (Mistral Workflows, raw Temporal) — pipelex-api's routes do **not** go through it (they hit the registry directly), but it must split consistently and pass `delivery`.-- `runtime_bridge/payloads.py` — `PipelexPipeRunInput.execution_mode: PipelexExecutionMode = DIRECT` (`30`). Splits into `orchestration_mode: str = "direct"` + `delivery: DeliveryMode = BLOCKING`. `PipelexPipeRunOutput` (`34-52`) already carries `workflow_id` / `is_completed` and needs no change.--**Temporal (`pipelex-temporal/pipelex_temporal/`):**--- `temporal_orchestrators.py` — `TemporalBlockingOrchestrator.run` calls `temporal_pipe_run.run(...)` (await completion, `58`) and reports `make_workflow_id(...)` (`69`); `TemporalFireAndForgetOrchestrator.run` calls `temporal_pipe_run.start(...)` (return immediately, `91`) and builds an `is_completed=False` output (`96-103`). **The only difference is `run` vs `start`** → one `TemporalOrchestrator` whose `run` branches on `delivery`. Both already guard the `pipelex[temporal]` extra via `_require_temporal_extra` (`32-40`).-- `temporal_plugin.py` — `register` (`127`): two `add_orchestrator` calls under the two temporal modes (`128-129`), two `add_bundle_validator` calls under both temporal modes (`136-138`), the F3 mapper (`145`, unchanged), and the `boot_orchestrator == self.name` slot-claims (`152-157`, unchanged — `boot_orchestrator` is the *separate* hub-slot gate, orthogonal to the orchestration token even though both read `"temporal"`).-- `temporal_bundle_validator.py` — `TemporalBundleValidator` (validate is inherently blocking; register once under `"temporal"`).--**API (`pipelex-api/api/`):**--- `api_config.py` — `ApiConfig.execution_mode: PipelexExecutionMode` + `allow_request_execution_mode_override: bool` (`51-52`); the `_reject_fire_and_forget_default` field-validator to **delete** (`54-74`); `resolve_execution_mode(requested, *, config)` (`103-121`) → rename `resolve_orchestration_mode`, logic unchanged (string compare; 403 on forbidden override); `api.toml` ships `execution_mode = "direct"`.-- `schemas/models.py` — `PipelineApiExtras.execution_mode: PipelexExecutionMode | None` (`153`); `PipelexApiExecuteRequest.execution_mode` (`204`); `_EXECUTION_MODE_DESCRIPTION` (`110-115`, lists the four enum values → open-token wording). Rename field → `orchestration_mode`.-- `routes/pipelex/pipeline.py` — `_async_start_mode` (`109-134`, **delete**); `ApiRunner.execute` resolves + rejects f&f (`208-214`, drop the rejection) and injects `_OrchestratorPipeRun` (`222`); `ApiRunner.start` calls `_async_start_mode(resolve_…(…))` (`283`, → resolve + capability-check); `_OrchestratorPipeRun` / `_pipe_output_from_run_output` (`73-153`, thread `delivery`); `validate_verdict` (`343-390`, rename the mode param). Routes `execute` (`530`) / `start` (`572`) thread the renamed extra.-- `error_types.py` — `EXECUTION_MODE_OVERRIDE_FORBIDDEN` (`23`, rename) and `FIRE_AND_FORGET_NOT_SUPPORTED` (`32`, **delete**); add the honest-`/start` rejection type (D-E).-- `routes/pipelex/validate.py` — threads `execution_mode` into `validate_verdict` (rename to `orchestration_mode`).-- The committed OpenAPI artifact (`docs/openapi/pipelex-api.openapi.yaml`) re-exports via `make openapi-export`; drift-gated by `make openapi-check`.--> **Mechanical-usage inventory** (every stray `PipelexExecutionMode` / `execution_mode` / mode-literal site across the three repos) is being swept and is appended in §9 — run that grep before declaring a phase done.-------## 3. The design (locked)--### 3.1 Core types (`_plugins/pipelex/runtime_bridge/`)--- **`orchestration_mode.py`** (replaces `execution_mode.py`):- - `OrchestrationMode: TypeAlias = str` — a *semantic* alias documenting intent at registry/protocol signatures (`dict[OrchestrationMode, …]`), assignment-compatible with plain `str` so plugins pass raw strings with no casts. (Chose the alias over `NewType` deliberately: a `NewType` forces `OrchestrationMode("temporal")` casts at every plugin boundary for zero validation benefit — the registry is the validator. See §8.)- - `DIRECT_ORCHESTRATION_MODE: Final[str] = "direct"` — core's one built-in token, referenced by core code instead of the literal.-- **`delivery_mode.py`** — `class DeliveryMode(StrEnum): BLOCKING = "blocking"; FIRE_AND_FORGET = "fire_and_forget"`. Closed; exhaustive `match`/`case`; no default arm.-- **`OrchestratorProtocol`** (`plugins/orchestrator_registry.py`):- - `async def run(self, *, pipe_job, delivery_assignment, delivery: DeliveryMode) -> PipelexPipeRunOutput`- - `supports_fire_and_forget: bool` — capability the runner reads *before* dispatch (D-E). A `@property` or a plain class attribute; the protocol declares it as a read-only attribute.-- **`OrchestratorRegistry`** / **`BundleValidatorRegistry`** — re-key `dict[OrchestrationMode, …]` (i.e. `dict[str, …]`); `get_optional(*, mode: OrchestrationMode)` / `has`. `BundleValidatorProtocol.validate_bundles` stays blocking — pass no `delivery` (or `DeliveryMode.BLOCKING` if uniformity reads better; validate has no async variant).-- **`PluginRegistrar`** — `orchestrators` / `bundle_validators` + `_*_sources` become `dict[str, …]`; `add_orchestrator(*, mode: OrchestrationMode, …)` / `add_bundle_validator(*, mode: OrchestrationMode, …)`. The `_add` dup-guard is unchanged (works on any hashable key).-- **`Missing*Error`** — drop the `match`; generic message keyed on the string (D-F). Optionally keep a one-line special case for `"direct"` (*"DIRECT is core; this indicates a boot/discovery problem"*) via a plain `if mode == DIRECT_ORCHESTRATION_MODE:` — now a string compare, not an enum, so it does not offend the no-`==`-on-enums rule.-- **`payloads.py`** — `PipelexPipeRunInput`: `execution_mode` → `orchestration_mode: str = DIRECT_ORCHESTRATION_MODE` + `delivery: DeliveryMode = DeliveryMode.BLOCKING`.-- **`bridge.py`** — `is_direct = input_payload.orchestration_mode == DIRECT_ORCHESTRATION_MODE`; pass `delivery=input_payload.delivery` into `orchestrator.run(...)`; `_validate_input` checks `input_payload.delivery is DeliveryMode.FIRE_AND_FORGET` for the delivery-target requirement.-- **`DirectOrchestrator`** — `supports_fire_and_forget = False`; `run(*, pipe_job, delivery_assignment, delivery)` accepts `delivery` for protocol uniformity and ignores it (in-process always blocks; `/start` never reaches it because the capability check rejects first — the bridge/Tier-3 path that *could* pass f&f to direct simply blocks, see §8).--### 3.2 Temporal plugin (`pipelex-temporal`)--- **One `TemporalOrchestrator`** (`temporal_orchestrators.py`) — `supports_fire_and_forget = True`; `run` branches:- - `DeliveryMode.BLOCKING` → `temporal_pipe_run.run(...)` then `make_workflow_id(...)` (today's `TemporalBlockingOrchestrator` body).- - `DeliveryMode.FIRE_AND_FORGET` → `temporal_pipe_run.start(...)` → `is_completed=False` output (today's `TemporalFireAndForgetOrchestrator` body).- - Keep the `_require_temporal_extra` guard and the `WorkflowExecutionError` wrapping in both arms.-- **`TEMPORAL_ORCHESTRATION_MODE: Final = "temporal"`** defined in `pipelex-temporal` (the token lives with the orchestrator). Distinct constant from the plugin's `name` / `boot_orchestrator` value even though both equal `"temporal"` — they are different concepts (registry key vs hub-slot gate).-- **Register once** — `add_orchestrator(mode=TEMPORAL_ORCHESTRATION_MODE, orchestrator=TemporalOrchestrator())`; `add_bundle_validator(mode=TEMPORAL_ORCHESTRATION_MODE, validator=TemporalBundleValidator())`. The two-mode duplication is gone. F3 mapper + boot-slot claims unchanged.--### 3.3 API (`pipelex-api`)--- **`ApiConfig`** — `orchestration_mode: str` + `allow_request_orchestration_mode_override: bool`; **delete** `_reject_fire_and_forget_default` (no f&f token can be configured anymore). `resolve_orchestration_mode(requested: str | None, *, config) -> str` — same default/override/403 logic. `api.toml`: `orchestration_mode = "direct"`, `allow_request_orchestration_mode_override = false`.-- **Wire** (`schemas/models.py`) — `PipelineApiExtras.orchestration_mode: str | None`; `PipelexApiExecuteRequest.orchestration_mode`; description reworded for the open token (*"e.g. `direct`, `temporal`; plugin-provided modes also accepted; an unregistered mode is refused at dispatch"*; drop the `temporal_fire_and_forget` value — no such token exists).-- **`/execute`** — resolve `orchestration_mode`; **no f&f rejection** (delete the `is_fire_and_forget` branch + `FIRE_AND_FORGET_NOT_SUPPORTED`); dispatch via `_OrchestratorPipeRun(orchestrator, delivery=DeliveryMode.BLOCKING)`.-- **`/start`** — resolve `orchestration_mode`; get orchestrator; **capability check (D-E):** `if not orchestrator.supports_fire_and_forget: raise_<4xx>(…, error_type=START_REQUIRES_ASYNC_ORCHESTRATION)`; dispatch with `delivery=DeliveryMode.FIRE_AND_FORGET`. **Delete `_async_start_mode`.**-- **`/validate`** — `validate_verdict(*, …, requested_orchestration_mode)`; resolve + dispatch through the validator registry (blocking). Verdict-as-value unchanged.-- **`error_types.py`** — rename `EXECUTION_MODE_OVERRIDE_FORBIDDEN` → `ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN`; **delete** `FIRE_AND_FORGET_NOT_SUPPORTED`; **add** `START_REQUIRES_ASYNC_ORCHESTRATION` (D-E). Status code: §8.-- **OpenAPI** — `make openapi-export`; the request schemas now advertise `orchestration_mode`.-------## 4. Phased implementation + checkpoints--Each phase: own repo, own branch, own commit, gates green, clean-context `/code-review` (hand the reviewer only the diff/SHA, never this plan). Ordering is load-bearing — core must land before its consumers.--### Phase 1 — core split (`pipelex`, `_plugins/`) ✅ DONE (`bd4e52827`)--Add `OrchestrationMode` (open `str` alias + `DIRECT_ORCHESTRATION_MODE`) and `DeliveryMode`; re-key both registries + the registrar stores on `str`; add `delivery` + `supports_fire_and_forget` to `OrchestratorProtocol`; update `DirectOrchestrator`; generic `Missing*Error`; split `PipelexPipeRunInput`; update `bridge.py` dispatch + `_validate_input`; delete `PipelexExecutionMode` and its three properties. Sweep §9. Gates: `make agent-check`, `make tb` (boot builds the re-keyed registries), `make agent-test`.--> **🛑 Checkpoint 1** — core green; registries keyed by string; boot still builds them; no orchestrator named in core error text. Capture an as-built (final type names/locations, the `OrchestrationMode` alias decision, the `supports_fire_and_forget` shape). Clean-context `/code-review` on the core diff.--### Phase 2 — Temporal collapse (`pipelex-temporal`) · **MAJOR** ✅ DONE (`01d7dc3`)--Collapse the two orchestrators into `TemporalOrchestrator` (delivery-branch + `supports_fire_and_forget=True`); define `TEMPORAL_ORCHESTRATION_MODE`; register orchestrator + validator **once** under `"temporal"`; F3 mapper + boot-slot claims unchanged. Update the in-memory integration tests to exercise both delivery values through the single orchestrator. Gates: `make agent-check`, `make agent-test` against the Phase 1 core.--> **🛑 Checkpoint 2** — the seam has a real consumer through the new shape: one `"temporal"` arm serving both deliveries; the base still has no idea Temporal exists. As-built + clean-context `/code-review` on both diffs.--### Phase 3 — API (`pipelex-api`) · **THE gate**--`ApiConfig` rename + drop the f&f validator + `resolve_orchestration_mode`; wire-model rename + open-token description; routes — delete `_async_start_mode`, `/execute` drop f&f-reject (delivery=BLOCKING), `/start` honest capability-reject (delivery=FIRE_AND_FORGET), `/validate` param rename; `error_types` changes; `make openapi-export`; docs + changelog. Sweep §9. Gates: `make agent-check`, `make agent-test`, `make openapi-check`.--> **🛑 Checkpoint 3** — `/execute`, `/start`, `/validate` all take `orchestration_mode`; delivery is endpoint-set; `/start` honestly rejects non-async modes; all the conflation workarounds are gone; OpenAPI in sync. **Verify the `/start`-honesty conformance question (§8) here.** As-built + clean-context `/code-review` on the API diff.--### Phase 4 — Mistral plugin migration (`pipelex-mistralai-workflows`) + bookkeeping ✅ DONE (`0eb3878`)--> **✅ DONE — see "### Phase 4 — as-built" in §10 for the full record** (incl. the load-bearing deviation: the plugin pins `../_workflows`, not the split worktree, so the split was first merged into `feature/mistralai-2x-bump`). The anchors below were re-derived against the live tip and were accurate; the migration matched this scope.--**⚠️ Scope correction (2026-06-22, from the Phases 1–2 `/code-review`): the Mistral work is NOT "token string + doc refs only" — that earlier estimate was wrong.** The external `pipelex-mistralai-workflows` plugin pins core editable, so it is already **red against Phase 1 core**: it carries live *code* that breaks under the `execution_mode → orchestration_mode` rename (`extra="forbid"`) + the `PipelexExecutionMode` deletion. This is the same Phase-2 shape the Temporal plugin took — a real code migration, not a string swap. **Anchors verified against tip `e508e73` (re-derive before starting — the tip will have moved):**--- `mistral_plugin.py:17` + `streaming.py:28` — `from pipelex.runtime_bridge.execution_mode import PipelexExecutionMode` → **`ImportError`** (module deleted). Re-point to `orchestration_mode` / `delivery_mode`.-- `mistral_plugin.py:31` — `add_orchestrator(mode=PipelexExecutionMode.MISTRAL_NATIVE, …)` registers under the **deleted enum member** → migrate to `add_orchestrator(mode="mistralai-workflows", …)` (the D-G token; the `"mistral_native"` literal lives at `mistral_plugin.py:21`).-- `mistral_orchestrator.py` — `MistralNativeOrchestrator.run` must adopt the new `OrchestratorProtocol.run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)` signature and declare `supports_fire_and_forget = False` (Mistral-native is blocking-only).-- `streaming.py:77,117,121` — its own `execution_mode: str` wire field + `input_payload.execution_mode` read + `is PipelexExecutionMode.DIRECT` compare → rename to `orchestration_mode`, drop the enum compare (string-compare `== DIRECT_ORCHESTRATION_MODE` or branch on `delivery`).-- `primitives/__init__.py`, `primitives/pipe_run.py`, `primitives/scoped_library.py` — docstring/comment/`mistral_native_…` string refs (the genuinely doc-only part).--**Then the bookkeeping:** update `_plugins/TODOS.md` (Phase C as-built + follow-on section), mark the two superseded sibling plans, `pipelex-api` `CHANGELOG.md`, and `docs/specs/` / `conformance/` if any heading or test asserts mode values (run `make check-spec-links`).-------## 5. Cross-repo state, pins, gates--- **Verify state FIRST (cold start).** Before cutting any branch, run the effort's orientation ritual — [`wip/plugins/resume-and-verify-state.md`](resume-and-verify-state.md) — and confirm each repo's working tree is clean and sits on the tip `_plugins/TODOS.md` "Current branches" names. This plan was written against those tips; if they have moved, re-derive the §2 / §9 anchors against the new tip rather than trusting the line numbers.-- **Branches:** cut fresh per phase off the verified current tips (`_plugins` core worktree; `pipelex-temporal`; `pipelex-api` off the `feature/Execute-per-request-mode` tip — this work stacks on the landed follow-ons it supersedes).-- **Pins:** `pipelex-api` and `pipelex-temporal` both pin `pipelex = { path = "../_plugins", editable = true }`, so a core edit is live in both immediately. `pipelex-api` does **not** depend on `pipelex-temporal` (Phase C dropped it) — test the API split with a stub orchestrator/validator registered for a non-`direct` test token; never import `temporalio` in `pipelex-api`. For a `pipelex-api` PR, the editable pin breaks CI — pin core to a git+https SHA + relock, flip back at release (the PR #27 `c60be1f` gotcha).-- **Gates:** core → `make agent-check` / `make tb` / `make agent-test`; temporal + api → `make agent-check` / `make agent-test`; api also `make openapi-check`; conformance → `make check-spec-links` if a spec heading is touched.-------## 6. Spec / conformance--- **No verdict/result wire contract changes** — `/validate`'s 200-always + `is_valid` discriminant + `validation_errors[]`, and the `/execute` / `/start` result shapes, are all backend-independent and untouched. The only wire change is the **request** field rename `execution_mode → orchestration_mode` (a pipelex-api extension field, not protocol-core).-- **The one real conformance question is D-E's `/start` honesty** (§8): if the MTHDS-Protocol conformance suite asserts `/start` returns a `202` on the reference implementation's default (direct) config, honest-reject breaks it. Resolve before Phase 3 lands.-- If any `docs/specs/` heading documents the mode values or the per-endpoint delivery behavior, update it + its `> Verified by:` link and run `make check-spec-links`.-------## 7. Testing--- **Core (Phase 1):** registry/registrar keyed by string (register + get + dup under arbitrary tokens incl. a synthetic `"acme"`); `MissingOrchestratorError` generic message for an unregistered token + the `"direct"` boot-bug special case; `DeliveryMode` exhaustive; `OrchestratorProtocol` round-trip with `delivery` + `supports_fire_and_forget`; `DirectOrchestrator.supports_fire_and_forget is False`; `PipelexPipeRunInput` defaults (`orchestration_mode="direct"`, `delivery=BLOCKING`); bridge `_validate_input` requires a delivery target iff `delivery is FIRE_AND_FORGET`. `make tb` green (boot builds the re-keyed registries).-- **Temporal (Phase 2):** in-memory-Temporal integration — the single `TemporalOrchestrator` returns a completed output for `delivery=BLOCKING` and an `is_completed=False` + `workflow_id` for `delivery=FIRE_AND_FORGET`; one registration serves both; validator registered once.-- **API (Phase 3, no temporal import):** `/execute` dispatches blocking with no f&f branch; `/start` honestly 4xxs when the resolved mode's stub orchestrator has `supports_fire_and_forget=False`, and acks when it's `True`; override policy on string tokens (default honored, different+allowed honored, different+forbidden → 403); an unregistered requested token → `MissingOrchestratorError` at dispatch; `/validate` unchanged verdict mapping; OpenAPI advertises `orchestration_mode`.-------## 8. Open sub-decisions / risks for the implementing session--- **[RISK — ✅ RESOLVED 2026-06-22, before Phase 3] `/start` honesty vs MTHDS-Protocol conformance.** D-E makes the runner's `/start` 4xx on a direct-only deployment (the OSS default). **Investigated and cleared:** (1) the protocol spec `docs/specs/pipelex-mthds-protocol.md` (§"operations") explicitly **defers `execute`/`start` artifacts to pipelex-api's OpenAPI + run-lifecycle docs** — it details only the `validate` and `models` artifacts, so the protocol mandates **no** `/start` status code; (2) the `conformance/tests/pipelex_api/` suite exercises only `validate` / `models` / `version` — **no test POSTs to the runner `/v1/start`**; every conformance `202` (`conformance/tests/pipelex_platform/test_runs_results.py`, `docs/specs/pipelex-platform-api.md`) is the **platform's run-results polling** (a RUNNING run → 202), a different endpoint untouched by the runner's dispatch semantics; (3) the platform's `/v1/start` (wallet-gated front door) is distinct from the runner's `/start`, and the hosted platform runs the `temporal` orchestration mode (async-capable → no 4xx). **Net: honest 4xx on a direct-only base breaks no conformance test and no protocol clause.** Proceed with D-E; status code **400** (§ next bullet). The only consumer to update is pipelex-api's own `/start` tests + its OpenAPI/docs (owned by Phase 3).-- **Honest-`/start` status code.** `400` (symmetric with the deleted `FIRE_AND_FORGET_NOT_SUPPORTED`), `409 Conflict` (request conflicts with deployment capability), or `422`. Recommend `400` with a crisp message; flag `409` as the more semantically precise alternative.-- **`OrchestrationMode` shape.** Recommended: `TypeAlias = str` + a `Final` `"direct"` constant (open, cast-free for plugins). Alternative: a `NewType` (stronger signatures, but forces casts at every plugin boundary for no validation gain since the registry validates). Decide in Phase 1; record in the as-built.-- **`supports_fire_and_forget` location.** On the orchestrator (recommended — it owns its capability) vs registry-level metadata. Keep it on the protocol unless a second consumer needs it elsewhere.-- **Unknown requested token validation timing.** Let dispatch raise `MissingOrchestratorError` (recommended — single validation point, keeps `resolve_orchestration_mode` decoupled from the hub) vs an early `get_orchestrator_registry().has(...)` check in resolve (earlier 4xx but couples config to the hub). Default: dispatch-time.-- **[RESOLVED 2026-06-22 — bridge-level guard added] Tier-3 bridge + incompatible (mode, delivery).** A raw bridge caller could pass `delivery=FIRE_AND_FORGET` to `direct`. The original disposition was "let it block — `DirectOrchestrator` blocks regardless." **The combined Phases 1–2 `/code-review` found that understated the harm:** with a valid `delivery_assignment_dump` the request passes `_validate_input`'s delivery-target gate, then `DirectOrchestrator.run` ignores `delivery`, runs to completion, and returns `is_completed=True` — so the caller **blocks** for the full run AND its delivery target is **never invoked** while completion is **falsely acked**. **Decided: add the bridge-level capability guard** (the Tier-3 counterpart of `/start`'s D-E check), not "let it block." The bridge now resolves the orchestrator up front and `_validate_input` rejects `FIRE_AND_FORGET` on any orchestrator with `supports_fire_and_forget = False` (raises `PipelexBridgeDispatchError`), regardless of whether a delivery target was supplied. As-landed shape + tests: see the post-checkpoint hardening note in §10. (This is pure core/`_plugins` work; pipelex-api routes hit the registry directly, not the bridge, so Phase 3 is unaffected.)-- **`boot_orchestrator` vs the `"temporal"` token.** Both strings equal `"temporal"` but are distinct concepts (hub-slot boot gate vs orchestrator-registry key). Keep separate named constants; do not unify them just because the values coincide today.-------## 9. Mechanical-usage inventory--> Swept across `_plugins/`, `pipelex-temporal/`, `pipelex-api/`. Actionable per-repo, keyed to the §3 decisions. **Re-run the grep at each phase boundary — do not trust a stale list:** `grep -rn 'PipelexExecutionMode\|execution_mode\|is_fire_and_forget\|requires_pipelex_temporal\|requires_mistral_workflows\|temporal_blocking\|temporal_fire_and_forget\|mistral_native' <repo>`.--### Core (`_plugins/pipelex/`)--**Source — split / rename / delete:**-- `runtime_bridge/execution_mode.py` → **replace** with `orchestration_mode.py` (`OrchestrationMode` alias + `DIRECT_ORCHESTRATION_MODE`) and `delivery_mode.py` (`DeliveryMode`). Delete the enum + its 3 properties.-- `runtime_bridge/payloads.py:16,30` — `PipelexPipeRunInput.execution_mode` → `orchestration_mode: str` + `delivery: DeliveryMode`.-- `runtime_bridge/bridge.py:36,85,92,94,164` — import; `is_direct` string-compare; registry dispatch; `MissingOrchestratorError`; `_validate_input` f&f-target check → `delivery is FIRE_AND_FORGET`; pass `delivery` to `run`.-- `runtime_bridge/exceptions.py:2,22-40,56-74` — `Missing{Orchestrator,BundleValidator}Error` — drop the per-mode `match`; generic message (D-F).-- `plugins/orchestrator_registry.py:3,21,31-42` — protocol `run(*, …, delivery)` + `supports_fire_and_forget`; registry `dict[str, …]`.-- `plugins/bundle_validator_registry.py:33,77-89` — registry `dict[str, …]` (validate stays blocking).-- `plugins/registrar.py:19,111-123,167-189` — `orchestrators`/`bundle_validators` + `_*_sources` → `dict[str, …]`; `add_orchestrator`/`add_bundle_validator` mode param typed `OrchestrationMode`.-- `runtime_bridge/direct_orchestrator.py:29,46-49` — `supports_fire_and_forget = False`; accept+ignore `delivery`.-- `plugins/direct/direct_plugin.py:5,20-22` — register under `DIRECT_ORCHESTRATION_MODE`.--**Tests — update/retarget:** `tests/unit/pipelex/runtime_bridge/test_execution_mode.py` (→ `test_orchestration_mode.py` + new `test_delivery_mode.py`; the property tests die with the properties), `test_input_models.py`, `test_dispatch.py`, `test_validation.py` (the f&f-target requirement now keys on `delivery`), `test_orchestrator_dispatch.py`, `test_direct_router_scoping.py`, `test_missing_bundle_validator_error.py` (parametrized over per-mode install hints → assert the generic message + the `"direct"` boot-bug case), `test_exceptions_disclosure.py`, `test_trace_context_contract.py`, `tests/unit/pipelex/plugins/test_bundle_validator_registry.py`, `test_plugin_discovery.py`, `tests/integration/pipelex/runtime_bridge/test_bridge_direct.py`.--**Docs:** the `execution_mode.py` module docstring (moves to the new modules); `TODOS.md` two-axis narrative (the `boot_orchestrator` vs `execution_mode` table, the f&f-derivation notes) — update at Phase 4 bookkeeping.--### Temporal (`pipelex-temporal/pipelex_temporal/`)--**Source:** `temporal_plugin.py:29,128-138` — collapse the two `add_orchestrator` + two `add_bundle_validator` calls to one each under `TEMPORAL_ORCHESTRATION_MODE`; `temporal_orchestrators.py` — merge `TemporalBlocking`/`TemporalFireAndForget` into one `TemporalOrchestrator` (`run` branches on `delivery`, `supports_fire_and_forget=True`); `temporal_bundle_validator.py` — unchanged impl, registered once; `temporal_cli.py` — docstring mention of `execution_mode`. `boot_orchestrator == self.name` gate (`:152`) is **unchanged** (separate concept — leave it).--**Tests:** `tests/unit/pipelex_temporal/test_bridge_temporal_dispatch.py` (both modes → one orchestrator + `delivery`), `test_trace_context_temporal.py`, `test_temporal_blocking_workflow_id.py`; the in-memory validator integration test (single registration).--### API (`pipelex-api/api/`)--**Source:** `api_config.py:27,51-74,103-122` — field rename + **delete** `_reject_fire_and_forget_default` + `.is_fire_and_forget` use; `resolve_execution_mode` → `resolve_orchestration_mode`. `schemas/models.py:19,110-115,153,204` — `PipelineApiExtras`/`PipelexApiExecuteRequest` field rename + open-token description. `routes/pipelex/pipeline.py:22,109-135,183,208,247,283,349,375` — **delete** `_async_start_mode`; `/execute` drop f&f-reject; `/start` capability-reject; `_OrchestratorPipeRun`/`_pipe_output_from_run_output` thread `delivery`; `validate_verdict` param rename. `routes/pipelex/validate.py:8,67-75,188` — `ValidateRequest.execution_mode` → `orchestration_mode` (confirm the model's home — route vs `models.py`). `error_types.py:13,23,32` — rename `EXECUTION_MODE_OVERRIDE_FORBIDDEN`; delete `FIRE_AND_FORGET_NOT_SUPPORTED`; add `START_REQUIRES_ASYNC_ORCHESTRATION`.--**Config:** `api/api.toml:15,19` — `execution_mode = "direct"` → `orchestration_mode = "direct"`; `allow_request_execution_mode_override` → `allow_request_orchestration_mode_override`; update the file comments.--**Tests:** `tests/unit/test_api_config.py` (drop the f&f-rejection test; rename), `test_execute_dispatch.py` (drop the 400-f&f case), `test_start_dispatch.py` (was "derives f&f variant" → now "honest-reject when not async-capable" + ack when capable), `test_validate_dispatch.py`, `test_protocol_conformance.py`.--**Docs + OpenAPI:** `api_config.py` / `api.toml` / `models.py` docstrings; `make openapi-export` regenerates `docs/openapi/pipelex-api.openapi.yaml`; `CHANGELOG.md`.--### Cross-cutting--- **`boot_orchestrator` is untouched** — it is the separate hub-slot boot gate (`registrar.config.plugins.boot_orchestrator`, the `temporal_plugin.py:152` claim guard), orthogonal to the orchestration token. Do not fold the two together even though both read `"temporal"`.-- **Fire-and-forget delivery-target validation** (`bridge.py:164-170`) moves from `execution_mode is TEMPORAL_FIRE_AND_FORGET` to `delivery is DeliveryMode.FIRE_AND_FORGET` — the only place that check should live post-split (the API config f&f-rejection and the `/execute` f&f-400 both delete).-------## 10. Checkpoint log / as-built (append per phase — keep cold-start-ready)--### ⏸️ COLD-START RESUME — paused before Phase 3 (2026-06-22) — ✅ SUPERSEDED (Phase 3 DONE)--> **✅ This section is historical — Phase 3 is now landed (`pipelex-api` `dd84a9a`).** Read **"### Phase 3 — as-built (API split) — Checkpoint 3 cleared"** further down in §10 for the as-built + the one carry-forward item. The locked decisions below were applied as-is. The next cold start is **Phase 4** (Mistral plugin migration + bookkeeping); see the head **Status** line and the revised §4 Phase 4.--**Read this first on resume.** Phases 1 + 2 are landed, reviewed clean, and committed. Phase 3 (`pipelex-api`, THE gate) is **not started** — its branch is cut and the tree is clean. Everything Phase 3 needs is in **§3.3** (the locked design) and **§9 → API** (the mechanical inventory); the decisions below are already settled — do not re-litigate.--**Exact state — verify with `git -C <repo> log --oneline -1` before touching anything:**--- `_plugins` (core): branch `feature/orchestration-mode-delivery-split`. As of this tracker update the tip is **`5b70c2699`** (the cold-start checkpoint, successor of the Phase-1 tracker commit `919c09348`); **this commit lands the post-checkpoint bridge-guard change on top** (always verify the real tip with `git log` — it moves with each tracker update). (Phase 1 = `bd4e52827`; docs = `9c7a7785d`.) The bridge-guard change adds the Tier-3 fire-and-forget capability guard to `runtime_bridge/bridge.py` (+ tests + the `orchestrator-plugins.md` worked-example fix) — see the post-checkpoint hardening note just below.-- `pipelex-temporal`: branch `feature/orchestration-mode-delivery-split`, tip **`01d7dc3`**, clean. (Phase 2.)-- `pipelex-api`: branch `feature/orchestration-mode-delivery-split`, tip **`06cd5e6`** (= the cut point; **no Phase 3 code**), clean. Two partial edits (`error_types.py`, `api.toml`) were made then **reverted** — start Phase 3 from scratch against §3.3/§9.--**Cross-repo wiring (already true):** both `pipelex-temporal` and `pipelex-api` pin core **editable** (`../_plugins`), so the Phase 1 core is already live in both. `pipelex-api` does **not** depend on `pipelex-temporal` — test the API split with a **stub** orchestrator/validator registered under a non-`direct` test token (one with `supports_fire_and_forget=True` and one `False`); never `import temporalio` in `pipelex-api`.--**§8 RESOLVED — honest `/start` is safe.** The MTHDS-Protocol spec defers `execute`/`start` to pipelex-api's OpenAPI/docs; no conformance test POSTs the runner `/v1/start`; the hosted platform runs the async-capable `temporal` mode. So the runner's `/start` may honestly **400** on a direct-only base. (Details in §8.)--**Locked Phase 3 design decisions (apply as-is):**--- **Honest `/start` status code = `400`** via `raise_bad_request(message=…, error_type=ErrorType.START_REQUIRES_ASYNC_ORCHESTRATION)`. (409 was the flagged alternative; 400 chosen per §8.)-- **Restructure `ApiRunner.start`** so the capability check runs **before** `pipeline_run_setup` (no library load on a doomed request): resolve `orchestration_mode` → `get_orchestrator_registry().get_optional(mode=…)` (→ `MissingOrchestratorError` if `None`) → **`if not orchestrator.supports_fire_and_forget: raise_bad_request(START_REQUIRES_ASYNC_ORCHESTRATION)`** → `pipeline_run_setup(...)` → build `delivery_assignment` → `orchestrator.run(pipe_job=…, delivery_assignment=…, delivery=DeliveryMode.FIRE_AND_FORGET)`. **Delete `_async_start_mode`** entirely.-- **`_OrchestratorPipeRun`** gains a `delivery: DeliveryMode` ctor field; its `run` passes `delivery=self._delivery` to `orchestrator.run`. `/execute` constructs it with `delivery=DeliveryMode.BLOCKING`.-- **`/execute`**: delete the `execution_mode.is_fire_and_forget` → 400 block (and `FIRE_AND_FORGET_NOT_SUPPORTED`); dispatch `BLOCKING`.-- **`api_config.py`**: `ApiConfig.orchestration_mode: str` + `allow_request_orchestration_mode_override: bool`; **delete** the `_reject_fire_and_forget_default` field-validator (and the now-unused `field_validator` import + the `PipelexExecutionMode` import). `resolve_execution_mode` → `resolve_orchestration_mode(requested: str | None, *, config) -> str` (same default/override/403 logic; `error_type=ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN`). `load_api_config`/`get_api_config` unchanged in shape.-- **`error_types.py`**: rename `EXECUTION_MODE_OVERRIDE_FORBIDDEN → ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN`; delete `FIRE_AND_FORGET_NOT_SUPPORTED`; add `START_REQUIRES_ASYNC_ORCHESTRATION` (400). _(These exact edits were drafted then reverted — redo them.)_-- **`schemas/models.py`**: rename `_EXECUTION_MODE_DESCRIPTION → _ORCHESTRATION_MODE_DESCRIPTION`, reworded for the open token (drop the `direct`/`temporal_blocking`/`temporal_fire_and_forget`/`mistral_native` value list; say "e.g. `direct`, `temporal`; plugin-provided tokens accepted; an unregistered token is refused at dispatch; delivery is endpoint-set, not requestable"). `PipelineApiExtras.execution_mode → orchestration_mode: str | None`; `PipelexApiExecuteRequest.execution_mode → orchestration_mode`. Drop the `PipelexExecutionMode` import.-- **`routes/pipelex/pipeline.py`**: `_validate_extras` reads the `orchestration_mode` body key; rename every `requested_execution_mode → requested_orchestration_mode` (params on `execute`/`start`/`validate_verdict` + the two route handlers); drop the `PipelexExecutionMode` import, add `from pipelex.runtime_bridge.delivery_mode import DeliveryMode`; update docstrings (the `/start` one: no `_async_start_mode`; direct now honest-4xx, not blocking-202). **Pre-existing, do NOT widen scope:** `_validate_extras` labels all three extras with `error_type=INVALID_CALLBACK_URLS` on a malformed value — leave it (flagged in the execute follow-on's deferred list).-- **`routes/pipelex/validate.py`**: `ValidateRequest.execution_mode → orchestration_mode` (reworded description); route passes `requested_orchestration_mode=request_data.orchestration_mode`; docstring "a forbidden `execution_mode` override is a 403" → `orchestration_mode`.-- **`api.toml`**: `orchestration_mode = "direct"` + `allow_request_orchestration_mode_override = false`; reworded comments (open token; delivery is endpoint-set; flavor sets `orchestration_mode = "temporal"`). _(drafted then reverted — redo.)_--**Gates + after-work (Phase 3):** `make fui && make c` (this repo's lint/type — Python 3.11+), `make tp` (or `make t`) for tests, **`make openapi-export`** then **`make openapi-check`** (the request schemas now advertise `orchestration_mode`), update docs (`docs/configuration.md` exec-mode section → orchestration/delivery; `docs/pipe-run.md` `/start` "blocks in-process on direct" note → now honest-4xx), and `CHANGELOG.md`. **Tests to update (§9):** `test_api_config.py` (drop the configured-f&f-rejection test; rename to orchestration), `test_execute_dispatch.py` (drop the 400-f&f case), `test_start_dispatch.py` (was "derives f&f variant" → now "honest-400 when the resolved mode's stub orchestrator has `supports_fire_and_forget=False`" + "acks when True"), `test_validate_dispatch.py`, `test_protocol_conformance.py`. **CI gotcha:** the editable `../_plugins` pin breaks any `pipelex-api` PR CI — pin core to a git+https SHA + relock `uv.lock` (flip back at release). Not needed for local gates.--**Then Checkpoint 3** (§4): as-built here + clean-context `/code-review` on the `pipelex-api` diff. **Phase 4** (Mistral plugin migration + bookkeeping) is still pending — and its Mistral scope is **bigger than first estimated, not smaller**. Phase 1 moved the core-doc Mistral prose + `pipe_classification.py` + the Mistral `execution-modes.md` to `"mistralai-workflows"`, but the external `pipelex-mistralai-workflows` plugin still has live `execution_mode` / `PipelexExecutionMode` **code** — a deleted-enum import, a registration under the deleted enum member, its own `execution_mode` wire field + an `is PipelexExecutionMode.DIRECT` compare, and an orchestrator on the old `run` signature — all of which must migrate to `orchestration_mode` + `delivery` + the new protocol (same shape as the Temporal collapse). See the revised §4 Phase 4 for the verified anchors. The trackers/superseded-plan markers + `make check-spec-links` remain.-------### Post-checkpoint hardening — Phases 1–2 combined `/code-review` (2026-06-22)--After Checkpoints 1 + 2 cleared, a clean-context `/code-review` over "what we've done so far" (Phases 1–2) surfaced a reachable behavioral gap in landed core code and a stale doc. Both fixed on the `_plugins` branch (uncommitted at time of writing — **commit before Phase 3**).--**Fix 1 — Tier-3 fire-and-forget capability guard (resolves the §8 Tier-3 open question).** The bridge forwarded `delivery=FIRE_AND_FORGET` to any resolved orchestrator without consulting `supports_fire_and_forget`, so `direct` + fire-and-forget + a valid `delivery_assignment_dump` passed `_validate_input`'s target gate, ran blocking to completion, returned `is_completed=True`, and **never invoked the delivery target** — block + false-ack + silent drop. As-landed shape:--- `run_pipe_via_bridge` now **resolves the orchestrator up front** (before opening the scoped library / building the pipe job), then calls `_validate_input`. A doomed request is rejected before any library/job work — symmetric with `/start`'s "capability check before `pipeline_run_setup`" (D-E). (Note: `MissingOrchestratorError` for an unregistered token now raises *before* the pipe-job build, not after — a behavior-equivalent reorder.)-- `_validate_input(input_payload, *, orchestrator, delivery_assignment)` gained the `orchestrator` param and a **capability gate checked first**: `if input_payload.delivery is FIRE_AND_FORGET and not orchestrator.supports_fire_and_forget → raise PipelexBridgeDispatchError` ("Orchestration mode '…' cannot honor fire-and-forget delivery…"), *before* the existing delivery-target gate. A valid target does not rescue a blocking-only orchestrator (the mode, not the target, is the problem). This is the first read site of `supports_fire_and_forget` in landed code (it was write-only after Phases 1–2).-- Tests: `test_validation.py` splits the FF tests into capability-gate vs target-gate (target-gate tests use an async-capable stand-in to reach that gate) + a real-path `run_pipe_via_bridge` rejection on `direct`+FF+target; `test_orchestrator_dispatch.py` adds a seam test proving a blocking-only orchestrator's `run` is **never awaited** for an FF request (`blocking_only.calls == []` — directly refutes the block/false-ack failure); `test_trace_context_contract.py`'s non-direct nulling test was updated to register a stand-in `temporal` orchestrator (it previously leaned on `MissingOrchestratorError` as a post-build stop-point, which the reorder removed).--**Fix 2 — stale worked-example doc.** `docs/under-the-hood/orchestrator-plugins.md` "Worked example" still described the deleted `TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` (the Phase 1 as-built had deferred this section to Phase 2, but Phase 2 — a different repo — missed the core-repo doc). Rewritten to the single collapsed `TemporalOrchestrator` (one `"temporal"` registration, delivery-branched `run`, `supports_fire_and_forget = True`).--**Deferred / folded elsewhere:** the review's other substantial cluster — `pipelex-api` + `pipelex-mistralai-workflows` unmigrated against Phase 1 core — is expected load-bearing-ordering breakage owned by Phases 3/4. The one correction it forced: **Phase 4's Mistral scope was underestimated** (real code, not a token rename) — folded into §4 Phase 4 + the cold-start resume note above. **Gates:** `make agent-check` green (pyright 0 / mypy 0 / ruff / plxt / keyword-only); targeted `runtime_bridge` + `plugins` suites green.-------### Phase 1 — as-built (core split) — Checkpoint 1 cleared--**Branch / commit.** `feature/orchestration-mode-delivery-split` in the `_plugins` core worktree, cut from `feature/Orchestrator-dispatched-validate` tip `fa1b46637`. One commit: **`bd4e52827`**. Nothing pushed.--**Open sub-decisions (§8) — resolved with the recommended defaults:**--- **`OrchestrationMode` shape** → `TypeAlias = str` + `DIRECT_ORCHESTRATION_MODE: Final[str] = "direct"` in the new `pipelex/runtime_bridge/orchestration_mode.py`. No `NewType` (cast-free for plugins; the registry is the validator). A `test_orchestration_mode.py` pins `OrchestrationMode is str` + the `"direct"` literal.-- **`supports_fire_and_forget` location** → on the orchestrator. Declared as a protocol **data member** (`supports_fire_and_forget: bool` on `OrchestratorProtocol`, still `@runtime_checkable`); `DirectOrchestrator.supports_fire_and_forget = False` (plain class attr).-- **Generic `Missing*Error`** → both `MissingOrchestratorError` and `MissingBundleValidatorError` delegate to a shared module-level `_build_missing_message(*, noun, mode)`; the `"direct"` boot-bug case is a plain `if mode == DIRECT_ORCHESTRATION_MODE:` **string** compare (not an enum `match`, so it does not offend the no-`==`-on-enums rule). Message: *"No {orchestrator|bundle validator} is registered for orchestration mode '{mode}'; is its plugin installed?"* / for direct: *"...for the core 'direct' mode; this indicates a boot or plugin-discovery problem."* Names no plugin.-- **Delivery is a `run` parameter; validate stays blocking** → `OrchestratorProtocol.run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)`. `BundleValidatorProtocol.validate_bundles` is unchanged (no `delivery` — validation is inherently blocking).-- **`_validate_input`** keys on `input_payload.delivery is DeliveryMode.FIRE_AND_FORGET`; message reworded "Fire-and-forget delivery requires …".--**Files.** New: `runtime_bridge/orchestration_mode.py`, `runtime_bridge/delivery_mode.py`. Deleted: `runtime_bridge/execution_mode.py`. Re-keyed `dict[str, …]`: `orchestrator_registry.py`, `bundle_validator_registry.py`, `registrar.py` (stores + `_*_sources` + `add_orchestrator`/`add_bundle_validator` mode params typed `OrchestrationMode`). Split `payloads.py` (`orchestration_mode: str` + `delivery: DeliveryMode`). `direct_plugin.py` registers under `DIRECT_ORCHESTRATION_MODE`. `direct_orchestrator.py` + `bridge.py` per §3.1.--**Deviations / extras beyond the literal §9 list (reconciled here):**--- **`plugins/exceptions.py`** — reworded `Duplicate{Orchestrator,BundleValidator}Error` prose "execution mode" → "orchestration mode" (their `mode` field was already `str`; only prose changed). Not in §9 but consistent with D-H.-- **`runtime_bridge/primitives/pipe_classification.py`** — two **docstring** prose mentions of the dead `MISTRAL_NATIVE` enum updated to `"mistralai-workflows"` (prose only; no token *registration* exists in core). The real D-G Mistral-token rename stays a Phase 4 item (deferred plugin).-- **`docs/under-the-hood/orchestrator-plugins.md`** — updated the **core-seam** sections (intro two-axis model, the seam-in-one-view pseudocode, the `OrchestratorProtocol` signature + `supports_fire_and_forget`, the `add_orchestrator(mode="temporal", …)` example, the generic-error section, the SPI table "Mode + delivery + errors" row pointing at the two new modules). **Deferred to Phase 2:** the "Worked example: the in-tree Temporal plugin" section still describes the two-orchestrator (`TemporalBlocking`/`TemporalFireAndForget`) shape — accurate until Phase 2 collapses them.--**Tests.** Deleted `test_execution_mode.py`; added `test_orchestration_mode.py` + `test_delivery_mode.py`. Updated `test_input_models.py`, `test_dispatch.py` (mistral→generic-hint), `test_validation.py` (delivery-centric f&f), `test_orchestrator_dispatch.py` (`_FakeOrchestrator` gains `supports_fire_and_forget` + `delivery` param; miss test asserts the generic message over tokens incl. a synthetic `"acme"`), `test_direct_router_scoping.py`, `test_missing_bundle_validator_error.py` (generic + `"direct"` boot-bug), `test_exceptions_disclosure.py` (generic hint survives STRICT), `test_trace_context_contract.py`, `test_bundle_validator_registry.py`, `test_plugin_discovery.py`, `test_bridge_direct.py`.--**Gates.** `make agent-check` (ruff/plxt/pyright 0/mypy 0/keyword-only ✓), `make tb` (boot builds the re-keyed registries), `make agent-test` — all green.--**⚠️ Cold-start note for Phase 2.** `pipelex-temporal` pins core **editable** (`../_plugins`), so this commit is live in it immediately. Its `TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator` do **not** yet implement the new `OrchestratorProtocol.run(*, …, delivery)` signature and lack `supports_fire_and_forget`, so `pipelex-temporal`'s suite is expected to fail against this core until Phase 2 adopts the new shape (collapse to one `TemporalOrchestrator`). This is the intended load-bearing ordering (core lands first), not a regression.--### Phase 2 — as-built (Temporal collapse) — Checkpoint 2--**Branch / commit.** `feature/orchestration-mode-delivery-split` in `pipelex-temporal`, cut from `feature/Orchestrator-dispatched-validate` tip `459e04d`. One commit: **`01d7dc3`**. Nothing pushed. (`pipelex-temporal` pins core editable, so it built against the Phase 1 core `bd4e52827` live.)--**The collapse.** `temporal_orchestrators.py` — the two classes (`TemporalBlockingOrchestrator` / `TemporalFireAndForgetOrchestrator`) merged into one **`TemporalOrchestrator`** with `supports_fire_and_forget = True`. Its `run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)` runs the shared prelude (`_require_temporal_extra()`, lazy `WorkflowExecutionError` / `make_temporal_pipe_run` imports, `dispatch_errors` tuple, `temporal_pipe_run = make_temporal_pipe_run()`) then `match delivery:` — `BLOCKING` = the old blocking body (`temporal_pipe_run.run(...)` → `make_workflow_id(...)` → `serialize_completed_output`), `FIRE_AND_FORGET` = the old f&f body (`.start(...)` → `is_completed=False` output). Exhaustive match, no default arm; `WorkflowExecutionError` + `PIPE_DISPATCH_ERRORS` wrapping kept in both arms; extra-guard runs before any temporalio import. Dispatch-error messages reworded to name the *delivery* ("temporal blocking delivery" / "temporal fire-and-forget delivery").--**The token.** `TEMPORAL_ORCHESTRATION_MODE: Final[str] = "temporal"` defined in `temporal_orchestrators.py` (lives with the orchestrator). **Distinct constant** from `temporal_activation.TEMPORAL_PLUGIN_NAME` / `boot_orchestrator` (the hub-slot gate) — both equal `"temporal"` but are different concepts; left separate per §8. `temporal_bundle_validator.py` imports the token from `temporal_orchestrators` (import-light; no cycle — orchestrators never imports the validator) and its missing-extra guard now raises `MissingBundleValidatorError(mode=TEMPORAL_ORCHESTRATION_MODE)`.--**Register once.** `temporal_plugin.py`: one `add_orchestrator(mode=TEMPORAL_ORCHESTRATION_MODE, …)` + one `add_bundle_validator(mode=TEMPORAL_ORCHESTRATION_MODE, …)` (the two-mode duplication is gone). F3 mapper + `boot_orchestrator == self.name` slot-claims **unchanged**. Smoke-confirmed: `registrar.orchestrators` / `bundle_validators` keys == `['temporal']`.--**Import-light preserved.** `register` imports no `temporalio` (the import-light subprocess test still passes: blocks `temporalio`, runs `register` under both `boot_orchestrator` arms). The subprocess assertion changed `len(orchestrators) == 2 → == 1`.--**Tests.** `test_bridge_temporal_dispatch.py` (now `orchestration_mode="temporal"` + `delivery=FIRE_AND_FORGET` for the f&f case — exercises both delivery branches through the **single** real `TemporalOrchestrator.run`, with `make_temporal_pipe_run` mocked), `test_temporal_blocking_workflow_id.py`, `test_trace_context_temporal.py`, `test_temporal_plugin_http_error_mapper.py` (count 2→1). The validator in-memory integration test (`test_temporal_bundle_validator_in_memory.py`) constructs the validator directly — no mode reference, unchanged. Minor docstring fix in `temporal_cli.py` (`execution_mode` → `orchestration_mode`).--**Gates.** `pipelex-temporal` `make agent-check` (ruff/pyright 0/mypy 0) + `make agent-test` (full suite incl. `temporal`-marked in-memory integration; the markers exclude only inference/llm/img_gen/extract/search/pipelex_api, NOT temporal) — all green against the Phase 1 core. **Bonus:** this repo runs Python 3.11, so the run also validated the new core modules' 3.10+ compatibility (`TypeAlias`/`Final`/`StrEnum`-from-`pipelex.types`).--**Cold-start note for Phase 3.** `pipelex-api` is the next consumer (THE gate). It pins core editable too, so Phase 1 core is live in it; `pipelex-api` does NOT depend on `pipelex-temporal` (test the API split with a stub orchestrator/validator under a non-`direct` token; never import `temporalio`). Resolve the §8 `/start`-honesty conformance question before Phase 3 lands. CI gotcha: the editable `../_plugins` pin breaks `pipelex-api` CI — pin core to a git+https SHA + relock for any PR.--**Code-review triage (Checkpoint 2).** One clean-context `pr-review-toolkit:code-reviewer` (pointed only at `01d7dc3`). **Verdict: no material issues, correct + standards-compliant.** It verified against parent `459e04d` that the collapse is byte-faithful (BLOCKING still recomputes/reports `make_workflow_id(...)` not the bare run id; FIRE_AND_FORGET still returns `is_completed=False` + `.start()`'s workflow id; `WorkflowExecutionError` + `PIPE_DISPATCH_ERRORS` wrapping kept in both arms; extra-guard runs before any temporalio import), the `match delivery:` is exhaustive with no default arm and no `==`/`is` on enum members, and **empirically confirmed import-light** (a `sys.meta_path` blocker proves importing the plugin + `temporal_bundle_validator` importing `TEMPORAL_ORCHESTRATION_MODE` from `temporal_orchestrators` pulls no `temporalio`, no cycle). Two non-defect observations (cosmetic error-string rewording; the test's `expected_claims = 4` structural count is the legitimate test-assertion exception to the no-hardcoded-counts rule). **No applied or deferred items.**-------**Code-review triage (Checkpoint 1).** One clean-context `pr-review-toolkit:code-reviewer` (no plan/rationale handed in; pointed only at `bd4e52827`). **Verdict: correct + standards-compliant, no bugs, tests not weakened.** It independently confirmed: the `is_direct` move to a `==` string compare and `_validate_input`'s move to `delivery is FIRE_AND_FORGET` are faithful (the latter a deliberate, intended broadening — *any* f&f run now needs a delivery target, not just the old Temporal-f&f member); defaults preserved (`direct` + `BLOCKING`, never f&f); the `"direct"` boot-bug `==` compare and generic messages are standards-clean; no `==`-on-enum-member introduced. **One actionable finding — APPLIED:** three user-facing Mistral-workflows doc pages still described the deleted `execution_mode` / `PipelexExecutionMode` model — `docs/distributed-execution/mistral-workflows/execution-modes.md` (rewritten to the two-axis `orchestration_mode` + `delivery` model, retitled "Orchestration & Delivery", generic error wording, `mistralai-workflows` token), `index.md`, `your-first-pipelex-workflow.md`, and `docs/reliability/durable-execution.md` (field/token refs + the now-stale link labels). My §9 grep had been `.py`-only — these `.md` pages slipped it; folded into this commit. **No deferred items.**-------### Phase 3 — as-built (API split) — Checkpoint 3 cleared (2026-06-22)--**Branch / commit.** `feature/orchestration-mode-delivery-split` in `pipelex-api`, cut from `06cd5e6` (the cut point named in the cold-start note). One commit: **`dd84a9a`**. Nothing pushed. THE gate — the third and last code repo of Direction A — is landed; all the conflation workarounds (`_async_start_mode`, the `/execute` f&f-400, the `ApiConfig` f&f field-validator) are gone.--**Applied exactly per §3.3 / §9 / §10 locked decisions:**--- **`error_types.py`** — `EXECUTION_MODE_OVERRIDE_FORBIDDEN → ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN` (wire value `OrchestrationModeOverrideForbidden`); **deleted** `FIRE_AND_FORGET_NOT_SUPPORTED`; **added** `START_REQUIRES_ASYNC_ORCHESTRATION` (wire value `StartRequiresAsyncOrchestration`, status **400** per §8).-- **`api_config.py`** — `ApiConfig.orchestration_mode: str` + `allow_request_orchestration_mode_override: bool`; **deleted** `_reject_fire_and_forget_default` (+ the now-unused `field_validator` and `PipelexExecutionMode` imports). `resolve_execution_mode → resolve_orchestration_mode(requested: str | None, *, config) -> str` (string compare; 403 via `ORCHESTRATION_MODE_OVERRIDE_FORBIDDEN`; an *unregistered* token is NOT rejected here — it surfaces at dispatch). Module/class docstrings reworded to the open-token + endpoint-set-delivery model.-- **`api.toml`** — `orchestration_mode = "direct"` + `allow_request_orchestration_mode_override = false`; comments reworded (open token; delivery endpoint-set; flavor sets `orchestration_mode = "temporal"`).-- **`schemas/models.py`** — `_EXECUTION_MODE_DESCRIPTION → _ORCHESTRATION_MODE_DESCRIPTION` (open-token wording, delivery-not-requestable note); `PipelineApiExtras.orchestration_mode: str | None`; `PipelexApiExecuteRequest.orchestration_mode: str | None`; dropped the `PipelexExecutionMode` import.-- **`routes/pipelex/pipeline.py`** — **deleted `_async_start_mode`**. `_OrchestratorPipeRun` gained a `delivery: DeliveryMode` ctor field threaded into `orchestrator.run`. `/execute` drops the f&f-reject and dispatches `delivery=BLOCKING`. **`/start` restructured** so resolve → `get_optional` → `MissingOrchestratorError` if None → **capability check `if not orchestrator.supports_fire_and_forget: raise_bad_request(START_REQUIRES_ASYNC_ORCHESTRATION)`** all run **before `pipeline_run_setup`** (no library load on a doomed request); dispatch `delivery=FIRE_AND_FORGET`. `validate_verdict` param `requested_execution_mode → requested_orchestration_mode`; `_validate_extras` reads the `orchestration_mode` body key; both route handlers thread `requested_orchestration_mode=extras.orchestration_mode`; docstrings updated (the `/start` one: honest-400, not blocking-202). Import swap: drop `PipelexExecutionMode`, add `DeliveryMode`; `resolve_execution_mode → resolve_orchestration_mode`.-- **`routes/pipelex/validate.py`** — `ValidateRequest.execution_mode → orchestration_mode: str | None` (open-token description); route passes `requested_orchestration_mode=request_data.orchestration_mode`; dropped the `PipelexExecutionMode` import; "no verdict" docstring `execution_mode → orchestration_mode`.-- **OpenAPI** — `make openapi-export` regenerated `docs/openapi/pipelex-api.openapi.yaml` (now advertises `orchestration_mode`, no residual `execution_mode` / old enum values); `make openapi-check` clean.--**⚠️ Deviation reconciled — the pin was NOT editable as the cold-start note claimed.** The cold-start note (and §5) asserted "`pipelex-api` pins core editable." **In reality the branch sat on a git-SHA pin** (`[tool.uv.sources] pipelex = { git = …, rev = "51ff0941…" }`, from parent commit `c60be1f` "pin pipelex to a git SHA so CI can resolve it (was a local editable path)") — and that SHA predates Phase 1, so it has `execution_mode.py` and lacks `delivery_mode`/`orchestration_mode`. Phase 3 therefore did not compile against it. **Resolution:** flipped `[tool.uv.sources]` back to `pipelex = { path = "../_plugins", editable = true }` (matching `pipelex-temporal`'s convention) + `make li` to relock/reinstall, so the Phase 1 core split is live. **CARRY-FORWARD (the one open item):** this editable local-path pin is unresolvable in CI, so **before opening any `pipelex-api` PR, push the Phase 1 core and flip the pin to that pushed git+https SHA + relock** (the documented CI gotcha). `uv.lock` + `pyproject.toml` are part of `dd84a9a`.--**Tests (all updated to the two-axis model; §9 list).** `test_api_config.py` (dropped `TestApiConfigRejectsFireAndForgetDefault`; renamed to `TestResolveOrchestrationMode`; string tokens; 403 asserts `OrchestrationModeOverrideForbidden`), `test_execute_dispatch.py` (dropped the 400-f&f case; `_StubOrchestrator.run` gains `delivery` + `supports_fire_and_forget`; asserts `delivery is BLOCKING`), `test_start_dispatch.py` (deleted the `_async_start_mode` pure-mapping class; new `TestStartCapabilityGate` — ack-when-async-capable asserts `delivery is FIRE_AND_FORGET`, honest-400 asserts `StartRequiresAsyncOrchestration` + `stub.calls == []` proving `run` is never awaited), `test_validate_dispatch.py` (string `"temporal"` token; renamed forbidden-override test + asserts the error_type), `test_protocol_conformance.py` (`mode="direct"`; `fake_run` gains `delivery` + an `assert delivery is FIRE_AND_FORGET`; `fake_orchestrator.supports_fire_and_forget = True` so the new gate is exercised).--**Docs + CHANGELOG.** `docs/configuration.md` ("Execution mode" → "Orchestration mode", two-axis model, honest-400), `docs/pipe-run.md` (`/execute` + `/start` notes; the `workflow_id` field note — a 202 is now only ever an async backend's, direct returns 400), `docs/pipe-validate.md` (`orchestration_mode`-aware, no delivery axis), `CHANGELOG.md` `[Unreleased]` rewritten to the orchestration_mode/delivery model (the never-released flat `execution_mode` bullets reshaped — no backward-compat concern).--**Gates.** `pipelex-api` `make c` (ruff / pyright **0** / mypy **0**) + `make tp` (**319 passed**) + `make openapi-export` then `make openapi-check` (artifact up to date) — all green against the live Phase 1 core (editable `../_plugins`). Python 3.11.--**Code-review triage (Checkpoint 3).** One clean-context `pr-review-toolkit:code-reviewer` (no plan/rationale handed in; pointed only at `dd84a9a`, cross-checked against the live core contract in `../_plugins`). **Verdict: no must-fix issues — correct, complete, standards-compliant.** It independently confirmed: the capability-check ordering is right (resolve → lookup → missing-check → `supports_fire_and_forget` 400, all before `pipeline_run_setup`; the orchestrator is looked up **once** and reused at dispatch so the gate can't diverge); the **honest-400 actually fires on the real base** (core `DirectOrchestratorPlugin` in `plugins/builtins.py` registers `DirectOrchestrator` with `supports_fire_and_forget = False` under `"direct"`, so `/start` hits the capability 400, not the `MissingOrchestratorError` fallback — exactly as `test_start_honestly_400s…` asserts); no StrEnum/`==`-on-enum violation (`resolve_orchestration_mode`'s `==` is `str`/`str`; `DeliveryMode` only ever compared with `is`); Protocol signatures match; RFC 7807 helpers correct; the field-validator deletion + its test removal are justified (the invariant no longer exists), not a weakening; the test assertions strengthened (delivery-axis + never-awaited proofs). **One should-consider — the editable-pin CI break — is the carry-forward item already captured above** (flagged by the author in the commit body + inline comment; flip to a pushed git SHA before a PR). **No applied or deferred code items.**--**Cold-start note for Phase 4.** Phase 4 = the **bigger-than-first-estimated** Mistral plugin migration (real code, not a token swap — see the revised §4 Phase 4 for the verified `pipelex-mistralai-workflows` anchors: deleted-enum import, registration under the deleted member, its own `execution_mode` wire field + `is PipelexExecutionMode.DIRECT` compare, an orchestrator on the old `run` signature) **plus** bookkeeping: `_plugins/TODOS.md`, the two superseded sibling-plan markers, and `make check-spec-links` if any spec heading/test asserts mode values. Re-derive the §4 anchors against the live `pipelex-mistralai-workflows` tip before starting (they will have moved). The three landed repos (`_plugins` `e9e73e0eb`, `pipelex-temporal` `01d7dc3`, `pipelex-api` `dd84a9a`) all sit on `feature/orchestration-mode-delivery-split`, pinned core editable for local dev.-------### Phase 4 — as-built (Mistral plugin migration + bookkeeping) — effort COMPLETE (2026-06-22)--**✅ This is the final phase. All four phases of the orchestration-mode/delivery split are landed.** The cold-start note just above is now historical — read this section for the as-built.--**⚠️ The deviation that shaped Phase 4 (the §4 anchors were re-derived and the pin assumption was wrong).** The plan (and §4/§5) assumed `pipelex-mistralai-workflows` pinned the Phase-1 core, so it would be "already red against Phase 1." **In reality the plugin pins core editable at `../_workflows`, NOT `../_plugins`** — and `_workflows` was on a *divergent* branch **`feature/mistralai-2x-bump`** that still carried the OLD `runtime_bridge` (had `execution_mode.py`, lacked `orchestration_mode`/`delivery_mode`). So the plugin was **green against its own core**, not red against the split — and could not be migrated in isolation. `feature/mistralai-2x-bump` and `feature/orchestration-mode-delivery-split` were divergent (neither an ancestor; common base `e460d072`); `mistralai-2x-bump` made **no** `runtime_bridge` changes relative to that base.--**Resolution (user-directed): merge the split into the plugin's core branch.** Merged `feature/orchestration-mode-delivery-split` → `feature/mistralai-2x-bump` in the `_workflows` worktree — **merge commit `a1448ad8f`, conflict-free** (the branches overlapped only in shared plugin-system work, which was identical). The merged core gives the plugin a runtime bridge carrying **both** the orchestration split and the mistralai-2x bump. Merged-core verification: `make tb` (boot/config) ✓, `make agent-check` (ruff/pyright 0/mypy 0/keyword-only) ✓, the merge-sensitive core suites (`runtime_bridge` + `plugins` + `test_plugin_config_loader` + `test_direct_bundle_validator`) all green.--**The Mistral migration — `pipelex-mistralai-workflows` commit `0eb3878` on `feature/Mistral-native`** (parent `e508e73`; nothing pushed). Same shape as the Phase 2 Temporal collapse — real code, not a token swap:--- **`mistral_orchestrator.py`** — defines `MISTRALAI_WORKFLOWS_ORCHESTRATION_MODE: Final[str] = "mistralai-workflows"` (the D-G token, **living with the orchestrator** exactly as `pipelex-temporal` puts `TEMPORAL_ORCHESTRATION_MODE` in `temporal_orchestrators.py`). `MistralNativeOrchestrator.run` adopts the new `OrchestratorProtocol.run(*, pipe_job, delivery_assignment, delivery: DeliveryMode)` signature and declares **`supports_fire_and_forget = False`** — Mistral-native always runs to completion in-place, so it is blocking-only like core `DirectOrchestrator`. It accepts `delivery` and ignores it; the bridge's pre-dispatch capability gate (the §10 "Fix 1" Tier-3 guard, present in the merged core) rejects a `FIRE_AND_FORGET` request against it before `run` is ever called, so `run` provably never receives f&f. (No `# noqa: ARG002` needed — unlike core, this repo's ruff `select` omits `ARG`, so the unused-arg rule does not fire; the linter auto-removed the noqa as unused.)-- **`mistral_plugin.py`** — registers `add_orchestrator(mode=MISTRALAI_WORKFLOWS_ORCHESTRATION_MODE, …)`; dropped the deleted-enum import. **The plugin name / entry-point `MISTRAL_NATIVE_PLUGIN_NAME = "mistral_native"` is KEPT, deliberately distinct from the orchestration token** — the plugin-discovery identity vs the registry key, the same two-concept separation `pipelex-temporal` keeps between `TEMPORAL_PLUGIN_NAME` and `TEMPORAL_ORCHESTRATION_MODE`. (D-G renamed the *token*, not the plugin name; renaming the entry-point would force a reinstall for zero gain.)-- **`streaming.py`** — wire field `execution_mode → orchestration_mode` (on `PipelexPipeRunStreamingState` + the read from the input); the `is PipelexExecutionMode.DIRECT` enum compare → `== DIRECT_ORCHESTRATION_MODE` (both plain `str` now — `==` is correct, matches how core's `bridge.py` compares it); docstrings reworded off `DIRECT`/`TEMPORAL_BLOCKING`/`TEMPORAL_FIRE_AND_FORGET` onto the `direct`/non-direct orchestration-mode framing. The per-step-streaming branch still correctly keys on "is this the in-process `direct` mode".-- **`primitives/`** — doc/comment refs to the deleted enum; the internal scoped-library id prefix `mistral_native_` → `mistralai_workflows_` (`scoped_library.py`; purely internal, ephemeral per-call id — no wire contract).-- **Tests** — `test_plugin_discovery.py` looks up + asserts the orchestrator under `MISTRALAI_WORKFLOWS_ORCHESTRATION_MODE` (real registration check via full boot; renamed `test_mistralai_workflows_orchestrator_is_registered`); the three `test_activities_*` construct `PipelexPipeRunInput(orchestration_mode=DIRECT_ORCHESTRATION_MODE)`; `test_activities_streaming.py` asserts the renamed `orchestration_mode` wire field; `test_scoped_library.py` asserts the renamed id prefix; doc-only refs in `test_router_dispatch.py` / `test_native_recursion.py` / `test_data/bridge_funcs.py`. No assertion weakened — field-for-field.-- **Docs** — `CLAUDE.md` public-surface contract (the deleted `execution_mode` module + the two other now-deleted symbols `PipelexBridgeRuntimeError` / `MissingPipelexTemporalExtraError` — stale since the Temporal externalization — replaced with the real surface: `payloads`, `orchestration_mode`, `delivery_mode`, `PipelexBridgeDispatchError`); `README.md` quick-start + a recast "Orchestration mode and delivery" section (also dropped the dead `[temporal]` extra — the README now says install `pipelex-temporal` for the `temporal` mode); the `wip/boundary-violation-mistral-native.md` allowed-surface list; `CHANGELOG.md` (new `### Changed` reconciliation bullet + reshaped the never-released bullets that named the old enum); `pyproject.toml` entry-point comment.--**Gates (Mistral plugin, against the merged editable `../_workflows` core).** `make agent-check` (ruff / pyright **0** / mypy **0**) + `make agent-test` + the full migrated integration suite (`test_plugin_discovery` + `test_activities_{direct,offloaded,streaming}` + `test_native_recursion` + `test_scoped_library`) — **all green** (12 passed in the targeted verbose run). Python 3.13.--**Code-review triage (Phase 4).** One clean-context `pr-review-toolkit:code-reviewer` (no plan handed in; pointed only at `c2013fb` — the pre-amend SHA — cross-checked against the live merged core in `../_workflows` + the Temporal reference). **Verdict: the code migration is correct and complete.** It independently confirmed: the `run` signature conforms to the live `OrchestratorProtocol`; `supports_fire_and_forget=False` + ignoring `delivery` is safe given the bridge's pre-dispatch FF rejection; registration under `"mistralai-workflows"` with the name/token split is sound (not a bug); the `streaming.py` `==` string compare is correct (vs `is`); tests not weakened; no StrEnum/`==`-on-enum-member violation, no `except Exception`, keyword-only respected, no `__init__.py` re-exports; no new pyright/mypy regression (the `reportUnknown*` on the untyped Mistral SDK return are pre-existing, identical count on the parent). **One must-fix — APPLIED:** `README.md` still imported the deleted `PipelexExecutionMode` and documented it in a "## Execution modes" section (my source/test grep had been `.py`+config-scoped — the README slipped, the same `.md`-slips-a-`.py`-grep the Phase 1 review caught). Rewritten to `orchestration_mode`/`delivery` + the dead `[temporal]`-extra fix. **One nit — APPLIED:** the `wip/boundary-violation-mistral-native.md` allowed-surface list named the deleted `execution_mode` module → updated. Both fixes folded into the commit by amend (`c2013fb` → **`0eb3878`**). **No deferred code items.**--**Bookkeeping done.** Superseded markers added to both sibling plans (`execute-per-request-execution-mode.md`, `orchestrator-dispatched-validate.md`) pointing here; `_plugins/TODOS.md` follow-on entry flipped to "DONE (Phases 1–4)" + the merge deviation recorded; the `pipelex-api` `CHANGELOG.md` was already on the orchestration_mode/delivery model from Phase 3 (`dd84a9a`) — no Phase 4 change needed there; `make check-spec-links` **not** triggered (verified: zero `docs/specs/` or `conformance/` references to mode values).--**Carry-forward / open items (none block the effort; flagged for whoever pushes/PRs):**--1. **`pipelex-api` editable-pin → pushed git-SHA flip (from Phase 3, still open).** `pipelex-api` pins core editable `../_plugins`; before any `pipelex-api` PR, push the core split and flip to that pushed git+https SHA + relock (the documented CI gotcha).-2. **`pipelex-mistralai-workflows` core pin.** It pins `../_workflows` editable; for a real release the same flip applies (push the merged core, pin a published/pinned ref). The plugin's own CHANGELOG already notes the `../_workflows` co-dev override is stripped at v0.1.0.-3. **`_workflows` ↔ `_plugins` doc divergence.** The `a1448ad8f` merge copied a *pre-Phase-4 snapshot* of the orchestration tracker docs (this plan, `TODOS.md`, the sibling plans) into `_workflows`. The Phase 4 bookkeeping edits above landed on **`_plugins`** (`feature/orchestration-mode-delivery-split`, the tracker SoT), so the `_workflows` copies are momentarily stale. They reconcile on the next merge of `feature/orchestration-mode-delivery-split` → `feature/mistralai-2x-bump` (or whenever the two core branches are integrated). Not a code concern — `_workflows`'s *code* is the superset (it contains the split); only the in-flight tracker prose differs.-4. **Two divergent core branches now exist** (`feature/orchestration-mode-delivery-split` in `_plugins`, `feature/mistralai-2x-bump` in `_workflows`, the latter containing the former via `a1448ad8f`). Integrating/landing them is a branch-strategy decision for the user — out of scope for this effort.diff --git a/wip/plugins/orchestrator-agnostic-runner-and-flavors.md b/wip/plugins/orchestrator-agnostic-runner-and-flavors.mddeleted file mode 100644index ae78c9b8e..000000000--- a/wip/plugins/orchestrator-agnostic-runner-and-flavors.md+++ /dev/null@@ -1,182 +0,0 @@-# Orchestrator-agnostic runner & deployment flavors--**Status:** plan + agent handoff. Decisions locked (see §3). Not yet started in any consumer repo.-**Owner effort:** the plugin-system externalization (see [`TODOS.md`](../../TODOS.md) and the Phase 5 cut-over notes). This doc is the forward-looking *consumer* migration: it explains how the new plugin system changes the world for projects that consume `pipelex` and choose an orchestrator (Temporal, Mistral, or none), and gives the concrete, sequenced handoff to make `pipelex-api` an orchestrator-agnostic base plus N thin deployment flavors.--> **Why this exists.** Once Temporal left core (it is now the private `pipelex-temporal` plugin, discovered via the `pipelex.plugins` entry point), a consumer no longer gets Temporal "for free" by installing an extra. More importantly, the recon for this migration found that `pipelex-api` is **not** an orchestrator-agnostic runner today — it is wired directly to Temporal in production code. Making it the shared base for multiple deployment flavors (`pipelex-api-hosted` = Temporal, a future `pipelex-api-mistral` = Mistral) is a real decoupling refactor, not a dependency bump. This doc captures the target model, the locked design decisions, and the per-repo work.-------## 1. The model shift (read this first)--The plugin system splits a previously-monolithic idea ("Temporal is on") into orthogonal concepts. Internalize these before touching any repo.--**Presence vs activation.**--- **Presence** = the plugin distribution is installed, so its `pipelex.plugins` entry point is discovered at boot and its `register()` runs. For an orchestrator plugin, `register()` always contributes its `execution_mode` arms to the orchestrator registry. Presence alone makes a mode *dispatchable*; it does not make it *used*.-- **Activation** = the runtime is actually told to use it. Activation decomposes into **two independent knobs** (this is the crux — see below).--**Two independent activation knobs.**--| Knob | Owns | Set by | Temporal | Mistral | Direct |-|---|---|---|---|---|---|-| `execution_mode` | which mode a *top-level* run dispatches as, through the bridge | runner config (default) + optional per-request override | `temporal_blocking` / `temporal_fire_and_forget` | `mistral_native` | `direct` |-| `plugins.boot_orchestrator` | whether to claim the process-global hub slots (content generator / pipe router / pipe run / task manager) at boot | deployment config TOML | `"temporal"` | *unset* | *unset* |--The two are **not derivable from each other** — and Mistral is the proof. A Temporal deployment needs *both* (the boot gate stands up the Temporal client/task-manager that the orchestrator uses to enqueue; the `execution_mode` tells the bridge to enqueue). A Mistral deployment needs **only** `execution_mode = mistral_native` — its plugin claims **no** hub slots (it is a self-contained per-call orchestrator), so there is no boot gate to set. A plain in-process deployment sets neither.--**Boot-global vs per-call orchestrators.** This is *why* the two knobs exist. A **boot-global** orchestrator (Temporal) replaces process-wide execution machinery — it must claim hub slots at boot, gated by `boot_orchestrator`. A **per-call** orchestrator (Mistral) only adds a dispatch arm — presence is enough; selection is per run via `execution_mode`. When you add a new orchestrator plugin, classify it first; the classification decides whether `boot_orchestrator` is even meaningful for it.--**Install channel follows license, not convenience.**--- `pipelex-temporal` is **private** → distributed by a pinned `git+ssh` ref (precedent: `pipelex-shared @ git+ssh://…/infra-python-tools.git@<ref>`). Only **private** images may pin it.-- `pipelex-mistralai-workflows` is **public** → plain PyPI install (`pipelex-mistralai-workflows==X`). (Currently pre-publish; gated on its own `mistralai-2.x` work — treat as PyPI once published.)--**Base-agnostic + flavor = base + one plugin.** The target: `pipelex/pipelex-api` (public, Docker Hub) names *no* orchestrator and ships only the in-process/direct path. Each deployment **flavor** is `base image + exactly one orchestrator plugin + that plugin's activation`. This is what lets a future `pipelex-api-mistral` exist at all — and it is the architecture the rest of this doc builds toward.-------## 2. Where we are today (the gap)--`pipelex-api` is a Temporal-only runner. Concrete coupling sites (line numbers drift; locations hold):--- **Execution has no agnostic path.** `ApiRunner.start()` (`api/routes/pipelex/pipeline.py`) calls `make_temporal_pipe_run().start()` directly. There is **zero** use of the agnostic bridge entry `run_pipe_via_bridge` / `PipelexPipeRunInput` anywhere in `api/`.-- **Validate reads a deleted field.** `ApiRunner.validate()` branches on `get_config().temporal.is_enabled` — a config field **core has removed** — and on the "enabled" arm dispatches `dispatch_dry_validate(DryValidateArg(...))` from `pipelex.temporal.tprl_pipe`. The "disabled" arm already runs `validate_bundles_in_process(...)` in-process.-- **Exception handling imports `temporalio`.** `api/exception_handlers.py` imports `temporalio.exceptions.TemporalError` and registers a dedicated `handle_temporal_error`. It also references `WorkflowExecutionError` (which is a `PipelexError` via `TemporalFlowError`, so it is already covered generically by `handle_pipelex_error`).--Because core deleted both the `temporal` extra and the `temporal` config field, `pipelex-api` against the current `pipelex` worktree is broken on all three sites. The fix is the refactor below, not a pin nudge.--The plugin system inverted **core pipelex's** couplings (Phases 0–5). It did **not** provide seams for the **runner/HTTP layer** — exception-handler contribution and validate dispatch are `pipelex-api` concerns. One small new core seam is required (§3 F3).-------## 3. Locked decisions--These were decided with the user. Do not re-litigate; if you believe one is wrong, raise it explicitly before deviating.--- **D1 — `boot_orchestrator` is set from the flavor's deployment TOML.** The Temporal flavor bakes `[plugins] boot_orchestrator = "temporal"` into its env-specific `pipelex_{env}.toml`. No change to `pipelex-api` for the gate. *Verified safe:* `Pipelex.setup()` only overrides `plugins.boot_orchestrator` when its param is non-`None` (`pipelex/pipelex.py` ~`:206`), so a TOML-provided value is **preserved**, not clobbered, and it is in place before `build_registrar` reads it. The core comment "set programmatically, not from pipelex.toml" describes core's *defaults*; a deployment override is a supported, distinct use.-- **D2 — plugin config files become env-aware, mirroring the main config.** Today `load_temporal_config()` resolves a *single* `temporal.toml` (project-then-global, no env layering), unlike the main config which layers `pipelex_{env}.toml` via `PIPELEX_ENV`. Add env-aware resolution so a plugin's config self-loads `name.toml` (packaged default) → `name_{env}.toml` → `name_override.toml`. Implement as a **reusable core helper** so every plugin inherits identical env semantics; `pipelex-temporal` uses it. One image bakes all envs; `PIPELEX_ENV` selects at runtime.-- **F1 — `execution_mode` = runner config default + optional per-request override, gated by a per-deployment policy.** `[api] execution_mode` sets the deployment default; the request may override it **only** when the deployment's policy allows it (so a locked-down Temporal runner can refuse a caller forcing `direct`). The runner builds `PipelexPipeRunInput(execution_mode=…)` and dispatches through `run_pipe_via_bridge`; the bridge output already carries `workflow_id`, so async-start semantics survive.-- **F2 — validate is always DIRECT in-process.** Drop the Temporal validate branch entirely; `ApiRunner.validate()` always calls `validate_bundles_in_process(...)`. Removes the `pipelex.temporal` imports (`dispatch_dry_validate` / `DryValidateArg`). **Behavior change to call out at handoff:** the API runner now loads the method library to validate, which the prior D10/D14 design deliberately pushed onto the worker. Acceptable per decision; note the resource/isolation implication on the hosted runner in that repo's docs.-- > **REVERSED / EXTENDED — orchestrator-dispatched `/validate`.** F2's "always DIRECT" is superseded: `/validate` is now `execution_mode`-aware through a new per-call `BundleValidatorRegistry` seam (mirroring the `OrchestratorRegistry`). `direct` keeps F2's in-process path (the agnostic base validates in-process); `temporal_*` dispatches the whole job to a worker, restoring what F2 removed — assembled back into the **same** canonical report API-side, so the verdict wire is byte-identical across backends. The library-load implication is now flavor-conditional, not absolute. Plan + as-built: [`orchestrator-dispatched-validate.md`](orchestrator-dispatched-validate.md).-- **F3 — orchestrator-specific HTTP error handling via a new plugin seam, contributed as a mapper (not a raw FastAPI handler).** Core's `PluginRegistrar` gains a framework-agnostic contribution: `add_http_error_mapper(*, exc_type: type[Exception], to_error_report: Callable[[Exception], ErrorReport])`. The orchestrator plugin registers its transport-fault mapping (e.g. `temporalio.TemporalError → ErrorReport[transient, RUNTIME]`). `pipelex-api` iterates the registrar's mappers at app construction and wraps each into a FastAPI handler using its existing RFC 7807 + `DisclosureMode` rendering. **FastAPI/Starlette stays only in `pipelex-api`; core and the plugin import neither.** This grows the plugin contract by one optional capability → bump `PLUGIN_API_VERSION` and document it in the orchestrator SPI.-------## 4. Image architecture (handles public + private plugins uniformly)--A **two-layer** build per flavor. This isolates the plugin-install complexity (private SSH, version-locking) into one place and keeps the env-config layer as trivial as it is today.--```-pipelex/pipelex-api:<ver> (public base, orchestrator-agnostic, Docker Hub)- │- ├─ flavored base = base + ONE plugin (built once per flavor → private ECR)- │ pipelex-api-temporal:<ver> = base + pipelex-temporal (git+ssh)- │ pipelex-api-mistral:<ver> = base + pipelex-mistralai-workflows (PyPI)- │- └─ env-config child = flavored base + COPY env TOMLs (per env; stays config-only)- pipelex-api-{dev,staging,prod} (Temporal)- pipelex-api-mistral-{dev,staging,prod}-```--**One parameterized install recipe across flavors.** The flavored-base Dockerfile takes a build arg `PLUGIN_SPEC` and always exposes a BuildKit SSH mount:--- PyPI plugin (Mistral): `PLUGIN_SPEC="pipelex-mistralai-workflows==X"` → plain install, SSH mount unused.-- Private plugin (Temporal): `PLUGIN_SPEC="pipelex-temporal @ git+ssh://[email protected]/Pipelex/pipelex-temporal.git@<ref>"` → install uses `--mount=type=ssh`, with the deploy key wired into the CI builder (`docker build --ssh default`). Never bake a key into a layer.--Install mechanics that bite (encode these in the Dockerfile):--- The base image's Python lives in a **uv project venv at `/app/.venv`**, run via `uv run uvicorn …`. Install into that venv (`cd /app && uv pip install …`), do **not** let the resolver swap the pinned `pipelex` already baked in (pin / `--no-deps` as appropriate).-- The base image **purges `git`/`build-essential`** in a late layer. Re-add `git` (+`openssh-client` for the private case) in the flavored-base layer before installing.-- **Version-lock is load-bearing.** A plugin imports `pipelex` internals deeply; the plugin ref and the base `pipelex` version are a coupled pair that bump together. Track the pair wherever the flavor repo records versions (e.g. `api-config.toml`).--Whether the flavored base is a *separately-published* ECR image or just an earlier stage in a single multi-stage Dockerfile is a minor call: publish it only if more than one downstream consumes it; otherwise a multi-stage build keeps it to one artifact. Either way the env-config child stays a pure `COPY` of TOMLs, exactly like `pipelex-api-hosted/Dockerfile` today.-------## 5. Phased plan--Order matters: enabling changes in `pipelex` and `pipelex-temporal` must land before `pipelex-api` can consume the new seam.--### Phase A — Core enabling changes (`pipelex`, the `_plugins` worktree)--1. **D2 env-aware plugin config loader.** Add a reusable helper on `config_manager` (e.g. `load_plugin_config(*, name, schema)`) that resolves `name.toml` (packaged default of the calling plugin) → `name_{environment}.toml` → `name_override.toml` with deep-merge, mirroring `load_config`'s env layering (`pipelex/system/configuration/config_loader.py`).-2. **F3 HTTP-error-mapper seam.** Add `add_http_error_mapper(*, exc_type, to_error_report)` to `PluginRegistrar` (`pipelex/plugins/registrar.py`) plus a read accessor for the collected mappers. Use `ErrorReport` (core type) in the signature — **no FastAPI import**. Bump `PLUGIN_API_VERSION` (`pipelex/plugins/contract.py`) and add the capability to the orchestrator SPI doc (`docs/under-the-hood/orchestrator-plugins.md`).-3. Tests: env-layered plugin-config round-trip; registrar collects mappers; import-light boot still green (the seam adds no SDK/HTTP import).--**Acceptance:** `make agent-check` clean · `make tb` green · `make agent-test` green. Core still names no orchestrator and imports no FastAPI.--### Phase B — `pipelex-temporal` (additive)--1. Switch `load_temporal_config()` to the D2 helper; ship `temporal_{env}.toml` resolution (the packaged default `temporal.toml` stays).-2. In `TemporalPlugin.register()`, register the F3 mapper: `temporalio.TemporalError → ErrorReport` classified transient / `RUNTIME` (port the classification currently living in `pipelex-api`'s `handle_temporal_error`). Keep `register` import-light — the mapper closure may import `temporalio` lazily when first invoked, matching the existing thunk discipline.--**Acceptance:** `pipelex-temporal` `make agent-check` + `make agent-test` green against the Phase A `pipelex`.--### Phase C — `pipelex-api` decoupling (the big one)--Make the base orchestrator-agnostic. No `pipelex.temporal.*`, no `temporalio`, anywhere in `api/`.--1. **Execution (F1).** Rewrite `ApiRunner.start()` / `execute` to build `PipelexPipeRunInput(…, execution_mode=…)` and call `run_pipe_via_bridge`; read `workflow_id` off the output payload. Resolve `execution_mode` from a new `[api] execution_mode` config default, with an optional per-request override gated by a per-deployment policy (e.g. `[api] allow_request_execution_mode_override` and/or an allowlist). Delete `make_temporal_pipe_run` usage + import.-2. **Validate (F2).** Collapse `ApiRunner.validate()` to the single in-process path (`validate_bundles_in_process`). Delete the `temporal.is_enabled` branch and the `dispatch_dry_validate` / `DryValidateArg` imports. Document the library-load-on-API behavior change in `pipelex-api/docs/`.-3. **Exception handlers (F3).** Remove the `temporalio` import and the hardcoded `handle_temporal_error` registration from `api/exception_handlers.py`. At app construction (`api/main.py` → `register_exception_handlers`), iterate `registrar` HTTP-error mappers and register a FastAPI handler per `exc_type` that runs the mapper then renders via the existing RFC 7807 + `DisclosureMode` path. Keep `handle_pipelex_error` (it already covers `WorkflowExecutionError`); drop the explicit `WorkflowExecutionError` import/catch in `validate.py` (now unreachable — validate is DIRECT and never raises it).-4. **Deps.** Drop the `temporal` extra from `pyproject.toml` (`pipelex[mistralai,anthropic,google,…]==<ver>`, no `temporal`). The base depends on **no** orchestrator plugin.-5. Tests: update `tests/unit/test_exception_handlers.py`, `test_validate_*` to the agnostic shape (a synthetic plugin contributing a mapper proves the seam without importing `temporalio`).--**Acceptance:** `pipelex-api` test suite green with **no** `pipelex.temporal` / `temporalio` import in `api/`; `grep -rn "temporal" api/` returns only incidental prose. A direct-mode run + validate works with no orchestrator plugin installed.--### Phase D — `pipelex-api-hosted` (Temporal flavor)--1. Flavored base (`pipelex-api-temporal`): `FROM pipelex/pipelex-api:<ver>` + install `pipelex-temporal @ git+ssh://…@<ref>` per §4 (re-add `git`+`openssh-client`, SSH mount, version-locked to `<ver>`). Wire the deploy key into CI (`--ssh default`).-2. Config migration in the baked `.pipelex/pipelex_{env}.toml`:- - Replace `[temporal] is_enabled = true` → `[plugins] boot_orchestrator = "temporal"`.- - Add `[api] execution_mode = "temporal_fire_and_forget"` (or `temporal_blocking` per route semantics) + the override policy you want.- - **Move** the connection tree (`temporal_config`, `worker_config`, `queue_options`, `search_attributes`) **out** of `pipelex_{env}.toml` into baked `.pipelex/temporal_{env}.toml` files (the D2 plugin-config file). The main `pipelex_{env}.toml` no longer carries any `[temporal.*]` keys (core would reject them).-3. The env-config child Dockerfile stays a `COPY` — now copying `pipelex_{env}.toml` **and** `temporal_{env}.toml`.--**Acceptance:** dev image boots with Temporal active; a run enqueues to Temporal Cloud; `PIPELEX_ENV` selects the right `temporal_{env}.toml`.--### Phase E — `pipelex-api-mistral` (Mistral flavor, new repo)--Scaffold mirroring `pipelex-api-hosted` conventions.--1. Flavored base (`pipelex-api-mistral`): `FROM pipelex/pipelex-api:<ver>` + `uv pip install pipelex-mistralai-workflows==X` (PyPI — no SSH, no `git` re-add needed).-2. Config in baked `.pipelex/pipelex_{env}.toml`: `[api] execution_mode = "mistral_native"`. **No** `boot_orchestrator` (per-call orchestrator), **no** plugin config file (the Mistral plugin self-loads none today).-3. Gated on `pipelex-mistralai-workflows` being published and on its in-flight `mistralai-2.x` work — coordinate, don't disrupt.--**Acceptance:** image boots with `mistral_native` dispatchable; a run routes through the Mistral orchestrator; no Temporal anything present.--### Phase F — `pipelex-worker` pin flip (from the existing cut-list)--Per the Phase 5 cut-list in [`TODOS.md`](../../TODOS.md): `pyproject.toml` `pipelex[dynamodb,s3,temporal]==X` → `pipelex[dynamodb,s3]==X` + `pipelex-temporal @ git+ssh://…@<ref>`; `Dockerfile` CMD `["pipelex","worker",…]` → `["pipelex-temporal","worker",…]`; `Makefile` `pipelex worker` → `pipelex-temporal worker`; set `boot_orchestrator=temporal` for the worker process. Land in the **same** commit as the `pipelex` pin bump.--### Checkpoints--- **Checkpoint 1 — after Phase B:** core + `pipelex-temporal` green together; the seam exists and the plugin contributes through it. Natural session boundary before the `pipelex-api` refactor.-- **Checkpoint 2 — after Phase C:** `pipelex-api` is agnostic and green with no orchestrator installed. THE gate — flavors are meaningless until the base is clean. Fan out `/code-review` on the `api/` diff.-- **Checkpoint 3 — after Phase D:** first real flavor boots end-to-end against Temporal Cloud (dev). Validates the whole chain before replicating for Mistral.-------## 6. Per-repo agent handoff summary--| Repo | Change | Channel |-|---|---|---|-| `pipelex` (`_plugins`) | D2 env-aware plugin-config helper; F3 `add_http_error_mapper` seam + `PLUGIN_API_VERSION` bump; SPI doc | normal release |-| `pipelex-temporal` | use D2 helper; register F3 `TemporalError → ErrorReport` mapper | private git+ssh |-| `pipelex-api` | drop `temporal` extra; route execution through bridge (F1); validate DIRECT (F2); consume F3 mappers; remove all `pipelex.temporal`/`temporalio` imports | public Docker Hub |-| `pipelex-api-hosted` | flavored-base install of `pipelex-temporal` (git+ssh); `boot_orchestrator=temporal` + `[api] execution_mode` in `pipelex_{env}.toml`; connection tree → `temporal_{env}.toml` | private ECR |-| `pipelex-api-mistral` (new) | scaffold; flavored-base install of `pipelex-mistralai-workflows` (PyPI); `[api] execution_mode=mistral_native`, no boot gate | private ECR |-| `pipelex-worker` | pin `pipelex-temporal` (git+ssh); CLI → `pipelex-temporal worker`; `boot_orchestrator=temporal` | private |-------## 7. Open items / risks to verify during execution--- **F3 — confirm transport faults are wrapped, or rely on the mapper.** Verify whether the Temporal orchestrator already wraps bare `temporalio` transport errors into `PipelexError`s (`TemporalServerError` etc.) before they reach HTTP. If it does, the F3 mapper is belt-and-suspenders; if not, it is the only thing keeping such errors off the catch-all 500. Either way the mapper makes the base correct without naming Temporal.-- **F1 — define the override policy precisely.** Decide the exact shape (`allow_request_execution_mode_override` bool vs a per-deployment allowlist of modes) and the default (recommend: override **off** on hosted flavors). A caller must never be able to force `direct` on a runner whose whole point is distributed execution.-- **F2 — quantify the library-load cost on the hosted runner.** Validate now loads the method library API-side. Confirm the hosted runner image/resources are sized for it, and document it in `pipelex-api`/`pipelex-api-hosted` docs.-- **Version-lock bookkeeping.** Record the (base `pipelex-api` version ⇄ plugin ref) pair in each flavor repo's version file so a bump can't drift them apart.-- **Mistral publish gating.** `pipelex-mistralai-workflows` must be published to PyPI and its `mistralai-2.x` work settled before Phase E.-- **Docs deploy.** The Temporal error pages left in core `docs/errors/` on purpose (type_uri dereference targets) still need a home decision when `pipelex-temporal` docs exist — tracked in the Phase 5 cut-over notes, not here.diff --git a/wip/plugins/orchestrator-dispatched-validate-review-findings.md b/wip/plugins/orchestrator-dispatched-validate-review-findings.mddeleted file mode 100644index 311850722..000000000--- a/wip/plugins/orchestrator-dispatched-validate-review-findings.md+++ /dev/null@@ -1,47 +0,0 @@-# Review findings — orchestrator-dispatched `/validate` (PR #998)--Captured from a pre-landing `/review` (gstack) of the `feature/Orchestrator-dispatched-validate` feature across all three repos (`pipelex` core = PR #998 vs `refactor/Plugins-4`; `pipelex-temporal` vs `refactor/own-temporal-config`; `pipelex-api` vs `refactor/orchestrator-agnostic-base`). Verdict: **landable.** No critical findings. Greptile P1 (verdict alias bound under `TYPE_CHECKING`) already fixed by `34a33633d`. The items below were surfaced by the testing + adversarial passes and are deferred (none is a clear, unambiguous win; the brief was clean-solid, no speculative guards). Recorded so they are not re-discovered cold.--## 1. MISTRAL_NATIVE `/validate` is unserviceable; its "missing" error names an already-installed package (deferred — out of scope by plan)--**Observation (adversarial, conf 8).** `pipelex-mistralai-workflows`'s plugin (`mistral_plugin.py:31`) registers only the MISTRAL_NATIVE *orchestrator*, never a bundle validator. So a deployment configured `execution_mode = mistral_native` (plugin installed, `/start` working) would fail every `/validate` with `MissingBundleValidatorError` → 5xx — and that error's MISTRAL_NATIVE branch (`pipelex/runtime_bridge/exceptions.py`) tells the operator to `pip install pipelex-mistralai-workflows`, a package that is already installed in that scenario. `/start` and `/validate` are asymmetric for the one orchestrator whose error message specifically promises serviceability.--**Why deferred, not fixed.** This is the **explicitly deferred Mistral arm**, not a regression this PR introduces:--- The plan (`orchestrator-dispatched-validate.md`) defers Mistral in three places: START-HERE ("Mistral is a gated follow-up — do **not** build it"), decision **C** ("Mistral deferred… gated on its own `mistralai-2.x` work"), and Phase **V-Mistral — DEFERRED** ("Add a `MistralBundleValidator` later… The generic seam means this lands without touching core or the API").-- `pipelex-mistralai-workflows` is itself gated from publish (HOLD on `mistralai-2.x` / instructor #2298), so `mistral_native` is not a shippable `/validate` config on any deployment that ships today. The dead path is unreachable in production.-- The error-message wording mirrors the parallel `MissingOrchestratorError` MISTRAL_NATIVE branch verbatim; once the Mistral validator lands, the install hint becomes directionally correct again. Touching only the message now (to soften the "install X" claim) is a judgment call, not a mechanical win, and would desync the two parallel error classes.--**When the Mistral validator lands (Phase V-Mistral):** register a `MistralBundleValidator` under MISTRAL_NATIVE (genuine dispatch, or an in-process passthrough — validation is backend-independent pure analysis). At that point `MissingBundleValidatorError`'s MISTRAL_NATIVE branch is reachable only as a true "plugin not installed" signal and its message is correct. If Mistral is *intentionally* never going to serve `/validate`, revisit the message then (e.g. point MISTRAL_NATIVE `/validate` at the DIRECT in-process validator as a fallback) — but that is a Phase-D/V-Mistral decision, not this PR's.--## 2. The `cast` at the verdict narrowing point launders a non-`PipelexValidationReport` valid arm (deferred — latent, loud-fail, future-plugin guard)--**Observation (adversarial, conf 6).** The seam's valid arm is typed at the protocol-level `ValidationReport` (a leaf type, to stay import-acyclic in core — see §9 divergence in the plan). `ApiRunner.validate_verdict` ends with `return cast("PipelexValidationReport | ErrorReport", verdict)`, and the route discriminates with `if not isinstance(verdict, PipelexValidationReport): return _invalid_report_response(verdict, ...)`. The protocol `ValidationReport` valid arm has neither `validation_errors` nor `message` (confirmed in `mthds/protocol/models.py`), and `_invalid_report_response` accesses `error_report.validation_errors` / `error_report.message`. So a *future* validator whose valid arm returned a bare `ValidationReport` (not the `PipelexValidationReport` subtype) would fail the `isinstance`, be routed to the invalid-arm handler, and crash on attribute access.--**Why not a bug today, why deferred.**--- Both shipped validators (`DirectBundleValidator`, `TemporalBundleValidator`) are annotated `-> PipelexValidationReport | ErrorReport` and in fact return the concrete envelope; the type checker enforces this for every in-tree validator. The `cast` is the single documented narrowing point the plan's §9 acceptance note already calls out.-- The failure mode for a hypothetical non-conforming external plugin is **loud (5xx crash), not silent data corruption** — the worst-case is an obvious server error, not a wrong verdict.-- Adding a runtime guard ("validator returned a non-Pipelex report → raise a clear fault") would broaden the seam's surface to defend a scenario no in-tree code can reach — exactly the speculative guarding the review brief excludes ("no guarding impossible scenarios, no broadening surface area").--**If revisited:** if external/third-party bundle validators ever become a supported extension point (not just core + the two first-party plugins), add a single explicit guard at the narrowing point in `ApiRunner.validate_verdict` — `if isinstance(verdict, ErrorReport) or isinstance(verdict, PipelexValidationReport): ...` else raise a clear "validator must produce a PipelexValidationReport" fault — so a misbehaving plugin gets a precise diagnostic instead of an `AttributeError` deep in the route. Until then the `cast` + the type-checked first-party validators are sufficient.--## 3. Test-coverage symmetry gaps (deferred — covered transitively / by convention)--Testing pass, all INFORMATIONAL, none a correctness gap:--- **Dispatched-*valid* arm has no ungated fast-suite test (conf 6).** `pipelex-api/tests/unit/test_validate_dispatch.py` exercises the non-direct *invalid* (`ErrorReport` → 200) and *fault* (→ 5xx) arms with a stub validator, but not the non-direct *valid* (`PipelexValidationReport` → 200 `is_valid: true`) arm. That valid dispatched path is covered by `pipelex-temporal/tests/integration/test_temporal_bundle_validator_in_memory.py`, which is `@pytest.mark.temporal`-gated (needs `temporalio` + an in-memory server) and excluded from the default fast `agent-test` run. The route mapping is backend-independent and the DIRECT valid arm is covered ungated by the existing `/validate` suite, so risk is low. This matches the plan's §7 V2 acceptance bar, which deliberately scoped the stub tests to invalid/fault/missing/403 and relied on V1 for the valid path. **If desired:** add a `TestValidateDispatch` case with a stub validator returning a real `PipelexValidationReport` for a temporal mode, asserting 200 + `is_valid: true` — proves the valid dispatched arm with no `temporalio` import. Nice-to-have symmetry, not a bug.-- **`DirectOrchestratorPlugin.register` validator line untested in core (conf 5)** and **`TemporalBundleValidator._require_temporal_extra` guard untested (conf 4).** Both are zero-coverage lines, but each is consistent with the pre-existing untested-by-convention sibling (the DIRECT *orchestrator* registration is likewise not directly asserted in core; the orchestrator's own `_require_temporal_extra` is likewise untested). Wiring is covered transitively by the cross-repo boot/envelope tests. Flagging only for the record; not worth a new test against the existing convention.--## Passes with no findings--- **Maintainability:** clean — no dead code/imports, F2 "always DIRECT" prose thoroughly reversed across docstrings + `docs/pipe-validate.md` + CHANGELOG + trackers, no DRY violation worth extracting (the `start`/`validate_verdict` duplication is the separately-noted deliberate mirror; the two `Missing*Error` classes are parallel-by-design), no module-boundary violation (the protocol-level typing is a *deliberate* import-cycle avoidance).-- **API contract:** verified byte-equivalent for the direct path; `execution_mode` field is backward-compatible (optional, default None → deployment default); the 200-always-for-verdict / non-2xx-for-no-verdict contract is preserved including the new forbidden-override 403; `isinstance(verdict, PipelexValidationReport)` is provably correct (disjoint MROs — neither `ErrorReport` nor `PipelexValidationReport` subclasses the other).-- **Enum completeness:** `MissingBundleValidatorError._build_message`'s `match` over `PipelexExecutionMode` is exhaustive (all four members, `-> str`, no `case _`); a 5th member would be caught by the linter. Mirrors the existing pattern in `execution_mode.py`.-- **Boot wiring:** `set_bundle_validator_registry` is called in the single boot block (`pipelex.py`) immediately after `set_orchestrator_registry`; identical lifetime guarantees to the already-production orchestrator registry — no partial-boot gap introduced.--## Already-captured (pre-existing) deferred notes — not re-opened--- `validate-dispatch-requires-boot-active.md` — dispatched `/validate` effectively requires `boot_orchestrator == "temporal"` at runtime (reused executor guard); identical to the run orchestrators, pre-existing, gated on Phase D's boot-model decision.-- `validate-verdict-subclass-parity-divergence.md` — the Temporal arm's `error_type != ValidateBundleError.__name__` string-equality vs the direct arm's `except ValidateBundleError` subclass-catch; latent (zero subclasses exist), forced by the wire boundary.-- `validate-start-mode-dispatch-duplication.md` — `start`/`validate_verdict` share a ~3-line resolve-then-dispatch shape; extraction deferred as added indirection, not removed complexity.diff --git a/wip/plugins/orchestrator-dispatched-validate.md b/wip/plugins/orchestrator-dispatched-validate.mddeleted file mode 100644index 3539f4950..000000000--- a/wip/plugins/orchestrator-dispatched-validate.md+++ /dev/null@@ -1,235 +0,0 @@-# Plan — orchestrator-dispatched `/validate` (reverse/extend F2)--> **⚠️ SUPERSEDED (2026-06-22) by [`orchestration-mode-and-delivery-split.md`](orchestration-mode-and-delivery-split.md).** This plan was implemented (V0–V3, committed across all three repos) on top of the **flat** `PipelexExecutionMode` enum, and registered the bundle validator under **both** temporal modes (`temporal_blocking` + `temporal_fire_and_forget`) — a double-registration that existed only because the flat enum lacked a delivery axis. The orchestration-mode/delivery split (now DONE, incl. Phase 4 — the `pipelex-mistralai-workflows` migration) **removed that flatness**: delivery became a `DeliveryMode` parameter to `OrchestratorProtocol.run`, the two Temporal orchestrators collapsed into one registered **once** under `"temporal"`, and the `/validate` double-registration collapsed to a single `"temporal"` validator. The orchestrator-dispatched `/validate` mechanism this plan built still stands; only the keying changed (open `orchestration_mode` token, single registration). Read the split plan for the current surface; this doc is kept for history. (The "nothing coded yet" status below is itself stale — this plan was completed; see its sibling `execute-per-request-execution-mode.md` for the as-built cross-references.)--**Status:** ready-to-execute plan for a fresh session. Design is locked (decisions below); nothing coded yet. This is **Plan 1 of 2** — its sibling is `wip/plugins/execute-per-request-execution-mode.md` (run this one first). It **reverses/extends locked decision F2** of the orchestrator-agnostic-runner effort (tracker: `_plugins/TODOS.md`; parent plan: `_plugins/wip/plugins/orchestrator-agnostic-runner-and-flavors.md`).--> **START HERE (cold start).** Read this whole doc, then §"Current state" and §"Background". Begin on the branches named in §"Cross-repo state". The work spans `pipelex` (core seam) → `pipelex-temporal` (wire the surviving validate machinery into the seam) → `pipelex-api` (mode-aware dispatch + route). Mistral is a gated follow-up (do **not** build it). Same one-commit-per-checkpoint / clean-context-`/code-review` discipline as the rest of the effort.-------## 1. The ask + locked decisions--**The ask (user, verbatim intent):** make `/validate` dispatchable through the active orchestrator, the way `/start` runs a pipe — instead of always validating in-process on the API runner.--1. **Temporal flavor** → `/validate` can dispatch the surviving `act_dry_validate` activity to a worker (restore the behavior F2 removed).-2. **Do both** → keep the in-process path for the agnostic base / `direct` mode AND add the dispatched path for orchestrator flavors. Validate becomes `execution_mode`-aware: `direct` → in-process; `temporal_*` → worker activity. The base validates in-process because it has no worker — expected.-3. **Mistral flavor** → the seam must be **generic across orchestrators**, not Temporal-specific, so a Mistral validate arm can be added later.--**Locked decisions (do not re-litigate):**--- **A — Verdict as value (return a union).** The seam **returns** the verdict, it does not raise it. Valid → the canonical report; invalid → the structured invalid report. Respect the existing `/validate` contracts in `docs/specs/` and `conformance/` and their spirit (200-always for any produced verdict; non-2xx = no verdict).-- **B — Seam returns the assembled report.** The worker returns the raw artifact bundle (`DryValidateResult`); the **orchestrator plugin** (running in-API) assembles the canonical report via core `build_validation_report`. The API route stays thin and uniform. (Revisit only if this turns out materially more complex than the in-route assembly alternative — see §"Open sub-decisions".)-- **C — Mistral deferred** as a follow-up (gated on its own `mistralai-2.x` work). Build the generic seam now so Mistral slots in later.-- **D — Reuse `PipelexExecutionMode`.** Validate is inherently blocking, so `temporal_blocking` and `temporal_fire_and_forget` collapse for validate: the Temporal plugin registers the **same** validator under **both** temporal modes. `/validate` reuses `resolve_execution_mode` + the override policy wholesale.-- **Per-call, presence-based, NOT boot-gated.** Validate dispatch joins the **per-call registry family** (like `/start`), not the boot-global hub-slot family (like `/execute` today). The validator is registered **unconditionally** by the plugin (exactly how `pipelex-temporal` already registers its run orchestrators); only hub-slot *claiming* is `boot_orchestrator`-gated, and validate claims no slot. This resolves the brief's old open question #5: `dispatch_dry_validate` needs only a reachable Temporal client + a worker with the validate workflow registered — **not** a runner booted-as-Temporal.-------## 2. Current state (post-Phase C — the thing we are reversing)--Phase C made `pipelex-api` orchestrator-agnostic (`pipelex-api` tip `a39841e` on `refactor/orchestrator-agnostic-base`). As part of it, **F2** collapsed `/validate` to a single in-process path. F2 verbatim (from the parent plan, §3):--> **F2 — Validate always DIRECT in-process.** Drop the Temporal validate branch entirely; `ApiRunner.validate()` always calls `validate_bundles_in_process(...)`. Removes the `pipelex.temporal` imports (`dispatch_dry_validate` / `DryValidateArg`). Behavior change to call out at handoff: the API runner now loads the method library to validate, which the prior D10/D14 design deliberately pushed onto the worker. Acceptable per decision; note the resource/isolation implication on the hosted runner in that repo's docs.--So today:--- `pipelex-api/api/routes/pipelex/pipeline.py` `ApiRunner.validate` (lines `182-217`) calls only `validate_bundles_in_process(...)` — always in-process, loads the method library on the runner, **no `execution_mode` awareness**. Docstring at line `73` and the route comment at `190-193` assert "always DIRECT in-process (F2)" — **these are the sentences to edit.**-- The route `api/routes/pipelex/validate.py` (lines `141-201`) catches only `ValidateBundleError` (its old `WorkflowExecutionError` catch was removed).-- **The worker machinery survives intact and tested in `pipelex-temporal`, just orphaned** (no longer called by the API): `pipelex_temporal/tprl_pipe/{act_dry_validate,wf_dry_validate,dry_validate_dispatch}.py`, plus unit + in-memory-Temporal integration tests. **This is a re-wiring, not a rebuild.**-- **Phase C dropped `pipelex-api`'s dependency on `pipelex-temporal` entirely.** `pipelex-api/pyproject.toml` deps are now just `pipelex`, `mthds`, `fastapi`, `pyjwt`, `uvicorn`. The agnostic base must stay temporal-free **even in tests** (see §"Testing").--The old dual-path `ApiRunner.validate` (the exact code to mirror) is one commit back: `git -C pipelex-api show a39841e^:api/routes/pipelex/pipeline.py`. Its Temporal arm dispatched **first** (`dispatch_dry_validate(DryValidateArg(mthds_contents, mthds_sources, allow_signatures))`) so every failure surfaced through the worker's `validate_bundle` cascade with the same categorized `ValidateBundleError` identity the direct path raises; then it parsed blueprints **cheaply (no library)** via `PipelexInterpreter.make_pipelex_bundle_blueprint(...)` and assembled the canonical report via `build_validation_report(blueprints=…, pipe_io_contracts=result.pipe_io_contracts, dry_run_result=result.dry_run_outputs, pending_signatures=result.pending_signatures, graph_spec=result.graph_spec)`. The route's old arm caught `WorkflowExecutionError`, recovered the `ValidateBundleError` report (→ 200 invalid) or re-raised a genuine fault (→ 5xx). **Mirror this split**, but as a returned verdict (decision A), not an exception.-------## 3. Background — the two existing dispatch seams (cold-start orientation)--The runner already has **two** orchestration mechanisms. Validate joins the first.--**(a) Per-call `OrchestratorRegistry`** (the model validate mirrors), all in core under `_plugins/pipelex/` (a worktree of `pipelex`):--- `pipelex/plugins/orchestrator_registry.py:21` — `OrchestratorProtocol.run(self, *, pipe_job, delivery_assignment) -> PipelexPipeRunOutput` (single-method `@runtime_checkable` Protocol).-- `pipelex/plugins/orchestrator_registry.py:24-42` — `OrchestratorRegistry(orchestrators: dict[PipelexExecutionMode, OrchestratorProtocol])` with `get_optional(*, mode) -> OrchestratorProtocol | None` and `has(*, mode) -> bool`.-- `pipelex/plugins/registrar.py:163-173` — `PluginRegistrar.add_orchestrator(*, mode, orchestrator)` (uses an internal `_add` helper that fail-louds on duplicates).-- `pipelex/plugins/direct/direct_plugin.py:18-19` — core `DirectOrchestratorPlugin.register` calls `registrar.add_orchestrator(mode=DIRECT, orchestrator=DirectOrchestrator())`; wired unconditionally via `pipelex/plugins/builtins.py:27-28`.-- `pipelex/pipelex.py:391-395` — registries are built **once at boot** from the registrar and stored on the hub; `:206` shows `boot_orchestrator` is a separate boot-global string (claims hub slots) — orthogonal to per-call mode dispatch.-- `pipelex/runtime_bridge/bridge.py:92-95` — per-call dispatch: `get_orchestrator_registry().get_optional(mode=...)` → `MissingOrchestratorError` if absent (no silent fallback) → `orchestrator.run(...)`.-- `pipelex/runtime_bridge/execution_mode.py:4-27` — `PipelexExecutionMode`: `DIRECT`, `TEMPORAL_BLOCKING`, `TEMPORAL_FIRE_AND_FORGET`, `MISTRAL_NATIVE`.-- `pipelex/runtime_bridge/exceptions.py:9-40` — `MissingOrchestratorError` (carries a per-mode pip-install hint).-- `pipelex/runtime_bridge/direct_orchestrator.py` — `DirectOrchestrator.run` returns `serialize_completed_output(pipe_output=…, workflow_id=None)`; scopes an in-process router so nested sub-pipes stay in-process even inside a Temporal worker.-- `pipelex-temporal/pipelex_temporal/temporal_plugin.py:126-147` — `TemporalPlugin.register` adds the run orchestrators **unconditionally** under `TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`; `claim_*` hub slots only `if boot_orchestrator == "temporal"`. Entry point: `[project.entry-points."pipelex.plugins"] temporal = "pipelex_temporal.temporal_plugin:TemporalPlugin"`.--**(b) Boot-global hub slots** — `get_pipe_run()` / `get_pipe_router()` etc., claimed at boot when `boot_orchestrator == "temporal"`. This is what `/execute` uses today (sibling plan converts it to (a)). Validate does **not** touch this.--**Core validate building blocks** (reused as-is, all under `_plugins/pipelex/`):--- `pipelex/pipeline/validate_in_process.py:44-51` — `async validate_bundles_in_process(*, mthds_contents, mthds_sources=None, library_dirs=None, allow_signatures=False, log_context="validate") -> PipelexValidationReport`. Loads + tears down the method library internally (heavy, lifecycle-managed). Raises `ValidateBundleError` on an invalid bundle.-- `pipelex/pipeline/validation_report.py:63-94` — `build_validation_report(*, blueprints, pipe_io_contracts, dry_run_result, pending_signatures, graph_spec=None) -> PipelexValidationReport`. The single assembly point.-- `pipelex/pipeline/validation_report.py:29-60` — `PipelexValidationReport` (extends the neutral `ValidationReport`; `is_valid: Literal[True]` is the valid-arm discriminant; Pipelex-branded envelope).-- `pipelex/pipeline/exceptions.py:72-161` — `ValidateBundleError`; `.to_error_report()` (line `129`) → `ErrorReport` with `validation_errors` populated.-- `pipelex/base_exceptions.py:309-349` — `ErrorReport` (`validation_errors: list[ValidationErrorItem] | None`); `:269-307` `ValidationErrorItem`.-- `pipelex/core/interpreter/interpreter.py` — `PipelexInterpreter.make_pipelex_bundle_blueprint(*, mthds_content, mthds_source)` (cheap parse, no library).--**Surviving Temporal validate machinery** (in `pipelex-temporal`, reused as-is):--- `pipelex_temporal/tprl_pipe/act_dry_validate.py:62-74` — `DryValidateArg(mthds_contents, mthds_sources, allow_signatures, pipe_code)` (plain `BaseModel`, no `temporalio`); `:77-98` — `DryValidateResult(dry_run_outputs, graph_spec, pending_signatures, pipe_io_contracts)`. **These fields map 1:1 onto `build_validation_report`.**-- `pipelex_temporal/tprl_pipe/dry_validate_dispatch.py:22-66` — `async dispatch_dry_validate(arg, *, task_queue=None, should_auto_connect_temporal=True) -> DryValidateResult`. Needs a reachable Temporal client + a worker with `WfDryValidate`/`act_dry_validate` registered; **not** boot-as-Temporal. An invalid bundle comes back as `WorkflowExecutionError` carrying the recovered `ErrorReport` (`error_type == "ValidateBundleError"`); a genuine fault recovers no such report.-- Tests to mirror: `tests/unit/pipelex_temporal/test_dry_validate_dispatch.py`, `tests/integration/pipelex_temporal/test_dry_validate_activity_in_memory.py`, `tests/integration/pipelex_temporal/test_validate_sweep_stays_in_process.py`.-------## 4. The design (locked)--A new **generic, per-call validate seam**, mirroring the orchestrator seam exactly. Core declares the contract (SDK-agnostic core types only); the plugin implements it; the API resolves the mode and dispatches. **The Temporal wire payloads (`DryValidateArg`/`DryValidateResult`) stay in `pipelex-temporal`** — the seam never exposes them, so no payload migration to core is needed.--### 4.1 Core seam (proposed names — finalize in implementation)--- **Verdict union** (core types only): `BundleValidationVerdict = PipelexValidationReport | ErrorReport`. Valid arm = `PipelexValidationReport`; invalid arm = `ErrorReport` (carrying `validation_errors`). This is exactly what the route already maps to the wire (`ValidReport` / `_invalid_report_response`). Discriminate in the route by `isinstance(verdict, PipelexValidationReport)`.-- **Protocol** `BundleValidatorProtocol` (e.g. `pipelex/plugins/bundle_validator_registry.py`):-- ```- async def validate_bundles(- self, *, mthds_contents, mthds_sources, allow_signatures, library_dirs,- ) -> BundleValidationVerdict: ...- ```-- Returns the verdict (valid report | invalid `ErrorReport`); **raises only** for no-verdict infra faults (→ global 5xx). Note: `library_dirs` is host context the **in-process** arm needs; the Temporal arm ignores it (the worker loads its own library).-- **Registry** `BundleValidatorRegistry(validators: dict[PipelexExecutionMode, BundleValidatorProtocol])` with `get_optional(*, mode)` / `has(*, mode)` — copy `OrchestratorRegistry`.-- **Registrar** `PluginRegistrar.add_bundle_validator(*, mode, validator)` — copy `add_orchestrator` (reuse the `_add` dup-guard helper).-- **Hub** `get_bundle_validator_registry()` + boot wiring in `pipelex/pipelex.py` next to the orchestrator registry (built once at boot from the registrar).-- **Missing-validator error** `MissingBundleValidatorError(mode=...)` — copy `MissingOrchestratorError`, with a per-mode install hint.-- **Core direct impl** `DirectBundleValidator` (registered by the core `direct` plugin, alongside `DirectOrchestrator`): calls `validate_bundles_in_process(mthds_contents=…, mthds_sources=…, library_dirs=…, allow_signatures=…, log_context="API validate")`; **catches `ValidateBundleError` → returns `exc.to_error_report()`**; returns the `PipelexValidationReport` on success. Any other exception propagates (→ 5xx). This gives the agnostic base in-process validate "for free" via the same seam.--### 4.2 Temporal impl (in `pipelex-temporal`)--- **`TemporalBundleValidator`** (implements `BundleValidatorProtocol`): builds `DryValidateArg(mthds_contents, mthds_sources, allow_signatures)` (ignores `library_dirs`) → `await dispatch_dry_validate(arg)`. On success (`DryValidateResult`): parse blueprints cheaply with `mthds_sources` threaded (so success-path `bundle_blueprint.source` matches the failure path), then `build_validation_report(blueprints=…, pipe_io_contracts=result.pipe_io_contracts, dry_run_result=result.dry_run_outputs, pending_signatures=result.pending_signatures, graph_spec=result.graph_spec)` → return the `PipelexValidationReport`. On `WorkflowExecutionError`: recover the `ErrorReport`; if it is a validation verdict (`error_type == ValidateBundleError.__name__`) → **return** the `ErrorReport` (invalid verdict); otherwise **re-raise** (genuine fault → 5xx).-- **Register it unconditionally** in `TemporalPlugin.register` under **both** `TEMPORAL_BLOCKING` and `TEMPORAL_FIRE_AND_FORGET` (decision D), right next to the existing `add_orchestrator` calls (which are already unconditional). Claims no hub slot.--### 4.3 API dispatch + route (in `pipelex-api`)--- The dispatch lives on `ApiRunner` (mirrors how `ApiRunner.start` resolves the mode and dispatches), but **returns the verdict union** so the route maps it as a value (decision A). Suggested shape: a method `ApiRunner.validate_verdict(*, mthds_contents, mthds_sources, allow_signatures, requested_execution_mode) -> BundleValidationVerdict` that does `resolve_execution_mode(...)` → `get_bundle_validator_registry().get_optional(mode=...)` (→ `MissingBundleValidatorError` if absent) → `validator.validate_bundles(mthds_contents=…, mthds_sources=…, allow_signatures=…, library_dirs=self.library_dirs)`. Keep `ApiRunner` as the holder of library config (unchanged `self.library_dirs` sourcing).-- **The protocol method** `ApiRunner.validate(...) -> PipelexValidationReport` (the `PipelexMTHDSProtocol` `@override`) stays raise-based for protocol conformance: it can delegate to `validate_verdict` and re-raise on the `ErrorReport` arm. The **route uses `validate_verdict`** for the verdict-as-value mapping. (If nothing besides the route calls the protocol `validate`, the cold session may simplify — confirm first.)-- **Route** `validate.py` (`141-201`): replace the `try/except ValidateBundleError` with: resolve render formats (unchanged) → `verdict = await ApiRunner(...).validate_verdict(..., requested_execution_mode=request_data.execution_mode_or_equivalent)` → `if isinstance(verdict, PipelexValidationReport): return <ValidReport built as today>` else `return _invalid_report_response(verdict, requested_formats=...)`. Genuine faults propagate to the global `problem+json` handler. **`ValidReport` / `InvalidReport` / `_invalid_report_response` are unchanged** (`validate.py:79-138, 204-228`) — only the control flow changes from catch-exception to match-on-returned-verdict.-- Wire the `execution_mode` extra into the validate request shape if not already present (the `/validate` request must carry the optional per-request mode the same way `/start` does, gated by `resolve_execution_mode`'s policy). Confirm `ValidateRequest` vs `PipelineApiExtras` and thread it consistently.--### 4.4 Why the wire contract is preserved (conformance-neutral)--The valid/invalid wire arms (`ValidReport`, `InvalidReport`), the 200-always verdict contract, and the structured `validation_errors[]` are **identical** regardless of backend. The dispatched arm assembles the **same** `PipelexValidationReport` via the **same** `build_validation_report`, and surfaces the **same** `ValidateBundleError`-derived `ErrorReport` on invalid. So the existing `/validate` conformance must pass **unchanged** — that byte-identity is the acceptance bar (§"Testing").-------## 5. Phased implementation + checkpoints--Each phase lands in its own repo on its own branch, one commit per checkpoint. Cut new branches from the named tips.--### Phase V0 — core seam (`pipelex`, in `_plugins/`, branch off `refactor/Plugins-4`)--- Add the protocol, registry, registrar method, hub getter, boot wiring, `MissingBundleValidatorError`, the `BundleValidationVerdict` type, and `DirectBundleValidator` (registered by the `direct` plugin). Mirror the orchestrator seam file-for-file.-- Gates: `make agent-check`, `make tb` (boot/config sanity — the new registry must build at boot), `make agent-test`.--### Phase V1 — Temporal validator (`pipelex-temporal`, branch off `refactor/own-temporal-config` @ `6d9adff`)--- Implement `TemporalBundleValidator`; register it unconditionally under both temporal modes. Reuse `dispatch_dry_validate` / `DryValidateResult` as-is.-- Add an in-memory-Temporal integration test (mirror `test_dry_validate_activity_in_memory.py`): the validator returns an assembled `PipelexValidationReport` for a valid bundle and an `ErrorReport` (with `validation_errors`) for an invalid one; a genuine fault re-raises.-- Gates: `make agent-check`, `make agent-test`.--> **Checkpoint V-A** (MAJOR boundary, natural session break): core + `pipelex-temporal` green together; the seam exists and the Temporal plugin contributes through it, verified against the in-memory Temporal env. The base still has no idea Temporal exists. Fan out a clean-context `/code-review` on the core seam diff + the `pipelex-temporal` diff.--### Phase V2 — API dispatch + route + F2 reversal (`pipelex-api`, branch off `refactor/orchestrator-agnostic-base` @ `a39841e`)--- Add `ApiRunner.validate_verdict`, rewire the route to verdict-as-value, thread `execution_mode` into the validate request, keep `ValidReport`/`InvalidReport`/`_invalid_report_response` unchanged.-- Reverse the F2 prose: the docstring at `pipeline.py:73` and the route comment at `190-193`; restore the dual-backend description in `pipelex-api/docs/pipe-validate.md` (the runner-sizing/library-load note becomes flavor-conditional, not absolute).-- Gates: `make agent-check`, `make agent-test`. **Keep `pipelex-temporal` out of `pipelex-api`'s deps** — test the dispatch+route+verdict-mapping with a **stub** `BundleValidatorProtocol` registered for a test mode (no temporal import). The real Temporal arm is covered in V1.--### Phase V3 — bookkeeping + spec/conformance--- Update the parent plan's **F2** decision (mark it reversed/extended; point to this doc) in `_plugins/wip/plugins/orchestrator-agnostic-runner-and-flavors.md`, and the Phase C as-built note in `_plugins/TODOS.md`. Add a `CHANGELOG.md` entry (dual-backend `/validate`).-- `docs/specs/`: the verdict contract does **not** change. Optionally add one clarifying sentence that the verdict is backend-independent (in-process **or** orchestrator-dispatched) to `pipelex-mthds-protocol.md` "Validation status codes" (`130-151`) and/or `command-surface-map.md` row 6 — **only if** added with a `> Verified by:` link + a matching `pytest.mark.spec(...)`. If you touch any spec heading, run `make check-spec-links` (in `conformance/`).-- `conformance/`: the existing `/validate` tests (`tests/pipelex_api/test_validate_{valid,diagnostic,render}.py`, `tests/pipelex_agent/test_validate_{envelope,errors}.py`, the `validate-error-qa/` corpus) must pass **unchanged** against the in-process (`direct`) arm. A conformance test of the **dispatched** arm needs Temporal in the conformance env — **defer** it (note the gap explicitly, like the other deferred HTTP arms; the in-memory coverage in V1 is the safety net meanwhile).--> **Checkpoint V-B** (THE gate for validate): `pipelex-api` dispatches validate by mode, the route returns verdict-as-value, F2 prose reversed, all gates green in all three repos, existing `/validate` conformance green unchanged, `make check-spec-links` green if specs touched. Clean-context `/code-review` on the `pipelex-api` diff. Capture an as-built (final names, signatures, divergences, test evidence).--### Phase V-Mistral — DEFERRED (decision C)--- Add a `MistralBundleValidator` later (genuine dispatch mirroring Temporal **or** an in-process passthrough — decide then). Gated on `pipelex-mistralai-workflows`' `mistralai-2.x` work. The generic seam means this lands without touching core or the API.-------## 6. Cross-repo state, pins, gates--**Branches / tips (verified 2026-06-21):**--- `_plugins` (pipelex core worktree): `refactor/Plugins-4` (this doc is the only dirty file — it overwrites the prior brief).-- `pipelex-api`: `refactor/orchestrator-agnostic-base` @ `a39841e` (clean).-- `pipelex-temporal`: `refactor/own-temporal-config` @ `6d9adff` (clean). **Private** (`LicenseRef-Proprietary` + `Private :: Do Not Upload`) — install via `git+ssh`, never PyPI.-- `pipelex-mistralai-workflows`: `feature/Mistral-native` (deferred — ignore).--**Editable pins (local dev):** `pipelex-api/pyproject.toml:84-85` and `pipelex-temporal/pyproject.toml` both pin `pipelex = { path = "../_plugins", editable = true }`. So a core (`_plugins`) edit is live in both immediately. **`pipelex-api` does not depend on `pipelex-temporal`** (Phase C dropped it) — for any *manual* end-to-end check of the Temporal arm in the API, install `pipelex-temporal` as an editable **dev-only** extra; never add it to the base runtime deps.--**Gates:** core `_plugins` → `make agent-check` / `make tb` / `make agent-test`. `pipelex-api` and `pipelex-temporal` → `make agent-check` / `make agent-test`. `conformance` → `make check-spec-links` (+ its pytest suite for the `/validate` arm).-------## 7. Testing--- **Core (V0):** unit-test the registry + `DirectBundleValidator` (valid → report; invalid bundle → `ErrorReport` with `validation_errors`; missing mode → `MissingBundleValidatorError`). `make tb` must stay green (boot builds the new registry).-- **Temporal (V1):** in-memory-Temporal integration test of `TemporalBundleValidator` (valid → assembled report; invalid → `ErrorReport`; genuine fault → re-raise). Mirror the existing in-memory validate tests.-- **API (V2):** dispatch+route+verdict-mapping tested against a **stub** validator (no temporal). Assert: `direct` → in-process verdict; a stubbed temporal-mode validator's returned `ErrorReport` → 200 `InvalidReport`; a stubbed raised fault → 5xx; missing-mode → `MissingBundleValidatorError`.-- **Conformance (V3):** existing `/validate` suite green **unchanged** (byte-identical verdicts) = the acceptance bar. Dispatched-arm conformance deferred (note the gap).-------## 8. Open sub-decisions for the implementing session--- **Final names/locations** for the seam (`BundleValidator*`, file homes). Proposed above; confirm against the orchestrator-seam layout.-- **Decision B sanity check:** if in-plugin assembly (Temporal arm calls `build_validation_report`) proves more tangled than expected, the fallback is in-route assembly (route gets `DryValidateResult` and assembles) — but that would re-expose worker payloads to the API, so prefer in-plugin. Flag if you hit friction.-- **Protocol `validate` fate:** keep as a raise-based conformance wrapper delegating to `validate_verdict`, or simplify if the route is its only caller. Confirm callers first.-- **`execution_mode` on the validate request:** confirm the request model/extras plumbing matches `/start`'s, and that the override policy (403 on forbidden override) applies identically.-------## 9. Progress / as-built (live)--> Appended at each checkpoint. Final names, divergences, test evidence — enough to cold-resume.--### Phase V0 — core seam — DONE (uncommitted at Checkpoint V-A)--Mirrors the orchestrator seam file-for-file. Final names/locations:--- `pipelex/plugins/bundle_validator_registry.py` — `BundleValidatorProtocol` (`@runtime_checkable`, single async `validate_bundles(*, mthds_contents, mthds_sources, allow_signatures, library_dirs)`), `BundleValidatorRegistry` (`get_optional` / `has` / `modes`), and the `BundleValidationVerdict` type alias.-- `pipelex/pipeline/direct_bundle_validator.py` — `DirectBundleValidator` (calls `validate_bundles_in_process(..., log_context="API validate")`; catches `ValidateBundleError` → returns `exc.to_error_report()`; other exceptions propagate).-- `pipelex/plugins/exceptions.py` — `DuplicateBundleValidatorError`. `pipelex/plugins/registrar.py` — `bundle_validators` store + `_bundle_validator_sources` + `add_bundle_validator` (reuses the `_add` dup-guard). `pipelex/plugins/direct/direct_plugin.py` — registers `DirectBundleValidator` under DIRECT alongside `DirectOrchestrator`.-- `pipelex/runtime_bridge/exceptions.py` — `MissingBundleValidatorError(mode=...)` (per-mode install hint, mirrors `MissingOrchestratorError`). `pipelex/hub.py` — field + setter + getter + module `get_bundle_validator_registry()`. `pipelex/pipelex.py` — boot wiring `set_bundle_validator_registry(BundleValidatorRegistry(plugin_registrar.bundle_validators))` next to the orchestrator registry.--**Divergence from §4.1 (the only one, forced by `reportImportCycles = true`):** the verdict's valid arm is typed at the **MTHDS-protocol base `ValidationReport`** (a leaf type), NOT the concrete `PipelexValidationReport`. Naming the concrete envelope from the hub-reachable registry closes a real import cycle (`hub → bundle_validator_registry → validation_report → bundle_validator → pipe_run → hub`) — pyright follows `TYPE_CHECKING` edges, so the cycle gates. This mirrors how the orchestrator seam returns the leaf `PipelexPipeRunOutput`, and is brand-correct (the generic seam speaks the protocol report; the Pipelex envelope is recovered at the API edge). `DirectBundleValidator` still returns the precise `PipelexValidationReport | ErrorReport` (a covariant narrowing); the API route recovers the precise valid arm for its `isinstance(verdict, PipelexValidationReport)` narrowing. Acceptance note for V2: `ApiRunner.validate_verdict` will narrow/cast the protocol's `ValidationReport` valid arm back to `PipelexValidationReport` (single documented point).--Tests: `tests/unit/pipelex/plugins/test_bundle_validator_registry.py` (registrar+registry+dup), `tests/unit/pipelex/pipeline/test_direct_bundle_validator.py` (verdict-as-value branches, mocked sweep), `tests/unit/pipelex/runtime_bridge/test_missing_bundle_validator_error.py` (per-mode hints). Gates: `make agent-check` green, `make tb` green, `make agent-test` green.--### Phase V1 — Temporal validator — DONE (uncommitted at Checkpoint V-A)--- `pipelex-temporal/pipelex_temporal/temporal_bundle_validator.py` — `TemporalBundleValidator` (import-light; `temporalio` + dispatch/wire/exception types resolved lazily inside `validate_bundles`, mirroring the orchestrators). Dispatches `dispatch_dry_validate(DryValidateArg(...))`; on success parses blueprints cheaply (`PipelexInterpreter.make_pipelex_bundle_blueprint`, threading `mthds_sources`) → `build_validation_report(...)`; on `WorkflowExecutionError` recovers the report and **returns** it iff `error_type == ValidateBundleError.__name__` (invalid verdict), else **re-raises** (genuine fault → 5xx). `library_dirs` ignored (worker loads its own library). Copied verbatim from the old dual-path `ApiRunner.validate` Temporal arm, but as a returned verdict (decision A).-- `temporal_plugin.py` — registers the SAME instance under **both** `TEMPORAL_BLOCKING` and `TEMPORAL_FIRE_AND_FORGET` (decision D), unconditionally, claims no hub slot.--Tests: `tests/integration/pipelex_temporal/test_temporal_bundle_validator_in_memory.py` (real worker, in-memory server: valid → assembled `PipelexValidationReport`; invalid → `ErrorReport` verdict; patches `is_temporal_boot_active` + `get_temporal_client` to route the validator's own `dispatch_dry_validate` through the in-memory env), `tests/unit/pipelex_temporal/test_temporal_bundle_validator.py` (verdict-vs-fault branching, mocked dispatch). Gates: `make agent-check` green, `make agent-test` green.--**Deferred observation:** dispatched `/validate` effectively requires `boot_orchestrator == "temporal"` (the reused executor's `is_temporal_boot_active()` guard), slightly at odds with the plan's decision #5 "not a runner booted-as-Temporal" — but identical to the run orchestrators' behavior, pre-existing, out of scope. Captured in [`validate-dispatch-requires-boot-active.md`](validate-dispatch-requires-boot-active.md).--> **Checkpoint V-A closed.** Core green + `pipelex-temporal` green together; the seam exists and the Temporal plugin contributes through it (verified against the in-memory Temporal env). Clean-context `/code-review` on both diffs: **zero findings, no fixes** (core: 3 independent passes; temporal: each flagged path verified against ground truth). One latent observation deferred ([`validate-verdict-subclass-parity-divergence.md`](validate-verdict-subclass-parity-divergence.md) — the `error_type` string-equality predicate, copied verbatim from the old route, would diverge from the direct arm's `except` only if a `ValidateBundleError` subclass is ever introduced; none exists). Commits: core `de709eea4`, `pipelex-temporal` `459e04d` (branch `feature/Orchestrator-dispatched-validate` in each).--### Phase V2 — API dispatch + route + F2 reversal — DONE (`pipelex-api`)--- `api/routes/pipelex/pipeline.py` — **dropped the `ApiRunner.validate` `@override`** (its only caller was the route; the parity test uses the base `PipelexMTHDSProtocol.validate`) and added **`ApiRunner.validate_verdict(*, mthds_contents, mthds_sources, allow_signatures, requested_execution_mode) -> PipelexValidationReport | ErrorReport`**. Mirrors `start`: `resolve_execution_mode(...)` FIRST (forbidden override → 403 before any dispatch) → `get_bundle_validator_registry().get_optional(mode=…)` (→ `MissingBundleValidatorError` if absent) → `validator.validate_bundles(..., library_dirs=…)`. The seam's valid arm is typed at the protocol-level `ValidationReport`, so `validate_verdict` recovers the concrete `PipelexValidationReport` with a single documented `cast` — the only narrowing point.-- `api/routes/pipelex/validate.py` — route rewired to **verdict-as-value**: `verdict = await ApiRunner().validate_verdict(...)`; `if not isinstance(verdict, PipelexValidationReport): return _invalid_report_response(verdict, ...)` else build `ValidReport`. Removed the `try/except ValidateBundleError`. Added `execution_mode: PipelexExecutionMode | None` to `ValidateRequest` (same plumbing as `/start`). `ValidReport`/`InvalidReport`/`_invalid_report_response` unchanged.-- **F2 prose reversed:** the `ApiRunner` class docstring, the `validate_verdict` docstring, the route docstring, and `docs/pipe-validate.md` ("Where validation runs" + the resource note + the 403 in the no-verdict list) now describe `execution_mode`-aware dual-backend validation (flavor-conditional library load) instead of "always DIRECT in-process."--**Sub-decision resolved (Protocol `validate` fate):** dropped, not kept as a raise-based wrapper. The route is the only `ApiRunner.validate` caller and switches to `validate_verdict`; re-raising from the returned `ErrorReport` would fabricate an exception no consumer catches (the original `ValidateBundleError` was already converted to a report by the validator). `ApiRunner` inherits the base `PipelexMTHDSProtocol.validate` (in-process — a sensible agnostic-base fallback, unused in practice).--Tests: `tests/unit/test_validate_dispatch.py` (NEW — stub validator for a non-direct mode proves registry dispatch + verdict mapping; invalid `ErrorReport` → 200, raised fault → 5xx, `MissingBundleValidatorError`, forbidden override → 403; no `temporalio` import). Three tests in `test_validate_errors.py` that mocked the removed `ApiRunner.validate` were re-pointed at `validate_verdict` (verdict-as-value: invalid arms now **return** the `ErrorReport`, the fault arm still **raises**) — the wire output is byte-identical. All other `/validate` tests (envelope, render, parity, conformance, allow_signatures) pass **unchanged** against the real direct arm. Gates: `make agent-check` green, `make agent-test` green (full suite).--### Phase V3 — bookkeeping + spec/conformance — DONE--- Parent plan F2 marked **REVERSED / EXTENDED** in [`orchestrator-agnostic-runner-and-flavors.md`](orchestrator-agnostic-runner-and-flavors.md) and in `_plugins/TODOS.md` (locked-decisions F2 + Phase C as-built). `pipelex-api` `CHANGELOG.md` `[Unreleased]`: the F2 "always DIRECT" bullet was rewritten in place (F2 never shipped) to the dual-backend verdict-as-value `/validate`; the `execution_mode` config bullet now names `/v1/validate` alongside `/v1/start`.-- **Spec / conformance:** no spec heading touched — the verdict contract (200-always, `is_valid` discriminant, structured `validation_errors[]`) is **unchanged**, and no spec or conformance test asserts the backend, so `make check-spec-links` is not needed. The direct-arm verdict wire is proven byte-identical by the `pipelex-api` suite (`test_validate_*`, `test_protocol_parity`, `test_protocol_conformance`, all green). The separate `conformance/` repo's live-wire `/validate` suite (`test_validate_{valid,diagnostic,render}.py`) is therefore expected green unchanged; a conformance test of the **dispatched** arm needs Temporal in the conformance env and is **deferred** (the V1 in-memory coverage is the safety net), exactly as the other deferred HTTP arms.--> **Checkpoint V-B — THE gate for validate.** `pipelex-api` dispatches validate by mode, the route returns verdict-as-value, F2 prose reversed across code + docs + trackers, all gates green in all three repos, existing `/validate` conformance green unchanged. Clean-context `/code-review` on the `pipelex-api` diff pending at this checkpoint.diff --git a/wip/plugins/orchestrators-as-plugins.md b/wip/plugins/orchestrators-as-plugins.mddeleted file mode 100644index 226abb881..000000000--- a/wip/plugins/orchestrators-as-plugins.md+++ /dev/null@@ -1,106 +0,0 @@-# Orchestrators as Pipelex plugins--**Status:** assessment / design notes (not started)-**Scope:** turn the *orchestrators* — the strategies that decide **where and how** a pipe runs (in-process, on a Temporal worker fleet, on Mistral Workflows) — into discoverable plugins, so core ships only the in-process orchestrator and every distributed one is an optional, separately-installable plugin wired through one seam.--> An **orchestrator** decides how a whole pipe job is scheduled and where it executes — distinct from a *pipe controller*, which sequences sub-pipes *within* a single run. (We deliberately avoid "backend" here: that word already names the inference SDK adapters in the [driver doc](inference-backends-as-plugins.md).) This is the **strategy** half of the plugin story: orchestrators *replace core orchestration behavior*. The **driver** half — inference SDK wrappers selected by a model's `sdk` handle — is in [`inference-backends-as-plugins.md`](inference-backends-as-plugins.md). Temporal's concrete seam-by-seam extraction lives in [`temporal-as-plugin.md`](temporal-as-plugin.md), which references this doc for the shared mechanism. Cross-cutting best practices are in [`README.md`](README.md).--## The category--An orchestrator answers *"where/how does this pipe run?"* — not *"which model/SDK?"* (that's a driver). Pipelex already models the choice as `PipelexExecutionMode` in `runtime_bridge/execution_mode.py`:--- **`DIRECT`** — in-process async. The trivial orchestrator. **Ships in core.**-- **`TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`** — distributed via Temporal. Lives in `pipelex/temporal/` today; destined for `pipelex-temporal`.-- **`MISTRAL_NATIVE`** — distributed via Mistral Workflows. Lives in the external `pipelex-mistralai-workflows` repo.--`DIRECT` is the only one that belongs in core. The other two are **peers**: optional, discovered, neither privileged. Today core fails that test — it names *both* external orchestrators by string (see the seam below). Mistral is the proof that "orchestrator" is a *category with ≥2 members*, so the seam must be generic and N-ary, not "Temporal plus a special case."--## Two worked instances--| | `DIRECT` (core) | `pipelex-temporal` | `pipelex-mistralai-workflows` |-|---|---|---|---|-| **Maps a pipe onto** | in-process async calls | Temporal workflows + activities | Mistral Workflows workflows + activities |-| **Pipelex contracts implemented** | core default | `PipeRouterProtocol`, `PipeRunProtocol`, `ContentGeneratorProtocol`, task manager | `PipeRunProtocol`, `PipeRouterProtocol` |-| **How core couples to it *today*** | n/a | lazy imports gated on `get_config().temporal.is_enabled` (hub injection in `pipelex.py`) + `bridge._run_temporal_*` + **hard** CLI imports in `_cli.py` | **hard-coded import** in `bridge._run_mistral_native` of `pipelex_mistralai_workflows.primitives.pipe_run` + a `requires_mistral_workflows_extra` gate |-| **Activation model** | default | **boot-global** (hub swap when enabled) **+ per-call** modes | **per-call** (`execution_mode`); installs its router *inside* the workflow at runtime |-| **Also a plugin for…** | — | its host: register workflows/activities with a Temporal worker | its host: register workflows/activities with a Mistral worker (`register_pipelex_primitives`) |--Two different couplings, same fix: invert them through one registry.--## The seam: an execution-mode → orchestrator registry--`bridge.py` currently does a `match input_payload.execution_mode:` whose arms hard-code both external orchestrators — the `TEMPORAL_*` arms lazily import `pipelex.temporal`, the `MISTRAL_NATIVE` arm lazily imports `pipelex_mistralai_workflows`. That is **two string couplings** to remove.--Replace the `match` with a registry keyed by execution mode. Core registers `DIRECT`; each orchestrator plugin registers its mode(s) through the shared `pipelex.plugins` entry point; the bridge collapses to a lookup:--```python-orchestrator = orchestrator_registry.get(input_payload.execution_mode)-if orchestrator is None:- raise MissingOrchestratorError(- mode=input_payload.execution_mode,- hint="install pipelex-temporal / pipelex-mistralai-workflows",- )-return await orchestrator.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)-```--**Keep the `PipelexExecutionMode` enum in core.** It is a small, stable *taxonomy of strategies*; core can know the *names* without the *implementations*. That is what lets the error above name the missing package, and keeps the `requires_pipelex_temporal` / `requires_mistral_workflows_extra` gates statically typed. Plugins supply *orchestrators* for existing enum values — they do not invent modes. (Third-party-defined modes would mean an open, string-keyed mode space; that's a larger change, defer it until something actually needs it.)--## Two activation models → the `Plugin` protocol is a menu of hooks--The orchestrators do not use the seam the same way, and the protocol must not be modelled on Temporal alone:--- **Per-call dispatch (primary).** The bridge picks an orchestrator by `execution_mode` for each run. Both Temporal modes *and* Mistral use this. Hook: **`register_orchestrators(registry)`**.-- **Boot-global swap (Temporal only).** When `temporal.is_enabled`, the default router / run / content-generator / task-manager are swapped at boot via hub injection — necessary because *inside a Temporal worker* the content generator must dispatch each operation as an activity. Hook: **`register_hub_implementations(hub, config)`** + lifecycle.-- **Mistral needs no boot swap.** It installs `MistralWorkflowsPipeRouter` *inside* the child workflow at runtime, per call — so it uses `register_orchestrators` and essentially nothing at boot. It is the counter-example that keeps the protocol honest: hooks are a **menu**, each plugin implements the subset it needs.-- **CLI is optional.** Temporal contributes `worker` + `setup-temporal-namespace`; Mistral contributes none (the worker is the user's Mistral worker). Hook: **`register_cli_commands(app)`**.--## The orchestrator SPI (published surface)--A clean plugin must depend only on a *designed* surface, never reach into internals. Where the two orchestrators sit today:--- **Temporal** depends on hub setters + the three protocols + its config field — mostly already protocol-shaped.-- **Mistral** reaches deeper, and says so: its own `wip/boundary-violation-mistral-native.md` documents that the native tier imports `pipelex.hub`, `pipelex.pipe_run.*`, `pipelex.core.*`, `pipelex.graph.*`, `pipelex.tracing.*` — well outside the declared `runtime_bridge.*`-only surface.--That is a **signal, not a sin**: running a pipe out-of-process genuinely needs library/crate access (`hub`), the `PipeJob` / `PipeOutput` types (`core`), the router/run protocols (`pipe_run`), and the trace/tracer hooks (`graph` / `tracing`). The fix is to **publish that set as an orchestrator SPI**, not to keep an honor-system "runtime_bridge only" rule that's already breached. Define the SPI as:--- the orchestrator registry + the `PipeRouterProtocol` / `PipeRunProtocol` / `ContentGeneratorProtocol`;-- `PipeJob` / `PipeOutput` and the boundary payload types;-- library-crate access (for per-call library hydration — Mistral already does this via `library_crate_dump`);-- the tracing/graph hooks an orchestrator must integrate with.--Anything an orchestrator imports *outside* the SPI is a design bug to resolve — promote it into the SPI, or remove the need. (This is the same theme as the OpenAI *substrate* becoming a deliberate contract in the [driver doc](inference-backends-as-plugins.md): once an out-of-tree consumer depends on you, the dependency surface must be designed, not accidental.)--## One repo, two hosts (the adapter shape)--`pipelex-mistralai-workflows` depends on **both** `pipelex` and `mistralai-workflows`. It is a bridge/adapter: a Pipelex orchestrator plugin on one side, a Mistral Workflows activity/workflow library on the other. The two "plugin" directions use **different mechanisms**, and conflating them causes confusion:--- **→ Pipelex: entry-point discovery.** Declare `[project.entry-points."pipelex.plugins"]`; installing the wheel makes `MISTRAL_NATIVE` available and core stops hard-coding the import.-- **→ Mistral: registration-by-import.** Mistral has no plugin registry; the user (or the package's `register_pipelex_primitives(workflows, activities)` helper) hands the workflow/activity classes to their worker. No entry point — and nothing to change. (Temporal is identical on *its* host: you always register workflows/activities with a worker explicitly.)--```toml-[project]-name = "pipelex-mistralai-workflows"-dependencies = ["pipelex>=X", "mistralai-workflows>=3.4.0"] # depends on BOTH hosts--# Pipelex side — discovery-[project.entry-points."pipelex.plugins"]-mistral-native = "pipelex_mistralai_workflows.plugin:MistralWorkflowsOrchestrator"--# Mistral side — NO entry point: users call register_pipelex_primitives(...) on their worker.-```--This is the natural shape for an adapter — **don't split it.** The host-agnostic decomposition plumbing (pipe classification) already lives in core under `runtime_bridge/primitives/`; everything left is inherently bi-lateral.--## Effort & risk--- **The per-call dispatch inversion is small** — one `match` → a registry lookup; both external orchestrators are already behind lazy/dynamic imports.-- **The SPI definition is the real design work** — settle what core publishes and resolve Mistral's boundary violation against it. This gates a *clean* extraction (the mechanical move is easy; the contract is the part worth getting right).-- **Temporal's boot-global hub-injection inversion is the heavier mechanical piece** — detailed seam-by-seam in [`temporal-as-plugin.md`](temporal-as-plugin.md).-- **Watch:** registration timing — `DIRECT` must always be registered, and a requested-but-absent orchestrator must produce the "install X" error, not a `None` deref. Keep the enum + `requires_*` gates as the typed source of truth for *which modes exist*.--## Suggested phasing--- **Phase 0 — define the seam.** `Plugin` protocol + entry-point discovery + the execution-mode → orchestrator registry + the published orchestrator SPI. Register `DIRECT` in core through it and collapse the bridge `match` to a registry lookup (external orchestrators still resolved as today, now *behind* the registry — no behaviour change). _Checkpoint: the bridge no longer matches on specific external modes; `DIRECT` flows through the registry; suite green._-- **Phase 1 (Temporal) — through the seam.** See [`temporal-as-plugin.md`](temporal-as-plugin.md) Phase 1 (register hub implementations + orchestrators + CLI; resolve the config cut line).-- **Phase 1′ (Mistral) — through the seam.** Replace `bridge._run_mistral_native`'s hard import with the orchestrator the `pipelex-mistralai-workflows` entry point registers; resolve its boundary violation against the published SPI. _Checkpoint: core has zero string references to `pipelex_mistralai_workflows`; `MISTRAL_NATIVE` works purely via discovery._-- **Phase 2 — repackage as needed.** Temporal: extract `pipelex/temporal/` → `pipelex-temporal` (its doc). Mistral: already external — just flip the dev path-pin to a published `pipelex` once the SPI lands.diff --git a/wip/plugins/phase-1-review-deferred.md b/wip/plugins/phase-1-review-deferred.mddeleted file mode 100644index acd779f45..000000000--- a/wip/plugins/phase-1-review-deferred.md+++ /dev/null@@ -1,6 +0,0 @@-# Phase 1 plugin-system review — deferred items--Items surfaced during the `/code-review` pass on the Phase 1 staged diff that are design tradeoffs (not bugs) and were intentionally NOT changed. Left here for later triage.--- `llm_worker_factory.make_llm_worker` computes `ModelHandle.make_for_inference_model(...)` to do the registry lookup by `model_handle.sdk`, and each vendor `_make_*_worker` closure recomputes it internally. The pre-migration `match` arms computed it once. The recompute is a trivial 3-field pydantic construction, off any hot loop (one per worker creation), so the cost is negligible — but if a later phase threads `model_handle` through the `MakeWorkerFn` signature for other reasons, this redundancy disappears for free. Not worth a seam-signature change on its own.-- `PluginDiscovery.targets_api` is recorded from `plugin.targets_api` on the disabled-plugin path (discovery.py) but from the literal `PLUGIN_API_VERSION` on the registered path (via `begin_plugin`). They are equal for any registered plugin (the version-mismatch check gates registration), so this is a stylistic asymmetry with no behavioral effect. Could be unified to always record `plugin.targets_api` if a future API allows a registered plugin to declare a value other than the supported one.diff --git a/wip/plugins/phase-2-review-deferred.md b/wip/plugins/phase-2-review-deferred.mddeleted file mode 100644index 2c3f645a1..000000000--- a/wip/plugins/phase-2-review-deferred.md+++ /dev/null@@ -1,29 +0,0 @@-# Phase 2 plugin-seam review — deferred items--Items surfaced by the xhigh `/code-review` pass over the current PR (the inference-backend plugin seam + all four migrated families: LLM/ImgGen/Extract/Search). These are latent footguns reachable only by external entry-point plugins or not-yet-written callers, plus quality cleanups — none produces wrong output on a live path today, so they were intentionally NOT changed and are left here for later triage.--The three CONFIRMED pre-merge gaps from the same pass were fixed in this PR (import-light blocklist completeness, structured lookup-miss error, multi-spec `require_sdk` naming); only the items below remain.--## Latent footguns — decide at the Phase 3 staging boundary whether to gate or document--- **`build_registrar` silently drops three of the five registrar collections.** `pipelex.py` `build_registrar` accumulates `cli_commands` / `slot_claims` / `teardown_callbacks`, but boot only consumes `inference_backends` and `orchestrators`. An external `pipelex.plugins` entry-point plugin that calls `add_cli_command` / `claim_pipe_run` / `add_teardown` registers with no error and shows up in `pipelex plugins list`, yet its command never appears, its hub slot is never wired, and **its teardown never runs at `Pipelex.teardown()`** — any SDK client / external resource it opened leaks for the process lifetime. The discovery + iteration path is fully live, so this is reachable today by a third-party plugin; only the in-tree builtins (which use none of those three menus yet) keep it latent. At minimum, boot should warn on an unconsumed contribution rather than silently ignore it; better, wire the three collections as the orchestrator seam lands in Phase 3.--- **`make_search_worker` can silently drop usage reporting.** `cogt/search/search_worker_factory.py` moved `reporting_delegate` from an unconditional in-factory `get_report_delegate()` fetch to a parameter that **defaults to `None`**. The single production caller (`search_generate.py`) passes it, but any future caller (a Temporal search-activity path, a host runtime, a test) that omits it gets a search worker emitting no usage/cost report — silent zero where the old code always reported. The other three families route their delegate the same way, but search is the one whose factory previously self-fetched, so it is the regression-prone one. Consider making the param non-defaulting (force the caller to decide) or fall back to `get_report_delegate()` when `None`.--- **A callable plugin *instance* is mistaken for a factory.** `plugins/discovery.py` `_load_external_plugin` uses `loaded() if callable(loaded) else loaded`, which will **call** a plugin object whose class defines `__call__` instead of using it as the plugin. If that `__call__` returns `None` or raises, a valid installed plugin is rejected as `BrokenPluginError` with a confusing "failed to load entry point" message. Tighten the factory-vs-instance discrimination (e.g. accept only a class or a plain function as a factory, not any `callable`). Only reachable via an external entry point, so latent until the discovery path has real out-of-tree consumers.--- **The coverage test leans on an unenforced contract.** `tests/unit/pipelex/plugins/test_inference_backend_coverage.py` constructs `PluginRegistrar(config=cast('PipelexConfig', None))` and calls each builtin's `register()`, relying on the invariant that `register()` never reads `self.config`. If a future builtin reads `registrar.config` (a plausible mistake — `config` is an `__init__` param), the test crashes with `AttributeError: 'NoneType'` on a path unrelated to the change under test, masking the real registration logic behind a confusing CI failure instead of a clear contract violation. Either pass a minimal real/stub config, or make the contract explicit (assert/typed null-object) so a violation fails loud and on-point.--## Quality cleanups — fold into the Phase 3 seam threading--- **Redundant `ModelHandle` double-construction.** Already captured in [`phase-1-review-deferred.md`](phase-1-review-deferred.md) (item 1): each family factory builds a `ModelHandle` only to read `.sdk` for the lookup, then every `_make_*_worker` closure rebuilds the identical handle from the same `inference_model`. Disappears for free if a later phase threads `model_handle` through the `MakeWorkerFn` signature; not worth a seam-signature change on its own. Cross-referenced here because the xhigh pass re-surfaced it across all four families and ~two dozen closures.--- **Copy-pasted worker-builder preamble.** The `model_handle = ModelHandle.make_for_inference_model(...)` + `sdk_clients.get_or_create(handle=..., build=lambda: ...)` dance repeats across ~two dozen `_make_*_worker` closures (`google_plugin.py`, `bedrock_plugin.py`, the openai/gateway/portkey families, …), varying only in the build lambda and the worker constructed. A shared helper — `build_cached_client(inference_model, *, sdk_clients, build) -> (model_handle, sdk_instance)` — would shrink each closure to its build lambda plus the worker call, and localize any future change to client caching / handle derivation / teardown registration. (Composes naturally with threading `model_handle` through `MakeWorkerFn` above.)--- **Duplicate img-gen worker builders on the OpenAI-completions substrate.** `_make_openrouter_img_gen_worker` (`openrouter_plugin.py`), `_make_blackboxai_img_gen_worker` (`blackboxai_plugin.py`), and the gateway-completions img-gen variant are the same closure — `OpenAIClientFactory.make_openai_client` → `get_or_create` → `OpenAICompletionsImgGenWorker(...)` — differing only in the completions-factory subclass and the `is_http_url_enabled` flag. The same shape recurs across the gateway/portkey/openai `_make_*_completions_worker` LLM closures. Parameterize on the factory class so a substrate fix lands in one place.--- **Search rebuilds its worker on every call.** `cogt/content_generation/search_generate.py` (`search_gen_sourced_answer` / `search_gen_structured`) constructs a fresh `SearchWorker` per search call — re-resolving model + backend, re-running the registry lookup, and (for the linkup arm) re-probing `importlib.util.find_spec` — where LLM/img_gen/extract return a worker cached in `InferenceManager`. The SDK client itself is cached in `SdkClientRegistry`, but the worker rebuild + resolution + `find_spec` repeat. The per-call `find_spec` on the linkup search arm is **new in this PR** (the migration routed search through `require_sdk`). Cache the search worker per handle the way the other three families do.--- **Dead `self.model_handle` field on `AzureImgGenWorker`.** `plugins/azure_rest/azure_img_gen_worker.py` `__init__` stores `self.model_handle` and reads it exactly once on the next line to re-resolve the backend via `get_models_manager()` — then never references it again. Leftover per-instance state from the `Plugin` → `ModelHandle` rename; the factory already resolved the backend and could hand it over directly, and the test fixtures had to set `worker.model_handle = MagicMock()` purely to satisfy construction. Drop the field and the redundant hub re-resolution; have the factory pass the `InferenceBackend` it already fetched.--- **Missing `-> None` return annotations.** `plugins/sdk_client_registry.py` `teardown` (and `setup` / `teardown` in `sdk_client_manager.py`) omit the `-> None` return annotation, against the "every function return must be typed" standard. Pyright in `make agent-check` does not flag missing return annotations by default, so it passes CI silently. Carried verbatim from the deleted `plugin_sdk_registry.py` / `plugin_manager.py` during the rename — a pre-existing untyped shape, not newly authored. Low cost, no runtime consequence; annotate when these files are next touched.diff --git a/wip/plugins/phase-3-critical-cli-harvest-fragility.md b/wip/plugins/phase-3-critical-cli-harvest-fragility.mddeleted file mode 100644index 95340cb13..000000000--- a/wip/plugins/phase-3-critical-cli-harvest-fragility.md+++ /dev/null@@ -1,104 +0,0 @@-# Phase 3 critical finding — the import-time CLI plugin-command harvest is an unguarded surface--> ✅ **RESOLVED BY REMOVAL (Option A).** This finding is moot: the CLI plugin-command harvest — and all four facets below — were **deleted**, not hardened. Temporal's `worker` / `setup-namespace` now ship as the standalone `pipelex-temporal` console script, so `pipelex --help` performs no config load, no entry-point scan, and no dynamic import; no plugin can shadow a core command or brick the recovery commands (`doctor` / `plugins` / `init`), because there is no longer any CLI-command contribution path. Decision + step-by-step: [`option-a-drop-cli-command-seam.md`](option-a-drop-cli-command-seam.md). Everything below is retained for historical context only.->-> **Status:** ~~open, unsolved~~ **RESOLVED.** **Purpose (historical):** a cold-start problem statement. This doc explains *what was wrong and why it mattered* in enough depth to start a fresh session with no prior context. **It deliberately stopped short of proposing a fix** — the solution space was explored separately (and the chosen answer was to remove the surface). Surfaced by the xhigh `/code-review` over the Phase 3 checkpoint commit (`19e6ca66b` / `5f61db323`).->-> Read the [TODOS cold-start primer](../../TODOS.md#cold-start-primer-read-this-first-if-youre-new-to-the-session) first for the plugin-seam vocabulary (registrar, `build_registrar`, `BUILTIN_PLUGINS`, slot-claims, D3) — this doc assumes it.--## TL;DR--Phase 3 (decision **D3**) harvests plugin-contributed CLI commands by running the full plugin-discovery pipeline (`build_registrar`) **at module-import time of `pipelex.cli._cli`**, on **every** `pipelex` invocation. That import path is now also doing config loading, external-plugin loading, and dynamic `import_module`/`getattr` — none of it guarded the way the boot path is, and one facet (command-name collision) has no guard *anywhere*. The net effect: a single broken/incompatible/colliding installed plugin, or an unreadable user config file, can make **every** `pipelex` command crash on import — including the very `--help` / `doctor` / `plugins` / `init` commands you would use to diagnose and disable the bad plugin. A fourth facet lets a plugin **silently shadow a core command** (`run`, `validate`, …) with no error or warning.--This is the single dominant correctness cluster of the Phase 3 review. It does not affect any in-tree-only install today (the built-ins contribute no CLI commands that collide, and a present-and-readable config parses), which is why the suite is green — but the surface is live for any external `pipelex.plugins` entry-point plugin, and that is exactly what Phase 5 ships.--## Where this lives--| Element | Location | What it does |-|---|---|---|-| The harvest entry point | `pipelex/cli/_cli.py` — `_register_discovered_cli_commands()` (~`:262`) and the module-level `_PLUGIN_COMMAND_NAMES = _register_discovered_cli_commands()` (~`:280`) | Runs at import of `_cli.py`. The `pipelex` console-script entry point is `pipelex.cli._cli:app`, so importing `app` runs this. |-| The config load for the harvest | `pipelex/cli/_cli.py` — `_config_for_cli_harvest()` (~`:246`) | Loads config without side effects; `except (TomlError, ValidationError)` → falls back to package defaults. |-| The discovery pipeline it calls | `pipelex/plugins/discovery.py` — `build_registrar()` (`:24`) | Iterates `BUILTIN_PLUGINS` then external `pipelex.plugins` entry points; **fail-loud** on every conflict (`BrokenPluginError`, `PluginApiVersionMismatchError`, `DuplicateInferenceBackendError`, `DuplicateOrchestratorError`, `HubSlotAlreadyClaimedError`, `CoreUnconditionalPluginDisabledError`). |-| The dynamic command resolve | `pipelex/cli/_cli.py` (~`:272-276`) | `module_path, _, attribute = import_path.partition(":")` → `getattr(importlib.import_module(module_path), attribute)` → `app.command(name=..., help=...)(command)`. |-| The registrar menu method with no guard | `pipelex/plugins/registrar.py` — `add_cli_command()` (`:135`) | Appends a `CliCommand` with **no** duplicate/collision check — unlike `add_orchestrator` (`:116`), `_claim` (`:145`), and the inference-backend add (`:110`), which all raise and name both contributors. |-| The command list that surfaces them | `pipelex/cli/_cli.py` — `PipelexCLI.list_commands` (~`:54`) returns `[*_CORE_COMMAND_ORDER, *_PLUGIN_COMMAND_NAMES]` | Plugin names are appended after core names. |--For contrast, the **boot** path runs the same `build_registrar` at `pipelex/pipelex.py` `setup()` (~`:390`) — but boot is allowed to fail loud; CLI startup is not.--## The contract this is supposed to honor--`_config_for_cli_harvest`'s own docstring states the intended invariant in plain words:--> "Runs on every `pipelex` invocation (including `--help` / `init`), so it must never create `~/.pipelex/` … and **must survive a broken user config** — a malformed override must not brick the very commands (`init`/`doctor`) that fix it. On any load or validation failure it falls back to the shipped package defaults, which always validate."--`build_registrar`'s docstring states the *other* intended invariant, which is in direct tension with the first:--> "is **fail-loud** on every conflict (duplicate backend/mode/slot, version mismatch, broken plugin)."--The Phase-1 boot-wiring note and the "Invariants that must survive every phase" section of TODOS both make **fail-loud** a deliberate, load-bearing property: *"installed but broken" must be loud, distinct from "not installed" (fine).* The problem is that Phase 3 moved that fail-loud pipeline onto the one path — CLI module import — where failing loud means failing *everything*, recovery commands included. The two invariants were never reconciled at this altitude.--## The problem, in four facets--All four are reachable through one surface: `_cli.py` import now performs config-load + plugin-discovery + dynamic import, none of it degraded for the "this must never brick `--help`" context.--### Facet 1 — `build_registrar()` is not wrapped; a broken external plugin bricks the whole CLI--`_register_discovered_cli_commands()` calls `build_registrar(config=...)` with no surrounding `try`/`except`. `build_registrar` is intentionally fail-loud: it raises on a broken entry point (`BrokenPluginError`), an API-version mismatch (`PluginApiVersionMismatchError`), and any duplicate `(family, sdk)` / duplicate mode / double-claimed slot across plugins. Pre-Phase-3, this pipeline ran **only inside `Pipelex.setup()`**, so a discovery failure could not touch `pipelex --help`.--**Failure scenario.** A user `pip install`s a broken / API-incompatible external pipelex plugin (or two plugins that collide on a mode/backend/slot). The exception now fires during `import pipelex.cli._cli`, before any command function runs. `pipelex --help`, `pipelex doctor`, `pipelex plugins`, and `pipelex init` all crash identically. The diagnostic commands (`doctor` / `plugins`) and the `plugins.disabled` denylist edit reachable via `init` — the documented recovery path for a broken installed plugin — are themselves unreachable. The CLI is fully bricked until the user manually `pip uninstall`s the offending dist by hand.--### Facet 2 — the config-load `except` is too narrow; an unreadable config file bricks the CLI--`_config_for_cli_harvest()` catches only `(TomlError, ValidationError)`. But the loader opens user TOML with `open(path, "rb")` (`pipelex/tools/misc/toml_utils.py:39`, `load_toml_from_path`), which catches only `tomli.TOMLDecodeError`. A file that is **present but unreadable** (wrong permissions, root-owned, on a flaky network mount) raises `PermissionError` / `OSError` from `open()` — neither is a `TomlError` nor a `ValidationError`, so it escapes the harvest's `except` uncaught and aborts the import.--**Failure scenario.** A user whose `~/.pipelex/pipelex.toml` or project `.pipelex/pipelex.toml` is present-but-unreadable gets every `pipelex` command — `--help`, `init`, `doctor` included — to abort at CLI module import with an uncaught `OSError`, instead of falling back to package defaults as the "bulletproof fallback" docstring promises. Again, the commands meant to fix config are unreachable.--> Note: the CHECKPOINT-3 review already added `test_harvest_config_falls_back_to_base_on_broken_user_config` (S1), but that test pins only the **malformed/parse** fallback (`TomlError`/`ValidationError`). The **unreadable/`OSError`** path is a distinct, still-uncovered gap — the existing test does not exercise it.--### Facet 3 — a malformed `import_path` produces an opaque crash, not a named one--`cli_command.import_path.partition(":")` on a path missing the `:attr` separator returns `("pkg.module", "", "")`, so the harvest calls `getattr(importlib.import_module("pkg.module"), "")`. There is no validation that `import_path` actually contains a `:`. `getattr(module, "")` raises an opaque `AttributeError` (or, on some objects, silently returns something unexpected) at `_cli.py` import time — again aborting the whole CLI, and without fail-loudly naming the offending plugin/command the way the rest of the registrar does.--**Failure scenario.** An external plugin registers `add_cli_command(import_path="pkg.module")` (forgetting the `:attr`). Every `pipelex` invocation crashes at import with an `AttributeError` that names neither the plugin nor the command, so the user cannot tell which installed plugin is at fault.--### Facet 4 — `add_cli_command` has no collision guard; a plugin can silently shadow a core command--This is the odd one out: it is not a crash, it is a **silent override**, and it is a correctness/trust problem rather than an availability one. Every *other* contribution type fail-loud on conflict and names both contributors: `add_orchestrator` (`:116`), `add_inference_backend` (`:110`), `_claim` (`:145`). `add_cli_command` (`:135`) performs **no** duplicate or core-name-collision detection at all. The harvest then calls `app.command(name="run", ...)(command)` at import time, registering a **second** `run` on the Typer app. Click resolves the last registration, so the plugin's callable silently wins; `list_commands` lists the name twice.--**Failure scenario.** A third-party (or buggy future in-tree) plugin advertises `add_cli_command(name="run", import_path="evil:run")`. On the next `pipelex run …`, the plugin's callable is dispatched **instead of** the core `run` command — no error, no warning — and the command appears twice in `pipelex --help`. Core execution is silently replaced by plugin code. The same applies to any core name (`validate`, `show`, `which`, `plugins`, …) and to two plugins colliding on the same custom name.--## Why this is the critical cluster (severity reasoning)--- **It defeats its own recovery path.** The defining property of all three crash facets is that the bricked commands *include the ones designed to recover from the brick* (`doctor`, `plugins`, `init` → `plugins.disabled`). A fail-loud error you cannot act on is worse than a fail-loud error you can.-- **It is live exactly where the project is heading.** It is latent for in-tree-only installs, but the entire point of the seam (Phase 5) is external `pipelex.plugins` dists. The surface becomes load-bearing precisely when externalization ships.-- **Facet 4 is a silent-override / trust hole**, which the "fail loud" invariant and the parallel guards on every sibling contribution type were specifically meant to prevent. CLI commands are the one unguarded menu, and the one that can shadow core behavior.-- **It is a regression in blast radius.** Pre-Phase-3, discovery failure was contained to boot (`Pipelex.setup()`); a user could still run `--help`/`doctor`. Phase 3 widened the blast radius to "every invocation."--## How to reproduce (write the failing tests first)--These sketch the minimal triggers; turn each into a red test before exploring fixes.--1. **Facet 1:** register a fake external `pipelex.plugins` entry point whose `register()` raises (or whose `targets_api` mismatches `PLUGIN_API_VERSION`), then invoke `pipelex --help`. Expect: today it raises at import; the desired contract is `--help` still works (core commands listed), with the breakage surfaced through `doctor`/`plugins`, not a hard crash.-2. **Facet 2:** create a `.pipelex/pipelex.toml`, `chmod 000` it (or simulate `open()` raising `PermissionError`), invoke `pipelex --help`. Expect: today it raises uncaught `PermissionError`; the contract wants a fallback to package defaults.-3. **Facet 3:** a fake plugin contributing `add_cli_command(name="x", help="…", import_path="some.module")` (no `:`). Invoke any `pipelex` command. Expect: today an opaque `AttributeError` at import; the contract wants a named, contextual error attributing the plugin/command (and ideally not bricking the rest of the CLI).-4. **Facet 4:** a fake plugin contributing `add_cli_command(name="run", help="…", import_path="…:something")`. Assert that `run` is registered twice / that core `run` is shadowed. Expect: today it silently overrides; the contract wants a fail-loud collision (parallel to `DuplicateOrchestratorError`).--The existing harvest tests to extend: `tests/.../test_plugin_cli_command_harvest.py` and `test_harvest_config_falls_back_to_base_on_broken_user_config` (the S1 test — covers Facet 2's *parse* path only).--## Solution space — open questions to explore (NOT answered here)--Listed only to map the decision space for the fresh session. Each is a genuine tension; none is pre-judged.--- **Altitude / timing.** Should the harvest run at import time at all, or be deferred to the point where the command list is actually needed (the `PipelexCLI` group already subclasses `TyperGroup` and overrides `list_commands`/`get_command`)? What does `--help` *need* vs. what does dispatching a plugin command *need*? Does shell tab-completion change the answer?-- **Reconciling the two invariants.** "Fail loud on a broken plugin" (boot) vs. "never brick the recovery commands" (CLI startup) are both deliberate. Where is the seam between them? Is the answer "fail loud, but only when the user actually invokes the broken plugin's command / a boot-requiring command, and degrade-with-a-visible-notice for `--help`/`doctor`/`init`"? What does "visible notice" mean for an LLM-facing vs. human-facing surface (see the workspace "format follows consumer" rule)?-- **Degrade vs. abort.** If discovery fails during the harvest, what is the right degraded state — core commands only, with the failure reported through `doctor`/`plugins`? Or is partial discovery (the built-ins succeeded, one external failed) meaningful?-- **Collision policy (Facet 4).** Should `add_cli_command` route through a `_claim`-style guard that refuses to shadow a core command name and refuses two plugins claiming the same name? Is "core command names are reserved" a hard rule? Is there *any* legitimate reason a plugin should override a core command? How does this interact with `_CORE_COMMAND_ORDER` being the source of truth for core names?-- **`import_path` validation.** Where should the `:`-presence (and attribute-exists) validation live — at `add_cli_command` registration time (fail-loud, names the plugin), or at harvest/dispatch time? Should `CliCommand` validate its own `import_path` shape as a pydantic model?-- **Single vs. double discovery.** Boot re-runs `build_registrar` (`pipelex.py:~390`) after the harvest already ran it, discarding the harvest result. Any fix that changes *when/whether* the harvest runs interacts with this redundancy (tracked separately in [phase-3-review-deferred.md](phase-3-review-deferred.md) under "double discovery / import-time cost"). Decide whether the two are solved together.--## Pointers--- Code: `pipelex/cli/_cli.py` (harvest), `pipelex/plugins/discovery.py` (`build_registrar`), `pipelex/plugins/registrar.py` (`add_cli_command` + the guarded siblings), `pipelex/tools/misc/toml_utils.py:39` (the `open()`).-- Decision: **D3** in [TODOS](../../TODOS.md) ("plugin CLI commands harvested by running the pure `build_registrar` at CLI-build; same fn runs again at boot").-- Invariants: TODOS "Invariants that must survive every phase" — *fail loud* on a broken plugin; "installed but broken" ≠ "not installed".-- Sibling deferred items (medium/low): [phase-3-review-deferred.md](phase-3-review-deferred.md).diff --git a/wip/plugins/phase-3-review-deferred.md b/wip/plugins/phase-3-review-deferred.mddeleted file mode 100644index 16ef907ce..000000000--- a/wip/plugins/phase-3-review-deferred.md+++ /dev/null@@ -1,31 +0,0 @@-# Phase 3 plugin-seam review — deferred items--Items surfaced by the xhigh `/code-review` over the Phase 3 checkpoint commit (`19e6ca66b` / `5f61db323`) — orchestrators through the seam: the bridge `match` collapse into `OrchestratorRegistry`, the DIRECT/Temporal orchestrators, the slot-claim apply-points, the CLI-command harvest, and `MissingOrchestratorError`. These are the medium-to-low findings left for later triage; none produces wrong output on a live in-tree path today, so none was changed in the checkpoint.--> **The critical finding is split out.** The dominant correctness cluster — the import-time CLI plugin-command harvest is an unguarded surface that can brick every `pipelex` command (recovery commands included) and can silently shadow a core command — has its own cold-start problem statement in **[phase-3-critical-cli-harvest-fragility.md](phase-3-critical-cli-harvest-fragility.md)**. Read that first; it is the one item worth blocking on. This doc holds everything else.->-> 19 of the 40 verified candidates were **refuted** by the verify pass (e.g. the `registrar.config.temporal.is_enabled` precondition — production callers always pass a full config; the exhaustive `MissingOrchestratorError._build_message` match — pyright-clean and correct per the no-default-case enum rule; several TASK_MANAGER-slot divergence claims that cause no functional bug today). Only the kept items are below.--## High — deferred by design, but the error message actively misleads--- **`MISTRAL_NATIVE` is currently a dead execution mode.** `runtime_bridge/bridge.py` deleted `_run_mistral_native` and flipped to registry discovery, but **no plugin contributes a `MISTRAL_NATIVE` orchestrator** — `pipelex-mistralai-workflows` has no `pipelex.plugins` entry point / `add_orchestrator` call (it exposes only `register_pipelex_primitives` + a `make_mistral_workflows_pipe_run` factory the *old* bridge imported directly). This is expected per the Phase-3 goal ("core names no integration — not by import, not by string"), and landing the MISTRAL_NATIVE plugin onto `dev` is tracked as its own sub-track ([../distributed-execution/mistral-workflows/](../distributed-execution/mistral-workflows/)). The deferred *defect* is the user-facing message: a consumer who **already has** `pipelex-mistralai-workflows` installed and submits `execution_mode=MISTRAL_NATIVE` now gets `MissingOrchestratorError` telling them to `pip install pipelex-mistralai-workflows` — a package they have. Until the plugin grows an entry point, MISTRAL_NATIVE is wholly unusable and the hint points at the wrong remedy. Secondary: the old arm returned `workflow_id=pipe_output.pipeline_run_id`; the seam provides no orchestrator preserving that non-null `workflow_id` contract.--## Medium — error-contract & teardown fragility at the new seam--- **The bridge no longer wraps orchestrator errors; the single-error-type contract now rests on each orchestrator's discipline.** `runtime_bridge/bridge.py` calls `orchestrator.run()` with no surrounding `try`/`except` — error-wrapping into `PipelexBridgeDispatchError` moved *into* each orchestrator. The in-tree DIRECT/TEMPORAL orchestrators kept their wrapping, but an **external** orchestrator (the future MISTRAL_NATIVE plugin) must now wrap its own pipe-execution errors; the old `_run_mistral_native` did that wrapping at the bridge. If an external orchestrator raises a raw `PipeRunError`/`PipeJobError`/`PipeExecutionError`, it propagates unwrapped through the bridge, and a host catching only the documented `PipelexBridgeDispatchError` fails to catch it. Decide whether the SPI documents "orchestrators MUST raise `PipelexBridgeDispatchError`" as a hard contract (with a conformance test), or whether the bridge keeps a thin outer wrap as a safety net.--- **`teardown()` runs plugin callbacks with no `try`/`finally`; one raising callback aborts all core teardown.** `pipelex/pipelex.py:teardown()` (~`:494`) iterates `reversed(self._plugin_registrar.teardown_callbacks)` and *then* runs core teardown (pipeline/telemetry/inference/reporting/sdk-client managers). If a plugin teardown callback raises — e.g. a Temporal worker's `task_manager.teardown()` or `temporal_hub.reset()` throwing during an unclean shutdown — the exception propagates out of the loop and **none** of the subsequent core teardown runs, leaking those resources/handles. The old code had this same shape for the single inlined Temporal block, so this is a pre-existing fragility now generalized across N callbacks — flagged per the "flag and fix pre-existing bugs" convention. Cleanup that must run on the error path belongs in `try`/`finally` (per the error-handling standard).--- **`_teardown_temporal` re-fetches the task manager from the global hub instead of the instance it built.** `pipelex/temporal/temporal_plugin.py` (~`:63`): `_setup_temporal_task_manager()` constructs a `TemporalTaskManager`, sets it on the module-global `temporal_hub`, and discards the local reference; `_teardown_temporal()` re-fetches via `temporal_hub.get_optional_task_manager()`. If any path between setup and teardown resets `temporal_hub` (a nested/aborted second boot, or a reentrant `TemporalPlugin` register that fails after claiming), the fetch returns `None`, teardown silently skips `task_manager.teardown()`, and `TemporalManager` session/sandbox state + the in-workflow content generator leak into the next worker boot. Holding the constructed instance and tearing it down directly (instead of round-tripping through the global) would forward the call unconditionally.--- ✅ **RESOLVED (Option A, step 14).** Folded into the orchestrator-plugins.md rewrite: the doc now says the Temporal plugin is discovered through `BUILTIN_PLUGINS` (pipelex declares **no** `pipelex.plugins` self-entry-point) and states the real Phase-5 delta (remove from `BUILTIN_PLUGINS` + add a `pipelex.plugins` entry point in the new dist). Original finding retained below. **The SPI doc is factually wrong about how the Temporal plugin is discovered.** `docs/under-the-hood/orchestrator-plugins.md` (~`:116`) states: "The Temporal plugin is in-tree today and discovered via an entry point pipelex declares on itself." False on two counts: (1) the Temporal plugin is in `BUILTIN_PLUGINS` (a hardcoded list in `pipelex/plugins/builtins.py`), and `pyproject.toml` declares **no** `[project.entry-points."pipelex.plugins"]` on pipelex itself; (2) it misstates the Phase-5 story — externalization is *not* "the same entry point shipped from its own dist," because there **is** no entry point today; it requires *adding* one and *removing* the builtin. An engineer doing Phase 5 (or debugging plugin-override pickup) would search `pyproject.toml` for a self-entry-point that does not exist. Fix the prose to describe the builtin-list mechanism and the real externalization delta. *(This is the same gap the Phase-3 as-built "Notes for Phase 4/5" already flags from the implementation side — keep the two in sync when fixing.)*--## Low — cleanup, non-blocking--- **Enum identity comparison instead of the property.** `runtime_bridge/bridge.py` `_validate_input` (~`:165`) singles out fire-and-forget via `input_payload.execution_mode is PipelexExecutionMode.TEMPORAL_FIRE_AND_FORGET` instead of the existing `PipelexExecutionMode.is_fire_and_forget` property added for exactly this case. Violates the python-standards rule "never test equality to an enum value; add an `@property` so the linter complains when a new value is added." A future fire-and-forget-like mode would silently miss this validation guard. Pre-existing (untouched by the diff but in the reshaped file) — flagged per "flag and fix pre-existing."--- ✅ **RESOLVED (Option A).** The import-time CLI harvest is gone, so the second discovery, the per-invocation config load (deep-merge + full `PipelexConfig` validation), and the `entry_points(group=…)` metadata scan no longer run on any `pipelex` invocation (including `--help` and shell tab-completion) — only boot discovers now. Original finding retained below. **Double discovery + import-time cost on every invocation.** On any boot-requiring command (`run`/`validate`/`build`), plugin discovery + config load runs **twice**: once in the import-time CLI harvest (`_cli.py:~280`), once again at boot (`pipelex.py:~390`, `build_registrar(config=get_config())`), with the harvest registrar discarded except for command names. Separately, the harvest adds a full config load (deep-merge of multiple TOML overrides + full `PipelexConfig` validation) **plus** an `importlib.metadata.entry_points(group=...)` metadata scan to **every** `pipelex` invocation — including `--help` and shell tab-completion, which did none of this pre-Phase-3. Both stem from the harvest running fully, at import, on every call. **Coupled to the critical finding:** any redesign that defers/memoizes the harvest (see [phase-3-critical-cli-harvest-fragility.md](phase-3-critical-cli-harvest-fragility.md) "Solution space → Single vs. double discovery") should resolve this for free — solve them together, don't apply a narrow perf patch here in isolation.--- **Dead `serialize_pipe_output` re-export.** `runtime_bridge/bridge.py` (~`:43`) keeps a `serialize_pipe_output` re-export commented "for host-runtime back-compat," but the real definition + every internal use moved to `serialization.py`, and grep across `pipelex/`, `tests/`, and the sibling runner repos (`pipelex-api` / `pipelex-worker` / `pipelex-mistralai-workflows`) finds **zero** importers of it from `bridge`. Under the workspace "No backward compatibility" policy this is unjustified duplicated surface that keeps `bridge.py` importing `serialization.py` purely to re-export, and silently rots. Drop it; in-tree callers and the orchestrator SPI already import from `serialization.py` directly.--- **Hardcoded count in a docstring.** `pipelex/temporal/temporal_plugin.py` (~`:11`) says the plugin "claims the **four** process-global hub slots," violating the workspace "never hardcode counts" rule. The next phase (Phase 4, the `model_lists` 5th seam) is precisely the kind of change that makes "four" go stale while the `claim_*` calls below it claim a different number. Write "the process-global hub slots" with no count.diff --git a/wip/plugins/phase-4-residual-core-vendor-couplings.md b/wip/plugins/phase-4-residual-core-vendor-couplings.mddeleted file mode 100644index 248600333..000000000--- a/wip/plugins/phase-4-residual-core-vendor-couplings.md+++ /dev/null@@ -1,36 +0,0 @@-# Phase 4 — residual core→vendor couplings outside the enumerated seams--Phase 4 inverted the 5th and last *dispatch* seam (`model_lists.py`'s `match sdk:`), so every seam the plan enumerated now names no integration: the four inference worker factories, model listing, orchestrator dispatch, the boot/teardown hub swap, and the CLI (removed in Option A). With those done, "core names no integration" holds **for the dispatch surfaces the effort was scoped to**.--The CHECKPOINT-4 verification grep is broader than those surfaces, though, and it surfaces three **pre-existing** `pipelex.plugins.<vendor>` references in core that were never part of the Phase 0–4 seam list. They are recorded here so the "core names no integration" claim stays precise and a future session can decide their fate. None were introduced by Phase 4 — all three are unchanged from `main`.--## 1. `cogt/config_cogt.py` — vendor config models (by design, not a leak)--```-from pipelex.plugins.anthropic.anthropic_config import AnthropicConfig-from pipelex.plugins.google.google_config import GoogleConfig-from pipelex.plugins.mistral.mistral_config import MistralConfig-from pipelex.plugins.openai.openai_config import OpenAIConfig-```--This is the **typed-config-in-core** pattern, an explicit decision: the design's "Deferred / out of scope" list keeps per-plugin *typed config* in core ("Generic per-plugin typed-config namespace — Temporal config stays typed in core (design D7)"). Inverting it would mean a generic per-plugin config namespace, which the design deliberately declined. **Leave as-is** unless that decision is revisited.--## 2. `cogt/img_gen/img_gen_args_factory.py` — `OpenAIImgGenFactory` (genuine, unscoped)--`ImgGenArgsFactory` imports `OpenAIImgGenFactory` at module top to build provider-specific image-generation API arguments. This is a real core→vendor coupling, but it is an **argument-shaping** concern, not one of the worker-dispatch / listing / orchestrator / boot / config / CLI seams the plan enumerated. Inverting it would need its own capability on the contract (an "img-gen args" SPI), i.e. genuine new design — not a checkpoint cleanup. **Deferred; needs a design decision before it becomes a seam.**--## 3. `cogt/model_backends/backend_factory.py` — `VertexAIFactory` (genuine, unscoped)--```python-match name:- case "vertexai":- from pipelex.plugins.openai.vertexai_factory import VertexAIFactory # noqa: PLC0415- endpoint, api_key = VertexAIFactory.make_endpoint_and_api_key(extra_config=extra_config)-```--A lazy, `match`-guarded import that resolves vertexai's special endpoint/api-key auth during backend construction. Like (2), a real coupling but a **backend-auth** concern outside the enumerated seams; inverting it would need a backend-auth capability on the contract. **Deferred; needs a design decision before it becomes a seam.**--## Disposition--- (1) is **intentional** per D7 — not a follow-up, just documented so the grep result reads honestly.-- (2) and (3) are **genuine additional seams** the original plan did not enumerate. They do not block CHECKPOINT 4 (whose primary assertion — `model_lists.py` names no integration — holds). If the goal is upgraded from "no integration in the enumerated seams" to a literal "no integration anywhere in core," they are the next two candidates, each requiring its own contract capability.diff --git a/wip/plugins/phase-4-review-followups.md b/wip/plugins/phase-4-review-followups.mddeleted file mode 100644index c3897ceca..000000000--- a/wip/plugins/phase-4-review-followups.md+++ /dev/null@@ -1,42 +0,0 @@-# Phase 4 — xhigh code-review follow-ups--Record of the xhigh `/code-review` run on the Phase 4 commit (`6a0796548` — "invert model listing, the 5th and last dispatch seam"). The review confirmed the inversion is **byte-equivalent to the old static `match`** for every shipped config — no crash or wrong-output bug. Adversarial verify kept a handful of findings; this note dispositions them.--Two cheap cleanups were **applied in the same pass** (see the bottom of this note). The four below are **design tradeoffs, not bugs** — deferred deliberately rather than reflexively "fixed," so the decision is on the record for a future session. Sibling to [`phase-4-residual-core-vendor-couplings.md`](phase-4-residual-core-vendor-couplings.md) (which covers core→vendor couplings *outside* the enumerated seams); this note covers review findings *on the model-listing seam itself*.--## 1. Model listing is now coupled to the plugin enable/disable switch — DEFERRED (arguably correct as-is)--`cogt/model_backends/model_lists.py`. The old static `match sdk:` listed `anthropic` / `mistral` / `bedrock` / `google` SDKs unconditionally. After the inversion, listing dispatches through the registry, and those four plugins are **not** in `CORE_UNCONDITIONAL_PLUGIN_NAMES` — so adding one to `config.plugins.disabled` removes its lister, and `pipelex show models <backend>` then reports that SDK as "we don't support for remote listing."--- **Real?** Yes — it is the one live behavior change in the commit.-- **Why deferred (and arguably *correct*):** a disabled plugin also registers no `make_worker`, so its models can't be *run* either. Listing models you can't use was the old anomaly; gating listing on the same enable switch as execution is the more consistent behavior. "Fixing" it (e.g. registering listers even for disabled plugins) would re-introduce the inconsistency.-- **Revisit if:** product decides `plugins.disabled` should mean "hide the driver but still enumerate its catalog" (a discovery-vs-activation split). That is a config-semantics decision, not a Phase 4 bug.--## 2. Bedrock lister key skew (`bedrock` / `bedrock_aioboto3` vs worker key `bedrock_boto3`) — DEFERRED (zero live impact)--`plugins/bedrock/bedrock_plugin.py`. The lister registers under `bedrock` (dead — no shipped model spec carries `sdk="bedrock"`) and `bedrock_aioboto3`, while the **inference backend** in the same `register()` body registers under `bedrock_boto3` / `bedrock_aioboto3`. So within one function the lister and worker key sets deliberately differ.--- **Real?** Byte-equivalent to the old `case "bedrock" | "bedrock_aioboto3"` arm — carried forward verbatim, **zero live impact**. The `bedrock` key is dead today.-- **Why deferred:** dropping the dead `bedrock` key is a tiny cleanup, but `tests/unit/pipelex/plugins/test_model_lister_coverage.py` pins the registered-key set as an *exact-match* contract (`EXPECTED_LISTER_SDKS` includes `bedrock`), so the cleanup is "drop the key **and** update the exact-set test + its docstring framing" — a coupled edit better done deliberately than as a checkpoint drive-by.-- **Footgun to watch:** a future bedrock variant whose model specs use `bedrock_boto3` would get "unsupported for listing" instead of a model list, because no lister is registered under that key. If/when bedrock keys are consolidated, align the lister keys with the worker keys in the same `register()` body.--## 3. `any_listed` leaks the listing loop's display state into the public SPI — DEFERRED (pre-existing, now on the public surface)--`plugins/*/*_plugin.py` lister closures + `ListModelsFn` + the authoring example in `docs/under-the-hood/inference-backend-plugins.md`. `any_listed` is CSV-header / separator suppression state owned by `ModelLister.list_models` — pure loop-presentation bookkeeping. The inversion re-published it as part of the *public* `ListModelsFn` contract, so every out-of-tree backend author must accept and correctly forward a flag whose meaning is internal to the loop; a lister that forwards it wrongly produces a duplicated/missing CSV header.--- **Real?** Pre-existing (the old per-arm calls already threaded it), but Phase 4 cemented it into the new SPI surface.-- **Why deferred:** the clean fix is to keep the lister contract to the data it needs (`sdk` / `backend_name` / `backend` / `flat`) and let the loop own the header decision (it already knows `any_listed` — pass an `is_first` hint or post-process). That is an SPI-shape change touching every lister + the doc + the display helpers — a focused refactor, not a checkpoint cleanup.-- **Best moment:** before any third-party backend ships against `ListModelsFn`, while the only implementers are in-tree.--## 4. `ListModelsFn = Callable[..., Awaitable[None]]` gives the loop's kwargs zero static checking — DEFERRED (follows Phase-1 precedent)--`plugins/model_lister_registry.py`. The whole byte-equivalence claim rests on every lister's kwargs matching the loop's `await` call, yet the `...` args mean pyright/mypy can't catch future drift: rename a lister param or change the loop's call kwargs and the mismatch surfaces only at runtime as a `TypeError` wrapped into a confusing `PipelexCLIError`.--- **Real?** Yes as a latent type-safety gap; not a live bug (the shipped kwargs match).-- **Why deferred:** a `Protocol` with the explicit keyword-only signature (the way `OrchestratorProtocol` already types the orchestrator seam) would catch drift statically at no runtime cost — but `MakeWorkerFn` uses the *same* loose `Callable[..., ...]` form, so tightening only `ListModelsFn` would split the convention. The right scope is "tighten both inference-family callables to Protocols together," a Phase-1-and-Phase-4 SPI-typing pass, not a model-listing-only edit.-- **Pairs with #3:** if `ListModelsFn` becomes a `Protocol`, drop `any_listed` from its signature at the same time.--## Applied in the same pass (not deferred)--- **Registrar `_add` helper** — `add_inference_backend` / `add_model_lister` / `add_orchestrator` each hand-rolled the identical dup-check → store → mirror-into-`_*_sources` → append-contribution shape. Generalized behind one `PluginRegistrar._add(...)` (each method supplies its keyed store, parallel sources dict, contribution label, and a factory that builds its distinctly-typed `Duplicate*Error`), mirroring the existing `_claim` helper on the slot menu. A future 6th registration seam now reuses `_add` instead of becoming a 4th copy.-- **Hardcoded counts in the Phase 4 as-built** — removed the item-count enumerations the workspace `CLAUDE.md` "Never hardcode counts" rule forbids (`9 expected sdk keys`, `4 soft-miss cases`, `2 new pages`, `274 unchanged`, `4 keys`, `2 keys`). Point-in-time test-run evidence (`715 passed`, pyright file count) was left as a verification snapshot.diff --git a/wip/plugins/phase-5-cutover-review-followups.md b/wip/plugins/phase-5-cutover-review-followups.mddeleted file mode 100644index 5ca94312f..000000000--- a/wip/plugins/phase-5-cutover-review-followups.md+++ /dev/null@@ -1,70 +0,0 @@-# Phase 5 local cut-over — `/code-review` follow-ups--**Source:** xhigh `/code-review` on commit `a8382d1a1` (the `_plugins`/core side of the Temporal externalization — `git rm` of `pipelex/temporal/` + the `temporal` extra, pre-publish). This is the triage for that review (TODOS Checkpoint 5 → "Fan-out `/code-review` … Triage here").--**State that frames every decision below:** the cross-repo move is **already done locally** — `pipelex-temporal` is scaffolded at `../pipelex-temporal` (top-level `pipelex_temporal` package per D8), the temporal tree + tests + marker + `--temporal-server` conftest moved there, editable pins flipped, suites green on both sides. Nothing is published/merged/deployed. So "the following steps" = flip downstream consumer pins (`pipelex-worker`, `pipelex-api-hosted`), repoint `pipelex-mistralai-workflows`, run consumer suites, then publish/merge. None of the items below block those steps.--## Finalize-local pass (2026-06-20, session 2) — core dead-tooling removal + pipelex-temporal review triage--A second pre-publish local pass (still gated before publish). Goal: get **core clean of the moved Temporal dev-tooling** and review/triage the **pipelex-temporal side** of the cut-over (session 1 reviewed the core *deletion* side; this reviews the *additions* side). All changes local, uncommitted at time of writing, on top of the `a8382d1a1`/`27da150a5` cut-over commits (core) and `c39fbdc` (pipelex-temporal).--**Done:**--- **Both Temporal skills relocated to `pipelex-temporal` and removed from core.** Closes item 1 **and extends it** — alongside `temporal-e2e-validate` (the only one item 1 named), `temporal-test-crate` (`.claude/skills/temporal-test-crate/SKILL.md`, also dead in core: it pointed at the moved `tests/integration/pipelex/temporal/library_crate/`) was found and relocated too. Mechanical rewrite on the copies (`tests/{integration,unit}/pipelex/temporal/` → `tests/{integration,unit}/pipelex_temporal/`, `pipelex.temporal` → `pipelex_temporal`, and the one cross-repo `../pipelex/tests/` → `../pipelex-temporal/tests/`). Verified: residue grep clean (only the `.pipelex/temporal-payload-store/` runtime-dir false-positives remain — that dir name is config, not the module), the 3 scripts `py_compile`, and `submit_dry_validate.py`'s rewritten `pipelex_temporal.{exceptions,tprl_pipe.*}` imports resolve in pipelex-temporal's venv. Skills live in `.claude/` → **not** in the publishable wheel.-- **Core Makefile temporal targets removed** (the "or delete from core" half of item 2): help block, `.PHONY` entries, the `test-temporal`/`ttm` target + its `SRV`/`MODE`/`REG`/`TEMPORAL_*` vars, and `temporal-server`/`-stop`/`-worker`(+`-router`/`-runner`)/`-run`/`-run-dry`. Verified `make help`/`make -n` parse and every removed target reports "No rule to make target." The `agent-test-debug` target's temporal *comments* + the no-op `pkill temporal-sdk-python` were **left intentionally**, paired with the deferred `debugging-hanging-pytest-runs.md` (item 6) so the target and its doc de-temporalize together.-- **pipelex-temporal review (additions side): SHIP — clean.** `pr-review-toolkit:code-reviewer` built the wheel offline and verified empirically: `py.typed` packaged, `tests/` excluded, both entry points emitted, the `[tool.uv.sources] pipelex = {path=../_plugins}` pin does **not** leak into wheel METADATA (clean `Requires-Dist: pipelex>=0.35.0`), import-light proven at runtime (importing `temporal_plugin`/building the CLI leaves `temporalio` out of `sys.modules`), end-to-end entry-point discovery resolves, the namespace rewrite is logic-drift-free across the whole tree, suite collects standalone (0 errors). **No BLOCKER / SHOULD-FIX.** Two NITs:- - **stray `.pipelex/traces/*.ndjson` (9 files) committed — FIXED.** Confirmed stray: tracing-guard tests write to `f"..._{uuid.uuid4().hex[:12]}"` dirs each run, so the committed fixed-uuid files are never read back. `git rm`'d + added `.pipelex/traces/` to `.gitignore`.- - **shipped `test_extras/wf_test_*` back-import from the non-shipped `tests/` tree — DEFERRED.** `pipelex_temporal/test_extras/wf_test_content_generator_{child,pdf_page_views}.py` do `from tests.integration.pipelex_temporal.test_data import PipeTestCases`. Pre-existing (byte-faithful to `933b6c35c`) and dormant in production (gated behind `is_unit_testing` in `temporal_task_manager.run_worker`). Eventual fix: relocate `PipeTestCases` into `pipelex_temporal.test_extras` so nothing shipped depends on `tests/`. Not a cut-over blocker.--**Gates (this pass):** core `make tb` green + full `make agent-test` green (exit 0, HEAD `27da150a5`) + Makefile parses; pipelex-temporal `make agent-check` clean.--**Still open (added by this pass):**--- **pipelex-temporal Makefile lacks the Temporal dev targets** (`temporal-server`/`test-temporal`/`temporal-worker`/`temporal-run`) removed from core. Item 2's "move to pipelex-temporal's Makefile" half is **not** done — its Makefile delegates to `Makefiles/Makefile_basics.mk` and the targets can't be verified without a running Temporal server, so adding them was deferred rather than shipped unverified right before publish. Add when the repo's Temporal dev loop is next exercised.--## Fixed in session 1 (cut-over review, commit `a8382d1a1`)--Reachable breakages + cheap core-hygiene that stay in core and the cross-repo move would not otherwise touch:--- **kit `agent_rules/commands.md`** — removed the "Temporal Integration Test Options" section (pointed at the deleted `tests/integration/pipelex/temporal/`; shipped into client `.pipelex/`). Regenerated the derived `CLAUDE.md` via `pipelex-dev kit rules`; `check-rules` PASSED.-- **`uv.lock`** — regenerated; dropped the stale `temporal` extra (`temporalio`, `nexus-rpc`, `types-protobuf`). `uv lock --check` now clean.-- **`runtime_bridge/bridge.py`** docstring — "in-tree Temporal plugin" → external `pipelex-temporal`; dropped the "an in-tree orchestrator whose extra is absent" framing (no extra exists anymore).-- **`tests/unit/pipelex/system/test_config_temporal_optional_dep.py`** — docstring + assertion message no longer reference the deleted `temporal` extra (now: temporalio is not a core dep, ships only with `pipelex-temporal`).-- **5 plugin/cogt test stubs** (`test_plugin_discovery`, `test_inference_backend_coverage`, `test_model_lister_coverage`, `test_llm_worker_factory`, `test_img_gen_worker_factory`) — dropped the now-vestigial `temporal=SimpleNamespace(is_enabled=False)` stub field (no builtin's `register()` reads `config.temporal` post-cut-over — verified) and corrected the "TemporalPlugin is a builtin" comments.-- **Restored two core-coverage tests** for behavior that stayed in core but lost its test:- - `test_trace_context_contract.py::test_non_direct_mode_nulls_host_trace_context` — pins the `trace_context if is_direct else None` guard in `bridge.py` (non-DIRECT modes must null the host trace_context; doubles as a check that no TEMPORAL orchestrator is registered in core).- - `test_dispatch_options_no_temporalio.py` — `DispatchOptions` must be constructible without temporalio (complement to the AST scan, which cannot catch a regression of the `RetryPolicy = Any` placeholder to a bare forward ref → `PydanticUserError` at construction).--Gates after the fixes: targeted unit suites green (104 passed), `make agent-check` clean (pyright 0/0, mypy 0, ruff/plxt/keyword-only pass).--## Deferred — finalize with the cross-repo publish--These describe the Temporal surface that **moved** to `pipelex-temporal`. They don't bite core CI and don't block the publish steps; do them as part of finalizing the move (the package now exists locally to reference concretely).--1. **✅ DONE (session 2 — relocated, + `temporal-test-crate`; see *Finalize-local pass* above).** **Skill `temporal-e2e-validate`** (`.claude/skills/temporal-e2e-validate/`) — `scripts/submit_dry_validate.py` imports the deleted `pipelex.temporal.{exceptions,tprl_pipe.act_dry_validate,tprl_pipe.dry_validate_dispatch}` and its `DEFAULT_BUNDLE` points at the deleted `tests/integration/pipelex/temporal/...`. The whole skill exercises the Temporal arm, which now lives in `pipelex-temporal`. **Relocate the skill to `pipelex-temporal`** (and repoint imports to `pipelex_temporal.*`); it cannot run from the core venv (temporalio absent). Until then it is dead in core.--2. **✅ core-removed (session 2); ⏳ pipelex-temporal add still open (see *Finalize-local pass* above).** **Makefile temporal targets** (CI-isolated — `agent-test`/CI never invoke them; verified `agent-test:` deps don't reach them): `temporal-worker`/`tw` (runs deleted `pipelex.temporal.worker_cli`), `test-temporal`/`ttm` (`TEMPORAL_TESTS_DIR = tests/integration/pipelex/temporal/`, deleted), `temporal-run`/`temporal-run-dry`, `temporal-worker-router/-runner` (`twr`/`twn`), `temporal-server`/`ts`, and the debug-run guidance referencing the deleted temporal conftest. Move to `pipelex-temporal`'s Makefile (or delete from core). Also drop the temporal lines from the `make help` text and `.PHONY`.--3. **`docs/under-the-hood/orchestrator-plugins.md`** — rewrite Temporal as the canonical **external** orchestrator plugin (symmetric with `pipelex-mistralai-workflows`). Stale spots: frontmatter + line 10 ("in-tree Temporal plugin"), line 43/47 ("extra absent"; the install hint is rendered single-quoted `'pipelex-temporal'` while the code emits unquoted `pip install pipelex-temporal`), lines 82–89 (declares the `pipelex-temporal` console script in **core's** `pyproject.toml` — now in `pipelex-temporal`), lines 110–119 worked example ("in-tree today", "discovered through `BUILTIN_PLUGINS`", "Externalizing … (Phase 5)" as a *future* event — it has happened).--4. **`docs/under-the-hood/temporal-integration.md`** — the module map (~lines 106–276) enumerates concrete deleted `pipelex/temporal/**.py` files. Rewrite to point at `pipelex-temporal`, or reduce to an overview that links the external repo.--5. **`docs/errors/` temporal pages + `generate-error-pages`** — **do NOT run `generate-error-pages` in core until this is settled**: it deletes any page whose class is no longer importable from core, 404-ing the `type_uri` those errors emit. Per the cut-over (TODOS Phase-5 as-built), the split is:- - **Stay in core** (classes in `system/configuration/{config_temporal,exceptions}.py`): `temporal-config-error.md` (`TemporalConfigError`), `worker-task-queue-unknown-error.md` (`WorkerTaskQueueUnknownError`). Pages valid; regen keeps them.- - **Moved to `pipelex-temporal`**: `temporal-flow-error`, `temporal-server-error`, `unrecoverable-workflow-failure-error`, `workflow-execution-error`, `workflow-input-error`, `search-attribute-registration-error`, `worker-profile-config-error`, `worker-scope-config-error`, `content-generation-error`. Decide where these publish so `https://docs.pipelex.com/.../errors/<x>/` resolves (pipelex-temporal generates/publishes its own, or core retains authored stubs with `<!-- pipelex:authored -->`). Reconcile the links in `docs/errors/execution-and-runtime.md` accordingly.--6. **`docs/agents/debugging-hanging-pytest-runs.md`** (~lines 122, 161) — references the deleted `tests/integration/pipelex/temporal/` and its `boot_temporal` conftest teardown as "the known fixture-teardown hang point." That hang moved with the tests. Update or drop the temporal-specific example (the general methodology stays valid).--## Deferred — design decision (not a bug)--7. **Boot does not eagerly validate `temporal.is_enabled=true` when `pipelex-temporal` is absent.** With `TemporalPlugin` gone from `BUILTIN_PLUGINS`, nothing reads `is_enabled` or claims worker-runtime slots at boot; boot succeeds silently and the misconfiguration only surfaces at the first `TEMPORAL_*` dispatch (`MissingOrchestratorError`, which carries the install hint). When `pipelex-temporal` *is* installed, its entry-point plugin reads `is_enabled` and claims slots correctly. **Decide:** add an eager boot warning/validation for the enabled-but-not-installed case, or accept the lazy dispatch-time error. Note `Pipelex.setup` still accepts `temporal_enabled` and flips the config flag (`pipelex.py:204`) — keep or reconsider as part of the same decision.--## Deferred — cross-repo coupling guards (lower priority; review verifier marked "refuted" but the gap is real)--8. **`pipelex.toml` `non_retryable_error_types` pins externalized class-name string literals.** `worker_config.retry_policy_config.non_retryable_error_types` contains class names whose defining classes moved to `pipelex-temporal`. A rename there silently desyncs from the literal baked into core's `pipelex.toml`, re-making a deterministic failure retryable — the retry-forever family. `test_non_retryable_baseline_pins.py` only asserts the `DryRun*` entries, so this is unguarded. Add a cross-repo conformance pin (or move those literals to `pipelex-temporal`'s config) during the move.--9. **Error-report "parity" framing is now single-arm in core.** `tests/integration/pipelex/error_handling/test_data.py` (+ `test_error_report_local_full_chain.py`, `test_search_error_report_local_full_chain.py`) still frame the constants as driving "local / Temporal parity," but only the local arm runs in core (the Temporal arm moved). The shared constants are duplicated cross-repo with nothing enforcing they stay in sync. Reword the framing, or add a cross-repo guard.--## Deferred — harmless cleanup (do anytime / during the move)--10. **Dead crate fixture in core.** `tests/e2e/pipelex/cv_batch_screening/library_crate/cv_batch_screening_inputs.json` + `CvBatchScreeningTemporalTestData.INPUTS_FILE` (`test_data.py:18`) have zero consumers in this repo — the direct-mode e2e builds inputs inline and reads only `BUNDLE_FILE`. **Not deleted now on purpose:** `test_data.py`'s header documents an *intentional* self-contained crate mirror shared with `pipelex-temporal`'s suite + the skill. Decide during the move: (a) delete the JSON + constant from core (crate copy becomes bundle-only), or (b) keep it a complete mirror. Cosmetic also: `CvBatchScreeningTemporalTestData` is now a direct-mode-only fixture in core — consider dropping "Temporal" from the name.diff --git a/wip/plugins/phase-c-http-error-mapper-consumer-path.md b/wip/plugins/phase-c-http-error-mapper-consumer-path.mddeleted file mode 100644index 6251a6c5e..000000000--- a/wip/plugins/phase-c-http-error-mapper-consumer-path.md+++ /dev/null@@ -1,33 +0,0 @@-# Phase C decision — how `pipelex-api` reaches the F3 HTTP-error mappers--**Status:** deferred design decision, to settle when Phase C (`pipelex-api` decoupling) starts. Not a bug. Surfaced by the Phase A `/code-review` (finding #1).--## Context--Phase A added the F3 seam exactly as the plan scoped it: `PluginRegistrar.add_http_error_mapper(*, exc_type, to_error_report)` plus the read accessor `PluginRegistrar.get_http_error_mappers()`. It deliberately did **not** push the collected mappers onto the hub.--This differs from how the *other* plugin contributions are exposed: orchestrators / inference backends / model listers each get a hub registry (`set_orchestrator_registry(...)` etc.) that any consumer reads via a module-level `get_*_registry()` in `pipelex/hub.py`. The HTTP-error mappers have no such hub surface, and the boot registrar is held privately as `Pipelex._plugin_registrar` (no public getter).--## The question for Phase C--How does `pipelex-api` obtain the mappers at app construction (`api/main.py` → `register_exception_handlers`)? Two viable paths:--1. **Rebuild in the API (no further core change).** Call the documented pure, repeatable `build_registrar(config=get_config())` and iterate `registrar.get_http_error_mappers()`. Precedent: `pipelex/cli/commands/plugins_cmd.py` already calls `build_registrar` standalone. Cost: a third `build_registrar` pass (after CLI-build harvest is gone, this is boot + api-construction) — pure and import-light, so cheap, but it re-runs every plugin's `register()` and is *a* registrar, not *the* boot registrar (fine for a pure function whose output is deterministic for a given config).--2. **Expose the boot registrar from core.** Either (a) a public `get_plugin_registrar()` getter on `Pipelex`/hub returning the already-built registrar, or (b) a full `HttpErrorMapperRegistry` on the hub mirroring the orchestrator pattern. Either keeps a single registrar instance but is a **second core change**, which the per-repo handoff table frames as "Phase C = pipelex-api only".--## Recommendation (not yet ratified)--Default to **option 1** — it keeps Phase C a pure `pipelex-api` change (matching the per-repo table) and `build_registrar`'s contract explicitly blesses repeated calls. Choose option 2(b) only if a second consumer of the mappers appears, or if rebuilding in the API proves to interleave badly with boot ordering. Whichever is chosen, record it in the Phase C as-built.--## Why not just do it now--Per the deferral convention: this is a design tradeoff, not a silent bug, and the plan scoped Phase A to the registrar method + accessor only. Pre-building a hub registry now would be speculative surface for a consumer that doesn't exist yet in this repo. See [[feedback_defer_design_tradeoff_findings]].--## Phase B seam refinement — two consequences to note in the Phase C as-built--The Phase A seam was refined during Phase B (core commit `75196e2e3`): `add_http_error_mapper` now takes a lazy `exc_type_provider` thunk, resolved by `get_http_error_mappers()` at the host's read time, so a plugin's `register()` never imports the orchestrator SDK to name its exception class. Two downstream consequences for the Phase C consumer (surfaced by the Phase B `/code-review`, both deferred — fail-loud either way, no code change):--1. **Duplicate-mapper detection now fires at the host's app-construction read time, not at boot discovery.** `get_http_error_mappers()` is what runs the providers and raises `DuplicateHttpErrorMapperError`. For the locked one-orchestrator-per-image flavor model a collision can't occur, so this is theoretical — but a mis-packaged image with two mapper-contributing plugins fails when `pipelex-api` builds the app, not at boot. Phase C should know the raise lands at its `register_exception_handlers` call site.--2. **`pipelex plugins list` shows a generic `"http error mapper"` contribution line** (no `{exc_type.__qualname__}`), unlike the `family:sdk` / `mode` detail the other contributions carry — an unavoidable consequence of not resolving the type at register. If the per-type label matters, add an optional `display_label: str` to `add_http_error_mapper` (the plugin passes `"temporalio.TemporalError"` as a plain string). Speculative surface today; skip until something needs it.diff --git a/wip/plugins/plugin-api-version-gating-semver.md b/wip/plugins/plugin-api-version-gating-semver.mddeleted file mode 100644index c5338477c..000000000--- a/wip/plugins/plugin-api-version-gating-semver.md+++ /dev/null@@ -1,71 +0,0 @@-# Follow-up: the plugin-API version gate guards nothing — build real version gating (semver)--Surfaced while reconciling the orchestration-mode / delivery split branch against the merged `#1000` (`refactor: split execution_mode into orchestration_mode + DeliveryMode`). The question was whether `#1000`'s breaking change to the orchestrator contract warranted bumping `PLUGIN_API_VERSION` from 2 → 3. Investigating that turned up a deeper problem: **the version gate, as wired today, cannot reject any plugin — it is purely documentary.** This is the task to fix it.--This is the implementation gap behind synthesis point **#13** in [`README.md`](README.md) ("Version the core↔plugin contract … make the contract break *visible*"). The mechanism exists; it just doesn't do what #13 promises.--## The finding: the gate is toothless--The contract is a single coarse integer with an exact-match gate:--- `pipelex/plugins/contract.py:14` — `PLUGIN_API_VERSION: int = 2`. A plugin declares the version it targets via the `targets_api: int` field (`contract.py:35`).-- `pipelex/plugins/discovery.py:100` — the gate:-- ```python- targets_api = getattr(plugin, "targets_api", None)- if targets_api != PLUGIN_API_VERSION:- raise PluginApiVersionMismatchError(...)- registrar.begin_plugin(name=plugin.name, origin=origin, targets_api=PLUGIN_API_VERSION)- ```--The mechanism can never fire, because **every plugin sets `targets_api = PLUGIN_API_VERSION` by importing the constant** — built-ins (`openai_plugin.py`, `anthropic_plugin.py`, `direct_plugin.py`, …) *and* both external orchestrator plugins:--- `pipelex-mistralai-workflows/pipelex_mistralai_workflows/mistral_plugin.py` — `from pipelex.plugins.contract import PLUGIN_API_VERSION` / `targets_api = PLUGIN_API_VERSION`.-- `pipelex-temporal/pipelex_temporal/temporal_plugin.py` — same pattern.--So a plugin's declared `targets_api` *always equals the core it is installed against*. `targets_api != PLUGIN_API_VERSION` is structurally unreachable for any cooperative plugin, no matter what the constant's value is. Nothing in the workspace pins a literal `targets_api` (a `grep -rE 'targets_api\s*=\s*[0-9]'` over every repo returns nothing). And note `discovery.py:105` even *records* the core's `PLUGIN_API_VERSION`, not the plugin's declared value — the system assumes the two are identical by construction.--## Worked proof that this already let a breaking change through silently--`#1000` reshaped the orchestrator SPI in ways that are each a hard break for a plugin built against the prior contract:--- `OrchestratorProtocol.run` gained a required `delivery: DeliveryMode` keyword (no default), and the bridge passes it **unconditionally** (`pipelex/runtime_bridge/bridge.py:102`) → a prior `run` raises `TypeError` at dispatch.-- A new required `supports_fire_and_forget: bool` attribute, read with **no getattr default** (`bridge.py:178`) → a prior orchestrator raises `AttributeError` at `/start`.-- `pipelex/runtime_bridge/execution_mode.py` (defining `PipelexExecutionMode`) was **deleted**; the registry is re-keyed by the open `OrchestrationMode` string token → a prior plugin importing `PipelexExecutionMode` fails at import.--That is unambiguously a breaking SPI change — yet `#1000` shipped with `PLUGIN_API_VERSION` still at **2**, and nothing flagged it. It couldn't: the two external orchestrator plugins were migrated to the new contract in lockstep and re-import the (still-`2`) constant, so the gate stayed satisfied. The version number now labels two incompatible contracts (pre- and post-`#1000`), which is exactly the silent-incompatibility footgun #13 says the gate exists to prevent.--## Why it's harmless *today* but must be fixed before externalization--No published, out-of-tree, third-party plugin exists, and no version of the plugin API has shipped — so there is no real artifact pinned to an old contract for the gate to protect. The bump-to-3 vs stay-at-2 question is therefore moot in practice (which is why the reconciliation just dropped the inert bump and matched origin).--It stops being moot the moment a third party publishes a plugin against a *released* pipelex and pins the version it built against. At that point the current design fails them twice: a plugin that imports the constant still can't detect skew, and a plugin that pins a literal hits an exact-match `!=` that also rejects *compatible* (additive) core versions.--## The proper design (semver)--Two distinct defects to fix — neither alone is sufficient:--1. **Plugins must pin the version they were built against as a literal, not `= PLUGIN_API_VERSION`.** Importing the constant is what makes the declared version auto-track core and the gate inert. The declared value has to be a fact about the plugin's *source* (the contract it was coded against), frozen at author time — not a value re-read from whatever core happens to be installed.--2. **Replace the exact-match gate with a semver compatibility rule.** Even with literals, `!=` is too brittle — it rejects a plugin built against an older *compatible* core.--Concretely:--- Core publishes the plugin-API version as `MAJOR.MINOR` (PATCH irrelevant to the contract). **MAJOR** bumps on a breaking change — a removed/renamed SPI symbol, a changed required signature, a changed registry key type (i.e. exactly `#1000`). **MINOR** bumps on a backward-compatible addition — e.g. v2's optional `add_http_error_mapper` was purely additive and would have been a MINOR bump that required no existing plugin to change.-- A plugin declares `built_against = (MAJOR, MINOR)` as a literal.-- Compatibility: `plugin.major == core.major AND plugin.minor <= core.minor`. Same major ⇒ no breaking change since the plugin was built; `minor <=` core ⇒ every capability the plugin relies on exists in the installed core. Reject on a major mismatch, **loud**, naming both versions and the remediation (rebuild against / upgrade to plugin API `N.x`) — that is the visible break #13 wants. (`PluginApiVersionMismatchError` already carries `targets_api` + `supported_api`; the message just needs to speak semver and name the fix.)-- Re-framed: `#1000` is a MAJOR bump `2.x → 3.0`; `pipelex-mistralai-workflows` and `pipelex-temporal` each declare `(3, 0)`; a stale third-party plugin pinned to `(2, x)` is cleanly rejected at boot with "rebuild against pipelex plugin API 3.x" instead of crashing at dispatch with a `TypeError`.--Optional refinement if needed later: let a plugin also declare a maximum supported major, so a plugin can advertise forward-compat across a range rather than a single built-against point. Start with built-against-min; add the range only if a real consumer needs it.--## Scope when this is picked up--- `pipelex/plugins/contract.py` — version constant → a `MAJOR.MINOR` pair (or a small `PluginApiVersion` value object) + the compatibility helper. Update the "bump this only on a breaking change" comment to define MAJOR vs MINOR.-- `pipelex/plugins/discovery.py:100` — gate calls the compatibility helper instead of `!=`; record the plugin's *declared* version, not core's.-- `pipelex/plugins/exceptions.py` — `PluginApiVersionMismatchError` message speaks semver and names the remediation.-- **Plugin authoring convention** — the real behavioral change: built-in and external plugins stop doing `targets_api = PLUGIN_API_VERSION` and pin a literal built-against version. Without this, no gate (semver or otherwise) has teeth. This is also a conformance-test opportunity: assert no plugin imports the constant for its declared version.-- Docs — `docs/under-the-hood/orchestrator-plugins.md` (the SPI/versioning section) and README #13.--## Status--Not started. Design task, deferred — no urgency while nothing is shipped, but it should land **before the first out-of-tree third-party plugin is published against a released pipelex**, since that is the first moment the gate is asked to do its job for real.diff --git a/wip/plugins/resume-and-verify-state.md b/wip/plugins/resume-and-verify-state.mddeleted file mode 100644index 73450d7e1..000000000--- a/wip/plugins/resume-and-verify-state.md+++ /dev/null@@ -1,63 +0,0 @@-# Resume & verify state — orchestrator-agnostic runner effort--Paste-ready orientation for picking up this multi-repo effort in a fresh session. It tells a new agent **what is true**, **how to verify it against the repos** (don't trust this doc over `git`), and **what's next**. Keep it current when the effort's state changes (it's a companion to `TODOS.md`, not a replacement).--The workspace is multi-repo under `/Users/lchoquel/repos/Pipelex/` (`_plugins` is the `pipelex` core worktree).-------## 1. Read the trackers first (source of truth)--- [`TODOS.md`](../../TODOS.md) — the execution tracker (Phases A–F + a "Follow-on extensions" section). Read the **Status** block, **Follow-on extensions**, and the **Cold-start primer**.-- [`orchestrator-agnostic-runner-and-flavors.md`](orchestrator-agnostic-runner-and-flavors.md) — the decision-locked plan (the *why/how*; locked decisions D1/D2/F1/F2/F3 — don't re-litigate).-- [`execute-per-request-mode-deferred.md`](execute-per-request-mode-deferred.md) — deferred/flagged items from the two follow-ons. **Read before touching `/execute`, `/start`, or the OpenAPI generation.**--## 2. Expected state — verify each, flag any mismatch--```-for r in _plugins pipelex-api pipelex-temporal; do d=/Users/lchoquel/repos/Pipelex/$r; \- echo "== $r =="; git -C "$d" branch --show-current; git -C "$d" log --oneline -1; \- git -C "$d" status -sb | head -1; echo "unpushed: $(git -C "$d" log --oneline @{u}..HEAD | wc -l)"; done-```--Expected (the exact tips drift as docs land; the load-bearing checks are *branch*, *clean tree*, *0 unpushed*):--- **`_plugins`** (core, `Pipelex/pipelex`): branch `feature/Orchestrator-dispatched-validate`, clean, 0 unpushed (tip at/after `db4194a98`).-- **`pipelex-api`**: branch `feature/Execute-per-request-mode`, clean, 0 unpushed (tip `191c900`). Stack: `a39841e` (Phase C) → `72c0efc` (validate) → `191c900` (execute).-- **`pipelex-temporal`**: branch `feature/Orchestrator-dispatched-validate`, clean, 0 unpushed (tip `459e04d`).--PR #27 (the `/execute` follow-on) still merge-ready:--```-cd /Users/lchoquel/repos/Pipelex/pipelex-api && \- gh pr view 27 --json state,mergeable,baseRefName,headRefName && \- (gh pr checks 27 | grep -viE 'pass|skipping' || echo '(CI all green)')-```--Expected: `OPEN`, `MERGEABLE`, base `feature/Orchestrator-dispatched-validate`, CI all green, 0 open review threads.--Optional gate sanity (proves `pipelex-api` builds against the git-pinned core):--```-cd /Users/lchoquel/repos/Pipelex/pipelex-api && make agent-test # silent on success-```--## 3. What's DONE--- **Phases A + B + C** — `pipelex-api` is the orchestrator-agnostic base (MAJOR GATE 2 / "THE gate"). Committed + pushed.-- **Follow-on 1** — orchestrator-dispatched `/validate` (reverses/extends F2; new per-call `BundleValidatorRegistry` seam). Done across all 3 repos.-- **Follow-on 2** — per-request `execution_mode` on `/execute` (extends F1). PR #27, merge-ready, **NOT merged** (user merges).--## 4. What's NEXT — confirm go/no-go before starting--**Phase D** = `pipelex-api-hosted` Temporal flavor (MAJOR GATE 3): flavored base image (`FROM pipelex/pipelex-api` + `pipelex-temporal` via `git+ssh`), config migration (`[plugins] boot_orchestrator = "temporal"`, `[api] execution_mode`, connection tree → `temporal_{env}.toml`). See `TODOS.md` → "Phase D" + its checkpoint. **This is where outward-facing / deploy-breaking work begins — confirm with the user before any actual deploy.**--## 5. Open threads to keep in mind (not blocking)--- PR #27 and the validate branch are **stacked and unmerged to `dev`**; merging is the user's call.-- `pipelex-api`'s `pipelex` dep is pinned to a **core git SHA** (`51ff09417`) so CI resolves (the editable `../_plugins` path can't resolve on a CI runner). Both the SHA pin and the validate-branch stacking flip to PyPI `==` pins **at release**.-- A pre-existing **`/start` DIRECT-path resource leak** is flagged (deferred) in `execute-per-request-mode-deferred.md` — fix it on its own surface, not as part of Phase D.-------**On resume:** run §2, then give a one-paragraph status confirmation (or a list of mismatches), and wait for the user to pick the next task.diff --git a/wip/plugins/temporal-as-plugin.md b/wip/plugins/temporal-as-plugin.mddeleted file mode 100644index 9f7332cf6..000000000--- a/wip/plugins/temporal-as-plugin.md+++ /dev/null@@ -1,90 +0,0 @@-# Temporal as a Pipelex plugin--**Status:** assessment / proposed plan (not started)-**Scope:** the concrete, seam-by-seam extraction of the Temporal integration into a first-class plugin — wired through the shared orchestrator seam rather than named by string from core. An independent release cadence becomes a *possible consequence* of this, not the goal.--> This is the **Temporal-specific** detail of one orchestrator. The shared mechanism — the `Plugin` protocol, entry-point discovery, the execution-mode → orchestrator registry, the published orchestrator SPI, and the activation-model menu of hooks — lives in [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md), where Temporal and `pipelex-mistralai-workflows` are the two worked instances. The **driver** half (inference SDK wrappers) is in [`inference-backends-as-plugins.md`](inference-backends-as-plugins.md); cross-cutting best practices are in [`README.md`](README.md). Read the category doc first; this one assumes it.--## Verdict--Highly separable — the runtime is already ~90% decoupled. All Temporal code lives in `pipelex/temporal/` (plus its tests under `tests/{unit,integration}/pipelex/temporal/`). The dependency direction is one-way: `pipelex/temporal/` imports *from* core, and core almost never names `pipelex.temporal` back. The runtime already injects the Temporal implementations through the hub (`set_content_generator`, `set_pipe_router`, `set_pipe_run`, plus `temporal_hub.set_task_manager`) behind existing protocols, and nearly every reference is a **lazy** import gated on `if get_config().temporal.is_enabled:`. `temporalio` is already an opt-in extra, and `config_temporal.py` is deliberately importable *without* the SDK (a `TYPE_CHECKING` / `Any` placeholder stands in for `RetryPolicy`).--So this is **not** a refactor of the Temporal code. It is about (1) defining a plugin seam in core, and (2) inverting the handful of places where core still names `pipelex.temporal` so they go through that seam instead.--## How Temporal uses the orchestrator seam--Temporal is the **heaviest** instance of the orchestrator category — it exercises nearly every hook the shared `Plugin` protocol offers (Mistral, the other instance, uses only a subset; that contrast is what keeps the protocol generic — see the category doc's "activation models" section). What Temporal needs:--- **`register_hub_implementations(hub, config)`** — inject `ContentGeneratorInWorkflow`, `TemporalPipeRouter`, `TemporalPipeRun`, `TemporalTaskManager` when its config slice is enabled. This is the **boot-global** activation model; Temporal is the only current backend that needs it (inside a Temporal worker the content generator must dispatch each operation as an activity).-- **`register_orchestrators(registry)`** — own the `TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET` orchestrators the bridge dispatches to (the **per-call** activation model, shared with Mistral).-- **`register_cli_commands(app)`** — add the `worker` and `setup-temporal-namespace` Typer commands.-- **`register_config_section(...)`** — contribute (or validate) its `temporal` config slice. (See config decision below — this hook is only load-bearing if the schema moves out of core.)-- **lifecycle** — `setup()` / `teardown()` so the worker/task-manager teardown currently inlined in `pipelex.py` moves behind the plugin.--## What's already in our favour--- **One package, one direction.** Everything Temporal-specific is under `pipelex/temporal/` (`tprl_pipe/` router+run workflows, `tprl_content_generation/` activities, `codec/`, `temporal_data_converter.py`, `temporal_task_manager.py`, `config_temporal.py`, `exceptions.py`, `worker_cli.py`). Core → temporal references are the only thing to cut.-- **Hub injection seams exist.** `PipelexHub` already accepts swappable implementations behind protocols (`set_content_generator`, `set_pipe_router`, `set_pipe_run`); `temporal_hub.set_task_manager` covers the task manager. The Temporal variants (`ContentGeneratorInWorkflow`, `TemporalPipeRouter`, `TemporalPipeRun`, `TemporalTaskManager`) all implement those protocols.-- **Protocols exist.** `ContentGeneratorProtocol`, `PipeRouterProtocol`, `PipeRunProtocol` already abstract the swap points.-- **Opt-in extra.** `pyproject.toml`: `temporal = ["temporalio==1.24.0", "aiohttp>=3.14.0"]`. The SDK is never a hard dependency of base.-- **SDK-free config.** `config_temporal.py` imports no `temporalio` at runtime (`if TYPE_CHECKING: from temporalio.common import RetryPolicy` / `else: RetryPolicy = Any`), so the config schema is importable on installs that skipped the extra.-- **SDK-free capability detection.** `reporting/reporting_manager.py` detects "am I inside a Temporal activity?" via `sys.modules.get("temporalio.activity")` — no import, so it never pulls the SDK onto the boot hot path. This pattern survives extraction unchanged.--## The complete list of seams to cut--| # | Site | Current coupling | Hard or lazy |-|---|------|------------------|--------------|-| 1 | `system/configuration/configs.py` | `from pipelex.temporal.config_temporal import Temporal` (top import) + `temporal: Temporal` field on `PipelexConfig` | **HARD** (the only non-lazy import of the package) |-| 2 | `pipelex.py` boot | content generator (`ContentGeneratorInWorkflowFactory`), task manager (`Tasks` + `temporal_hub` + `TemporalTaskManager`), pipe router (`make_temporal_pipe_router`), pipe run (`make_temporal_pipe_run`) — all gated on `get_config().temporal.is_enabled`, injected via the hub | lazy |-| 3 | `pipelex.py` teardown | `temporal_hub` reset + `TemporalTaskManager.teardown()`, gated on a stored handle | lazy |-| 4 | `runtime_bridge/bridge.py` | `_run_temporal_blocking` / `_run_temporal_fire_and_forget` (lazy imports of `temporal.exceptions` + `make_temporal_pipe_run`) + `_require_pipelex_temporal_extra()` availability check | lazy |-| 5 | `cli/_cli.py` | top imports of `worker_cmd` / `setup_temporal_namespace_cmd` + their `app.command(...)` registration | **HARD** |-| 6 | `cli/commands/worker_cmd.py` | lazy `from pipelex.temporal.temporal_hub import get_task_manager` + `temporal_enabled=True` boot flag | lazy (rides with the command, moves wholesale) |-| 7 | `reporting/reporting_manager.py` | `sys.modules.get("temporalio.activity")` — introspection only, never imports | already decoupled — leave as-is |--Only **#1 and #5 are hard imports.** Everything else is already behind a lazy/gated seam and just needs to call through a registry instead of a literal module path.--## Recommended approach: invert Temporal's seams onto the shared registry--The generic plumbing — the `Plugin` protocol, entry-point discovery, the execution-mode → orchestrator registry, and the published orchestrator SPI — is defined once in [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md) (Phase 0 there). It does **not** belong to Temporal. This section assumes that seam exists and covers only what's Temporal-specific.--1. **Invert seams #2–#6 onto the seam.** Replace the four `if is_enabled: from pipelex.temporal...` boot blocks and the matching teardown with `register_hub_implementations` / lifecycle hooks. The bridge's run-mode dispatch (`TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`) becomes an orchestrator lookup the plugin populated, so `bridge.py` no longer names `pipelex.temporal` at all. CLI registration in `_cli.py` becomes plugin-driven `app.command(...)` instead of two hard imports.--2. **Decide the config-field cut line (#1) — the one real decision.** Two options:- - **(Recommended) Keep the `Temporal` config schema in open `pipelex`** and move only the *implementation*. The schema is pure data with no `temporalio` runtime dependency; the genuine logic (workflows, activities, worker, converter, codec) is what moves. Lowest churn, keeps the root config statically typed, makes seam #1 a no-op. The `register_config_section` hook then stays dormant for Temporal.- - **Move config out too** for a truly zero-`pipelex.temporal` open repo: replace the typed `temporal: Temporal` field with a pluggable section the plugin parses its own slice of. Cleaner boundary, but loses static typing on that subtree and adds validation plumbing — this is the only thing that makes `register_config_section` load-bearing.--3. **Repackage as a real plugin dist.** Move `pipelex/temporal/` and its tests into `pipelex-temporal`, depending on `pipelex` + `temporalio`, declaring the `pipelex.plugins` entry point. Downstream consumers (`pipelex-worker`, `pipelex-api-hosted`) pin `pipelex-temporal` instead of `pipelex[temporal]`. The base `pipelex` `temporal` extra is dropped (or reduced to just `temporalio` for anyone integrating against the protocols without our implementation). **This is the step that makes an independent release cadence trivial** — but it's optional: the plugin can equally well stay in-repo as long as it's wired through the seam.--## Open question to settle before coding--- **Does the config schema (`config_temporal.py`) stay open/in-core or move into the plugin?** This drives whether seam #1 is a no-op (schema stays, only implementation moves) or requires a pluggable-config mechanism. Recommendation above is to keep it in core.--## The data converter / codec--`temporal_data_converter.py` + `codec/` (incl. `codec_server_cli.py`, `storage_payload_codec.py`) is the most logic-heavy piece, but it is self-contained and Temporal-specific — it moves wholesale with the package. It registers through Temporal's own client/worker config, not through any core seam, so it adds no new coupling to cut.--## Distribution mechanics--- New package `pipelex-temporal`, depends on `pipelex` + `temporalio`, declares `[project.entry-points."pipelex.plugins"]`. Published wherever fits (PyPI or `git+ssh`) — the seam is identical either way.-- The `temporal` pytest marker and the `--temporal-server` CLI option plumbing (in `conftest.py`) relocate with the tests.-- Downstream pins flip from `pipelex[temporal]==X` to `pipelex-temporal==Y` (which itself pins a compatible `pipelex`).-- The base `pipelex` `temporal` extra is dropped or reduced to just `temporalio` for protocol-level integrators.--## Effort & risk--- **Low-to-moderate.** The hard work (protocol-based injection, lazy gating, SDK-free config, SDK-free activity detection) already landed with the runtime-bridge work. The remaining work is mostly mechanical: build the plugin/registry seam, rewire the small number of call sites, make CLI + run-mode registration dynamic, lift the package + tests.-- **Watch-items:**- - The config-field decision above.- - The data converter / codec is logic-heavy but self-contained.- - The Temporal test suite (unit + integration) moves wholesale, with the marker and the `--temporal-server` option.- - `reporting_manager.py`'s `sys.modules` check must keep working without importing `pipelex.temporal` (it already does).- - Don't build the seam to Temporal's shape. Temporal exercises every hook (boot-global hub swap + per-call modes + CLI + config), which makes it tempting to over-fit. `pipelex-mistralai-workflows` uses only per-call mode registration and no boot swap — it's the counter-example that keeps the protocol a menu of hooks. See [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md).--## Suggested phasing--Assumes the shared seam from [`orchestrators-as-plugins.md`](orchestrators-as-plugins.md) Phase 0 (Plugin protocol, entry-point discovery, execution-mode → orchestrator registry, SPI) already exists.--- **Phase 0 — decide config cut line.** Settle the open question. _Checkpoint: cut line agreed before any code moves._-- **Phase 1 — wire Temporal through the seam, in-tree.** Convert seams #2–#6: hub injection → `register_hub_implementations`; run-mode dispatch → registered handlers; CLI → `register_cli_commands`; teardown → lifecycle hook. No behaviour change (Temporal still in-repo, now registered through the seam instead of hard/lazy imports). _Checkpoint: full suite green with Temporal still in-repo but wired through the seam — proves it before extraction._-- **Phase 2 — extract.** Move `pipelex/temporal/` + tests into `pipelex-temporal`, wire its entry point, flip downstream pins, drop/reduce the base extra. _Checkpoint: base `pipelex` has zero `pipelex.temporal` references; `pipelex-temporal` test suite green against the published base._diff --git a/wip/plugins/temporal-config-out-of-core.md b/wip/plugins/temporal-config-out-of-core.mddeleted file mode 100644index ec32a7b69..000000000--- a/wip/plugins/temporal-config-out-of-core.md+++ /dev/null@@ -1,137 +0,0 @@-# Move Temporal config out of core (fix the core→`temporalio` type-check coupling)--**Status:** **Phases 1 & 2 DONE + verified green (Checkpoint 1 + Checkpoint 2) + COMMITTED (not pushed). Phase 3 (re-merge) deferred to a fresh session.** Core commit `eaa671d10` on `refactor/Plugins-3` (worktree `_plugins`); pipelex-temporal commit `187e846` on `refactor/own-temporal-config` (branched off `main`). **Priority:** ship-blocker for the plugin-system effort — it falsifies the effort's headline invariant ("core names no integration — not by import, not by string — anywhere").--> **As-built summary (see the "As-built" section at the bottom for detail).** The `temporalio` type-coupling is gone from core: `config_temporal.py` deleted, no `temporal` field on `PipelexConfig`, core `make agent-check` (pyright + mypy) clean, `make tb` / `make agent-test` green, `make gep` + `make check-config-sync` green. `pipelex-temporal` now owns the rich config (`config_temporal.py` + `temporal.toml` + `load_temporal_config` + `temporal_hub` cache) and gates on the generic core `plugins.boot_orchestrator` selector; its `make agent-check` + `make agent-test` are green against the editable new core. Both CLIs smoke clean (`--temporal/--no-temporal` → `--orchestrator`).--## The defect--`pipelex/system/configuration/config_temporal.py` lives in **core** but type-imports `from temporalio.common import RetryPolicy` (under `TYPE_CHECKING`, with a runtime `RetryPolicy = Any` placeholder). Core declares **no** `temporalio` dependency anywhere — not runtime, not an extra, not `dev`. So a clean type-check cannot resolve the import:--```-config_temporal.py:15 - error: Import "temporalio.common" could not be resolved (reportMissingImports)-… 13 pyright errors total, + the equivalent mypy import-not-found-```--It only ever passed because both the `_plugins` and `_workflows` venvs carried a **stale** `temporalio 1.24.0` left over from before the Phase 5 Temporal externalization. `make li` (lock + `uv sync --all-extras`) removes it and `make check` fails identically in `_plugins` itself. The merge of `refactor/Plugins-3` into `feature/mistralai-2x-bump` surfaced it because that merge correctly **drops the `temporal` extra** (public-repo rule), and the subsequent clean sync removed `temporalio`.--### Root cause--The plan's **D7** ("Temporal config stays typed in core") plus **Step 0b** (which *relocated* the Temporal config schema *into* core to break the `configs.py → pipelex.temporal` import cycle). That broke the cycle by importing Temporal's whole schema into core instead of inverting the dependency. Now that Temporal is an external plugin (`pipelex-temporal`), its config schema has no business in core.--### Rejected alternative--Adding `temporalio` to core's `dev` extra so the type-only import resolves: re-couples core's dev/CI environment to a SDK we just externalized and leaves the schema in core. Perpetuates exactly the coupling this effort exists to remove. Rejected.--## Decision (locked with the user)--**Move the Temporal config out of core entirely**, mirroring how `backends.toml` / `routing_profiles.toml` already work (separate `.pipelex/` TOML, resolved via `config_manager.resolve_config_file(...)`, parsed by their own subsystem at setup — *not* fields on `PipelexConfig`).--**Activation gate = generic core gate** (chosen over keeping a `--temporal` flag or dropping the flag): replace the temporal-specific `--temporal/--no-temporal` + `temporal_enabled` + `config.temporal.is_enabled` plumbing with a backend-agnostic selector. Core names **no** orchestrator; the plugin matches on its own name.--## Target architecture--The key property: the rich Temporal schema (the one with the `temporalio` import) **never loads in core or in `register`**. It loads only inside the boot thunks and `orchestrator.run` — all in `pipelex-temporal`, at runtime, where `temporalio` is a real dependency.--- **Generic gate in core config:** `PluginsConfig` gains `boot_orchestrator: str | None` (Optional → `= None` in the class def per repo rule; absent from `pipelex.toml`). It means "boot *this process* under the orchestrator plugin of this name" (`None`/any non-matching name = in-process default). This is plugin-system wiring, legitimately core's concern (core owns the orchestrator registry + boot slots) and carries no integration name.-- **`register` gates generically:** the Temporal plugin's `register` claims the boot slots iff `registrar.config.plugins.boot_orchestrator == self.name`. No temporal config read, no I/O, no `temporalio`. The TEMPORAL_* orchestrators are still added unconditionally (they need no config).-- **Rich config self-loads in the plugin:** `pipelex-temporal` ships its own default `temporal.toml`, reads user overrides from `.pipelex/temporal.toml` (layered project/global via `config_manager.resolve_config_file("temporal.toml")`), deep-merges, and parses into its `Temporal` schema — inside the boot thunks (apply-point, I/O fine) and `orchestrator.run` (runtime). The `temporalio` type-import travels with the schema into `pipelex-temporal`.-- **CLI:** `--temporal/--no-temporal` → `--orchestrator TEXT`; `temporal_enabled: bool | None` → `boot_orchestrator: str | None` through `cli_factory` → `Pipelex.setup()`. The override site (`pipelex.py:206-207`) sets `config.plugins.boot_orchestrator` instead of mutating `config.temporal.is_enabled`. `--orchestrator temporal` claims temporal's slots; `--orchestrator direct` (or anything ≠ a boot-orchestrator plugin name) → no claim → in-process. No sentinel needed: the gate is a name match.--## Core footprint (small)--- `configs.py:14` import of `Temporal` + `configs.py:257` `temporal: Temporal` field → removed.-- `pipelex.py:206-207` — the only core code reading `config.temporal` (the override) → rewired to `boot_orchestrator`.-- `--temporal/--no-temporal` flags threaded through: `cli/cli_factory.py`, `cli/commands/validate/{pipe,bundle,method}_cmd.py` + `_validate_core.py`, `cli/commands/run/bundle_cmd.py`, `pipelex.py` setup signatures.-- `[temporal]` TOML blocks in `pipelex/pipelex.toml`, `.pipelex/pipelex.toml`, `.pipelex/pipelex_override.toml`, **and** the kit templates under `pipelex/kit/configs/` (they seed `~/.pipelex`; `make check-config-sync` enforces `.pipelex ↔ kit` parity, so both sides move together). `tracing_config.temporal_dynamodb` is the separate tracing axis and **stays**.-- `config_temporal.py` (the 760-line schema) + the two exceptions it owns (`TemporalConfigError`, `WorkerTaskQueueUnknownError` in `pipelex/system/configuration/exceptions.py`) → moved to `pipelex-temporal`. `pipelex-temporal` already consumes the schema from core (its `temporal_connect`/`temporal_tasks`/`task_manager`/`temporal_task_manager`/`temporal_manager`/`namespace_check` import `pipelex.system.configuration.config_temporal`), so those imports become local.--## Plan (phased, cross-repo)--### Phase 1 — `pipelex-temporal` owns its config (additive, no core change yet)--- Add `config_temporal.py` to `pipelex_temporal/` (copy from core; the `temporalio` type-import now sits where `temporalio` is a real dep). Move `TemporalConfigError` + `WorkerTaskQueueUnknownError` into `pipelex_temporal`'s own `exceptions.py`.-- Add `pipelex_temporal/temporal.toml` = the default config (the relocated `[temporal]` block), shipped in the wheel.-- Add a config loader (`load_temporal_config() -> Temporal`) using `config_manager.resolve_config_file("temporal.toml")` + the packaged defaults, deep-merged.-- Repoint `pipelex-temporal`'s internal imports to its own `config_temporal` module.-- Keep the plugin reading core config for the gate *temporarily* (still `is_enabled`) so this phase stays green standalone.--🛑 **Checkpoint 1** — pipelex-temporal `make agent-check` + `make agent-test` green with its own schema (core still has its copy + field; nothing broken).--### Phase 2 — core sheds Temporal config + the generic gate (the breaking, atomic change)--- `PluginsConfig` += `boot_orchestrator: str | None = None`.-- Remove `temporal: Temporal` field + the `config_temporal` import from `configs.py`.-- Delete core `config_temporal.py` + the two exceptions from `pipelex/system/configuration/exceptions.py`.-- Remove every `[temporal]` block from core TOMLs (base + `.pipelex/*` + kit configs); keep `tracing_config.temporal_dynamodb`. Run `make check-config-sync`.-- Rewire CLI: `--temporal/--no-temporal` → `--orchestrator`; `temporal_enabled` → `boot_orchestrator` through `cli_factory` + `Pipelex.setup()`; the `pipelex.py` override site sets `config.plugins.boot_orchestrator`.-- Update the Temporal plugin `register` gate → `registrar.config.plugins.boot_orchestrator == self.name`; boot thunks + orchestrators self-load the rich config via the Phase-1 loader.-- Regenerate error pages (`make gep`) — the two relocated exceptions leave core's error catalog.--🛑 **Checkpoint 2** — **core `make check` pyright/mypy CLEAN (the `temporalio` import is gone from core), `make tb` green, `make agent-test` green**; `pipelex-temporal` `make agent-check` + `make agent-test` green (editable core). This is the gate that proves the defect fixed.--### Phase 3 — re-merge + re-sync--- Re-merge `refactor/Plugins-3` → `feature/mistralai-2x-bump` (worktree `_workflows`); the only expected conflict is `CHANGELOG.md` (union, `[Unreleased]` on top). `uv sync --all-extras` in `_workflows`. `make tb` + `make agent-check` + `make agent-test`.-- Then resume the original task (make `pipelex-mistralai-workflows` a discoverable `pipelex.plugins` entry-point plugin) against the corrected base.--## Risks / watch-items--- **`extra="forbid"` is unforgiving:** a single missed `[temporal]` key in any loaded TOML hard-fails boot. The TOML sweep (base + `.pipelex/*` + kit + tests) must be exhaustive; `make tb` is the fast catch.-- **Cross-repo ordering:** `pipelex-temporal` must have its own schema (Phase 1) before core deletes its copy (Phase 2), or the editable `pipelex-temporal` build breaks. Verify both green together under the editable pin before committing either.-- **Tests in core that assert `--temporal` / `config.temporal`:** update to `--orchestrator` / `boot_orchestrator`. Tests that reference the Temporal *schema* should already live in `pipelex-temporal`.--## Deferred (out of scope, follow-ups)--- **Tracing naming axis:** `TracingBackend.TEMPORAL_DYNAMODB`, `TemporalDynamoDBTracingConfig`, `tracing_config.temporal_dynamodb` are string/naming couplings (no `temporalio` import, don't break the build). Whether a fully-externalized Temporal should still have core's tracing system name a `temporal_dynamodb` backend is a separate question — leave for later.-- **Exact `--orchestrator` vocabulary** for force-in-process (`direct` vs `none`) — settled: the flag takes any plugin name; `--orchestrator temporal` boots Temporal, omitting it (or any non-boot-orchestrator name) runs in-process. No sentinel keyword is reserved.-- **Whether other (inference) plugin config should also leave `PipelexConfig`** — user explicitly scoped this pass to Temporal only.--## As-built (Phases 1 & 2 complete, uncommitted)--### What landed--**`pipelex-temporal` now owns the rich Temporal config** (new files):--- `pipelex_temporal/config_temporal.py` — the 760-line schema, relocated from core. The `temporalio.common.RetryPolicy` type-import now sits where `temporalio` is a real dependency. `is_enabled` dropped from the `Temporal` model (the gate moved to core).-- `pipelex_temporal/temporal.toml` — the packaged default (the old core `[temporal]` block, prefix-stripped to root tables, `is_enabled` removed). Shipped in the wheel; loaded as the base layer.-- `pipelex_temporal/temporal_config_loader.py` — `load_temporal_config()`: packaged default deep-merged with an optional `.pipelex/temporal.toml` override (project→global via `config_manager.resolve_config_file`). Mirrors `backends.toml` / `routing_profiles.toml`.-- `pipelex_temporal/temporal_activation.py` — `is_temporal_boot_active()` + `TEMPORAL_PLUGIN_NAME = "temporal"`. The runtime gate (`config.plugins.boot_orchestrator == "temporal"`), replacing the old `config.temporal.is_enabled` guard.-- `temporal_hub` caches the loaded `Temporal` (lazy-load + `set_temporal_config` for tests, cleared by `reset()`). All ~130 runtime `get_config().temporal.X` reads → `get_temporal_config().X`. Exceptions `TemporalConfigError` / `WorkerTaskQueueUnknownError` moved into `pipelex_temporal/exceptions.py`.-- Plugin gate flipped to `registrar.config.plugins.boot_orchestrator == self.name`; `worker_cli` / `worker_cmd` / `setup_namespace_cmd` / `codec_server_cli` boot with `Pipelex.make(boot_orchestrator=TEMPORAL_PLUGIN_NAME)`. The old `worker_cli` `is_enabled` force-on block is gone (redundant under the explicit boot gate).--**Core (`pipelex`) shed everything Temporal-config**:--- `PluginsConfig += boot_orchestrator: str | None = None` (generic gate; absent from `pipelex.toml`, set programmatically).-- Deleted `pipelex/system/configuration/config_temporal.py` **and** `pipelex/system/configuration/exceptions.py` (its only importer was `config_temporal`); removed `temporal: Temporal` from `PipelexConfig`.-- CLI: `--temporal/--no-temporal` (`bool|None`) → `--orchestrator` (`str|None`); `temporal_enabled` → `boot_orchestrator` through `cli_factory` → `Pipelex.make`/`setup`; the `pipelex.py` override sets `config.plugins.boot_orchestrator`.-- Removed every `[temporal]` block from `pipelex/pipelex.toml`, `.pipelex/pipelex.toml`, `.pipelex/pipelex_override.toml`, `pipelex/kit/configs/pipelex.toml`; `tracing_config.temporal_dynamodb` stays. `make check-config-sync` green.-- `make gep` regenerated: removed the 2 relocated core exceptions **and** 9 pre-existing stale `pipelex-temporal` flow-error pages (left over from the Phase-5 externalization) — core's error catalog now names no Temporal error.--### Deviations from the written plan (deliberate)--- **Phase 1 did more than "repoint imports."** Repointing only the schema *imports* to the local module while still reading `get_config().temporal` would create nominal-type conflicts (core `WorkerScope` vs local `WorkerScope`). So Phase 1 also rewired all rich reads to `get_temporal_config()` (local). The gate/guards stayed on core `is_enabled` through Phase 1 (faithful copy keeps `is_enabled` in the local schema); `is_enabled` was dropped in Phase 2 with the gate flip. Checkpoint 1 (pipelex-temporal green, core untouched) held.-- **Phase 2 = core cut-over (Step B) + pipelex-temporal gate flip (Step C), landed together.** Core green on its own; the editable pin means pipelex-temporal's gate flip had to land with it. Both verified green together at Checkpoint 2.--### Local-environment migration (one-off, done)--The config loader also merges the **global** `~/.pipelex/pipelex.toml`, which was seeded from an old kit and still carried `[temporal]` → `extra="forbid"` boot failure. Backed up to `~/.pipelex/pipelex.toml.pre-temporal-cut.bak` and removed the stale block. CI is unaffected (fresh global is kit-seeded from the now-clean kit). **Any developer/deployment upgrading past this change must drop `[temporal]` from their `~/.pipelex/pipelex.toml`** (breaking change, no back-compat per repo policy).--### Headline invariant — verified--Core has **no** `from temporalio... import` anywhere and **no** `temporal` config field. Remaining `temporal` mentions are all out of scope: the intentional `sys.modules.get("temporalio.activity")` *sniff* in `reporting_manager` (a string lookup that explicitly avoids importing — the crash-free-without-temporalio pattern), the execution-mode enum's `requires_pipelex_temporal` mode vocabulary, prose comments/docstrings, and the deferred `temporal_dynamodb` tracing axis. None is a type-check coupling.--### Gates (all green)--- core: `make agent-check` (pyright 0 / mypy 0 / keyword-only / plxt), `make tb`, `make agent-test` (exit 0), `make gep`, `make check-config-sync`; `pipelex run pipe --help` / `validate pipe --help` show `--orchestrator`.-- pipelex-temporal: `make agent-check` (pyright 0 / mypy 0), `make agent-test` (unit + integration time-skipping) against editable new core; `pipelex-temporal --help` smokes.--### Committed (not pushed)--- Core: `eaa671d10` on `refactor/Plugins-3` (`_plugins`) — the doc update lands in the immediate follow-up commit.-- pipelex-temporal: `187e846` on `refactor/own-temporal-config` (branched off `main`).--### Phase 3 — pickup for a fresh session (NOT started)--Re-merge `refactor/Plugins-3` → `feature/mistralai-2x-bump` (worktree `_workflows`); the only expected conflict is `CHANGELOG.md` (union, `[Unreleased]` on top). `uv sync --all-extras` in `_workflows`, then `make tb` + `make agent-check` + `make agent-test`. The base is now the corrected post-temporal-cut tree, so the clean sync that previously surfaced the defect (dropping `temporalio`) will pass. Then resume the original task: make `pipelex-mistralai-workflows` a discoverable `pipelex.plugins` entry-point plugin.-- **Docs follow-up (deferred).** Updated `docs/under-the-hood/orchestrator-plugins.md` (gate + module paths). Still stale and not touched this pass: `docs/under-the-hood/pipe-routing-and-execution.md` and `docs/distributed-execution/task-routing.md` still show `temporal.is_enabled` and `[temporal.worker_config.*]` / `[temporal.queue_options.*]` TOML examples. Those config tables now live at **root** in `pipelex-temporal`'s `temporal.toml` (no `[temporal.]` prefix), so the examples need rewriting — and the distributed-execution config docs arguably belong in the `pipelex-temporal` repo now. Left as a separate docs pass (involves a relocation decision).diff --git a/wip/plugins/validate-dispatch-requires-boot-active.md b/wip/plugins/validate-dispatch-requires-boot-active.mddeleted file mode 100644index 09cf0fd28..000000000--- a/wip/plugins/validate-dispatch-requires-boot-active.md+++ /dev/null@@ -1,34 +0,0 @@-# Deferred: dispatched `/validate` effectively requires `boot_orchestrator == "temporal"`--**Status:** observation surfaced during Phase V1 of `orchestrator-dispatched-validate.md`. Not a bug introduced by this work — a pre-existing property of the reused dispatch machinery. Deferred, not fixed.--## What the plan says--The plan's "Per-call, presence-based, NOT boot-gated" decision (and its resolution of old open question #5) states:--> `dispatch_dry_validate` needs only a reachable Temporal client + a worker with the validate workflow registered — **not** a runner booted-as-Temporal.--## What the code actually does--`TemporalBundleValidator.validate_bundles` reuses `dispatch_dry_validate` as-is. That calls `WorkflowExecutorFactory().create_executor(...).execute_workflow(...)`, and `WorkflowExecutor.temporal_client()` (`pipelex_temporal/tprl/workflow_caller.py`) opens with:--```python-if not is_temporal_boot_active():- raise AsyncExecutionNotEnabledError.with_default_message()-```--`is_temporal_boot_active()` is `config.plugins.boot_orchestrator == "temporal"`. So the executor refuses to connect unless **this process booted under Temporal** — i.e. dispatched `/validate` does in practice require `boot_orchestrator == "temporal"`, contrary to the plan's decision #5.--## Why this is consistent (and why it's only an observation)--The **registration** is correctly presence-based / unconditional — the validator is contributed under both temporal modes regardless of `boot_orchestrator`, exactly like the Temporal run orchestrators. The runtime boot-active requirement is **identical** for the run orchestrators: `TemporalBlockingOrchestrator.run` → `make_temporal_pipe_run().run` → the same `WorkflowExecutor.temporal_client()` boot-active guard. So validate dispatch is no more boot-coupled than pipe-run dispatch already is; the validator faithfully mirrors the established orchestrator behavior.--The plan's decision #5 was about **the seam not being boot-*gated* at registration** (true — no hub slot is claimed, no `boot_orchestrator` branch in `register`). The phrase "not a runner booted-as-Temporal" overreaches for the *runtime* connect path, which the reused `dispatch_dry_validate`/executor enforces.--## Why not fix it here--Relaxing the `is_temporal_boot_active()` guard so a non-Temporal-booted runner can still open a Temporal client for dispatch is a change to the **shared executor** that governs *all* Temporal dispatch (run + validate), not validate alone. It is out of scope for this feature, would broaden blast radius well beyond `/validate`, and is only meaningful once the hosted runner's boot story (does the agnostic runner boot as temporal, or dispatch per-request without booting?) is settled in the parent effort's Phase D. Until then this is a no-op for correctness: a Temporal-flavor runner boots as Temporal anyway.--## If revisited--Decide the hosted runner's boot model first (Phase D). If a single agnostic runner must dispatch *some* requests to Temporal without booting as Temporal, the fix belongs in `WorkflowExecutor.temporal_client()` (gate on client reachability, not `boot_orchestrator`) and must be validated against the run-dispatch path too — then update the plan's decision #5 wording to match.diff --git a/wip/plugins/validate-start-mode-dispatch-duplication.md b/wip/plugins/validate-start-mode-dispatch-duplication.mddeleted file mode 100644index f39b9ff98..000000000--- a/wip/plugins/validate-start-mode-dispatch-duplication.md+++ /dev/null@@ -1,34 +0,0 @@-# Deferred: `start` / `validate_verdict` mode-dispatch duplication (pipelex-api)--Surfaced during the `/code-review` of the `feature/Orchestrator-dispatched-validate` staged diff in `pipelex-api`. Deferred (not fixed) as a design tradeoff, not a clear win.--## The observation--`ApiRunner.start` and `ApiRunner.validate_verdict` (both in `api/routes/pipelex/pipeline.py`) now share the same three-line dispatch shape:--1. `execution_mode = resolve_execution_mode(requested_execution_mode, config=get_api_config())` — resolve the effective mode FIRST, so a forbidden per-request override is refused with a 403 before any library load / registry lookup.-2. `arm = get_<orchestrator|bundle_validator>_registry().get_optional(mode=execution_mode)`.-3. `if arm is None: raise Missing<Orchestrator|BundleValidator>Error(mode=execution_mode)`.--The "resolve mode FIRST … refused (403) here, before any … load" comment is also near-duplicated between the two methods.--The two sites differ only in the registry getter (`get_orchestrator_registry` vs `get_bundle_validator_registry`) and the error class (`MissingOrchestratorError` vs `MissingBundleValidatorError`).--## Why deferred (not fixed)--A helper such as `_resolve_arm(registry_getter, missing_error_cls)` would collapse the two sites to one, but:--- It is only ~4 lines × 2, and the two registries / error types are genuinely distinct seams.-- The helper would take a registry-getter callable and an exception class as parameters — that is *added* indirection and surface area, the opposite of removing complexity.-- The user's review brief for this change was explicit: clean, solid software, no speculative abstractions, and "in case of any doubt, do NOT fix." This is a judgment call, not an unambiguous win.--So the code is left as-is.--## When this becomes worth doing--Revisit the extraction only if a concrete pull arrives:--- A third surface acquires the same resolve-then-dispatch shape (the contract would then live in three places).-- The override-policy ordering guarantee changes (e.g. the "403 before any load" point moves), so a maintainer would otherwise have to remember to edit every site and keep the parallel comments in sync.--At that point a single small helper carries its weight; until then the duplication is the lower-churn, lower-surface choice.diff --git a/wip/plugins/validate-verdict-subclass-parity-divergence.md b/wip/plugins/validate-verdict-subclass-parity-divergence.mddeleted file mode 100644index 3330f9c6d..000000000--- a/wip/plugins/validate-verdict-subclass-parity-divergence.md+++ /dev/null@@ -1,50 +0,0 @@-# `/validate` verdict/fault discrimination: latent subclass parity divergence between the two arms--Status: deferred observation from the `/code-review --fix` of the `TemporalBundleValidator` staged diff (`pipelex-temporal`). **Not a present bug — no code change applied.**--## The two arms discriminate "verdict vs fault" differently--`/validate` produces a verdict-as-value: an invalid bundle's `ValidateBundleError` is *returned* as a structured `ErrorReport` (the invalid arm), while a genuine infra fault *propagates* (a no-verdict 5xx). The two registered bundle validators implement that split with different predicates:--- **Direct (in-process) arm** — `pipelex/pipeline/direct_bundle_validator.py`:-- ```python- except ValidateBundleError as exc:- return exc.to_error_report()- ```-- An `except ValidateBundleError` clause catches `ValidateBundleError` **and any subclass of it**.--- **Temporal (dispatched) arm** — `pipelex_temporal/temporal_bundle_validator.py`:-- ```python- recovered_report = workflow_error.to_error_report()- if recovered_report.error_type != ValidateBundleError.__name__:- raise- return recovered_report- ```-- The verdict crosses the worker boundary as a structured `ErrorReport` whose `error_type` is the raised exception's `type(self).__name__` (see `pipelex/base_exceptions.py`, `to_error_report` sets `error_type=type(self).__name__`). The arm recognizes a verdict by **exact string equality** with `"ValidateBundleError"`. A subclass would carry the *subclass* name, so this predicate would **not** match it.--## Why it is not a bug today--The validation cascade in `pipelex/pipeline/validate_bundle.py` always raises the concrete `ValidateBundleError` class itself for every content verdict, and there are **no `ValidateBundleError` subclasses anywhere in the tree** (`grep -rn "class .*(ValidateBundleError)"` is empty). So for every reachable invalid-bundle verdict today, `error_type == "ValidateBundleError"` and both arms agree: return the verdict.--## The latent divergence--If someone later introduces a `ValidateBundleError` subclass and raises it from the validation cascade:--- the **direct arm** would still return it as a verdict (its `except` catches subclasses), but-- the **Temporal arm** would `raise` it as a no-verdict 5xx (its name-equality rejects the subclass name).--Same bundle, same intended verdict, two different HTTP outcomes depending on which orchestrator served the request. That is a cross-backend parity break, of exactly the kind the dispatched-`/validate` design is meant to avoid (both backends must answer identically for the same bundle).--## Why deferred, not fixed--- It is latent: zero subclasses exist, so there is no failing input today.-- The string-equality predicate is forced by the wire boundary — the Temporal arm only has the recovered `ErrorReport` (`error_type` string), not the live exception type, so it *cannot* use `isinstance`/`except` the way the in-process arm does without a richer recovery contract.-- The clean fix is not local to the reviewed file: it would mean teaching the recovery layer to carry a "is-this-a-ValidateBundleError-or-subclass" signal (e.g. an `is_validate_verdict` flag on the recovered report, or a verdict marker class hierarchy the recovery preserves), then having **both** arms branch on that signal — a seam change touching `pipelex` and `pipelex-temporal`, well outside the staged diff.--## If/when a `ValidateBundleError` subclass is introduced--Re-open this. The minimal correct fix is to make verdict-recognition subclass-aware on the wire side — e.g. have `convert_pipelex_errors` / `recover_error_report` stamp a stable "validate verdict" discriminator that does not depend on the exact leaf class name, and have both bundle validators branch on that discriminator instead of on `ValidateBundleError.__name__` (Temporal) / `except ValidateBundleError` (direct). That keeps the two arms provably in lockstep regardless of the concrete exception class.diff --git a/wip/recursivity/review-followups.md b/wip/recursivity/review-followups.mddeleted file mode 100644index 889d36425..000000000--- a/wip/recursivity/review-followups.md+++ /dev/null@@ -1,21 +0,0 @@-# Deferred polish — additive multi-file library (from pre-landing /review)--> **Status: DEFERRED (P3 only).** Surfaced by the pre-landing `/review` on `feature/Support-recursive-design` (PR #970) via independent adversarial + testing + maintainability reviewers. None are bugs; the review verdict was **ship**. Both PR bots (greptile 5/5, cubic clean) and the full suite were green. These are recorded so they aren't lost — pick up only if touching this area again. Not worth re-triggering a CI/bot cycle on their own.--## Documentation / clarity nits--- **`domain_metadata_merge.py` summary line slightly overclaims "order-independent."** The merge is order-independent for the omission cases (a membership-only sibling never overrides a declared value), but two files declaring *different* non-empty values is genuinely first-seen-wins (it always warns). The docstring body already lists this case correctly ("both non-empty and different -> keep the first, warn"); only the one-line summary reads as if every case were order-independent. If editing this file, soften the summary to name the warned-conflict exception.--- **`library_crate_factory.py` `_reconcile_pipe_collision` tie-break drops a losing signature's `description` silently.** When two `PipeSignature`s for the same pipe have matching contracts but different `description`s, the deterministic `min(...)` tie-break keeps one and discards the other's description with no warning. This is harmless in practice — signature descriptions are advisory and a concrete definition's description wins outright once it arrives — but it's a silent drop. If touching reconciliation, add a one-line comment noting header descriptions are advisory (or warn, only if a real need appears; a warning here would be over-engineering for the DAG-forward-declared-by-multiple-callers edge).--- **`library_crate_factory.py` `or ""` coercion asymmetry.** The crate-factory merge coerces `merge_domain_metadata_field(...) or ""` because `DomainBlueprint.description` is non-optional, while the `DomainLibrary` merge of the same field needs no coercion because `Domain.description` is `str | None`. Behavior is correct; a one-line comment at the `or ""` would save a future reader the double-take.--## Test-coverage gap (low confidence)--- **`validate_bundle.py` `LibraryLoadingError` structured-forwarding branch is exercised but not asserted with a *populated* payload.** The `except LibraryError` arm forwards `blueprint_validation_errors` / `pipe_concept_validation_errors` when the error is a `LibraryLoadingError`. The generic-`LibraryError` `else` branch is well covered (undeclared concept/pipe). The `isinstance(LibraryLoadingError)` branch is reached only by the cycle-detection test, whose error carries *empty* lists — so no test proves a populated structured payload survives onto the resulting `ValidateBundleError` (which the CLI renders). A regression that collapsed the two branches would not be caught. If revisiting: one integration test driving a `LibraryLoadingError` with non-empty validation-error lists through `validate_bundle` and asserting the forwarded fields are populated (not `None`).--## Explicitly NOT worth doing (recorded so they aren't re-raised)--- `contracts_match` input-name divergence (same arity, same specs, different variable name): transitively covered by the dict comparison; a dedicated case would be over-testing.-- The `contract_match` `match is None` defensive return for a cross-package spec in `inputs`/`output`: documented as an impossible-in-blueprints input handled defensively; do not add a test for it.-- The inline undeclared-ref message in `concept_reference_validation.py`: fine as a one-off; only centralize next to `duplicate_ref_msg` if a third such message appears.diff --git a/wip/runtime-bridge/README.md b/wip/runtime-bridge/README.mddeleted file mode 100644index c1fd82732..000000000--- a/wip/runtime-bridge/README.md+++ /dev/null@@ -1,91 +0,0 @@-# runtime-bridge — review follow-ups--Triage of the SWE-agent review comments on **PR #959** (`feat: extract framework-agnostic Pipelex runtime bridge`, branch `feature/Runtime-bridge-extraction`). The PR extracts `pipelex/runtime_bridge/` — a host-runtime-agnostic surface (`run_pipe_via_bridge`) that lets Mistral Workflows / raw Temporal / future plugins invoke Pipelex pipes from inside their own activities.--Two review bots left 12 unresolved threads (**greptile-apps** + **cubic-dev-ai**). All 12 were verified against the code. **All are now resolved in the working tree** — the only thing left is replying to / resolving the GitHub threads (see "Remaining" at the bottom).--## 1. Design forks — all resolved--Each was a **confirmed** finding whose resolution was a genuine design choice. Each doc retains its full triage as the record of why.--- **[direct-mode-nested-router-leak.md](direct-mode-nested-router-leak.md)** — ✅ **RESOLVED (Option A).** (greptile P1, + cubic P2 enabler) In `DIRECT` mode, nested controller sub-pipes resolved the *hub default* router and leaked to Temporal when a worker has `[temporal] is_enabled`. Fixed with a `scoped_pipe_router` helper in `hub.py` wrapping `_run_direct` (also resolves cubic's hub.py:615/teardown-clobber). The observer sub-question that gated the fork is written up in [`../observer-and-telemetry/observer-telemetry-posthog.md`](../observer-and-telemetry/observer-telemetry-posthog.md).-- **[graph-context-temporal-contract.md](graph-context-temporal-contract.md)** — ✅ **RESOLVED (Option A).** (greptile P2) `graph_context` was threaded into the `PipeJob` for *all* modes despite the DIRECT-only docstring, and `WfPipeRouter` consumes it. Fixed: `run_pipe_via_bridge` now nulls `graph_context` for the Temporal modes (honoring the contract); docstring corrected. + regression test.-- **[trace-flush-blocking-io.md](trace-flush-blocking-io.md)** — ✅ **RESOLVED (false positive / by-design).** (cubic P2) `flush_trace_events_to_backend` is `async` with blocking boto3/file I/O — but it runs in an *activity*, which is exactly where Temporal allows blocking I/O. No code change. The optional `asyncio.to_thread` throughput-offload note is kept on file for if/when Temporal ships and we profile worker contention.-- **[bridge-error-name-collision.md](bridge-error-name-collision.md)** — ✅ **RESOLVED (renamed).** The near-mirror `PipelexBridgeRuntimeError` (leaf) vs `PipelexRuntimeBridgeError` (base) was renamed: the leaf is now **`PipelexBridgeDispatchError`**. Updated the class, `bridge.py` raises, `test_validation.py`, regenerated `docs/errors/` (old slug removed, new written; no mistral churn), and the `TODOS.md` prose.--## 2. Mechanical fixes — all applied--- **Boot race** — ✅ `pipelex/runtime_bridge/bootstrap.py`. Double-checked locking with a module-level `threading.Lock` (re-check inside the lock). Regression test: `tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py` (threaded barrier; asserts make runs once, no raise). **Follow-up (PR #966 review):** this closed the write-write race but not a lock-free read of a half-built singleton mid-`setup()` — **now also fixed** via Approach B (the bridge gates on `Pipelex.is_fully_booted()`; an instance-level `is_ready` is published only at the end of `make()`). See §3 / [`bootstrap-half-built-singleton-race.md`](bootstrap-half-built-singleton-race.md).-- **Exceptions caller-facing flag** — ✅ `pipelex/runtime_bridge/exceptions.py`. Added `_authors_caller_facing_message = True` to `MissingPipelexTemporalExtraError` and `MissingMistralWorkflowsPluginError` so the pip-install hint survives STRICT disclosure. Regression test: `test_exceptions_disclosure.py`.-- **streaming.mdx missing import** — ✅ added `import asyncio` to the example's import block.-- **observability.mdx frontmatter id** — ✅ `id: observabilities` → `id: observability`.-- **your-first-workflow.mdx grammar** — ✅ "It coordinate" → "It coordinates".-- **your-first-workflow.mdx broken image** — ✅ removed the missing-PNG image, kept the instruction text.-- **durable-execution.md overgeneralization** — ✅ rewrote the line to distinguish the two durable backends (the `[temporal] is_enabled` flag is the Pipelex-on-Temporal backend only; the Mistral Workflows path runs pipes via the runtime bridge inside Workflows activities).--## 3. PR #966 pre-landing review--A later `/review` pass on PR #966 (the bridge + the trace-event read hardening). One deferred item plus four cheap hardening fixes applied to the working tree.--- **Bootstrap half-built-singleton race** — ✅ **applied (P1)**, see [`bootstrap-half-built-singleton-race.md`](bootstrap-half-built-singleton-race.md). The #959 lock closed the write-write race; this closes the remaining lock-free read of a mid-`setup()` singleton. Approach B (instance-level `is_ready`, gated via `Pipelex.is_fully_booted()`, published only at the end of `make()`) — chosen over a module-global flag so the existing `teardown_if_needed()` lifecycle auto-resets it. Regression tests added (mid-setup window + setup-failure re-boot).-- **`library_id` widened** — ✅ `bridge.py` + `primitives/submitter_hydration.py`. Dropped the `uuid4().hex[:8]` truncation (32 bits) to full hex. `open_library` silently *reuses* a colliding id (`library_manager.py:148-149`), so a 32-bit collision between two overlapping calls would share and prematurely tear down one library; full hex matches the collision-safe Temporal path (full `workflow_id`).-- **`act_assemble_graph` comment corrected** — ✅ the lifted primitive catches only a specific exception tuple, so the old "swallows every failure / no error ever crosses the boundary" comment was stale; reworded to say expected failures degrade to None while programming bugs deliberately propagate (pinned by `test_propagates_unexpected_keyerror`).-- **Scoped-library open guarded** — ✅ `bridge.py::_scoped_library_for_crate` now opens the library inside the `try` (with a `library_opened` flag) so a throw between open and yield can't leak the manager entry — matches the sibling `rehydrate_pipe_output_with_crate`.-- **Input decode wrapped** — ✅ `bridge.py` decodes `library_crate_dump` / `delivery_assignment_dump` once, translating raw `pydantic.ValidationError` into `PipelexBridgeDispatchError` so a malformed dump no longer escapes the entry point as a non-`PipelexError`; also removes the double-decode of the delivery dump. `_validate_input` now takes the decoded assignment (its unit tests updated).--`make agent-check` (pyright 0/0/0, mypy clean) and `tests/unit/pipelex/runtime_bridge/` green after these.--## Verification--`make agent-check` (ruff, plxt, pyright 0/0/0, mypy) and `make agent-test` (full suite) both green.--## 4. PR #969 review rounds (live PR)--PR #969 is the live PR (#959 and #966 closed). Successive bot rounds (greptile / cubic / codex) each re-examined the prior round's fixes plus new surface. Round-4 dispositions:--- **Keyed tracers through the bridge** (`bridge.py:179`, greptile P1 + cubic P2) — ✅ **RESOLVED (refuted as unreachable; no code change).** Full triage + the deferred structural option in [`bridge-keyed-tracer-unsupported.md`](bridge-keyed-tracer-unsupported.md). Correct abstract analysis but unreachable: keyed tracers (`tracer_key != graph_id`) are only opened by `wf_pipe_router`, which never calls the bridge; non-DIRECT modes null `trace_context`; on every reachable path `lookup_key == graph_id`. `lookup_key` is kept (round-3 fix; `graph_id` would re-break `close_tracer`) with a tightened comment documenting the unsupported keyed-tracer contract. A loud boundary guard was tried and **backed out** as over-engineering (defended a non-occurring config, was only partial, and stacked churn on the round-3 change defending the same case).-- **Temporal blocking `workflow_id`** (`bridge.py:342`, codex P2 + cubic P1) — ✅ **RESOLVED (real fix).** Blocking reported the bare `pipeline_run_id`; the workflow is actually started with `make_workflow_id(...)`, which prefixes in non-NORMAL run modes (`ut-`/`ci-`/`cc-`/`cct-`). The bridge now reports `make_workflow_id(...)` — the same id fire-and-forget returns from `start()`, single source of truth. In `RunMode.NORMAL` (prod) the prefix is empty, so behavior is unchanged there. Wiring test: `tests/unit/pipelex/runtime_bridge/test_temporal_blocking_workflow_id.py`; prefix table covered by `tests/unit/pipelex/temporal/test_workflow_id_construction.py`.-- **trace_flush pre-dedup** (`trace_flush.py:28`, cubic P2) — ➖ **False positive.** Dedup is a **read-side** contract: `EventLogProtocol.read_events` returns events deduplicated by `(workflow_id, writer_id, event_type, sequence)`, and the DynamoDB backend is write-idempotent via PK+SK ("natural deduplication for Temporal replay re-emissions"). NDJSON appends on retry but `read_events` collapses duplicates on assembly. The proposed pre-dedup key also **omits `pipeline_run_id`**, so it would wrongly collapse identical sequences across different runs. No code change. (Related async/blocking-IO note already on file: [`trace-flush-blocking-io.md`](trace-flush-blocking-io.md).)-- **Mistral-workflows docs** (`index.md:29` cubic P2, `choosing-a-backend.md:8` cubic P3) — ✅ **RESOLVED (accuracy).** "Automatic activity retries — each leaf operator retries independently" overclaimed: in `direct` mode the whole pipe runs in one host activity. Reworded to be mode-aware (`direct` retries as a unit; `mistral_native` retries leaves independently). The "replay" guarantee line was reworded to state replay re-runs the workflow from history reusing stored activity results, not re-executing completed activities.--Round-5 dispositions:--- **Self-review back-out** — the round-4 keyed-tracer guard was removed (see the keyed-tracer bullet above; details in [`bridge-keyed-tracer-unsupported.md`](bridge-keyed-tracer-unsupported.md)).-- **Gateway `transport_max_retries`** (`gateway_completions_factory.py` / `gateway_responses_factory.py`, cubic P2) — ✅ **RESOLVED (real fix).** The gateway LLM client factories built `openai.AsyncOpenAI` without `max_retries`, silently ignoring a `transport_max_retries` override on the primary inference path (the near-identical `Portkey*Factory` siblings pass it). Wired it into both. The gateway extract/img-gen path uses `AsyncPortkey` (no `max_retries` param — verified), so the `automatic-retries.md` doc was corrected to scope the claim precisely.--Round-6 dispositions:--- **Gateway wiring test missing** (`gateway_completions_factory.py:128`, cubic P2) — ✅ **RESOLVED (real gap, my round-5 omission).** `test_transport_retry_wiring.py` has a dedicated test per factory; the round-5 gateway fix shipped without the matching cases. Added `test_gateway_completions_factory_passes_max_retries` / `test_gateway_responses_factory_passes_max_retries`. (A wiring test is exactly what would have caught the round-5 gateway bug — so it earns its keep here, contra the general "don't test trivial lib-config swaps" rule.)-- **Durable-execution retry claim** (`durable-execution.md:14`, cubic P2) — ✅ **RESOLVED (accuracy).** Same flavor as the round-4 docs fix: per-leaf retry is the Temporal model; Mistral `direct` mode retries the pipe as a unit. Qualified the bullet.-- **choosing-a-backend table** (`choosing-a-backend.md:17`, cubic P3 ×2) — ✅ **RESOLVED.** Added the `.pipelex/pipelex.toml` config-path for `[temporal] is_enabled`, and reworded the Mistral cell to "invoke … through the runtime bridge inside your Mistral Workflows activity".-- **`PipelexBridgeDispatchError` caller-facing flag** (`exceptions.py:22`, cubic P2) — ➖ **Declined.** The class is raised for BOTH caller-facing validation (malformed `library_crate_dump`/`delivery_assignment_dump`) AND execution wrapping that embeds downstream `{exc}` internals (sites 321/341/369/452). Marking the whole class `_authors_caller_facing_message=True` would expose execution internals under STRICT — the exact over-broad-disclosure leak greptile caught in round 3 for `PipelexSetupError`. STRICT is never the active render mode in prod (default VERBOSE — grep-confirmed no prod call site passes STRICT), so the validation detail already shows. If STRICT is ever adopted at a boundary AND validation detail is wanted there, the right fix is splitting the class, not flagging the mixed one — deferred.-- **Bootstrap concurrent-boot race** (`bootstrap.py:39`, cubic P2 conf 8) — ✅ **RESOLVED (reachable cases) + deferred (residual).** Full triage in [`bridge-bootstrap-external-race.md`](bridge-bootstrap-external-race.md). The reachable case (two `ensure_pipelex_booted` threads) is lock-serialized and tested — `make()` runs inside `_boot_lock`, so no second `make()` races. The genuine residual is a *concurrent external* `Pipelex.make()` (not under `_boot_lock`) mid-setup; narrow/non-standard (hosts boot before serving). Tightened the docstring to state the real contract; the bounded-wait fix is recorded as deferred (poll loop + TOCTOU catch = over-engineering for a non-occurring setup).--Round-7 disposition:--- **Tracing assembly drops DynamoDB construction failures** (`tracing_assembly.py:98`, cubic P2 conf 8) — ✅ **RESOLVED (real regression from this PR's refactor).** Pre-PR `assemble_tracing` caught botocore directly, covering BOTH `make_event_log()` construction and `read_events()`. The refactor pushed the botocore→`EventLogReadError` conversion into `DynamoDBEventLog.read_events` only, but `__init__` eagerly builds the boto3 client — so a construction-time botocore failure (e.g. a misconfigured region) escaped the catch and would abort a completed run instead of degrading. Fixed: new `EventLogSetupError(EventLogError)`; `__init__` translates construction botocore → `EventLogSetupError`; `assemble_tracing` catches the `EventLogError` base (read + setup). Regression tests both sides; error docs regenerated.--## 5. PR #969 `/review` final pass (gstack)--The finalization `/review` pass (specialist army + adversarial Claude/Codex) on the full diff against `dev`. 0 critical. Findings triaged:--- **WorkflowExecutionError escapes the dispatch contract** (`bridge.py:339,367`, red-team conf 8) — **decided with the user.** `_run_temporal_blocking` / `_run_temporal_fire_and_forget` catch `(PipeRunError, PipeJobError, PipeRouterError, PipeExecutionError, PipelineExecutionError)`, but `TemporalPipeRun.run()`/`.start()` actually raise `WorkflowExecutionError` (a `TemporalFlowError(PipelexError)`, not in the tuple) on a pipe failure (`workflow_caller.py:133/176`). So a Temporal-mode failure escapes `run_pipe_via_bridge` un-wrapped — asymmetric with DIRECT/mistral, and the catch tuple is dead for the dominant failure mode. The catch's presence shows uniform-wrapping was the intent. (Disposition recorded inline at finalization.)-- **`pipe_classification` docstring falsely claimed `pipelex.temporal` uses it** — ✅ **RESOLVED (accuracy).** Zero in-tree callers; the helpers are forward-looking `MISTRAL_NATIVE` plumbing (sibling host package, not landed). Docstring corrected to say so; module kept (it is in-scope bridge-side plumbing per `TODOS.md`).-- **`_serialize_completed_output(pipe_job=...)` unused param** (`bridge.py:383`, maintainability conf 7) — kept-for-future YAGNI; decided with the user at finalization.-- **`LibraryManager._pipe_source_map` not per-call scoped + unlocked** (red-team conf 7/6) — ➖ **Deferred.** Pre-existing manager behaviour, observability-only impact, Temporal not in prod. Full mechanism + fix shape in [`library-manager-pipe-source-map-concurrency.md`](library-manager-pipe-source-map-concurrency.md).-- **`output_dict` python-mode dump** (`bridge.py:80`, api-contract conf 8) — ➖ **Declined.** `dump_for_temporal()` uses `model_dump(serialize_as_any=True)`, but the model is JSON-safe via pydantic's own `model_dump(mode="json")`/`model_dump_json()` (and the kajson Temporal converter); only the naive `json.dumps(output.model_dump())` misuse fails, loudly. Usage caveat, not a defect. `extra="forbid"` skew, `trace_flush` async-IO, `delivery` pass-through, and the scoped-library 3× duplication were all re-confirmed as previously-considered/by-design.-- **Test gaps** (testing conf 5–7) — `_decode_*` error path, `trace_flush` guards, `_run_mistral_native` success path are untested "lake" gaps; noted, added per the user's finalization call.--## Open deferred follow-ups (not bugs)--Carried forward, each recorded so review tooling doesn't re-raise them as new findings:--- **[execution-mode-property-helpers-unused.md](execution-mode-property-helpers-unused.md)** — the three `PipelexExecutionMode` `@property` helpers are tested but not yet consumed by `bridge.py` (which inlines the equivalent checks). Wire them in or remove; no correctness impact.-- **[library-manager-pipe-source-map-concurrency.md](library-manager-pipe-source-map-concurrency.md)** — pre-existing `LibraryManager._pipe_source_map` is manager-level, not per-call scoped. Observability-only, Temporal not in prod.-- **[bridge-bootstrap-external-race.md](bridge-bootstrap-external-race.md)** — residual race against a *concurrent external* `Pipelex.make()` mid-setup; narrow/non-standard, bounded-wait fix deferred.-- **[cost-reporting-test-cleanup-followups.md](cost-reporting-test-cleanup-followups.md)** — §3 (open, PR #977 re-review): the cross-worker cost assert (`assert_cross_worker_cost.py`) selects its CSV by globbing the *global* reports dir (lexicographically-latest), not the run being asserted, so a concurrent/earlier run's report can be compared against the current NDJSON. Dev/test helper only.-- **[distributed-cost-reporting-test-coverage.md](distributed-cost-reporting-test-coverage.md)** — audit (from a full `/temporal-e2e-validate` Mode 2 run) of cost-reporting test coverage in split-worker mode. No automated test asserts real-inference distributed cost numbers; the real-provider *capture* branch is asserted nowhere; cross-child sum and img-gen/extract/search usage are uncovered; plus two real risks (R2 retry over-count — documented/accepted; zero-token silent emit). Code-grounded map + prioritized test additions by layer, written as the input for a follow-up planning session.--## Remaining--Code/docs are done. What's left is the GitHub side, per the `/review-pr-agents` skill: reply on each PR #969 thread (`✅ Fixed` / `➖ False positive`) and resolve them. No threads need to stay open — every fork was decided and applied.diff --git a/wip/runtime-bridge/bootstrap-half-built-singleton-race.md b/wip/runtime-bridge/bootstrap-half-built-singleton-race.mddeleted file mode 100644index a5489526d..000000000--- a/wip/runtime-bridge/bootstrap-half-built-singleton-race.md+++ /dev/null@@ -1,92 +0,0 @@-# `ensure_pipelex_booted` can publish a half-built Pipelex singleton under concurrent first-boot--**Status:** ✅ **APPLIED** — Approach B (instance-level `is_ready`). Real P1 race in new code, now fixed: `ensure_pipelex_booted` gates on `Pipelex.is_fully_booted()`, and the singleton is published as ready only at the very end of `make()` (after `setup()` + the optional `validate_model_deck()`). See [As applied](#as-applied) below.-**Source:** PR #966 pre-landing review (`/review`), adversarial pass + source verification.-**Severity:** real, but **scoped**: only bites when two threads race the *first* boot from inside activities; the recommended "boot once at the worker entry-point" path never hits it. Fails with a loud `RuntimeError`, not silent corruption.-**Relation to the #959 "Boot race" fix:** this **extends** it. The double-checked `threading.Lock` added in #959 (`bootstrap.py`, see README §2) correctly serializes the two *writers* so the loser no longer raises `PipelexSetupError("already initialized")`. It does **not** close the second hole below: a *reader* taking the lock-free fast path while the winner is mid-`setup()`.--## The finding--`ensure_pipelex_booted` (`pipelex/runtime_bridge/bootstrap.py:34`) has a lock-free fast path:--```python-if Pipelex.get_optional_instance() is not None: # line 34 — no lock- return-with _boot_lock:- if Pipelex.get_optional_instance() is None:- Pipelex.make(config_overrides=config_overrides)-```--The premise that makes this unsafe is that **the singleton becomes publicly visible before it is configured**:--1. `MetaSingleton.__call__` registers the instance the moment `__init__` returns:- - `pipelex/system/registries/singleton.py:17` — `cls.instances[cls] = super().__call__(*args, **kwargs)`-2. `Pipelex.make()` constructs **then** runs the slow `setup()`:- - `pipelex/pipelex.py:577` — `pipelex_instance = cls(config_overrides=config_overrides)` (registers now; `__init__:107` even swaps in a fresh **empty** hub via `set_pipelex_hub`)- - `pipelex/pipelex.py:579` — `pipelex_instance.setup(...)` (this is what assigns the library manager, class registry, pipe router, pipeline manager — slow: loads libraries, validates the model deck)-3. `get_optional_instance()` reads exactly that registry:- - `pipelex/pipelex.py:611` — `return MetaSingleton.instances.get(cls)`--**Chain:** Thread A holds `_boot_lock` and is inside `make()`, past line 577 (instance registered, empty hub installed) but still inside `setup()`. Thread B arrives fresh, hits the lock-free check at `bootstrap.py:34`, sees a non-`None` instance, returns immediately, and proceeds into `run_pipe_via_bridge` → `get_required_pipe()` / `get_library_manager()` against the still-empty hub → `RuntimeError` ("LibraryManager is not initialized" or similar partial-state failure).--There is a second nasty interleaving: `make()` wraps `setup()` in `try/except BaseException` that **deletes** the instance on setup failure (`pipelex.py:601-605`). If A's `setup()` fails, B — which already returned thinking boot succeeded — is now running against an instance A just removed from the registry.--## Why this is the multi-thread scenario the module targets--`bootstrap.py:18-20` already calls out "Temporal's sync-activity thread pool / multi-thread workers" as the reason for the lock. The bridge is built for embedding (`pipelex-mistralai-workflows` calls `run_pipe_via_bridge` directly from inside an activity, with `ensure_pipelex_booted` as the documented in-activity first-boot). So concurrent first-boot across worker threads is a supported path, not a corner case — the lock proves the author intended it to be safe.--The window is wide: `setup()` does real work (library load + model-deck validation), so it is realistically open for hundreds of ms on a cold worker.--## Why the existing test does not catch it--`tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py::test_concurrent_first_calls_boot_once_without_error` starts both worker threads at the same `threading.Barrier`, so both miss the fast path (no instance yet) and both queue on `_boot_lock`. The test asserts `make_calls == 1` and no raise — it verifies the *writers* are serialized. It never has a thread arrive **fresh during another thread's `setup()` window**, which is the only way to exercise the lock-free read of a half-built instance. The `fake_make` also conflates registration and setup (it appends to `instance_holder` then `time.sleep`), but the timing never lets a third party observe the in-between state.--## Recommended fix--> **Note:** the shape sketched here (a module-global `_booted`) is **not** what shipped — see [As applied](#as-applied). The applied fix is the `is_ready`-on-instance alternative called out in the caveat, chosen against the test lifecycle. This section is kept as the record of the original sketch and its trade-off.--Gate the fast path on a "fully booted" flag flipped **only after** `make()` returns, and move the existence check inside the lock so the lock-free path can never read a half-built instance:--```python-_booted = False--def ensure_pipelex_booted(config_overrides: dict[str, Any] | None = None) -> None:- global _booted- if _booted:- return- with _boot_lock:- if _booted:- return- if Pipelex.get_optional_instance() is None:- Pipelex.make(config_overrides=config_overrides)- _booted = True-```--This preserves the documented "adopt an externally-created singleton" contract: the first bridge call finds `_booted == False`, takes the lock, sees a fully-built external instance, skips `make()`, and sets `_booted = True`. The one requirement (already implicit today) is that any external boot must *complete* before the first bridge call — adopting a singleton another thread is mid-`make()`-ing externally is inherently racy and out of scope.--### Caveat that makes this worth a deliberate decision, not a reflex fix--`_booted` is module-global, so it survives across tests. The existing concurrency test (and any test that boots/tears down Pipelex) needs a reset hook (e.g. a fixture that sets `bootstrap._booted = False`), or the flag has to live somewhere resettable. A "check the instance is fully set up" alternative (an `is_ready` flag on `Pipelex` flipped at the end of `setup()`) avoids the module global but is more invasive (touches `pipelex.py`). Pick the shape against the boot/teardown lifecycle the test suite expects — hence deferred rather than auto-applied in the review pass.--## As applied--Shipped the `is_ready`-on-`Pipelex` alternative from the caveat, **not** the module-global `_booted` sketch above. The deciding factor is exactly the test-lifecycle point the caveat raised: the suite's module-scoped autouse fixture (`tests/conftest.py`) tears Pipelex down with `Pipelex.teardown_if_needed()`, which deletes the singleton from `MetaSingleton.instances`. An instance-level flag dies with the instance, so it auto-resets between test modules — no module-global to leak, no reset fixture to add, no second source of truth that can drift from the registry.--Changes:--- `pipelex/pipelex.py` — `__init__` initializes `self.is_ready = False` (set before the metaclass registers the instance, so it always exists before the instance is observable); `make()` flips `pipelex_instance.is_ready = True` on its success tail, **after** the `try/except`, so it is set only once `setup()` and the optional `validate_model_deck()` have both succeeded and the delete-on-failure handler is behind us; new classmethod `is_fully_booted()` returns `instance is not None and instance.is_ready`.-- `pipelex/runtime_bridge/bootstrap.py` — `ensure_pipelex_booted` gates both the lock-free fast path and the in-lock re-check on `Pipelex.is_fully_booted()` instead of `get_optional_instance() is not None`. A registered-but-not-ready instance (mid-`setup()`, or about to be deleted on setup failure) is now treated as not-booted, so the reader blocks on the lock and re-checks. The "adopt an externally-created singleton" contract is preserved: a fully-booted external instance is `is_ready` → no-op.--## Tests (done)--`tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py` (all mock at the `Pipelex` classmethod boundary — never touching `MetaSingleton.instances`, so they coexist with the autouse session singleton):--- `test_reader_arriving_mid_setup_blocks_until_ready` — three-thread test: A is held inside a patched `make()` that registers an `is_ready=False` instance, then flips it ready only just before returning; B and C arrive during the window and must block on the lock until A finishes (asserted via `is_alive()` mid-window). Fails against the pre-fix bare-presence check.-- `test_setup_failure_lets_next_thread_reboot` — A's patched `make()` registers then clears + raises (mirroring delete-on-failure); B must fall through to the lock, find nothing booted, and re-boot rather than proceed against the deleted instance.-- The original `test_concurrent_first_calls_boot_once_without_error` keeps the `make_calls == 1` assertion (no regression of the #959 write-write fix).--## Related--- README §2 "Boot race — ✅ applied" (the #959 double-checked lock this extends).-- `pipelex/system/registries/singleton.py` (`MetaSingleton` — registration-at-construction).-- `pipelex/pipelex.py:577-607` (`make` construct-then-setup, with delete-on-failure).diff --git a/wip/runtime-bridge/bridge-bootstrap-external-race.md b/wip/runtime-bridge/bridge-bootstrap-external-race.mddeleted file mode 100644index 89b2f840e..000000000--- a/wip/runtime-bridge/bridge-bootstrap-external-race.md+++ /dev/null@@ -1,50 +0,0 @@-# Bridge bootstrap vs a concurrent external Pipelex.make() — narrow, documented, deferred--**Status:** ✅ RESOLVED for the reachable/standard cases (lock-serialized, tested). The residual race against a *concurrent external* boot is narrow/non-standard; the docstring now states the real contract, and the bounded-wait fix is recorded below as deferred (adding it now would be over-engineering for a config standard hosts don't produce).--**Raised by:** PR #969 round-6 review — cubic-dev-ai **P2** (confidence 8), `pipelex/runtime_bridge/bootstrap.py:39`: "`ensure_pipelex_booted()` can raise `Pipelex is already initialized` during concurrent startup when another thread is already mid-boot, instead of safely adopting/waiting for that boot."--## What's actually true--`ensure_pipelex_booted()` is:-```python-if Pipelex.is_fully_booted():- return-with _boot_lock:- if not Pipelex.is_fully_booted():- Pipelex.make(config_overrides=config_overrides)-```-`Pipelex.make()` raises `PipelexSetupError("Pipelex is already initialized")` if any instance is already registered (pipelex.py). `is_fully_booted()` is `instance is not None and instance.is_ready`; `is_ready` flips True only at the tail of `make()` (after setup + optional deck validation; a failed boot deletes the instance).--**Reachable cases are safe** — verified against the three tests in `tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py`:--- *Two `ensure_pipelex_booted` threads.* `make()` runs **inside** `with _boot_lock`, so thread A holds the lock for its entire boot. Thread B blocks on the lock and, when it finally enters, sees `is_fully_booted()==True` and skips. No second `make()`, no raise. (`test_concurrent_first_calls_boot_once_without_error`)-- *Reader arriving mid-setup.* B/C gate on `is_fully_booted()` (not bare presence), so they block on the lock through A's half-built window and never read a not-ready instance. (`test_reader_arriving_mid_setup_blocks_until_ready`)-- *Setup failure.* A registers half-built, fails, deletes the instance; B (held on the lock) then re-boots cleanly. (`test_setup_failure_lets_next_thread_reboot`)--The cubic write-up's two-`ensure_pipelex_booted`-threads scenario is **not** a real race: it assumes B can run `make()` while A is mid-`make()`, but the lock makes that impossible.--## The genuine residual (scenario B)--The lock only serializes the bridge's *own* callers. A **separate external** `Pipelex.make()` (e.g. `worker_cli.py`, `codec_server_cli.py`, a CLI factory) does **not** take `_boot_lock`. If such an external boot is mid-setup (`is_ready=False`) on one thread while the bridge's `ensure_pipelex_booted()` runs on another, the bridge sees `is_fully_booted()==False`, acquires the (uncontended) lock, calls `make()`, and hits "already initialized".--**Reachability: narrow.** Each external `make()` call site is a process entry point that boots Pipelex *before* serving work; activities (which reach the bridge) run after boot, when `is_fully_booted()` is already True and the bridge fast-paths. Hitting scenario B requires a non-standard host that boots Pipelex on one thread while *concurrently* serving bridge traffic on another. No code in the repo does this.--## Decision: document the contract, defer the bounded-wait--The docstring previously over-promised ("adopts that singleton without re-initializing") without qualifying the mid-boot window. It now states the real contract: the bridge serializes its own boot and adopts a *fully-booted* external singleton; it does **not** support racing a concurrent external `make()` that is mid-initialization — boot Pipelex before bringing up concurrent bridge callers. No code change: the failure mode is a loud raise (not a silent corruption) in a setup nothing standard produces.--## Deferred: bounded-wait adoption--If a real host ever needs to boot Pipelex concurrently with serving bridge traffic, make `ensure_pipelex_booted()` wait out a mid-flight external boot instead of colliding. Because a failed boot deletes the instance, the wait terminates:--```python-with _boot_lock:- while not Pipelex.is_fully_booted():- if Pipelex.get_optional_instance() is None:- Pipelex.make(config_overrides=config_overrides) # still needs to tolerate a TOCTOU raise vs external- else:- time.sleep(...) # external boot mid-setup: wait for is_ready, or for it to fail+disappear-```--This adds a poll loop, holds `_boot_lock` while waiting (acceptable — other bridge callers would block anyway), and must also catch the `make()` "already initialized" raise for the external-registers-between-check-and-make TOCTOU. That fragility for a non-occurring setup is why it's deferred, not done.diff --git a/wip/runtime-bridge/bridge-error-name-collision.md b/wip/runtime-bridge/bridge-error-name-collision.mddeleted file mode 100644index fe245c2dd..000000000--- a/wip/runtime-bridge/bridge-error-name-collision.md+++ /dev/null@@ -1,42 +0,0 @@-# Bridge error name collision — `PipelexRuntimeBridgeError` vs `PipelexBridgeRuntimeError`--**Status:** ✅ **RESOLVED — leaf renamed to `PipelexBridgeDispatchError`.** (Option A.) Applied: renamed the class in `exceptions.py`; updated the import + raise sites in `bridge.py` and the `pytest.raises` references in `test_validation.py`; regenerated `docs/errors/` (`pipelex-bridge-runtime-error.md` removed, `pipelex-bridge-dispatch-error.md` written, `index.md` updated — no spurious mistral-page churn); updated the `TODOS.md` prose. `make agent-check` + `make agent-test` green. Original triage retained below.--**Spotted:** during the `dev` → `feature/Runtime-bridge-extraction` merge triage. Not a merge regression and not a stale duplicate — verified both classes are live and their docs are in sync (`pipelex-dev generate-error-pages` reports `Removed: 0`, no orphans).--## The smell--`pipelex/runtime_bridge/exceptions.py` declares two classes whose names differ **only by word order**:--```python-class PipelexRuntimeBridgeError(PipelexError): # :4 — base for the whole runtime-bridge surface-class PipelexBridgeRuntimeError(PipelexRuntimeBridgeError): # :16 — concrete leaf: a bridge-dispatched pipe execution failed-```--"Runtime Bridge" vs "Bridge Runtime". They are genuinely distinct — the base also parents `MissingPipelexTemporalExtraError` and `MissingMistralWorkflowsPluginError` — but the near-mirror names are easy to misread, mistype, and mis-`except`. The leaf reads almost like its own base, which is exactly backwards. The two also generate two almost-identical doc-page slugs (`pipelex-runtime-bridge-error` / `pipelex-bridge-runtime-error`) that a reader can't tell apart at a glance.--This is a readability/API-clarity issue only: the kebab slugs are distinct (no `type_uri` collision), and nothing is functionally wrong.--## The fork--- **(A) Rename the concrete leaf** to a self-evident name decoupled from the base's word-order — e.g. `BridgeDispatchError`, `BridgePipeExecutionError`, or `BridgeRunError`. The base `PipelexRuntimeBridgeError` (the namespace error) stays. This is the recommended direction: the leaf is the one at the call sites and the one whose meaning ("a pipe dispatched through the bridge blew up") should be obvious from its name.-- **(B) Rename the base** instead (e.g. `BridgeError` as the family root) and keep the leaf — less compelling, since the base is rarely referenced directly and the family-prefix convention (`Pipelex…`) is intentional.-- **(C) Leave it.** Zero churn; the confusion persists for every future reader and `except` site.--## Recommendation--Take **(A)**. Pick a leaf name that states what failed, not where it sits in the hierarchy. Per project policy there's no back-compat to preserve, so it's a clean rename.--## Blast radius (small, contained)--- `pipelex/runtime_bridge/exceptions.py` — the class definition.-- `pipelex/runtime_bridge/bridge.py` — one import + the `raise` sites.-- `tests/unit/pipelex/runtime_bridge/test_validation.py` — the `pytest.raises(...)` references.-- `docs/errors/pipelex-bridge-runtime-error.md` — regenerated, old slug page removed (it's `<!-- pipelex:generated -->`, safe to delete).-- `TODOS.md` mentions the name in prose — update if still relevant.--## How to apply--1. Rename the class in `exceptions.py`; update the import + raises in `bridge.py` and the references in `test_validation.py`.-2. `pipelex-dev generate-error-pages` to refresh `docs/errors/` (writes the new slug page, removes the old one) and the `docs/errors/index.md` entry. ⚠️ This worktree runs `mistralai==2.4.4`; the generator's import-walk has tolerated it so far, but eyeball the diff for spurious mistral-page churn.-3. `make agent-check` + `make agent-test`.diff --git a/wip/runtime-bridge/bridge-keyed-tracer-unsupported.md b/wip/runtime-bridge/bridge-keyed-tracer-unsupported.mddeleted file mode 100644index a849d963d..000000000--- a/wip/runtime-bridge/bridge-keyed-tracer-unsupported.md+++ /dev/null@@ -1,36 +0,0 @@-# Bridge + keyed tracers (tracer_key ≠ graph_id) — unreachable, unsupported by contract--**Status:** ✅ RESOLVED (refuted as unreachable; `lookup_key` kept with a precise comment; **no guard code** — that would be over-engineering for a configuration nothing produces). Bigger-hammer structural option recorded below for if keyed tracers ever need to flow through the bridge.--**Raised by:** PR #969 round-4 review — greptile-apps **P1** (`bridge.py:179`, "Separate tracer keys") + cubic-dev-ai **P2** (same line, confidence 6).--## The finding--`build_pipe_job_from_input` derives the run id, when no explicit `pipeline_run_id` is given, as `trace_context.lookup_key` (= `tracer_key or graph_id`). The reviewers observed: when a host's `tracer_key != graph_id`, trace events are emitted under the tracer's *event* `pipeline_run_id` (which falls back to `graph_id`), **not** under `tracer_key`. But `PipeRun.run` assembles tracing by `job_metadata.pipeline_run_id` (now = `tracer_key`). So assembly would read the `tracer_key` partition, find no events, and the run would silently return an empty graph + no cost data — even though events were emitted.--This is **correct analysis** of an abstract data-flow. It is **not a reachable bug**, and the convenient "fix" is wrong — so the resolution is a tightened comment + refutation, not new code.--## Why it is not a reachable bug--Verified against the code (`pipe_run.py`, `bridge.py`, `wf_pipe_router.py`, `trace_context.py`):--- The in-process `PipeRun.run` keys **both** `close_tracer` (pipe_run.py:60) **and** `assemble_tracing_on_output` (pipe_run.py:78) off the **single** `job_metadata.pipeline_run_id` (pipe_run.py:35). One key, not two — and events are emitted under `pipeline_run_id or graph_id`.-- The bridge **nulls `trace_context` for all non-DIRECT modes** (bridge.py:133), so only DIRECT runs ever receive one.-- The **only** place that opens a keyed tracer (`tracer_key != graph_id`) is the Temporal child path `wf_pipe_router` — and it **never calls the bridge** (`grep`-confirmed: no `build_pipe_job_from_input` / `run_pipe_via_bridge` under `pipelex/temporal/`). It uses its own activity-based assembly, not `PipeRun.run`.-- Every bridge-reachable caller opens its tracer the way `pipeline_run_setup` does — `graph_id == pipeline_run_id`, `tracer_key` unset — so `lookup_key == graph_id == event-partition` and all three keys coincide.--So the divergence requires a *host integrator* to hand-build a Temporal-style keyed `TraceContext` and force it through the DIRECT bridge — which nothing does, and which the contract ("open a tracer for this pipeline run"; `graph_id` "typically the pipeline run id") does not anticipate. No internal Pipelex code reaches the bridge with a keyed trace_context.--## Why `lookup_key` is the right value (and `graph_id` is not)--The convenient counter-"fix" — switch back to `trace_context.graph_id` — is **wrong**: it re-breaks `close_tracer` for any keyed tracer (`close_tracer` needs the registration key, `lookup_key`). `lookup_key` and `graph_id` differ only for a divergent keyed tracer, which (per above) never reaches the bridge — so on every reachable path `lookup_key == graph_id` and `lookup_key` is correct for registration, close, event-emission, and assembly simultaneously.--## Decision: comment, not guard--An earlier round-4 attempt added a loud boundary guard that rejected a divergent `tracer_key` with `PipelexBridgeDispatchError`. It was **backed out** on review (this PR) as over-engineering: it defended a configuration no code path produces (only host misuse), it was only *partial* (it caught `tracer_key ≠ graph_id` but could not catch a host opening with `tracer_key=None, pipeline_run_id ≠ graph_id`, which the bridge cannot see from `graph_id` + `tracer_key` alone), and it sat on top of the round-3 `lookup_key` change that was itself defending the same unreachable case. The net of two rounds of churn over a non-occurring config was the smell.--Resolution: keep `lookup_key`, document the reachability + the unsupported keyed-tracer contract in the code comment at the derivation site, and refute the review thread with the reachability proof. No behavior change, no new code.--## Deferred: if keyed tracers ever need bridge support--The structural fix, only worth it once a real caller needs `tracer_key != graph_id` through the bridge: give `TraceContext` an explicit event-partition field (the value the host passed to `open_tracer(pipeline_run_id=...)`) so the bridge stops *inferring* it from `graph_id`/`tracer_key`. Then `JobMetadata.pipeline_run_id` (assembly/close key) and the event-partition can be carried independently end-to-end. This touches `TraceContext` across `graph/`, `temporal/`, and the in-process tracing path, so it is over-engineering for a configuration nothing currently produces — hence deferred, not done.diff --git a/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md b/wip/runtime-bridge/cost-reporting-test-cleanup-followups.mddeleted file mode 100644index 4791be30d..000000000--- a/wip/runtime-bridge/cost-reporting-test-cleanup-followups.md+++ /dev/null@@ -1,71 +0,0 @@-# Cost-reporting test additions — cleanup follow-ups--**Status:** 🧹 Optional cleanups, no correctness impact. Items #1–#2 surfaced by a code review of the distributed-cost-reporting test additions (see [`distributed-cost-reporting-test-coverage.md`](distributed-cost-reporting-test-coverage.md), "Resolution (as-built)") and are resolved. Item #3 surfaced on the v0.32.0 release PR #977 and is deferred (dev/test helper only). The reviews found **no shipped-code bugs**; these are cosmetic dead-code / weak-assertion / test-helper-correctness items.-------## 1. Dead CSV fallback + never-present NDJSON key in the skill helper ✅ DONE--**Resolution:** Verified against `pipelex/cogt/usage/cost_registry.py` and applied the fix. `INPUT_JOINED` is written into an `nb_tokens_by_category` dict only inside `complete_cost_report` (which `pop`s `INPUT` first), and the CSV is written exclusively from completed reports — so the column is always `nb_tokens_input_joined`, never `nb_tokens_input`. Raw NDJSON `UsageReportEvent`s only ever carry `input`. Dropped the `nb_tokens_input` CSV fallback and set `_INPUT_KEYS = ("input",)`. (Note: a `NB_TOKENS_INPUT = "nb_tokens_input"` enum value does exist in `llm_report.py`, but it is never emitted into a record, so the conclusion holds.)--**File:** `.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py`--Two bits of defensive code reference a data shape that never actually occurs, which misleads a reader into thinking a second format exists:--- **`_csv_token_totals` (~line 224):** `int(float(row.get("nb_tokens_input_joined") or row.get("nb_tokens_input") or 0))`. The cost-report CSV is written by `CostRegistry.save_to_csv(to_records(...))`, whose records only ever carry the column `nb_tokens_input_joined` — never `nb_tokens_input`. The `or row.get("nb_tokens_input")` branch is unreachable.-- **`_sum_tokens` / `_INPUT_KEYS = ("input", "input_joined")`:** sums an `input_joined` key that is never present in a *raw* `UsageReportEvent`. `INPUT_JOINED` is synthesized later, inside `CostRegistry.complete_cost_report` (it pops `INPUT` and renames it on a copy used only for costing). The emitted event's `nb_tokens_by_category` always has `input`, never `input_joined`, so the extra key always contributes 0.--**Why it's harmless today:** neither produces a wrong total — the CSV column genuinely matches the NDJSON `input` sum, and the missing keys add 0.--**The real (latent) risk:** if the CSV schema ever *did* emit both `nb_tokens_input` and `nb_tokens_input_joined`, the union-sum logic would silently double-count input. Worth simplifying to just the column that exists.--**Fix:** drop the `nb_tokens_input` fallback; set `_INPUT_KEYS = ("input",)`. Re-confirm the CSV column name in `pipelex/cogt/usage/cost_registry.py` (`save_to_csv` + `to_records`, and the `*TokenCostReportField` enums where `NB_TOKENS_INPUT_JOINED = "nb_tokens_input_joined"`) before editing, in case it has since changed.-------## 2. Vacuous + duplicated sentinel guard in the real-inference test--**File:** `tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py`--The test declares a local `_FAKE_RUNNER_MODEL_NAME = "split_runner_fake"` and asserts (near the end of `test_real_provider_usage_aggregates_to_nonzero_cost`):--```python-assert _FAKE_RUNNER_MODEL_NAME not in model_names, "Real inference must report a real model handle, not the fake sentinel"-```--**Problem:** this test explicitly installs `runner_act_llm_gen_text=_real_runner_act_llm_gen_text` (real inference), so the *default* fake substitute — the only thing that ever emits `"split_runner_fake"` — is never wired in. The asserted string can never appear regardless of whether real inference actually ran, so the guard can never fail (false confidence).--**Drift risk:** `"split_runner_fake"` is re-declared as a local literal instead of imported from the helper that owns it (`tests/integration/pipelex/temporal/tracing/helpers.py`, `_RUNNER_FAKE_INFERENCE_MODEL_NAME`). If the helper's sentinel is renamed, this assertion silently checks a stale string and keeps "passing" while guarding nothing.--**Fix options (pick one):**--- Drop the assertion — the adjacent `assert all(model_names)` (every usage record names a real handle) plus the non-zero-token assertions already prove real inference ran. This is the cleaner option; the sentinel-absence check adds nothing once the default substitute isn't in play.-- Or, if a guard against accidentally falling back to the fake substitute is still wanted, import `_RUNNER_FAKE_INFERENCE_MODEL_NAME` from `helpers.py` rather than re-declaring it, so a rename can't rot it.-------## 3. Global CSV selection in the cross-worker cost assert ⏭️ DEFERRED--**File:** `.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py` (`_csv_token_totals`, ~lines 54-75)--**Status:** ⏭️ Deferred — dev/test helper only (not shipped library code). Surfaced by **greptile (P2)** on the v0.32.0 release PR #977; left open on that PR for a fast-follow.--`_csv_token_totals(reports_dir)` picks the cost-report CSV to compare against with `sorted(reports_dir.glob("cost_report*.csv"))[-1]` — the lexicographically-latest file in the **global** reports dir — even though the caller passes the run-specific `--run-dir` for the NDJSON side of the comparison. If another run already wrote a later-numbered `cost_report*.csv`, the assert compares the current run's NDJSON usage against a **stale CSV from a different run**, which can fail a good run or pass a bad one.--This is the same helper as item #1, but a distinct concern: item #1 was about the *column* read inside a row (`nb_tokens_input` fallback, ~line 224); this is about *which file* is selected (~lines 54-75).--**Why deferred, not fixed:** it's a `.claude/skills/` test helper, out of scope for the docs-only v0.32.0 release-branch pass. Real enough to fix on `dev`.--**Fix:** tie CSV selection to the run being asserted — either point `--reports-dir` at a per-run subdir (e.g. `run_dir / "reports"`) and glob there, or filter `cost_report*.csv` by the run id derived from `run_dir.name`. Re-confirm where the runner writes `cost_report*.csv` relative to `--run-dir` before choosing.--## Verification--These files' tests are green as-is. After either change, re-run:--```bash-# helper (#1) — no test, just exercise it against a fresh run dir, or eyeball-# real-inference test (#2) is marked inference/llm (opt-in spend); the no-spend lane:-.venv/bin/pytest -p no:cacheprovider -q \- tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py \- tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py-```diff --git a/wip/runtime-bridge/cost-reporting-validation-prompts.md b/wip/runtime-bridge/cost-reporting-validation-prompts.mddeleted file mode 100644index 417e8d74b..000000000--- a/wip/runtime-bridge/cost-reporting-validation-prompts.md+++ /dev/null@@ -1,50 +0,0 @@-# Cost-reporting validation — prompts for `/temporal-e2e-validate`--Operator cheat-sheet for driving the distributed cost-reporting validation through the `/temporal-e2e-validate` skill. The skill's cost route is **scope-aware**: a bare cost request runs only the free deterministic arms; an explicit spend opt-in runs every arm including the live (real-money) ones. The arms (A–G) and their numeric assertions are defined in the skill's scope manifest at `.claude/skills/temporal-e2e-validate/references/mode-2-tiers.md` (Step 5b', Tier 8b). The gap audit that motivated them is the sibling [`distributed-cost-reporting-test-coverage.md`](distributed-cost-reporting-test-coverage.md).--## Prompt menu--### Full thorough run — real spend (arms A–G)--```text-/temporal-e2e-validate cost reporting full-```--Routes to Tier 8b and runs every arm: mock primary + cross-child fan-out + CSV cross-check + `--no-costs` negative gate + live LLM + live img-gen + live extract — each asserted numerically with `assert_cross_worker_cost.py`, PASS/FAIL surfaced per arm. `thorough`, `every`, and `with-spend` are aliases of `full`:--```text-/temporal-e2e-validate distributed cost — every arm-```--### Free, deterministic pass — no spend (arms A–B)--```text-/temporal-e2e-validate cost reporting-```--Mock primary + cross-child fan-out only (`--mock-inference`). The quick regression to run before committing or before paying for the live arms.--### Spend, but ask first--```text-/temporal-e2e-validate validate distributed cost on a live run-```--`live` / `all` are deliberately **not** treated as a spend opt-in (too easily incidental; `live` also collides with the default mock arm already running in LIVE mode). When that's the only signal, the agent runs the free arms and **confirms with you before** the real-money arms.--### Full run, LLM cost path only--```text-/temporal-e2e-validate cost reporting full, LLM arms only (skip img-gen/extract)-```--Use when only LLM inference is wired up in your env (see caveat below).--## Caveat — live non-LLM arms need backends--Arms F (img-gen) and G (extract) are **live-only** — `--mock-inference` can't reach them (their mock leaves raise `MockInferenceUnsupportedError`), so they're the sole proof that non-LLM usage crosses the runner fallback and aggregates into the cost report. They need real img-gen and extract backends configured (extract = Azure Doc Intel via the Pipelex Gateway). If those aren't set up, the two arms fail at the **run** step, not the assertion — and the manifest's "stop and report" rule means you'll see exactly which arm and why.--## Recommended sequence--1. `/temporal-e2e-validate cost reporting` — free, fast, confirms the capture → emit → fallback → aggregate → render plumbing.-2. `/temporal-e2e-validate cost reporting full` — spend on the live arms once the cheap ones are green.diff --git a/wip/runtime-bridge/direct-mode-nested-router-leak.md b/wip/runtime-bridge/direct-mode-nested-router-leak.mddeleted file mode 100644index 66268f133..000000000--- a/wip/runtime-bridge/direct-mode-nested-router-leak.md+++ /dev/null@@ -1,129 +0,0 @@-# DIRECT mode: nested controller sub-pipes leak to the hub default router--**Status:** ✅ **RESOLVED — Option A applied.** (was: confirmed bug, deferred pending decision)-**Source:** PR #959 review — greptile-apps (P1, `bridge.py:252-258`); the enabler is cubic-dev-ai (P2, `hub.py:615`, "prefer a `scoped_pipe_router` context manager").-**Severity:** real but **latent** — only bites when a worker runs with `[temporal] is_enabled = true`, and Temporal hasn't shipped to prod. In the common (non-Temporal) worker it was silently harmless *except* for an observer inconsistency (see the fork, now resolved).--## Resolution (Option A)--Decision: **Option A** — scope the bridge's bare `PipeRouter()` for the whole DIRECT run. The observer sub-question was investigated and cleared the way for A: the router-level observer's *only* job is best-effort PostHog telemetry, it's inert by default and only live for Gateway users, the DIRECT root pipe already emitted none, and Temporal mode already uses `ObserverNoOp`. Threading the observer in (Option B) would have required a hub `get_observer` getter that doesn't exist — re-coupling the bridge to internals. Full analysis: [`../observer-and-telemetry/observer-telemetry-posthog.md`](../observer-and-telemetry/observer-telemetry-posthog.md).--Applied:--- Added `scoped_pipe_router(pipe_router)` to `pipelex/hub.py` (mirrors `scoped_current_library`: captures the prior `_current_pipe_router` override and restores it on exit — fixes cubic's hub.py:615/teardown-clobber point too). The raw `set_pipe_router` / `teardown_current_pipe_router` pair is kept for the external `pipelex-mistralai-workflows` plugin.-- Wrapped `_run_direct` (`pipelex/runtime_bridge/bridge.py`) in `with scoped_pipe_router(direct_router):` so root + nested sub-pipes both resolve the in-process router. Kills the Temporal leak.-- Tests: `tests/unit/pipelex/pipe_run/test_scoped_pipe_router.py` (set/restore/nested/raise semantics) and `tests/unit/pipelex/runtime_bridge/test_direct_router_scoping.py` (nested dispatch resolves the scoped router, not the hub default; override torn down after the run). `make agent-check` + `make agent-test` green.--Everything below is the original triage write-up, retained as the record of why.-------## The finding--> `DIRECT` mode creates a fresh `PipeRouter`, so only the root pipe uses the bridge-local direct runner. Controller pipes dispatch their sub-pipes through `get_pipe_router()`, which resolves to the hub or context router. In a worker booted with Temporal enabled, a direct bridge run of a sequence or batch can unexpectedly send nested pipes to Temporal instead of running in-process.--## Verified behavior (code trace)--1. `_run_direct` builds a **bare** router and hands it only to `PipeRun`:- - `pipelex/runtime_bridge/bridge.py:256` — `pipe_run = PipeRun(pipe_router=PipeRouter())`-2. `PipeRun.run` uses that router **only for the root pipe**, and never installs it into the routing ContextVar:- - `pipelex/pipe_run/pipe_run.py:42` — `pipe_output = await self._pipe_router.run(pipe_job)`-3. Controllers dispatch every sub-pipe via `get_pipe_router()` — *not* the parent's router instance:- - `pipelex/pipe_controllers/sub_pipe.py:179` (batch), `:195` (condition), `:229` (normal) — all `await get_pipe_router().run(...)`-4. `get_pipe_router()` returns the ContextVar override if set, else the **hub default**:- - `pipelex/hub.py:633-637`- ```python- def get_pipe_router() -> "PipeRouterProtocol":- override = _current_pipe_router.get()- if override is not None:- return override- return get_pipelex_hub().get_required_pipe_router()- ```-5. The hub default is **Temporal-aware** when Temporal is enabled:- - `pipelex/pipelex.py:446-452`- ```python- if pipe_router:- self.pipelex_hub.set_pipe_router(pipe_router)- elif get_config().temporal.is_enabled:- self.pipelex_hub.set_pipe_router(make_temporal_pipe_router()) # TemporalPipeRouter- else:- self.pipelex_hub.set_pipe_router(PipeRouter(observer=multi_observer))- ```--**Chain:** `_run_direct` never sets `_current_pipe_router` → nested `get_pipe_router()` sees no override → returns the hub default → which is `TemporalPipeRouter` in a Temporal-enabled worker → **nested sub-pipes of a DIRECT-mode sequence/batch get dispatched to Temporal**, contradicting the whole point of DIRECT (run in-process, here, now).--This is not intentional: `set_pipe_router`'s own docstring (`pipelex/hub.py:618-619`) describes exactly this use case — *"Used by host runtimes that want controllers to dispatch sub-pipes through their own router."* DIRECT mode simply forgets to use it.--The greptile comment also notes a Mistral-native variant of the same root cause: a `set_pipe_router` override installed by a host is honored by nested pipes (via `get_pipe_router()`) but bypassed by the root pipe (which runs on the bare `PipeRouter()`), so root vs nested dispatch is inconsistent.--## Provenance — new in #959, not pre-existing in dev--Checked against `dev` (`../pipelex`, HEAD `fa15d15c` — the commit this branch was cut from, no `runtime_bridge/`). **This bug is introduced by the bridge PR.** Both ingredients are new:--- dev's `get_pipe_router()` (hub.py:612-613) has **no override layer** — it just `return get_pipelex_hub().get_required_pipe_router()`. The PR added `_current_pipe_router` + the module-level `set_pipe_router`/`teardown_current_pipe_router`.-- dev never builds a fresh bare router for a root run: the hub's `PipeRun` uses the *hub's* router (`pipelex.py:461` — `PipeRun(pipe_router=self.pipelex_hub.get_required_pipe_router())`), and nested sub-pipes resolve the *same* hub router. So in dev root and nested always agree — non-Temporal → both `PipeRouter(observer=multi_observer)` (no observer split either); Temporal-enabled → both the Temporal router. There is no "in-process root while Temporal is enabled" concept in dev.--The bridge introduces that concept (`DIRECT` = force in-process even in a Temporal worker) via the bare `PipeRouter()` in `_run_direct`, but doesn't install it into the new ContextVar — hence the leak. The **observer split** is likewise bridge-introduced.--Pre-existing look-alike (out of scope, flagged for honesty, not #1): `pipelex/pipeline/bundle_validator.py:106` builds `PipeRun(pipe_router=PipeRouter(observer=ObserverNoOp()))` — same "fresh root router, nested falls to `get_pipe_router()`" shape, but it's the validation/dry-run path, predates the bridge, and is low-risk. Worth its own look someday, separately.--## The mechanism for the fix (resolves cubic's hub.py:615 too)--There's an established pattern in the same file — `scoped_current_library` (`pipelex/hub.py:517-532`) — a context manager that captures the prior ContextVar value, sets the new one, and restores on exit:--```python-@contextmanager-def scoped_current_library(library_id: str) -> Generator[None, None, None]:- prev = _library_id.get()- _library_id.set(library_id)- try:- yield- finally:- _library_id.set(prev)-```--The router ContextVar has only the raw pair today, and the teardown clobbers any outer override:--- `pipelex/hub.py:615` `set_pipe_router(router)` → `_current_pipe_router.set(router)`-- `pipelex/hub.py:628-630` `teardown_current_pipe_router()` → `_current_pipe_router.set(None)` (unconditional — **does not restore a prior override**, cubic's P2)--**Proposed shared fix:** add `scoped_pipe_router(router)` mirroring `scoped_current_library`, then wrap the DIRECT run in it so the in-process router is the active router for the *whole* call (root + nested). Keep the raw `set_pipe_router` / `teardown_current_pipe_router` — the external `pipelex-mistralai-workflows` plugin depends on them (separate repo, can't edit here) — but prefer the scoped helper internally.--```python-# bridge.py _run_direct-direct_router = PipeRouter()-with scoped_pipe_router(direct_router):- pipe_run = PipeRun(pipe_router=direct_router)- pipe_output = await pipe_run.run(pipe_job=pipe_job, delivery_assignment=delivery_assignment)-```--## The fork — what to do about the observer--The hub default in-process router is `PipeRouter(observer=multi_observer)` (`pipelex.py:452`), but the bridge builds a bare `PipeRouter()` (`observer=None`, see `pipe_router.py:10`). So **today**, in a non-Temporal worker:--- root pipe → bridge's bare router (no observer)-- nested pipes → hub default router (**with** `multi_observer`)--Scoping the bridge's bare router makes nested pipes consistent with the root — but they'd **lose** the router-level `multi_observer`. Open sub-question: does that matter? The hub *also* sets the observer separately (`pipelex.py:424` `set_observer(multi_observer)`), and the report delegate is reachable via the hub — so the router-level observer may be partly redundant. Needs a quick check of what `MultiObserver` does on a `PipeRouter` vs via the hub before choosing.--**Option A — Fix now, bare router (recommended).** Scope the bridge's bare `PipeRouter()` for the whole DIRECT run.-- Pros: smallest change; root + nested become consistent; matches what `_run_direct` already builds for the root; kills the Temporal-leak. Also delivers cubic's `scoped_pipe_router` request.-- Cons: nested pipes lose the router-level `multi_observer` in non-Temporal workers (behavior change — possibly fine if the hub observer covers it, possibly a telemetry regression).--**Option B — Fix now, preserve observer.** Scope an in-process router that carries `multi_observer`.-- Pros: no observer behavior change; still kills the leak.-- Cons: more plumbing — the bridge must source `multi_observer` (it lives in `Pipelex.make`, not trivially reachable from the framework-agnostic bridge); risks coupling the bridge back to internals it was extracted to avoid.--**Option C — Defer.** Leave routing untouched; this doc is the record.-- Pros: zero risk now; the bug can't fire until a Temporal-enabled worker ships.-- Cons: leaves a known P1 latent in new code; the root-vs-nested observer split persists.--## Test ideas (when fixed)--- Unit: `scoped_pipe_router` — `get_pipe_router()` returns the scoped router inside the block; nested `scoped_pipe_router` restores the *outer* override (not `None`) on inner exit; restores prior on outer exit.-- Bridge: install a sentinel/recording router as the hub default, run a sequence pipe in `DIRECT` mode via `run_pipe_via_bridge`, assert nested dispatch went through the scoped in-process router and **not** the hub default. (Optionally with `temporal_enabled=True` to prove no Temporal dispatch.)--## Related--- `graph-context-temporal-contract.md` — sibling bridge contract issue (#3).-- The raw `set_pipe_router`/`teardown_current_pipe_router` consumer lives in `pipelex-mistralai-workflows` (other repo) — if `scoped_pipe_router` lands, that plugin can adopt it later to fix its own unconditional teardown.diff --git a/wip/runtime-bridge/distributed-cost-reporting-test-coverage.md b/wip/runtime-bridge/distributed-cost-reporting-test-coverage.mddeleted file mode 100644index 619d532eb..000000000--- a/wip/runtime-bridge/distributed-cost-reporting-test-coverage.md+++ /dev/null@@ -1,214 +0,0 @@-# Cost reporting in distributed (split-worker) mode — test-coverage audit--**Status:** ✅ **RESOLVED — coverage added.** The prioritized additions below were implemented and are green. See "Resolution (as-built)" for the exact files and the decisions taken. The audit body is retained as the code-grounded map and rationale.-**Source:** Audit triggered during `/temporal-e2e-validate` (full Mode 2 run on `feature/Runtime-bridge-extraction`). Question asked: *do we have comprehensive testing of cost reporting in distributed mode with real inference — emission and aggregation — and what should we add?*-**Scope:** the token-usage → cost-report path when inference runs on a **separate Temporal `runner` worker process** (split-worker topology), with **real** provider inference. Touches `pipelex/reporting/`, `pipelex/tracing/`, `pipelex/cogt/usage/`, `pipelex/temporal/tprl_pipe/`, and the test tree under `tests/`.-------## Resolution (as-built)--All four audit questions now have automated coverage. No production code changed — these are characterization/regression tests plus one test-helper extension and a skill upgrade. **R2 decision: keep accepting** the activity-retry over-count (matches the existing documented stance); it is now regression-visible at the cost-total level rather than fixed.--**Capture (stage 1 — the biggest blind spot, real-provider branch previously asserted nowhere):**--- `tests/unit/pipelex/plugins/openai/test_openai_completions_usage_capture.py` — factory mapping (incl. cached/audio/reasoning/prediction detail fields) + `_gen_text` worker wiring reads `response.usage` into `nb_tokens_by_category`; plus the no-`usage` case pinned (stays empty → the documented silent zero-token emit).-- `tests/unit/pipelex/plugins/openai/test_openai_responses_usage_capture.py` — same for the Responses API.-- `tests/unit/pipelex/plugins/openai/test_openai_img_gen_worker_usage_capture.py` — `_gen_image_list` reads `ImagesResponse.usage`.-- `tests/unit/pipelex/cogt/extract/test_extract_worker_usage_capture.py` — extract page-count fallback, both directions (fills when provider reports none; not overridden when provider sets usage).--**Aggregate + non-LLM (stage 3, the discriminated-union wire concern):**--- `tests/unit/pipelex/tracing/test_non_llm_usage_aggregation.py` — img-gen/extract/search usage JSON-round-trip through `UsageReportEvent` (discriminator intact) → `UsageAggregator.aggregate` → `CostRegistry.aggregate_costs` with exact per-type totals.-- `tests/unit/pipelex/reporting/test_emit_runner_fallback_non_llm.py` — the previously-untested `_report_img_gen_job` / `_report_extract_job` dispatch branches emit through the real runner fallback to NDJSON and read back as the right concrete type.--**Distributed (split-worker) summed totals + cross-child + R2:**--- `tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py` — added `test_runner_usage_sums_to_expected_cost_total` (summed total via `aggregate_costs`, computed independently from the cross-worker events — no longer just landing/dedup/passthrough).-- `tests/integration/pipelex/temporal/tracing/test_split_worker_cross_child_usage.py` — PipeParallel fan-out: usage from ≥2 distinct child workflows aggregates into one parent total over the shared `pipeline_run_id` partition.-- `tests/unit/pipelex/reporting/test_emit_runner_fallback.py` — added `test_retried_activity_double_counts_cost_total_documenting_r2` pinning R2 at the billing level (retry → doubled tokens/cost). Flip these assertions if the idempotent-resequence fix ever lands.--**E2E (real inference, gated/opt-in — the thing that was invisible):**--- `tests/integration/pipelex/temporal/tracing/test_split_worker_real_inference_cost.py` — marked `inference`/`llm` (default lanes deselect it; passes when run). Real `llm_gen_text` runs on the runner queue with the context cache cleared, so real provider tokens are captured → runner fallback → aggregated to a non-zero total with a real model handle. Enabled by a backward-compatible `runner_act_llm_gen_text` param on `make_split_workers` in `helpers.py`.--**`/temporal-e2e-validate` skill — manual eyeball → numeric assertion:**--- `.claude/skills/temporal-e2e-validate/scripts/assert_cross_worker_cost.py` — sums input/output tokens straight from the NDJSON usage events, checks event count + `act_*` fallback + optional un-truncated CSV cross-check, and asserts model-type / non-zero. Wired into Tier 8b (`references/mode-2-tiers.md`) for the mock, cross-child, live, and img-gen/extract-live arms.--**Deferred / accepted (unchanged by design):**--- **R2 over-count** — accepted; pinned, not fixed (decision above).-- **Zero-token silent emit** — capture side pinned (no-`usage` → empty); the downstream zero-token `UsageReportEvent` emission remains accepted behavior.-- **Best-effort drops** — unchanged; the existing `_emit_best_effort` / child-flush WARNING-and-drop behavior is by design and already has gating tests.-- **Non-LLM full Temporal activity hop** — the runner-fallback emit path is type-agnostic (proven for LLM via the split-worker tests) and non-LLM usage is covered at the `ReportingManager`/aggregator level + the skill's live arm; a dedicated split-worker bundle with fake img-gen/extract activities was judged redundant.-------## Context for a cold-start session--Read this first if you're picking this up cold to write the detailed plan.--**Why this matters now.** The runtime-bridge work makes Pipelex pipes runnable from inside a host's own Temporal activities, and the hosted platform runs Pipelex on Temporal with **split workers** — a `router` process runs the workflows, a `runner` process runs the inference activities. Cost reporting (token usage → priced cost table / `cost_report` JSON) is a billing-adjacent surface. In split-worker mode the usage numbers are produced on one process (the runner, where inference happens) and assembled on another (the router/submitter). That cross-process assembly is the thing under audit.--**The cost path has three stages — keep them distinct when planning.**--1. **Capture** — inside the inference activity on the runner, read the provider response's token counts into `LLMJob.llm_tokens_usage.nb_tokens_by_category` (and the img-gen / extract / search equivalents).-2. **Emit** — turn that into a `UsageReportEvent` and persist it to the trace partition. In split-worker mode the runner has no registered event-log context, so emission goes through the **runner fallback** (`writer_id = act_{pid}_{uuid8}`, written to a `wf_*__w_act_*.ndjson` file or the DynamoDB partition).-3. **Aggregate + render** — the parent `WfPipeRun` runs `act_assemble_tracing` (with `assemble_usage=True`), which reads *every* event in the `pipeline_run_id` partition (router + all runners + all child workflows), passes them through `UsageAggregator.aggregate` → `PipeOutput.tokens_usages`, and the submitter prices and renders the cost table.--**The one-line finding.** Stages 2 and 3 are correct-by-construction and well unit-tested **in isolation / direct mode**. What is **not** tested anywhere by assertion is: (a) stage 1 with a *real* provider response, (b) the *summed total* end-to-end in distributed mode, (c) cross-child fan-out summation, and (d) any non-LLM usage (img-gen / extract / search). Real-inference distributed cost is exercised **only manually** via this skill's Tier 8b "live arm", which just eyeballs that a table rendered.--**Two things that are real bugs, not just missing tests** — see "Correctness risks" below. R2 (activity-retry over-count) is already documented and test-pinned as *current accepted behavior*; the planning session should decide whether to fix it or keep accepting it.--**How to use this doc.** The "How the path works" section is the map (with file:line anchors). "What is / isn't tested" is the gap inventory. "Prioritized additions" is a menu, not a committed plan — the planning session should sequence it and decide the R2 question. Nothing here has been implemented.-------## TL;DR — the four questions, answered--- **Comprehensive testing of distributed real-inference cost reporting?** No. Zero automated tests run real inference through split workers and assert the cost numbers. Only manual eyeballing (skill Tier 8b live arm).-- **Do we correctly emit usage event logs?** For LLM: the emission plumbing is correct and well-tested, but the **real-capture branch is asserted by nothing** — mock-inference diverges from real *before* the worker is built, so a passing mock test does not prove real provider tokens are read. For img-gen / extract / search: emission is **untested in any mode**.-- **Do we correctly aggregate them?** The aggregator + cost registry are correct and have exact-total unit tests — **but only in no-temporal/direct mode**. In distributed mode no test asserts a summed total (only "events land / no per-node dup / 1:1 passthrough"). Cross-child aggregation has **zero** automated assertions.-- **What to add?** Real-provider capture unit tests; split-worker summed-total + cross-child + non-LLM integration tests; one gated real-inference e2e; and turn the skill's manual Tier 8b into a numeric assertion. Details below.-------## How the path works (code-grounded map)--### Capture (runner process, inside the activity)--`LLMJob.llm_tokens_usage.nb_tokens_by_category` is seeded empty in `llm_job_before_start` (`pipelex/cogt/llm/llm_job.py:36-42`), then populated from the provider response inside the plugin worker:--- OpenAI completions / gateway / Portkey (gateway subclasses completions — `gateway_completions_factory.py:39`): `openai_completions_llm_worker.py:201-202` (text), `:266-267` (object); mapping in `openai_completions_factory.py:101-111` (`usage.prompt_tokens`, `usage.completion_tokens`, cached/reasoning details).-- OpenAI responses API: `openai_responses_llm_worker.py:179-180`; factory `openai_responses_factory.py:100-106`.-- Google: `google_llm_worker.py:236-237, 313-325`; factory `google_factory.py:144-162`.-- Bedrock: `bedrock_llm_worker.py:85-86`.-- Img-gen: `openai_img_gen_worker.py:134, 149-150, 164`; reported at `img_gen_worker_abstract.py:69-70 / 108-110`.-- Extract: provider sets `nb_tokens_by_category`, else a page-count fallback fills it (`extract_worker_abstract.py:96-101`); reported at `:105-106`.-- Search: hardcoded (`linkup_search_worker.py:90, 154`); reported at `search_worker_abstract.py:49, 72`.--All of this runs synchronously inside the activity on the runner process, so the populated `tokens_usage` is what the runner-side fallback emits.--**Mock diverges before capture.** `llm_gen_text` short-circuits to the mock helper at the top (`pipelex/cogt/content_generation/llm_generate.py:14-15`; object/list at `:29-30`, `:50-51`) — it never builds an `LLMJob`, never gets a worker, never reads a provider response. Mock and real **re-converge only at `report_inference_job`** (the *emit* half). The mock helper hand-builds synthetic usage `MOCK_INFERENCE_NB_TOKENS_BY_CATEGORY = {INPUT: 100, OUTPUT: 50}` (`dry_mock.py:58-61, 94-108`). **Consequence: mock passing proves emit/aggregate, not capture.** And img-gen / extract leaves *raise* `MockInferenceUnsupportedError` under mock (`img_gen_generate.py:12-13`, `extract_generate.py:11-13`) — so mock cannot cover them at all.--### Emit (runner fallback)--`ReportingManager.report_inference_job` → `_emit_usage_event` (`pipelex/reporting/reporting_manager.py:149-185`):--```python-if not trace_context.emit_usage_events: return # :173 — the --costs gate, before context lookup-context = self._event_log_contexts.get(trace_context.lookup_key) # :176-if context is not None:- self._emit_via_registered_context(...) # FAST PATH, writer_id="primary"- return # :179 — hard return-self._emit_usage_event_runner_fallback(...) # FALLBACK, writer_id="act_*"-```--On the runner, `_event_log_contexts` is empty (`set_event_log` is only ever called on the router — `wf_pipe_router.py:114-120`), so the fallback always wins. `writer_id` is generated once per process under a double-checked lock in `ActivityEventLogCache.get_or_create` as `f"act_{os.getpid()}_{uuid4().hex[:8]}"` (`activity_event_log.py:64-76`), and the one-shot WARNING "Runner-side usage event emission engaged" comes from `activity_event_log.py:93-99`. Event type: `UsageReportEvent` (`pipelex/tracing/trace_events.py:162-167`), carrying `tokens_usage: AnyTokensUsage` + a top-level `writer_id`.--### Aggregate + render (parent / submitter)--`act_assemble_tracing` (`pipelex/temporal/tprl_pipe/act_assemble_tracing.py:36-50`) → `assemble_tracing` (`pipelex/pipe_run/tracing_assembly.py:50-131`):--- Reads events **once**, keyed by `pipeline_run_id` only (`:95-99`). The NDJSON read globs **every** `*.ndjson` in `{traces_dir}/{pipeline_run_id}/` (`ndjson_event_log.py:133`), so router `wf_*.ndjson` and all runner `wf_*__w_act_*.ndjson` files are picked up by the same glob. Read-side dedup key: `(workflow_id, writer_id, type(event).__name__, sequence)` (`ndjson_event_log.py:150`); DynamoDB equivalent `SK = EVENT#{workflow_id}#{writer_id}#{sequence:010d}` overwrites idempotently (`dynamodb_event_log.py:99-118`). This is what makes Temporal **replay** safe.-- `if assemble_usage: result.tokens_usages = UsageAggregator.aggregate(events)` (`:112-113`). `UsageAggregator.aggregate` is a 1:1 passthrough — `[evt.tokens_usage for evt in events if isinstance(evt, UsageReportEvent)]` (`usage_aggregator.py:12-22`), **no dedup/grouping of its own** (dedup already happened in `read_events`).-- `TracingAssembly` rides back; `wf_pipe_run.py:91-114` copies `tokens_usages` onto `PipeOutput.tokens_usages` (`pipe_output.py:32`).-- Render: `cost_report_renderer.py:21-82` (called from `_run_core.py:320-324`); per-model grouping + pricing in `cost_registry.py:216-279`; price comes from `tokens_usage.unit_costs` captured at inference time, **not** re-priced from a catalog (`costs_per_token.py:4-31`). Suppression: `has_reportable_usage` = `total_nb_tokens > 0 or total_cost > 0` (`cost_registry.py:41-49`), so dry-run (zero) is suppressed; a real run on a free model still reports (tokens > 0, cost 0).--### Cross-child fan-out--Join is **by `pipeline_run_id` on a shared partition** — there is no per-workflow join. Each child gets a distinct `workflow_id` (`temporal_pipe_router.py:69`) but the **same `pipeline_run_id`** (`TraceContext.copy_for_child` preserves it — `trace_context.py:66-84`; the fallback stamps it from the job metadata — `reporting_manager.py:270`). Only the top-level `WfPipeRun` runs `act_assemble_tracing`, and its single `read_events(pipeline_run_id)` sweeps the whole partition. Load-bearing invariant: every child's `pipeline_run_id` must equal the parent's, and (NDJSON backend) `traces_dir` must be shared across all hosts — the hosted platform must use the DynamoDB backend for multi-host (`ndjson_event_log.py:38-40`).-------## What is tested vs what only "runs without crashing"--### Distributed (true split-worker) topology--Only `tests/integration/pipelex/temporal/tracing/test_split_worker_usage.py` runs a real two-queue split — and it **substitutes** `act_llm_gen_text` with a helper that hand-builds `LLMTokensUsage` (INPUT=1, OUTPUT=1) (`helpers.py:195-246`). So even here there is no real worker and no provider read. It asserts:--- an `act_*` usage event **lands** in a runner file (`test_runner_usage_event_lands_in_same_ndjson_dir`),-- exactly **one** event per `(node_id, workflow_id)` — fast-path/fallback de-dup (`test_no_double_emit_in_split_worker_pool`),-- the aggregator returns one record per event, **none dropped, order preserved** (`test_runner_usage_aggregates_to_tokens_usages` → `tokens_usages == [evt.tokens_usage for evt in usage_events]`).--None of them sums tokens or asserts `total == N×{input,output}`.--### Numeric correctness — exists, but only no-temporal--Exact-total assertions live entirely outside Temporal, on hand-built usage:--- `tests/unit/pipelex/cogt/usage/test_cost_registry.py` — per-model groupby totals, `compute_total_cost==3.5`, `aggregate_costs` exact `total_cost==0.2` / `total_nb_tokens==150`, cached/non-cached splits. Richest numeric coverage.-- `tests/unit/pipelex/reporting/test_cost_report_renderer.py` — free model `total_cost==0.0 / 150`, console `total_cost==0.2`.-- `tests/integration/pipelex/pipeline/test_cost_report_rendering.py` — rendered string contains `"0.2000"` (DIRECT).-- `tests/unit/pipelex/reporting/test_reporting_event_emission.py` — single emitted event has `nb_tokens_by_category[INPUT]==100` (fast path, in-memory log).--### Mock-inference (synthetic tokens, emit-path only)--- `test_mock_inference_temporal.py` — real `act_llm_gen_text` with `is_mock_inference=True`, but **single task queue** (not split-worker), so it doesn't even hit the runner fallback. Asserts `tokens_usages` reportable under the sentinel `mock_inference` model; no exact total.-- `test_mock_inference_direct.py` — DIRECT; asserts non-None / `has_reportable_usage`; no exact total.--### Graph-only / serialization-only (zero cost assertions despite real cross-child fan-out)--- `test_wf_graph_tracing_batch.py`, `test_wf_graph_tracing_parallel.py` — real fan-out to child workflows, **dry-run**, assert only GraphSpec structure (node statuses, BATCH/CONTAINS/DATA edges, multiple NDJSON files). Never touch `tokens_usages`.-- `test_split_worker_object_gen.py`, `test_split_worker_extract_pages.py` — in-process single worker; assert serialization / activity-id dispatch; **no usage/cost assertion**.--### Precise answers--- **A. Exact aggregated total in split-worker mode (even mock)?** No. The three split-worker tests assert landing / per-node de-dup / 1:1 passthrough, never a summed total.-- **B. Real-inference cost correctness in distributed mode?** No automated test. Only manual via the skill's Tier 8b live arm ("expect a table with the real model handle and real token counts") — eyeball, no assertion.-- **C. Cross-child sum → single parent total?** No automated assertion, mock or real. The fan-out tracing tests are graph-only/dry-run; the skill's "A+B+C → one report" block is manual.-- **D. Img-gen / extract / search usage in distributed mode?** Zero coverage in **any** mode. Grep for `ImgGenTokensUsage` / `ExtractTokensUsage` / `SearchTokensUsage` across `tests/` returns nothing; every usage test uses `LLMTokensUsage` only.-- **E. Fast-path vs fallback de-dup?** Covered by `test_no_double_emit_in_split_worker_pool` (buckets by `(node_id, workflow_id)`, asserts count==1 per key) plus unit gating tests. It does **not** assert token totals and does **not** cover the retry over-count (R2 below).-------## Correctness risks (real bugs, not just missing tests)--- **R2 — activity-retry over-count.** A Temporal-**retried** activity (not replay) re-emits the same inference's usage at sequence N+1 instead of overwriting N, defeating the `(workflow_id, writer_id, sequence)` dedup → counted twice. Documented at `docs/.../tracing-cost-reporting.md:51`, pinned as *current accepted behavior* by `tests/unit/pipelex/reporting/test_emit_runner_fallback.py:367-388` (asserts two events at sequences `[0,1]`). The `strict_mode` / idempotent-resequence fix is deferred. With M children doing retryable inference, the over-count probability scales with fan-out × retry rate. **The planning session must decide: fix or keep accepting.**-- **Zero-token silent emit.** If a provider returns no `usage`, the worker skips the assignment (`and (usage := response.usage)` guards everywhere, e.g. `openai_completions_llm_worker.py:201`), but the job still has a truthy `LLMTokensUsage`, so `_report_llm_job` does not hit its `if not llm_tokens_usage` guard (`reporting_manager.py:116`) — a `UsageReportEvent` is emitted **with zero tokens**. Silent under-count, no flag.-- **Best-effort drops.** Emit (`_emit_best_effort`, `reporting_manager.py:208-221`) and child flush (`wf_pipe_router.py:175-177`) swallow infra failures with a WARNING; an unwritable dir / DDB reject / unshared NDJSON `traces_dir` across hosts silently drops a slice. The run still succeeds; the report is quietly short. A whole-read failure is at least visible (`usage_assembly_error` → `tokens_usages=None` → no report), but per-line/per-file drops are not.--The aggregator and renderer themselves are lossless — all risk lives in the capture/emit/persist/read layer.-------## Prioritized additions (the original menu — now implemented; see "Resolution (as-built)" above for the landed files)--### Unit — cheapest, closes the biggest blind spot (real capture)--- **Real-provider capture tests.** Substitute the OpenAI/gateway client with a canned response carrying `usage.input_tokens=…` (and the cached/reasoning detail fields); assert `nb_tokens_by_category` is populated correctly. One each for completions, responses API, **img-gen, extract**. Deterministic, no spend — this is the single highest-value gap, since the real-capture branch is asserted nowhere today.-- **No-`usage` provider case.** Decide intended behavior (today: silent zero-token emit) and pin it.-- **Non-LLM through the aggregator.** A test routing `ImgGenTokensUsage` / `ExtractTokensUsage` / `SearchTokensUsage` through `UsageReportEvent → aggregate_costs` and asserting the per-type total (currently never exercised).--### Integration — split-worker, fixture/mock (no spend)--- **Summed-total assertion.** Extend `test_split_worker_usage.py`: inject known synthetic per-call tokens, assert `aggregate_costs(tokens_usages).total_nb_tokens == expected sum` (not just landing/dedup/passthrough).-- **Cross-child sum.** PipeBatch/PipeParallel split-worker test where each child emits known usage; assert the parent's single total = sum across children. Highest-risk uncovered path (also the natural home for an R2 regression).-- **Non-LLM cross-worker.** Split-worker img-gen + extract usage crossing the runner fallback via a fixture worker returning canned `ImgGenTokensUsage` / `ExtractTokensUsage`; assert aggregate + render. Closes D in distributed mode.-- **R2 at the total level.** Force an activity retry and assert the cost total double-counts — make the known over-count regression-visible (and, if fixed, flip the assertion).--### E2E — real inference, gated/opt-in lane--- One marked real-inference split-worker test: assert `tokens_usages` non-empty, every entry non-zero, model handles match the real models, usage-record count == number of inference calls, total > 0. Even loose bounds catch the entire "real provider usage not captured/emitted/aggregated in distributed mode" class — the exact thing that's invisible today (and that I could not verify by eyeball this session because the live cost tables were column-truncated).--### temporal-e2e-validate skill — make the manual check assert--- Strengthen Tier 8b's live arm: parse the NDJSON usage events and assert **summed input/output across all `__w_act_*` files == rendered table total**, and usage-event count == inference-call count. Dump costs to CSV (un-truncated) so real non-zero numbers are actually visible — the current terminal table truncates to `0 … …`, making the eyeball check nearly worthless.-- Add an img-gen / extract live cross-worker **cost** assertion (Tiers 4/5/10c run those live but never check the numbers).-------## Key files--- Capture: `pipelex/cogt/content_generation/llm_generate.py`, `dry_mock.py`, `img_gen_generate.py`, `extract_generate.py`; `pipelex/cogt/llm/llm_job.py`, `llm_worker_abstract.py`; `pipelex/plugins/openai/openai_completions_llm_worker.py` (+ `openai_completions_factory.py`, `openai_responses_factory.py`), `pipelex/plugins/openai/openai_img_gen_worker.py`; `pipelex/cogt/extract/extract_worker_abstract.py`.-- Emit: `pipelex/reporting/reporting_manager.py`, `pipelex/tracing/activity_event_log.py`, `pipelex/tracing/trace_events.py`.-- Aggregate + render: `pipelex/temporal/tprl_pipe/act_assemble_tracing.py`, `pipelex/pipe_run/tracing_assembly.py`, `pipelex/tracing/usage_aggregator.py`, `pipelex/tracing/ndjson_event_log.py`, `pipelex/tracing/dynamodb_event_log.py`, `pipelex/cogt/usage/cost_registry.py`, `pipelex/cogt/usage/costs_per_token.py`, `pipelex/reporting/cost_report_renderer.py`, `pipelex/core/pipes/pipe_output.py`.-- Tests today: `tests/integration/pipelex/temporal/tracing/{test_split_worker_usage.py,test_mock_inference_temporal.py,test_split_worker_object_gen.py,test_split_worker_extract_pages.py,helpers.py}`; `tests/unit/pipelex/reporting/{test_emit_runner_fallback.py,test_emit_usage_event_gating.py,test_reporting_event_emission.py,test_cost_report_renderer.py}`; `tests/unit/pipelex/cogt/usage/test_cost_registry.py`; `tests/unit/pipelex/tracing/{test_usage_aggregator.py,test_writer_id_schema.py,test_trace_events.py}`; `tests/unit/pipelex/pipe_run/test_tracing_assembly.py`.--## Related--- [`cost-reporting-validation-prompts.md`](cost-reporting-validation-prompts.md) — operator cheat-sheet of `/temporal-e2e-validate` prompts that drive the Tier 8b arms this audit motivated.-- `../observer-and-telemetry/` — sibling tracing/telemetry follow-ups.-- `docs/.../tracing-cost-reporting.md` (R2 documentation) and the [`wip/history/registry/cost-report-deferred-decisions.md`](../history/registry/cost-report-deferred-decisions.md) deferred items (e.g. `tracing_config.is_enabled=false` + `--costs` -> silent no-report).-- This audit was produced cross-checking three independent code sweeps (emission, aggregation, test inventory); the file:line anchors above are the load-bearing evidence — re-verify any that look stale before building on them.diff --git a/wip/runtime-bridge/execution-mode-property-helpers-unused.md b/wip/runtime-bridge/execution-mode-property-helpers-unused.mddeleted file mode 100644index dfac43258..000000000--- a/wip/runtime-bridge/execution-mode-property-helpers-unused.md+++ /dev/null@@ -1,24 +0,0 @@-# `PipelexExecutionMode` property helpers — tested but not yet consumed--**Status:** deferred follow-up (not a bug). Intentionally kept as a tested, intention-revealing surface; wiring them into `bridge.py` (or removing them) is a follow-up. Migrated here from the PR #969 reviewer's guide (`TODOS.md`) when that file was retired at merge — it was the only live item in that guide not already recorded under `wip/`.--## What--`PipelexExecutionMode` (`pipelex/runtime_bridge/execution_mode.py`) exposes three `@property` helpers:--- `requires_pipelex_temporal` — True for `TEMPORAL_BLOCKING` / `TEMPORAL_FIRE_AND_FORGET`.-- `requires_mistral_workflows_extra` — True for `MISTRAL_NATIVE`.-- `is_fire_and_forget` — True for `TEMPORAL_FIRE_AND_FORGET`.--They are covered by `tests/unit/pipelex/runtime_bridge/test_execution_mode.py`, but **`bridge.py` does not consume them** — it inlines the equivalent `match` / identity checks at each dispatch and validation site instead.--## Why it's on file, not fixed--The helpers were written as the intention-revealing form of the mode predicates, ahead of `bridge.py` being refactored to call them. Keeping them tested-but-unused is a deliberate, low-risk surface (the `match` bodies are exhaustive, so the linter guards them if a mode is added), not a silent gap. There is no behavioural bug: the inlined checks in `bridge.py` and the helpers agree.--## Follow-up (pick one when next touching the bridge)--- **Wire them in** — replace the inlined predicates in `bridge.py` (`_validate_input`'s fire-and-forget guard, the per-mode extra-dependency guards) with the helpers, so the predicate lives in one place. Lowest-churn, removes the duplication the reviewer flagged.-- **Or remove them** — if `bridge.py`'s `match execution_mode:` dispatch stays the single source of truth and the helpers earn no second caller, delete them with their tests.--Not urgent either way; no correctness impact.diff --git a/wip/runtime-bridge/graph-context-temporal-contract.md b/wip/runtime-bridge/graph-context-temporal-contract.mddeleted file mode 100644index 899bbf041..000000000--- a/wip/runtime-bridge/graph-context-temporal-contract.md+++ /dev/null@@ -1,75 +0,0 @@-# `graph_context` threaded into Temporal modes despite a DIRECT-only contract--**Status:** ✅ **RESOLVED — Option A applied.** (was: confirmed contract/docstring mismatch, deferred pending decision)-**Source:** PR #959 review — greptile-apps (P2, `bridge.py:111-128`, "Temporal modes inherit tracing").-**Severity:** real but **latent** — only mattered when a host passed a non-None `graph_context` *and* selected a Temporal mode.--## Resolution (Option A)--`run_pipe_via_bridge` now computes `is_direct = input_payload.execution_mode is PipelexExecutionMode.DIRECT` and passes `graph_context=graph_context if is_direct else None` to `build_pipe_job_from_input`, so a host `graph_context` is honored for DIRECT and **nulled for the Temporal modes** — honoring the documented contract and removing the cross-contamination foot-gun (`WfPipeRouter`'s `graph_context is not None` guard makes the None a clean no-op). The now-inaccurate "would be ignored anyway" docstring clause was rewritten to state the nulling plainly. Regression test: `tests/unit/pipelex/runtime_bridge/test_graph_context_contract.py` (DIRECT forwards the host context; TEMPORAL_BLOCKING nulls it). `make agent-check` + `make agent-test` green.--The original triage below is retained as the record of why Option A over B/C.-------## The finding--> The docstring says `graph_context` is only honored for `DIRECT`, but this code builds the `PipeJob` with that context before dispatching every mode. For `TEMPORAL_BLOCKING` and `TEMPORAL_FIRE_AND_FORGET`, `WfPipeRouter` consumes the non-null context and opens a workflow tracer under the caller's graph id and parent node. A host activity that passes its own graph context can therefore merge or duplicate Pipelex Temporal trace events into the host activity graph, even though the bridge contract says Temporal modes ignore that context.--## Verified behavior (code trace)--**The documented contract (`pipelex/runtime_bridge/bridge.py:100-103`):**--> The optional `graph_context` is only honored for `DIRECT` execution mode — TEMPORAL modes already have their own event-log infrastructure via `pipeline_run_setup` and a passed-in context **would be ignored anyway**.--**What the code does** — `graph_context` is baked into the `PipeJob` *before* the mode `match`, so it flows into every branch:--- `pipelex/runtime_bridge/bridge.py:112-116` — `build_pipe_job_from_input(..., graph_context=graph_context)` is called unconditionally, then `:118` `match input_payload.execution_mode:` dispatches DIRECT / TEMPORAL_BLOCKING / TEMPORAL_FIRE_AND_FORGET / MISTRAL_NATIVE.-- `pipelex/runtime_bridge/bridge.py:161-165` — `graph_context` is stored into `JobMetadata`.--**`WfPipeRouter` *does* consume it** (so "would be ignored anyway" is false):--- `pipelex/temporal/tprl_pipe/wf_pipe_router.py:49` — `graph_context = workflow_arg.job_metadata.graph_context`-- `:80` — `if tracing_config.is_enabled and graph_context is not None:`-- `:87-94` — `open_tracer(graph_id=graph_context.graph_id, data_inclusion=graph_context.data_inclusion, ...)`-- `:95-100` — preserves `graph_context.parent_node_id` so "CONTAINS edges link back to the parent workflow's controller node."--So if a host passes its own `graph_context` with a Temporal mode, Pipelex's Temporal workflow opens its tracer under the **host's** `graph_id` and links edges to the **host's** `parent_node_id` — i.e. Pipelex Temporal trace events get merged into the host's graph. That is exactly the cross-contamination the docstring's contract was trying to forbid.--Note the asymmetry that makes this a *contract* bug rather than a feature: for DIRECT the host's graph_context is intended (per-step events flow into the host's tracer — `bridge.py:97-99, 142-146`); for Temporal the bridge bypasses Pipelex's own `pipeline_run_setup` graph_context creation and instead forwards the host's, which `WfPipeRouter` then treats as if it were Pipelex's own.--## The fork--**Option A — Honor the contract (recommended).** Pass `graph_context` only for DIRECT; pass `None` for the Temporal modes. E.g. compute it per-mode:--```python-is_direct = input_payload.execution_mode is PipelexExecutionMode.DIRECT-pipe_job = build_pipe_job_from_input(- input_payload=input_payload,- library_crate=library_crate,- graph_context=graph_context if is_direct else None,-)-```--- Pros: makes the code match the documented intent; stops a host's graph_context from contaminating Pipelex's Temporal trace graph; tiny, low-risk; the `None` guard at `wf_pipe_router.py:80` already makes a None context a clean no-op. Aligns with the docstring author's clear intent.-- Cons: if someone *wanted* host→Pipelex Temporal trace linking, this removes the only path to it. (No evidence anyone does, and `pipeline_run_setup` is the documented Temporal tracing path.)-- Tidy-up if chosen: drop the now-inaccurate "would be ignored anyway" clause and state plainly that Temporal modes deliberately do not forward a host `graph_context`.--**Option B — Fix the docstring, keep threading it.** Treat the current behavior as intended and document that Temporal modes consume `graph_context` (parent-linking via CONTAINS edges).-- Pros: no behavior change; preserves any future host→Pipelex Temporal trace stitching.-- Cons: keeps the cross-contamination foot-gun (a host that opens its own tracer and picks a Temporal mode silently merges Pipelex's Temporal events into its graph); the "their own event-log infrastructure via `pipeline_run_setup`" rationale is then misleading, since the bridge's Temporal path doesn't go through `pipeline_run_setup` for graph_context.--**Option C — Defer.** This doc is the record; revisit when Temporal ships and the host-tracing story for Temporal modes is actually designed.--## Recommendation--Option A. It is the smaller change, removes a latent foot-gun, and makes the code obey its own stated contract. Revisit only if a concrete need for host→Pipelex Temporal trace stitching appears — at which point it should be designed explicitly (own graph id / explicit opt-in), not inherited by accident.--## Test ideas (when fixed)--- Bridge unit: call `run_pipe_via_bridge(input_payload=<TEMPORAL_*>, graph_context=<non-None>)` and assert the resulting `PipeJob.job_metadata.graph_context is None`; for `DIRECT` assert it is preserved. (Can test `build_pipe_job_from_input` selection logic without a live Temporal server.)--## Related--- `direct-mode-nested-router-leak.md` — sibling bridge issue (#1); both are about the bridge's contract for DIRECT vs Temporal dispatch.-- `wf_pipe_router.py:80-103` is shared Temporal machinery — changing *it* would affect normal Pipelex Temporal runs; Option A keeps the change confined to the bridge boundary, which is correct.diff --git a/wip/runtime-bridge/library-manager-pipe-source-map-concurrency.md b/wip/runtime-bridge/library-manager-pipe-source-map-concurrency.mddeleted file mode 100644index 9805bd603..000000000--- a/wip/runtime-bridge/library-manager-pipe-source-map-concurrency.md+++ /dev/null@@ -1,40 +0,0 @@-# LibraryManager `_pipe_source_map` is not per-call scoped (deferred)--**Status:** deferred. Pre-existing `LibraryManager` behaviour, surfaced more by the bridge's per-call scoped-library path. Observability-only impact, Temporal not in production. Not fixed in PR #969.--## What--`LibraryManager._pipe_source_map` (`pipelex/libraries/library_manager.py:98`) is a **manager-level** dict keyed by `pipe_ref` (`domain.pipe_code`):--```python-self._pipe_source_map: dict[str, Path] = {} # pipe_ref (domain.pipe_code) -> source .mthds file-```--Its siblings — `_blueprints`, `_crate_cache`, `_loaded_fingerprints` — are all keyed by `library_id` (one entry per open library). `_pipe_source_map` is the odd one out: it is shared across every open library.--`load_from_crate` writes `self._pipe_source_map[pipe_ref] = Path(source)`, and the keyed `teardown(library_id=...)` iterates the torn-down library's pipes and `pop()`s those `pipe_ref`s out of the shared map (`library_manager.py:122`).--## Why it can interfere across concurrent bridge calls--`run_pipe_via_bridge` opens a fresh per-call scoped library (`bridge._scoped_library_for_crate`) with a unique `library_id`, loads the caller's crate into it, and tears it down on exit. Two **concurrent** bridge calls whose crates define the **same** `pipe_ref` will:--- overwrite each other's source path (last writer wins), and-- one call's teardown can `pop()` the `pipe_ref` out from under the other still-running call, so the survivor's `get_pipe_source()` returns `None` (or, after an overwrite, the other tenant's path).--`get_pipe_source` feeds source attribution into the graph / observability layer only — it does **not** affect pipe resolution or run correctness. So the blast radius is a wrong-or-missing source path in a trace graph, not a wrong result.--There is also a broader, lower-confidence concern: `LibraryManager` mutates these plain dicts (`open_library` / `teardown` / `load_from_crate`) with no lock, while the bridge advertises a concurrent-activity execution model. The no-arg `teardown()` branch reassigns the dicts wholesale; concurrent mutation during the keyed teardown's iterate-and-`pop` could in principle raise `dictionary changed size during iteration`. Each bridge call uses a unique `library_id` (isolating `_libraries`/`_blueprints`/etc.), which keeps the realistic blast radius confined to `_pipe_source_map`.--## Why defer (not fix here)--- **Pre-existing.** This is `LibraryManager` behaviour, not introduced by this PR. The pre-existing Temporal `wf_pipe_router` scoped-library path uses the same map. The bridge only makes the concurrent-scoped-library pattern more prominent.-- **Observability-only.** Worst case is a wrong/missing source path in a trace graph. No run-correctness or data-loss impact.-- **Not in production.** The concurrent-multi-tenant path that triggers it is the Temporal/Mistral worker model, which is not in production.-- **Right fix touches `library_manager.py` beyond this PR's surface** and wants a deliberate design pass, not a reflexive patch on the way to landing the bridge.--## Fix shape when picked up--- Scope the source map per `library_id`: `dict[str, dict[str, Path]]` keyed by `library_id` (matching the sibling caches) so the keyed teardown pops only the calling library's entries and concurrent crates cannot clobber each other. Alternatively, store the source path on the `Library` object itself.-- If `LibraryManager` is to formally support concurrent callers, add a lock around the structural mutations in `open_library` / `teardown` / `load_from_crate` (or document that mutation is not thread-safe and require callers to serialise). At minimum, guard the `_pipe_source_map` iterate-and-`pop` in the keyed teardown.--Found by the `/review` adversarial (red-team) pass on PR #969.diff --git a/wip/runtime-bridge/temporalio-bump-1.24-provenance.md b/wip/runtime-bridge/temporalio-bump-1.24-provenance.mddeleted file mode 100644index 8a5316de0..000000000--- a/wip/runtime-bridge/temporalio-bump-1.24-provenance.md+++ /dev/null@@ -1,15 +0,0 @@-# `temporalio==1.24.0` bump on the bridge branch--**Status: RESOLVED — keep `1.24.0`** (maintainer decision, 2026-06-07). The bump lands with this PR; `dev` moves from `1.23.0` to `1.24.0` on merge. No code change needed; this note is kept as the provenance record.--## What--`pyproject.toml` on `feature/Runtime-bridge-extraction` pins `temporal = ["temporalio==1.24.0", ...]`, while `dev` is on `temporalio==1.23.0`. The bump was introduced by the branch's first commit (`204cd749`, the bridge extraction carved off the `_workflows` worktree), not by any later bridge work.--## Why it's a question--The runtime bridge does not depend on a specific Temporal SDK version — `temporalio` is lazy-imported only inside the `TEMPORAL_*` branches and the existing `pipelex.temporal` glue, none of which references a 1.24-only API. So this bump looks like it rode along from the upstream `_workflows` branch rather than being required by this PR.--## Decision--Keep `1.24.0` — confirmed wanted on `dev`. The bridge itself is version-agnostic (lazy import, no 1.24-only API), but staying current is fine and CI exercises it on the in-process test server.diff --git a/wip/runtime-bridge/trace-flush-blocking-io.md b/wip/runtime-bridge/trace-flush-blocking-io.mddeleted file mode 100644index 17c6ce7db..000000000--- a/wip/runtime-bridge/trace-flush-blocking-io.md+++ /dev/null@@ -1,54 +0,0 @@-# `flush_trace_events_to_backend` — async with blocking I/O--**Status:** ✅ **RESOLVED — false positive / by-design.** No code change. The `asyncio.to_thread` offload below is kept on file for if/when Temporal ships and we profile activity-worker contention.-**Source:** PR #959 review — cubic-dev-ai (P2, `pipelex/runtime_bridge/primitives/trace_flush.py:14`).--## The finding--> `async def` with `# noqa: RUF029` but contains synchronous blocking I/O — the emit loop calls `DynamoDBEventLog.emit()` (boto3 `put_item` — synchronous HTTP call) and `NdjsonEventLog.emit()` (synchronous file write+flush) without yielding the event loop. Because the caller `act_flush_trace_events` is an `async def` Temporal activity that awaits this coroutine, the blocking I/O holds Temporal's async activity worker event loop for the aggregate duration of all `emit()` calls.--## Why this is acceptable (the activity vs workflow distinction)--The facts are correct, but the conclusion ("holds the event loop") is the *intended and correct* place for blocking I/O in a Temporal app.--- The function is **designed to run in an activity**, never a workflow. Its module docstring (`trace_flush.py:1-6`) says so: *"Called from a host-runtime activity (Temporal / Mistral) after a workflow's `BufferingEventLog` has drained, since synchronous I/O (boto3, file writes) cannot run inside a workflow thread."*-- The emit loop is synchronous on purpose (`trace_flush.py:26-32`): `make_event_log(...)` then `for event in events: event_log.emit(event)`. The `# noqa: RUF029` (`trace_flush.py:14`) documents the deliberate async-with-no-await.-- Workflows must stay deterministic and I/O-free, so they buffer in memory via `BufferingEventLog` and hand the drained events to this activity:- - `pipelex/temporal/tprl_pipe/wf_pipe_router.py:152-167` — drains `event_log` and `await workflow.execute_activity(act_flush_trace_events, ... start_to_close_timeout=30s, retry_policy=max 3)`.-- The activity is a thin wrapper: `pipelex/temporal/tprl_pipe/act_flush_trace_events.py:22-25` — `@activity.defn(name="act_flush_trace_events")` / `async def act_flush_trace_events(arg)` → `await flush_trace_events_to_backend(arg.events)`.-- `DynamoDBEventLog.emit` even warns against being called from a workflow (its docstring says the synchronous boto3 HTTP call would block the workflow thread and trip the deadlock detector) and points callers to exactly this `BufferingEventLog` + `act_flush_trace_events` pattern.--Activities are the part of Temporal that is *allowed* to block — that's their whole purpose. So this is the textbook pattern, not a bug.--## The kernel of truth (only if throughput ever matters)--Temporal's **async** activities share the worker's event loop. A blocking call inside one async activity does stall *sibling* async activities on the same worker for its duration. Temporal's own guidance: an async activity doing blocking I/O should either be registered as a **sync** activity (runs in a thread-pool executor) or offload the blocking part via `asyncio.to_thread` / `run_in_executor`.--For trace flushing this is negligible today: best-effort, bounded (30s `start_to_close_timeout`, ≤3 retries), batched per workflow, low volume, and Temporal isn't in prod. So **no change now.**--If/when activity throughput becomes a real concern, the minimal mitigation is:--```python-# trace_flush.py — offload the blocking emit loop off the event loop-import asyncio--async def flush_trace_events_to_backend(events: list[TraceEvent]) -> None:- ...- def _emit_all() -> None:- try:- for event in events:- event_log.emit(event)- finally:- event_log.close()- await asyncio.to_thread(_emit_all)-```--(That also lets the `# noqa: RUF029` go away, since the function would then genuinely await.)--## Recommendation--Reply on the thread as a **false positive / by-design**, citing the activity-vs-workflow distinction, and resolve it. Keep the `asyncio.to_thread` mitigation above on file here for if/when Temporal goes to prod and we profile activity-worker contention.--## Related--- The other two PR #959 forks are real bugs: `direct-mode-nested-router-leak.md` (#1), `graph-context-temporal-contract.md` (#3). This one is the only "false positive" of the three.diff --git a/wip/runtime-code-fixes.md b/wip/runtime-code-fixes.mddeleted file mode 100644index b6a155c9a..000000000--- a/wip/runtime-code-fixes.md+++ /dev/null@@ -1,19 +0,0 @@-# Runtime code fixes--Concrete fixes verified against the code, not yet done. Each is independent.--## `request_id` on delivery failure messages--Append the already-in-scope `request_id_suffix` to the `StorageDeliveryError` failure message and to both `WebhookDeliveryError` branches in `pipelex/pipe_run/delivery_executor.py`, mirroring the success paths; add unit assertions that the messages carry `request_id=`. Full analysis — exact sites, the case for/against, and a sketch commit — is in [`error-handling/track-delivery-error-path-request-id.md`](error-handling/track-delivery-error-path-request-id.md).--## Cross-worker cost report assembly wiring--The single genuine functional gap from the tracing work: wire `UsageAggregator.aggregate(events)` → `ReportingManager.inject_tokens_usages(...)` → `generate_report` into the post-run readback, parallel to the existing graph readback, for both direct mode (`pipe_run/pipe_run.py` / `graph_assembly.py`) and Temporal (`act_assemble_graph` / post-workflow), with a cross-worker test. Tracked as P1 in [the distributed-execution plan](distributed-execution/README.md); the as-built context and the open T2/T3 gaps are in [`distributed-execution/tracing-cost-reporting.md`](distributed-execution/tracing-cost-reporting.md).--## Deferred / out of scope--Surfaced alongside the two fixes but out of scope for them:--- GraphSpec causal ordering for parent/child topologies (observability-only).-- kajson class-registry race under pytest-xdist (test-hygiene; needs runtime repro).-- `get_config()` replay-determinism — the cheap parts (a `docs/distributed-execution` note on the config-edit-while-in-flight constraint, plus a Replayer regression test) are file-able; the full fix is Worker Versioning.diff --git a/wip/structured-logging.md b/wip/structured-logging.mddeleted file mode 100644index 3e915f264..000000000--- a/wip/structured-logging.md+++ /dev/null@@ -1,136 +0,0 @@-# Structured logging refactor — kickoff briefing--**Status:** kicked off, not started. Sequenced AFTER the error-handling stack merges. Will get its own branch and its own PR.--**Purpose of this doc:** cold-start briefing for the session that picks this up. Captures *why* the refactor exists, *what good looks like*, *what the current code constrains us to*, and the first open questions to resolve. **Not a plan** — the plan is the first artifact of that session.--## TL;DR--Pipelex's `log` API is string-shaped today (`log.info("a message string with stuff baked in")`). That shape forced the recent `request_id` correlation work into per-line text interpolation, which is leaky, fragile, and unqueryable downstream. The right answer is **structured logging** (each record is a dict / JSON object with named fields) combined with **contextvars for request-scoped attributes** (`request_id`, `user_id`, `pipeline_run_id`) bound once at the entry boundary and auto-stamped onto every log record in scope. That replaces the current threading-and-interpolation pattern entirely, and removes a whole class of "we forgot to add `request_id` on the failure path" bugs by construction.--## What triggered this--`wip/error-handling/track-delivery-error-path-request-id.md` opened the question of whether to thread `request_id` into the failure-path messages of `DeliveryExecutor` (mirroring the success-path threading already in place). The proximate decision was "do it the manual way OR do the structural fix." We chose to defer the structural fix to a dedicated branch — this one.--Earlier work did manual per-line threading on the success path:--- `DeliveryExecutor.execute` threads `request_id` into the success log lines.-- `PipeRun.run` forwards `request_id` from the direct-mode dispatcher to `DeliveryExecutor`.--Both commits also introduced the ad-hoc `request_id_suffix = f", request_id={request_id}" if request_id is not None else ""` pattern. That pattern is exactly what this refactor exists to delete.--## The deeper insight (the senior-engineer take)--Text interpolation of contextual identifiers into log message strings is a 2010-era anti-pattern. In 2025:--- **Each log record is a dict.** Message body is human-readable narrative; identifiers, request scope, user scope, error codes, durations are *fields*. Sinks consume them as fields, not as substrings to grep.-- **Request-scoped attributes ride on contextvars.** Bind once at the entry boundary; let a logging filter / processor stamp them onto every record in scope. The author of an individual `log.info(...)` call shouldn't be able to forget the request id, because they were never asked to include it in the first place.-- **f-strings stay for the narrative.** `log.info("Storage delivery completed", files=n)` — not `log.info(f"Storage delivery completed: files={n}, request_id={r}, pipeline_run_id={p}")`.--Why a senior systems engineer cares (each row is a property the current text-mode design fails at):--| Property | Text-interpolated | Structured + contextvar |-|---|---|---|-| Queryable in Datadog / Honeycomb / Loki | grep over message strings | exact field match, joins, aggregations |-| Survives a field rename (`pipeline_run_id` → `run_id`) | breaks every dashboard | one schema bump |-| Correlates with traces / spans | regex parse | attribute join via OTel |-| New log line in the request path | each author must remember to interpolate the context | auto-stamped by filter, can't be forgotten |-| Long / binary / escape-sensitive values | breaks formatting | structured field handles it |-| Sampling / routing by attribute | not possible | first-class |-| Asymmetry between success and failure paths | requires discipline; we already drifted | impossible by construction |--The asymmetry row is the load-bearing one: it's precisely the gap the track-delivery doc was trying to close on the failure path. With structured + contextvar, that gap can't exist.--## What good looks like (destination shape, not the plan)--Sketch of the steady-state API and behavior we're aiming at — leave room for the kickoff session to refine the exact ergonomics:--```python-# Entry boundary (API request handler, Temporal activity start, CLI command):-with bind_log_context(request_id=req_id, user_id=user_id):- await pipe_run.run(pipe_job)--# Inside the request path — any module, any depth:-log.info("Storage delivery completed", files=len(result_files), pipeline_run_id=pipeline_run_id)-# Sink receives: {"msg": "Storage delivery completed", "files": 7, "pipeline_run_id": "...",-# "request_id": "...", "user_id": "...", "level": "INFO", "ts": "..."}-```--Key properties of the destination state:--- `pipelex.log` gains a structured-field surface (likely `**fields` kwargs, or an explicit `extra: dict` — to settle in the kickoff session).-- A `ContextVar`-backed context manager (`bind_log_context`) sets request-scoped attributes; a logging filter / processor reads them and merges into every record.-- The two boundaries that bind today's `request_id` continue to bind it — but at the *context* level, not by threading kwargs through 4-deep call stacks. The API request handler binds; `PipeRun.run` binds for non-API direct-mode invocations (whatever metadata it has); the Temporal activity entry re-binds from `DeliveryActivityArg.*` (the wire-format fields stay; only the propagation mechanism inside the activity changes).-- The manual `request_id` kwargs on `DeliveryExecutor.execute` / `_store_results` / `_notify_webhook` are **deleted**. So is the `request_id_suffix` interpolation. The success log lines auto-pick up `request_id` from the filter; the failure log lines do too; the exception messages can stop carrying it (and stop pretending to).-- Non-API invocations (CLI, direct library use, tests) simply don't bind `request_id`; the filter sees None; the field is absent from the record. No conditional suffix code anywhere.-- **Event-name emission on delivery paths.** Add an `event` field to the delivery log lines — `event=webhook_delivery` / `event=webhook_failure` on the webhook path, `event=storage_delivery` / `event=storage_failure` on the storage path. This is folded into this refactor rather than done standalone because the lines it touches (`delivery_executor.py` around 239 / 243 / 280 / 282 / 285) are exactly the `request_id_suffix` interpolation / kwarg-threading sites listed as deletion targets in "Reality of the current code" below — a separate event-name pass would touch them, then this refactor would touch them again. Emit the four `event=...` fields once the structured-field surface lands.--## Reality of the current code (what we found during the trigger discussion)--The `pipelex.log` public API is **string-shaped**:--```python-# pipelex/tools/log/log.py-def info(self, content: str | Any, title: str | None = None, inline: str | None = None): ...-def warning(self, content: str | Any, ..., problem_id: str | None = None): ...-def error(self, content: str | Any, ..., include_exception: bool = False, problem_id: str | None = None): ...-```--No `extra=` kwarg, no `**fields`, no structured pathway. Dispatch flows through `pipelex/tools/log/log_dispatch.py` → `log_formatter.py`. **Whether the dispatch + formatter layers could thread structured attrs through to the underlying stdlib `LogRecord.extra` without breaking existing sinks is the first thing to verify in the kickoff session** (see open questions).--Specific sites currently doing manual interpolation / threading of `request_id` and `pipeline_run_id` (these become the regression-test fixtures for the refactor, and the places we delete code):--- `pipelex/pipe_run/delivery_executor.py`- - `execute(... request_id: str | None = None)` — the threaded kwarg. To delete.- - `_store_results(... request_id: str | None = None)` — same. To delete.- - `_notify_webhook(... request_id: str | None = None)` — same. To delete.- - Success log lines (lines around 239 and 280) interpolate `request_id_suffix`. To revert to plain narrative.- - Failure exception messages (lines 243, 282, 285) interpolate `pipeline_run_id`. Probably revert to plain narrative; the catch site's log carries the structured fields.-- `pipelex/pipe_run/pipe_run.py`- - Debug entry log (around line 79) interpolates `pipeline_run_id` and `status`.- - Pipe-execution-failed log (around line 55) interpolates `pipeline_run_id`.- - Delivery-also-failed and tracer-close-also-failed logs (around lines 64-68 and 93-97) interpolate `pipeline_run_id`.- - The kwarg pass-through `request_id=pipe_job.job_metadata.request_id` on line 88 — to delete (binding happens at context-manager level instead).--These are the sites *we found during the trigger discussion*. A full sweep across the codebase for `f"...pipeline_run_id={...}..."` and similar patterns is part of the kickoff session's discovery step.--## Constraints, risks, and unknowns to size--- **Log sink consumers.** Whatever currently consumes pipelex log output (Temporal activity logs, container stdout, ECS log driver to CloudWatch, dev-time console formatter) needs to handle records with new structured attrs. The container path is probably fine (CloudWatch ingests JSON nicely); the local dev formatter (`log_formatter.py`) needs review — it likely pretty-prints strings today and will need a structured-aware mode.-- **Temporal worker boundary.** ContextVars do NOT propagate across the activity wire automatically. The current code passes `request_id` as an explicit field on `DeliveryActivityArg` — that stays. What changes is that the activity's entry code re-binds the contextvar from the arg, so everything *inside the activity* gets auto-stamped. Same pattern for any other cross-process boundary.-- **Async / task boundaries within a single process.** ContextVars do propagate across `asyncio` tasks created from a context where the var is bound (Python 3.7+ semantics). Worth a focused test, especially around `asyncio.create_task` and the `PipeRun.run` finally-block delivery dispatch.-- **Non-API callers.** CLI, tests, direct library use — these have no `request_id`. Filter must tolerate unbound contextvars cleanly (emit records with the field absent, not with the literal string "None"). Tests should pin this.-- **Backwards compatibility on `log.error` / `log.warning`.** They have extra kwargs today (`include_exception`, `problem_id`). The new structured-field surface needs to coexist with those without ambiguity. Likely cleanest: explicit `extra: dict[str, Any] | None = None` rather than `**fields`, to keep the existing kwarg surface unambiguous.-- **Existing tests that grep log strings.** Any test that does `assert "request_id=" in caplog.text` or similar needs to migrate to `assert caplog.records[0].request_id == ...`. That's a one-time mechanical fix but it's broad.-- **No backward-compat shim.** Per CLAUDE.md, we don't ship deprecation transitions. The refactor lands as a single coherent change; manual threading on `DeliveryExecutor` is deleted, not double-supported.--## First open questions for the kickoff session--The session should answer these before writing the plan:--1. **Does `log_dispatch` + `log_formatter` already preserve `LogRecord.extra` end-to-end, or does the formatter layer drop it?** This determines whether the refactor is half-a-day (preserve and surface) or several days (rebuild the formatter / sink pathway). Read `pipelex/tools/log/log_dispatch.py` and `log_formatter.py` first.-2. **What's the destination wire format for production sinks?** If we're emitting to CloudWatch / Datadog / Honeycomb, the formatter needs a JSON mode for prod and a human-readable mode for local dev. What's wired up today?-3. **Structured-field API shape:** explicit `extra: dict | None` parameter, or `**fields: Any` kwargs? The explicit-dict variant is unambiguous against existing kwargs (`title`, `inline`, `problem_id`, `include_exception`); the `**fields` variant reads cleaner at call sites. Pick one consciously.-4. **Bind-once boundaries:** confirm the entry points. API request handler is one. `PipeRun.run` for direct-mode is another. Temporal activity entry is a third. CLI command entries are a fourth. Anything else?-5. **Field naming convention.** `request_id` or `request.id` (dotted)? Plural / singular for collections? Standardize before stamping it across the codebase.-6. **Existing observability conventions.** Does pipelex already emit anything OTel-flavored? If so, align field names with the OTel semantic conventions where they exist (`http.request.id`, `user.id`, etc.) so traces and logs join cleanly downstream.--## Sequencing--- **Do not start until the API-readiness work merges.** That is the trigger context; this refactor touches the same files.-- **Do not start until the error-handling stack settles.** Several `wip/error-handling/track-*.md` items are still open; they shouldn't be racing this refactor through the same files.-- **Branch:** `refactor/structured-logging` (or similar — confirm at branch-creation time).-- **Scope discipline:** the refactor is the API extension + the contextvar machinery + the migration of the request-scoped fields we know about (`request_id`, `pipeline_run_id`, `user_id`). It is NOT "rewrite every log line in the codebase." Other call sites migrate opportunistically as they're touched.--## References--- `wip/error-handling/track-delivery-error-path-request-id.md` — the trigger doc; closing rationale references this refactor as the deferred altitude fix.-- The success-path threading in `DeliveryExecutor` — to be partially reverted.-- The direct-mode forwarding from `PipeRun.run` to `DeliveryExecutor` — to be partially reverted.-- `pipelex/tools/log/log.py` — current string-shaped Log API. The surface to extend.-- `pipelex/tools/log/log_dispatch.py` — dispatch layer. First file to investigate (open question 1).-- `pipelex/tools/log/log_formatter.py` — formatter layer. Second file to investigate (open questions 1 and 2).-- `pipelex/pipe_run/delivery_executor.py` — site of the manual threading. Deletion target.-- `pipelex/pipe_run/pipe_run.py` — site of additional uncorrelated failure logs (lines 55, 64-68, 79, 93-97 approximately). Deletion / migration target.-- Python `contextvars` docs — https://docs.python.org/3/library/contextvars.html-- OpenTelemetry log data model + semantic conventions — for field-naming alignment (look up at session start).diff --git a/wip/structured-validation-errors-deferred-findings.md b/wip/structured-validation-errors-deferred-findings.mddeleted file mode 100644index 9d6263d8b..000000000--- a/wip/structured-validation-errors-deferred-findings.md+++ /dev/null@@ -1,52 +0,0 @@-# Deferred: structured validation-errors follow-ups--Code-review follow-ups on the structured wiring/concept/type validation-errors work (`feature/Validate-api-render-format`). The two consumer-facing findings from that review (concept-owned refs mislabeled as `pipe_validation`; the missing dependency only in the message) were **fixed** in that change. The items below were judged lower-severity and **deferred** here rather than fixed inline — each is a contained, independent pickup. None is a silent correctness regression; they are coverage gaps, asymmetries, and one cleanup.--The surface: `validate_bundle` → `_translate_to_validate_bundle_error` (`pipelex/pipeline/validate_bundle.py`) → `build_validation_error_items` (`pipelex/pipeline/validation_errors.py`) → `ErrorReport.validation_errors[]` (`ValidationErrorItem` in `pipelex/base_exceptions.py`). The three categorizing sites are `categorize_blueprint_validation_error` (`pipelex/core/interpreter/validation_error_categorizer.py`), `validate_concept_references_in_blueprints` (`pipelex/libraries/concept_reference_validation.py`), and `Library.validate_pipe_library_with_libraries` (`pipelex/libraries/library.py`).--## 1. Missing-`type` pipe not categorized (`union_tag_not_found`)--`categorize_blueprint_validation_error` only categorizes `error["type"] == "union_tag_invalid"` as `UNKNOWN_PIPE_TYPE` (the case where a pipe declares a `type` matching no known operator/controller). A pipe block that **omits `type` entirely** raises pydantic `union_tag_not_found`, which the branch does not match — so it falls through to the uncategorized `log.warning(...)` residual with no `error_type` / `pipe_code`.--This is asymmetric with the present-but-unknown-`type` case the feature fixes: both are "bad/absent pipe type" from the author's view, but only one is categorized.--**Shape when picked up:** match both `union_tag_invalid` and `union_tag_not_found` in that branch (the loc is still `("pipe", <code>)`, so `pipe_code` populates the same way), and add a fixture with a `[pipe.foo]` block that omits `type`. Verify pydantic's exact error code for the missing-discriminator case first.--## 2. Dependency validation fails fast; only the first missing sub-pipe is reported--In `Library.validate_pipe_library_with_libraries`, the `raise LibraryLoadingError(...)` for an unresolved dependency sits **inside** the `for sub_pipe_code in pipe.pipe_dependencies()` loop, so a controller with several missing dependencies surfaces only **one** `unresolved_pipe_dependency` item. This contrasts with `validate_concept_references_in_blueprints`, which deliberately accumulates every unresolved reference and reports them together ("the author sees all of them at once").--**Shape when picked up:** accumulate `PipesAndConceptValidationErrorData` items across the dependency loop (and likely across pipes) and raise once at the end, mirroring the concept-ref accumulation. Note this is a behavioral change to a fail-fast path that predates the feature; scope the blast radius (other callers of `validate_pipe_library_with_libraries` rely on it raising on the first failure) before changing it.--## 3. Qualified same-domain ref yields a qualified `concept_code`--In `validate_concept_references_in_blueprints`, the pipe-owned `unresolved_concept` item sets `concept_code=concept_ref_or_code` — the **raw** ref. For a same-domain ref that the author happened to qualify (e.g. `output = "mydomain.Foo"` in domain `mydomain`), `ref.is_external_to("mydomain")` is `False`, so it reaches the unresolved branch and `concept_code` becomes the domain-qualified string `"mydomain.Foo"` rather than the bare `Foo`. The categorizer's concept path and the bare-ref test (`concept_code == "NonExistentConcept"`) assume the bare code, so a consumer keying on a bare `concept_code` mismatches when the author qualified the ref.--**Shape when picked up:** decide the canonical form for `concept_code` (bare local code vs. as-written) and normalize at the construction site — likely `ref.local_code` for the bare form — then add a fixture with a qualified same-domain output ref.--## 4. `unresolved_pipe_dependency` carries no `source`--The `unresolved_pipe_dependency` item omits `source`, while the sibling `unresolved_concept` and `unknown_pipe_type` items carry it. Root cause: at library-validation time `library.py` builds the item from the `pipe` object, and `PipeAbstract` has no `source` field — so there is no per-pipe declaring-file path to attach. A consumer mapping `validation_errors[]` to declaring files (the vscode cross-file diagnostics path) can attribute the concept/type failures but not the dependency one.--This is a **model limitation**, not a regression. Fixing it means threading the declaring source onto the pipe model (or looking it up by domain at validation time) — non-trivial.--Related but distinct: [`validate-parse-level-source-attribution.md`](validate-parse-level-source-attribution.md) covers the *parse-level* (malformed-TOML) source-less residual. Both are about `source` on `validation_errors[]`, but that one is in the parse comprehension and this one is in library validation — they would be fixed in different places.--## 5. (cleanup) Per-reference failure sentence is built twice--In `validate_concept_references_in_blueprints`, each unresolved reference builds two near-identical sentences in the same iteration: the `undeclared` aggregate-message entry (`'{ref}' in {context} is not declared in domain '{domain}' (source: '{source}')`) and the per-item `item_message` (`Concept '{ref}' in {context} is not declared in domain '{domain}' and is not native.`). A future wording change must touch both or the aggregate `ConceptLibraryError` message and the structured item silently diverge.--**Shape when picked up:** derive one from the other (or from a single small formatter), so there is a single phrasing of "this reference does not resolve."--## Known trade-off baked in by the concept-owned fix (revisit only if it bites)--The concept-owned-ref fix made `concept_code` mean different things across categories for `error_type == unresolved_concept`:--- pipe-owned (`pipe_validation`): `pipe_code` = owning pipe, `concept_code` = **missing** concept.-- concept-owned (`blueprint_validation`): `concept_code` = **owning** concept (consistent with how every other `blueprint_validation` concept error is categorized); the missing reference is named in the `message`.--A consumer disambiguates via `category`, and both branches carry comments. The clean-but-larger alternative would be distinct "owner" and "missing/referenced" locator fields on the wire item (and on `PipesAndConceptValidationErrorData` / `PipelexBundleBlueprintValidationErrorData`) so `concept_code` need not double as both. Deferred as a wire-model design question, not worth the churn unless a consumer needs the owner concept structurally in the pipe-owned case (or the missing concept structurally in the concept-owned case).--## Cross-repo note--Any change here that alters the `validation_errors[]` wire shape (new locator fields, normalized `concept_code`) must be mirrored in `conformance/conformance/validation_contract.py` + `conformance/conformance/assertions/json_output.py`, per the spec/conformance sync rule, and the conformance suite run on both arms.diff --git a/wip/stuffs-as-nodespec.md b/wip/stuffs-as-nodespec.mddeleted file mode 100644index b3c2ebc8f..000000000--- a/wip/stuffs-as-nodespec.md+++ /dev/null@@ -1,63 +0,0 @@-# Stuffs as first-class NodeSpec--## Why this exists--When we promoted `description` and `domain_code` onto `NodeSpec` for pipes, we deliberately stopped at pipes. Concepts have the same metadata (`domain_code`, `description`) and the same problem: today, the only way for a renderer to discover the domain or human description of a stuff's concept is to join with `GraphSpec.concept_registry` — and that registry is empty when `data_inclusion.pipe_and_concept_registry = False`.--So the symmetric fix is to make stuffs self-describing in the graph, just like pipe nodes now are.--This is a starter, not a worked plan. Treat the design below as a starting point — the code will likely have moved by the time we pick this up, and several of the choices below are reversible.--## Current shape (snapshot)--- `NodeKind` already defines `INPUT`, `OUTPUT`, `ARTIFACT` — so the type system says `NodeSpec` is meant to span pipes and stuffs.-- In practice today, only pipe-shaped `NodeSpec`s ever get emitted. `GraphTracer.on_pipe_start` and `GraphSpecAssembler._handle_pipe_start` create `PIPE_CALL` / `CONTROLLER` / `OPERATOR` nodes only.-- Stuffs live as `IOSpec` entries inside `NodeSpec.node_io.inputs` / `.outputs`.-- Renderers (mermaidflow, reactflow) synthesize stuff "nodes" inline at render time from `IOSpec` data, using digest-derived IDs — they never enter the model.--That inline synthesis is the asymmetry: pipe NodeSpecs are model-level, stuff nodes are render-level.--## What this rework would do--Make stuffs first-class `NodeSpec` instances:--- Tracer (and replay assembler) emit a `NodeSpec` per stuff with `kind` ∈ {`INPUT`, `OUTPUT`, `ARTIFACT`}.-- The stuff's concept metadata lives on the same `NodeSpec.description` / `NodeSpec.domain_code` fields we just added — same shape, semantics depend on `kind`:- - On a pipe-call node: pipe's `description` and `domain_code`.- - On a stuff node: the concept's `description` and `domain_code`.-- Pipe ↔ stuff relationships move out of `IOSpec` lists and into `EdgeSpec` entries (data edges already exist; this would extend their role).-- `IOSpec` goes away (or becomes a thin reference to the stuff node by `node_id`). One source of truth.--## Decisions taken in the discussion (revisable)--These were chosen quickly in chat and should be re-examined when picking this up:--- **One source of truth.** Stuffs as `NodeSpec` replaces `IOSpec`, not in parallel with it. Two-source designs were rejected to avoid drift between the in-model graph and the rendered one.-- **`NodeSpec.description` / `.domain_code` are polymorphic by `NodeKind`.** Same fields, semantics change by kind. The alternative (separate `concept_description` / `concept_domain_code` fields) was considered uglier given the taxonomy already exists.-- **`INPUT` / `OUTPUT` / `ARTIFACT` are semantic, not visual-only.** `INPUT` = pipeline entrypoint, `OUTPUT` = pipeline final, `ARTIFACT` = intermediate stuff. The tracer/assembler is responsible for classifying — renderers should not need to derive this from edge topology.--If any of these no longer fits when we revisit, change them. The starter doesn't lock anything in.--## Likely impact surface--Not exhaustive — just the obvious ones to scope from before committing:--- `pipelex/graph/graph_tracer.py` and `pipelex/tracing/graphspec_assembler.py` — emit stuff nodes, classify kind, drop or rewrite `IOSpec` aggregation.-- `pipelex/graph/graphspec.py` — `NodeSpec` field semantics doc, possibly retire `IOSpec` / `NodeIOSpec`.-- `pipelex/tracing/trace_events.py` — `PipeStartEvent.input_specs` rethink (events probably emit stuff-node ids instead of inline `IOSpec`).-- Every renderer that today synthesizes stuff nodes at render time (mermaidflow, reactflow JS bundle, HTML).-- All `GraphSpec` JSON fixtures under `tests/data/graphs/` — breaking change, will need regeneration.-- The existing pipe-side `description` / `domain_code` work has already landed — confirm semantics stay consistent when extended to stuff kinds.--## Open questions to settle before starting--- Stuff node identity: derive `node_id` from `stuff_code` (digest), or assign sequenced IDs like pipe nodes?-- Concept reference on the stuff node: still keep a `concept` string (qualified `domain.code`)? Drop it now that `domain_code` is a real field?-- Migration story for older `GraphSpec` JSON files. Probably "no back-compat" per repo policy, but worth confirming.-- `ERROR` `NodeKind` — same treatment as data kinds, or stays pipe-attached?--## Out of scope for this future work--- The pipe-side metadata wiring is already done — don't redo it.-- Adding new `NodeKind`s.-- Cross-pipeline concept dedup beyond what `concept_registry` already does.diff --git a/wip/validate-parse-level-source-attribution.md b/wip/validate-parse-level-source-attribution.mddeleted file mode 100644index 60e472744..000000000--- a/wip/validate-parse-level-source-attribution.md+++ /dev/null@@ -1,32 +0,0 @@-# Deferred: thread `mthds_source` through parse-level validate failures--Code-review follow-up (greptile P1) on pipelex PR #992 (`feature/Tweaks-for-validation-api`). **Deferred** — recorded here for scoping rather than fixed in this PR, because it is a non-trivial runtime parse-path change, not a clear/small win.--## The gap--In `pipelex/pipeline/validate_bundle.py` (~line 286–291), the multi-file load builds blueprints per content with the source threaded in:--```python-content_sources = list(mthds_sources) if mthds_sources is not None else [None] * len(mthds_contents)-loaded_blueprints = [- PipelexInterpreter.make_pipelex_bundle_blueprint(mthds_content=content, mthds_source=source)- for content, source in zip(mthds_contents, content_sources, strict=True)-]-```--When one content has **malformed TOML**, `make_pipelex_bundle_blueprint()` raises *before* it seeds `mthds_source` into the blueprint data. The exception propagates out of the comprehension to the enclosing `_translate_to_validate_bundle_error(category="pipe")` context manager, which (with the structured-info GATE) yields a last-resort `blueprint_validation` residual **with no `source`**. So `validate_bundle(mthds_contents=[good, bad], mthds_sources=["a.mthds","bad.mthds"])` returns a structured error that cannot be attributed to `bad.mthds`.--## Why it matters--The vscode cross-file diagnostics path routes each `validation_errors[].source` to the owning editor file. A source-less parse error is unroutable (it also has no `pipe_code`/`concept_code` to fall back on), so a syntax error in a multi-file API request lands nowhere useful in the editor. (The PR #50 path-boundary fix in `crossFileDiagnostics.ts` only helps once a `source` exists.)--## Why it's deferred (not a quick patch)--- It needs the comprehension restructured into a per-item loop with a `try`/`except` around `make_pipelex_bundle_blueprint`, catching the **specific** interpreter parse exception (must be identified — no speculative `except`), so the current `source` can be attached.-- The residual category is currently `pipe` for this block (the `with _translate_to_validate_bundle_error(category="pipe")` scope); a parse-level failure is arguably `blueprint_validation`. Fixing attribution may also mean correcting the category — a second judgment call.-- The source must be plumbed into the residual item built by `build_validation_error_items` (`pipeline/validation_errors.py`) — i.e. convert the per-content parse failure into a `blueprint_validation` item carrying that source, rather than the generic translated residual.-- All of this sits on top of the Phase-2 + GATE logic; it carries regression risk against the just-landed structured-info invariant and deserves its own focused change + tests.--## Suggested shape (when picked up)--Loop per (content, source); on the specific parse exception, raise/emit a `blueprint_validation` `ValidationErrorItem` with `source=<this source>` and the parse message, so the GATE residual is no longer source-less. Add a unit test: two contents, the second malformed, assert the returned `validation_errors[]` has one item whose `source` is the second source. Then the conformance HTTP arm can assert parse-level source attribution on the real wire.diff --git a/wip/webhook-signing.md b/wip/webhook-signing.mddeleted file mode 100644index e6a3f4a52..000000000--- a/wip/webhook-signing.md+++ /dev/null@@ -1,228 +0,0 @@-# Webhook signing — security track--This plan covers webhook signing for the `pipelex → pipelex-api → external receiver` completion-callback flow. It is **independent of the error-handling refactor** ([`error-handling/`](error-handling/)) — they have no functional dependency on each other.--Splitting signing into its own track lets reviewers evaluate it on security merit: trust topology, threat model, tamper detection, rollout safety.-------## Background — what exists today--### Dispatcher-side signing in pipelex-api--`pipelex-api/api/routes/pipelex/pipeline.py:44-56` computes a signature **at dispatch time** when wiring up the webhook target. The signed bytes are **only the `pipeline_run_id`**, and the resulting hex is baked into the `WebhookTarget.headers` dict that rides through pipelex unchanged:--```python-def _completion_signature(pipeline_run_id: str) -> str:- secret = get_required_env("COMPLETION_CALLBACK_SECRET")- return hmac.new(secret.encode("utf-8"), pipeline_run_id.encode("utf-8"), hashlib.sha256).hexdigest()--# At dispatch:-WebhookTarget(- url=url,- headers={"X-Completion-Signature": _completion_signature(resolved_pipeline_run_id)},-)-```--### Pipelex itself--Pipelex has **zero signing code**. `pipelex/pipe_run/delivery_executor.py:_notify_webhook` POSTs the payload with whatever headers it was handed and never reads a secret. It is a generic forwarder.--### Threat model gap--A MitM who can rewrite the webhook body in transit can change `status="completed"` → `status="failed"`, mutate `result_url`, and — after the error-handling refactor's Item D-2 lands — rewrite the entire `error` classification payload (`error_type`, `retryable`, `provider`, `user_action`, etc.). The current header signature does nothing to detect this; the signed bytes are just `pipeline_run_id`, which is in the URL anyway.-------## Trust topology — today vs. proposed--| | Today | Proposed |-|---|---|---|-| **Who signs** | pipelex-api dispatcher | pipelex worker |-| **What's signed** | `pipeline_run_id` only | Full serialized request body |-| **Where the secret lives** | One process (the dispatcher) | Every pipelex worker process |-| **Trust blast radius if compromised** | Dispatcher | Any worker |-| **Header format** | `X-Completion-Signature: <hex>` (bare hex) | `X-Completion-Signature: sha256=<hex>` (algorithm-prefixed) |--The proposed pattern matches Stripe / GitHub / standard webhook-signing best practice. The cost is broader secret distribution — every worker needs it. This is acceptable because workers already run in trusted infrastructure; their attack surface dominates the marginal risk of also holding this secret.-------## Decisions locked in--- **Secret source: env var only, no config field.** The worker reads `PIPELEX_WEBHOOK_SIGNING_SECRET` from its environment. There is no `pipelex.toml` field for it. Rationale: pipelex's repo policy is "no secrets in any committed config file." Consistent with how provider API keys (`OPENAI_API_KEY`, etc.) are handled.-- **Same env-var name on both sides.** Worker and receiver both read `PIPELEX_WEBHOOK_SIGNING_SECRET`. The shared-secret nature is named explicitly. The old `COMPLETION_CALLBACK_SECRET` is renamed in pipelex-api's deployment manifests as part of the migration.-- **Algorithm: HMAC-SHA256.** Standard, fast, well-supported.-- **Header: `X-Completion-Signature: sha256=<hex>`.** Algorithm-prefixed format. The bare-hex format used today becomes incompatible — see rollout.-- **Verification: constant-time via `hmac.compare_digest`.** Prevents timing-side-channel leaks.-- **Failure mode: loud, not silent.** If a webhook is configured but the secret is missing, raise `PipelexConfigError` at delivery time. No unsigned send, ever.-- **Empty webhooks list does not require the secret.** Cost of running pipelex without webhooks should not increase.-- **Body bytes signed = body bytes sent.** Sign the exact `bytes` posted to httpx (`content=body_bytes`), not the dict (`json=payload`). `httpx`'s JSON serialization is otherwise free to add whitespace, breaking signature verification.-------## Approach — 3-step rollout--The signature format changes from bare hex to `sha256=<hex>`, and from `pipeline_run_id`-signing to body-signing. A naive single-PR merge breaks any in-flight webhooks during the deploy window. The correct sequence is three deploys, each reversible on its own.--### Step 1 — Receiver supports both old and new formats--The pipelex-api receiver verifies webhooks **both ways**:--- If the `X-Completion-Signature` header starts with `sha256=`, take body bytes, recompute, compare.-- Otherwise (bare hex, no algorithm prefix), assume legacy format, recompute from `pipeline_run_id`, compare. Log a deprecation warning identifying the request.--Ship and deploy first. Existing pipelex workers (still signing the old way at dispatch time) keep working.--### Step 2 — Pipelex worker switches to body-signing--`pipelex/pipe_run/delivery_executor.py:_notify_webhook` computes HMAC-SHA256 over the request body bytes and sets `X-Completion-Signature: sha256=<hex>`. The dispatcher (`pipelex-api/api/routes/pipelex/pipeline.py`) **stops** populating the header at dispatch time — the worker is now the signer end-to-end.--Ship and deploy. Receivers verify via the new sha256-prefixed branch.--### Step 3 — Remove the legacy fallback--The pipelex-api receiver drops the bare-hex verification branch and the `_completion_signature` helper.--**Pre-condition:** Step 2 has been deployed across all pipelex workers, AND the deprecation log line from Step 1 has gone quiet for at least one full retention window (typically a week). The deprecation log is the safety net — if any legacy sender is still in production, the log surfaces it before the fallback is removed.--This rollout is the **single biggest difference** from the original Item F framing in the error-handling plan, which proposed lockstep merge of both repos. Lockstep merge breaks any in-flight webhook during the deploy window; the 3-step rollout does not.-------## The work--### [ ] Item 1 — Receiver-side dual-format verification--- **Files:**- - `pipelex-api/api/routes/pipelex/pipeline.py` — replace `_completion_signature(pipeline_run_id)` with `_verify_signature(header, body, pipeline_run_id) -> bool` that handles both formats.- - Deployment manifests + `.env.example` — rename `COMPLETION_CALLBACK_SECRET` → `PIPELEX_WEBHOOK_SIGNING_SECRET`. Value unchanged; same shared secret.-- **Surface:**- ```python- def _verify_signature(header: str | None, body: bytes, pipeline_run_id: str) -> bool:- if header is None:- return False- secret = get_required_env("PIPELEX_WEBHOOK_SIGNING_SECRET").encode("utf-8")- if header.startswith("sha256="):- expected = hmac.new(secret, body, hashlib.sha256).hexdigest()- return hmac.compare_digest(header.removeprefix("sha256="), expected)- # Legacy: bare-hex over pipeline_run_id. Drops in Step 3.- log.warning("Legacy webhook signature format received; sender should upgrade.")- expected = hmac.new(secret, pipeline_run_id.encode("utf-8"), hashlib.sha256).hexdigest()- return hmac.compare_digest(header, expected)- ```-- **Tests:**- - sha256-prefixed format verifies a body-bytes signature correctly.- - Bare-hex format still verifies a pipeline_run_id signature correctly.- - Bare-hex format triggers the deprecation log.- - Wrong signature in either format returns `False`.- - Missing header returns `False`.-- **Acceptance:** receiver works with both old and new pipelex workers during the rollout window.--### [ ] Item 2 — Pipelex-side body-bytes signing--- **Files:**- - `pipelex/pipe_run/delivery_executor.py:_notify_webhook` — compute HMAC-SHA256 over the request body bytes; set `X-Completion-Signature: sha256=<hex>` header before POST. Read `PIPELEX_WEBHOOK_SIGNING_SECRET` from `os.environ` at delivery time. Raise `PipelexConfigError` if the secret is missing **and** webhooks are configured.- - `pipelex-api/api/routes/pipelex/pipeline.py` — **stop** populating `X-Completion-Signature` at dispatch time. The `WebhookTarget.headers` no longer carries it. Pipelex now owns signing end-to-end.-- **Surface (sketch):**- ```python- async def _notify_webhook(self, pipeline_run_id, status, result_url, webhook) -> None:- payload: dict[str, Any] = dict(webhook.payload)- payload["pipeline_run_id"] = pipeline_run_id- payload["status"] = status- if result_url is not None:- payload["result_url"] = result_url-- # Sign the exact bytes we send. `content=body_bytes` below — not `json=payload` —- # so httpx cannot reformat the JSON after signing.- body_bytes = json.dumps(payload, separators=(",", ":")).encode("utf-8")- secret = os.environ.get("PIPELEX_WEBHOOK_SIGNING_SECRET")- if secret is None:- msg = (- "Webhook signing secret not configured. "- "Set PIPELEX_WEBHOOK_SIGNING_SECRET in the worker environment."- )- raise PipelexConfigError(msg)- signature = hmac.new(secret.encode("utf-8"), body_bytes, hashlib.sha256).hexdigest()- headers = {- **webhook.headers,- "X-Completion-Signature": f"sha256={signature}",- "Content-Type": "application/json",- }-- async with httpx.AsyncClient() as client:- response = await client.post(webhook.url, content=body_bytes, headers=headers, timeout=30.0)- response.raise_for_status()- ```-- **Tests** (`tests/unit/pipelex/pipe_run/test_delivery_executor.py`):- - Signature is deterministic for fixed (secret, body) pairs.- - Signature matches `hmac.new(secret, body, sha256).hexdigest()` (compare against the receiver-side computation in Item 1).- - Flipping a single byte in the body produces a different signature.- - Missing-secret-with-webhook-configured raises `PipelexConfigError` and the message names the env var.- - Webhook with no recipients (empty `delivery_assignment.webhooks`) succeeds without requiring the secret.- - The exact bytes posted to httpx match the exact bytes signed (no whitespace drift from JSON serialization).-- **Acceptance:** rewriting `status`, `result_url`, or `error` in transit causes signature verification to fail on the receiver.--### [ ] Item 3 — Drop legacy fallback in receiver--- **Files:**- - `pipelex-api/api/routes/pipelex/pipeline.py` — remove the bare-hex branch and the legacy verification path; `_verify_signature` becomes the sha256-only path.-- **Pre-conditions:**- - Item 2 deployed to all pipelex workers.- - Deprecation log line from Item 1 has been quiet for one full retention window (typically a week).-- **Tests:** existing tests for sha256-prefixed verification continue to pass; bare-hex tests are deleted.-- **Acceptance:** code surface is now small. One signing pattern, one secret, one verification path.-------## End-to-end acceptance test--Independent of per-item tests, this scenario pins the cross-repo contract once Items 1 and 2 are deployed:--1. Dispatch a real pipeline run that triggers a webhook.-2. Observe the webhook arrive at a test receiver wired to `_verify_signature` from Item 1.-3. Verify the receiver accepts the sha256-prefixed signature.-4. **Flip one byte in the body** between worker-side signing and receiver-side verification (e.g. via an httpx-MockTransport interceptor).-5. Verify the receiver rejects the tampered body.-------## Deferred follow-ups (out of scope)--### Replay attack mitigation--The current design protects against tamper-in-transit but **not against replay** — an attacker who captures a valid signed webhook can resubmit it. Standard mitigations: include a timestamp in the signed payload, have the receiver enforce a freshness window (e.g. ±5 minutes), reject duplicates by `pipeline_run_id` if the receiver maintains a dedupe cache.--**Why deferred.** No current consumer has asked. The active threat model is "MitM rewrites body in transit," not "attacker captures and resubmits." Pick this up when a partner integration asks for it or when the threat model evolves.--### Secret rotation--Rotating `PIPELEX_WEBHOOK_SIGNING_SECRET` today requires coordinated env-var updates on worker and receiver, with a brief window where webhooks fail verification. Standard mitigation: support **two** secrets simultaneously (current + previous) — worker signs with the current, receiver tries both. Rotate by setting the previous secret, deploying, then dropping it in a follow-up deploy.--**Why deferred.** Secret rotation isn't operationally needed yet (no schedule, no compromise event). The dual-secret pattern is a small addition to the verifier when it's needed.-------## Tracking--| ID | Item | Pre-condition | Status |-|---|---|---|---|-| 1 | Receiver-side dual-format verification | — | [ ] |-| 2 | Pipelex-side body-bytes signing | Item 1 deployed | [ ] |-| 3 | Drop legacy fallback in receiver | Item 2 deployed + deprecation log quiet | [ ] |-------## Cross-repo coordination--This plan spans the pipelex and pipelex-api repos. The 3-step rollout is intentionally ordered across separate PRs in separate repos — no lockstep deploy.--The pipelex-api companion plan (when written) lives at `pipelex-api/wip/security/webhook-signing.md` and describes the receiver-side changes from its own perspective.-------## How this relates to the error-handling refactor--The error-handling work (in [`error-handling/`](error-handling/)) puts a structured `error` dict on the webhook body. That meaningfully expands the tamper surface — a MitM body rewrite previously could change `status` / `result_url`, now it can also rewrite the classification.--That expanded tamper surface is **the motivating context** for accelerating this signing work, but the signing fix itself is correct independently. The two can land on independent schedules; there is no functional dependency.