Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Release v0.37.0
Louis Choquel96ee96c7f218
diff --git a/.badges/tests.json b/.badges/tests.jsonindex 07637ab1c..a51b2bced 100644--- a/.badges/tests.json+++ b/.badges/tests.json@@ -1,7 +1,7 @@{"schemaVersion": 1,"label": "tests",- "message": "7544",+ "message": "7814","color": "blue","cacheSeconds": 300}\ No newline at end of filediff --git a/CHANGELOG.md b/CHANGELOG.mdindex 60aa01490..facddb85d 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,41 +1,48 @@# Changelog-## [Unreleased]+## [v0.37.0] - 2026-07-04-### Changed+### Added-- **`PipeParallel` always combines (breaking):** A parallel now always combines its branch outputs into its declared `output` concept and stamps the combination as its main output — a pipe run always delivers a main stuff. The `combined_output` field is deleted from the MTHDS language: `output` is now the (single, truthful) combination concept. Migration: pipes declaring both fields just drop the `combined_output` line; `add_each_output`-only pipes replace their placeholder `output` with `Composite` or a structured concept matching the branch `result` names. `add_each_output` keeps its meaning (also expose each branch output by name) and now defaults to `false` with no one-of-two constraint.-- **Static validation of parallel outputs:** The declared `output` of a `PipeParallel` is validated at author time: it must be `Composite` or a structured concept whose fields are compatible with the branch `result` names (required fields ⊆ result names; result names ⊆ declared fields) **and types** — when a field is typed with a content class, the branch feeding it must produce that concept or one refining it (conceptual compatibility: a concept refining `Text` fits a `TextContent` field). Scalar native concepts (`Text`, `Image`, ...), `Dynamic`, `Anything`, multiplicity suffixes (`Foo[]`, `Foo[N]`), and type-incompatible branches are rejected by `/validate` and library loading instead of failing at runtime inside the combine step.-- **The main-stuff invariant is now enforced at every post-run boundary (breaking):** With every pipe stamping a main output, the defensive "maybe there is no main stuff" branches for completed runs are gone. Wire models make the field required (`main_stuff_name: str` on `PipelexPipeRunOutput` and on the `/execute` response extension), result delivery always writes the `main_stuff.*` artifact files (a completed run with no main stuff now fails delivery loudly instead of storing an empty envelope), and the graph tracer and OTel/Langfuse output capture read the main stuff directly. `PipeOutput.optional_main_stuff` is deleted — use `PipeOutput.main_stuff`; `WorkingMemory.get_optional_main_stuff()` remains for pre-run/empty memories.-- **Orchestrator SPI: the delivery axis is now two methods (breaking):** `OrchestratorProtocol.run` no longer takes a `delivery` parameter — it is the BLOCKING arm and returns the completed-run `PipelexPipeRunOutput` (which therefore requires `main_stuff_name` with no "not finished yet" escape hatch). The FIRE_AND_FORGET arm is the new `OrchestratorProtocol.start`, returning the new ids-only `PipelexPipeDispatchAck` (`pipeline_run_id` + `workflow_id`). `PipelexPipeRunOutput.is_completed` is deleted (it is always a completed output now); `DeliveryMode` remains the wire-input vocabulary that picks which method the endpoint calls. Orchestrators that cannot do genuine async (core's DIRECT) implement `start` by raising, unreachable behind the `supports_fire_and_forget` gate. Migration: orchestrator plugins split their `match delivery` bodies into the two methods; `/start`-serving runners call `orchestrator.start(...)` and read the ack's `workflow_id`.-- **`PipeParallel` honors `final_stuff_code`:** A requested final stuff code (e.g. pre-allocated by a `PipeBatch` for its branch outputs) is now stamped on the parallel's combined stuff instead of being discarded, so the combined output keeps its graph identity when a parallel runs as a batch branch.-- **`LLMWorkerInternalAbstract` folded into `LLMWorkerAbstract` (breaking):** The LLM worker family now matches the other three (img_gen, extract, search): the abstract base holds the `InferenceModelSpec` (`__init__(inference_model, reporting_delegate=None)`) and owns the whole job lifecycle — `llm_job_before_start` (started_at stamp + usage-report init), spec-driven capability checks (vision, documents), constraint application, and spec-based OTel naming. Previously those lived on the intermediate `LLMWorkerInternalAbstract` (a remnant of the pre-plugin-system "external plugin" era), so an out-of-tree worker subclassing the documented `LLMWorkerAbstract` silently inherited a template method with a hole in it: forgetting to call `llm_job_before_start` crashed reporting on a `None` duration. That state is now unrepresentable — a plugin worker implements `_gen_text`/`_gen_object` and nothing else. `LLMWorkerInternalAbstract` is gone; subclasses now extend `LLMWorkerAbstract` directly. The prompt/response debug dumps moved from the worker base to the `llm_generate` funnel (the dump helpers in `llm_utils` are now config-gated internally), keeping the worker ABC free of `pipelex.config` (the hub imports the ABC at module level, and config imports the hub — a config import from the ABC would be an import cycle).-- **Legacy `set_llm_worker_from_external_plugin` removed (breaking):** This pre-plugin-system path on the inference manager registered a spec-less worker class by handle — the old meaning of "external plugin". The real seam (a `pipelex.plugins` entry point registering `(family, sdk)` backends) fully replaces it.-- **Docs — cookbook inference-plugins page rewritten for the plugin system:** `docs/cookbook/using-inference-plugins.md` now documents the real cookbook example (an installable `pipelex.plugins` entry-point package serving the zero-key `hello-1` model) instead of the retired pre-plugin-system "plugin = config entry" walkthrough that referenced a model handle defined nowhere. Also fixed the stale claim that model config lives in `.pipelex/pipelex.toml` (it lives in `.pipelex/inference/`), and corrected the lookup-miss error named in `docs/under-the-hood/inference-backend-plugins.md` (`InferenceBackendNotFoundError`, not `NotImplementedError`).-- **Dry-run mock lists shrink from 3 to 2 items:** When a pipe input is declared as a list without a fixed size, the dry-run mock now fabricates 2 items instead of 3. Cuts dry-run work on list inputs without changing correctness.-- **Security: `transformers` bumped past CVE-2026-4372 (GHSA-29pf-2h5f-8g72):** The lockfile no longer resolves the vulnerable `transformers < 5.3.0` (a high-severity RCE through malicious model `config.json` files, pulled in transitively by the `docling` extra). To unblock the upgrade, the `huggingface` extra now requires `huggingface_hub>=1.5.0,<2.0.0` (breaking: was `>=0.23,<1.0.0`) — the hub 1.x line is requests→httpx based, but the surfaces Pipelex uses (`AsyncInferenceClient`, `HfHubHTTPError`, `InferenceTimeoutError`, provider literals) are unchanged.-- **Repo lint validates `.mthds` files against the locally generated MTHDS schema:** `plxt lint` in this repo now uses `derived/mthds_schema.json` (regenerated automatically by the lint make targets) instead of the released schema bundled with plxt, so language-surface additions like `size` are lintable here before they propagate to a plxt release.+- **Portable image size (`size`) for `PipeImgGen`:** New `size` parameter accepting portable tiers (`"1k"`, `"2k"`, `"4k"`) or exact pixel dimensions (e.g. `"2048x1152"`). A tier means "this pixel class at my chosen `aspect_ratio`", mapped to each provider's own grid; an exact size is deterministic (declaring `aspect_ratio` alongside it is a validation error). Unsatisfiable requests fail as hard validation errors at blueprint-load time, never warn-and-ignore. When `size` is unset, no size intent is sent and the provider default applies. The MTHDS JSON Schema exposes the field, and an optional `size` default is supported in `[cogt.img_gen_config.img_gen_param_defaults]`.+- **`native.Composite` concept:** New native concept backed by `CompositeContent` — an untyped, named composition holding sub-contents as top-level fields, serving as the default combination vehicle for parallel branches. Supports the full content surface: `smart_dump`, kajson/transport round-trip, and markdown/HTML rendering.+- **Gemini image-to-image (img2img) support:** The Google img-gen worker now forwards input images to the Gemini API as inline parts, enabling image editing and multi-image composition for the `nano-banana` models (including the new `nano-banana-2-lite`). Every img-gen worker now validates input images against the model's declared capability at job start, rejecting img2img on unsupported models with a clean `ImgGenParameterError` before any provider call.+- **Banner aspect ratios:** Support for extreme banner formats (`landscape_4_1`, `landscape_8_1`, `portrait_1_4`, `portrait_1_8`) from Gemini 3.1 image models. All other image-gen backends reject them with a clean parameter error.+- **Static validation for Google image models:** Google image models now use `rules` blocks (geometry taxonomies like `gemini_3_flash`) in the backend deck, catching unsupported aspect-ratio and size combinations at blueprint-load time. The Google img-gen factory is keyed by taxonomy instead of hardcoded model names (Breaking: the `GoogleImageGenModel` name enum is removed — model handles are deck config, not code constants).-### Added+### Changed-- **`native.Composite` concept:** New native concept backed by `CompositeContent` — an untyped named composition holding its sub-contents as top-level fields (no wrapper key). It is the combination vehicle for parallels whose authors don't want to declare a bespoke concept, and supports the full content surface: `smart_dump`, kajson/transport round-trip, and markdown/HTML rendering.-- **Portable image size for image generation (`size` on `PipeImgGen`):** A single `size` field carries a portable size intent across providers — either a size tier (`"1k"`, `"2k"`, `"4k"`: "this pixel class at my chosen `aspect_ratio`", mapped to each provider's own grid or computed dimensions) or an exact pixel size like `"2048x1152"` (deterministic; setting `aspect_ratio` alongside an exact size is a validation error since the size implies the ratio). Unsatisfiable requests are hard validation errors at blueprint-load time (or at runtime for late-resolved aliases/presets), never warn-and-ignore: Gemini 3-class models accept `1k`/`2k`/`4k`, `gpt-image-2` accepts `1k`/`2k` and arbitrary exact sizes within OpenAI's constraints, Gemini 2.5 / Flash Lite / legacy GPT Image / Flux / SDXL / Qwen accept `1k` only. On tier-grid (Gemini) models an exact size runs only on an exact grid-cell match, with near-misses rejected naming the nearest valid cells. When `size` is unset no size intent is sent (Gemini jobs omit `image_size`; OpenAI models keep their fixed 1K-class preset for the chosen ratio) and the provider default applies. The MTHDS JSON Schema exposes the field as tier-enum-or-`WxH`-pattern, and an optional `size` default is supported in `[cogt.img_gen_config.img_gen_param_defaults]`.-- **Static validation for Google image models:** Google image models now carry `rules` blocks in the backend deck (new geometry taxonomies `gemini_2_5`, `gemini_3_pro`, `gemini_3_flash`, `gemini_3_flash_lite`), so unsupported (aspect ratio × size) combinations on Gemini models are caught at blueprint-load time instead of only failing at the provider call. The Google img-gen factory is keyed by taxonomy instead of hardcoded model names (breaking: the `GoogleImageGenModel` name enum is removed — model handles are deck config, not code constants).-- **Gemini image-to-image (img2img) support:** The Google img-gen worker now forwards input images to the Gemini API as inline parts alongside the text prompt, enabling image editing and multi-image composition with the Nano Banana models. All Google-backend image models declare image inputs (`nano-banana`, `nano-banana-pro`, `nano-banana-2`, `nano-banana-2-lite` — matching the same models' declarations on the OpenRouter backend and the Pipelex gateway); `nano-banana-2` (`gemini-3.1-flash-image-preview`) is supported by the dimensions factory (same 1K/2K/4K resolution grid as Nano Banana Pro, per Google's published tables); added `nano-banana-2-lite` (`gemini-3.1-flash-lite-image`, 1K only) to the Google backend. Every img-gen worker now validates input images against the model's declared capability at job start: a model without image inputs (no `"images"` in its `inputs`) rejects img2img jobs with a clean `ImgGenParameterError` before any provider call — enforced once in the worker base class, so direct-request workers (Google Gemini native, chat-completions image models) and args-factory workers all fail early and explicitly instead of forwarding an unsupported request to the provider.-- **Banner aspect ratios (`landscape_4_1`, `landscape_8_1`, `portrait_1_4`, `portrait_1_8`):** New `AspectRatio` values for the extreme banner/strip formats introduced by the Gemini 3.1 image models, supported by `nano-banana-2` and `nano-banana-2-lite` with Google's published pixel grids. All other image-gen backends reject them with a clean parameter error. The Google dimension tables were restructured by model generation (Gemini 2.5 grid vs Gemini 3 grid) with per-model capability gating, so unsupported ratio/size combinations raise `ImgGenParameterError` instead of crashing.+- **`PipeParallel` always combines (Breaking):** A `PipeParallel` controller now always combines its branch outputs into its declared `output` concept and stamps it as the main output. The declared `output` is strictly validated at author time and must be `Composite` or a structured concept whose fields and types match the branch `result` names. Combination replaces the removed `combined_output` field. Migration: pipes declaring both fields just drop the `combined_output` line; `add_each_output`-only pipes replace their placeholder `output` with `Composite` or a matching structured concept. `add_each_output` keeps its meaning and now defaults to `false`.+- **Main-stuff invariant enforced (Breaking):** Every pipe run now guarantees a `main_stuff`; defensive "maybe there is no main stuff" branches are removed, wire models make the field required (`main_stuff_name: str` on `PipelexPipeRunOutput` and the `/execute` response extension), and `PipeOutput.optional_main_stuff` is replaced by `PipeOutput.main_stuff`.+- **Orchestrator SPI delivery split (Breaking):** `OrchestratorProtocol.run` is now strictly the blocking arm (returning a completed `PipelexPipeRunOutput`). A new `OrchestratorProtocol.start` handles fire-and-forget, returning a `PipelexPipeDispatchAck` (IDs only). `PipelexPipeRunOutput.is_completed` is deleted — it is always a completed output now.+- **`LLMWorkerInternalAbstract` folded (Breaking):** Removed and folded into `LLMWorkerAbstract`, which now owns the whole job lifecycle (capability checks, constraints, telemetry). Subclasses extend `LLMWorkerAbstract` directly and implement `_gen_text`/`_gen_object`, nothing else.+- **Type preservation across transport:** `Composite` components (and their nested lists) now retain strict types across transport boundaries (`dump_for_transport` / hydration) via private class markers.+- **`PipeParallel` honors `final_stuff_code`:** A requested final stuff code (e.g. from a `PipeBatch`) is now correctly stamped on the parallel's combined output.+- **Docs — inference plugins:** Rewrote `using-inference-plugins.md` to demonstrate a genuine `pipelex.plugins` entry-point package instead of a legacy config workaround.+- **Linting pipeline:** `plxt lint` now validates `.mthds` files against the locally generated schema (`derived/mthds_schema.json`) rather than the released schema bundled with `plxt`.+- **Dry-run mocks:** Reduced the default dry-run mock list generation from 3 items to 2 to cut dry-run processing time.+- Bumped `mthds` dependency from `>=0.6.0` to `>=0.7.0`.### Fixed-- **`Composite` components keep their types across transport:** `WorkingMemory.dump_for_transport()` now stamps each composite component with the pipelex-private `__pipelex_class__`/`__pipelex_module__` markers (the same convention as list items), and hydration rebuilds typed `StuffContent` components from them — including `ListContent` components with typed items. Previously a delivered `native.Composite` hydrated with plain `dict` components, breaking typed access and rendering after an API/worker round-trip.-- **`add_each_output` is optional in the builder spec:** `PipeParallelSpec` now defaults `add_each_output` to `false` like the blueprint does, so a generated always-combine spec with just `branches` and `output` no longer fails validation before `to_blueprint()`.-- **`PipeCondition` `continue` with nothing to pass through fails loud:** The `continue` outcome delivers the current main stuff by passing it through. When there is no main stuff to pass through (e.g. the condition is the entry pipe over named inputs only), the run now raises a clear, actionable `PipeRunError` at the pipe level — in live and dry runs alike — instead of completing without a main output and crashing downstream surfaces (graph tracer, delivery, telemetry).-- **Stale main stuff after an `add_each_output`-only parallel:** A pipeline ending in an `add_each_output`-only `PipeParallel` used to silently report the *previous* step's output (or even an input) as its main result, and such a parallel as the top-level pipe on the API path completed with no main stuff at all. The parallel now always stamps its combined output, in live and dry runs alike.-- **Provider error-body reader tolerates unread httpx streaming responses:** `_parse_response_text_body` (shared by the HuggingFace, FAL, Azure REST, and Mistral error classifiers) no longer crashes with `httpx.ResponseNotRead` when an SDK error carries a streaming `httpx.Response` whose body was never buffered (possible with `huggingface_hub` 1.x async streaming errors); it now degrades to a body-less metadata payload, keeping status code, request id, and retry-after intact.-- **Img2img support check keyed on declared inputs:** `ImgGenParamSupport.check_job_params` now takes the full model spec and checks image-to-image support against the model's declared `inputs` (the universal capability field) instead of requiring an `input_images` rule. The rule is an args-factory formatting concern that Gemini-family models — whose request path never reads it — legitimately don't carry, so the old rules-keyed check wrongly reported the whole family as not supporting img2img.-- **Google img-gen worker no longer hardcodes 1K:** The native Google worker and the gateway path now send the requested `image_size` to the Gemini API (omitted entirely when `size` is unset); previously the worker always sent `"1K"` regardless of intent, making 2K/4K generation unreachable.-- **Routing-profile `optional_routes` were silently dropped:** `RoutingProfileFactory.make_routing_profile` parsed and validated the `optional_routes` table from `routing_profiles.toml` but never passed it to the built `RoutingProfile`, so optional routes declared in config had no effect — models relying on one fell through to the profile's default backend and could silently vanish from the model deck. The factory now carries `optional_routes` through, with unit coverage for the pass-through and for the enabled/disabled-backend gating.-- **Unknown boot orchestrator now fails loud:** Requesting a boot orchestrator no installed plugin provides — via `--orchestrator <name>` or `Pipelex.make(boot_orchestrator=...)` — now raises `UnknownBootOrchestratorError` at boot instead of silently falling back to in-process execution. A typo or a missing orchestrator plugin (e.g. `--orchestrator temporal` without the Temporal plugin installed) is reported rather than quietly running the job on the wrong runtime. The requested name is matched against registered plugin names, the same namespace the boot gate uses.-- **Failed boot no longer leaks process-global state:** A `Pipelex.make` that raises during setup now releases the process-global singletons a partial boot acquired (config, logging, the kajson class registry, template registries), mirroring `teardown`. Previously a failed boot could poison a subsequent boot in the same process — surfacing as a "LogConfig is already set" error or a stale, half-populated class registry.+- **`PipeCondition` pass-through failure:** A `continue` outcome with no pre-existing main stuff now fails loudly and actionably at the pipe level instead of crashing downstream surfaces.+- **Stale main stuff in parallels:** A pipeline ending in an `add_each_output`-only `PipeParallel` no longer silently reports the previous step's output as its main result.+- **`add_each_output` optional in the builder spec:** `PipeParallelSpec` now defaults `add_each_output` to `false` like the blueprint does, so a generated always-combine spec with just `branches` and `output` no longer fails validation before `to_blueprint()`.+- **Google img-gen size handling:** The native Google worker and gateway path now send the requested `image_size` to the Gemini API instead of hardcoding `"1K"`, making 2K/4K generation reachable.+- **Img2img support checks:** The capability check now keys off the model's declared `inputs` rather than requiring an `input_images` rule, which had falsely reported Gemini models as unsupported.+- **Routing profile optional routes:** `optional_routes` declared in `routing_profiles.toml` are no longer silently dropped by the factory.+- **Unknown boot orchestrator fails loud:** Requesting a boot orchestrator no installed plugin provides — via `--orchestrator <name>` or `Pipelex.make(boot_orchestrator=...)` — now raises `UnknownBootOrchestratorError` at boot instead of silently falling back to in-process execution.+- **Failed boot no longer leaks process-global state:** A `Pipelex.make` that raises during setup now releases the process-global singletons a partial boot acquired (config, logging, the kajson class registry, template registries), so a failed boot no longer poisons a subsequent boot in the same process.+- **HuggingFace streaming errors:** The provider error-body reader now safely tolerates unread `httpx` streaming responses without crashing with `httpx.ResponseNotRead`.++### Removed++- **`combined_output` field (Breaking):** Removed from the MTHDS language for `PipeParallel`; pipes now declare their combination target directly in the `output` field.+- **Legacy external plugin setter (Breaking):** Removed `set_llm_worker_from_external_plugin` from the inference manager, fully replaced by the `pipelex.plugins` entry point.++### Security++- **Transformers vulnerability:** Bumped `transformers` past CVE-2026-4372 (GHSA-29pf-2h5f-8g72) to resolve a high-severity RCE. To unblock this, the `huggingface` extra now requires `huggingface_hub>=1.5.0,<2.0.0` (Breaking).## [v0.36.0] - 2026-06-30diff --git a/pyproject.toml b/pyproject.tomlindex 8ff0ed095..ab13116ec 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[project]name = "pipelex"-version = "0.36.0"+version = "0.37.0"description = "Execute composable AI methods declared in the MTHDS open standard"authors = [{ name = "Evotis S.A.S.", email = "[email protected]" }]maintainers = [{ name = "Pipelex staff", email = "[email protected]" }]diff --git a/uv.lock b/uv.lockindex 54c9ff214..7165b61a9 100644--- a/uv.lock+++ b/uv.lock@@ -3693,7 +3693,7 @@ wheels = [[[package]]name = "pipelex"-version = "0.36.0"+version = "0.37.0"source = { editable = "." }dependencies = [{ name = "aiofiles" },