Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
Release v0.38.0
Louis Choquel623c0f7736f5
diff --git a/.badges/tests.json b/.badges/tests.jsonindex 952e007e3..95990360b 100644--- a/.badges/tests.json+++ b/.badges/tests.json@@ -1,7 +1,7 @@{"schemaVersion": 1,"label": "tests",- "message": "7826",+ "message": "8214","color": "blue","cacheSeconds": 300}\ No newline at end of filediff --git a/CHANGELOG.md b/CHANGELOG.mdindex 113168b28..158241c53 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,35 +1,42 @@# Changelog-## [Unreleased]+## [v0.38.0] - 2026-07-06++### Highlights++- **Optionality is a first-class language feature** — Concept references in pipe `inputs`/`outputs` can be marked `?` (optional) or `!` (force), so absent values become tracked, first-class data with provenance instead of crashes or silent nulls. Absence flows through a runtime trichotomy — a plain input skips its pipe, a `?` input absorbs it and runs, a `!` input fails loudly — and a static taint pass proves at author time that every maybe-absent value reaches an explicit sink.+- **Storage and secrets are now plugin seams** — Both backends are selected at boot from a config-keyed registry, so third parties can ship `pipelex-storage-<backend>` / `pipelex-secrets-<backend>` packages via the `pipelex.plugins` entry point — the same discovery machinery the inference and orchestrator seams already use.### Added-- **Storage provider is now a plugin seam:** The storage backend (`local` / `in_memory` / `s3` / `gcp`) is selected at boot from a `StorageProviderRegistry` keyed by the open `storage_config.method` token, populated by the always-on built-in `StoragePlugin`. A third party can now ship a `pipelex-storage-<backend>` package that advertises itself under the `pipelex.plugins` entry-point group and is selected via `storage_config.method` — the same discovery/denylist machinery the inference and orchestrator seams use. `storage_config.method` is now an open `str`: an unknown method fails loudly at boot with `UnknownStorageMethodError` (listing the registered methods), not at config parse. See the new `docs/under-the-hood/storage-provider-plugins.md`.-- **Secrets provider is now a plugin seam:** The secrets backend (`env`) is selected at boot from a `SecretsProviderRegistry` keyed by the new open `secrets_config.method` token (default `"env"`), populated by the always-on built-in `SecretsPlugin`. A third party can now ship a `pipelex-secrets-<backend>` package (Vault, AWS Secrets Manager, …) that advertises itself under the `pipelex.plugins` entry-point group and is selected via `secrets_config.method` — the same mechanism the storage seam uses. Out-of-the-box behavior is unchanged: `env` stays the default. An unknown method fails loudly at boot with `UnknownSecretsMethodError` (listing the registered methods), not at config parse. See the new `docs/under-the-hood/secrets-provider-plugins.md`.-- **TOML pipeline inputs:** The `--inputs` file passed to `pipelex run` and `pipelex-agent run` (pipe/bundle/method) can now be TOML in addition to JSON, discriminated by file extension — a `.toml` suffix is parsed as TOML, every other value stays JSON. Both formats produce the same input dictionary; TOML's multi-line strings make text-heavy inputs much easier to author. `run bundle <dir>` auto-detects `inputs.toml` alongside `inputs.json` when `--inputs` is omitted, erroring if both exist. Inline JSON (`{...}`) and agent-CLI stdin inputs stay JSON-only. A bare TOML datetime/date/time literal is rejected with an explicit error (quote it as a string) — native datetime concept support is a separate track.-- **`--format toml` for inputs-template generation:** `pipelex build inputs` and `pipelex-agent inputs` (pipe/bundle/method) accept `--format json|toml` (default `json`) to emit the generated inputs template as TOML. On the main CLI, `toml` defaults the output filename to `inputs.toml`; on the agent CLI, `toml` prints the raw TOML template to stdout (the no-inputs case prints a TOML comment line that loads back as an empty dict).-- **Absent results are first-class on the wire and every surface (optionals Step E):** A run whose declared `?` output resolves absent is a success everywhere. `main_stuff_name` on `PipelexPipeRunOutput` and the `/execute` response now names the declared output slot even when it resolved absent — consumers branch on the absence record carried in the serialized working memory (`absences`), never on transport. The absence ledger round-trips cross-process (`hydrate_working_memory` reconstructs it, so a recorded absence no longer degrades to a hard miss after transit). The CLI run surfaces render an explicit absence document instead of crashing: `pipelex run` prints the absence and `--save-main-stuff` writes the `"absent": true` artifact (json + md, no viewer), the agent CLI's `run` returns the absence document as its result payload, and OTel/Langfuse output capture serializes the absence for both inputs and outputs.-- **IO contracts carry optionality (`optional` on `pipe_io_contracts`):** `PipeInputContract` and `PipeOutputContract` gain an `optional: bool` — `true` for `?`-declared inputs (may be omitted; the pipe handles absence) and `?`-declared outputs (may resolve absent).-- **Advisory `warnings` on the validation report + the useless-`!` lint:** The validation report gains a general `warnings` array — same item shape as `validation_errors`, but advisory: it never flips `is_valid`. Its first occupant is `optional_force_redundant`: a `!` (force) input whose slot is guaranteed present in every analyzed flow — the assertion can never fire, so the marker is dead weight. The lint aggregates across flows before warning (a `!` that is meaningful in one flow is never flagged because another flow guarantees the slot). Every whole-bundle validate surface carries it: the protocol report, the agent-CLI validate envelopes (rendered as a "Warnings" section in markdown), the human `pipelex validate` (yellow advisory lines — never the exit code), and the builder validate operations.-- **Execution graphs show skips and optional data edges:** `GraphSpec` nodes gain the `skipped` status with a `skip_reason` (a lifted pipe ends its node as skipped — "why did my workflow produce nothing?" is now answerable from the graph), and data edges fed by a declared-optional (`?`) output carry `optional: true`. A skipped pipe with a plural output still registers the real empty list it wrote, so downstream data edges resolve like any produced value. Both flow through the in-process tracer and the distributed event-replay assembler (new `pipe_end_skipped` trace event).-- **Static optionality validation — the absence-taint pass (breaking):** Validation now proves the absence safety theorem at author time: every maybe-absent slot must reach an explicit sink. A taint pass over each controller's dataflow computes per-slot presence (`guaranteed` / `maybe-absent`) and rejects escapes with new typed validation errors: `optional_not_handled` (a maybe-absent slot ends a sequence — or feeds a condition through a mapped `?`-output pipe — without the controller declaring a `?` output; the message names the absence origin, the propagation path, and the three fixes), `optional_output_required` (**breaking**: a `continue`-reachable `PipeCondition` must declare its output optional, e.g. `output = "Constraint?"` — `continue` resolves the output absent, so a plain output would hide the no-output path), and `optional_branch_required_field` (a required structure field of a `PipeParallel` output fed by a maybe-absent branch — make the field non-required or sink the absence upstream). Plural slots are never tainted: a lifted plural output is the empty list and a batched step compacts, so taint stops at `[]`.-- **Template guard-lint for optional inputs (`optional_input_unguarded`):** Every template reference to a declared-optional (`?`) input must be guarded — `@?var`, a `{% if var %}...{% endif %}` block, or an inline presence conditional (`... if var is defined else ...`). An unguarded reference (including a plain `@var` sigil on an optional input, a deep `{{ var.attr }}`, or use in an `{% else %}` arm) fails validation with the precise fix. Applies to `PipeLLM` prompts and system prompts, `PipeCompose` templates, and `PipeCondition` expressions. In the same motion, `@?` finally means what it says: a variable declared optional on a pipe is no longer presence-required by the controller miss-gates — the pipe runs and its guarded templates take the absent arm.-- **Liftable-pipe inventory on the valid report (`liftable_pipes`):** The validation report now lists every pipe that may be skipped (lifted) when an optional slot resolves absent — the liftable pipe, the controller it lifts within, the trigger slots, and the absence source. This is the build-time visibility that makes implicit lifting acceptable: authors see "may be skipped when X is absent" at `/validate` time, not just in the run report.-- **Absence at runtime — the ledger and the trichotomy (`?` / `!` become live):** Working memory gains an absence ledger: an `AbsenceRecord` (variable, producing pipe, kind = declared-absent / skipped / not-provided, reason, upstream chain) records *why* a slot holds no value, with provenance captured at the moment absence is produced. At run time, a pipe whose plain input is fed a recorded absence is **skipped** (implicit lifting): its own output is recorded absent, chaining back to the origin — write `?` once where absence enters and the chain short-circuits like Swift's `a?.b.c`. A `?` input absorbs absence (the pipe runs and handles both arms); a `!` input fed absence raises the new typed `OptionalValueAbsentError`, whose report names the variable, the consuming pipe, the producing pipe, and the original reason. A skipped pipe with a plural output normalizes to an empty list (with a ledger note) instead of an absent slot — `[]`-emptiness stays the absence story for plurals. Optional **method** inputs may now be omitted from the pipeline inputs: the slot starts as a recorded not-provided absence instead of raising a missing-inputs error, and the missing-required-inputs message now names the optional inputs a caller may omit (it also no longer claims "Dry run" on live runs). Post-run reads get a tri-state resolved accessor (`WorkingMemory.resolve_stuff` / `resolve_main_stuff`: a value or a recorded absence); the run's working memory enumerates all recorded absences. A slot with neither a value nor a record remains a hard error — never-produced is a bug, not an absence. Controller boundaries stay authoritative: a sequence declaring `X?` over a step that needs `X` plain runs and lets its steps lift (the declared-vs-needed input check no longer compares presence markers).-- **Presence markers `?` and `!` on pipe inputs/outputs — grammar and static carriers:** The io-ref grammar gains presence markers after the multiplicity suffix: `inputs = { clause = "PenaltyClause?" }` (optional input), `clause = "PenaltyClause!"` (force), `output = "PenaltyClause?"` (optional output). Markers parse, are carried through blueprints, builder specs, and IO spec carriers, round-trip through bundle representations, and signature-vs-definition contract matching compares them. Grammar misuse is rejected at blueprint parse with the new `optional_marker_invalid` validation error type: markers never combine with multiplicity (`X[]?` is invalid — an absent plural is just the empty list), `!` is inputs-only, and markers stay illegal on concept definitions, `refines`, structure-field refs, and package refs. No runtime behavior change yet: absence semantics (skip/absorb/force at run time, taint validation, wire surfaces) land in the following steps of the optionals feature.+- **Optionals & presence markers (`?`, `!`):** Concept references in pipe `inputs`/`outputs` can now be marked `?` (optional) or `!` (force), e.g. `clause = "PenaltyClause?"`. Absent values are first-class data, tracked in working memory via an `AbsenceRecord` ledger (variable, producing pipe, kind = declared-absent / skipped / not-provided, reason, upstream chain) with provenance captured at the moment absence is produced:++ - A plain input fed an absence causes the pipe to be **skipped** (implicitly lifted); its own output is recorded absent, chaining back to the origin so the miss short-circuits like Swift's `a?.b.c`.+ - A `?` input **absorbs** the absence and the pipe runs, handling both arms.+ - A `!` input fed an absence **fails loudly** with a typed `OptionalValueAbsentError` naming the variable, the consuming pipe, the producing pipe, and the original reason.+ - Callers may omit `?`-marked method inputs; the slot starts as a recorded `not_provided` absence rather than raising a missing-inputs error, and the missing-required-inputs message now names the optional inputs a caller may omit.+ - Post-run reads get a tri-state resolved accessor (`WorkingMemory.resolve_stuff` / `resolve_main_stuff`: a value or a recorded absence). A slot with neither a value nor a record is still a hard error — never-produced is a bug, not an absence. Markers round-trip through blueprints, builder specs, IO contracts, and bundle representations; grammar misuse (`X[]?`, `!` on an output, markers on concept definitions) is rejected at parse with `optional_marker_invalid`.+- **Static optionality validation (the absence-taint pass):** Validation proves the absence-safety theorem at author time — a taint pass over each controller's dataflow computes per-slot presence (`guaranteed` / `maybe-absent`) and rejects any maybe-absent slot that escapes without an explicit sink, with typed errors (`optional_not_handled`, `optional_output_required`, `optional_branch_required_field`) that each name the absence origin, the propagation path, and the fixes. The validation report also lists every liftable (skippable) pipe (`liftable_pipes`) — build-time visibility for "may be skipped when X is absent" — and gains a general `warnings` array for advisory lints that never flip `is_valid` (first occupant: `optional_force_redundant`, a `!` whose slot is guaranteed present in every flow).+- **Template guard-lint (`optional_input_unguarded`):** Every template reference to a declared-optional (`?`) input must be guarded — `@?var`, a `{% if var %}…{% endif %}` block, or an inline presence conditional — or validation fails with the precise fix. Applies to `PipeLLM` prompts and system prompts, `PipeCompose` templates, and `PipeCondition` expressions. In the same motion, `@?` finally means what it says: an optional variable is no longer presence-required by the controller miss-gates, so the pipe runs and its guarded templates take the absent arm.+- **Execution graph updates:** `GraphSpec` nodes gain the `skipped` status with a `skip_reason` — "why did my workflow produce nothing?" is now answerable from the graph — and data edges fed by a declared-optional (`?`) output carry `optional: true`. Both flow through the in-process tracer and the distributed event-replay assembler (new `pipe_end_skipped` trace event).+- **Storage provider plugin seam:** The storage backend (`local` / `in_memory` / `s3` / `gcp`) is selected at boot from a `StorageProviderRegistry` keyed by the open `storage_config.method` token, populated by the always-on built-in `StoragePlugin`. Third parties can ship a `pipelex-storage-<backend>` package that advertises itself under the `pipelex.plugins` entry-point group — the same discovery/denylist machinery the inference and orchestrator seams use. `storage_config.method` is now an open `str`: an unknown method fails loudly at boot with `UnknownStorageMethodError` (listing the registered methods), not at config parse. See the new `docs/under-the-hood/storage-provider-plugins.md`.+- **Secrets provider plugin seam:** The secrets backend is now a plugin seam selected via the new open `secrets_config.method` token (default `"env"`) from a `SecretsProviderRegistry`, populated by the always-on built-in `SecretsPlugin`. Third parties can ship a `pipelex-secrets-<backend>` package (Vault, AWS Secrets Manager, …) under the `pipelex.plugins` entry-point group — the same mechanism the storage seam uses. Out-of-the-box behavior is unchanged: `env` stays the default. An unknown method fails loudly at boot with `UnknownSecretsMethodError`. See the new `docs/under-the-hood/secrets-provider-plugins.md`.+- **TOML pipeline inputs & templates:** The `--inputs` file passed to `pipelex run` and `pipelex-agent run` (pipe/bundle/method) can now be TOML in addition to JSON, discriminated by file extension — TOML's multi-line strings make text-heavy inputs much easier to author. `run bundle <dir>` auto-detects `inputs.toml` alongside `inputs.json` when `--inputs` is omitted, erroring if both exist. Complementarily, `pipelex build inputs` and `pipelex-agent inputs` accept `--format json|toml` (default `json`) to emit the generated inputs template as TOML. Inline JSON (`{…}`) and agent-CLI stdin inputs stay JSON-only; a bare TOML datetime/date/time literal is rejected with an explicit "quote it as a string" error (native datetime concept support is a separate track).+- **Gateway models:** Added support for `nano-banana-2-lite` through Pipelex Gateway.### Changed-- **Breaking: dropped Python 3.10 support — `requires-python` is now `>=3.11,<3.15`.** The 3.10 compatibility bridges are gone: `StrEnum`, `Self`, and `Traversable` are now imported directly from the stdlib (`enum` / `typing` / `importlib.resources.abc`) at every call site, the `pipelex.types` re-export module that existed only to paper over 3.10 is deleted, and the `backports.strenum` conditional dependency (and its type stub) is dropped. CI test/lint matrices, the `package-check` `requires-python` floor gate, and the contributor docs now start at 3.11. Install on Python 3.11 or newer.-- **Plugin API version bumped to 3 (Breaking):** `PLUGIN_API_VERSION` is now `3` (was `2`) to add the `add_storage_provider` and `add_secrets_provider` registrar menu methods. Plugin discovery version-checks with strict equality, so every external plugin must re-declare `targets_api = 3`. `pipelex-mistralai-workflows` registers no storage or secrets provider and imports none of the removed symbols, so the `targets_api` bump is its only change. `pipelex-temporal` needs the `targets_api` bump **and** a code migration: its payload-codec factory imports the now-removed `make_storage_provider_from_config` (see Removed) and must switch to resolving the provider via the storage-provider registry (`get_storage_provider_registry().get_required(method=...)`).-- **`PipeSignature` is not a pipe type (Breaking):** A signature is now declared by **omitting** `type` — a `[pipe.x]` section with no `type` and nothing but the contract (`description`, `output`, optional `inputs`, optional `signature_for`) *is* a `PipeSignature`. Writing `type = "PipeSignature"` explicitly is rejected with a migration error ("`PipeSignature` is no longer a pipe type — delete the `type` line"), and a typeless section that declares any non-contract field is a hard error naming the field. The `pipelex-agent pipe` authoring command mirrors this: a typeless JSON spec (no `type`) renders a `[pipe.x]` section with no type line, and `--type PipeSignature` is rejected. **MTHDS JSON Schema shape change:** the signature arm no longer carries a `type` property (an explicit tag now fails schema validation), so the downstream schema copies (`mthds`, `vscode-pipelex`, `mthds-ui`) must be re-synced on the next release via the `mthds-schema-sync` skill.-- **`PipeCondition` `continue` now resolves the output as absent (breaking):** The `continue` special outcome no longer passes the current main stuff through (and no longer errors when there is none). It records a declared-absent `AbsenceRecord` for the condition's declared output — with the evaluated expression as the reason — and returns success, memory otherwise unchanged. Dry-run parity: an all-special-outcomes condition records the same absence instead of raising when `continue` is reachable; a fail-only condition (every outcome and the default map to `fail`) still raises in dry run, since every live path can only fail. Migration: the previous value stays in the working memory under its own name, so downstream pipes consume it explicitly by that name (declared `?` when it may be absent); a coalescing operator is the planned ergonomic replacement for pass-through.-- **Controllers combine under absence:** `PipeParallel` no longer crashes when a branch result resolves absent (lifted branch or `continue`): with a `Composite` output the absent component is omitted (ledger note kept for observability); with a structured output a non-required field absorbs the absence as its field default (`None` unless the author declared another default), while a required field fed an absent branch raises a typed error naming the branch, the field, and the fixes. `PipeBatch` compacts: absent branch results are dropped from the aggregated list, so batching a "keep or skip" condition over items yields only the kept results.-- **Result delivery of an absent output:** An absent main output is a first-class success at the delivery boundary — the `main_stuff.json/md/html` artifact files become an explicit absence artifact (JSON carries `"absent": true` plus the absence record with its provenance chain) on both the typed and the raw (cross-process) delivery paths. A working memory with neither a value nor a recorded absence still fails delivery loudly.+- **Dropped Python 3.10 support (Breaking):** `requires-python` is now `>=3.11,<3.15`. The 3.10 compatibility bridges are gone — `StrEnum`, `Self`, and `Traversable` are imported directly from the stdlib at every call site, the `pipelex.types` re-export module that existed only to paper over 3.10 is deleted, and the `backports.strenum` conditional dependency (and its type stub) is dropped. CI test/lint matrices, the `package-check` `requires-python` floor gate, and the contributor docs now start at 3.11.+- **Plugin API version bumped to 3 (Breaking):** `PLUGIN_API_VERSION` is now `3` (was `2`) to add the `add_storage_provider` and `add_secrets_provider` registrar menu methods. Plugin discovery version-checks with strict equality, so every external plugin must re-declare `targets_api = 3`. `pipelex-mistralai-workflows` registers no storage or secrets provider, so the `targets_api` bump is its only change. `pipelex-temporal` needs the bump **and** a code migration: its payload-codec factory imports the now-removed `make_storage_provider_from_config` (see Removed) and must switch to resolving the provider via the registry (`get_storage_provider_registry().get_required(method=...)`).+- **`PipeSignature` is not a pipe type (Breaking):** A signature is now declared by **omitting** `type` — a `[pipe.x]` section with no `type` and nothing but the contract (`description`, `output`, optional `inputs`, optional `signature_for`) *is* a `PipeSignature`. Writing `type = "PipeSignature"` explicitly is rejected with a migration error ("`PipeSignature` is no longer a pipe type — delete the `type` line"), and a typeless section declaring any non-contract field is a hard error naming the field. The `pipelex-agent pipe` authoring command mirrors this. **MTHDS JSON Schema shape change:** the signature arm no longer carries a `type` property (an explicit tag now fails schema validation), so the downstream schema copies (`mthds`, `vscode-pipelex`, `mthds-ui`) must be re-synced on the next release via the `mthds-schema-sync` skill.+- **`PipeCondition` `continue` now resolves the output as absent (Breaking):** The `continue` special outcome no longer passes the current main stuff through (and no longer errors when there is none). It records a declared-absent `AbsenceRecord` for the condition's declared output — with the evaluated expression as the reason — and returns success, memory otherwise unchanged. A `continue`-reachable `PipeCondition` must therefore declare its output optional (e.g. `output = "Constraint?"`), or validation rejects it (`optional_output_required`). Dry-run parity holds. Migration: the previous value stays in working memory under its own name, so downstream pipes consume it explicitly by that name (declared `?` when it may be absent); a coalescing operator is the planned ergonomic replacement for pass-through.+- **Controllers combine under absence:** `PipeParallel` no longer crashes when a branch result resolves absent (lifted branch or `continue`): with a `Composite` output the absent component is omitted (ledger note kept), and with a structured output a non-required field absorbs the absence as its default (`None` unless the author declared another), while a required field fed an absent branch raises a typed error naming the branch, the field, and the fixes. `PipeBatch` compacts: absent branch results are dropped from the aggregated list, so batching a "keep or skip" condition over items yields only the kept results.+- **Absent output delivery is now a first-class success:** A run whose declared `?` output resolves absent succeeds everywhere. `main_stuff_name` on `PipelexPipeRunOutput` and the `/execute` response names the declared output slot even when it resolved absent — consumers branch on the absence record in the serialized working memory (`absences`), never on transport, and the ledger round-trips cross-process (`hydrate_working_memory` reconstructs it). The `main_stuff.json/md/html` artifact files become an explicit absence document (`{"absent": true}` plus the provenance chain) on both the typed and raw delivery paths, `pipelex run` / `--save-main-stuff` and the agent CLI's `run` surface it instead of crashing, and OTel/Langfuse capture serializes it. A working memory with neither a value nor a recorded absence still fails delivery loudly.### Fixed-- **Template truth-tests on singular values no longer crash:** `{% if var %}` (and `@?var`, which expands to it) on a present singular value used to raise `TypeError: '...' content does not support len().` — Jinja2's truth test fell through to the artefact's list-only `__len__`. `StuffArtefact` now defines `__bool__`: a present non-list artefact is truthy, a list artefact follows list emptiness (an empty list is falsy). This makes the optionals guard idiom safe on both arms.+- **Template truth-tests on singular values no longer crash:** `{% if var %}` (and `@?var`, which expands to it) on a present singular value used to raise `TypeError: '…' content does not support len().` — Jinja2's truth test fell through to the artefact's list-only `__len__`. `StuffArtefact` now defines `__bool__`: a present non-list artefact is truthy, a list artefact follows list emptiness. This makes the optionals guard idiom safe on both arms.+- **GitHub Actions:** Removed an invalid `environment` block from the `manual-trigger-tests-check.yml` workflow.### Documentationdiff --git a/pyproject.toml b/pyproject.tomlindex 7ea93db32..a51de09d6 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[project]name = "pipelex"-version = "0.37.0"+version = "0.38.0"description = "Execute composable AI methods declared in the MTHDS open standard"authors = [{ name = "Evotis S.A.S.", email = "[email protected]" }]maintainers = [{ name = "Pipelex staff", email = "[email protected]" }]diff --git a/uv.lock b/uv.lockindex fc1f39cea..7530cf41a 100644--- a/uv.lock+++ b/uv.lock@@ -3242,7 +3242,7 @@ wheels = [[[package]]name = "pipelex"-version = "0.37.0"+version = "0.38.0"source = { editable = "." }dependencies = [{ name = "aiofiles" },