Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
docs(changelog): polish v0.31.0 entry — highlights-first ordering
Louis Choquel424ea758c838
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 10842cd4b..993e7bacf 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,57 +2,58 @@## [v0.31.0] - 2026-06-04-### Added+This release hardens Pipelex at its edges. The headliners: a full **error-handling overhaul** that gives every error a stable, RFC 7807-shaped identity and carries it all the way out to webhooks and external surfaces; **lenient validation with pipe signatures**, so you can sketch and dry-run a whole pipeline top-down before a single pipe is implemented; and a first, **experimental cut of CSV tabular support** for reading and writing typed lists straight from `.csv`.-- **CSV input & output for pipelines.** A `.csv` file can now be read directly into a typed `ListContent[YourConcept]` and a list output can be written back out as CSV. On input, an `inputs.json` reference whose `url` ends in a tabular suffix under a flat structured concept (e.g. `{ "people": { "concept": "demo.Person", "content": { "url": "people.csv" } } }`) loads as one list whose items are instances of that concept — column headers must match the concept's field names, cells are coerced via Pydantic (`birth_year` → `int`, ISO dates → `date`), and an empty cell becomes `None` (so the field must be optional). On output, the new `--save-csv <path>` flag on `pipelex run pipe|bundle|method` writes the main stuff to a literal, cwd-relative CSV path when that output is a flat list. The row concept must be flat (scalar fields only — `text`/`integer`/`number`/`boolean`/`date`, plus optionals and choice-constrained scalars); a nested/list/dict/concept-typed field is rejected with a clear error naming the offending field. CSV reading uses the Python standard library; `.xlsx` is recognized but routed to a clear "needs `pipelex[tabular]`" message (the Excel backend is not built yet). v1 reads local file paths only — a remote `url` (`http(s)`/`s3`/`gs`/…) with a tabular suffix is actively rejected. See the [CSV Input & Output guide](https://docs.pipelex.com/latest/building-methods/pipes/csv-input-and-output/).-- **`PipeSignature` — contract-only pipes for top-down pipeline design.** A new pipe type (`type = "PipeSignature"`) declares a pipe's `inputs`, `output`, and `description` without an implementation, so an author or agent can sketch a complete pipeline before committing to the operator that will eventually do the work. Signatures slot into `PipeBlueprintUnion` and `PipeSpecUnion`, are produced by the new `PipeSignatureFactory`, and ship with their own runtime (`PipeSignature` in `pipelex/pipe_signature/`). At dry-run time a signature mints a mock `Stuff` matching its declared output (multiplicity included); at live-run time it raises `PipeSignatureNotExecutableError`. Multiplicity is supported on inputs (`Doc[]`, `Image[3]`) just like every other pipe. Optional `signature_for` field hints the downstream pipe type the signature stands in for (tooling-only; cannot itself be `PipeSignature`). New docs page: [Signature Pipes](https://docs.pipelex.com/latest/building-methods/pipes/signature-pipes/). Cross-link: `wip/signature-based-validation/signature-based-validation.md`.-- **`--allow-signatures` flag on `validate`.** `pipelex validate pipe`, `pipelex validate bundle`, and every `pipelex-agent validate` subcommand (`pipe`, `bundle`, `method`) accept `--allow-signatures` (default off) to dry-run pipelines that still contain `PipeSignature` placeholders. Without the flag, strict validation refuses any pipeline whose dependency graph reaches a signature, raising `SignaturesNotAllowedError` and reporting every reachable signature plus the controller chain (`signature_refs` and `dep_paths`) so the user knows exactly which pipes are still stubs. The main CLI and the agent CLI are both strict by default.-- **Graph walk for signature detection.** `collect_signature_refs()` and `collect_signature_paths()` (in `pipelex/pipe_signature/signature_walk.py`) walk a pipe's dependency graph (via the new default `pipe_dependencies()` method on `PipeAbstract` and per-controller overrides), aggregating signature pipe_refs and the controller chains that reach them. Used by the strict-mode pre-check in `BundleValidator` (`_signature_pre_pass`, run from `validate_pipes`) so `validate_bundle` surfaces a single `SignaturesNotAllowedError` listing every offender across the bundle.+### Added-- **`PipelexError.title()` and `PipelexError.type_uri()` — every error class now exposes a human-readable title and a stable RFC 7807 `type` URI.** Both auto-derive from the class name; a subclass overrides them only when the derived value is wrong by declaring `_declared_title` / `_declared_type_uri`. `type_uri()` is a pure function resolving to `https://docs.pipelex.com/latest/errors/<kebab-class-name>/`. Every `ErrorReport` now carries `title` and `type_uri` as populated fields, so consumers (CLI, API, SDKs) read `report.title` / `report.type_uri` directly instead of humanizing class names themselves.+- **Stable error identity (RFC 7807).** Every `PipelexError` now exposes `title()` and a stable `type_uri()` — a real, dereferenceable `https://docs.pipelex.com/latest/errors/<error>/` URL — and every `ErrorReport` carries both as populated fields, so consumers read `report.title` / `report.type_uri` directly instead of humanizing class names. `ErrorReport.to_problem_document()` renders an `application/problem+json` document with no web-framework dependency, and a new `DisclosureMode` (`VERBOSE` / `STRICT`) controls how much leaks onto external surfaces: `STRICT` drops provider/model attribution and redacts any message that wasn't authored to be caller-facing, keeping only the stable identifiers.-- **`ErrorReport.to_problem_document()` and `DisclosureMode` for rendering errors on external surfaces.** `to_problem_document(*, instance=None, request_id=None, disclosure_mode=VERBOSE)` returns an RFC 7807 `application/problem+json` dict (`type`, `title`, `status`, `detail`, `instance`, plus Pipelex classification fields as extension members) with no web-framework dependency. `to_dict(disclosure_mode=...)` projects the same report: `STRICT` drops `provider` / `model` unconditionally and projects `provider_metadata` through a curated subset (only `status_code` and `retry_after_seconds` — actionable HTTP client hints, not provider attribution), then redacts the `message` by *provenance* — a report whose error class authored caller-facing copy (`PipelexInterpreterError`, `ValidateBundleError`) keeps its `message` and `user_action`, while every other report has `message` replaced with a generic placeholder and `user_action` dropped, keeping only the stable identifiers. The keep/redact decision uses a per-class `caller_facing_message` flag rather than `error_domain` (which is inherited up the `__cause__` chain and so cannot distinguish a wrapper's own message from an inherited classification).+- **Structured error payloads on failure webhooks.** When a run fails, the delivery webhook now includes an `error` object — the full `ErrorReport` as a dict — so receivers can rehydrate it with `ErrorReport.from_dict(...)`, render an RFC 7807 response, or route on `error_domain` / `retryable`. Applies to both Temporal and direct execution. A `WebhookTarget.payload` that collides with a Pipelex-reserved key (`pipeline_run_id`, `status`, `result_url`, `error`) is now rejected at construction.-- **Per-class error documentation pages.** Every `PipelexError` subclass gets a generated reference page under `docs/errors/`, surfaced in the docs site as "Error Reference" — so a `type_uri` dereferences straight to a populated page. The new `pipelex-dev generate-error-pages` command regenerates them; pages a maintainer claims with a `<!-- pipelex:authored -->` marker are preserved across runs. The generator's discovery walks every `exceptions.py` / `*_exceptions.py` module the Phase 6 filename convention enforces — including modules whose home file is only reached via a deferred import path (plugin worker / factory chains for anthropic, bedrock, gateway, google, mistral, openai/vertex, azure, portkey, plus standalone classes like `FalCredentialsError`, `GraphSpecError`, `PipeBatchFactoryError`, `PyPdfium2RendererError`, `ConceptSpecError`). No manifest to maintain; adding a new error class is one step.+- **Per-class error reference pages.** Every `PipelexError` subclass now has a generated reference page under `docs/errors/`, surfaced in the docs site as "Error Reference" — so a `type_uri` dereferences straight to a populated page. Regenerate with the new `pipelex-dev generate-error-pages` command (`make generate-error-pages`, alias `make gep`); pages a maintainer claims with a `<!-- pipelex:authored -->` marker are preserved across runs.- **`request_id` on `JobMetadata`.** An optional caller-supplied request id, threaded through every activity / workflow / submitter hop and into the Temporal log context. Set it at dispatch with `pipeline_run_setup(..., request_id="...")` and read it back off `job_metadata.request_id`.-- **Failed runs now carry a structured error on the delivery webhook.** When a pipeline run fails, the webhook payload includes an `error` object — the full `ErrorReport` as a dict — so receivers can rehydrate it with `ErrorReport.from_dict(...)`, render an RFC 7807 response, or route on `error_domain` / `retryable`. Applies to both Temporal and direct execution modes. A `WebhookTarget.payload` that declares a Pipelex-owned key (`pipeline_run_id` / `status` / `result_url` / `error`) is now rejected at construction time, so a caller's static payload key can no longer silently shift meaning with delivery status.+- **`PipeSignature` — contract-only pipes for top-down design.** A new pipe type (`type = "PipeSignature"`) declares a pipe's `inputs`, `output`, and `description` with no implementation, so an author or agent can sketch a complete pipeline before committing to the operator that will eventually do the work. At dry-run time a signature mints a mock output matching its declared type and multiplicity; at runtime it raises `PipeSignatureNotExecutableError`. The optional `signature_for` field hints which pipe type the stub stands in for. See [Signature Pipes](https://docs.pipelex.com/latest/building-methods/pipes/signature-pipes/).++- **Lenient validation with `--allow-signatures`.** `pipelex validate pipe|bundle` and every `pipelex-agent validate` subcommand take `--allow-signatures` (default off) to dry-run a pipeline that still contains `PipeSignature` stubs. Strict by default: without the flag, validation refuses any pipeline whose dependency graph reaches a signature and raises `SignaturesNotAllowedError`, reporting every reachable stub plus the controller chain that leads to it — so you know exactly which pipes are still placeholders.-- **`--traceback` CLI flag for full stack traces.** By default CLI commands print a friendly one-line error; passing `--traceback` also prints the Rich-rendered stack trace before that message, for debugging. The flag is intercepted in `PipelexCLI.make_context` (mirroring `--no-logo`), so it is position-agnostic — it works both before and after the subcommand (`pipelex run --traceback pipe ...` and `pipelex run pipe ... --traceback`) without changing any command signature.+- **CSV tabular support (experimental).** Read a `.csv` straight into a typed `ListContent[YourConcept]`: point an `inputs.json` reference at a `.csv` whose column headers match the concept's field names, and each row becomes an instance, with cells coerced via Pydantic (`birth_year` → `int`, ISO dates → `date`; an empty cell becomes `None`, so the field must be optional). Write a flat-list output back out with the new `--save-csv <path>` flag on `pipelex run pipe|bundle|method`. v1 is deliberately narrow — local file paths only (remote URLs with a tabular suffix are rejected), the row concept must be flat (scalar fields only; a nested/list/concept-typed field is rejected with a clear error naming it), and `.xlsx` is recognized but routed to a "needs `pipelex[tabular]`" message (the Excel backend isn't built yet). See the [CSV Input & Output guide](https://docs.pipelex.com/latest/building-methods/pipes/csv-input-and-output/).++- **`--traceback` CLI flag for full stack traces.** By default CLI commands print a friendly one-line error; `--traceback` also prints the Rich-rendered stack trace before it. It's position-agnostic — `pipelex run --traceback pipe ...` and `pipelex run pipe ... --traceback` both work.++- **Test tooling for hanging runs.** New `make agent-test-debug` (alias `make atd`) runs the suite with upfront stale-process cleanup, an outer wall-clock timeout, and live per-test logging — for when `make agent-test` hangs or fails opaquely. Paired with a debugging playbook at `docs/agents/debugging-hanging-pytest-runs.md`.### Changed-- **Gemini deck: `gemini-3.0-pro` removed — Google shut down the underlying `gemini-3-pro-preview`.** Use `gemini-3.1-pro` (→ `gemini-3.1-pro-preview`) instead; it was already in the deck and is now also registered on the `portkey` backend for parity with `google`. The dead `gemini-3.0-pro` handle is dropped from the `google`, `portkey`, and `openrouter` backends and from the test-profile collections. `gemini-3.0-flash` and the `nano-banana-pro` image model (`gemini-3-pro-image-preview`) are unaffected. Anyone referencing `gemini-3.0-pro` must switch to `gemini-3.1-pro` (or the `gemini-pro-latest` alias). The Pipelex Gateway lists models from its own remote config — drop `gemini-3.0-pro` there separately.+- **`ErrorReport` is now a frozen Pydantic `BaseModel`** (was a frozen Pydantic dataclass) — still immutable and still round-trips through `to_dict()` / `from_dict()`, but an attempted mutation now raises `pydantic.ValidationError` instead of `dataclasses.FrozenInstanceError`. Relatedly, **`recover_error_report()` is now total**: it always returns an `ErrorReport`, synthesizing one from the new `UnrecoverableWorkflowFailureError` (surfacing the deepest worker-side cause) when a Temporal failure carries no embedded report — callers no longer branch on `None`.-- **`validate_bundle`, `validate_bundles_from_directory`, and `BundleValidator.validate_pipes` / `acquire_and_validate` now accept `allow_signatures: bool = False`.** Strict by default. The same flag is threaded through `pipelex/cli/commands/validate/_validate_core.py` (the CLI entry point) and `pipelex/cli/agent_cli/commands/validate/_validate_core.py` (the agent CLI), both strict by default. `ValidateBundleError` gained a `signature_check_error: SignaturesNotAllowedError | None` field so callers (and the CLI error renderer) can show the signature dep chain when validation fails on a signature.+- **Dry-run and validation consolidated into `BundleValidator`.** The standalone `dry_run_pipe` / `dry_run_pipes` functions and the modules `pipelex/pipe_run/dry_run.py`, `dry_run_with_graph.py`, and `dry_pipe_router.py` are gone; their work now lives in `BundleValidator`, and `validate_bundle` / `BundleValidator` gained the `allow_signatures: bool = False` flag (strict by default). Validation ordering and outputs are unchanged — only the import surface moved. Callers importing from `pipelex.pipe_run.dry_run` must switch to `BundleValidator`.-- **Dry-run and validation consolidated into `BundleValidator`; the standalone `pipelex/pipe_run/dry_run.py` is removed.** The free functions `dry_run_pipe` / `dry_run_pipes` and the modules `pipelex/pipe_run/dry_run.py`, `dry_run_with_graph.py`, and `dry_pipe_router.py` are gone. Their work now lives in `BundleValidator` (`pipelex/pipeline/bundle_validator.py`) with two lifecycles — `acquire_and_validate` (opens a fresh library, sweeps every pipe, owns teardown; the standalone `validate --all` path) and `validate_pipes` (the inner sweep against an already-open library, called by `validate_bundle`) — and the library/job setup steps are extracted into `pipelex/pipeline/execution_seams.py` (`acquire_library`, `prepare_pipe_job`). The mock-minting and stuff-spec converters (`convert_to_working_memory_format`, `convert_stuff_spec_to_typed_named`) moved onto `WorkingMemoryFactory` as classmethods. No behavior change — validation ordering (wiring pass → signature pre-pass → per-pipe dry-run sweep) and outputs are preserved; only the import surface moved. Callers importing from `pipelex.pipe_run.dry_run` must switch to `BundleValidator` / `WorkingMemoryFactory`.+- **`validated_pipes` now identifies every pipe by its qualified `pipe_ref` (`domain.code`).** Previously `validate pipe` reported the bare code while `validate bundle` / `validate all` reported the namespaced ref, so the same pipe could appear under two identities depending on which command produced it. Every validate surface now emits the qualified ref, which can't collide across domains. Consumers that parsed `pipe_code` expecting the bare form (e.g. the MTHDS skills) must match on the namespaced ref.-- **The `validated_pipes` JSON envelope now identifies every pipe by its namespaced `pipe_ref` (`domain.code`) on every validate surface.** Previously `pipelex-agent validate all` / `validate bundle` reported namespaced refs while `validate pipe` (and the builder validate ops) reported the bare code, so the same pipe could appear under two identifiers depending on which command produced it. All surfaces now emit the qualified ref, which cannot collide across domains. The shared projection helper `build_validated_pipes` dropped its `use_ref` parameter accordingly, and its entry type is now a named `ValidatedPipeEntry` `TypedDict`. Consumers that parsed `pipe_code` from `validated_pipes` expecting the bare form (e.g. the MTHDS skills) must match on the namespaced ref.+- **Gemini deck refresh.** Google shut down `gemini-3-pro-preview`, so the `gemini-3.0-pro` handle is removed across the `google`, `portkey`, and `openrouter` backends (and the test-profile collections) — use `gemini-3.1-pro` (→ `gemini-3.1-pro-preview`) instead, now registered on `portkey` too for parity with `google`. The `gemini-3.0-flash-preview` handle is renamed to `gemini-3.0-flash`. The Pipelex Gateway lists models from its own remote config, so drop `gemini-3.0-pro` there separately.-- **Filesystem path helpers in `pipelex.tools.misc.file_utils` and `pipelex.tools.misc.json_utils` now take and return `pathlib.Path` instead of `str`.** Helpers such as `save_text_to_path`, `load_text_from_path`, `load_binary` / `load_binary_async`, `load_json_from_path` / `load_json_dict_from_path` / `load_json_list_from_path`, `save_as_json_to_path`, `copy_file`, `remove_file` / `remove_folder`, `ensure_directory_exists` / `ensure_directory_for_file_path`, and `get_incremental_directory_path` / `get_incremental_file_path` now operate on `Path`. Path handling is `Path`-based throughout `pipelex/`, converting to/from `str` only at boundaries (CLI arguments, environment variables, JSON/TOML serialization). Callers passing bare strings must now wrap them with `Path(...)`.+- **Filesystem path helpers moved to `pathlib.Path`.** Helpers in `pipelex.tools.misc.file_utils` and `pipelex.tools.misc.json_utils` (`save_text_to_path`, `load_json_from_path`, `load_binary`, `copy_file`, `ensure_directory_exists`, `get_incremental_file_path`, and the rest) now take and return `Path` instead of `str`; path handling is `Path`-based throughout `pipelex/`, converting to/from `str` only at boundaries. Callers passing bare strings must wrap them with `Path(...)`.-- **`ErrorReport` is now a frozen Pydantic `BaseModel`** (previously a frozen Pydantic dataclass). It is still immutable and still round-trips through `to_dict()` / `from_dict()`, but an attempted mutation now raises `pydantic.ValidationError` instead of `dataclasses.FrozenInstanceError`.+- **Safer Temporal defaults.** `[temporal.search_attributes]` now ships `enabled = false` (opt in per env) so a local boot can't fail on the unregistered-attribute check, and the default `[temporal.queue_options.temporal_task_queue]` block is removed to avoid leaving an orphan overlay when a queue is renamed.-- **`recover_error_report()` is now a total function.** It returns an `ErrorReport` rather than `ErrorReport | None`: when a Temporal failure carries no embedded report — or its payload fails to rehydrate — it synthesizes one from the new `UnrecoverableWorkflowFailureError`, surfacing the deepest worker-side cause message. Callers no longer branch on `None`.+- **`teardown_current_library()` renamed to `clear_current_library()`** in `pipelex.hub`, pairing it cleanly with `set_current_library()`.-- **Dev tooling: `pyright` bumped `1.1.408 → 1.1.410`.** Drove two internal, behavior-neutral adjustments: `@contextmanager` / `@asynccontextmanager` helpers now annotate their return as `Generator` / `AsyncGenerator` (was `Iterator` / `AsyncIterator`), and `AsyncAnthropicBedrock` is imported from `anthropic.lib.bedrock` (relocated in the SDK).+- **Dev tooling: `pyright` bumped `1.1.408 → 1.1.410`** (drove two behavior-neutral internal adjustments).### Fixed-- **The generated MTHDS schema now requires `type` on every pipe.** In the schema consumed by `plxt` lint and the VS Code Taplo LSP, each pipe blueprint variant declared `type` with a literal default — so it was optional, and because the Draft-4 export drops the union discriminator, a pipe table written without `type` matched several `oneOf` branches at once and was rejected with an ambiguous multi-match (worse, a type-less table carrying fields unique to one variant validated silently). `type` is now required on every variant, so a type-less pipe table fails with a clear "missing type" and a typed table resolves to exactly one variant. The patched set is derived from `PipeBlueprintUnion`, so newly added pipe types are covered automatically. Regenerate with `pipelex-dev generate-mthds-schema`; the bundled copy ships in `pipelex-tools` 0.6.0 and the VS Code extension.+- **The generated MTHDS schema now requires `type` on every pipe.** In the schema consumed by `plxt` lint and the VS Code Taplo LSP, each pipe blueprint variant declared `type` with a literal default — so it was optional, and because the Draft-4 export drops the union discriminator, a pipe table written without `type` matched several `oneOf` branches at once and was rejected with an ambiguous multi-match (worse, a type-less table carrying fields unique to one variant validated silently). `type` is now required on every variant, so a type-less pipe table fails with a clear "missing type" and a typed table resolves to exactly one variant. The patched set derives from `PipeBlueprintUnion`, so new pipe types are covered automatically. Regenerate with `pipelex-dev generate-mthds-schema`.-- **`InputStuffSpecsFactoryError` was shadowed by a duplicate class definition.** `pipelex/core/pipes/inputs/input_stuff_specs_factory.py` declared a local class with the same name as the canonical one in the package's `exceptions.py`, leaving two distinct class objects in play so an `except` on one would miss the other. Consolidated to the single canonical class.+- **`InputStuffSpecsFactoryError` was shadowed by a duplicate class definition.** `input_stuff_specs_factory.py` declared a local class with the same name as the canonical one in `exceptions.py`, leaving two distinct class objects in play so an `except` on one would miss the other. Consolidated to the single canonical class.### Security-- **urllib3 floor `>=2.7.0`** to patch two high-severity issues: CVE-2026-44431 (GHSA-qccp-gfcp-xxvc) forwards sensitive headers across origins on proxied low-level redirects, and CVE-2026-44432 (GHSA-mf9v-mfxr-j63j) bypasses the decompression-bomb safeguards in parts of the streaming API. Floor added to the runtime `dependencies` in `pyproject.toml` so downstream installs cannot resolve a vulnerable version.-- **pymdown-extensions floor `>=10.21.3`** to patch CVE-2026-46338 (GHSA-62q4-447f-wv8h, medium): a regression in `pymdownx.snippets` reintroduced the sibling-prefix path-traversal bypass despite `restrict_base_path`. Docs-only dependency; floor added to the `docs` extra.+- **urllib3 floor `>=2.7.0`** to patch two high-severity issues: CVE-2026-44431 (GHSA-qccp-gfcp-xxvc) forwards sensitive headers across origins on proxied low-level redirects, and CVE-2026-44432 (GHSA-mf9v-mfxr-j63j) bypasses the decompression-bomb safeguards in parts of the streaming API. Floor added to the runtime `dependencies`.+- **pymdown-extensions floor `>=10.21.3`** to patch CVE-2026-46338 (GHSA-62q4-447f-wv8h, medium): a regression in `pymdownx.snippets` reintroduced the sibling-prefix path-traversal bypass despite `restrict_base_path`. Docs-only; floor added to the `docs` extra.- **idna floor `>=3.15`** (lockfile resolves to 3.18) to patch CVE-2026-45409 (GHSA-65pc-fj4g-8rjx, medium): specially crafted inputs to `idna.encode()` could bypass the CVE-2024-3651 fix. Floor added to the runtime `dependencies`.-- **Proactive floor bumps in the same hardening pass** — past known-vulnerable releases, so fresh resolves can't pick a bad version: pillow `>=12.1.1`, protobuf `>=6.33.5`, python-dotenv `>=1.2.2`, and requests `>=2.33.0` (runtime); aiohttp `>=3.12.14` (`temporal` extra); Pygments `>=2.20.0` (`docs` extra).--Together these close the open GitHub Dependabot alerts on the default branch — urllib3 (two high), pymdown-extensions (medium), and idna (medium); the alerts remain visible on `main` only until this branch merges.+- **Proactive floor bumps in the same hardening pass**, past known-vulnerable releases: pillow `>=12.1.1`, protobuf `>=6.33.5`, python-dotenv `>=1.2.2`, requests `>=2.33.0` (runtime); aiohttp `>=3.12.14` (`temporal` extra); Pygments `>=2.20.0` (`docs` extra).## [v0.30.3] - 2026-05-28