Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

Stabilization fixes and changelog polish (#1047)

Louis Choquelc76d7ff8980c
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bd7f61cc4..c1ff8f5b6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,58 +2,83 @@
## [Unreleased]
+### Highlights
+
+- **From resolved library to typed clients: `pipelex resolve` and `pipelex codegen`** — `pipelex resolve` distills a bundle's whole closure into the normalized library crate — a flat, fully-qualified, fingerprinted snapshot — and `pipelex codegen types` projects it into TypeScript (`ts-zod`) or Python (`python-pydantic`, `python-structures`) artifacts. A trust chain of stamped headers and a `codegen.lock` makes drift detectable offline: `pipelex codegen check` verifies artifacts by pure hashing — no engine, no network, no API key.
+- **Validation now fixes what it finds** — When an error has a deterministic safe fix, every surface says so: a 💡 suggested-fix line in the CLI output, a structured `suggested_fix` object in agent and API payloads. The new `pipelex fix bundle` / `pipelex-agent fix bundle` commands apply those fixes in place through a shared fix loop, with `--diff` to preview the changes without writing anything.
+- **Smart Inputs: the signature decides the shape** — Inputs are now interpreted top-down against the pipe's declared input signature instead of bottom-up from their shape alone: a bare string becomes the declared concept (never silently degrading to generic `Text`), lists shape element-wise, bare URLs/paths resolve for `Image`/`Document` inputs, and generated input templates default to the light bare-value form.
+- **New native concepts: `Date`, `Time`, and `YesNo`** — Built-in concepts for calendar dates (preserving source precision — no fabricated times of day), times of day, and boolean judgments, plus a `time` structure-field type completing the temporal triple. Every native now materializes from its pinned normative definition, so crate fingerprints byte-agree across implementations.
+
### Added
-- **`pipelex resolve` and the `pipelex codegen` family:** `pipelex resolve` assembles a bundle closure and emits the **normalized library crate** — a flat, fully-qualified, natives-expanded, self-contained, fingerprinted snapshot of a resolved library — as JSON or TOML. `pipelex codegen types --target ts-zod|python-pydantic|python-structures` projects the crate's concept set into typed artifacts, and `pipelex codegen inputs --pipe <ref>` renders a runnable inputs template for a pipe (defaulting to the closure's `main_pipe`). The `ts-zod` target emits a pure `types.ts` (imports only `zod`, with wire-native snake_case keys so a schema validates a wire payload directly) plus a `binder.ts` exposing a typed `parse`/`serialize` pair per concept; `python-pydantic` emits self-contained `BaseModel`s and `python-structures` emits runtime `StructuredContent` classes. Generated files are never edited — each carries an `AUTOGENERATED` header pointing at the sibling extension-file mechanism.
+
+- **`pipelex resolve` and the `pipelex codegen` family:** `pipelex resolve` assembles a bundle closure and emits the **normalized library crate** — a flat, fully-qualified, natives-expanded, self-contained, fingerprinted snapshot of a resolved library — as JSON or TOML. On top of the crate:
+
+ - `pipelex codegen types --target ts-zod|python-pydantic|python-structures` projects the crate's concept set into typed artifacts. The `ts-zod` target emits a pure `types.ts` (imports only `zod`, with wire-native snake_case keys so a schema validates a wire payload directly) plus a `binder.ts` exposing a typed `parse`/`serialize` pair per concept; `python-pydantic` emits self-contained `BaseModel`s; `python-structures` emits runtime `StructuredContent` classes.
+ - `pipelex codegen inputs --pipe <ref>` renders a runnable inputs template for a pipe (defaulting to the closure's `main_pipe`).
+ - Generated files are never edited — each carries an `AUTOGENERATED` header pointing at the sibling extension-file mechanism.
- **Codegen trust chain (stamps · `codegen.lock` · offline `codegen check`):** Every file `pipelex codegen types` writes now carries a self-describing **stamp** header (source crate fingerprint, engine version, projection, options, and a content hash of the body below it), and the artifact set is recorded in a sibling **`codegen.lock`**. `pipelex codegen check [DIR]` verifies generated artifacts are current **offline** — pure hashing, no engine, network, or API key — reporting drift by category (missing, modified, hand-edited, orphan) with the verdict on the exit code (`0` current · `1` drift · `2` no lock). Emission is idempotent (write-if-changed — unchanged output leaves files untouched) and prunes previously-generated files that drop out of the set, so a deleted concept never lingers as a stale generated file. Input templates (`codegen inputs`) are deliberately not stamped or locked — they are user-editable scaffolds, not tracked generated code.
+- **Agent-CLI codegen mirror (`pipelex-agent codegen types|check`):** The codegen family is now available to machine consumers through the agent CLI's two-stream envelopes (`--format markdown|json` success stream, `--error-format` error stream). `codegen types` runs the same engine and write-if-changed emission as the bare CLI; `codegen check` presents the offline drift verdict structurally — drift is a `CodegenDriftError` envelope enumerating `drifts[]` by category (exit `1`), a missing or unreadable `codegen.lock` is a no-verdict error (exit `2`). With the pipelex runner, `mthds-agent codegen …` forwards to these commands.
+- **Host-facing resolve/codegen engine cores (what the HTTP routes ride):** `pipelex.pipeline.resolve_bundle.resolve_crate_from_contents` resolves **in-memory** MTHDS contents into the normalized crate — same engine, same `ValidateBundleError` verdict vocabulary, and the same loaded-on-success library contract as `validate_bundle` — so a host (the `pipelex-api` `/v1/resolve` + `/v1/codegen` routes) serves resolution without touching the filesystem. `pipelex.codegen.emission.build_stamped_projection` is the new **pure** stamping core (stamped files + `codegen.lock` content, no disk writes) that `write_stamped_projection` now rides, so artifacts served over HTTP are byte-identical to locally written ones and pass the offline `codegen check` verbatim. The shared bundle-loading error cascade `translate_to_validate_bundle_error` is now public (renamed from its underscore-private form) since it gained a third entry point.
- **Agent bundle fixer:** Added `pipelex-agent fix bundle`, a deterministic in-place fixer over the shared validation fix loop. The command supports `--format`/`--error-format`, repeatable `--select`/`--ignore` fix-rule filters, `--allow-signatures`, `--max-iterations`, and the same bundle-file or directory resolution as `pipelex-agent validate bundle`. Successful fix results report `pending_signatures` and `is_runnable`; by default the command exits non-zero when a fixed bundle remains valid-but-not-runnable.
- **Human bundle fixer:** Added `pipelex fix bundle`, the human counterpart of the agent fixer over the same fix loop, with Rich-rendered output naming every change made (fix descriptions, files written, iteration count), the same `--select`/`--ignore`/`--allow-signatures`/`--max-iterations` options, and the same bundle-file or directory resolution as `pipelex validate bundle`. A new `--diff` flag previews the changes as a unified diff without writing anything — the real fix loop runs against a temp-copy sandbox, and exit codes keep the verdict semantics so `--diff` answers "would it converge?".
- **Suggested fixes surfaced in `pipelex validate`:** When a validation error has a deterministic safe fix, the human error output now shows a per-error `💡 Suggested fix:` line and ends with an actionable footer naming the exact `pipelex fix bundle` command (echoing the invocation's `-L` dirs) in place of the generic tip.
- **`suggested_fix` field in `/validate` payloads:** Each validation error that has a deterministic safe fix now carries an additive structured `suggested_fix` object — `fix_code` (the kebab-case rule id), `description` (the author-facing 💡 text), `safety` (`safe`/`unsafe`), optional `source` file, and `ops` (the semantic patch operations) — on every `validation_errors[]` item, both in the `pipelex-agent validate`/`fix` JSON envelopes and in the API `/validate` response body. This is the machine-actionable counterpart of the 💡 line, letting a software consumer apply the fix directly from the structured payload.
- **Smart Inputs (signature-driven input shaping):** Inputs are now interpreted top-down against the pipe's declared input signature rather than bottom-up from their shape alone.
- - Bare values (strings, numbers, booleans, dicts, lists) are automatically shaped into the declared concept (e.g., a bare string becomes a `legal.Question`, not a generic `native.Text`).
- - Lists shape element-wise into the declared item concept; single values auto-wrap into lists where required, and empty lists are now legal.
- - Bare URLs/paths for `Image` or `Document` inputs are resolved relative to the inputs file's directory.
- - Declared structured lists (e.g., `Person[]`) accept a direct path to a `.csv` file by signature, without requiring an envelope.
+
+ - Bare values (strings, numbers, booleans, dicts, lists) are automatically shaped into the declared concept (e.g., a bare string becomes a `legal.Question`, not a generic `native.Text`).
+ - Lists shape element-wise into the declared item concept; single values auto-wrap into lists where required, and empty lists are now legal.
+ - Bare URLs/paths for `Image` or `Document` inputs are resolved relative to the inputs file's directory.
+ - Declared structured lists (e.g., `Person[]`) accept a direct path to a `.csv` file by signature, without requiring an envelope.
+- **Comprehensive input error handling:** A suite of typed errors for input shaping failures (`WrongScalarKindError`, `ListWhereSingularError`, `MultiplicityCountMismatchError`, `StructureValidationError`, `UnknownInputNameError`) that provide clear hints and render the expected JSON shape.
+- **CLI `--explicit` flag:** Added `--explicit` to `pipelex build inputs` and `pipelex-agent inputs` to generate the legacy `{"concept", "content"}` envelope templates.
- **Native `YesNo` concept:** Built-in concept (backed by `YesNoContent`) for boolean judgments. Requires a strict boolean, renders as `yes`/`no` in downstream prompts, and can be read via `pipe_output.main_stuff_as_yes_no.yes_no`.
- **Native `Date` concept:** Built-in concept (backed by `DateContent`) for calendar dates. Requires a `datetime.date` and accepts an optional `datetime.time`. Preserves source precision (never forcing the LLM to fabricate a time of day) and retains UTC offsets verbatim.
-- **Native `Time` concept:** Built-in concept (backed by `TimeContent`) for a time of day, optionally with a UTC offset — never attached to an invented date. A top-level TOML time-of-day literal in an inputs file now maps to `Time` (breaking: `Time` joins the reserved native concept codes, and the former bare-time rejection — `InputsTimeOnlyNotSupportedError` — is gone).
+- **Native `Time` concept:** Built-in concept (backed by `TimeContent`) for a time of day, optionally with a UTC offset — never attached to an invented date. A top-level TOML time-of-day literal in an inputs file now maps to `Time` (Breaking: `Time` joins the reserved native concept codes, and the former bare-time rejection — `InputsTimeOnlyNotSupportedError` — is gone).
- **Structure-field `type = "time"`:** The concept structure language gains a `time` field type (a time of day, optionally with a UTC offset), completing the temporal triple beside `date` and `datetime`. Generates a `datetime.time` field; emitted by every codegen target.
- **Pinned normative native definitions:** Native concepts now materialize into the normalized crate by **lookup into the standard's pinned per-version definitions** (`pipelex/core/concepts/native/pinned_blueprints.py`, mirroring the mthds spec's new `native-concepts.md`), never by reflection over the runtime content classes — so crate fingerprints over materialized natives byte-agree across implementations. The old reflection is retained solely as a consistency test proving each runtime content class still matches its pinned blueprint.
-- **Agent-CLI codegen mirror (`pipelex-agent codegen types|check`):** the codegen family is now available to machine consumers through the agent CLI's two-stream envelopes (`--format markdown|json` success stream, `--error-format` error stream). `codegen types` runs the same engine and write-if-changed emission as the bare CLI; `codegen check` presents the offline drift verdict structurally — drift is a `CodegenDriftError` envelope enumerating `drifts[]` by category (exit `1`), a missing or unreadable `codegen.lock` is a no-verdict error (exit `2`). With the pipelex runner, `mthds-agent codegen …` forwards to these commands.
-- **Host-facing resolve/codegen engine cores (what the HTTP routes ride):** `pipelex.pipeline.resolve_bundle.resolve_crate_from_contents` resolves **in-memory** MTHDS contents into the normalized crate — same engine, same `ValidateBundleError` verdict vocabulary, and the same loaded-on-success library contract as `validate_bundle` — so a host (the `pipelex-api` `/v1/resolve` + `/v1/codegen` routes) serves resolution without touching the filesystem. `pipelex.codegen.emission.build_stamped_projection` is the new **pure** stamping core (stamped files + `codegen.lock` content, no disk writes) that `write_stamped_projection` now rides, so artifacts served over HTTP are byte-identical to locally written ones and pass the offline `codegen check` verbatim. The shared bundle-loading error cascade `translate_to_validate_bundle_error` is now public (renamed from its underscore-private form) since it gained a third entry point.
-- **Comprehensive input error handling:** A suite of typed errors for input shaping failures (`WrongScalarKindError`, `ListWhereSingularError`, `MultiplicityCountMismatchError`, `StructureValidationError`, `UnknownInputNameError`) that provide clear hints and render the expected JSON shape.
-- **CLI `--explicit` flag:** Added `--explicit` to `pipelex build inputs` and `pipelex-agent inputs` to generate the legacy `{"concept", "content"}` envelope templates.
-- **Subject-grants registry — the keyword-only convention hardened:** a positional first parameter ("subject") in `pipelex/` source is now legal only under an explicit grant recorded in `subject_grants.toml` at the repo root, carrying the subject's param name and an honest, def-specific review rationale — every grant in the registry has been individually reviewed against the grant rubric. `pipelex-dev subject-grant` is the sole registry writer; the `check-keyword-only` AST guard enforces the registry symmetrically (an ungranted subject fails, and so does a stale grant whose def was renamed, moved, demoted, or newly carved out) in `make check`, CI, and the edit-time hook. Subjects typed `bool`/`int`/`float` are banned outright — no grant can cover them. The registry schema is strict: exactly `param` + `rationale`, unknown keys fail the check. See `docs/contribute/keyword-only-arguments.md`.
-- **Drift contracts (`pipelex-dev drift plan|check|ack`):** Deterministic review obligations between code and docs. A root `drift.toml` declares contracts ("when these trigger files change, these review targets must be re-examined and the examination recorded"); committed ack files under `.drift/acks/` record each fulfilled review with a digest, reviewer, and rationale. A contract is fulfilled iff its stored ack digest equals the digest recomputed from the current tree (staged blob OIDs from the git index — no base ref, no timestamps), so trigger edits, matched-file adds/deletes, and contract-definition edits all reopen it. `make drift-check` gates the `make check` aggregate and runs as an advisory CI job; `make drift-plan` prints per-contract Markdown packets with exact per-file changes since the last ack; `make drift-ack CONTRACT=… RATIONALE="…"` runs the contract's verify commands then records the review and stages the ack file it writes, so the local gate checks the same index the commit is built from (and if staging that ack fails, the ack file is removed rather than left unstaged, so a later check reports a missing ack instead of a false green). For a contract with verify commands, a trigger file left unstaged-modified or untracked is a hard ack error (the verify run would certify content the digest does not cover) — checked both before and after the verify commands, so a trigger the commands themselves format or generate is caught too; for other contracts it stays a warning. Seeded with contracts for the config docs, the CLI docs, and the keyword-only convention spec; workflow documented in `docs/contribute/drift-contracts.md`.
+- **Subject-grants registry — the keyword-only convention hardened:** A positional first parameter ("subject") in `pipelex/` source is now legal only under an explicit grant recorded in `subject_grants.toml` at the repo root, carrying the subject's param name and an honest, def-specific review rationale — every grant in the registry has been individually reviewed against the grant rubric. `pipelex-dev subject-grant` is the sole registry writer; the `check-keyword-only` AST guard enforces the registry symmetrically (an ungranted subject fails, and so does a stale grant whose def was renamed, moved, demoted, or newly carved out) in `make check`, CI, and the edit-time hook. Subjects typed `bool`/`int`/`float` are banned outright — no grant can cover them. The registry schema is strict: exactly `param` + `rationale`, unknown keys fail the check. See `docs/contribute/keyword-only-arguments.md`.
+- **Drift contracts (`pipelex-dev drift plan|check|ack`):** Deterministic review obligations between code and docs. A root `drift.toml` declares contracts ("when these trigger files change, these review targets must be re-examined and the examination recorded"); committed ack files under `.drift/acks/` record each fulfilled review with a digest, reviewer, and rationale. A contract is fulfilled iff its stored ack digest equals the digest recomputed from the current tree (staged blob OIDs from the git index — no base ref, no timestamps), so trigger edits, matched-file adds/deletes, and contract-definition edits all reopen it.
+
+ - `make drift-check` gates the `make check` aggregate and runs as an advisory CI job; `make drift-plan` prints per-contract Markdown packets with exact per-file changes since the last ack.
+ - `make drift-ack CONTRACT=… RATIONALE="…"` runs the contract's verify commands, then records the review and stages the ack file it writes, so the local gate checks the same index the commit is built from. If staging that ack fails, the ack file is removed rather than left unstaged, so a later check reports a missing ack instead of a false green.
+ - For a contract with verify commands, a trigger file left unstaged-modified or untracked is a hard ack error (the verify run would certify content the digest does not cover) — checked both before and after the verify commands, so a trigger the commands themselves format or generate is caught too. For other contracts it stays a warning.
+ - Seeded with contracts for the config docs, the CLI docs, and the keyword-only convention spec; workflow documented in `docs/contribute/drift-contracts.md`.
### Changed
-- (Breaking) **Keyword-only demotions across the public API:** the subject-grant review demoted every positional first parameter that failed the grant rubric to keyword-only, and external code calling these surfaces positionally (or subclassing them with positional signatures) breaks. Notable demoted families: the `PipeAbstract` run family (`run_pipe`, `dry_run_pipe`, `validate_before_run`, … — `job_metadata`/`visited_pipes` now keyword-only), `StuffContent.rendered_for_prompt`/`rendered_for_template_async` (`text_format`) and the `TextFormatRenderable`/`ImageRenderable` protocols with all their implementations, the `LibraryManagerAbstract` load family (`library_id`), `ensure_pipelex_booted` (`config_overrides`), `pipeline_run_setup` (`execution_config`), `ConceptLibrary.is_compatible` (`tested_concept`), `GraphTracerProtocol.setup` (`graph_id`), the `PromptImageFactory`/`PromptDocumentFactory` content factories (`uri`), the model-manager/backend-library setup+load (`secrets_provider`), the reporting protocol's `set_event_log`/`clear_event_log` (`context_key`), and assorted tools helpers (`pluralize`/`count_with_noun`, `LogLevel.from_int`, `set_dry_run_forced`, …).
-- (Breaking) **`pipelex build structures` is now an alias of `codegen types --target python-structures`:** the legacy always-qualified per-file generator is deleted. The alias resolves the target's whole directory closure and emits one stamped `structures.py` module plus a `codegen.lock` (bare-when-unique class names, declared imprecision instead of the old silent `TextContent` guess for structureless concepts). The `--force` flag is gone (emission is write-if-changed), a `.mthds` file target now resolves its parent directory's closure, and concepts-only loading of invalid-pipe bundles is no longer supported (the crate requires a valid library). The concepts-only loading API (`load_concepts_only`, `load_concepts_only_from_directory`, and the library-manager methods behind them) is removed with it.
-- (Breaking) **`pipelex build runner` emits its structures through the codegen engine:** the scaffolded `structures/` directory now contains the stamped single-module projection (`structures.py` + `codegen.lock`) instead of per-concept files, and the generated runner script imports and instantiates the emitted class spellings (`from structures.structures import Invoice`) — fixing dropped imports for nested custom concepts along the way. A user class backing a `structure = "<ClassName>"` concept is imported from its real module.
+
+- **Keyword-only demotions across the public API (Breaking):** The subject-grant review demoted every positional first parameter that failed the grant rubric to keyword-only, and external code calling these surfaces positionally (or subclassing them with positional signatures) breaks. Notable demoted families: the `PipeAbstract` run family (`run_pipe`, `dry_run_pipe`, `validate_before_run`, … — `job_metadata`/`visited_pipes` now keyword-only), `StuffContent.rendered_for_prompt`/`rendered_for_template_async` (`text_format`) and the `TextFormatRenderable`/`ImageRenderable` protocols with all their implementations, the `LibraryManagerAbstract` load family (`library_id`), `ensure_pipelex_booted` (`config_overrides`), `pipeline_run_setup` (`execution_config`), `ConceptLibrary.is_compatible` (`tested_concept`), `GraphTracerProtocol.setup` (`graph_id`), the `PromptImageFactory`/`PromptDocumentFactory` content factories (`uri`), the model-manager/backend-library setup+load (`secrets_provider`), the reporting protocol's `set_event_log`/`clear_event_log` (`context_key`), and assorted tools helpers (`pluralize`/`count_with_noun`, `LogLevel.from_int`, `set_dry_run_forced`, …).
+- **`pipelex build structures` is now an alias of `codegen types --target python-structures` (Breaking):** The legacy always-qualified per-file generator is deleted. The alias resolves the target's whole directory closure and emits one stamped `structures.py` module plus a `codegen.lock` (bare-when-unique class names, declared imprecision instead of the old silent `TextContent` guess for structureless concepts). The `--force` flag is gone (emission is write-if-changed), a `.mthds` file target now resolves its parent directory's closure, and concepts-only loading of invalid-pipe bundles is no longer supported (the crate requires a valid library). The concepts-only loading API (`load_concepts_only`, `load_concepts_only_from_directory`, and the library-manager methods behind them) is removed with it.
+- **`pipelex build runner` emits its structures through the codegen engine (Breaking):** The scaffolded `structures/` directory now contains the stamped single-module projection (`structures.py` + `codegen.lock`) instead of per-concept files, and the generated runner script imports and instantiates the emitted class spellings (`from structures.structures import Invoice`) — fixing dropped imports for nested custom concepts along the way. A user class backing a `structure = "<ClassName>"` concept is imported from its real module.
- **Human validation error rendering routed through the shared structured items:** `pipelex validate`'s error details now render from the same `ValidationErrorItem`s the agent CLI and API emit, which surfaces information the old renderer silently dropped: pipe-factory errors (e.g. an unknown concept) get their own section, a parse-level failure (e.g. a TOML syntax error) now shows its message instead of no detail at all, and pipe validation errors name their source file. The dry-run message remains visible alongside categorized errors.
- **Validation error messages now speak MTHDS author syntax:** Pipe validation errors state the problem and the fix the way an author writes them — a sequence output-multiplicity mismatch now reads "declares its output as 'StoryIdea', but its last step yields 'StoryIdea[]'. Update the sequence's output to 'StoryIdea[]'", and an input-type mismatch reads "input 'dish' is declared as 'Number' but its step needs 'Text'. Update the input to 'Text'" — instead of leaking Python internals (`multiplicity=None`, `Concept(...)`/`PresenceMarker` reprs). The misleading `(required: […], provided: …)` suffix is suppressed for multiplicity errors (where both sides are identical by definition) and rendered as joined author-syntax refs — never Python list-repr brackets — elsewhere.
- **Agent `validate` / `fix` output renders prose, not a JSON dump:** `pipelex-agent validate bundle` and `pipelex-agent fix bundle` markdown output now renders validation errors as category-grouped prose (a humanized title, identity fields, the message, and a `💡 Suggested fix:` line) followed by a fix-aware footer naming the exact `pipelex-agent fix bundle` command when a safe fix exists — mirroring the human panel. The machine-facing JSON stream (`--format json`) is unchanged, so software consumers keep branching on the same structured fields.
- **Clean validation summary message on every surface:** The top-level error `message` (agent JSON, the API `/validate` body) is now a clean, author-facing summary derived from the structured errors, instead of the raw pydantic dump (`Validation error(s): … Value errors: '<field>': Value error, …`). The API's opt-in invalid-verdict `rendered_markdown` now carries the same per-error `💡 Suggested fix` prose as the CLI. A still-invalid fix that stops with no progress now names the re-proposed fix in author terms rather than dumping an internal fingerprint string.
-- (Breaking) **Input templates default to "light" shape:** `pipelex build inputs` and `pipelex-agent inputs` now generate light templates by default, outputting the bare values expected by the signature rather than the verbose `{"concept", "content"}` envelopes. TOML templates include the declared concept as a `# concept: ...` comment.
-- (Breaking) **Structure-field `type = "date"` behavior:** A `date` field now generates a `datetime.date` (JSON schema `format: date`), preventing LLMs from hallucinating times for calendar dates. A new `type = "datetime"` field handles full timestamps (`format: date-time`), and default-value validation now rejects a `datetime` default on a `date` field.
-- (Breaking) **Strict input validation:** Undeclared input names are now a hard error (previously silently ignored), and top-level `null` values are rejected—absence must be expressed by omitting the key.
-- (Breaking) **Explicit envelope validation:** The `{"concept", "content"}` escape hatch is now compatibility-checked against the pipe's declared signature; incompatible concepts raise a typed error.
-- (Breaking) **TOML date/time handling:** TOML date and datetime literals are natively supported and map directly to the `Date` concept; a bare time-of-day literal maps to the new `Time` concept (it never silently becomes a `Date` — shaping a `Time` into a `Date` slot fails the compatibility check).
-- (Breaking) **`ImageContent` flattens its pixel dimensions:** `size: ImageSize | None` is replaced by paired optional `width` / `height` integer fields (both-or-neither, enforced by a validator) — a named nested shape in the crate is a concept, and the size grouping did not earn that citizenship. `ImageSize` remains as the internal img-gen parameter type. Image wire payloads change shape accordingly.
-- (Breaking) **`native.Date` declares real structure:** materialized `Date` is no longer structureless — its pinned definition carries a required `date` field plus an optional `time` field (the new `time` field type), so generated clients get real Date types. `DateContent`'s wire form is unchanged.
-- **Documentation overhaul:** Updated `provide-inputs.md`, `executing-pipelines.md`, and `native-concepts.md` for the Smart Inputs paradigm, leading with the bare-value approach and demoting the envelope format to an escape hatch.
-
-### Fixed
-- **`resolve` / `codegen` rejected bundles that pin a model:** the resolve and codegen commands booted without model specs, so library validation could not check pipe model pins (`model = "gpt-4o-mini"`) against the deck and rejected any bundle carrying one. They now boot like `validate` — offline, but with model specs loaded.
-- **Native `Image` no longer materializes structureless:** every native now materializes from its pinned normative definition, so generated clients for image-producing pipes expose `url`, `public_url`, `caption`, `width`, `height`, etc. (The brief interim dict-with-imprecision mapping for the nested size model is superseded by the `ImageContent` flatten above.)
-- **Deterministic input-template placeholders:** mock URL placeholders in generated inputs templates (`codegen inputs` / `build inputs`) are now deterministic (`https://mock.invalid/<field>`), so committed templates no longer churn on every regeneration.
-- **`codegen inputs` is write-if-changed:** like `codegen types`, an already-current inputs template is left untouched (no mtime churn) and the console reports `Unchanged` instead of claiming `Generated`.
-- **Opaque generated classes no longer strip content:** a structureless or Python-class-backed concept's generated Python class now carries `model_config = ConfigDict(extra="allow")`, so `model_validate` keeps the payload verbatim instead of pydantic's default `extra="ignore"` silently dropping every field. Opaque is pass-through, never lossy — matching the ts-zod `z.unknown()` behavior.
+- **Input templates default to "light" shape (Breaking):** `pipelex build inputs` and `pipelex-agent inputs` now generate light templates by default, outputting the bare values expected by the signature rather than the verbose `{"concept", "content"}` envelopes. TOML templates include the declared concept as a `# concept: ...` comment.
+- **Structure-field `type = "date"` behavior (Breaking):** A `date` field now generates a `datetime.date` (JSON schema `format: date`), preventing LLMs from hallucinating times for calendar dates. A new `type = "datetime"` field handles full timestamps (`format: date-time`), and default-value validation now rejects a `datetime` default on a `date` field.
+- **Strict input validation (Breaking):** Undeclared input names are now a hard error (previously silently ignored), and top-level `null` values are rejected — absence must be expressed by omitting the key.
+- **Explicit envelope validation (Breaking):** The `{"concept", "content"}` escape hatch is now compatibility-checked against the pipe's declared signature; incompatible concepts raise a typed error.
+- **TOML date/time handling (Breaking):** TOML date and datetime literals are natively supported and map directly to the `Date` concept; a bare time-of-day literal maps to the new `Time` concept (it never silently becomes a `Date` — shaping a `Time` into a `Date` slot fails the compatibility check).
+- **`ImageContent` flattens its pixel dimensions (Breaking):** `size: ImageSize | None` is replaced by paired optional `width` / `height` integer fields (both-or-neither, enforced by a validator) — a named nested shape in the crate is a concept, and the size grouping did not earn that citizenship. `ImageSize` remains as the internal img-gen parameter type. Image wire payloads change shape accordingly.
+- **`native.Date` declares real structure (Breaking):** Materialized `Date` is no longer structureless — its pinned definition carries a required `date` field plus an optional `time` field (the new `time` field type), so generated clients get real Date types. `DateContent`'s wire form is unchanged.
+
+### Fixed
+
+- **Library loaders now bind the target library for the whole load:** `load_from_blueprints` / `load_from_crate` / `load_libraries` took an explicit `library_id` but resolved concepts and the class registry through the ambient *current-library* ContextVar one hop below — so loading into a library that wasn't current read (and registered generated structure classes into) whatever library the ambient pointer happened to name. Production hosts always set the target current before loading, so this only bit multi-library callers — exactly the isolation the loaders exist to provide. The loaders (and blueprint removal) now wrap their body in `scoped_current_library(library_id)`, restoring the caller's binding on exit, so everything beneath a load resolves against the load's target by construction. A regression test loads into a deliberately non-current library.
+- **`PipeExtract` broke crate normalization (500s on every static-core route):** `PipeExtractBlueprint.validate_output` only accepted the authoring spelling `Page[]`, but normalization qualifies concept refs (`Page[]` → `native.Page[]`) and then rebuilds the blueprints — so the validator rejected its own normalized output, and any bundle containing a `PipeExtract` raised a raw pydantic `ValidationError` out of `resolve_crate_from_contents`, 500-ing `/v1/build/inputs`, `/v1/build/output`, `/v1/resolve`, and `/v1/codegen`. The validator now compares the *parsed* concept ref and multiplicity instead of the raw string, so it holds for both spellings. The output contract is unchanged and still exact — a variable-length list of native `Page` — since a refined output would be an unverifiable promise: `PipeExtract` mechanically produces native pages and cannot make them more specific.
+- **`resolve` / `codegen` rejected bundles that pin a model:** The resolve and codegen commands booted without model specs, so library validation could not check pipe model pins (`model = "gpt-4o-mini"`) against the deck and rejected any bundle carrying one. They now boot like `validate` — offline, but with model specs loaded.
+- **Native `Image` no longer materializes structureless:** Every native now materializes from its pinned normative definition, so generated clients for image-producing pipes expose `url`, `public_url`, `caption`, `width`, `height`, etc. (The brief interim dict-with-imprecision mapping for the nested size model is superseded by the `ImageContent` flatten above.)
+- **Deterministic input-template placeholders:** Mock URL placeholders in generated inputs templates (`codegen inputs` / `build inputs`) are now deterministic (`https://mock.invalid/<field>`), so committed templates no longer churn on every regeneration.
+- **`codegen inputs` is write-if-changed:** Like `codegen types`, an already-current inputs template is left untouched (no mtime churn) and the console reports `Unchanged` instead of claiming `Generated`.
+- **Opaque generated classes no longer strip content:** A structureless or Python-class-backed concept's generated Python class now carries `model_config = ConfigDict(extra="allow")`, so `model_validate` keeps the payload verbatim instead of pydantic's default `extra="ignore"` silently dropping every field. Opaque is pass-through, never lossy — matching the ts-zod `z.unknown()` behavior.
- **Human CLI `~` expansion:** `pipelex validate bundle` (and the new `pipelex fix bundle`) now expand `~` in the bundle path and `-L/--library-dir` values, matching the agent CLI.
-- **Validate telemetry label:** the `CLI_COMMAND` telemetry tag for `pipelex validate` subcommands no longer double-suffixes (previously "validate bundle bundle").
+- **Validate telemetry label:** The `CLI_COMMAND` telemetry tag for `pipelex validate` subcommands no longer double-suffixes (previously "validate bundle bundle").
- **Silent mistyping of inputs:** Providing a bare string for a refined text concept (e.g., `legal.Question`) no longer silently degrades the type to a generic `native.Text`; the declared concept type is now retained.
- **TOML inline table formatting:** Light templates now use inline tables for structured values, preventing trailing scalars from being swallowed into `[sections]`.
+### Documentation
+
+- **Smart Inputs guides:** Updated `provide-inputs.md`, `executing-pipelines.md`, and `native-concepts.md` for the Smart Inputs paradigm, leading with the bare-value approach and demoting the envelope format to an escape hatch.
+
## [v0.38.0] - 2026-07-06
### Highlights
diff --git a/pipelex/codegen/emitters/python_common.py b/pipelex/codegen/emitters/python_common.py
index b41450197..1a0acd8f4 100644
--- a/pipelex/codegen/emitters/python_common.py
+++ b/pipelex/codegen/emitters/python_common.py
@@ -71,12 +71,16 @@ def format_default_value(value: Any) -> str:
def class_docstring(description: str, *, extra_line: str | None = None, indent: str = " ") -> str:
- """Render a class docstring; append `extra_line` (e.g. an imprecision caveat) on its own line."""
+ """Render a triple-quoted class docstring; append `extra_line` (e.g. an imprecision caveat) on its own line.
+
+ The body goes through `escape_py_string`, which escapes every `"` (and backslashes, newlines, control
+ chars), so a hostile description can never form a bare `\"\"\"` and break out of the docstring.
+ """
text = description or ""
if extra_line:
- body = f"{text}\n\n{indent}{extra_line}" if text else extra_line
- return f"{indent}{escape_py_string(body)}"
- return f"{indent}{escape_py_string(text)}"
+ text = f"{text}\n\n{indent}{extra_line}" if text else extra_line
+ quoted = escape_py_string(text)
+ return f'{indent}"""{quoted[1:-1]}"""'
def field_line(field: ResolvedField, *, annotation: str) -> str:
diff --git a/pipelex/codegen/emitters/target.py b/pipelex/codegen/emitters/target.py
index 61c5e5888..81a2867d4 100644
--- a/pipelex/codegen/emitters/target.py
+++ b/pipelex/codegen/emitters/target.py
@@ -1,11 +1,11 @@
"""The two codegen axes (`kind` and `target`) and the emitted-file unit shared by every emitter.
Codegen has exactly two explicit axes (see `docs/specs/pipelex-codegen.md` → "Two axes"): `kind` is
-*what* to project (`types` over the concept set, `inputs` per pipe, …), `target` is *for whom* (a
-language / idiom flavor). An emitter returns one or more `EmittedFile`s — a filename relative to the
-output root plus its content — so a single projection can span more than one file (the ts-zod purity
-split: a pure types file plus a thin binder file). The `kind`/`target` pair is what a generated file's
-stamp records as its projection.
+*what* to project (`types`, over the crate's concept set), `target` is *for whom* (a language / idiom
+flavor). An emitter returns one or more `EmittedFile`s — a filename relative to the output root plus
+its content — so a single projection can span more than one file (the ts-zod purity split: a pure
+types file plus a thin binder file). The `kind`/`target` pair is what a generated file's stamp records
+as its projection.
"""
from enum import StrEnum
@@ -14,12 +14,17 @@ from pydantic import BaseModel, ConfigDict
class CodegenKind(StrEnum):
- """A codegen projection `kind` — the *what* axis. Only the kinds that emit tracked artifacts today
- are listed: `types` (over the concept set) and `inputs` (per pipe). Recorded in a file's stamp.
+ """A codegen projection `kind` — the *what* axis, as recorded in a generated file's stamp.
+
+ This is the **stamped-kind vocabulary**, deliberately narrower than the spec's `kind` axis: a kind
+ is a member exactly when its artifacts ride the stamp/lock/offline-check trust chain. Input
+ templates are the standing counter-example — they are user-editable scaffolds, not tracked
+ generated code, so `pipelex codegen inputs` writes them unstamped and unlocked on purpose (see
+ `pipelex/codegen/emission.py`). Nothing can ever stamp one, hence no `inputs` member. A future
+ per-pipe kind (`docs`, `tools`, `tests`) joins this enum exactly when its artifacts are tracked.
"""
TYPES = "types"
- INPUTS = "inputs"
class CodegenTarget(StrEnum):
diff --git a/pipelex/libraries/library_manager.py b/pipelex/libraries/library_manager.py
index 8500327c0..d577544c3 100644
--- a/pipelex/libraries/library_manager.py
+++ b/pipelex/libraries/library_manager.py
@@ -28,7 +28,7 @@ from pipelex.core.pipes.pipe_factory import PipeFactory
from pipelex.core.qualified_ref import QualifiedRef
from pipelex.core.stuffs.structured_content import StructuredContent
from pipelex.core.validation import report_validation_error
-from pipelex.hub import get_class_registry, get_current_library, get_current_library_id_or_none
+from pipelex.hub import get_class_registry, get_current_library, get_current_library_id_or_none, scoped_current_library
from pipelex.libraries.collision_messages import duplicate_ref_msg
from pipelex.libraries.concept.exceptions import ConceptLibraryError
from pipelex.libraries.concept_reference_validation import validate_concept_references_in_blueprints
@@ -286,55 +286,59 @@ class LibraryManager(LibraryManagerAbstract):
msg = f"Trying to load a library that does not exist: '{library_id}'"
raise LibraryError(msg)
- if not library_dirs:
- library_dirs = []
-
- all_dirs: list[Path] = []
- all_mthds_paths: list[Path] = []
- all_dirs.extend(library_dirs)
- all_mthds_paths.extend(get_pipelex_mthds_files_from_dirs(set(library_dirs)))
-
- if library_file_paths:
- all_mthds_paths.extend(library_file_paths)
+ # Bind the target library as current for the whole load: class registration and the
+ # blueprint loads beneath resolve through the ambient current library, so a load into
+ # a non-current library must not read the caller's binding.
+ with scoped_current_library(library_id=library_id):
+ if not library_dirs:
+ library_dirs = []
+
+ all_dirs: list[Path] = []
+ all_mthds_paths: list[Path] = []
+ all_dirs.extend(library_dirs)
+ all_mthds_paths.extend(get_pipelex_mthds_files_from_dirs(set(library_dirs)))
+
+ if library_file_paths:
+ all_mthds_paths.extend(library_file_paths)
+
+ # Combine and deduplicate
+ seen_absolute_paths: set[str] = set()
+ valid_mthds_paths: list[Path] = []
+ for mthds_path in all_mthds_paths:
+ try:
+ absolute_path = str(mthds_path.resolve())
+ except (OSError, RuntimeError):
+ # For paths that can't be resolved (e.g., in zipped packages), use string representation
+ absolute_path = str(mthds_path)
+
+ if absolute_path not in seen_absolute_paths:
+ valid_mthds_paths.append(mthds_path)
+ seen_absolute_paths.add(absolute_path)
+
+ # Import modules and register in global registries
+ # Import from user directories
+ for library_dir in all_dirs:
+ # Only import files that contain StructuredContent subclasses (uses AST pre-check)
+ ClassRegistryUtils.import_modules_in_folder(
+ folder_path=library_dir,
+ base_class_names=[StructuredContent.__name__],
+ force_include_dirs=[Path(builder_pkg.__file__).parent],
+ )
+ # Only import files that contain @pipe_func decorated functions (uses AST pre-check)
+ FuncRegistryUtils.register_funcs_in_folder(
+ folder_path=library_dir,
+ force_include_dirs=[Path(builder_pkg.__file__).parent],
+ )
- # Combine and deduplicate
- seen_absolute_paths: set[str] = set()
- valid_mthds_paths: list[Path] = []
- for mthds_path in all_mthds_paths:
- try:
- absolute_path = str(mthds_path.resolve())
- except (OSError, RuntimeError):
- # For paths that can't be resolved (e.g., in zipped packages), use string representation
- absolute_path = str(mthds_path)
-
- if absolute_path not in seen_absolute_paths:
- valid_mthds_paths.append(mthds_path)
- seen_absolute_paths.add(absolute_path)
-
- # Import modules and register in global registries
- # Import from user directories
- for library_dir in all_dirs:
- # Only import files that contain StructuredContent subclasses (uses AST pre-check)
- ClassRegistryUtils.import_modules_in_folder(
- folder_path=library_dir,
- base_class_names=[StructuredContent.__name__],
- force_include_dirs=[Path(builder_pkg.__file__).parent],
- )
- # Only import files that contain @pipe_func decorated functions (uses AST pre-check)
- FuncRegistryUtils.register_funcs_in_folder(
- folder_path=library_dir,
- force_include_dirs=[Path(builder_pkg.__file__).parent],
+ # Auto-discover and register all StructuredContent classes from sys.modules
+ num_registered = ClassRegistryUtils.auto_register_all_subclasses(
+ base_class=StructuredContent,
)
+ log.verbose(f"Auto-registered {num_registered} StructuredContent classes from loaded modules")
- # Auto-discover and register all StructuredContent classes from sys.modules
- num_registered = ClassRegistryUtils.auto_register_all_subclasses(
- base_class=StructuredContent,
- )
- log.verbose(f"Auto-registered {num_registered} StructuredContent classes from loaded modules")
-
- # Load MTHDS files into the specific library
- log.verbose(f"Loading MTHDS files from: {[str(p) for p in valid_mthds_paths]}")
- return self._load_mthds_files_into_library(library_id=library_id, valid_mthds_paths=valid_mthds_paths)
+ # Load MTHDS files into the specific library
+ log.verbose(f"Loading MTHDS files from: {[str(p) for p in valid_mthds_paths]}")
+ return self._load_mthds_files_into_library(library_id=library_id, valid_mthds_paths=valid_mthds_paths)
@override
def load_from_crate(self, *, library_id: str, crate: LibraryCrate) -> list[PipeAbstract]:
@@ -355,80 +359,84 @@ class LibraryManager(LibraryManagerAbstract):
Returns:
List of all pipes that were loaded, or empty list if already loaded
"""
- # Fingerprint idempotency: skip if this crate was already loaded into this library
- fingerprint = crate.fingerprint
- loaded_set = self._loaded_fingerprints.setdefault(library_id, set())
- if fingerprint in loaded_set:
- log.verbose(f"Crate with fingerprint {fingerprint[:12]}... already loaded into '{library_id}', skipping")
- return []
- library = self.get_library(library_id=library_id)
-
- # Load domains from crate metadata
- all_domains: list[Domain] = []
- for domain_blueprint in crate.domains.values():
- domain = DomainFactory.make_from_blueprint(blueprint=domain_blueprint)
- all_domains.append(domain)
- library.domain_library.add_domains(domains=all_domains)
-
- # Load concepts in topological order
- all_concepts = self._load_concepts_from_crate(crate.concepts)
- library.concept_library.add_concepts(concepts=all_concepts)
-
- # Resolve forward references in dynamically generated structure classes
- self._rebuild_models_with_forward_refs(all_concepts)
-
- # Detect cycles in concept references (A -> B -> A is forbidden)
- self._detect_concept_cycles(all_concepts)
-
- # Precompute domain -> concept local codes mapping (avoids O(N*M) re-parsing per pipe).
- # Sourced from the LIVE library (native concepts + concepts from prior load batches + this
- # batch's concepts, already added above), not just this crate — so a pipe can reference, by
- # bare code, a same-domain concept that a prior batch (e.g. a -L library directory) loaded.
- # This is the pipe-factory counterpart to the loader's cross-batch concept-reference check:
- # without it, that check would pass a bare cross-batch ref only for the factory to reject
- # it. Cross-package aliased entries ('alias->...') are skipped — they resolve through the
- # dependency resolver, not by bare code, and would not parse as a concept ref.
- domain_concept_codes: dict[str, list[str]] = {}
- for concept_ref in library.concept_library.root:
- if QualifiedRef.has_cross_package_prefix(concept_ref):
- continue
- parsed_concept = QualifiedRef.parse_concept_ref(raw=concept_ref)
- if parsed_concept.domain_path is not None:
- domain_concept_codes.setdefault(parsed_concept.domain_path, []).append(parsed_concept.local_code)
+ # Bind the target library as current for the whole load: PipeFactory and the concept
+ # factories resolve concepts and the class registry through the ambient current library,
+ # so a load into a non-current library must not read the caller's binding.
+ with scoped_current_library(library_id=library_id):
+ # Fingerprint idempotency: skip if this crate was already loaded into this library
+ fingerprint = crate.fingerprint
+ loaded_set = self._loaded_fingerprints.setdefault(library_id, set())
+ if fingerprint in loaded_set:
+ log.verbose(f"Crate with fingerprint {fingerprint[:12]}... already loaded into '{library_id}', skipping")
+ return []
+ library = self.get_library(library_id=library_id)
+
+ # Load domains from crate metadata
+ all_domains: list[Domain] = []
+ for domain_blueprint in crate.domains.values():
+ domain = DomainFactory.make_from_blueprint(blueprint=domain_blueprint)
+ all_domains.append(domain)
+ library.domain_library.add_domains(domains=all_domains)
+
+ # Load concepts in topological order
+ all_concepts = self._load_concepts_from_crate(crate.concepts)
+ library.concept_library.add_concepts(concepts=all_concepts)
+
+ # Resolve forward references in dynamically generated structure classes
+ self._rebuild_models_with_forward_refs(all_concepts)
+
+ # Detect cycles in concept references (A -> B -> A is forbidden)
+ self._detect_concept_cycles(all_concepts)
+
+ # Precompute domain -> concept local codes mapping (avoids O(N*M) re-parsing per pipe).
+ # Sourced from the LIVE library (native concepts + concepts from prior load batches + this
+ # batch's concepts, already added above), not just this crate — so a pipe can reference, by
+ # bare code, a same-domain concept that a prior batch (e.g. a -L library directory) loaded.
+ # This is the pipe-factory counterpart to the loader's cross-batch concept-reference check:
+ # without it, that check would pass a bare cross-batch ref only for the factory to reject
+ # it. Cross-package aliased entries ('alias->...') are skipped — they resolve through the
+ # dependency resolver, not by bare code, and would not parse as a concept ref.
+ domain_concept_codes: dict[str, list[str]] = {}
+ for concept_ref in library.concept_library.root:
+ if QualifiedRef.has_cross_package_prefix(concept_ref):
+ continue
+ parsed_concept = QualifiedRef.parse_concept_ref(raw=concept_ref)
+ if parsed_concept.domain_path is not None:
+ domain_concept_codes.setdefault(parsed_concept.domain_path, []).append(parsed_concept.local_code)
- # Load pipes with domain-filtered concept codes
- all_pipes: list[PipeAbstract] = []
- for pipe_ref, pipe_blueprint in crate.pipes.items():
- parsed = QualifiedRef.parse_pipe_ref(raw=pipe_ref)
- if parsed.domain_path is None:
- msg = f"Crate pipe_ref '{pipe_ref}' must be domain-qualified"
- raise PipeLibraryError(msg)
- domain_code = parsed.domain_path
- pipe_code = parsed.local_code
+ # Load pipes with domain-filtered concept codes
+ all_pipes: list[PipeAbstract] = []
+ for pipe_ref, pipe_blueprint in crate.pipes.items():
+ parsed = QualifiedRef.parse_pipe_ref(raw=pipe_ref)
+ if parsed.domain_path is None:
+ msg = f"Crate pipe_ref '{pipe_ref}' must be domain-qualified"
+ raise PipeLibraryError(msg)
+ domain_code = parsed.domain_path
+ pipe_code = parsed.local_code
- concept_codes_for_domain = domain_concept_codes.get(domain_code, [])
+ concept_codes_for_domain = domain_concept_codes.get(domain_code, [])
- pipe = PipeFactory[PipeAbstract].make_from_blueprint(
- domain_code=domain_code,
- pipe_code=pipe_code,
- blueprint=pipe_blueprint,
- concept_codes_from_the_same_domain=concept_codes_for_domain,
- )
- all_pipes.append(pipe)
+ pipe = PipeFactory[PipeAbstract].make_from_blueprint(
+ domain_code=domain_code,
+ pipe_code=pipe_code,
+ blueprint=pipe_blueprint,
+ concept_codes_from_the_same_domain=concept_codes_for_domain,
+ )
+ all_pipes.append(pipe)
- # Track source file for this pipe (used by get_pipe_source)
- source = crate.source_map.get(pipe_ref)
- if source:
- self._pipe_source_maps.setdefault(library_id, {})[pipe_ref] = source
+ # Track source file for this pipe (used by get_pipe_source)
+ source = crate.source_map.get(pipe_ref)
+ if source:
+ self._pipe_source_maps.setdefault(library_id, {})[pipe_ref] = source
- library.pipe_library.add_pipes(pipes=all_pipes)
+ library.pipe_library.add_pipes(pipes=all_pipes)
- library.validate_library()
+ library.validate_library()
- # Only cache fingerprint after the entire load succeeds — if loading fails
- # with an exception, subsequent retries must not be skipped.
- loaded_set.add(fingerprint)
- return all_pipes
+ # Only cache fingerprint after the entire load succeeds — if loading fails
+ # with an exception, subsequent retries must not be skipped.
+ loaded_set.add(fingerprint)
+ return all_pipes
@override
def load_from_blueprints(self, *, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> list[PipeAbstract]:
@@ -444,32 +452,36 @@ class LibraryManager(LibraryManagerAbstract):
Returns:
List of all pipes that were loaded
"""
- # Accumulate blueprints for later crate construction via get_crate()
- self._blueprints.setdefault(library_id, []).extend(blueprints)
- self._crate_cache.pop(library_id, None)
-
- # Discover and load address-based cross-package dependencies before loading pipes
- self._load_address_based_dependencies(
- library_id=library_id,
- blueprints=blueprints,
- )
+ # Bind the target library as current for the whole load: the factories beneath
+ # (concept resolution, class-registry access) resolve through the ambient current
+ # library, so a load into a non-current library must not read the caller's binding.
+ with scoped_current_library(library_id=library_id):
+ # Accumulate blueprints for later crate construction via get_crate()
+ self._blueprints.setdefault(library_id, []).extend(blueprints)
+ self._crate_cache.pop(library_id, None)
- # Build the crate (merges, qualifies, detects duplicates)
- crate = LibraryCrateFactory.make_from_blueprints(blueprints=blueprints)
+ # Discover and load address-based cross-package dependencies before loading pipes
+ self._load_address_based_dependencies(
+ library_id=library_id,
+ blueprints=blueprints,
+ )
- # Validate same-domain concept references against the live library: the concepts declared
- # in this batch plus those already loaded by prior batches (e.g. via a -L library
- # directory). This is done in the loader, not the crate factory, so a bare concept ref can
- # resolve across separate load batches into the same library. The batch's own concepts are
- # not in the library yet (load_from_crate adds them below), so they come from `blueprints`.
- library = self.get_library(library_id=library_id)
- validate_concept_references_in_blueprints(
- blueprints=blueprints,
- already_loaded_concept_refs=set(library.concept_library.root.keys()),
- )
+ # Build the crate (merges, qualifies, detects duplicates)
+ crate = LibraryCrateFactory.make_from_blueprints(blueprints=blueprints)
+
+ # Validate same-domain concept references against the live library: the concepts declared
+ # in this batch plus those already loaded by prior batches (e.g. via a -L library
+ # directory). This is done in the loader, not the crate factory, so a bare concept ref can
+ # resolve across separate load batches into the same library. The batch's own concepts are
+ # not in the library yet (load_from_crate adds them below), so they come from `blueprints`.
+ library = self.get_library(library_id=library_id)
+ validate_concept_references_in_blueprints(
+ blueprints=blueprints,
+ already_loaded_concept_refs=set(library.concept_library.root.keys()),
+ )
- # Load from crate (domains, concepts, pipes, validation)
- return self.load_from_crate(library_id=library_id, crate=crate)
+ # Load from crate (domains, concepts, pipes, validation)
+ return self.load_from_crate(library_id=library_id, crate=crate)
def _load_concepts_from_blueprints(
self,
@@ -1113,8 +1125,11 @@ class LibraryManager(LibraryManagerAbstract):
@override
def _remove_from_blueprint(self, *, library_id: str, blueprint: PipelexBundleBlueprint) -> None:
- self._remove_pipes_from_blueprint(blueprint=blueprint)
- self._remove_concepts_from_blueprint(blueprint=blueprint)
+ # Bind the target library as current: the helpers below resolve through the ambient
+ # current library, so a removal from a non-current library must honor library_id.
+ with scoped_current_library(library_id=library_id):
+ self._remove_pipes_from_blueprint(blueprint=blueprint)
+ self._remove_concepts_from_blueprint(blueprint=blueprint)
@override
def _remove_from_blueprints(self, *, library_id: str, blueprints: list[PipelexBundleBlueprint]) -> None:
diff --git a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
index 05eb9e816..a639e3c51 100644
--- a/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
+++ b/pipelex/pipe_operators/extract/pipe_extract_blueprint.py
@@ -3,7 +3,11 @@ from typing import Literal
from typing_extensions import override
from pipelex.cogt.extract.extract_setting import ExtractModelChoice
+from pipelex.core.concepts.native.concept_native import NativeConceptCode
+from pipelex.core.domains.domain import SpecialDomain
from pipelex.core.pipes.pipe_blueprint import PipeBlueprint
+from pipelex.core.pipes.variable_multiplicity import parse_concept_with_multiplicity
+from pipelex.core.qualified_ref import QualifiedRef
class PipeExtractBlueprint(PipeBlueprint):
@@ -35,6 +39,20 @@ class PipeExtractBlueprint(PipeBlueprint):
@override
def validate_output(self):
- if self.output != "Page[]":
- msg = f"PipeExtract output should be a 'native.Page[]' concept, but is {self.output}"
+ # PipeExtract mechanically produces native Pages: the declared output is exactly a
+ # variable-length list of `native.Page` — no refinement of it, since an output declaration is
+ # a promise to downstream consumers and the operator cannot make its pages any more specific.
+ # This compares the *parsed* ref and multiplicity rather than the authoring spelling "Page[]",
+ # so it still holds once crate normalization qualifies the ref to "native.Page[]". The
+ # authoritative check on the resolved concept lives in PipeExtract.validate_output_with_library.
+ output_parse_result = parse_concept_with_multiplicity(self.output)
+ output_ref = QualifiedRef.parse(output_parse_result.concept_ref_or_code)
+ is_native_page = output_ref.local_code == NativeConceptCode.PAGE and (
+ output_ref.domain_path is None or SpecialDomain.is_native(domain_code=output_ref.domain_path)
+ )
+ if not is_native_page or output_parse_result.multiplicity is not True:
+ msg = (
+ f"PipeExtract output must be '{NativeConceptCode.PAGE.as_output_multiple_indeterminate}' — "
+ f"a variable-length list of native Pages — but is '{self.output}'."
+ )
raise ValueError(msg)
diff --git a/pipelex/pipeline/validate_bundle.py b/pipelex/pipeline/validate_bundle.py
index b5e56c475..751fb8ab2 100644
--- a/pipelex/pipeline/validate_bundle.py
+++ b/pipelex/pipeline/validate_bundle.py
@@ -226,7 +226,7 @@ async def validate_bundle(
mthds_file_path: Path | None = None,
*,
mthds_contents: list[str] | None = None,
- mthds_sources: list[str] | None = None,
+ mthds_sources: Sequence[str | None] | None = None,
library_dirs: Sequence[Path] | None = None,
allow_signatures: bool = False,
dry_run_pipe_codes: list[str] | None = None,
diff --git a/tests/conftest.py b/tests/conftest.py
index c143d3cf9..a31fbaf84 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -7,7 +7,7 @@ import shortuuid
from pytest_mock import MockerFixture
from pipelex import log
-from pipelex.hub import get_library_manager, set_current_library
+from pipelex.hub import clear_current_library, get_current_library_id_or_none, get_library_manager, set_current_library
from pipelex.pipelex import Pipelex
from pipelex.pipeline.job_metadata import JobMetadata
from pipelex.system.pipelex_service.pipelex_service_config import (
@@ -125,10 +125,13 @@ def reset_pipelex_config_fixture():
@pytest.fixture(scope="class")
def load_test_library() -> Generator[Callable[[list[Path]], None], None, None]:
library_id = None
+ prev_library_id = None
def _load(library_dirs: list[Path]) -> None:
- nonlocal library_id
+ nonlocal library_id, prev_library_id
library_manager = get_library_manager()
+ if library_id is None:
+ prev_library_id = get_current_library_id_or_none()
library_id, _ = library_manager.open_library()
set_current_library(library_id=library_id)
@@ -144,16 +147,26 @@ def load_test_library() -> Generator[Callable[[list[Path]], None], None, None]:
if library_id is not None:
library_manager = get_library_manager()
library_manager.teardown(library_id=library_id)
+ # Restore the binding that existed before _load ran (same pattern as scoped_current_library),
+ # so an outer scope's current library survives. prev was captured before open_library minted
+ # the new id, so this can never resurrect the torn-down library.
+ if prev_library_id is not None:
+ set_current_library(library_id=prev_library_id)
+ else:
+ clear_current_library()
log.verbose(f"Torn down library: {library_id}")
@pytest.fixture(scope="class")
def load_empty_library() -> Generator[Callable[[], str], None, None]:
library_id = None
+ prev_library_id = None
def _load() -> str:
- nonlocal library_id
+ nonlocal library_id, prev_library_id
library_manager = get_library_manager()
+ if library_id is None:
+ prev_library_id = get_current_library_id_or_none()
library_id, _ = library_manager.open_library()
set_current_library(library_id=library_id)
@@ -165,6 +178,13 @@ def load_empty_library() -> Generator[Callable[[], str], None, None]:
if library_id is not None:
library_manager = get_library_manager()
library_manager.teardown(library_id=library_id)
+ # Restore the binding that existed before _load ran (same pattern as scoped_current_library),
+ # so an outer scope's current library survives. prev was captured before open_library minted
+ # the new id, so this can never resurrect the torn-down library.
+ if prev_library_id is not None:
+ set_current_library(library_id=prev_library_id)
+ else:
+ clear_current_library()
log.verbose(f"Torn down library: {library_id}")
diff --git a/tests/integration/pipelex/libraries/test_crate_normalization_round_trip.py b/tests/integration/pipelex/libraries/test_crate_normalization_round_trip.py
index 33c735fa2..d8c2ce796 100644
--- a/tests/integration/pipelex/libraries/test_crate_normalization_round_trip.py
+++ b/tests/integration/pipelex/libraries/test_crate_normalization_round_trip.py
@@ -30,6 +30,17 @@ model = "$quick-reasoning"
prompt = "Compute a score from $data"
"""
+INTAKE_MTHDS = """\
+domain = "intake"
+description = "Document intake domain"
+
+[pipe.extract_pages]
+type = "PipeExtract"
+description = "Extract the pages of a document"
+inputs = { doc = "Document" }
+output = "Page[]"
+"""
+
class TestCrateNormalizationRoundTrip:
"""Integration: a normalized crate (natives materialized) loads back into a live library."""
@@ -65,3 +76,33 @@ class TestCrateNormalizationRoundTrip:
assert "report.make_score" in library.pipe_library.root
# Natives remain singly-registered (pre-registered natives, not the crate's copies).
assert "native.Text" in library.concept_library.root
+
+ def test_extract_pipe_survives_normalization(self, load_empty_library: Callable[[], str]):
+ """A PipeExtract declares `output = "Page[]"`, which normalization qualifies to
+ `native.Page[]` before rebuilding the blueprints — which re-runs PipeExtractBlueprint's own
+ output validator on the normalized spelling. A validator that only accepted the authoring
+ spelling rejected its own normalized output, raising a raw pydantic ValidationError out of
+ normalize_crate and 500-ing every static-core API route.
+ """
+ library_manager = get_library_manager()
+
+ with tempfile.TemporaryDirectory() as tmp_dir:
+ intake_path = Path(tmp_dir) / "intake.mthds"
+ intake_path.write_text(INTAKE_MTHDS, encoding="utf-8")
+ blueprints = [PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=intake_path)]
+
+ crate = LibraryCrateFactory.make_from_blueprints(blueprints=blueprints)
+ normalized = normalize_crate(crate, mthds_version=MTHDS_TEST_VERSION)
+
+ assert normalized.pipes["intake.extract_pages"].output == "native.Page[]"
+
+ # Normalization is idempotent: re-normalizing re-validates the already-normalized
+ # blueprints, which is exactly the round trip the authoring-only validator broke.
+ renormalized = normalize_crate(normalized, mthds_version=MTHDS_TEST_VERSION)
+ assert renormalized.pipes["intake.extract_pages"].output == "native.Page[]"
+ assert renormalized.fingerprint == normalized.fingerprint
+
+ library_id = load_empty_library()
+ library_manager.load_from_crate(library_id=library_id, crate=normalized)
+ library = library_manager.get_library(library_id=library_id)
+ assert "intake.extract_pages" in library.pipe_library.root
diff --git a/tests/integration/pipelex/libraries/test_load_from_crate.py b/tests/integration/pipelex/libraries/test_load_from_crate.py
index 3017beaf6..e90a238eb 100644
--- a/tests/integration/pipelex/libraries/test_load_from_crate.py
+++ b/tests/integration/pipelex/libraries/test_load_from_crate.py
@@ -3,7 +3,7 @@ from collections.abc import Callable
from pathlib import Path
from pipelex.core.interpreter.interpreter import PipelexInterpreter
-from pipelex.hub import clear_current_library, get_library_manager, set_current_library
+from pipelex.hub import clear_current_library, get_current_library, get_library_manager, set_current_library
from pipelex.libraries.library_crate_factory import LibraryCrateFactory
SCORING_MTHDS = """\
@@ -130,3 +130,36 @@ class TestLoadFromCrate:
clear_current_library()
library_manager.teardown(library_id=first_library_id)
library_manager.teardown(library_id=second_library_id)
+
+ def test_load_into_non_current_library(self, tmp_path: Path) -> None:
+ """Loading must target the given library_id even when a different library is current.
+
+ States the loader invariant directly: once load_from_blueprints is handed a library_id,
+ everything beneath it resolves against that library — the ambient current-library binding
+ is neither consulted for resolution nor clobbered by the load.
+ """
+ bundle_path = tmp_path / "scoring.mthds"
+ bundle_path.write_text(SCORING_MTHDS, encoding="utf-8")
+ blueprint = PipelexInterpreter.make_pipelex_bundle_blueprint(bundle_path=bundle_path)
+ library_manager = get_library_manager()
+ current_library_id, _ = library_manager.open_library()
+ target_library_id, _ = library_manager.open_library()
+
+ try:
+ set_current_library(library_id=current_library_id)
+ library_manager.load_from_blueprints(library_id=target_library_id, blueprints=[blueprint])
+
+ # The load went into the target library, not the ambient current one
+ target_library = library_manager.get_library(library_id=target_library_id)
+ assert target_library.pipe_library.get_optional_pipe(pipe_code="scoring.compute_score") is not None
+ assert target_library.concept_library.is_concept_exists(concept_ref="scoring.ScoreResult")
+ current_library = library_manager.get_library(library_id=current_library_id)
+ assert current_library.pipe_library.get_optional_pipe(pipe_code="scoring.compute_score") is None
+ assert not current_library.concept_library.is_concept_exists(concept_ref="scoring.ScoreResult")
+
+ # The ambient binding survived the load untouched
+ assert get_current_library() == current_library_id
+ finally:
+ clear_current_library()
+ library_manager.teardown(library_id=current_library_id)
+ library_manager.teardown(library_id=target_library_id)
diff --git a/tests/unit/pipelex/codegen/test_python_pydantic_emitter.py b/tests/unit/pipelex/codegen/test_python_pydantic_emitter.py
index 385e31c15..bd6190b75 100644
--- a/tests/unit/pipelex/codegen/test_python_pydantic_emitter.py
+++ b/tests/unit/pipelex/codegen/test_python_pydantic_emitter.py
@@ -24,6 +24,10 @@ class TestPythonPydanticEmitter:
assert "from typing import Optional" not in content
assert "from typing import List" not in content
+ def test_class_docstrings_are_triple_quoted(self, pipeline_crate: LibraryCrate):
+ content = emit_python_pydantic(resolve_concepts_from_crate(pipeline_crate))[0].content
+ assert '"""A report with a score and a label"""' in content
+
def test_generated_module_compiles_and_validates(self, pipeline_crate: LibraryCrate, tmp_path: Path):
content = emit_python_pydantic(resolve_concepts_from_crate(pipeline_crate))[0].content
module = load_generated_module(content, tmp_path=tmp_path, name="gen_models")
diff --git a/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_blueprint.py b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_blueprint.py
index 61f7b804d..fef303112 100644
--- a/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_blueprint.py
+++ b/tests/unit/pipelex/pipe_operators/pipe_extract/test_pipe_extract_blueprint.py
@@ -2,40 +2,69 @@ import pytest
from pydantic import ValidationError
from pipelex.pipe_operators.extract.pipe_extract_blueprint import PipeExtractBlueprint
+from tests.unit.pipelex.pipe_operators.pipe_extract.data import PipeExtractInputTestCases
class TestPipeExtractBlueprint:
- def test_force_output_correct(self):
- blueprint = PipeExtractBlueprint(
- description="lorem ipsum",
- inputs={"document": "Document"},
- output="Page[]",
- )
- assert blueprint.output == "Page[]"
+ @pytest.mark.parametrize(("test_id", "blueprint"), PipeExtractInputTestCases.VALID_CASES)
+ def test_valid_blueprints(self, test_id: str, blueprint: PipeExtractBlueprint):
+ assert blueprint.nb_inputs == 1, f"Expected exactly one input for {test_id}"
+ assert blueprint.output == "native.Page[]", f"Unexpected output for {test_id}"
- blueprint = PipeExtractBlueprint(
- description="lorem ipsum",
- inputs={"image": "Image"},
- output="Page[]",
- )
- assert blueprint.output == "Page[]"
-
- def test_validate_inputs_correct(self):
+ @pytest.mark.parametrize(
+ "output",
+ [
+ "Page[]", # authoring spelling
+ "native.Page[]", # normalized spelling, as rewritten by crate normalization
+ ],
+ )
+ def test_validate_output_accepts_both_spellings(self, output: str):
+ """The validator must accept the normalized ref as well as the authored one: crate
+ normalization qualifies `Page[]` to `native.Page[]` and then rebuilds the blueprints, which
+ re-runs this validator on its own output.
+ """
blueprint = PipeExtractBlueprint(
description="lorem ipsum",
inputs={"document": "Document"},
- output="Page[]",
+ output=output,
)
- assert blueprint.nb_inputs == 1
- assert blueprint.input_names == ["document"]
+ assert blueprint.output == output
+ @pytest.mark.parametrize(
+ "output",
+ [
+ "Page", # missing multiplicity: extract always emits a list
+ "native.Page",
+ "Page[3]", # fixed count: page count is data-dependent
+ "Text[]", # wrong concept
+ "native.Text[]",
+ "intake.Page[]", # a domain concept named Page is not the native Page
+ ],
+ )
+ def test_validate_output_rejects_anything_but_a_native_page_list(self, output: str):
+ with pytest.raises(ValidationError) as exc_info:
+ PipeExtractBlueprint(
+ description="lorem ipsum",
+ inputs={"document": "Document"},
+ output=output,
+ )
+ assert "PipeExtract output must be 'Page[]'" in str(exc_info.value)
+
+ @pytest.mark.parametrize(
+ "inputs",
+ [
+ {"document": "Document"},
+ {"image": "Image"},
+ ],
+ )
+ def test_validate_inputs_correct(self, inputs: dict[str, str]):
blueprint = PipeExtractBlueprint(
description="lorem ipsum",
- inputs={"image": "Image"},
+ inputs=inputs,
output="Page[]",
)
assert blueprint.nb_inputs == 1
- assert blueprint.input_names == ["image"]
+ assert blueprint.input_names == list(inputs)
def test_validate_inputs_incorrect(self):
with pytest.raises(ValidationError) as exc_info:
@@ -53,19 +82,3 @@ class TestPipeExtractBlueprint:
output="Page[]",
)
assert "Too many inputs provided for PipeExtract" in str(exc_info.value)
-
- def test_validate_output_correct(self):
- blueprint = PipeExtractBlueprint(
- description="lorem ipsum",
- inputs={"document": "Document"},
- output="Page[]",
- )
- assert blueprint.output == "Page[]"
-
- def test_validate_output_incorrect(self):
- blueprint = PipeExtractBlueprint(
- description="lorem ipsum",
- inputs={"document": "Document"},
- output="Page[]",
- )
- assert blueprint.output == "Page[]"