Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.
feat(plugins): storage & secrets provider plugin seams (#1024)
Louis Choquel2a7cd2f96418
diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.tomlindex 25d68f91d..a508997f6 100644--- a/.pipelex/pipelex.toml+++ b/.pipelex/pipelex.toml@@ -119,6 +119,15 @@ bucket_name = ""project_id = ""signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs+####################################################################################################+# Secrets Config+####################################################################################################++[pipelex.secrets_config]+# Secrets provider method: "env" (built-in, reads secrets from environment variables).+# An external pipelex-secrets-<backend> plugin can add its own method (e.g. "vault").+method = "env"+##################################################################################################### Scan Config####################################################################################################diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f75e88e8f..f7ead2a92 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,13 +4,20 @@### Added+- **Storage provider is now a plugin seam:** The storage backend (`local` / `in_memory` / `s3` / `gcp`) is selected at boot from a `StorageProviderRegistry` keyed by the open `storage_config.method` token, populated by the always-on built-in `StoragePlugin`. A third party can now ship a `pipelex-storage-<backend>` package that advertises itself under the `pipelex.plugins` entry-point group and is selected via `storage_config.method` — the same discovery/denylist machinery the inference and orchestrator seams use. `storage_config.method` is now an open `str`: an unknown method fails loudly at boot with `UnknownStorageMethodError` (listing the registered methods), not at config parse. See the new `docs/under-the-hood/storage-provider-plugins.md`.+- **Secrets provider is now a plugin seam:** The secrets backend (`env`) is selected at boot from a `SecretsProviderRegistry` keyed by the new open `secrets_config.method` token (default `"env"`), populated by the always-on built-in `SecretsPlugin`. A third party can now ship a `pipelex-secrets-<backend>` package (Vault, AWS Secrets Manager, …) that advertises itself under the `pipelex.plugins` entry-point group and is selected via `secrets_config.method` — the same mechanism the storage seam uses. Out-of-the-box behavior is unchanged: `env` stays the default. An unknown method fails loudly at boot with `UnknownSecretsMethodError` (listing the registered methods), not at config parse. See the new `docs/under-the-hood/secrets-provider-plugins.md`.- **TOML pipeline inputs:** The `--inputs` file passed to `pipelex run` and `pipelex-agent run` (pipe/bundle/method) can now be TOML in addition to JSON, discriminated by file extension — a `.toml` suffix is parsed as TOML, every other value stays JSON. Both formats produce the same input dictionary; TOML's multi-line strings make text-heavy inputs much easier to author. `run bundle <dir>` auto-detects `inputs.toml` alongside `inputs.json` when `--inputs` is omitted, erroring if both exist. Inline JSON (`{...}`) and agent-CLI stdin inputs stay JSON-only. A bare TOML datetime/date/time literal is rejected with an explicit error (quote it as a string) — native datetime concept support is a separate track.- **`--format toml` for inputs-template generation:** `pipelex build inputs` and `pipelex-agent inputs` (pipe/bundle/method) accept `--format json|toml` (default `json`) to emit the generated inputs template as TOML. On the main CLI, `toml` defaults the output filename to `inputs.toml`; on the agent CLI, `toml` prints the raw TOML template to stdout (the no-inputs case prints a TOML comment line that loads back as an empty dict).### Changed+- **Plugin API version bumped to 3 (Breaking):** `PLUGIN_API_VERSION` is now `3` (was `2`) to add the `add_storage_provider` and `add_secrets_provider` registrar menu methods. Plugin discovery version-checks with strict equality, so every external plugin must re-declare `targets_api = 3`. `pipelex-mistralai-workflows` registers no storage or secrets provider and imports none of the removed symbols, so the `targets_api` bump is its only change. `pipelex-temporal` needs the `targets_api` bump **and** a code migration: its payload-codec factory imports the now-removed `make_storage_provider_from_config` (see Removed) and must switch to resolving the provider via the storage-provider registry (`get_storage_provider_registry().get_required(method=...)`).- **`PipeSignature` is not a pipe type (Breaking):** A signature is now declared by **omitting** `type` — a `[pipe.x]` section with no `type` and nothing but the contract (`description`, `output`, optional `inputs`, optional `signature_for`) *is* a `PipeSignature`. Writing `type = "PipeSignature"` explicitly is rejected with a migration error ("`PipeSignature` is no longer a pipe type — delete the `type` line"), and a typeless section that declares any non-contract field is a hard error naming the field. The `pipelex-agent pipe` authoring command mirrors this: a typeless JSON spec (no `type`) renders a `[pipe.x]` section with no type line, and `--type PipeSignature` is rejected. **MTHDS JSON Schema shape change:** the signature arm no longer carries a `type` property (an explicit tag now fails schema validation), so the downstream schema copies (`mthds`, `vscode-pipelex`, `mthds-ui`) must be re-synced on the next release via the `mthds-schema-sync` skill.+### Removed++- **`make_storage_provider_from_config` (Breaking):** The module-level storage-provider factory (`pipelex/tools/storage/storage_provider_factory.py`) is removed — storage is now resolved through the config-selected `StorageProviderRegistry` at boot. Downstream code that imported this helper (notably `pipelex-temporal`'s payload-codec factory) must select through the registry instead.+## [v0.37.0] - 2026-07-04### Highlightsdiff --git a/TODOS.md b/TODOS.mdindex 3438ef2fc..9d20e33e2 100644--- a/TODOS.md+++ b/TODOS.md@@ -1,72 +1,234 @@-# TOML pipeline inputs — feature/Inputs+# Master plan — Provider plugins: storage & secrets-Status: **CHECKPOINT 3 CLEARED (2026-07-06)** — all four phases done and committed. Phases 1+2 (`9041c6986`), Phase 3 (`1fdfd26d0`), Phase 4 docs + changelog (`5ebd9bef2`), with full `make agent-check` + `make agent-test` green. Feature is ready for PR to dev — nothing pushed. This tracker replaces the retired hello-plugin tracker (that track completed and merged to dev via PR #1015).+Status: **TRACK COMPLETE. All 5 phases DONE + all 5 Checkpoints CLEARED.** Both provider verticals (storage + secrets) shipped as config-selected plugin seams; Phase 5 (release gating / cross-repo bookkeeping) done and Checkpoint 5 (final whole-branch clean-room review) cleared with its triage applied. Branch: `feature/More-plugins-2` (worktree `/Users/lchoquel/repos/Pipelex/_plugins`). **Nothing pushed** (no upstream) — the PR to the intended base is Louis's call. This file is the **conductor**; the granular per-seam procedures live in linked docs and should not be duplicated here.-## Goal+Commits on the branch (newest substantive first; run `git log` for the exact tip — later `docs(plugins)` bookkeeping commits may sit on top):-Accept pipeline inputs as TOML in addition to JSON, discriminated by file extension, on both CLI surfaces (`pipelex run pipe|bundle|method` and `pipelex-agent run ...`). Also let the input-template generators (`pipelex build inputs ...`, `pipelex-agent inputs ...`) emit TOML via `--format toml`. TOML's multi-line strings make text-heavy inputs much more pleasant to author than JSON.+- `0bbc39474` — fix: Checkpoint-5 whole-branch review triage (**Phase 5 / Checkpoint 5 — the review-triage commit**)+- `65603e912` — test: secrets-provider seam tests + docs (**Phase 4 — the review target**)+- `fdf3e114b` — fix: Phase-3 review triage (kit-config sync + stale boot comment)+- `8a0b32668` — feat: secrets provider → config-selected plugin seam (**Phase 3 code — the review target**)+- `3940eb7a2` — docs: Phase 2 checkpoint bookkeeping — storage vertical done+- `37a3b72f1` — fix: Phase-2 review triage (doc tense + stronger gcp secret-wiring assert)+- `f08193e81` — test: storage-provider seam tests + docs (**Phase 2**)+- `977c73811` — docs: this plan + Phase 1 checkpoint bookkeeping+- `58961848a` — fix: contract.py v3 comment (Checkpoint-1 review triage)+- `8268ff08f` — feat: storage provider → config-selected plugin seam (**Phase 1 code**)+- `04434f785` — branch base (release v0.37.0 merge). Whole-branch diff for the final Checkpoint 5 = `git diff 04434f785..HEAD`.-## Current state (verified against this tree)+**Cold-start (resume here):** **The whole track is COMPLETE — nothing left to build.** Both provider verticals shipped (storage Phases 1–2, secrets Phases 3–4) and Phase 5 (release gating) + Checkpoint 5 (final whole-branch clean-room review + triage) are done. All gates green on the final tip: `agent-check`, `tb`, `check-config-sync`, full `agent-test`. The **only remaining action is Louis's**: open the PR `feature/More-plugins-2` → intended base, and (when the pipelex version carrying this lands) execute the cross-repo obligations recorded in the CHANGELOG — `pipelex-mistralai-workflows` bumps `targets_api` to 3 (bump-only); `pipelex-temporal` bumps `targets_api` to 3 **and** migrates its payload-codec factory off the removed `make_storage_provider_from_config` to `get_storage_provider_registry().get_required(...)`. Deferred design follow-ups (D3 external config passthrough incl. the external-method `uri_format` gap, first real external provider, cookbook example) are captured in `wip/plugins/`. `PLUGIN_API_VERSION` is 3.-- Both surfaces take `--inputs` and discriminate by content prefix, not extension: `startswith("{")` → inline JSON, else → JSON file via `load_json_dict_from_path` + `resolve_inputs_paths`. Main CLI: `pipelex/cli/commands/run/_run_core.py:140-164`. Agent CLI: `pipelex/cli/agent_cli/commands/run/stdin_resolver.py` (`parse_cli_inputs` → `_parse_inputs_arg`).-- The agent CLI additionally reads JSON from stdin (flat dict or `working_memory` envelope) — `stdin_resolver.py:_read_stdin_inputs`.-- `run bundle <dir>` auto-detects `inputs.json` (`DEFAULT_INPUTS_FILE_NAME` in `pipelex/builder/conventions.py`) — main `cli/commands/run/bundle_cmd.py:149`, agent `cli/agent_cli/commands/run/bundle_cmd.py:118`.-- TOML loading machinery already exists and is unused on this path: `pipelex/tools/misc/toml_utils.py` (`load_toml_from_path` via `tomli`, raising `TomlError`; `tomlkit` for writing).-- Template generation: main CLI `build inputs` uses `render_inputs` (`pipelex/core/pipes/inputs/input_renderer.py`) → `InputStuffSpecs.render_inputs` (`input_stuff_specs.py:153`) which builds a dict then `json.dumps` — the dict is trivially extractable for a TOML serializer. Agent CLI `inputs` uses `build_inputs_for_pipe` (`pipelex/builder/operations/inputs_ops.py`) which already returns the dict.-- The API runner receives the already-parsed dict, so this feature is purely client-side parsing — no wire or server change.+> ⚠️ **Phase-4 heads-up (learned in Phase 2):** the s3/gcp "MissingDependencyError when *selected*" phrasing in the plan is imprecise — the SDK guard is deferred to *use*, not *selection*. Expect the secrets detail doc to carry the same imprecision for SDK-backed secrets providers (Vault/AWS) and treat it the same way (pin *import-light registration*, not a select-time raise). See "Phase 2 — as-built".-## Decisions (settled with Louis, 2026-07-06)+**Environment notes for the next session (avoid rediscovering these):**-- **D1 — extension discrimination.** `.toml` suffix → TOML loader; every other value (including `.json` and extensionless) keeps today's JSON behavior. No content sniffing.-- **D2 — inline and stdin stay JSON-only.** The `{`-prefix inline path is untouched; inline TOML is ambiguous with file paths. Agent-CLI stdin (flat dict + `working_memory` envelope) stays JSON-only.-- **D3 — auto-detect both, error on ambiguity.** `run bundle <dir>` auto-detects `inputs.toml` alongside `inputs.json`; if both exist, hard error telling the user to pass `--inputs` explicitly.-- **D4 — TOML datetimes rejected for now.** `tomli` parses TOML datetime/date/time into Python objects with no JSON equivalent. Adding `DATETIME` to `NativeConceptCode` is deferred to its own track; until then the loader rejects any datetime-typed value with an explicit not-implemented error.-- **D5 — template generation gets `--format json|toml`** (default `json`) on both surfaces; when `toml` is selected and no explicit output path is given, the default filename becomes `inputs.toml`.-- **D6 — everything lands in pipelex first.** No `docs/specs/` or `conformance/` edits ride this branch: conformance pins the released pipelex, so its tests can't go green until this ships in a release. Spec + conformance + skills + mthds-js sweep is an explicitly deferred follow-up wave (see "Deferred" below).+- **Targeted tests for the storage seam:** `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/unit/pipelex/plugins/ tests/unit/pipelex/tools/storage/ tests/integration/pipelex/tools/storage/` (Phase 2 also needs `tests/integration/pipelex/plugins/` for the external-plugin harness). But Checkpoint 2 requires the **full** `make agent-test`.+- **This machine's global `~/.pipelex/pipelex.toml` sets `storage_config.method = "s3"`**, so a plain `Pipelex.make(needs_inference=False)` boot smoke test yields an `S3StorageProvider`, *not* `local` (base + repo `.pipelex/` both say `local`; the global override wins). To smoke a specific method faithfully, pass `config_overrides={"pipelex": {"storage_config": {"method": "local"}}}` in a fresh interpreter. This is a machine/global-config condition, **not** a bug in the seam — don't chase it.+- **Empirically confirmed** (so Phase 2 boot tests can rely on it): `StorageProviderConfig.method: str = Field(strict=False)` coerces a `StrEnum` input to a plain `str`, accepts an external token (e.g. `"azure"`) at parse, and `StrEnum` registry keys resolve against a plain-str `config.method` in dict lookup.-## Phases+> Replaces the retired cookbook hello-plugin tracker (complete; recoverable in git history).-### Phase 1 — Shared inputs-file loader — DONE+## What & why (cold-start summary)-- [x] New module `pipelex/cli/commands/run/_inputs_file_loader.py`: `load_inputs_dict_from_path(path: Path) -> dict[str, Any]`. Suffix match: `.toml` → `load_toml_from_path`; else → `load_json_dict_from_path`. Also hosts the shared auto-detect probe `find_default_inputs_file(directory)` (D3).-- [x] Datetime guard: recursive walk rejecting `datetime.datetime` / `date` / `time` instances, run on both formats. New `pipelex/cli/commands/run/exceptions.py` with `InputsDatetimeNotSupportedError` and `AmbiguousInputsFilesError` — both carry class-level `error_domain = INPUT` + `user_action` (CHANGE_INPUT), so the agent envelope self-describes and they must NOT get `AGENT_ERROR_HINTS`/`AGENT_ERROR_DOMAINS` entries (enforced by `test_agent_output_drift.py`). Messages include the offending key path (e.g. `record.entries[0].when`) and the quote-as-string workaround.-- [x] `pipelex-dev generate-error-pages` run — new pages `docs/errors/inputs-datetime-not-supported-error.md`, `docs/errors/ambiguous-inputs-files-error.md` (+ index update).-- [x] Unit tests: `tests/unit/pipelex/cli/commands/run/test_inputs_file_loader.py` (loads, extension rule, TomlError, datetime top-level/nested/all four TOML datetime flavors, JsonTypeError, multi-line string) and `test_inputs_file_probe.py` (json-only / toml-only / neither / both-ambiguous).+Promote the two dependency-injection seams that pass all plugin criteria — **storage provider** and **secrets provider** — to formal `pipelex.plugins` entry-point plugins, so third parties can ship `pipelex-storage-<backend>` / `pipelex-secrets-<backend>` packages selected at deploy time. Chosen after a full boot-audit of DI seams; everything else is already a plugin, a boot-orchestrator-owned hub slot, or a population/config point that should stay one.-### Phase 2 — Wire into both run surfaces — DONE+**Read these three before starting** (they hold the detailed steps this master plan sequences):-- [x] Main CLI `_run_core.py`: swapped to the shared loader; new except branches for `TomlError` and `InputsDatetimeNotSupportedError`; also fixed a pre-existing gap — invalid JSON in an inputs *file* (`json.JSONDecodeError`) previously escaped to the generic traceback handler, now gets a friendly message on both surfaces. The `JsonTypeError` wording stays JSON-specific (it can only fire on the JSON branch).-- [x] Agent CLI `stdin_resolver.py::_parse_inputs_arg`: same swap; `agent_error` branches for `TomlError` / `InputsDatetimeNotSupportedError` / `json.JSONDecodeError`. `AGENT_ERROR_HINTS` + `AGENT_ERROR_DOMAINS` got `TomlError` entries (it has no class-level metadata — kept generic since TomlError also fires outside inputs loading); the two new errors self-describe (see Phase 1).-- [x] Auto-detect + ambiguity (D3): `DEFAULT_INPUTS_TOML_FILE_NAME` in `conventions.py`; both `bundle_cmd.py` sites probe via the shared `find_default_inputs_file`. The probe is skipped entirely when `--inputs` is passed, so the ambiguity rule only bites on auto-detect (both surfaces).-- [x] Verified `run method --inputs relative/path.toml` comes free: main-CLI `method_cmd.py` joins `method_dir / inputs_path` preserving the suffix. Also fixed a pre-existing asymmetry (Louis's call): the *agent* CLI `run method` used to treat `--inputs` as cwd-relative; both surfaces now resolve relative `--inputs` file paths against the method dir via the shared `resolve_inputs_arg_against_dir` helper in `_inputs_file_loader.py` (inline JSON and absolute paths pass through).-- [x] Tests: e2e subprocess dry run with `inputs.toml` + relative `url` resolution (`tests/e2e/pipelex/cli/test_toml_inputs_run.py`, staged csv_demo fixture); main-CLI auto-detect/ambiguity/explicit-bypass (`tests/unit/pipelex/cli/commands/run/test_bundle_cmd_auto_inputs.py`); agent-CLI auto-detect + ambiguity envelope + bypass (`tests/unit/pipelex/cli/agent_cli/test_run_bundle_auto_inputs.py`); `.toml` branch + error envelopes in `test_stdin_resolver.py`; TOML load/resolve + error exits in `test_run_core_execution.py`; method-dir `--inputs` resolution helper (`test_inputs_arg_resolution.py`) + agent `run method` relative-path resolution (`test_run_method_inputs_resolution.py`).+- `wip/plugins/README.md` — the track's shared decisions + boot-audit rationale + the new mechanism.+- `wip/plugins/storage-provider-plugin.md` — storage detail (defines the shared mechanism). **Lands first.**+- `wip/plugins/secrets-provider-plugin.md` — secrets detail (reuses the mechanism; **Decision W already RESOLVED = W-A**).-**CHECKPOINT 1 — CLEARED 2026-07-06.** TOML inputs load end-to-end on both surfaces; `make agent-check` fully green (pyright/mypy/ruff/keyword-only); the full CLI unit, integration, and e2e test paths pass, including the new subprocess TOML dry run against the real binary. Work is uncommitted — commit before starting Phase 3.+**The new mechanism (both seams share it):** a *keyed registry + config-selected singleton*. Plugins register N provider factories keyed by an open method token (mirroring `OrchestrationMode`); at boot, core reads one config field (`storage_config.method` / `secrets_config.method`), looks the token up in the registry, and calls the factory to produce the one provider set on the hub. Deliberately **not** a `HubSlot` (those are orchestrator-coupled).-### Phase 3 — Template generation `--format toml` — DONE+## Locked decisions (do not re-litigate)-- [x] `InputStuffSpecs.build_inputs_template() -> dict` exposed; `render_inputs` (both the method and the module-level function in `input_renderer.py`) is now the JSON serializer over it.-- [x] TOML serializer `serialize_inputs_template_to_toml` in `pipelex/core/pipes/inputs/input_renderer.py` (tomlkit), plus `render_inputs_toml(pipe)` and the `InputsTemplateFormat` StrEnum (`json|toml`). Verified the generator never emits `None` placeholders (Optionals are unwrapped to their inner type), but the pinned defensive policy is: None → `""` recursively, so keys stay visible in the template (`test_inputs_template_toml.py`).-- [x] Main CLI `pipelex build inputs pipe|bundle|method`: `--format json|toml` (param `template_format`, default `json`); `toml` defaults the output filename to `inputs.toml` (next to bundle / `results/` / method `results/` — all three default-path sites).-- [x] Agent CLI `pipelex-agent inputs pipe|bundle|method`: same `--format json|toml`; `toml` prints the raw TOML template to stdout via shared `emit_inputs_result`/`emit_no_inputs_result` helpers in agent `_inputs_core.py` (no-inputs case prints a TOML comment line — valid TOML, loads as `{}`). Deviation documented in `pipelex/cli/agent_cli/CLAUDE.md` (output-format section + commands table); `test_inputs_format_unaffected.py` reworked to pin the new contract (InputsTemplateFormat, still no `--error-format`).-- [x] Round-trip tests: unit (`test_inputs_template_roundtrip.py` — TOML template loads through the Phase-1 loader identically to its JSON twin) and e2e (`test_toml_inputs_build.py` — real binary generates `inputs.toml` from the csv_demo fixture, then a dry run consumes it).+- **DX-1 — one API bump.** `PLUGIN_API_VERSION` 2→3, done once on this branch. Both seams land before any release, so external plugins (`pipelex-temporal`, `pipelex-mistralai-workflows`) re-declare `targets_api=3` exactly once. Discovery uses strict-equality, so any menu addition is a breaking bump.+- **DX-2 — unconditional builtins.** `StoragePlugin` and `SecretsPlugin` join `CORE_UNCONDITIONAL_PLUGIN_NAMES` — required infra can't be disabled into a broken boot.+- **DX-3 — external-provider config surface is a follow-up**, not built here (fixed typed sub-configs only in Phase scope).+- **D1 / S1 — open method token.** `method` config fields accept an open `str`, validated at registry lookup (unknown → `UnknownStorageMethodError` / `UnknownSecretsMethodError`), not at parse.+- **W-A (secrets boot ordering) — RESOLVED, refined in Phase 3.** The `# needed for gateway check` comment was stale; the gateway path is secrets-free, so the hardcoded `EnvSecretsProvider()` is gone and secrets is now config-selected from the registry. **Placement deviation (deliberate, see "Phase 3 — as-built"):** the plan's literal W-A said move the `build_registrar` call *above the gateway block* (after line 209 / before old 212). Phase 3 instead placed it *after* the gateway service/terms-check block and *just before the telemetry factory* (secrets' true first consumer). Both satisfy W-A's only hard constraint (secrets resolved before telemetry); the after-the-gate placement additionally preserves precondition-gate-first semantics (an unaccepted-terms / first-run boot fails fast before any discovery work) — a contract two `__new__`-based gateway-terms unit tests encode, which the above-the-gate placement broke.-**CHECKPOINT 2 — CLEARED 2026-07-06.** Template generation done on both surfaces; `make agent-check` fully green; targeted CLI + core + builder unit/integration/e2e suites pass (incl. the generate→dry-run e2e chain and a manual raw-TOML smoke of the agent CLI). Committed at this boundary.+## Sequencing-### Phase 4 — Docs, changelog, final gates — DONE+Storage vertical (Phases 1–2) → Secrets vertical (Phases 3–4) → Release gating (Phase 5). One coherent commit per phase; a mandatory checkpoint after each.-- [x] `docs/tools/cli/run.md`: reworked "Input JSON Format" → "Input File Formats" section (extension rule, JSON + TOML examples with multi-line strings, datetime-limitation warning, auto-detect/ambiguity note); all three `--inputs` option lines and the `run bundle` intro updated to mention JSON-or-TOML + inputs-file auto-detect.-- [x] `docs/building-methods/pipes/provide-inputs.md`: new "Input Files: JSON or TOML" section (extension rule, equivalent JSON/TOML example, datetime-limitation warning, pointer to run CLI reference); `build inputs` tip mentions `--format toml`.-- [x] `docs/tools/cli/build/inputs.md`: `--format json|toml` on all three subcommands, `--output` default-filename behavior (`inputs.json`/`inputs.toml`), `--format toml` example, TOML output + multiplicity examples; frontmatter/intro reworded.-- [x] `docs/tools/cli/agent-cli.md`: `run --inputs` JSON-or-TOML + stdin-stays-JSON/auto-detect note; `inputs --format json|toml` deviation note; Output Contract updated for inputs' dual output. (`pipelex/cli/agent_cli/CLAUDE.md` was already updated in Phase 3.)-- [x] `CHANGELOG.md` under `[Unreleased]`: Added — TOML inputs (extension-discriminated) + `--format toml` template generation; TOML-datetime limitation noted.-- [x] Full `make agent-check` (pyright/mypy/ruff/plxt/keyword-only all green) + `make agent-test` (full suite green).+----**CHECKPOINT 3 (final) — CLEARED 2026-07-06.** All phases done; gates fully green; docs + changelog landed. Phase 4 changes uncommitted at this boundary — commit, then open the PR to dev. Nothing pushed.+## ⛔ Checkpoint protocol (MANDATORY at every checkpoint — do not skip, do not merge phases)-## Deferred — follow-up wave AFTER this ships in a pipelex release (NOT this branch)+At each `CHECKPOINT`, the agent **must stop** and do all three, in order:-- **Spec + conformance sync (D6).** Grep `docs/specs/` (workspace root) for the `--inputs` surface (command-surface-map, protocol spec); add the extension rule + `--format toml` to the spec prose and matching conformance tests with the bidirectional `> Verified by:` ↔ `pytest.mark.spec` links. Gated on the conformance venv's pipelex pin reaching the release that carries this feature (same de-gate pattern as the Optionals track).-- **Skills sweep.** `mthds-plugins` skills (`mthds-run`, `mthds-inputs`) document `inputs.json`; update after release (published plugin, separate repo).-- **mthds-js check.** Verify `mthds-agent` passes `--inputs` through opaquely (no client-side `.json` assumption) — expected no-op.-- **DATETIME native concept.** Add `DATETIME` to `NativeConceptCode`, then replace the Phase-1 datetime guard with real support. Own track.+1. **Verify progress.** Run the gates for the phase and paste real results:+ - Always: `make agent-check` (pyright/ruff/mypy/plxt/keyword-only) + `make tb` (boot sequence — critical whenever config/registry wiring changed).+ - Test phases (2, 4) and the final phase: full `make agent-test`. Intermediate phases may scope to the phase's tests but must still pass `agent-check` + `tb`.+ - If a gate fails: fix before proceeding; a checkpoint is not cleared with a red gate.++2. **Commit, then update this file for cold start.** Commit the phase as one commit. Then edit `TODOS.md` (and/or the linked wip doc) so a brand-new session could resume with no lost context: tick the phase's checkboxes, record the **commit SHA**, any decisions taken or deviations from the linked plan, the current state of the code, and the exact next action. Treat this as a handoff you won't be present to explain.++3. **Fan-out a Sonnet-5 `/code-review` sub-agent — with NO inherited context.** Spawn a **fresh** sub-agent (never a fork) whose entire input is a *pointer to the phase's changes* — the commit SHA / `git diff <base>..HEAD` / the working-tree file list — and nothing else. Do **not** hand it the plan, the rationale, the decisions, or your conclusions; a clean-room review is the point (we want clean solid software, not over-engineering). Spawn template:++ ```+ Agent(+ subagent_type: "general-purpose", # fresh context — NOT "fork"+ model: "sonnet", # Sonnet-5+ description: "code-review phase N",+ prompt: "Run the /code-review skill on the changes in commit <SHA> "+ "(inspect via `git diff <SHA>^..<SHA>` in /Users/lchoquel/repos/Pipelex/_plugins). "+ "Review ONLY those changes. You have no prior context and should assume none. "+ "Focus: correctness bugs, and over-engineering / unnecessary abstraction / "+ "speculative generality. Report findings ranked most-severe first; "+ "if nothing substantive, say so plainly."+ )+ ```++ Then **triage** the findings: apply genuine bug/simplification fixes in a follow-up commit; capture design-tradeoff findings (not silent bugs) as a deferred note under `wip/plugins/` rather than reflexively applying the convenient fix. Record the triage outcome in this file. Only then move to the next phase.++---++## Phase 1 — Storage: mechanism + builtin plugin + boot wiring+*(Detail: `wip/plugins/storage-provider-plugin.md` Phases 1–3. Delivers a working config-selected storage plugin.)*++- [x] `PLUGIN_API_VERSION` 2→3 in `pipelex/plugins/contract.py` (the one batched bump — DX-1).+- [x] Registrar (`pipelex/plugins/registrar.py`): `storage_providers` dict field + `add_storage_provider(*, method, factory)` via the `_add` helper; `StorageProviderFactoryFn` alias.+- [x] New `pipelex/plugins/storage_provider_registry.py`: `StorageProviderRegistry` (mirror `OrchestratorRegistry`: `get_optional`/`get_required`/`has`/`methods`).+- [x] New `UnknownStorageMethodError` in `pipelex/plugins/exceptions.py`.+- [x] Hub (`pipelex/hub.py`): `set_/get_storage_provider_registry` + module accessor.+- [x] New `pipelex/plugins/storage/storage_plugin.py`: `StoragePlugin` (`name="storage"`) + module-level factory closures (`_make_local/_in_memory/_s3/_gcp_storage_provider`) holding the exact bodies moved from the deleted `make_storage_provider_from_config` `match` arms (keep `lazy_validate()`; keep GCP's `get_secrets_provider()` read — legal at the boot apply-point).+- [x] `pipelex/plugins/builtins.py`: register `StoragePlugin()`; add `"storage"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES`.+- [x] Boot (`pipelex.py`): build + `set_storage_provider_registry(...)` alongside the other registries; replace the `make_storage_provider_from_config` block with registry selection (explicit param > `registry.get_required(method=config.method)(config)`).+- [x] Delete `pipelex/tools/storage/storage_provider_factory.py` (grep for other importers first — and re-home them).+- [x] D1: relax `StorageProviderConfig.method` to open `str`.++### Phase 1 — as-built (deviations & decisions, for cold start)++- **Boot ordering (resolved a plan ambiguity).** The old factory ran at `pipelex.py:307-310`, *before* `build_registrar` (~393), but the registry only exists after `build_registrar`. So storage selection **moved down** to sit right beside the other registry constructions (after the 4 `set_*_registry` calls): build `StorageProviderRegistry(plugin_registrar.storage_providers)` → `set_storage_provider_registry` → `if storage_provider is None: select via registry` → `set_storage_provider`. Verified safe: no `get_storage_provider()` consumer runs during `setup()` before that point (all consumers are run-time), and secrets is on the hub (line 306) so the GCP factory's secret read works. This is byte-equivalent to the old factory and forward-compatible with the Phase 3 W-A move (which relocates only the `build_registrar` *call*, not the storage registry construction).+- **`DuplicateStorageProviderError` added** (not in the plan checklist but required by the `_add` `on_duplicate` contract; mirrors `DuplicateOrchestratorError`).+- **D1 blast radius.** `method: StorageMethod` → `method: str = Field(strict=False)` forced a `case _` in `storage_path` + `uri_format` (and the model-validator) because `reportMatchNotExhaustive` fires on open `str`. Empirically confirmed the field coerces a `StrEnum` input to a plain `str` and accepts an external token (e.g. `"azure"`) at parse. Same `case _` added to the two test-side matches (`test_storage_config.py`, `generator_fixtures.py`).+- **Test re-homing.** `test_storage_provider_factory.py` **deleted** (its SUT is gone; Phase 2 adds the registry + parametrized-boot seam tests that supersede it). `generator_fixtures.py` now selects through `get_storage_provider_registry().get_required(...)`. `test_storage_provider_config.py`'s strict-coercion test rewritten to the D1 open-token behavior (`method == "local"` as plain str; `"azure"` accepted at parse).+- **Built-in tokens** registered as the `StorageMethod` enum values (`StrEnum` keys are interchangeable with their plain-str form in dict lookup, so boot's plain-str `config.method` resolves them).++### ⛔ CHECKPOINT 1 — run the protocol above (verify → commit + update → clean-room /code-review → triage)+- [x] Gates green: `make agent-check` (ruff/plxt/pyright/mypy/keyword-only all pass), `make tb` (9 passed), `pipelex plugins list` shows `storage` (builtin, API 3, 4 contributions), boot `method="local"`→`LocalStorageProvider` / `method="in_memory"`→`InMemoryStorageProvider`, unknown token → `UnknownStorageMethodError`. Targeted `tests/unit/pipelex/plugins/ tests/unit+integration/.../tools/storage/` = 1036 passed; re-homed fixture consumers = 4 passed.+- [x] Commit SHA recorded here: `8268ff08f`+- [x] Cold-start state updated in this file.+- [x] Sonnet-5 clean-room `/code-review` fanned out on `8268ff08f`; findings triaged. **Outcome:** review verified correctness (byte-identical factory bodies, safe `StorageMethod→str` D1 relaxation, correct boot ordering, keyword-only compliant, no bare `except`). 3 findings:+ 1. *contract.py v3 comment implied secrets registry already exists* → **FIXED** in follow-up `58961848a` (reworded to "pre-reserved / lands in a follow-up").+ 2. *storage-selection path has no direct automated test after deleting the factory test* → **deferred to Phase 2 as already planned** (Phase 2 checklist adds registry hit/miss + duplicate + parametrized-boot). Manually smoke-verified this phase.+ 3. *`StorageProviderRegistry.get_optional`/`has` have no caller* → **kept (no change).** Confirmed by grep that all sibling read-views (orchestrator/bundle_validator/model_lister) expose the identical shape; `has` is family-wide unused and `get_optional` used by exactly one. It's an established, plan-mandated registry-family convention, not speculative surface — trimming would make storage the odd one out.+ Reviewer also noted a benign boot-ordering nuance: a bad storage config now fails *after* `models_manager.setup()` rather than immediately after secrets (necessary consequence of needing the registrar built first; behavior otherwise byte-equivalent).++---++## Phase 2 — Storage: tests + docs+*(Detail: `wip/plugins/storage-provider-plugin.md` Phases 4–5.)*++- [x] Unit: `StorageProviderRegistry` hit/miss (`UnknownStorageMethodError`) + duplicate-registration fail-loud via `_add`. → `test_storage_provider_registry.py`.+- [x] Built-in factories: each built-in `method` yields the right provider (registry selection); gcp reads its credentials from the hub secrets provider. → `test_storage_plugin.py`. *(Reframed from "parametrized boot" to direct factory-selection — see as-built; the boot-select line is covered end-to-end by the integration test.)*+- [x] Integration: an external test plugin registering a fake `method` (entry-point discovered) is selectable via config, and lands on the hub. → `test_storage_external_plugin.py`.+- [x] Import-light registration pinned: s3/gcp factories register without importing their SDK. → `test_import_light_boot.py` (added `google.cloud.storage` to BLOCKED + assert `storage_providers`).+- [x] New `docs/under-the-hood/storage-provider-plugins.md` (mirrors `orchestrator-plugins.md`); mkdocs nav (both blocks); cross-links from `inference-backend-plugins.md` Related. *(No "the plugin seams are …" enumeration exists to update — grep-verified.)*+- [x] CHANGELOG `[Unreleased]`: "breaking" `PLUGIN_API_VERSION` 2→3; storage is now a plugin seam. *(Added a fresh `[Unreleased]` — none existed, branch base was v0.37.0.)*++### Phase 2 — as-built (deviations & decisions, for cold start)++- **The s3/gcp SDK guard is deferred to *use*, not *selection* (plan text was imprecise).** The plan/detail-doc said "s3/gcp arms raise `MissingDependencyError` only when *selected* with the SDK absent." Not so: `S3StorageProvider.__init__` / `GcpStorageProvider.__init__` only store fields; the `find_spec` guard lives in `_get_session` / `_get_bucket`, invoked inside the I/O methods. So *selecting* (constructing) s3/gcp NEVER raises `MissingDependencyError` even with the SDK absent — the error is deferred to first load/store. The correct invariant Phase 2 pins is therefore **import-light registration** (the s3/gcp factories register without importing their SDK), done by extending `test_import_light_boot.py`. The `MissingDependencyError`-on-use path is already covered by the dedicated `test_{s3,gcp}_storage_provider.py`, so it is not re-tested in the seam tests (avoids redundancy). Both aioboto3 and google-cloud-storage happen to be installed in this dev venv, so an in-process "SDK absent" test would need to mock `find_spec` anyway.+- **Phase-1 latent breakage found + fixed (Phase-1 escapee).** `tests/integration/pipelex/system/test_hub_slot_injection_precedence.py` booted with an empty fake registrar and no explicit `storage_provider`; Phase 1's new selection path (`get_required(method=config.method)`) now rejects that with `UnknownStorageMethodError`. Checkpoint 1 only ran the plugins/storage dirs, not `system/`, so it escaped — the full `make agent-test` at Checkpoint 2 caught it. Fixed by injecting a dependency-free `InMemoryStorageProvider()` in the four boots (this suite tests hub slots, not storage). Grep-verified it was the *only* test with the empty-registrar-boot pattern (`test_plugin_discovery` / `test_import_light_boot` call `build_registrar` directly, never `Pipelex.make`, so they're unaffected).+- **Test layout** (one TestClass per module): registry read-view + duplicate (`test_storage_provider_registry.py`, mirrors `test_bundle_validator_registry.py`); built-in factories + gcp secret-wiring (`test_storage_plugin.py`); integration discovery→selection→hub (`test_storage_external_plugin.py`). Config helpers reused from `tests/unit/pipelex/tools/storage/test_storage_provider_config.py` (`make_{local,in_memory,s3,gcp}_config`) — local construction touches no filesystem (no tmp_path needed).+- **Integration harness** = `mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[SimpleNamespace(name=..., load=lambda: FakePluginClass)])` (discovery instantiates a callable/class), keep the real `BUILTIN_PLUGINS`, boot `Pipelex.make(needs_inference=False, config_overrides={"pipelex": {"storage_config": {"method": "test_mem"}}})`, assert `get_storage_provider()` is the fake + registry `.has(method="test_mem")`. An external method token loads fine (the config validator's `case _: pass`). Per-module autouse `reset_pipelex_config_fixture` (teardown-first) overrides the global module fixture (mirror of the two existing `system/` boot-test modules). This is also the harness template for the Phase 4 secrets external-plugin test.+- **GCP secret-wiring assertion (Phase-2 review triage).** The first cut only asserted the secret was *requested*; the clean-room reviewer flagged it didn't assert the *value* flowed. Strengthened via `mocker.patch(..., wraps=GcpStorageProvider)` + `assert_called_once_with(credentials_file_path=fake_secrets.credentials_path, …)` — the repo-idiomatic way (`reportPrivateUsage`/`SLF001` both hard-block a `provider._credentials_file_path` read, so no private access / suppressions).++### ⛔ CHECKPOINT 2 — CLEARED+- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), **full `make agent-test` (exit 0, "All tests passed")**. Targeted storage-seam + at-risk boot test also green in isolation.+- [x] Commit SHAs: `f08193e81` (Phase 2 code/tests/docs/changelog) + `37a3b72f1` (review triage).+- [x] Cold-start state updated in this file.+- [x] Sonnet clean-room `/code-review` fanned out on `f08193e81`; findings triaged. **Outcome:** 2 findings, **both genuine and applied** in `37a3b72f1`. (1) *High — doc/code contradiction:* the new page reasserted "the same mechanism backs the secrets provider seam" as present tense, but secrets is not yet registry-selected (and commit `58961848a` had just retracted the same claim in `contract.py`) → reworded to planned/future. (2) *Medium — test gap:* the gcp test checked the secret was requested but not that its value reached the provider → added the wraps-ctor `assert_called_once_with` above. Reviewer otherwise verified: no correctness bugs, no over-engineering, mocks faithful to the real contracts, no tautological assertions, no redundancy across the three new files, all doc snippets match source, MkDocs/pytest conventions respected.++---++## Phase 3 — Secrets: mechanism + config + builtin plugin + boot wiring (W-A)+*(Detail: `wip/plugins/secrets-provider-plugin.md` Phases 1–3. W-A is locked — this is a build instruction, not an investigation.)*++- [x] Registrar: `secrets_providers` dict + `add_secrets_provider(*, method, factory)` via `_add`; `SecretsProviderFactoryFn` alias. *(API already at 3 from Phase 1 — no second bump.)*+- [x] New `pipelex/plugins/secrets_provider_registry.py`: `SecretsProviderRegistry`.+- [x] New `UnknownSecretsMethodError` in `pipelex/plugins/exceptions.py`. *(Also added `DuplicateSecretsProviderError` — required by the `_add` `on_duplicate` contract, mirroring storage.)*+- [x] Hub: `set_/get_secrets_provider_registry` + module accessor.+- [x] New config `SecretsProviderConfig` (`pipelex/tools/secrets/secrets_config.py`), `method: str = Field(strict=False)`, **no class default**. Placed as sibling of `storage_config` under `pipelex.*` (single flat model, no `SecretsConfig` subclass needed — secrets has no is_fetch/is_upload analogue).+- [x] `pipelex/pipelex.toml`: added `[pipelex.secrets_config] method = "env"`; wired `secrets_config: SecretsProviderConfig` into the `Pipelex` config model. Same real-valued block added to `.pipelex/pipelex.toml` override. `make tb` green.+- [x] New `pipelex/plugins/secrets/secrets_plugin.py`: `SecretsPlugin` + `_make_env_secrets_provider(config)` → `EnvSecretsProvider()` (unused `config` param carries `# noqa: ARG001`, the plugins-dir convention for a callback-signature-conforming factory).+- [x] `builtins.py`: registered `SecretsPlugin()` in `BUILTIN_PLUGINS`; added `"secrets"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES`.+- [x] **W-A boot edit** (`pipelex.py`): `build_registrar` + boot-orchestrator gate + secrets registry/selection now sit **just before the telemetry factory** (not above the gateway block — see as-built). Downstream registry constructions still reference the same `plugin_registrar`.+- [x] **Deleted** the stale `# needed for gateway check` + hardcoded `EnvSecretsProvider()`; replaced with registry selection (explicit `setup(secrets_provider=...)` still wins). Dropped the now-unused `EnvSecretsProvider` import from `pipelex.py`.++### Phase 3 — as-built (deviations & decisions, for cold start)++- **W-A boot placement refined — after the precondition gate, not above the gateway block (the one real deviation).** The plan's literal W-A said move `build_registrar(config=get_config())` up to *before the gateway service block* (after line 209 / old line 212). Doing exactly that broke two unit tests — `tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py::{test_first_run_raises_inference_setup_required, test_needs_inference_true_raises_when_terms_not_accepted}`. Those tests exercise the gateway terms/first-run gate in isolation via `Pipelex.__new__(Pipelex)` (skipping `__init__`, so no config/hub is set) and rely on `setup()` reaching the terms-check `raise` **before** touching `get_config()` / `self.pipelex_hub`. Moving `build_registrar(config=get_config())` above the gate made `setup()` call `get_config()` first → `RuntimeError: Config instance is not set`. **Resolution:** placed the whole plugin-discovery + boot-orchestrator gate + secrets-registry/selection block **after** the gateway service/terms-check block and **immediately before the telemetry factory** — telemetry (`make_telemetry_manager`) is the *true first consumer* of `secrets_provider`, which is W-A's only hard constraint. This is strictly better than the plan's literal placement: the terms/first-run precondition gate fails fast before any discovery work, and no `__new__` test needed changing for the gate. (The boot-orchestrator gate also moved to this spot — still far earlier than its original post-`models_manager.setup` position, so fail-fast is preserved.)+- **`DuplicateSecretsProviderError` added** (not in the plan checklist) — required by the `_add` `on_duplicate` contract; mirrors `DuplicateStorageProviderError` / `DuplicateOrchestratorError`.+- **`SecretsProviderConfig` is a single flat model**, not a base+subclass like storage's `StorageProviderConfig`/`StorageConfig`. Storage's subclass exists only to add `is_fetch_remote_content_enabled`/`is_upload_local_content_enabled`; secrets has no such extra fields, so a lone `SecretsProviderConfig(ConfigModel)` with just `method` is the clean-solid shape (no speculative base).+- **Phase-1 escapee proactively found + fixed (same class as storage's).** `test_hub_slot_injection_precedence.py` boots a fake **empty** registrar (patched into `build_registrar`) and passed an explicit `storage_provider` but no `secrets_provider`; the new secrets selection path (`get_required(method="env")`) rejects the empty registry with `UnknownSecretsMethodError`. Fixed by injecting `EnvSecretsProvider()` in the four boots (+ docstring updated). Grep-verified this was the only `Pipelex.make` boot with a mocked empty/partial registrar (the plugins-dir tests call `build_registrar` directly; `test_storage_external_plugin` keeps the real `BUILTIN_PLUGINS`, so its `env` secrets provider is present). Caught here via the full `make agent-test` rather than escaping to a later checkpoint.+- **The env factory ignores config.** `_make_env_secrets_provider(config)` returns a bare `EnvSecretsProvider()`; the `config` param exists only to conform to `SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], ...]` (an SDK-backed external provider would read its own settings from it). Marked `# noqa: ARG001`, the established plugins-dir convention (see linkup/bedrock/azure_rest workers).++### ⛔ CHECKPOINT 3 — CLEARED+- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), **full `make agent-test` (exit 0, "All tests passed")**, `check-config-sync` PASSED (after the triage fix). Manual smokes: `pipelex plugins list` shows `secrets` (builtin, API 3, `secrets provider env`); default `method="env"`→`EnvSecretsProvider`; unknown token→`UnknownSecretsMethodError`.+- [x] Commit SHAs: `8a0b32668` (Phase 3 code) + `fdf3e114b` (review triage).+- [x] Cold-start state updated in this file.+- [x] Sonnet clean-room `/code-review` fanned out on `8a0b32668` (fresh general-purpose agent, no context, pointed only at the commit); findings triaged. **Outcome:** reviewer confirmed **no correctness bug in the boot-order move** (traced every consumer: `build_registrar` is pure, runs after the gateway/terms gate and before the telemetry factory, its only pre-hub consumer; `UnknownBootOrchestratorError` moving earlier is a correct fail-fast improvement). 4 findings:+ 1. *High — kit-config drift:* `pipelex/kit/configs/pipelex.toml` (the `pipelex init` template) missing the new `[pipelex.secrets_config]` block ⇒ `make check-config-sync` fails (CI + `make check` gate; **not** in `agent-check`, which is why Checkpoint-3 gates missed it). → **FIXED** in `fdf3e114b` via `make up-kit-configs`. *(Reconciled against the `worker_scopes` "not in kit copies" false-positive memory: that gate compares kit-configs ↔ `.pipelex` only, not the base `pipelex.toml`; a net-new section in `.pipelex` genuinely diverges — real, not a false positive.)*+ 2. *Medium — stale comment:* the secrets-hub-set comment still said the registrar is "built below," but the reorder moved that build above it. → **FIXED** in `fdf3e114b` (reworded).+ 3. *Over-engineering flag (registry+plugin+config+exceptions for a one-impl seam)* → **kept, no change.** Reviewer explicitly called it "consistent architecture applied deliberately across provider verticals, not a bug" — it is the plan-mandated byte-for-byte mirror of the accepted storage vertical.+ 4. *Docs + dedicated tests missing* → **deferred to Phase 4 as planned** (Phase 3 = mechanism/boot; Phase 4 = tests + docs, exactly as storage split 1↔2).++---++## Phase 4 — Secrets: tests + docs+*(Detail: `wip/plugins/secrets-provider-plugin.md` Phases 4–5.)*++- [x] Unit: `SecretsProviderRegistry` hit/miss (`UnknownSecretsMethodError`) + duplicate fail-loud. → `test_secrets_provider_registry.py` (byte-for-byte mirror of `test_storage_provider_registry.py`).+- [x] Boot: default config yields `EnvSecretsProvider` on the hub; explicit `setup(secrets_provider=...)` overrides. → `test_secrets_external_plugin.py` (first two methods). Plus a `test_secrets_plugin.py` unit (plugin registers the `env` factory; factory constructs `EnvSecretsProvider`), mirroring `test_storage_plugin.py` but lean (single builtin, no optional-dep matrix).+- [x] Integration: external test plugin registering a fake secrets `method` (`"test_secret"`, entry-point discovered) is selectable via config. → `test_secrets_external_plugin.py`.+- [x] Ordering guard: a boot with a non-env secrets method has secrets on the hub before storage's GCP arm reads it. → `test_secrets_external_plugin.py::test_config_selected_secrets_provider_is_read_by_storage_gcp_arm_at_boot`.+- [x] New `docs/under-the-hood/secrets-provider-plugins.md` (emphasizes the lazy optional-dep closure for SDK-backed Vault/AWS providers + the boot-ordering relationship); mkdocs nav (both blocks); cross-links (storage↔secrets both directions, inference-backend Related, secrets-provider-injection); updated the storage page's stale "planned" forward-ref.+- [x] CHANGELOG `[Unreleased]`: secrets is now a plugin seam; folded `add_secrets_provider` into the existing `PLUGIN_API_VERSION` 2→3 note.++### Phase 4 — as-built (deviations & decisions, for cold start)++- **The ordering guard is stronger than "assert boot order" implied — telemetry ALSO consumes the config-selected secrets provider, earlier.** First cut asserted `requested_secret_ids == ["GCP_CREDENTIALS_FILE_PATH"]` (exact). It failed: the recorded list led with `POSTHOG_ENDPOINT`, `POSTHOG_API_KEY`, … — the **telemetry factory** reads secrets from the same provider *before* storage is selected (secrets resolves right before the telemetry factory; storage much later). This is extra confirmation of correct ordering, not a bug. Fixed to a **membership** check: `"GCP_CREDENTIALS_FILE_PATH" in requested_secret_ids`. That token is requested **only** by storage's gcp factory (grep-confirmed), so its presence uniquely proves the gcp arm read the external provider from the hub. (Telemetry reads secrets even with `needs_inference=False` — it resolves the POSTHOG keys to build the manager, then the disabled path ignores them; the fake returning a canned gcp path for POSTHOG_* doesn't break boot.)+- **The gcp override needs BOTH `bucket_name` and `project_id`.** Base `pipelex.toml` ships `[pipelex.storage_config.gcp]` with `bucket_name = ""` **and** `project_id = ""`; `StorageGcpConfig.lazy_validate()` (run inside the gcp factory) rejects either empty. So the ordering-guard boot passes `config_overrides={"pipelex": {"storage_config": {"method": "gcp", "gcp": {"bucket_name": "test-bucket", "project_id": "test-project"}}}}` (deep-merge fills the rest). `GcpStorageProvider.__init__` only stores fields (the `find_spec`/creds-file check lives in `_get_bucket`), so constructing it at boot with a fake creds path is import-light and does no I/O — safe even though `google-cloud-storage` is installed here.+- **Default-env boot is deterministic on this branch.** `test_default_config_yields_the_env_provider_on_the_hub` boots with no `secrets_config` override and asserts `env`. Verified the machine global `~/.pipelex/pipelex.toml` has **no** `secrets_config` block (it's net-new here), so unlike storage's `method=s3` global-override gotcha, secrets has no machine override to fight — default→`env` holds locally and in CI.+- **Import-light test NOT extended for secrets (deliberate, unlike storage Phase 2).** `test_import_light_boot.py` gained a storage assert because s3/gcp are SDK-backed and BLOCKED there — a real import-light risk. The only builtin secrets method is `env` (no SDK), so an added `assert registrar.secrets_providers` would restate the registry unit test, not pin an import-light invariant. Left untouched to avoid over-engineering; an external SDK-backed secrets plugin would carry its own import-light guarantee.+- **Test module layout** (one TestClass per module): `test_secrets_provider_registry.py` (registry read-view + duplicate), `test_secrets_plugin.py` (builtin plugin + env factory), `test_secrets_external_plugin.py` (integration: default/override/external/ordering — one class, four methods). Integration harness identical to `test_storage_external_plugin.py`: `mocker.patch("pipelex.plugins.discovery._external_entry_points", …)` keeps the real `BUILTIN_PLUGINS`, per-module autouse teardown-first `reset_pipelex_config_fixture`. The fake external provider is a `SecretsProviderAbstract` subclass recording requested secret ids.+- **The Phase-3 escapee stayed fixed.** `test_hub_slot_injection_precedence.py` (fixed in Phase 3 to inject `EnvSecretsProvider()` in its four boots) passed under the full suite — no new escapee surfaced.++### ⛔ CHECKPOINT 4 — CLEARED+- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), `check-config-sync` PASSED, **full `make agent-test` (exit 0, "All tests passed")**. Targeted secrets-seam trio green in isolation (11 passed).+- [x] Commit SHA recorded here: `65603e912`+- [x] Cold-start state updated in this file.+- [x] Sonnet-5 clean-room `/code-review` fanned out on `65603e912` (fresh general-purpose agent, no context, pointed only at the commit). **Outcome: no substantive findings — nothing to fix, nothing to defer.** The reviewer ran all three test files against the tree (11 passed) and traced every factual claim to source: confirmed the secrets-before-telemetry-before-storage boot ordering in `pipelex.py`, the `gcp` factory's lazy credential check (fake creds path never touches disk / needs no GCP SDK at boot), telemetry's always-on `${POSTHOG_*}` secret resolution (validating the ordering-guard's membership-check rationale), `PLUGIN_API_VERSION` still `3` (changelog correctly folds in, no second bump), and identical nav placement in both mkdocs blocks. Verdict: assertions are concrete and non-tautological (the ordering guard is a genuine regression guard — reordering secrets-after-storage would raise or read the wrong provider inside the gcp factory), no over-engineering (secrets tests deliberately leaner than storage's — no optional-dep matrix, accurate to the single-impl `env` seam), no doc/code contradictions, all cross-links resolve.++---++## Phase 5 — Release gating & cross-repo (documentation in this branch; execution is release-gated)++- [x] Record in the CHANGELOG / release notes the cross-repo `targets_api=3` obligation. **Corrected during Checkpoint-5 triage (F2):** the original wording ("they register no provider; the bump is the only change") was wrong for `pipelex-temporal`. `pipelex-mistralai-workflows` is bump-only, but `pipelex-temporal`'s payload-codec factory imports the now-removed `make_storage_provider_from_config`, so it needs the `targets_api` bump **and** a code migration to `get_storage_provider_registry().get_required(...)`. CHANGELOG now says this + carries a `### Removed` bullet for the deleted factory. Did NOT touch those repos here.+- [x] Confirmed the whole branch is one clean per-phase commit sequence (feat/test → triage → bookkeeping per phase; `git log 04434f785..HEAD`). Opening the PR to the intended base is **left to Louis** (nothing pushed; no upstream).++### ⛔ CHECKPOINT 5 (final) — CLEARED+- [x] Gates green on the final tip: `make agent-check` (ruff/plxt/pyright 0 errors/mypy 2084 files/keyword-only), `make tb` (9 passed), `make check-config-sync` (passed), **full `make agent-test` (exit 0, "All tests passed")** — run both before the triage (tip `65603e912`+bookkeeping) and again after the triage commit `0bbc39474`.+- [x] Final Sonnet-5 clean-room `/code-review` over the **whole-branch diff** (`git diff 04434f785..HEAD`) — fresh general-purpose agent, no inherited context, 10 finder angles + line-by-line + a test-quality pass. **Outcome: migration verified largely clean and faithful** (behavior-preserving factory→plugin split, correct secrets→hub→telemetry→storage→gcp-reads-secrets boot ordering, safe `StorageMethod`→`str` widening, conventions honored). 9 findings, all triaged in `0bbc39474`:+ - **F2 [MED-HIGH] — FIXED (genuine, was actively misleading):** deleting `make_storage_provider_from_config` breaks `pipelex-temporal` at import (its codec factory calls it); CHANGELOG claimed a `targets_api` bump was temporal's only change. Corrected CHANGELOG + wip doc + added Removed bullet.+ - **F6 [LOW] — FIXED (simplification):** removed dead `get_optional` from both registries (no production consumer; `get_required` doesn't delegate to it; only its own unit tests used it). This reverses Checkpoint-1's keep-decision, which rested on a factual misread — siblings expose `get_optional`+`has`, while storage/secrets hard-fail via `get_required`+`methods`, so `get_optional` was a vestigial mechanical copy, not a convention match.+ - **F3 [MED] — FIXED:** added symmetric storage injection-precedence test (was tested for secrets only).+ - **F5 [LOW] — FIXED:** parametrized `test_disabling_core_unconditional_plugin_raises` over `direct/storage/secrets/openai`.+ - **F8 [LOW] — FIXED:** `contract.py` v3 comment no longer says `add_secrets_provider` is "in a follow-up commit" (it landed in Phase 3).+ - **F9 [LOW] — FIXED:** `orchestrator-plugins.md` no longer claims `PLUGIN_API_VERSION` "is now 2".+ - **F1 [HIGH per reviewer] — DEFERRED to D3 (design tradeoff, not a silent bug):** external storage method boots/selects cleanly but crashes on the first *generated-content* store because `uri_format` is defined only for the built-in methods (raises loudly on `case _:`). External config surface is explicitly the D3 follow-up; strengthened the D3 note + added a docs-page caveat. External providers are usable today for their own storage API, not yet as the generated-content backing store.+ - **F7 [LOW] — KEPT:** `Field(strict=False)` on the `method` fields is cosmetically vestigial but removal risks a config-parse regression at enum-member call sites for zero benefit.+ - **F4 [MED] — SKIPPED:** a `lazy_validate` negative-path factory test (trivial 3-line wiring; per-config validation already covered) — avoided test bloat.+- [x] This file updated to DONE with all SHAs; deferred follow-ups (D3 external config passthrough incl. the `uri_format` gap, first real external provider, cookbook example) captured under `wip/plugins/`.++---++## Deferred follow-ups (not in this plan — see wip docs)++- DX-3 external-provider config passthrough (storage D3 / secrets S4).+- First real external provider (e.g. Vault / AWS Secrets Manager) as proof-of-seam — likely a private package if hosted-platform-targeted.+- Cookbook `pipelex-storage-hello` external-plugin example, mirroring the hello-inference-plugin track.diff --git a/docs/under-the-hood/inference-backend-plugins.md b/docs/under-the-hood/inference-backend-plugins.mdindex 0e641a325..57c2b053d 100644--- a/docs/under-the-hood/inference-backend-plugins.md+++ b/docs/under-the-hood/inference-backend-plugins.md@@ -241,6 +241,9 @@ The SPI is a documented, versioned **module/symbol list** gated by `PLUGIN_API_V## Related+- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, riding the same discovery/denylist machinery+- [Storage Provider Plugins](storage-provider-plugins.md) — the config-selected-singleton seam (storage backend by `storage_config.method`)+- [Secrets Provider Plugins](secrets-provider-plugins.md) — the config-selected-singleton seam (secrets backend by `secrets_config.method`)- [Pipe Routing & Execution](pipe-routing-and-execution.md) — where worker construction sits in the run path- [Error Model](error-model.md) — how these errors render and dereference- [Architecture Overview](architecture-overview.md)diff --git a/docs/under-the-hood/orchestrator-plugins.md b/docs/under-the-hood/orchestrator-plugins.mdindex 879beb20f..ff65234f4 100644--- a/docs/under-the-hood/orchestrator-plugins.md+++ b/docs/under-the-hood/orchestrator-plugins.md@@ -119,7 +119,7 @@ The contract is deliberately split so no layer overreaches:- **Core** owns *transport* — `registrar.get_http_error_mappers()` runs every provider, builds the `{exc_type: mapper}` dict, and is fail-loud on a duplicate *resolved* exception type (naming both plugins). `ErrorReport` is a core type, so the seam carries **no** web-framework import.- **The host runtime** owns *presentation* — at app construction it iterates the resolved mappers and wraps each into one framework error handler (FastAPI, …) that runs the mapper, then renders the `ErrorReport` through its own RFC 7807 + `DisclosureMode` path. FastAPI / Starlette stays only in the host; core and the plugin import neither.-This is what lets the public `pipelex-api` base be orchestrator-agnostic and still render a Temporal (or Mistral) transport fault correctly: install the flavor's plugin and its mapper rides in; install none and there is simply nothing to wrap. The capability is optional, so it grew the plugin contract by one method → `PLUGIN_API_VERSION` is now **2**.+This is what lets the public `pipelex-api` base be orchestrator-agnostic and still render a Temporal (or Mistral) transport fault correctly: install the flavor's plugin and its mapper rides in; install none and there is simply nothing to wrap. The capability is optional, contributing one method to the plugin contract.---diff --git a/docs/under-the-hood/secrets-provider-plugins.md b/docs/under-the-hood/secrets-provider-plugins.mdnew file mode 100644index 000000000..06e49ebfb--- /dev/null+++ b/docs/under-the-hood/secrets-provider-plugins.md@@ -0,0 +1,169 @@+---+title: "Secrets Provider Plugins"+description: "How Pipelex selects a secrets backend by config, the keyed-registry-plus-config-selected-singleton seam a secrets plugin rides, and how to author an out-of-tree pipelex-secrets-<backend> package."+---++# Secrets Provider Plugins++Every credential Pipelex reads — an API key, a service-account file path, a signing secret — is resolved through a single **secrets provider** set on the hub at boot. Which provider that is comes entirely from data: one config field, `secrets_config.method`, names a backend, and a **secrets plugin** is what teaches Pipelex how to build the provider for that method.++Core names no secrets backend by import or by string. The built-in `env` provider (reads secrets from environment variables) is a plugin too — the always-on `SecretsPlugin` — riding the exact same seam an out-of-tree `pipelex-secrets-<backend>` package (Vault, AWS Secrets Manager, GCP Secret Manager, …) would. This page documents that seam, the factory contract a plugin registers, and how to write one.++This is the second application of the mechanism the [storage provider](storage-provider-plugins.md) seam introduced; secrets reuses it wholesale, so the two pages describe the same shape with the nouns swapped.++---++## A different shape from the inference/orchestrator seams++The [inference-backend](inference-backend-plugins.md) and [orchestrator](orchestrator-plugins.md) registries are selected **per call** — a model's `sdk`, a request's `orchestration_mode`. Secrets is not: it is a **process-global singleton selected by its own config key**, independent of the orchestrator. So it rides the same **keyed registry + config-selected singleton** mechanism storage uses:++> Plugins register N provider factories into a registry keyed by an open `method` token. At boot, core reads `secrets_config.method`, looks that token up in the registry, and calls the factory to produce the one provider set on the hub.++This is deliberately **not** a hub slot (those are orchestrator-coupled, claimed only when `boot_orchestrator == plugin.name`). Secrets selection has nothing to do with the orchestrator, so it gets its own registry and its own config key — the sibling of `storage_config.method`.++---++## The seam in one view++```+boot (Pipelex.setup)+ └─ build_registrar(config) # pure, import-light+ ├─ for each plugin in BUILTIN_PLUGINS (SecretsPlugin is one)+ └─ for each installed "pipelex.plugins" entry point+ └─ plugin.register(registrar) # side-effect-free+ └─ registrar.add_secrets_provider(method=…, factory=…)+ └─ SecretsProviderRegistry(registrar.secrets_providers) # stored on the hub+ └─ secrets_provider = registry.get_required(method=secrets_config.method)(secrets_config)+ └─ set_secrets_provider(secrets_provider) # the one provider every consumer reads+```++There is no `match secrets_config.method:` anywhere in boot — the token set is open, so validation *is* the registry lookup. Adding a backend means registering a factory for its token; nothing in core changes. Every downstream consumer (`get_secret(...)`, telemetry credentials, the storage `gcp` arm, search/extract workers) keeps calling `get_secrets_provider()` and is unaffected by which method was selected.++### Where in boot secrets is resolved++Secrets is resolved **early** — right after plugin discovery, after the gateway/terms precondition gate but before the telemetry factory (its first in-process consumer) and well before storage selection. Two consequences that this ordering guarantees:++- **Telemetry** receives the config-selected secrets provider when it is constructed.+- **Storage's `gcp` factory** reads the config-selected secrets provider from the hub at *its* apply-point (it resolves `GCP_CREDENTIALS_FILE_PATH`) — secrets is already on the hub by then. This ordering is pinned by an integration test that boots a non-`env` secrets method alongside `gcp` storage and asserts the gcp arm read the external provider.++### Injection precedence++`Pipelex.setup` resolves the secrets provider in this order:++1. an explicit `setup(secrets_provider=...)` parameter (test/host injection) — always wins;+2. the config-selected registry factory (`get_required(method=secrets_config.method)(config)`).++There is no separate core default: the built-in `SecretsPlugin` *is* the default supplier of the `env` method, so an ordinary boot always resolves through step 2.++---++## The contract: `SecretsProviderFactoryFn`++A secrets backend is a typed **callable** — whole secrets config in, provider out (`pipelex/plugins/secrets_provider_registry.py`):++```python+SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], SecretsProviderAbstract]+```++Passing the whole `SecretsProviderConfig` (not a pre-resolved sub-config) is deliberate: it lets a factory read whatever it needs — an SDK-backed provider (Vault, AWS Secrets Manager) reads its own settings — at the boot apply-point, never at registration. The built-in `env` factory ignores the config entirely.++A plugin contributes one factory per method it serves by calling the registrar menu in its `register`, passing the token as a raw string:++```python+registrar.add_secrets_provider(method="vault", factory=_make_vault_secrets_provider)+```++`register` is **side-effect-free** — it may call the registrar menu and nothing else (no hub access, no I/O, no SDK import). That is what keeps `build_registrar` safe to run more than once (at boot, and again for the `pipelex plugins list` diagnostic).++### Import-light registration, deferred dependency++The factory is a plain callable stored at registration and **invoked only at the boot apply-point**. Two invariants follow, and they are what let an SDK-backed provider ship without weighing down every boot:++- **Import-light.** The plugin module must import no backend SDK at module load. The built-in `SecretsPlugin` needs none (the `env` provider reads environment variables); an out-of-tree `pipelex-secrets-vault` plugin must keep its `hvac`/`boto3`/etc. import *inside* the provider's methods (or its factory), never at module top-level, so discovery stays import-light even when the extra is installed.+- **Fail at use, not at boot.** An optional dependency raises `MissingDependencyError` (naming the package and the `pipelex[<extra>]` install hint) when the backend is *used*, not when it is registered or selected.++---++## The built-in `SecretsPlugin`++`pipelex/plugins/secrets/secrets_plugin.py` is the reference secrets plugin. It is **core-unconditional** — a secrets provider is required infra, so it joins `CORE_UNCONDITIONAL_PLUGIN_NAMES` and cannot be disabled into a boot with no secrets provider:++```python+class SecretsPlugin:+ name = "secrets"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_secrets_provider(method="env", factory=_make_env_secrets_provider)+```++The `env` factory ignores its config and returns a bare `EnvSecretsProvider` — the `config` parameter exists only to conform to `SecretsProviderFactoryFn`; an SDK-backed external provider is where reading the config earns its keep:++```python+def _make_env_secrets_provider(config: SecretsProviderConfig) -> SecretsProviderAbstract:+ return EnvSecretsProvider()+```++`env` is a single-impl backend: `EnvSecretsProvider` resolves `get_required_secret` / `get_optional_secret` from environment variables, and its versioned reads (`*_specific_version`) raise `NotImplementedError` because environment variables have no version history. A full external provider (Vault, AWS Secrets Manager) implements those versioned reads.++---++## Selecting a method by config++`secrets_config.method` is an **open `str` token** (Decision S1), not a closed enum. The built-in uses `"env"`; an external `pipelex-secrets-<backend>` plugin registers its own (e.g. `"vault"`). A config naming an external method **parses fine** — the token is stored verbatim and its installability is validated later, at registry lookup:++```toml+# .pipelex/pipelex.toml+[pipelex.secrets_config]+method = "vault" # an out-of-tree provider — selected iff its plugin is installed+```++Whether that token names an *installed* provider is validated at **registry lookup**, not at parse: an unknown method surfaces as `UnknownSecretsMethodError` at boot, which is the right layer — it lists the registered methods so the fix is obvious.++!!! note "External-provider config surface is a scoped follow-up"+ `SecretsProviderConfig` today carries only `method` — the built-in `env` backend needs no per-method sub-config. An out-of-tree `vault` provider has nowhere to read *its* structured config yet; a generic passthrough for external providers is a captured follow-up (Decision S4), not built speculatively. Until it lands, an external provider reads its own config from the environment or its own file.++---++## Fail-loud guarantees++| Condition | Error |+|-----------|-------|+| `secrets_config.method` names no registered provider | `UnknownSecretsMethodError` (lists the registered methods) |+| two plugins register the same `method` | `DuplicateSecretsProviderError` (names both plugins) |+| `name` (`"secrets"`) in `plugins.disabled` | `CoreUnconditionalPluginDisabledError` |+| entry point raises while loading/registering | `BrokenPluginError` |+| optional SDK missing at use | `MissingDependencyError` (package + `pipelex[<extra>]` hint) |++The duplicate detection is the same fail-loud `_add` helper the inference/orchestrator/storage menus use — a conflict names both contributing plugins, so a collision between two installed secrets packages is unambiguous.++---++## Authoring an out-of-tree secrets plugin++A third-party secrets plugin is a distribution that:++1. implements a `SecretsProviderAbstract` subclass — `get_required_secret` / `get_optional_secret`, the versioned `*_specific_version` reads, and `set_secret_as_env_var` — deferring any SDK import into those methods (import-light);+2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_secrets_provider(method="<token>", factory=...)` — and nothing else;+3. advertises itself under the `pipelex.plugins` entry-point group:++```toml+# pyproject.toml of your plugin package+[project.entry-points."pipelex.plugins"]+vault_secrets = "pipelex_secrets_vault.plugin:VaultSecretsPlugin"+```++Installing the distribution makes the method selectable (`secrets_config.method = "vault"`); uninstalling removes it. No core change, no central registration list — *presence* is the source of truth. A discovered plugin can be quarantined without uninstalling via the `plugins.disabled` denylist (matched against the entry-point name *before* load, so a broken install can still be disabled to recover startup — see [Inference Backend Plugins](inference-backend-plugins.md) for the shared discovery/denylist machinery).++Use `pipelex plugins list` to see every discovered plugin, what each contributed, and its denylist state.++---++## Related++- [Storage Provider Plugins](storage-provider-plugins.md) — the sibling config-selected-singleton seam this one mirrors; its `gcp` arm reads the secrets provider at boot+- [Secrets Provider Injection](../advanced/secrets-provider-injection.md) — supplying a custom secrets provider from your own vault or secrets manager+- [Inference Backend Plugins](inference-backend-plugins.md) — the per-call sibling seam and the shared discovery/denylist machinery+- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, plus the hub-slot mechanism these config-selected seams deliberately avoid+- [Error Model](error-model.md) — how these errors render and dereference+- [Architecture Overview](architecture-overview.md)diff --git a/docs/under-the-hood/storage-provider-plugins.md b/docs/under-the-hood/storage-provider-plugins.mdnew file mode 100644index 000000000..c5a1a5212--- /dev/null+++ b/docs/under-the-hood/storage-provider-plugins.md@@ -0,0 +1,170 @@+---+title: "Storage Provider Plugins"+description: "How Pipelex selects a storage backend by config, the keyed-registry-plus-config-selected-singleton seam a storage plugin rides, and how to author an out-of-tree pipelex-storage-<backend> package."+---++# Storage Provider Plugins++Every asset Pipelex persists — a generated image, an uploaded document, a payload blob — is written through a single **storage provider** set on the hub at boot. Which provider that is comes entirely from data: one config field, `storage_config.method`, names a backend, and a **storage plugin** is what teaches Pipelex how to build the provider for that method.++Core names no storage backend by import or by string. The built-in providers (`local`, `in_memory`, `s3`, `gcp`) are a plugin too — the always-on `StoragePlugin` — riding the exact same seam an out-of-tree `pipelex-storage-<backend>` package would. This page documents that seam, the factory contract a plugin registers, and how to write one.++---++## A different shape from the inference/orchestrator seams++The [inference-backend](inference-backend-plugins.md) and [orchestrator](orchestrator-plugins.md) registries are selected **per call** — a model's `sdk`, a request's `orchestration_mode`. Storage is not: it is a **process-global singleton selected by its own config key**, independent of the orchestrator. So it rides a net-new mechanism the track calls a **keyed registry + config-selected singleton**:++> Plugins register N provider factories into a registry keyed by an open `method` token. At boot, core reads `storage_config.method`, looks that token up in the registry, and calls the factory to produce the one provider set on the hub.++This is deliberately **not** a hub slot (those are orchestrator-coupled, claimed only when `boot_orchestrator == plugin.name`). Storage selection has nothing to do with the orchestrator, so it gets its own registry and its own config key. The same mechanism also backs the [secrets provider](secrets-provider-plugins.md) seam (`secrets_config.method`).++---++## The seam in one view++```+boot (Pipelex.setup)+ └─ build_registrar(config) # pure, import-light+ ├─ for each plugin in BUILTIN_PLUGINS (StoragePlugin is one)+ └─ for each installed "pipelex.plugins" entry point+ └─ plugin.register(registrar) # side-effect-free+ └─ registrar.add_storage_provider(method=…, factory=…)+ └─ StorageProviderRegistry(registrar.storage_providers) # stored on the hub+ └─ storage_provider = registry.get_required(method=storage_config.method)(storage_config)+ └─ set_storage_provider(storage_provider) # the one provider every consumer reads+```++There is no `match storage_config.method:` anywhere in boot — the token set is open, so validation *is* the registry lookup. Adding a backend means registering a factory for its token; nothing in core changes. Every downstream consumer (input normalizer, content generator, PDF/image utilities, delivery executor) keeps calling `get_storage_provider()` and is unaffected by which method was selected.++The provider is resolved **after** the secrets provider is on the hub, so a factory may perform a hub secrets read at its apply-point (the `gcp` factory does — see below).++### Injection precedence++`Pipelex.setup` resolves the storage provider in this order:++1. an explicit `setup(storage_provider=...)` parameter (test/host injection) — always wins;+2. the config-selected registry factory (`get_required(method=storage_config.method)(config)`).++There is no separate core default: the built-in `StoragePlugin` *is* the default supplier of every method, so an ordinary boot always resolves through step 2.++---++## The contract: `StorageProviderFactoryFn`++A storage backend is a typed **callable** — whole storage config in, provider out (`pipelex/plugins/storage_provider_registry.py`):++```python+StorageProviderFactoryFn = Callable[[StorageProviderConfig], StorageProviderAbstract]+```++Passing the whole `StorageProviderConfig` (not a pre-resolved sub-config) is deliberate: it lets a factory read whatever it needs — including a hub secrets read — at the boot apply-point, never at registration.++A plugin contributes one factory per method it serves by calling the registrar menu in its `register`, passing the token as a raw string:++```python+registrar.add_storage_provider(method="azure", factory=_make_azure_storage_provider)+```++`register` is **side-effect-free** — it may call the registrar menu and nothing else (no hub access, no I/O, no SDK import). That is what keeps `build_registrar` safe to run more than once (at boot, and again for the `pipelex plugins list` diagnostic).++### Import-light registration, deferred dependency++The factory is a plain callable stored at registration and **invoked only at the boot apply-point**. Two invariants follow, and they are what let an SDK-backed provider ship without weighing down every boot:++- **Import-light.** The plugin module must import no backend SDK at module load. Registering the built-in `s3`/`gcp` factories pulls in neither `aioboto3` nor `google-cloud-storage` — the SDK guard lives *inside* the provider's I/O methods (`_get_session` / `_get_bucket`), not its `__init__`. Selecting `s3` therefore *constructs* an `S3StorageProvider` even with the extra absent; the guard fires only on the first actual load/store.+- **Fail at use, not at boot.** An optional dependency raises `MissingDependencyError` (naming the package and the `pipelex[<extra>]` install hint) when the backend is *used*, not when it is registered or selected.++---++## The built-in `StoragePlugin`++`pipelex/plugins/storage/storage_plugin.py` is the reference storage plugin. It is **core-unconditional** — storage is required infra, so it joins `CORE_UNCONDITIONAL_PLUGIN_NAMES` and cannot be disabled into a boot with no storage:++```python+class StoragePlugin:+ name = "storage"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_storage_provider(method=StorageMethod.LOCAL, factory=_make_local_storage_provider)+ registrar.add_storage_provider(method=StorageMethod.IN_MEMORY, factory=_make_in_memory_storage_provider)+ registrar.add_storage_provider(method=StorageMethod.S3, factory=_make_s3_storage_provider)+ registrar.add_storage_provider(method=StorageMethod.GCP, factory=_make_gcp_storage_provider)+```++Each factory is a module-level closure that None-checks its sub-config, calls `lazy_validate()`, and constructs the provider. The built-in tokens are the `StorageMethod` enum values (`"local"`, `"in_memory"`, `"s3"`, `"gcp"`); because a `StrEnum` key compares and hashes equal to its plain-`str` form, boot's plain-`str` `config.method` resolves them.++The `gcp` factory is the one that reads a secret from the hub — legal because the factory runs at the boot apply-point, after the secrets provider is set, never inside `register`:++```python+def _make_gcp_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:+ ...+ return GcpStorageProvider(+ bucket_name=config.gcp.bucket_name,+ project_id=config.gcp.project_id,+ credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),+ signed_urls_lifespan=config.gcp.signed_urls_lifespan,+ )+```++---++## Selecting a method by config++`storage_config.method` is an **open `str` token** (Decision D1), not a closed enum. The built-ins use the `StorageMethod` values; an external `pipelex-storage-<backend>` plugin registers its own (e.g. `"azure"`). A config naming an external method **parses fine** — the per-method sub-config validator requires a matching sub-config only for the four built-in tokens and lets any other token through:++```toml+# .pipelex/pipelex.toml+[pipelex.storage_config]+method = "azure" # an out-of-tree provider — no built-in sub-config required+```++Whether that token names an *installed* provider is validated at **registry lookup**, not at parse: an unknown method surfaces as `UnknownStorageMethodError` at boot, which is the right layer — it lists the registered methods so the fix is obvious.++!!! note "External-provider config surface is a scoped follow-up"+ `StorageProviderConfig` has fixed, typed per-method sub-models, so an out-of-tree `azure` provider has nowhere to read *its* structured config yet. The seam for built-in methods lands first; a generic passthrough sub-config for external providers is a captured follow-up (Decision D3), not built speculatively. Until it lands, an external provider reads its own config from the environment or its own file. **One live gap:** `StorageProviderConfig.uri_format` (read by `GeneratedContentFactory._build_storage_key` on every content store) is only defined for the four built-in methods and raises `StorageConfigError` for any other token — so an external provider selects and boots cleanly but a *generated-content store* through it fails until D3 gives external methods a `uri_format`. External providers are therefore usable today for their own direct storage API, not yet as the backing store for generated content.++---++## Fail-loud guarantees++| Condition | Error |+|-----------|-------|+| `storage_config.method` names no registered provider | `UnknownStorageMethodError` (lists the registered methods) |+| two plugins register the same `method` | `DuplicateStorageProviderError` (names both plugins) |+| `name` (`"storage"`) in `plugins.disabled` | `CoreUnconditionalPluginDisabledError` |+| entry point raises while loading/registering | `BrokenPluginError` |+| optional SDK missing at use | `MissingDependencyError` (package + `pipelex[<extra>]` hint) |++The duplicate detection is the same fail-loud `_add` helper the inference/orchestrator menus use — a conflict names both contributing plugins, so a collision between two installed storage packages is unambiguous.++---++## Authoring an out-of-tree storage plugin++A third-party storage plugin is a distribution that:++1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light);+2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_storage_provider(method="<token>", factory=...)` — and nothing else;+3. advertises itself under the `pipelex.plugins` entry-point group:++```toml+# pyproject.toml of your plugin package+[project.entry-points."pipelex.plugins"]+azure_storage = "pipelex_storage_azure.plugin:AzureStoragePlugin"+```++Installing the distribution makes the method selectable (`storage_config.method = "azure"`); uninstalling removes it. No core change, no central registration list — *presence* is the source of truth. A discovered plugin can be quarantined without uninstalling via the `plugins.disabled` denylist (matched against the entry-point name *before* load, so a broken install can still be disabled to recover startup — see [Inference Backend Plugins](inference-backend-plugins.md) for the shared discovery/denylist machinery).++Use `pipelex plugins list` to see every discovered plugin, what each contributed, and its denylist state.++---++## Related++- [Secrets Provider Plugins](secrets-provider-plugins.md) — the sibling config-selected-singleton seam riding the same mechanism; the `gcp` factory above reads its provider+- [Inference Backend Plugins](inference-backend-plugins.md) — the per-call sibling seam and the shared discovery/denylist machinery+- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, plus the hub-slot mechanism storage deliberately avoids+- [Error Model](error-model.md) — how these errors render and dereference+- [Architecture Overview](architecture-overview.md)diff --git a/mkdocs.yml b/mkdocs.ymlindex 0db0e4f8b..d84b6140e 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -316,6 +316,8 @@ plugins:- under-the-hood/pipe-routing-and-execution.md: "Pipe Routing & Execution"- under-the-hood/inference-backend-plugins.md: "Inference Backend Plugins"- under-the-hood/orchestrator-plugins.md: "Orchestrator Plugins"+ - under-the-hood/storage-provider-plugins.md: "Storage Provider Plugins"+ - under-the-hood/secrets-provider-plugins.md: "Secrets Provider Plugins"- under-the-hood/error-model.md: "Error Model"- under-the-hood/runtime-bridge-and-transport.md: "Runtime Bridge & Transport"- under-the-hood/distributed-content-generation.md: "Content Generation Across Boundaries"@@ -547,6 +549,8 @@ nav:- Pipe Routing & Execution: under-the-hood/pipe-routing-and-execution.md- Inference Backend Plugins: under-the-hood/inference-backend-plugins.md- Orchestrator Plugins: under-the-hood/orchestrator-plugins.md+ - Storage Provider Plugins: under-the-hood/storage-provider-plugins.md+ - Secrets Provider Plugins: under-the-hood/secrets-provider-plugins.md- Error Model: under-the-hood/error-model.md- Runtime Bridge & Transport: under-the-hood/runtime-bridge-and-transport.md- Content Generation Across Boundaries: under-the-hood/distributed-content-generation.mddiff --git a/pipelex/hub.py b/pipelex/hub.pyindex d998ddef7..1d5bf8fea 100644--- a/pipelex/hub.py+++ b/pipelex/hub.py@@ -53,6 +53,8 @@ if TYPE_CHECKING:from pipelex.plugins.inference_backend_registry import InferenceBackendRegistryfrom pipelex.plugins.model_lister_registry import ModelListerRegistryfrom pipelex.plugins.orchestrator_registry import OrchestratorRegistry+ from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry+ from pipelex.plugins.storage_provider_registry import StorageProviderRegistryfrom pipelex.tracing.event_log_protocol import EventLogProtocol@@ -89,6 +91,8 @@ class PipelexHub:self._model_lister_registry: ModelListerRegistry | None = Noneself._orchestrator_registry: OrchestratorRegistry | None = Noneself._bundle_validator_registry: BundleValidatorRegistry | None = None+ self._storage_provider_registry: StorageProviderRegistry | None = None+ self._secrets_provider_registry: SecretsProviderRegistry | None = Noneself._inference_manager: InferenceManagerProtocolself._report_delegate: ReportingProtocolself._content_generator: ContentGeneratorProtocol | None = None@@ -212,6 +216,12 @@ class PipelexHub:def set_bundle_validator_registry(self, bundle_validator_registry: "BundleValidatorRegistry"):self._bundle_validator_registry = bundle_validator_registry+ def set_storage_provider_registry(self, storage_provider_registry: "StorageProviderRegistry"):+ self._storage_provider_registry = storage_provider_registry++ def set_secrets_provider_registry(self, secrets_provider_registry: "SecretsProviderRegistry"):+ self._secrets_provider_registry = secrets_provider_registry+def set_inference_manager(self, inference_manager: InferenceManagerProtocol):self._inference_manager = inference_manager@@ -353,6 +363,18 @@ class PipelexHub:raise RuntimeError(msg)return self._bundle_validator_registry+ def get_storage_provider_registry(self) -> "StorageProviderRegistry":+ if self._storage_provider_registry is None:+ msg = "StorageProviderRegistry is not initialized"+ raise RuntimeError(msg)+ return self._storage_provider_registry++ def get_secrets_provider_registry(self) -> "SecretsProviderRegistry":+ if self._secrets_provider_registry is None:+ msg = "SecretsProviderRegistry is not initialized"+ raise RuntimeError(msg)+ return self._secrets_provider_registry+def get_inference_manager(self) -> InferenceManagerProtocol:return self._inference_manager@@ -534,6 +556,14 @@ def get_bundle_validator_registry() -> "BundleValidatorRegistry":return get_pipelex_hub().get_bundle_validator_registry()+def get_storage_provider_registry() -> "StorageProviderRegistry":+ return get_pipelex_hub().get_storage_provider_registry()+++def get_secrets_provider_registry() -> "SecretsProviderRegistry":+ return get_pipelex_hub().get_secrets_provider_registry()++def get_inference_manager() -> InferenceManagerProtocol:return get_pipelex_hub().get_inference_manager()diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.tomlindex 25d68f91d..a508997f6 100644--- a/pipelex/kit/configs/pipelex.toml+++ b/pipelex/kit/configs/pipelex.toml@@ -119,6 +119,15 @@ bucket_name = ""project_id = ""signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs+####################################################################################################+# Secrets Config+####################################################################################################++[pipelex.secrets_config]+# Secrets provider method: "env" (built-in, reads secrets from environment variables).+# An external pipelex-secrets-<backend> plugin can add its own method (e.g. "vault").+method = "env"+##################################################################################################### Scan Config####################################################################################################diff --git a/pipelex/pipelex.py b/pipelex/pipelex.pyindex 9c6fa44bb..a70c2b884 100644--- a/pipelex/pipelex.py+++ b/pipelex/pipelex.py@@ -56,6 +56,8 @@ from pipelex.plugins.model_lister_registry import ModelListerRegistryfrom pipelex.plugins.orchestrator_registry import OrchestratorRegistryfrom pipelex.plugins.registrar import HubSlot, PluginRegistrarfrom pipelex.plugins.sdk_client_manager import SdkClientManager+from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry+from pipelex.plugins.storage_provider_registry import StorageProviderRegistryfrom pipelex.reporting.reporting_manager import ReportingManagerfrom pipelex.reporting.reporting_protocol import ReportingProtocolfrom pipelex.system.configuration.config_loader import config_manager@@ -87,10 +89,8 @@ from pipelex.test_extras.registry_test_models import TestRegistryModelsfrom pipelex.tools.jinja2.jinja2_template_loader import TemplateLoaderfrom pipelex.tools.jinja2.jinja2_template_registry import TemplateRegistryfrom pipelex.tools.misc.package_utils import get_package_info-from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProviderfrom pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstractfrom pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract-from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_configfrom pipelex.types import Selffrom pipelex.urls import URLs@@ -208,9 +208,6 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.if boot_orchestrator is not None:get_config().plugins.boot_orchestrator = boot_orchestrator- # Initialize secrets provider early - needed for gateway check- secrets_provider = secrets_provider or EnvSecretsProvider()-# --- Pipelex Service and Telemetry --------------------------------------------------# Check if Pipelex Gateway is enabled@@ -278,6 +275,38 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.stacklevel=2,)+ # --- Plugin discovery -----------------------------------------------------------------+ # Build the plugin registrar from the fully-resolved config (pure and import-light:+ # registering the built-ins imports no backend SDK, constructs no client, touches no hub).+ # Built here — after the gateway service/terms precondition gate above (so an unaccepted-terms or+ # first-run boot fails fast before any discovery work) and before the telemetry factory below,+ # which is the first consumer of the secrets provider. Secrets is now a config-selected plugin+ # seam: the built-in SecretsPlugin's factory (and any external pipelex-secrets-<backend>) is+ # looked up from the registrar-derived SecretsProviderRegistry just below. The other registries+ # (inference, storage, …) are still built later at their own hub-set points, all referencing this+ # same already-built registrar; the slot-claim thunks / teardown callbacks it also accumulates are+ # applied at their ordered apply-points in later phases.+ plugin_registrar = build_registrar(config=get_config())+ self._plugin_registrar = plugin_registrar+ # Reject an unknown boot orchestrator before falling through to the core defaults. The requested+ # name (CLI --orchestrator / setup(boot_orchestrator=...) / config) is matched against registered+ # plugin names — the same namespace the slot-claim gate uses (boot_orchestrator == plugin.name).+ # When no plugin of that name registered (not installed, disabled, or a typo) nothing claims the+ # hub slots, so without this guard the run would silently execute in-process instead of under the+ # requested runtime. Checked here to fail fast, before the telemetry/model work.+ requested_boot_orchestrator = get_config().plugins.boot_orchestrator+ if requested_boot_orchestrator is not None and requested_boot_orchestrator not in plugin_registrar.registered_plugin_names:+ raise UnknownBootOrchestratorError(requested=requested_boot_orchestrator)++ # Secrets provider precedence: explicit setup() param > config-selected registry factory.+ # The built-in SecretsPlugin supplies the "env" method, so there is no separate core default.+ # Resolved here because the telemetry factory just below (and the model setup further down) consume it.+ secrets_provider_registry = SecretsProviderRegistry(plugin_registrar.secrets_providers)+ self.pipelex_hub.set_secrets_provider_registry(secrets_provider_registry)+ if secrets_provider is None:+ secrets_config = get_config().pipelex.secrets_config+ secrets_provider = secrets_provider_registry.get_required(method=secrets_config.method)(secrets_config)+# Disable Pipelex telemetry when:# - inference is not needed (no live runs to track), OR# - the gateway config came from the cache (stale specs imply potentially stale model@@ -304,10 +333,10 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.self.func_registry = func_registry or FuncRegistry()self.pipelex_hub.set_func_registry(func_registry=self.func_registry)self.pipelex_hub.set_secrets_provider(secrets_provider=secrets_provider)- if storage_provider is None:- storage_config = get_config().pipelex.storage_config- storage_provider = make_storage_provider_from_config(storage_config)- self.pipelex_hub.set_storage_provider(storage_provider)+ # Storage is selected from the config-driven StorageProviderRegistry, built from the plugin+ # registrar (constructed above, just before the telemetry factory). Its resolution and hub-set+ # still happen later at the plugin-derived-registries block — after secrets is on the hub here,+ # so the GCP factory's secret read works.# Register stuff templates first (used by mermaid, reactflow, and stuff_viewer)stuff_name, stuff_package, stuff_templates = STUFF_TEMPLATE_SET@@ -383,28 +412,25 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.# backend-keyed unconditionally (eng review D4) — a keyless Temporal submitter must still# dispatch activities and mock inside them, so `needs_inference` plays no role in picking the# generator. Its other boot roles (gateway/model setup, credentials, telemetry) are unchanged.- # --- Plugin discovery ------------------------------------------------------------------ # Build the plugin registrar from the fully-resolved config (pure and import-light:- # registering the built-ins imports no backend SDK). Runs after the gateway/model setup- # checks and before the hub setup points below — the family worker factories look their- # backends up on the hub registries at run time. The slot-claim thunks, CLI commands and- # teardown callbacks the registrar also accumulates are applied at their ordered- # apply-points in a later phase (no orchestrator plugin contributes any of them yet).- plugin_registrar = build_registrar(config=get_config())- self._plugin_registrar = plugin_registrar- # Reject an unknown boot orchestrator before falling through to the core defaults. The- # requested name (CLI --orchestrator / setup(boot_orchestrator=...) / config) is matched- # against registered plugin names — the same namespace the slot-claim gate uses- # (boot_orchestrator == plugin.name). When no plugin of that name registered (not installed,- # disabled, or a typo) nothing claims the hub slots, so without this guard the run would- # silently execute in-process instead of under the requested runtime.- requested_boot_orchestrator = get_config().plugins.boot_orchestrator- if requested_boot_orchestrator is not None and requested_boot_orchestrator not in plugin_registrar.registered_plugin_names:- raise UnknownBootOrchestratorError(requested=requested_boot_orchestrator)+ # --- Plugin-derived registries --------------------------------------------------------+ # The plugin registrar was built earlier (with the boot-orchestrator gate checked and the+ # config-selected secrets provider resolved) just before the telemetry factory. Turn its+ # accumulated contributions into the hub registries here — after the gateway/model setup checks+ # and before the hub setup points below — the family worker factories look their backends up on+ # these at run time.self.pipelex_hub.set_inference_backend_registry(InferenceBackendRegistry(plugin_registrar.inference_backends))self.pipelex_hub.set_model_lister_registry(ModelListerRegistry(plugin_registrar.model_listers))self.pipelex_hub.set_orchestrator_registry(OrchestratorRegistry(plugin_registrar.orchestrators))self.pipelex_hub.set_bundle_validator_registry(BundleValidatorRegistry(plugin_registrar.bundle_validators))+ storage_provider_registry = StorageProviderRegistry(plugin_registrar.storage_providers)+ self.pipelex_hub.set_storage_provider_registry(storage_provider_registry)+ # Storage provider precedence: explicit setup() param > config-selected registry factory.+ # The built-in StoragePlugin supplies every method, so there is no separate core default.+ # Resolves here (after secrets is on the hub) so the GCP factory's secret read works.+ if storage_provider is None:+ storage_config = get_config().pipelex.storage_config+ storage_provider = storage_provider_registry.get_required(method=storage_config.method)(storage_config)+ self.pipelex_hub.set_storage_provider(storage_provider)self.pipelex_hub.set_dry_run_forced(not needs_inference)# Injection precedence (codex C8): explicit setup() param > plugin slot-claim thunk > core default.diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.tomlindex ddb1c75f9..8952bf3f4 100644--- a/pipelex/pipelex.toml+++ b/pipelex/pipelex.toml@@ -26,6 +26,9 @@ bucket_name = ""project_id = ""signed_urls_lifespan_seconds = 3600+[pipelex.secrets_config]+method = "env" # env (built-in); an external plugin may add e.g. vault+[pipelex.observer_config]observer_dir = "results/observer"diff --git a/pipelex/plugins/builtins.py b/pipelex/plugins/builtins.pyindex 690178120..c00627b63 100644--- a/pipelex/plugins/builtins.py+++ b/pipelex/plugins/builtins.py@@ -15,12 +15,16 @@ from pipelex.plugins.openai.openai_plugin import OpenAIPluginfrom pipelex.plugins.openrouter.openrouter_plugin import OpenRouterPluginfrom pipelex.plugins.portkey.portkey_plugin import PortkeyPluginfrom pipelex.plugins.pypdfium2.pypdfium2_plugin import Pypdfium2Plugin+from pipelex.plugins.secrets.secrets_plugin import SecretsPlugin+from pipelex.plugins.storage.storage_plugin import StoragePlugin# The plugins Pipelex ships with — discovered at boot ahead of any external# entry-point plugin. Each is import-light: importing this module imports no# backend SDK (the SDKs load lazily inside the make_worker closures).BUILTIN_PLUGINS: list[PipelexPlugin] = [DirectOrchestratorPlugin(),+ StoragePlugin(),+ SecretsPlugin(),OpenAIPlugin(),GatewayPlugin(),PortkeyPlugin(),@@ -41,6 +45,9 @@ BUILTIN_PLUGINS: list[PipelexPlugin] = [# Built-in plugins that core requires unconditionally — naming one in# ``plugins.disabled`` is a configuration error, not a no-op. ``direct`` owns the# DIRECT orchestrator (you cannot boot without an in-process execution mode);-# ``openai`` is the always-on default inference driver (no optional SDK to avoid),-# so disabling it would only break the out-of-the-box experience.-CORE_UNCONDITIONAL_PLUGIN_NAMES: frozenset[str] = frozenset({"direct", "openai"})+# ``storage`` supplies every built-in storage backend (``storage_config.method`` must+# resolve to a registered factory or boot fails loud); ``secrets`` supplies the built-in+# ``env`` secrets backend (``secrets_config.method`` must likewise resolve or boot fails loud);+# ``openai`` is the always-on default inference driver (no optional SDK to avoid), so disabling it+# would only break the out-of-the-box experience.+CORE_UNCONDITIONAL_PLUGIN_NAMES: frozenset[str] = frozenset({"direct", "storage", "secrets", "openai"})diff --git a/pipelex/plugins/contract.py b/pipelex/plugins/contract.pyindex 2ea89c730..5acf8c7d7 100644--- a/pipelex/plugins/contract.py+++ b/pipelex/plugins/contract.py@@ -11,7 +11,12 @@ if TYPE_CHECKING:# v2 added the optional ``add_http_error_mapper`` capability (a framework-agnostic# transport-fault → ``ErrorReport`` mapping a host runtime renders into its own# HTTP error response).-PLUGIN_API_VERSION: int = 2+#+# v3 added ``add_storage_provider`` and ``add_secrets_provider`` — two config-selected,+# process-global provider registries (``storage_config.method`` / ``secrets_config.method`` pick+# the factory at boot). DX-1 batches both menu additions under this single bump so external plugins+# re-declare ``targets_api`` only once.+PLUGIN_API_VERSION: int = 3@runtime_checkablediff --git a/pipelex/plugins/exceptions.py b/pipelex/plugins/exceptions.pyindex 57ec3416e..65a34fa0a 100644--- a/pipelex/plugins/exceptions.py+++ b/pipelex/plugins/exceptions.py@@ -95,6 +95,34 @@ class DuplicateBundleValidatorError(PluginError):super().__init__(message)+class DuplicateStorageProviderError(PluginError):+ """Two plugins registered a storage provider for the same method."""++ def __init__(self, *, method: str, first_plugin: str, second_plugin: str):+ self.method = method+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"Storage provider for method '{method}' is registered by both plugin "+ f"'{first_plugin}' and plugin '{second_plugin}'. Each method must have a single provider."+ )+ super().__init__(message)+++class DuplicateSecretsProviderError(PluginError):+ """Two plugins registered a secrets provider for the same method."""++ def __init__(self, *, method: str, first_plugin: str, second_plugin: str):+ self.method = method+ self.first_plugin = first_plugin+ self.second_plugin = second_plugin+ message = (+ f"Secrets provider for method '{method}' is registered by both plugin "+ f"'{first_plugin}' and plugin '{second_plugin}'. Each method must have a single provider."+ )+ super().__init__(message)++class DuplicateHttpErrorMapperError(PluginError):"""Two plugins registered an HTTP-error mapper for the same exception type."""@@ -167,3 +195,45 @@ class UnknownBootOrchestratorError(PluginError):"Core provides only in-process execution; booting under a distributed orchestrator requires installing its plugin.")super().__init__(message)+++class UnknownStorageMethodError(PluginError):+ """A configured storage method has no registered provider factory.++ ``storage_config.method`` selects a storage provider from the registry the built-in+ ``StoragePlugin`` (and any external ``pipelex-storage-<backend>`` plugin) populates. When+ the token names no registered factory — a typo, or an external provider plugin that is not+ installed or was disabled via ``plugins.disabled`` — boot fails loud here rather than+ starting with no storage. The message lists the registered methods so the fix is obvious.+ """++ def __init__(self, *, method: str, registered_methods: list[str]):+ self.method = method+ self.registered_methods = registered_methods+ available = ", ".join(sorted(registered_methods)) or "(none)"+ message = (+ f"No storage provider is registered for method '{method}'. Registered methods: {available}. "+ "Check storage_config.method, or install/enable the plugin that provides that method."+ )+ super().__init__(message)+++class UnknownSecretsMethodError(PluginError):+ """A configured secrets method has no registered provider factory.++ ``secrets_config.method`` selects a secrets provider from the registry the built-in+ ``SecretsPlugin`` (and any external ``pipelex-secrets-<backend>`` plugin) populates. When+ the token names no registered factory — a typo, or an external provider plugin that is not+ installed or was disabled via ``plugins.disabled`` — boot fails loud here rather than+ starting with no secrets provider. The message lists the registered methods so the fix is obvious.+ """++ def __init__(self, *, method: str, registered_methods: list[str]):+ self.method = method+ self.registered_methods = registered_methods+ available = ", ".join(sorted(registered_methods)) or "(none)"+ message = (+ f"No secrets provider is registered for method '{method}'. Registered methods: {available}. "+ "Check secrets_config.method, or install/enable the plugin that provides that method."+ )+ super().__init__(message)diff --git a/pipelex/plugins/registrar.py b/pipelex/plugins/registrar.pyindex 7321c3872..5ac094957 100644--- a/pipelex/plugins/registrar.py+++ b/pipelex/plugins/registrar.py@@ -11,11 +11,15 @@ from pipelex.plugins.exceptions import (DuplicateInferenceBackendError,DuplicateModelListerError,DuplicateOrchestratorError,+ DuplicateSecretsProviderError,+ DuplicateStorageProviderError,HubSlotAlreadyClaimedError,)from pipelex.plugins.inference_backend_registry import InferenceFamily, MakeWorkerFnfrom pipelex.plugins.model_lister_registry import ListModelsFnfrom pipelex.plugins.orchestrator_registry import OrchestratorProtocol+from pipelex.plugins.secrets_provider_registry import SecretsProviderFactoryFn+from pipelex.plugins.storage_provider_registry import StorageProviderFactoryFnfrom pipelex.runtime_bridge.orchestration_mode import OrchestrationModefrom pipelex.types import StrEnum@@ -115,6 +119,8 @@ class PluginRegistrar:self.model_listers: dict[str, ListModelsFn] = {}self.orchestrators: dict[OrchestrationMode, OrchestratorProtocol] = {}self.bundle_validators: dict[OrchestrationMode, BundleValidatorProtocol] = {}+ self.storage_providers: dict[str, StorageProviderFactoryFn] = {}+ self.secrets_providers: dict[str, SecretsProviderFactoryFn] = {}# Ordered list (not a type-keyed dict) because the exception types are# resolved lazily — only ``get_http_error_mappers`` invokes the providers,# so duplicate-by-type detection is deferred to resolution time too.@@ -126,6 +132,8 @@ class PluginRegistrar:self._model_lister_sources: dict[str, str] = {}self._orchestrator_sources: dict[OrchestrationMode, str] = {}self._bundle_validator_sources: dict[OrchestrationMode, str] = {}+ self._storage_provider_sources: dict[str, str] = {}+ self._secrets_provider_sources: dict[str, str] = {}self._slot_sources: dict[HubSlot, str] = {}# Reassigned per plugin by build_registrar; the floating default keeps the# menu methods safe to call outside a registration loop (e.g. a focused unit test).@@ -193,6 +201,48 @@ class PluginRegistrar:),)+ def add_storage_provider(self, *, method: str, factory: StorageProviderFactoryFn) -> None:+ """Contribute a factory for one storage backend, keyed by an open ``method`` token.++ The built-in ``StoragePlugin`` registers the ``local`` / ``in_memory`` / ``s3`` / ``gcp``+ methods; an external ``pipelex-storage-<backend>`` plugin registers its own token (e.g.+ ``"azure"``). Boot reads ``storage_config.method`` and calls the looked-up factory to+ produce the one storage provider set on the hub. ``factory`` is invoked at that boot+ apply-point, never here — so a factory may do heavy work (SDK import, a hub secrets read)+ while ``register`` stays import-light. Fail-loud on a duplicate method, naming both plugins.+ """+ self._add(+ store=self.storage_providers,+ sources=self._storage_provider_sources,+ key=method,+ value=factory,+ contribution=f"storage provider {method}",+ on_duplicate=lambda first_plugin, second_plugin: DuplicateStorageProviderError(+ method=method, first_plugin=first_plugin, second_plugin=second_plugin+ ),+ )++ def add_secrets_provider(self, *, method: str, factory: SecretsProviderFactoryFn) -> None:+ """Contribute a factory for one secrets backend, keyed by an open ``method`` token.++ The built-in ``SecretsPlugin`` registers the ``env`` method; an external+ ``pipelex-secrets-<backend>`` plugin registers its own token (e.g. ``"vault"``). Boot reads+ ``secrets_config.method`` and calls the looked-up factory to produce the one secrets provider+ set on the hub. ``factory`` is invoked at that boot apply-point, never here — so a factory may+ do heavy work (SDK import) while ``register`` stays import-light. Fail-loud on a duplicate+ method, naming both plugins.+ """+ self._add(+ store=self.secrets_providers,+ sources=self._secrets_provider_sources,+ key=method,+ value=factory,+ contribution=f"secrets provider {method}",+ on_duplicate=lambda first_plugin, second_plugin: DuplicateSecretsProviderError(+ method=method, first_plugin=first_plugin, second_plugin=second_plugin+ ),+ )+def add_http_error_mapper(self, *, exc_type_provider: HttpErrorTypeProviderFn, to_error_report: HttpErrorMapperFn) -> None:"""Contribute a mapping from a transport/runtime exception to a structured ``ErrorReport``.diff --git a/pipelex/plugins/secrets/__init__.py b/pipelex/plugins/secrets/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/pipelex/plugins/secrets/secrets_plugin.py b/pipelex/plugins/secrets/secrets_plugin.pynew file mode 100644index 000000000..82d4b5b31--- /dev/null+++ b/pipelex/plugins/secrets/secrets_plugin.py@@ -0,0 +1,26 @@+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider+from pipelex.tools.secrets.secrets_config import SecretsProviderConfig+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract+++def _make_env_secrets_provider(config: SecretsProviderConfig) -> SecretsProviderAbstract: # noqa: ARG001 - env provider reads no config; uniform SecretsProviderFactoryFn signature+ return EnvSecretsProvider()+++class SecretsPlugin:+ """Always-on built-in provider of the ``env`` secrets backend (reads secrets from env vars).++ Core-unconditional: secrets is required infra, so this plugin cannot be disabled into a boot+ with no secrets provider (see ``CORE_UNCONDITIONAL_PLUGIN_NAMES``). It registers the one built-in+ ``env`` method; ``secrets_config.method`` selects which factory boot invokes. Importing this module+ is import-light — no SDK loads at register (the env provider needs none, and an external+ ``pipelex-secrets-<backend>`` plugin defers its SDK import to its own factory).+ """++ name = "secrets"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_secrets_provider(method="env", factory=_make_env_secrets_provider)diff --git a/pipelex/plugins/secrets_provider_registry.py b/pipelex/plugins/secrets_provider_registry.pynew file mode 100644index 000000000..16f39ecdc--- /dev/null+++ b/pipelex/plugins/secrets_provider_registry.py@@ -0,0 +1,37 @@+from collections.abc import Callable++from pipelex.plugins.exceptions import UnknownSecretsMethodError+from pipelex.tools.secrets.secrets_config import SecretsProviderConfig+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract++# A plugin's factory for one secrets backend: whole secrets config in, provider out. The whole+# config (not a pre-resolved sub-config) is passed so a factory can read whatever it needs — an+# SDK-backed provider (Vault, AWS Secrets Manager) reads its own settings — at the boot apply-point,+# never at registration. The built-in ``env`` factory ignores the config entirely.+SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], SecretsProviderAbstract]+++class SecretsProviderRegistry:+ """Read view over the secrets-provider factories contributed by discovered plugins.++ Keyed by the open secrets ``method`` token (a ``str``; the built-in ``SecretsPlugin`` registers+ ``"env"``, an external plugin registers e.g. ``"vault"``). Built once at boot from the registrar's+ accumulated ``secrets_providers`` and stored on the hub; core reads ``secrets_config.method`` and+ calls the looked-up factory to produce the one provider. Mirrors ``StorageProviderRegistry``.+ """++ def __init__(self, secrets_providers: dict[str, SecretsProviderFactoryFn]):+ self._secrets_providers: dict[str, SecretsProviderFactoryFn] = dict(secrets_providers)++ def get_required(self, *, method: str) -> SecretsProviderFactoryFn:+ factory = self._secrets_providers.get(method)+ if factory is None:+ raise UnknownSecretsMethodError(method=method, registered_methods=self.methods)+ return factory++ def has(self, *, method: str) -> bool:+ return method in self._secrets_providers++ @property+ def methods(self) -> list[str]:+ return list(self._secrets_providers)diff --git a/pipelex/plugins/storage/__init__.py b/pipelex/plugins/storage/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/pipelex/plugins/storage/storage_plugin.py b/pipelex/plugins/storage/storage_plugin.pynew file mode 100644index 000000000..bd7141088--- /dev/null+++ b/pipelex/plugins/storage/storage_plugin.py@@ -0,0 +1,79 @@+from pathlib import Path++from pipelex import log+from pipelex.hub import get_secrets_provider+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.registrar import PluginRegistrar+from pipelex.tools.storage.exceptions import StorageConfigError+from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider+from pipelex.tools.storage.storage_config import StorageMethod, StorageProviderConfig+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract+++def _make_local_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:+ if config.local is None:+ msg = "local config is required when method is local"+ raise StorageConfigError(msg)+ config.local.lazy_validate()+ log.verbose(f"Using local storage at: {config.local.local_storage_path}")+ return LocalStorageProvider(root_path=Path(config.local.local_storage_path))+++def _make_in_memory_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:+ if config.in_memory is None:+ msg = "in_memory config is required when method is in_memory"+ raise StorageConfigError(msg)+ config.in_memory.lazy_validate()+ log.verbose("Using in-memory storage")+ return InMemoryStorageProvider()+++def _make_s3_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:+ if config.s3 is None:+ msg = "S3 config is required when method is s3"+ raise StorageConfigError(msg)+ config.s3.lazy_validate()+ log.verbose(f"Using S3 storage: bucket={config.s3.bucket_name}, region={config.s3.region}")+ return S3StorageProvider(+ bucket_name=config.s3.bucket_name,+ region=config.s3.region,+ signed_urls_lifespan=config.s3.signed_urls_lifespan,+ )+++def _make_gcp_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:+ if config.gcp is None:+ msg = "GCP config is required when method is gcp"+ raise StorageConfigError(msg)+ config.gcp.lazy_validate()+ log.verbose(f"Using GCP storage: bucket={config.gcp.bucket_name}, project={config.gcp.project_id}")+ return GcpStorageProvider(+ bucket_name=config.gcp.bucket_name,+ project_id=config.gcp.project_id,+ # Hub secrets read is legal here: the factory runs at the boot apply-point, after the+ # secrets provider is on the hub — never in register().+ credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),+ signed_urls_lifespan=config.gcp.signed_urls_lifespan,+ )+++class StoragePlugin:+ """Always-on built-in provider of the local / in_memory / s3 / gcp storage backends.++ Core-unconditional: storage is required infra, so this plugin cannot be disabled into a+ boot with no storage (see ``CORE_UNCONDITIONAL_PLUGIN_NAMES``). It registers one factory per+ built-in method; ``storage_config.method`` selects which one boot invokes. Importing this+ module is import-light — the s3/gcp SDKs load lazily inside their providers, not at register.+ """++ name = "storage"+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_storage_provider(method=StorageMethod.LOCAL, factory=_make_local_storage_provider)+ registrar.add_storage_provider(method=StorageMethod.IN_MEMORY, factory=_make_in_memory_storage_provider)+ registrar.add_storage_provider(method=StorageMethod.S3, factory=_make_s3_storage_provider)+ registrar.add_storage_provider(method=StorageMethod.GCP, factory=_make_gcp_storage_provider)diff --git a/pipelex/plugins/storage_provider_registry.py b/pipelex/plugins/storage_provider_registry.pynew file mode 100644index 000000000..d49150875--- /dev/null+++ b/pipelex/plugins/storage_provider_registry.py@@ -0,0 +1,36 @@+from collections.abc import Callable++from pipelex.plugins.exceptions import UnknownStorageMethodError+from pipelex.tools.storage.storage_config import StorageProviderConfig+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract++# A plugin's factory for one storage backend: whole storage config in, provider out. The whole+# config (not a pre-resolved sub-config) is passed so a factory can read whatever it needs —+# including a hub secrets read (GCP) — at the boot apply-point, never at registration.+StorageProviderFactoryFn = Callable[[StorageProviderConfig], StorageProviderAbstract]+++class StorageProviderRegistry:+ """Read view over the storage-provider factories contributed by discovered plugins.++ Keyed by the open storage ``method`` token (a ``str``; the built-ins keep the+ ``StorageMethod`` values, an external plugin registers e.g. ``"azure"``). Built once at+ boot from the registrar's accumulated ``storage_providers`` and stored on the hub; core+ reads ``storage_config.method`` and calls the looked-up factory to produce the one provider.+ """++ def __init__(self, storage_providers: dict[str, StorageProviderFactoryFn]):+ self._storage_providers: dict[str, StorageProviderFactoryFn] = dict(storage_providers)++ def get_required(self, *, method: str) -> StorageProviderFactoryFn:+ factory = self._storage_providers.get(method)+ if factory is None:+ raise UnknownStorageMethodError(method=method, registered_methods=self.methods)+ return factory++ def has(self, *, method: str) -> bool:+ return method in self._storage_providers++ @property+ def methods(self) -> list[str]:+ return list(self._storage_providers)diff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.pyindex a4cd63827..52768c88a 100644--- a/pipelex/system/configuration/configs.py+++ b/pipelex/system/configuration/configs.py@@ -13,6 +13,7 @@ from pipelex.system.configuration.config_model import ConfigModelfrom pipelex.system.configuration.config_root import ConfigRootfrom pipelex.tools.aws.aws_config import AwsConfigfrom pipelex.tools.log.log_config import LogConfig+from pipelex.tools.secrets.secrets_config import SecretsProviderConfigfrom pipelex.tools.storage.storage_config import StorageConfigfrom pipelex.types import Self, StrEnum@@ -198,6 +199,7 @@ class PipelineExecutionConfig(ConfigModel):class Pipelex(ConfigModel):storage_config: StorageConfig+ secrets_config: SecretsProviderConfiglog_config: LogConfigaws_config: AwsConfigdiff --git a/pipelex/tools/secrets/secrets_config.py b/pipelex/tools/secrets/secrets_config.pynew file mode 100644index 000000000..5f23e754e--- /dev/null+++ b/pipelex/tools/secrets/secrets_config.py@@ -0,0 +1,17 @@+from pydantic import Field++from pipelex.system.configuration.config_model import ConfigModel+++class SecretsProviderConfig(ConfigModel):+ """Provider-selection config for the secrets-provider seam.++ Mirrors ``StorageProviderConfig``'s open-token selection (S1): the built-in ``SecretsPlugin``+ registers the ``env`` method; an external ``pipelex-secrets-<backend>`` plugin registers its own+ (e.g. ``"vault"``). An unknown token is validated at registry lookup (``UnknownSecretsMethodError``+ at boot), not at parse — so a config naming an external method still loads, and only the registry+ decides what is installable. The built-in ``env`` method needs no per-method sub-config; a future+ external-provider config passthrough is a scoped follow-up (S4).+ """++ method: str = Field(strict=False)diff --git a/pipelex/tools/storage/storage_config.py b/pipelex/tools/storage/storage_config.pyindex 98ddc61f5..95ccaa62a 100644--- a/pipelex/tools/storage/storage_config.py+++ b/pipelex/tools/storage/storage_config.py@@ -171,7 +171,11 @@ class StorageGcpConfig(StorageBucketConfig):class StorageProviderConfig(ConfigModel):"""Provider-selection config shared by asset storage and payload codec storage."""- method: StorageMethod = Field(strict=False)+ # Open str token (D1): the built-ins use the StorageMethod values; an external+ # ``pipelex-storage-<backend>`` plugin registers its own (e.g. "azure"). An unknown token is+ # validated at registry lookup (UnknownStorageMethodError at boot), not at parse — so a config+ # naming an external method still loads, and only the registry decides what is installable.+ method: str = Field(strict=False)local: StorageLocalConfig | None = Nonein_memory: StorageInMemoryConfig | None = Nones3: StorageS3Config | None = None@@ -196,6 +200,10 @@ class StorageProviderConfig(ConfigModel):if not self.gcp:msg = "gcp config is required when method is gcp"raise StorageConfigError(msg)+ case _:+ # External method token (D1): no built-in sub-config is required. Whether the token+ # names an installed provider is validated at registry lookup, not here.+ passreturn self@property@@ -206,7 +214,7 @@ class StorageProviderConfig(ConfigModel):msg = "local config is required to access storage_path"raise StorageConfigError(msg)return self.local.local_storage_path- case StorageMethod.IN_MEMORY | StorageMethod.S3 | StorageMethod.GCP:+ case _:msg = f"storage_path is only available when method is local, but method is '{self.method}'"raise StorageConfigError(msg)@@ -233,6 +241,9 @@ class StorageProviderConfig(ConfigModel):msg = "gcp config is required to access uri_format"raise StorageConfigError(msg)return self.gcp.uri_format+ case _:+ msg = f"uri_format is not available for method '{self.method}' (no built-in sub-config)"+ raise StorageConfigError(msg)class StorageConfig(StorageProviderConfig):diff --git a/pipelex/tools/storage/storage_provider_factory.py b/pipelex/tools/storage/storage_provider_factory.pydeleted file mode 100644index 4fff5088f..000000000--- a/pipelex/tools/storage/storage_provider_factory.py+++ /dev/null@@ -1,64 +0,0 @@-from pathlib import Path--from pipelex import log-from pipelex.hub import get_secrets_provider-from pipelex.tools.storage.exceptions import StorageConfigError-from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider-from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider-from pipelex.tools.storage.local_storage_provider import LocalStorageProvider-from pipelex.tools.storage.s3_storage_provider import S3StorageProvider-from pipelex.tools.storage.storage_config import StorageMethod, StorageProviderConfig-from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract---def make_storage_provider_from_config(storage_provider_config: StorageProviderConfig) -> StorageProviderAbstract:- """Create a storage provider based on the provided storage configuration.-- Args:- storage_provider_config: The storage configuration specifying method and provider-specific settings.-- Returns:- A configured storage provider instance.-- Raises:- StorageConfigError: If the required provider-specific config is missing or fails its lazy_validate checks.- """- match storage_provider_config.method:- case StorageMethod.LOCAL:- if storage_provider_config.local is None:- msg = "local config is required when method is local"- raise StorageConfigError(msg)- storage_provider_config.local.lazy_validate()- log.verbose(f"Using local storage at: {storage_provider_config.local.local_storage_path}")- return LocalStorageProvider(root_path=Path(storage_provider_config.local.local_storage_path))- case StorageMethod.IN_MEMORY:- if storage_provider_config.in_memory is None:- msg = "in_memory config is required when method is in_memory"- raise StorageConfigError(msg)- storage_provider_config.in_memory.lazy_validate()- log.verbose("Using in-memory storage")- return InMemoryStorageProvider()- case StorageMethod.S3:- if storage_provider_config.s3 is None:- msg = "S3 config is required when method is s3"- raise StorageConfigError(msg)-- storage_provider_config.s3.lazy_validate()- log.verbose(f"Using S3 storage: bucket={storage_provider_config.s3.bucket_name}, region={storage_provider_config.s3.region}")- return S3StorageProvider(- bucket_name=storage_provider_config.s3.bucket_name,- region=storage_provider_config.s3.region,- signed_urls_lifespan=storage_provider_config.s3.signed_urls_lifespan,- )- case StorageMethod.GCP:- if storage_provider_config.gcp is None:- msg = "GCP config is required when method is gcp"- raise StorageConfigError(msg)- storage_provider_config.gcp.lazy_validate()- log.verbose(f"Using GCP storage: bucket={storage_provider_config.gcp.bucket_name}, project={storage_provider_config.gcp.project_id}")- return GcpStorageProvider(- bucket_name=storage_provider_config.gcp.bucket_name,- project_id=storage_provider_config.gcp.project_id,- credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),- signed_urls_lifespan=storage_provider_config.gcp.signed_urls_lifespan,- )diff --git a/tests/integration/pipelex/fixtures/generator_fixtures.py b/tests/integration/pipelex/fixtures/generator_fixtures.pyindex 5f6d53d27..352a864d5 100644--- a/tests/integration/pipelex/fixtures/generator_fixtures.py+++ b/tests/integration/pipelex/fixtures/generator_fixtures.py@@ -6,8 +6,8 @@ from pipelex.cogt.content_generation.content_generator import ContentGeneratorfrom pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocolfrom pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactoryfrom pipelex.config import get_config+from pipelex.hub import get_storage_provider_registryfrom pipelex.tools.storage.storage_config import StorageMethod-from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_configS3_TEST_BUCKET = "pipelex-storage-test"GCP_TEST_BUCKET = "pipelex-storage-test"@@ -50,8 +50,11 @@ def generated_content_factory(tmp_path: Path) -> GeneratedContentFactory:patched_provider_config = storage_provider_config.model_copy(update={"local": patched_local_config})case StorageMethod.IN_MEMORY:patched_provider_config = storage_provider_config+ case _:+ # External method token: no built-in sub-config to patch; select it as configured.+ patched_provider_config = storage_provider_config- storage_provider = make_storage_provider_from_config(patched_provider_config)+ storage_provider = get_storage_provider_registry().get_required(method=patched_provider_config.method)(patched_provider_config)return GeneratedContentFactory(storage_provider=storage_provider)diff --git a/tests/integration/pipelex/plugins/test_secrets_external_plugin.py b/tests/integration/pipelex/plugins/test_secrets_external_plugin.pynew file mode 100644index 000000000..aca8949a2--- /dev/null+++ b/tests/integration/pipelex/plugins/test_secrets_external_plugin.py@@ -0,0 +1,158 @@+"""End-to-end secrets-provider selection at boot: default env, explicit override, external discovery,+and the storage↔secrets ordering guard.++The built-in ``env`` method is the default and is selected onto the hub by an ordinary boot; an+explicit ``setup(secrets_provider=...)`` still wins ahead of registry selection; a fake external+method token discovered through a ``pipelex.plugins`` entry point is selectable via+``secrets_config.method``. The final test is the ordering guard: with a non-env secrets method AND+``gcp`` storage, storage's gcp factory reads the config-selected secrets provider from the hub —+proving secrets is resolved and set on the hub before storage selection runs.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING++import pytest+from typing_extensions import override++from pipelex.hub import get_secrets_provider, get_secrets_provider_registry, get_storage_provider+from pipelex.pipelex import Pipelex+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.system.runtime import IntegrationMode, runtime_manager+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract+from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider++if TYPE_CHECKING:+ from collections.abc import Generator++ from pytest_mock import MockerFixture++ from pipelex.plugins.registrar import PluginRegistrar+ from pipelex.tools.secrets.secrets_config import SecretsProviderConfig++EXTERNAL_SECRETS_METHOD = "test_secret"+EXTERNAL_PLUGIN_NAME = "test_secrets_ext"+CANNED_CREDENTIALS_PATH = "/secrets/gcp-creds-from-external.json"+++class _RecordingSecretsProvider(SecretsProviderAbstract):+ """An out-of-tree provider — its distinct type proves config-selection, and it records which+ secrets were asked of it so the ordering guard can prove storage's gcp arm read *this* provider.+ """++ def __init__(self) -> None:+ self.requested_secret_ids: list[str] = []++ @override+ def get_required_secret(self, secret_id: str) -> str:+ self.requested_secret_ids.append(secret_id)+ return CANNED_CREDENTIALS_PATH++ @override+ def get_optional_secret(self, secret_id: str) -> str | None:+ return None++ @override+ def get_required_secret_specific_version(self, secret_id: str, *, version_id: str) -> str:+ raise NotImplementedError++ @override+ def get_optional_secret_specific_version(self, secret_id: str, *, version_id: str) -> str | None:+ raise NotImplementedError++ @override+ def set_secret_as_env_var(self, secret_id: str, *, version_id: str = "latest") -> None:+ pass+++def _make_recording_secrets_provider(_config: SecretsProviderConfig) -> SecretsProviderAbstract:+ return _RecordingSecretsProvider()+++class _FakeSecretsPlugin:+ """An external plugin contributing one secrets method under the ``test_secret`` token."""++ name = EXTERNAL_PLUGIN_NAME+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_secrets_provider(method=EXTERNAL_SECRETS_METHOD, factory=_make_recording_secrets_provider)++[email protected](autouse=True)+def reset_pipelex_config_fixture() -> Generator[None, None, None]:+ """Override the global module fixture: this module boots per test and tears down."""+ Pipelex.teardown_if_needed()+ yield+ Pipelex.teardown_if_needed()+++def _test_integration_mode() -> IntegrationMode:+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST+++def _patch_external_secrets_plugin(mocker: MockerFixture) -> None:+ # The entry point resolves to the plugin class (a zero-arg factory); discovery instantiates it.+ fake_entry_point = SimpleNamespace(name=EXTERNAL_PLUGIN_NAME, load=lambda: _FakeSecretsPlugin)+ mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[fake_entry_point])+++class TestSecretsExternalPlugin:+ def test_default_config_yields_the_env_provider_on_the_hub(self) -> None:+ """An ordinary boot selects the built-in ``env`` method: an EnvSecretsProvider lands on the hub."""+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)++ assert get_secrets_provider_registry().has(method="env")+ assert isinstance(get_secrets_provider(), EnvSecretsProvider)++ def test_explicit_secrets_provider_wins_over_config_selection(self) -> None:+ """An explicit setup() param is the exact provider on the hub, ahead of registry selection."""+ explicit = EnvSecretsProvider()+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, secrets_provider=explicit)++ assert get_secrets_provider() is explicit++ def test_external_secrets_method_is_discovered_and_selected_onto_the_hub(self, mocker: MockerFixture) -> None:+ """A fake entry point + a config naming its token boots with that external provider on the hub."""+ _patch_external_secrets_plugin(mocker)++ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ config_overrides={"pipelex": {"secrets_config": {"method": EXTERNAL_SECRETS_METHOD}}},+ )++ assert get_secrets_provider_registry().has(method=EXTERNAL_SECRETS_METHOD)+ assert isinstance(get_secrets_provider(), _RecordingSecretsProvider)++ def test_config_selected_secrets_provider_is_read_by_storage_gcp_arm_at_boot(self, mocker: MockerFixture) -> None:+ """Ordering guard: with a non-env secrets method AND gcp storage, storage's gcp factory reads the+ config-selected external secrets provider from the hub. Secrets must be resolved and set on the hub+ before storage selection — a regression that reordered them would leave the gcp arm without its+ provider (or reading a stale env default), and this recorded request would not be the external one.+ """+ _patch_external_secrets_plugin(mocker)++ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ config_overrides={+ "pipelex": {+ "secrets_config": {"method": EXTERNAL_SECRETS_METHOD},+ "storage_config": {"method": "gcp", "gcp": {"bucket_name": "test-bucket", "project_id": "test-project"}},+ }+ },+ )++ storage_provider = get_storage_provider()+ assert isinstance(storage_provider, GcpStorageProvider)+ secrets_provider = get_secrets_provider()+ assert isinstance(secrets_provider, _RecordingSecretsProvider)+ # GCP_CREDENTIALS_FILE_PATH is requested only by storage's gcp factory, so its presence proves the+ # gcp arm read *this* config-selected external provider from the hub at the boot apply-point. (The+ # recorded list also carries the telemetry factory's POSTHOG_* reads — telemetry resolves secrets+ # even earlier — which is why this is a membership check, not an exact-match.)+ assert "GCP_CREDENTIALS_FILE_PATH" in secrets_provider.requested_secret_idsdiff --git a/tests/integration/pipelex/plugins/test_storage_external_plugin.py b/tests/integration/pipelex/plugins/test_storage_external_plugin.pynew file mode 100644index 000000000..94fd2c36b--- /dev/null+++ b/tests/integration/pipelex/plugins/test_storage_external_plugin.py@@ -0,0 +1,98 @@+"""End-to-end: an out-of-tree storage plugin discovered through a ``pipelex.plugins`` entry point is+selectable via ``storage_config.method`` and lands on the hub.++This exercises the whole discovery → selection → hub chain the built-in providers ride, with a fake+external method token: a fake entry point loads a plugin registering ``method="test_mem"``, the boot+config selects that token, and the resolved provider is the one set on the hub.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING++import pytest+from typing_extensions import override++from pipelex.hub import get_storage_provider, get_storage_provider_registry+from pipelex.pipelex import Pipelex+from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.system.runtime import IntegrationMode, runtime_manager+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract, StoredData++if TYPE_CHECKING:+ from collections.abc import Generator++ from pytest_mock import MockerFixture++ from pipelex.plugins.registrar import PluginRegistrar+ from pipelex.tools.storage.storage_config import StorageProviderConfig++EXTERNAL_STORAGE_METHOD = "test_mem"+EXTERNAL_PLUGIN_NAME = "test_storage_ext"+++class _FakeExternalStorageProvider(StorageProviderAbstract):+ """A minimal out-of-tree provider — only its distinct type matters to the selection assertion."""++ @override+ async def _load_with_metadata(self, key: str) -> StoredData:+ raise NotImplementedError++ @override+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:+ raise NotImplementedError++ @override+ async def public_url(self, uri: str) -> str | None:+ return None+++def _make_fake_external_provider(_config: StorageProviderConfig) -> StorageProviderAbstract:+ return _FakeExternalStorageProvider()+++class _FakeStoragePlugin:+ """An external plugin contributing one storage method under the ``test_mem`` token."""++ name = EXTERNAL_PLUGIN_NAME+ targets_api = PLUGIN_API_VERSION++ def register(self, registrar: PluginRegistrar) -> None:+ registrar.add_storage_provider(method=EXTERNAL_STORAGE_METHOD, factory=_make_fake_external_provider)++[email protected](autouse=True)+def reset_pipelex_config_fixture() -> Generator[None, None, None]:+ """Override the global module fixture: this module boots per test and tears down."""+ Pipelex.teardown_if_needed()+ yield+ Pipelex.teardown_if_needed()+++def _test_integration_mode() -> IntegrationMode:+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST+++class TestStorageExternalPlugin:+ def test_explicit_storage_provider_wins_over_config_selection(self) -> None:+ """An explicit setup() param is the exact provider on the hub, ahead of registry selection."""+ explicit = _FakeExternalStorageProvider()+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, storage_provider=explicit)++ assert get_storage_provider() is explicit++ def test_external_storage_method_is_discovered_and_selected_onto_the_hub(self, mocker: MockerFixture) -> None:+ """A fake entry point + a config naming its token boots with that external provider on the hub."""+ # The entry point resolves to the plugin class (a zero-arg factory); discovery instantiates it.+ fake_entry_point = SimpleNamespace(name=EXTERNAL_PLUGIN_NAME, load=lambda: _FakeStoragePlugin)+ mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[fake_entry_point])++ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ config_overrides={"pipelex": {"storage_config": {"method": EXTERNAL_STORAGE_METHOD}}},+ )++ assert get_storage_provider_registry().has(method=EXTERNAL_STORAGE_METHOD)+ assert isinstance(get_storage_provider(), _FakeExternalStorageProvider)diff --git a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.pyindex 2d929bcb7..44fd94a2d 100644--- a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py+++ b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py@@ -18,6 +18,8 @@ from pipelex.hub import get_content_generator, get_pipe_routerfrom pipelex.pipelex import Pipelexfrom pipelex.plugins.registrar import HubSlot, PluginRegistrarfrom pipelex.system.runtime import IntegrationMode, runtime_manager+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProviderif TYPE_CHECKING:from pipelex.system.configuration.configs import PipelexConfig@@ -36,7 +38,13 @@ def _test_integration_mode() -> IntegrationMode:def _fake_registrar(mocker: MockerFixture) -> PluginRegistrar:- """A registrar with empty registries (boot stores them but doesn't consume them during setup)."""+ """A registrar with empty registries (boot stores them but doesn't consume them during setup).++ Storage and secrets are the exceptions: boot *does* select each provider from the registrar during+ setup, so every boot below passes an explicit ``storage_provider`` and ``secrets_provider`` — each+ wins ahead of the (empty) registry — keeping this suite focused on hub slots rather than+ storage/secrets selection.+ """registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(temporal=SimpleNamespace(is_enabled=False))))mocker.patch("pipelex.pipelex.build_registrar", return_value=registrar)return registrar@@ -49,7 +57,12 @@ class TestHubSlotInjectionPrecedence:registrar = _fake_registrar(mocker)registrar.slot_claims[HubSlot.CONTENT_GENERATOR] = lambda: claimed- Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)+ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ storage_provider=InMemoryStorageProvider(),+ secrets_provider=EnvSecretsProvider(),+ )assert get_content_generator() is claimed@@ -60,7 +73,13 @@ class TestHubSlotInjectionPrecedence:registrar = _fake_registrar(mocker)registrar.slot_claims[HubSlot.CONTENT_GENERATOR] = lambda: claimed- Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, content_generator=explicit) # pyright: ignore[reportArgumentType]+ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ content_generator=explicit, # pyright: ignore[reportArgumentType]+ storage_provider=InMemoryStorageProvider(),+ secrets_provider=EnvSecretsProvider(),+ )assert get_content_generator() is explicit@@ -73,7 +92,13 @@ class TestHubSlotInjectionPrecedence:# Claim PIPE_RUN too so boot does not build a default PipeRun from the sentinel router.registrar.slot_claims[HubSlot.PIPE_RUN] = lambda: mocker.sentinel.claimed_run- Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, pipe_router=explicit_router) # pyright: ignore[reportArgumentType]+ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ pipe_router=explicit_router, # pyright: ignore[reportArgumentType]+ storage_provider=InMemoryStorageProvider(),+ secrets_provider=EnvSecretsProvider(),+ )assert get_pipe_router() is explicit_router@@ -89,7 +114,12 @@ class TestHubSlotInjectionPrecedence:])- Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)+ Pipelex.make(+ integration_mode=_test_integration_mode(),+ needs_inference=False,+ storage_provider=InMemoryStorageProvider(),+ secrets_provider=EnvSecretsProvider(),+ )Pipelex.teardown_if_needed()assert order == ["third", "second", "first"]diff --git a/tests/integration/pipelex/tools/storage/test_storage_config.py b/tests/integration/pipelex/tools/storage/test_storage_config.pyindex 1a2e58096..e76b666fb 100644--- a/tests/integration/pipelex/tools/storage/test_storage_config.py+++ b/tests/integration/pipelex/tools/storage/test_storage_config.py@@ -41,3 +41,6 @@ class TestStorageConfigIntegration:case StorageMethod.GCP:assert storage_config.gcp is not Noneassert storage_config.uri_format == storage_config.gcp.uri_format+ case _:+ # External method token (D1): no built-in sub-config to compare against.+ passdiff --git a/tests/unit/pipelex/plugins/test_import_light_boot.py b/tests/unit/pipelex/plugins/test_import_light_boot.pyindex ce4c44fe1..38e70ded3 100644--- a/tests/unit/pipelex/plugins/test_import_light_boot.py+++ b/tests/unit/pipelex/plugins/test_import_light_boot.py@@ -18,6 +18,7 @@ _GUARD_SCRIPT = textwrap.dedent("anthropic","mistralai","google.genai",+ "google.cloud.storage","boto3","aioboto3","fal_client",@@ -58,6 +59,10 @@ _GUARD_SCRIPT = textwrap.dedent(assert registrar.inference_backends, "expected the built-in LLM backends to be registered"assert registrar.model_listers, "expected the built-in model listers to be registered import-light"assert registrar.orchestrators, "expected the built-in orchestrators to be registered"+ # The built-in StoragePlugin registers all four methods without importing aioboto3 /+ # google-cloud-storage: the s3/gcp SDK guards live inside the providers' I/O methods, so+ # registration stays import-light even though those SDKs are BLOCKED above.+ assert registrar.storage_providers, "expected the built-in storage providers to be registered import-light"print("import-light OK")""")diff --git a/tests/unit/pipelex/plugins/test_plugin_discovery.py b/tests/unit/pipelex/plugins/test_plugin_discovery.pyindex a6884e1a4..5dc04a1ce 100644--- a/tests/unit/pipelex/plugins/test_plugin_discovery.py+++ b/tests/unit/pipelex/plugins/test_plugin_discovery.py@@ -191,11 +191,12 @@ class TestPluginDiscovery:disabled_discovery = next(discovery for discovery in registrar.discoveries if discovery.name == "optional")assert disabled_discovery.status == PluginStatus.DISABLED- def test_disabling_core_unconditional_plugin_raises(self) -> None:- """Denylisting a plugin core requires unconditionally (e.g. openai) is a startup error."""+ @pytest.mark.parametrize("plugin_name", ["direct", "storage", "secrets", "openai"])+ def test_disabling_core_unconditional_plugin_raises(self, plugin_name: str) -> None:+ """Denylisting any plugin core requires unconditionally (storage/secrets included) is a startup error."""with pytest.raises(CoreUnconditionalPluginDisabledError) as exc_info:- build_registrar(config=_fake_config(["openai"]))- assert "openai" in str(exc_info.value)+ build_registrar(config=_fake_config([plugin_name]))+ assert plugin_name in str(exc_info.value)def test_discoveries_describe_builtins(self) -> None:"""build_registrar records each built-in with origin/status/contributions for `plugins list`."""diff --git a/tests/unit/pipelex/plugins/test_secrets_plugin.py b/tests/unit/pipelex/plugins/test_secrets_plugin.pynew file mode 100644index 000000000..cd4c1fa2e--- /dev/null+++ b/tests/unit/pipelex/plugins/test_secrets_plugin.py@@ -0,0 +1,49 @@+"""The built-in SecretsPlugin: registers the one ``env`` factory, which constructs an EnvSecretsProvider.++Secrets is a single-impl seam today — the ``env`` provider reads secrets from env vars and needs no+config, so its factory ignores the config it is handed. Registration is import-light (no SDK); an+out-of-tree ``pipelex-secrets-<backend>`` plugin would defer its SDK import into its own factory.+Mirrors ``test_storage_plugin`` (kept lean: one built-in method, no optional-dep matrix).+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar+from pipelex.plugins.secrets.secrets_plugin import SecretsPlugin+from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider+from pipelex.tools.secrets.secrets_config import SecretsProviderConfig++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig+++def _build_secrets_registry() -> SecretsProviderRegistry:+ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+ SecretsPlugin().register(registrar)+ return SecretsProviderRegistry(registrar.secrets_providers)+++class TestSecretsPlugin:+ def test_registers_the_env_factory(self) -> None:+ """SecretsPlugin is a named, API-versioned builtin registering exactly the built-in ``env`` method."""+ plugin = SecretsPlugin()+ assert plugin.name == "secrets"+ assert plugin.targets_api == PLUGIN_API_VERSION++ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+ discovery = registrar.begin_plugin(name="secrets", origin=PluginOrigin.BUILTIN, targets_api=PLUGIN_API_VERSION)+ plugin.register(registrar)++ assert set(registrar.secrets_providers) == {"env"}+ assert "secrets provider env" in discovery.contributions++ def test_env_factory_constructs_an_env_secrets_provider(self) -> None:+ """Selecting ``env`` through the registry constructs an EnvSecretsProvider (the factory ignores its config)."""+ registry = _build_secrets_registry()+ provider = registry.get_required(method="env")(SecretsProviderConfig(method="env"))+ assert isinstance(provider, EnvSecretsProvider)diff --git a/tests/unit/pipelex/plugins/test_secrets_provider_registry.py b/tests/unit/pipelex/plugins/test_secrets_provider_registry.pynew file mode 100644index 000000000..e0af20edc--- /dev/null+++ b/tests/unit/pipelex/plugins/test_secrets_provider_registry.py@@ -0,0 +1,96 @@+"""The secrets-provider seam: the registrar collects per-method provider factories into a registry.++Pins the seam independent of the real built-in ``env`` provider: a factory registered for a method is+the one the registry hands back; a miss raises ``UnknownSecretsMethodError`` listing the registered+methods; a second registration for the same method fails loud naming both plugins; an empty registry+misses every method. Mirrors ``test_storage_provider_registry``.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.exceptions import DuplicateSecretsProviderError, UnknownSecretsMethodError+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar+from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig+ from pipelex.tools.secrets.secrets_config import SecretsProviderConfig+ from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract+++def _make_registrar() -> PluginRegistrar:+ return PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+++def _fake_factory(_config: SecretsProviderConfig) -> SecretsProviderAbstract:+ """Stand-in factory: identity is all the registry tests assert (never actually invoked here)."""+ return EnvSecretsProvider()+++class TestSecretsProviderRegistry:+ def test_registered_factory_is_retrievable_by_method(self) -> None:+ """A factory registered for a method is the exact callable the built registry returns for it."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)++ registry = SecretsProviderRegistry(registrar.secrets_providers)+ assert registry.get_required(method="vault") is _fake_factory+ assert registry.has(method="vault")+ assert registry.methods == ["vault"]++ def test_contribution_recorded_on_the_active_plugin(self) -> None:+ """Registering a provider records a ``secrets provider <method>`` contribution on the plugin's discovery."""+ registrar = _make_registrar()+ discovery = registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ registrar.add_secrets_provider(method="vault", factory=_fake_factory)++ assert "secrets provider vault" in discovery.contributions++ def test_get_required_miss_raises_listing_registered_methods(self) -> None:+ """A miss names the requested method and lists the registered ones (the boot-time actionable error)."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_secrets_provider(method="env", factory=_fake_factory)+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)+ registry = SecretsProviderRegistry(registrar.secrets_providers)++ with pytest.raises(UnknownSecretsMethodError) as exc_info:+ registry.get_required(method="aws")++ assert exc_info.value.method == "aws"+ assert set(exc_info.value.registered_methods) == {"env", "vault"}+ message = str(exc_info.value)+ assert "env" in message+ assert "vault" in message++ def test_empty_registry_misses_every_method(self) -> None:+ """A registry with no factories misses every method — soft via has, loud via get_required."""+ registry = SecretsProviderRegistry({})++ assert not registry.has(method="env")+ assert registry.methods == []+ with pytest.raises(UnknownSecretsMethodError):+ registry.get_required(method="env")++ def test_duplicate_method_fails_loud_naming_both_plugins(self) -> None:+ """Two plugins registering a provider for the same method is a fail-loud conflict naming both."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ with pytest.raises(DuplicateSecretsProviderError) as exc_info:+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)++ assert exc_info.value.first_plugin == "alpha"+ assert exc_info.value.second_plugin == "beta"+ assert exc_info.value.method == "vault"diff --git a/tests/unit/pipelex/plugins/test_storage_plugin.py b/tests/unit/pipelex/plugins/test_storage_plugin.pynew file mode 100644index 000000000..c25eb8129--- /dev/null+++ b/tests/unit/pipelex/plugins/test_storage_plugin.py@@ -0,0 +1,115 @@+"""The built-in StoragePlugin: registers a factory for every built-in method, and each factory+constructs the matching provider at the boot apply-point.++The construction is deferred — selecting ``s3``/``gcp`` builds the provider without importing its+optional SDK (the SDK guard lives inside the provider's I/O methods, not its ``__init__``), and the+``gcp`` factory reads its credentials from the hub secrets provider at that apply-point. Registration+itself never touches an SDK; the import-light guarantee is pinned by ``test_import_light_boot``.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, Any, Callable, cast++import pytest++from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar+from pipelex.plugins.storage.storage_plugin import StoragePlugin+from pipelex.plugins.storage_provider_registry import StorageProviderRegistry+from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider+from pipelex.tools.storage.storage_config import StorageMethod, StorageProviderConfig+from tests.unit.pipelex.tools.storage.test_storage_provider_config import (+ make_gcp_config,+ make_in_memory_config,+ make_local_config,+ make_s3_config,+)++if TYPE_CHECKING:+ from pytest_mock import MockerFixture++ from pipelex.system.configuration.configs import PipelexConfig+ from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract+++class _FakeSecretsProvider:+ """Records which secrets the gcp factory asks for and returns a canned credentials path."""++ def __init__(self, credentials_path: str) -> None:+ self.credentials_path = credentials_path+ self.requested_secret_ids: list[str] = []++ def get_required_secret(self, *, secret_id: str) -> str:+ self.requested_secret_ids.append(secret_id)+ return self.credentials_path+++def _build_storage_registry() -> StorageProviderRegistry:+ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+ StoragePlugin().register(registrar)+ return StorageProviderRegistry(registrar.storage_providers)+++class TestStoragePlugin:+ def test_registers_a_factory_for_every_builtin_method(self) -> None:+ """StoragePlugin is a named, API-versioned builtin registering one factory per StorageMethod value."""+ plugin = StoragePlugin()+ assert plugin.name == "storage"+ assert plugin.targets_api == PLUGIN_API_VERSION++ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+ discovery = registrar.begin_plugin(name="storage", origin=PluginOrigin.BUILTIN, targets_api=PLUGIN_API_VERSION)+ plugin.register(registrar)++ # StrEnum keys compare equal to their plain-str form, so boot's plain-str config.method resolves them.+ assert set(registrar.storage_providers) == {"local", "in_memory", "s3", "gcp"}+ assert set(registrar.storage_providers) == set(StorageMethod)+ for method in StorageMethod:+ assert f"storage provider {method}" in discovery.contributions++ @pytest.mark.parametrize(+ ("method", "field_name", "sub_config_factory", "expected_type"),+ [+ pytest.param(StorageMethod.LOCAL, "local", make_local_config, LocalStorageProvider, id="local"),+ pytest.param(StorageMethod.IN_MEMORY, "in_memory", make_in_memory_config, InMemoryStorageProvider, id="in-memory"),+ pytest.param(StorageMethod.S3, "s3", make_s3_config, S3StorageProvider, id="s3"),+ ],+ )+ def test_factory_constructs_the_matching_provider(+ self,+ method: StorageMethod,+ field_name: str,+ sub_config_factory: Callable[[], Any],+ expected_type: type[StorageProviderAbstract],+ ) -> None:+ """Selecting a method through the registry constructs its provider (s3 needs no aioboto3 to build)."""+ registry = _build_storage_registry()+ kwargs: dict[str, Any] = {"method": method, field_name: sub_config_factory()}+ provider = registry.get_required(method=method)(StorageProviderConfig(**kwargs))+ assert isinstance(provider, expected_type)++ def test_gcp_factory_reads_credentials_from_the_hub_secrets_provider(self, mocker: MockerFixture) -> None:+ """The gcp factory resolves GCP_CREDENTIALS_FILE_PATH from the hub secrets provider and passes it through."""+ fake_secrets = _FakeSecretsProvider(credentials_path="/secrets/gcp-creds.json")+ mocker.patch("pipelex.plugins.storage.storage_plugin.get_secrets_provider", return_value=fake_secrets)+ # Spy the constructor (wraps → the real provider is still built and returned) to assert the+ # resolved secret value flows through as credentials_file_path, not just that it was requested.+ gcp_ctor = mocker.patch("pipelex.plugins.storage.storage_plugin.GcpStorageProvider", wraps=GcpStorageProvider)++ gcp_config = make_gcp_config()+ registry = _build_storage_registry()+ provider = registry.get_required(method=StorageMethod.GCP)(StorageProviderConfig(method=StorageMethod.GCP, gcp=gcp_config))++ assert isinstance(provider, GcpStorageProvider)+ assert fake_secrets.requested_secret_ids == ["GCP_CREDENTIALS_FILE_PATH"]+ gcp_ctor.assert_called_once_with(+ bucket_name=gcp_config.bucket_name,+ project_id=gcp_config.project_id,+ credentials_file_path=fake_secrets.credentials_path,+ signed_urls_lifespan=gcp_config.signed_urls_lifespan,+ )diff --git a/tests/unit/pipelex/plugins/test_storage_provider_registry.py b/tests/unit/pipelex/plugins/test_storage_provider_registry.pynew file mode 100644index 000000000..2139967b3--- /dev/null+++ b/tests/unit/pipelex/plugins/test_storage_provider_registry.py@@ -0,0 +1,96 @@+"""The storage-provider seam: the registrar collects per-method provider factories into a registry.++Pins the seam independent of the real built-in providers: a factory registered for a method is the+one the registry hands back; a miss raises ``UnknownStorageMethodError`` listing the registered+methods; a second registration for the same method fails loud naming both plugins; an empty registry+misses every method.+"""++from __future__ import annotations++from types import SimpleNamespace+from typing import TYPE_CHECKING, cast++import pytest++from pipelex.plugins.contract import PLUGIN_API_VERSION+from pipelex.plugins.exceptions import DuplicateStorageProviderError, UnknownStorageMethodError+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar+from pipelex.plugins.storage_provider_registry import StorageProviderRegistry+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider++if TYPE_CHECKING:+ from pipelex.system.configuration.configs import PipelexConfig+ from pipelex.tools.storage.storage_config import StorageProviderConfig+ from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract+++def _make_registrar() -> PluginRegistrar:+ return PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))+++def _fake_factory(_config: StorageProviderConfig) -> StorageProviderAbstract:+ """Stand-in factory: identity is all the registry tests assert (never actually invoked here)."""+ return InMemoryStorageProvider()+++class TestStorageProviderRegistry:+ def test_registered_factory_is_retrievable_by_method(self) -> None:+ """A factory registered for a method is the exact callable the built registry returns for it."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_storage_provider(method="azure", factory=_fake_factory)++ registry = StorageProviderRegistry(registrar.storage_providers)+ assert registry.get_required(method="azure") is _fake_factory+ assert registry.has(method="azure")+ assert registry.methods == ["azure"]++ def test_contribution_recorded_on_the_active_plugin(self) -> None:+ """Registering a provider records a ``storage provider <method>`` contribution on the plugin's discovery."""+ registrar = _make_registrar()+ discovery = registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ registrar.add_storage_provider(method="azure", factory=_fake_factory)++ assert "storage provider azure" in discovery.contributions++ def test_get_required_miss_raises_listing_registered_methods(self) -> None:+ """A miss names the requested method and lists the registered ones (the boot-time actionable error)."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_storage_provider(method="local", factory=_fake_factory)+ registrar.add_storage_provider(method="s3", factory=_fake_factory)+ registry = StorageProviderRegistry(registrar.storage_providers)++ with pytest.raises(UnknownStorageMethodError) as exc_info:+ registry.get_required(method="azure")++ assert exc_info.value.method == "azure"+ assert set(exc_info.value.registered_methods) == {"local", "s3"}+ message = str(exc_info.value)+ assert "local" in message+ assert "s3" in message++ def test_empty_registry_misses_every_method(self) -> None:+ """A registry with no factories misses every method — soft via has, loud via get_required."""+ registry = StorageProviderRegistry({})++ assert not registry.has(method="local")+ assert registry.methods == []+ with pytest.raises(UnknownStorageMethodError):+ registry.get_required(method="local")++ def test_duplicate_method_fails_loud_naming_both_plugins(self) -> None:+ """Two plugins registering a provider for the same method is a fail-loud conflict naming both."""+ registrar = _make_registrar()+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)+ registrar.add_storage_provider(method="azure", factory=_fake_factory)+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)++ with pytest.raises(DuplicateStorageProviderError) as exc_info:+ registrar.add_storage_provider(method="azure", factory=_fake_factory)++ assert exc_info.value.first_plugin == "alpha"+ assert exc_info.value.second_plugin == "beta"+ assert exc_info.value.method == "azure"diff --git a/tests/unit/pipelex/tools/storage/test_storage_provider_config.py b/tests/unit/pipelex/tools/storage/test_storage_provider_config.pyindex 37fda6344..bb52dee33 100644--- a/tests/unit/pipelex/tools/storage/test_storage_provider_config.py+++ b/tests/unit/pipelex/tools/storage/test_storage_provider_config.py@@ -88,11 +88,16 @@ class TestStorageProviderConfig:StorageProviderConfig(method=method)assert expected_message in str(exc_info.value)- def test_method_string_coercion_despite_strict_mode(self):- """The method field is Field(strict=False), so a plain string coerces to StorageMethod under strict config."""+ def test_method_is_open_str_token(self):+ """D1: method is an open str token stored verbatim. A built-in name is a plain string (equal to its+ StorageMethod value), and an unknown/external name is accepted at parse — its installability is+ validated later at registry lookup, not at parse — so a config naming an out-of-tree provider loads.+ """provider_config = StorageProviderConfig.model_validate({"method": "local", "local": make_local_config()})- assert isinstance(provider_config.method, StorageMethod)+ assert provider_config.method == "local"assert provider_config.method == StorageMethod.LOCAL+ external_config = StorageProviderConfig.model_validate({"method": "azure"})+ assert external_config.method == "azure"@pytest.mark.parametrize(("method", "field_name", "sub_config_factory", "expected_uri_format"),diff --git a/tests/unit/pipelex/tools/storage/test_storage_provider_factory.py b/tests/unit/pipelex/tools/storage/test_storage_provider_factory.pydeleted file mode 100644index 051abc73f..000000000--- a/tests/unit/pipelex/tools/storage/test_storage_provider_factory.py+++ /dev/null@@ -1,52 +0,0 @@-from pathlib import Path--import pytest--from pipelex.tools.storage.exceptions import StorageConfigError-from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider-from pipelex.tools.storage.local_storage_provider import LocalStorageProvider-from pipelex.tools.storage.storage_config import (- StorageInMemoryConfig,- StorageLocalConfig,- StorageMethod,- StorageProviderConfig,-)-from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_config---class TestStorageProviderFactory:- def test_local_method_returns_local_provider(self, tmp_path: Path):- """A valid local config yields a LocalStorageProvider."""- provider_config = StorageProviderConfig(- method=StorageMethod.LOCAL,- local=StorageLocalConfig(uri_format="assets/{hash}", local_storage_path=str(tmp_path / "store")),- )- provider = make_storage_provider_from_config(provider_config)- assert isinstance(provider, LocalStorageProvider)-- def test_in_memory_method_returns_in_memory_provider(self):- """A valid in-memory config yields an InMemoryStorageProvider."""- provider_config = StorageProviderConfig(- method=StorageMethod.IN_MEMORY,- in_memory=StorageInMemoryConfig(uri_format="assets/{hash}"),- )- provider = make_storage_provider_from_config(provider_config)- assert isinstance(provider, InMemoryStorageProvider)-- def test_local_config_is_validated_before_provider_construction(self, tmp_path: Path):- """A local uri_format without a {hash} placeholder is rejected when the provider is built."""- provider_config = StorageProviderConfig(- method=StorageMethod.LOCAL,- local=StorageLocalConfig(uri_format="assets/constant", local_storage_path=str(tmp_path / "store")),- )- with pytest.raises(StorageConfigError, match="uri_format must contain a"):- make_storage_provider_from_config(provider_config)-- def test_in_memory_config_is_validated_before_provider_construction(self):- """An in-memory uri_format without a {hash} placeholder is rejected when the provider is built."""- provider_config = StorageProviderConfig(- method=StorageMethod.IN_MEMORY,- in_memory=StorageInMemoryConfig(uri_format="assets/constant"),- )- with pytest.raises(StorageConfigError, match="uri_format must contain a"):- make_storage_provider_from_config(provider_config)diff --git a/wip/plugins/README.md b/wip/plugins/README.mdnew file mode 100644index 000000000..19e8654d2--- /dev/null+++ b/wip/plugins/README.md@@ -0,0 +1,34 @@+# Provider plugins track — storage & secrets++Two features that promote existing dependency-injection seams to formal `pipelex.plugins` entry-point plugins, so third parties can ship `pipelex-storage-azure` / `pipelex-secrets-vault` packages selected at deploy time.++- **[storage-provider-plugin.md](storage-provider-plugin.md)** — the storage provider seam (local / in_memory / s3 / gcp today). **Land this first** — it has the real multi-implementation need and exercises the new pattern end to end.+- **[secrets-provider-plugin.md](secrets-provider-plugin.md)** — the secrets provider seam (env only today). Lands second, mirroring the storage pattern; validates that the mechanism generalizes to a second seam.++Both were selected after a full audit of the boot-time DI seams (see the classification below). They are the only two seams that pass all three plugin criteria; everything else is either already a plugin, a boot-orchestrator-owned hub slot, or a population/config point that should stay one.++## The shared new pattern these introduce++Neither seam maps onto an existing plugin pattern. The current keyed registries (inference backends, model listers, orchestrators, bundle validators) are selected **per-request/per-model** by a wire or model field. The hub slots (content generator, pipe router, pipe run, task manager, isolated-execution probe) are singletons but selected by **one name gate** — `plugins.boot_orchestrator` — because they move as a coherent set with the runtime flavor.++Storage and secrets are neither. Each is a **process-global singleton selected by its *own* config key**, independent of the orchestrator. So they need a net-new mechanism:++> **Keyed registry + config-selected singleton.** Plugins register N provider factories into a registry keyed by an open method token (mirroring `OrchestrationMode`). At boot, core reads a single config field (`storage_config.method` / `secrets_config.method`), looks that token up in the registry, and calls the factory to produce the one provider set on the hub.++This is deliberately **not** a `HubSlot` — hub slots are orchestrator-coupled (claimed inside `register` iff `boot_orchestrator == plugin.name`). Storage/secrets selection has nothing to do with the orchestrator, so reusing the slot machinery would wrongly couple them. The storage plan defines this pattern in full (§ "The new mechanism"); the secrets plan references it.++## Cross-cutting decisions (apply to both)++- **DX-1 — Batch the plugin-API version bump.** Discovery version-checks `targets_api` against `PLUGIN_API_VERSION` with **strict equality** (`pipelex/plugins/discovery.py`, `_register_plugin`), so *any* additive menu growth is a breaking bump that forces every external plugin to re-declare `targets_api`. Introduce **both** new registrar menu methods (`add_storage_provider` and `add_secrets_provider`) under a **single** `PLUGIN_API_VERSION` bump (2 → 3), even though the two features land in sequence, to avoid double-breaking the external plugins. The known external plugins that must bump their `targets_api` to 3: `pipelex-temporal`, `pipelex-mistralai-workflows`. (Per the workspace "no backward compatibility" principle, that's acceptable — note it in changelogs; there is no transition shim.)+ - *Option not taken:* relax discovery to accept `targets_api <= PLUGIN_API_VERSION` for additive growth. That would make menu additions non-breaking but changes the plugin-system contract semantics — out of scope for this track; raise separately if the coarse-bump churn becomes painful.+- **DX-2 — The builtin provider plugin is unconditional.** Storage and secrets are required infra: if the config-selected method has no registered factory, boot must fail loud. The builtin `StoragePlugin` / `SecretsPlugin` therefore join `CORE_UNCONDITIONAL_PLUGIN_NAMES` (`pipelex/plugins/builtins.py`) so they can't be disabled into a broken boot.+- **DX-3 — External-provider config surface is a scoped follow-up.** `StorageProviderConfig` / the new `SecretsProviderConfig` have fixed, typed per-method sub-models. An out-of-tree provider (`azure`, `vault`) needs somewhere to read *its* config. Phase 1 of each plan lands the seam for the built-in methods only; a generic passthrough sub-config for external providers is captured as a follow-up in each plan, not built speculatively.+- **DX-4 — Parallel keyed registries stay un-abstracted (declined in this track; possible whole-pattern cleanup later).** `SecretsProviderRegistry` and `StorageProviderRegistry` are near-identical (a PR-review finding on [#1024](https://github.com/Pipelex/pipelex/pull/1024) proposed extracting a shared generic base so lookup/error behavior can't silently diverge). **Decision: not done here.** The genuinely-shared logic is ~5 trivial lines (`.get()` → None-check → raise → return); against that, six sibling plugin registries already exist (`storage`, `secrets`, `orchestrator`, `bundle_validator`, `model_lister`, `inference_backend`) and **none share a base** — parallel registries are the established, tolerated convention. Their behavior deliberately diverges in load-bearing ways: miss-semantics differ (hard-raise via `get_required` vs soft-`None` via `get_optional`), key types differ (`str` / `tuple[InferenceFamily, str]` / `OrchestrationMode`), and each raises a distinct, per-domain exception with its own `type_uri` doc page. Abstracting only storage+secrets would leave 2-of-6 sharing a base and 4 not — an asymmetry more confusing than uniform duplication. **Escape hatch:** if the coarse duplication ever bites, do it as a *separate whole-pattern* cleanup — a `KeyedRegistry[K, V]` generic (dict copy + `has` + keys accessor + `get_optional`) with a per-domain `on_missing` hook so each concrete registry keeps its own exception and keys-property name, touching all six `pipelex/plugins/*_registry.py`. Not this PR; matches the "smallest correct surface, no speculative additions" principle.++## Boot-audit classification (why only these two)++- **Already plugins:** inference backends (LLM/img-gen/extract/search, incl. OCR & web search), model listers, orchestrators, bundle validators, HTTP error mappers.+- **Boot-orchestrator-owned hub slots (leave as-is):** content generator, pipe router, pipe run, task manager, isolated-execution probe — they move as a set with the runtime flavor.+- **Population / config points (NOT plugins):** class registry, func registry, model deck, template sets. Where under-modular (templates especially, which lack even an injection param today), the fix is a normal injection seam, not entry-point discovery.+- **Prove-a-second-impl-first:** telemetry (don't double-wrap OTel's own exporter plugins), reporting/cost delegate (looks host-runtime-owned, not shelf-installable).+- **Promote (this track):** **secrets** and **storage**.diff --git a/wip/plugins/secrets-provider-plugin.md b/wip/plugins/secrets-provider-plugin.mdnew file mode 100644index 000000000..cdb382236--- /dev/null+++ b/wip/plugins/secrets-provider-plugin.md@@ -0,0 +1,135 @@+# Secrets provider → `pipelex.plugins` plugin++Status: **COMPLETE.** All phases delivered — this vertical (Phases 3–4 of the master plan) mirrors the storage pattern ([storage-provider-plugin.md](storage-provider-plugin.md), which defines the shared "keyed registry + config-selected singleton" mechanism this plan reuses wholesale). See [TODOS.md](../../TODOS.md) for the execution record (Phases 3–4 code + tests + docs, Phase 5 release gating). This plan reads as "mirror storage, swap the nouns."++Goal: turn the secrets provider into a formal plugin seam so third parties can ship `pipelex-secrets-<backend>` packages (Vault, AWS Secrets Manager, GCP Secret Manager, …) discovered via `pipelex.plugins` and selected at deploy time. The built-in `EnvSecretsProvider` becomes an unconditional builtin plugin.++Read [README.md](README.md) for the track's shared decisions (DX-1 API bump, DX-2 unconditional builtin, DX-3 external config follow-up).++---++## Cold-start context (the seam as it is today)++**Abstract base** — `pipelex/tools/secrets/secrets_provider_abstract.py`+- `SecretsProviderAbstract(ABC)` (line 6). Abstract: `get_required_secret(secret_id)`, `get_optional_secret(secret_id)`, `get_required_secret_specific_version(secret_id, *, version_id)`, `get_optional_secret_specific_version(secret_id, *, version_id)`, `set_secret_as_env_var(secret_id, *, version_id=LATEST_SECRET_VERSION_NAME)`. Concrete convenience `get_secret(secret_id) -> str` delegates to `get_required_secret`.+- `LATEST_SECRET_VERSION_NAME = "latest"` (line 3).++**The one implementation** — `pipelex/tools/secrets/env_secrets_provider.py`+- `EnvSecretsProvider` (line 9). Reads env vars via `get_required_env` / `get_optional_env`. The two `*_specific_version` methods `raise NotImplementedError`; `set_secret_as_env_var` is a no-op `pass`. **This is a single-impl seam** — the whole point of the plan is to open it.++**Boot injection** — `pipelex/pipelex.py`+- Import line 90; `setup()` param `secrets_provider: SecretsProviderAbstract | None = None` (line 186).+- **Hardcoded default:** line 212 `secrets_provider = secrets_provider or EnvSecretsProvider()` — constructed **early**, before the gateway check (comment: "needed for gateway check"), and before config-driven subsystems. This early construction is the main structural wrinkle (see § The wrinkle).+- Passed to telemetry factory (288) and to two setup helpers (345, 375). Set on hub line 306. `make()` mirror: param 560, forwarded 636.++**Config** — **none today.** No field selects the provider; it is hardcoded to Env via the `or EnvSecretsProvider()` fallback. There is no `secrets_config` in `pipelex.toml` or `configs.py`. This plan introduces one.++**Hub accessors** — `pipelex/hub.py`: `set_secrets_provider` (183), `get_required_secrets_provider()` (294), module-level `get_secrets_provider()` (472) and `get_secret(secret_id)` (607). Consumers (~4 files): `storage_provider_factory.py:62` (GCP creds — see the storage↔secrets coupling in the storage plan), `linkup_search_worker.py`, `linkup_extract_worker.py`, `show_cmd.py`. `secrets_utils.py` takes a `secrets_provider` param explicitly rather than pulling from the hub. **None of these change.**++**Pattern to mirror:** identical to the storage plan — `PipelexPlugin` contract, the registrar `_add` helper, `OrchestratorRegistry` as the read-view template, `docling_plugin.py` for the lazy optional-dep closure shape, `CORE_UNCONDITIONAL_PLUGIN_NAMES`. See [storage-provider-plugin.md](storage-provider-plugin.md) § Cold-start context for the exact refs.++---++## The wrinkle: secrets is constructed before config-driven subsystems — but this is fine (Decision W RESOLVED → W-A)++Storage is selected *after* config load and *after* secrets is on the hub — clean. Secrets looked different: `EnvSecretsProvider()` is built at `pipelex.py:212`, **early**, with the comment `# needed for gateway check`. That comment turned out to be **stale**, which is what makes W-A clean.++**Verified 2026-07-06 (locks Decision W):**++- **The gateway-check path is entirely secrets-free.** Neither `pipelex/system/pipelex_service/remote_config_fetcher.py` (`fetch_remote_config`, `make_dummy_remote_config`) nor `pipelex/system/pipelex_service/pipelex_service_config.py` (`is_pipelex_gateway_enabled`) references secrets in any form — no param, no hub read. Grep for `get_secret|secrets_provider|get_secrets_provider` in both is empty. The line-211 comment is wrong; nothing between line 212 and the telemetry factory needs secrets.+- **The true first consumer of `secrets_provider` in `setup()` is `make_telemetry_manager` at line 288** (passed as a param). The hub secrets slot isn't set until line 306, so nothing in between could be reading it from the hub either.+- **`build_registrar` is pure over config.** `PluginRegistrar.__init__(*, config)` takes only config; `build_registrar(config=get_config())` touches no hub, no models, no telemetry, constructs no SDK. `get_config()` is valid at `setup()` entry (config is loaded in `__init__` via `setup_config`, before `setup` runs). It's called exactly once today (line 393) and only re-read later by `_resolve_hub_slot` (line 505) and teardown (line 516).++**⇒ Decision W = W-A, confirmed, zero ordering risk.** Move the single `build_registrar(config=get_config())` call (currently `pipelex.py:393-394`) up to **right after `boot_orchestrator` is set (after line 209), before line 212**. Then:++1. Build `SecretsProviderRegistry(plugin_registrar.secrets_providers)`, set it on the hub.+2. Resolve secrets: `if secrets_provider is None: secrets_provider = registry.get_required(method=get_config().pipelex.secrets_config.method)(get_config().pipelex.secrets_config)`.+3. **Delete** the hardcoded `secrets_provider or EnvSecretsProvider()` line 212 and its stale comment.+4. Everything downstream is untouched — telemetry (288), hub set (306), and the existing registry constructions / slot application at 404-418 keep referencing the already-built `self._plugin_registrar`. Only the *build call* moves up; its downstream consumers stay put.++Bonus (recommended, not required): move the `boot_orchestrator` validation gate (`pipelex.py:401-403`, `UnknownBootOrchestratorError`) up alongside the build call — it's a pure check on the registrar and moving it fails fast on a typo'd orchestrator before any secrets/gateway work.++**W-B is not needed** and is not the plan — recorded only so a future reader knows it was considered: keep Env hardcoded for a pre-gateway bootstrap, then re-resolve and overwrite the hub. Rejected because W-A is clean; W-B would leave a transient provider and a hub re-set for no benefit.++---++## The mechanism (reuse storage's)++Same "keyed registry + config-selected singleton" as the storage plan:++- New registrar menu method `add_secrets_provider(*, method: str, factory: SecretsProviderFactoryFn)` where `SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], SecretsProviderAbstract]`, into `registrar.secrets_providers: dict[str, SecretsProviderFactoryFn]`, via `_add`.+- New read-view `SecretsProviderRegistry` (`pipelex/plugins/secrets_provider_registry.py`) mirroring `OrchestratorRegistry` / the new `StorageProviderRegistry`.+- New `UnknownSecretsMethodError` in `pipelex/plugins/exceptions.py`.+- New config: `SecretsProviderConfig(ConfigModel)` with `method: str = Field(strict=False)` (default `"env"`); built-in token `"env"` needs no sub-config. Add `[pipelex.secrets_config]\nmethod = "env"` to `pipelex/pipelex.toml`, and wire `secrets_config` into the config model where storage_config lives (`pipelex/tools/secrets/secrets_config.py` new file + a field on the owning config — confirm placement with the config-owner; the storage one hangs under `pipelex.storage_config`, so `pipelex.secrets_config` is the natural sibling).+- Builtin `SecretsPlugin` (`name="secrets"`, unconditional per DX-2) whose `register` calls `add_secrets_provider(method="env", factory=_make_env_secrets_provider)`; `_make_env_secrets_provider(config)` just returns `EnvSecretsProvider()`.+- Boot selects via the registry at the point Decision W lands; the explicit `setup(secrets_provider=...)` param still wins ahead of registry selection.++Because there's only one built-in method, introducing a config selector for a single impl looks like over-engineering — but the selector *is* the feature: it's what lets an external `vault` provider be chosen. `"env"` stays the default and the unconditional builtin, so out-of-the-box behavior is byte-identical.++---++## Decisions (recommended defaults — confirm before building)++- **W — Selection point. ✅ RESOLVED = W-A (verified 2026-07-06).** Move the pure `build_registrar(config=get_config())` call up to after line 209 / before line 212; registry-select secrets there; delete the hardcoded `or EnvSecretsProvider()` and its stale `# needed for gateway check` comment. Confirmed zero ordering risk — the gateway path is secrets-free (see § The wrinkle for the evidence). No Phase-0 investigation needed; this is now a build instruction.+- **S1 — Open method token + config selector.** Mirror storage D1: `SecretsProviderConfig.method` is an open `str`, validated at registry lookup, default `"env"`.+- **S2 — One unconditional builtin `SecretsPlugin`.** `name="secrets"`, added to `CORE_UNCONDITIONAL_PLUGIN_NAMES` (DX-2).+- **S3 — API bump batched with storage (DX-1).** `add_secrets_provider` ships in the same `PLUGIN_API_VERSION` 2→3 bump as `add_storage_provider`. If secrets lands in a much later PR than storage, that PR does its own bump and external plugins take a second `targets_api` update — acceptable but avoid if the two land close together.+- **S4 — External config surface (follow-up).** Same as storage D3: `SecretsProviderConfig` gets a generic passthrough for out-of-tree providers as a scoped follow-up, not Phase 1. `"env"` needs no sub-config, so Phase 1 has nothing to add.++---++## Phased checklist++### Phase 0 — Confirm design+- [x] **Decision W RESOLVED = W-A** (verified 2026-07-06) — gateway path is secrets-free; `build_registrar` is pure over config; move the build call ahead of line 212. See § The wrinkle. No further tracing needed.+- [ ] Confirm with the config owner where `secrets_config` hangs (sibling of `storage_config` under `pipelex.*`).+- [ ] Confirm the `PLUGIN_API_VERSION` bump is shared with storage (DX-1 / S3) or standalone.++### Phase 1 — The mechanism (core) — *skip the parts storage already landed*+- [ ] If storage already bumped `PLUGIN_API_VERSION` to 3 **and** added `add_storage_provider`: only add `add_secrets_provider` + `secrets_providers` field to `PluginRegistrar`. If storage hasn't landed: do the 2→3 bump here too.+- [ ] `add_secrets_provider(*, method, factory)` menu method via `_add`; `SecretsProviderFactoryFn` alias.+- [ ] New `pipelex/plugins/secrets_provider_registry.py`: `SecretsProviderRegistry` (`get_optional` / `get_required` / `has` / `methods`).+- [ ] New `UnknownSecretsMethodError` in `pipelex/plugins/exceptions.py`.+- [ ] Hub: `set_secrets_provider_registry` / `get_secrets_provider_registry` + module accessor.++### Phase 2 — Config+- [ ] New `SecretsProviderConfig(ConfigModel)` (`pipelex/tools/secrets/secrets_config.py`), `method: str = Field(strict=False)`; no default in the class per config rules — default lives in TOML.+- [ ] Add `[pipelex.secrets_config] method = "env"` to `pipelex/pipelex.toml`; wire `secrets_config` into the owning config model.+- [ ] Add the same block to `.pipelex/pipelex.toml` override **only if** it's a genuine client-override case — secrets backend selection plausibly is (a deployer picks vault vs env), so include it with the real `method = "env"` value, never commented out (house rule).+- [ ] `make tb` after config changes — config model ↔ TOML must stay in sync or boot fails.++### Phase 3 — The builtin plugin + boot wiring (Decision W = W-A)+- [ ] New `pipelex/plugins/secrets/secrets_plugin.py`: `SecretsPlugin` + `_make_env_secrets_provider(config)` returning `EnvSecretsProvider()`.+- [ ] Register `SecretsPlugin()` in `BUILTIN_PLUGINS`; add `"secrets"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES`.+- [ ] **Move the `build_registrar(config=get_config())` call** (`pipelex.py:393-394`) up to after line 209 / before line 212; keep `self._plugin_registrar = plugin_registrar`. Downstream registry constructions + slot application (404-418) stay put referencing the already-built registrar. Optionally move the `boot_orchestrator` gate (401-403) up with it for fail-fast.+- [ ] Build + `set_secrets_provider_registry(...)` right after the moved build call.+- [ ] **Delete** `pipelex.py:211-212` (the stale `# needed for gateway check` comment + `secrets_provider = secrets_provider or EnvSecretsProvider()`); replace with `if secrets_provider is None: secrets_provider = get_secrets_provider_registry().get_required(method=get_config().pipelex.secrets_config.method)(get_config().pipelex.secrets_config)` — explicit `setup(secrets_provider=...)` param still wins.+- [ ] Drop the now-unused `EnvSecretsProvider` import from `pipelex.py` (it moves to the plugin module).++### Phase 4 — Tests+- [ ] Unit: `SecretsProviderRegistry` hit/miss (`UnknownSecretsMethodError`) + duplicate fail-loud.+- [ ] Boot: default config yields `EnvSecretsProvider` on the hub; explicit `setup(secrets_provider=...)` still overrides.+- [ ] Integration: external test plugin registering a fake `method="test_secret"` is selectable via config (mirror the inference external-plugin test).+- [ ] Guard the storage↔secrets ordering: a boot with a non-env secrets method must still be on the hub before storage's GCP arm reads it (add/extend a boot-order assertion).+- [ ] `pipelex plugins list` shows `secrets`.++### Phase 5 — Docs & changelog+- [ ] New `docs/under-the-hood/secrets-provider-plugins.md` (mirror the storage/orchestrator doc; emphasize the lazy optional-dep closure for SDK-backed providers like Vault/AWS). mkdocs nav.+- [ ] Update any "the plugin seams are …" enumerations to include secrets.+- [ ] CHANGELOG `[Unreleased]`: secrets provider is now a plugin seam; `PLUGIN_API_VERSION` bump note if not already recorded by the storage change.++---++## Verification / gates+- [ ] `make agent-check`.+- [ ] `make tb` — critical (new `secrets_config` ↔ TOML ↔ selection).+- [ ] `make agent-test`.+- [ ] Manual: `pipelex plugins list`; a run with default `method="env"`; confirm `get_secret(...)` still resolves env vars.++## Cross-repo consequence (release-gated)+- Same as storage: `pipelex-temporal` / `pipelex-mistralai-workflows` bump `targets_api` to 3 (only once, if batched with storage per DX-1).++## Follow-ups (do not build now)+- **S4 external config passthrough** for out-of-tree secrets providers.+- **First real external provider** (Vault or AWS Secrets Manager) as the proof-of-seam, likely as a private/closed package if it targets the hosted platform.+- **Retire the `*_specific_version` `NotImplementedError`s** in `EnvSecretsProvider` only if a consumer needs them — out of scope here.diff --git a/wip/plugins/storage-provider-plugin.md b/wip/plugins/storage-provider-plugin.mdnew file mode 100644index 000000000..c0e6a0b4c--- /dev/null+++ b/wip/plugins/storage-provider-plugin.md@@ -0,0 +1,130 @@+# Storage provider → `pipelex.plugins` plugin++Status: **COMPLETE.** All phases delivered — this vertical (Phases 1–2 of the master plan) defines the shared "keyed registry + config-selected singleton" mechanism (§ The new mechanism) that the secrets plan reuses. See [TODOS.md](../../TODOS.md) for the execution record (Phases 1–2 code + tests + docs, Phase 5 release gating).++Goal: turn the storage provider into a formal plugin seam so third parties can ship `pipelex-storage-<backend>` packages discovered via the `pipelex.plugins` entry-point group and selected at deploy time by `storage_config.method`. The built-in providers (local / in_memory / s3 / gcp) become a single unconditional builtin plugin registering their factories — exactly as `OpenAIPlugin` registers several inference backends.++Read [README.md](README.md) first for the track's shared decisions (DX-1 API bump, DX-2 unconditional builtin, DX-3 external config follow-up) and the boot-audit rationale.++---++## Cold-start context (the seam as it is today)++Everything below is verified against the tree (worktree `_plugins`, branch is whatever's checked out — this is `pipelex/` code).++**Abstract base** — `pipelex/tools/storage/storage_provider_abstract.py`+- `StorageProviderAbstract` (line 16). Template methods `load` / `load_with_metadata` / `store`; subclass hooks `_load_with_metadata`, `_store`, `public_url` (abstract).+- Also defines `PIPELEX_STORAGE_SCHEME = "pipelex-storage://"` and `StoredData(NamedTuple)` (`data: bytes`, `mime_type: str | None`).++**Built-in implementations** — all under `pipelex/tools/storage/`+- `LocalStorageProvider` (`local_storage_provider.py:12`) — ctor `(root_path: Path)`; hard-imports `aiofiles`; path-traversal guarded; `public_url` → `file://`.+- `InMemoryStorageProvider` (`in_memory_storage_provider.py:13`) — a `RootModel[dict[str, bytes]]` **and** `StorageProviderAbstract`; no deps; `public_url` → `None`.+- `S3StorageProvider` (`s3_storage_provider.py:15`) — ctor `(bucket_name, region, signed_urls_lifespan)`; optional dep **`aioboto3`** (extra `"s3"`) + botocore, lazy-imported inside methods, guarded by `find_spec` → `MissingDependencyError`.+- `GcpStorageProvider` (`gcp_storage_provider.py:21`) — ctor `(bucket_name, project_id, credentials_file_path, signed_urls_lifespan)`; optional dep **`google-cloud-storage`** (extra `"gcp-storage"`), lazy-imported; sync GCS wrapped in `asyncio.to_thread`.++**The factory being replaced** — `pipelex/tools/storage/storage_provider_factory.py`+- `make_storage_provider_from_config(storage_provider_config)` (line 14) is a `match storage_provider_config.method:` over `StorageMethod`, each arm: `None`-check the sub-config → `sub.lazy_validate()` → construct the provider.+- Imports all four concrete classes at module top (importing the factory imports all providers; optional SDKs still stay lazy inside the providers).+- The **GCP arm** calls `get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH")` (line 62) — a storage→secrets→hub dependency. Safe at boot because storage is set on the hub (`pipelex.py:310`) *after* secrets (`pipelex.py:306`); **not** safe inside a plugin `register` (hub access is forbidden there). ⇒ construction must happen in a lazily-invoked factory closure, never in `register`.++**Config** — `pipelex/tools/storage/storage_config.py`+- `StorageMethod(StrEnum)` (line 13): `LOCAL`, `IN_MEMORY`, `S3`, `GCP`.+- Per-method sub-models: `StorageLocalConfig`, `StorageInMemoryConfig`, `StorageS3Config`, `StorageGcpConfig`, each with `lazy_validate()` raising `StorageConfigError`.+- `StorageProviderConfig` (line 171): `method: StorageMethod = Field(strict=False)` + one optional sub-field per method + a `model_validator(mode="after")` requiring the sub-config matching `method`.+- `StorageConfig(StorageProviderConfig)` (line 238) adds `is_fetch_remote_content_enabled` / `is_upload_local_content_enabled`.+- TOML `[pipelex.storage_config]` at `pipelex/pipelex.toml:5-27`; reached as `get_config().pipelex.storage_config`.++**Boot injection** — `pipelex/pipelex.py`+- `setup()` param `storage_provider: StorageProviderAbstract | None = None` (line 187).+- Boot block (lines 307-310): `if storage_provider is None: … make_storage_provider_from_config(get_config().pipelex.storage_config)` then `set_storage_provider(...)`.+- Also read at line 416 for the `CONTENT_GENERATOR` default thunk (`GeneratedContentFactory(storage_provider=...)`).+- `make()` mirror: param line 561, forwarded line 637.++**Hub accessors** — `pipelex/hub.py`: `set_storage_provider` (186), `get_storage_provider()` (306, raises if unset), module-level `get_storage_provider()` (476). Consumed by ~9 files / ~20 sites (input normalizer, pdf renderer, pypdfium2 worker, gateway extract worker, delivery executor, content generator, file/image/document prompt utils). **None of these change** — they keep calling `get_storage_provider()`.++**The plugin pattern to mirror** (verified):+- Contract `pipelex/plugins/contract.py` — `PipelexPlugin` protocol (`name`, `targets_api`, `register`); `PLUGIN_API_VERSION` (currently 2); `register` is **side-effect-free** (menu calls only; no hub, no I/O, no SDK import).+- Registrar `pipelex/plugins/registrar.py` — menu methods like `add_orchestrator` / `add_bundle_validator` funnel through the generic `_add` helper (line 279) which fail-louds on duplicate keys naming both plugins. Keyed registries live as `dict` fields on the registrar.+- Builtins `pipelex/plugins/builtins.py` — `BUILTIN_PLUGINS` is a list of instantiated plugin objects; `CORE_UNCONDITIONAL_PLUGIN_NAMES = frozenset({"direct", "openai"})`.+- Discovery `pipelex/plugins/discovery.py` — `build_registrar(config=...)` iterates builtins then external `pipelex.plugins` entry points; strict-equality `targets_api` check; denylist via `config.plugins.disabled`.+- Reference builtin registering several factories: `pipelex/plugins/openai/openai_plugin.py`. Reference optional-dep lazy closure: `pipelex/plugins/docling/docling_plugin.py` (`register` just calls `add_inference_backend`; the SDK import + `require_sdk` live in the module-level `_make_..._worker` closure).+- Example registry read-view to copy: `pipelex/plugins/orchestrator_registry.py` (`OrchestratorRegistry`, `get_optional` / `has` / `modes`).++---++## The new mechanism (defines the shared pattern)++A **keyed registry + config-selected singleton**. New, because no existing pattern selects a process-global singleton by its own config key (see [README.md](README.md) § The shared new pattern).++- **Open method token.** Make the registry key an open `str` token, mirroring `OrchestrationMode` (an open string, not a closed enum). The built-in tokens stay the `StorageMethod` values (`"local"`, `"in_memory"`, `"s3"`, `"gcp"`); an external plugin registers e.g. `"azure"`. **Decision D1** below covers making `storage_config.method` accept arbitrary tokens.+- **Registration side (in the plugin):** a new registrar menu method+ `add_storage_provider(*, method: str, factory: StorageProviderFactoryFn)` where+ `StorageProviderFactoryFn = Callable[[StorageProviderConfig], StorageProviderAbstract]`.+ Accumulates into `registrar.storage_providers: dict[str, StorageProviderFactoryFn]`, fail-loud on duplicate via the existing `_add` helper.+- **Read view:** `StorageProviderRegistry` (new, `pipelex/plugins/storage_provider_registry.py`) mirroring `OrchestratorRegistry` — `get_optional(method=...)` / `get_required(method=...)` (raising a new `UnknownStorageMethodError`) / `has` / `methods`.+- **Selection side (in core boot):** replace the `make_storage_provider_from_config` call at `pipelex.py:307-310` with:+ register-built `StorageProviderRegistry` on the hub → read `config.method` → `registry.get_required(method=config.method)(config)` → `set_storage_provider(...)`. The explicit `setup(storage_provider=...)` param still wins ahead of this (same three-tier precedence as content generator: explicit param > registry selection > — there is no separate core default; the builtin plugin *is* the default supplier).+- **Where the per-method construction goes:** each `match` arm body from today's factory (None-check → `lazy_validate()` → construct) moves verbatim into a module-level factory closure in the builtin `StoragePlugin`, e.g. `_make_local_storage_provider(config)`, `_make_s3_storage_provider(config)`. The GCP closure keeps its `get_secrets_provider()` hub read — legal because the closure runs at the boot apply-point, not in `register`.++Net effect: `make_storage_provider_from_config` and its `match` are deleted; the four arms become four registered closures; core boot does a registry lookup instead of a hardcoded factory call. `StorageProviderAbstract`, the four provider classes, the config models, and every consumer are unchanged.++---++## Decisions (recommended defaults — confirm before building)++- **D1 — Open method token.** Make `StorageProviderConfig.method` accept an open `str` while keeping the `StorageMethod` enum for the built-ins (validate against the registry at boot, not against the enum at parse time — an unknown token surfaces as `UnknownStorageMethodError` from `get_required`, which is the right layer). *Recommended:* yes; without it external providers can be registered but never selected. *Alternative:* keep the closed enum and defer external selection — rejected, it guts the feature.+- **D2 — One builtin `StoragePlugin`, unconditional.** A single builtin plugin (`name = "storage"`) registers all built-in methods, and joins `CORE_UNCONDITIONAL_PLUGIN_NAMES` (DX-2) so it can't be disabled into a broken boot. *Alternative:* one plugin per method — rejected as needless fragmentation; `OpenAIPlugin` already precedents "one plugin, many factories."+- **D3 — External-provider config surface (follow-up, not Phase 1).** `StorageProviderConfig` has fixed typed sub-fields, so an `azure` provider has nowhere to read its config. Phase 1 lands the seam for built-in methods only. Follow-up: a generic passthrough (e.g. `extra: dict[str, dict[str, Any]]` on `StorageProviderConfig`, handed to the factory) so out-of-tree providers get structured config. Captured in § Follow-ups; do not build speculatively.+- **D4 — Factory signature.** `Callable[[StorageProviderConfig], StorageProviderAbstract]` (whole config in, provider out) — lowest-churn move of the existing arm bodies, and keeps the GCP secrets read where it already is. *Alternative:* pass a pre-resolved typed sub-config + secrets provider explicitly (cleaner/more testable) — deferred; revisit if D3's passthrough makes "whole config in" awkward.+- **D5 — API version bump batched with secrets (DX-1).** Introduce `add_storage_provider` **and** `add_secrets_provider` in the same `PLUGIN_API_VERSION` 2→3 bump even though storage lands first, so external plugins re-declare `targets_api` once. If secrets is genuinely far behind, fall back to bumping per-feature and accept the double external-plugin update — but default to batching.++---++## Phased checklist++### Phase 0 — Confirm design+- [ ] Walk D1–D5 with the user; lock the token-openness (D1) and the API-bump batching (D5) — those two shape everything downstream.++### Phase 1 — The mechanism (core)+- [ ] Add `PLUGIN_API_VERSION` bump to 3 in `pipelex/plugins/contract.py` (batched with secrets per DX-1; do the bump once).+- [ ] Add `storage_providers: dict[str, StorageProviderFactoryFn]` field + `add_storage_provider(*, method, factory)` menu method to `PluginRegistrar` (`pipelex/plugins/registrar.py`), routed through `_add`. Add the `StorageProviderFactoryFn` type alias.+- [ ] New `pipelex/plugins/storage_provider_registry.py`: `StorageProviderRegistry` (mirror `OrchestratorRegistry`) with `get_optional` / `get_required` / `has` / `methods`.+- [ ] New `UnknownStorageMethodError` in `pipelex/plugins/exceptions.py` (message lists registered methods, mirroring `UnknownBootOrchestratorError`).+- [ ] Hub: `set_storage_provider_registry` / `get_storage_provider_registry` on `PipelexHub` + module-level accessor (mirror the four existing registry accessors in `pipelex/hub.py`).++### Phase 2 — The builtin plugin+- [ ] New `pipelex/plugins/storage/storage_plugin.py`: `StoragePlugin` (`name="storage"`, `targets_api=PLUGIN_API_VERSION`, `register` calling `add_storage_provider` for each built-in method).+- [ ] Module-level factory closures (`_make_local_storage_provider`, `_make_in_memory_storage_provider`, `_make_s3_storage_provider`, `_make_gcp_storage_provider`) holding the exact bodies moved from the deleted `match` arms (keep `lazy_validate()`; keep GCP's `get_secrets_provider()` read).+- [ ] Register `StoragePlugin()` in `BUILTIN_PLUGINS` and add `"storage"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES` (`pipelex/plugins/builtins.py`).++### Phase 3 — Wire core boot to the registry+- [ ] In `pipelex.py` setup: after `build_registrar(...)`, build + `set_storage_provider_registry(StorageProviderRegistry(plugin_registrar.storage_providers))` alongside the other four registry sets (near lines 404-407).+- [ ] Replace the `make_storage_provider_from_config` block (lines 307-310) with the registry-selection path (explicit param > `registry.get_required(method=config.method)(config)`). Note ordering: storage still resolves after secrets is on the hub (keep the GCP secrets read working).+- [ ] Delete `make_storage_provider_from_config` and `pipelex/tools/storage/storage_provider_factory.py` if nothing else imports it (grep first — `show_cmd`, tests). Move any residual import of the provider classes into the plugin module.+- [ ] D1: relax `StorageProviderConfig.method` to open `str` (drop the strict enum bind at parse; validation happens at registry lookup).++### Phase 4 — Tests+- [ ] Unit: `StorageProviderRegistry` (`get_required` hit/miss → `UnknownStorageMethodError`; duplicate registration fail-loud through `_add`).+- [ ] Unit/boot: booting with each built-in `method` yields the right provider on the hub (parametrized). Include the s3/gcp arms with SDK absent → the factory raising `MissingDependencyError` only when selected, not at registration.+- [ ] Integration: an external test plugin registering a fake `method="test_mem"` (entry-point discovered) is selectable via config — mirror `tests/.../test_external_plugin.py` (the inference-backend external-plugin test) for the harness.+- [ ] `pipelex plugins list` shows `storage` (origin builtin) with its contributions.++### Phase 5 — Docs & changelog+- [ ] New `docs/under-the-hood/storage-provider-plugins.md`, mirroring `docs/under-the-hood/orchestrator-plugins.md` structure (seam-in-one-view, the contract, the factory closure + lazy optional-dep guard, config selection, authoring an out-of-tree provider). Add to mkdocs nav.+- [ ] Update `docs/under-the-hood/inference-backend-plugins.md` cross-links / any "the plugin seams are: inference, orchestrator" enumerations to include storage.+- [ ] CHANGELOG `[Unreleased]`: "breaking" — `PLUGIN_API_VERSION` 2→3 (external plugins must re-declare `targets_api`); storage provider is now a plugin seam.++---++## Verification / gates+- [ ] `make agent-check` (pyright, ruff, mypy, plxt, keyword-only guard).+- [ ] `make tb` — **critical here:** it tests the boot sequence incl. config loading; the `storage_config` model ↔ TOML ↔ selection path must stay in sync.+- [ ] `make agent-test` — full suite; catches the framework-positional and discovery edges the type checker can't.+- [ ] Manual: `pipelex plugins list`; a real run with `method="local"`; flip to `method="in_memory"` and confirm selection.++## Cross-repo consequence (release-gated, not this PR)+- `pipelex-mistralai-workflows` must bump `targets_api` to 3 (DX-1); it registers no storage provider and imports none of the removed symbols, so the bump is its only change. `pipelex-temporal` needs the bump **and** a code migration: its payload-codec factory (`pipelex_temporal/codec/codec_factory.py`) imports the now-removed `make_storage_provider_from_config` and must switch to `get_storage_provider_registry().get_required(method=...)`. Do both when the pipelex version carrying this lands, not before.++## Follow-ups (do not build now)+- **D3 external config passthrough** — generic `extra` sub-config on `StorageProviderConfig` for out-of-tree providers. This must also fix the **`uri_format` gap**: `StorageProviderConfig.uri_format` (read by `GeneratedContentFactory._build_storage_key` on every content store) is defined only for the four built-in methods and raises `StorageConfigError` on the `case _:` arm for any external token — so an external provider selects and boots cleanly but crashes on the first *generated-content* store. D3 must give external methods a `uri_format` (e.g. promote it to a top-level field, or a passthrough default). Until then the docs page flags external providers as usable for their own storage API but not yet as the generated-content backing store.+- **Cookbook example** — a `pipelex-storage-hello` external plugin mirroring the hello-inference-plugin cookbook track, once the seam is proven.