Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feat(plugins): storage & secrets provider plugin seams (#1024)

Louis Choquel2a7cd2f96418
diff --git a/.pipelex/pipelex.toml b/.pipelex/pipelex.toml
index 25d68f91d..a508997f6 100644
--- a/.pipelex/pipelex.toml
+++ b/.pipelex/pipelex.toml
@@ -119,6 +119,15 @@ bucket_name = ""
project_id = ""
signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs
+####################################################################################################
+# Secrets Config
+####################################################################################################
+
+[pipelex.secrets_config]
+# Secrets provider method: "env" (built-in, reads secrets from environment variables).
+# An external pipelex-secrets-<backend> plugin can add its own method (e.g. "vault").
+method = "env"
+
####################################################################################################
# Scan Config
####################################################################################################
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f75e88e8f..f7ead2a92 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,13 +4,20 @@
### Added
+- **Storage provider is now a plugin seam:** The storage backend (`local` / `in_memory` / `s3` / `gcp`) is selected at boot from a `StorageProviderRegistry` keyed by the open `storage_config.method` token, populated by the always-on built-in `StoragePlugin`. A third party can now ship a `pipelex-storage-<backend>` package that advertises itself under the `pipelex.plugins` entry-point group and is selected via `storage_config.method` — the same discovery/denylist machinery the inference and orchestrator seams use. `storage_config.method` is now an open `str`: an unknown method fails loudly at boot with `UnknownStorageMethodError` (listing the registered methods), not at config parse. See the new `docs/under-the-hood/storage-provider-plugins.md`.
+- **Secrets provider is now a plugin seam:** The secrets backend (`env`) is selected at boot from a `SecretsProviderRegistry` keyed by the new open `secrets_config.method` token (default `"env"`), populated by the always-on built-in `SecretsPlugin`. A third party can now ship a `pipelex-secrets-<backend>` package (Vault, AWS Secrets Manager, …) that advertises itself under the `pipelex.plugins` entry-point group and is selected via `secrets_config.method` — the same mechanism the storage seam uses. Out-of-the-box behavior is unchanged: `env` stays the default. An unknown method fails loudly at boot with `UnknownSecretsMethodError` (listing the registered methods), not at config parse. See the new `docs/under-the-hood/secrets-provider-plugins.md`.
- **TOML pipeline inputs:** The `--inputs` file passed to `pipelex run` and `pipelex-agent run` (pipe/bundle/method) can now be TOML in addition to JSON, discriminated by file extension — a `.toml` suffix is parsed as TOML, every other value stays JSON. Both formats produce the same input dictionary; TOML's multi-line strings make text-heavy inputs much easier to author. `run bundle <dir>` auto-detects `inputs.toml` alongside `inputs.json` when `--inputs` is omitted, erroring if both exist. Inline JSON (`{...}`) and agent-CLI stdin inputs stay JSON-only. A bare TOML datetime/date/time literal is rejected with an explicit error (quote it as a string) — native datetime concept support is a separate track.
- **`--format toml` for inputs-template generation:** `pipelex build inputs` and `pipelex-agent inputs` (pipe/bundle/method) accept `--format json|toml` (default `json`) to emit the generated inputs template as TOML. On the main CLI, `toml` defaults the output filename to `inputs.toml`; on the agent CLI, `toml` prints the raw TOML template to stdout (the no-inputs case prints a TOML comment line that loads back as an empty dict).
### Changed
+- **Plugin API version bumped to 3 (Breaking):** `PLUGIN_API_VERSION` is now `3` (was `2`) to add the `add_storage_provider` and `add_secrets_provider` registrar menu methods. Plugin discovery version-checks with strict equality, so every external plugin must re-declare `targets_api = 3`. `pipelex-mistralai-workflows` registers no storage or secrets provider and imports none of the removed symbols, so the `targets_api` bump is its only change. `pipelex-temporal` needs the `targets_api` bump **and** a code migration: its payload-codec factory imports the now-removed `make_storage_provider_from_config` (see Removed) and must switch to resolving the provider via the storage-provider registry (`get_storage_provider_registry().get_required(method=...)`).
- **`PipeSignature` is not a pipe type (Breaking):** A signature is now declared by **omitting** `type` — a `[pipe.x]` section with no `type` and nothing but the contract (`description`, `output`, optional `inputs`, optional `signature_for`) *is* a `PipeSignature`. Writing `type = "PipeSignature"` explicitly is rejected with a migration error ("`PipeSignature` is no longer a pipe type — delete the `type` line"), and a typeless section that declares any non-contract field is a hard error naming the field. The `pipelex-agent pipe` authoring command mirrors this: a typeless JSON spec (no `type`) renders a `[pipe.x]` section with no type line, and `--type PipeSignature` is rejected. **MTHDS JSON Schema shape change:** the signature arm no longer carries a `type` property (an explicit tag now fails schema validation), so the downstream schema copies (`mthds`, `vscode-pipelex`, `mthds-ui`) must be re-synced on the next release via the `mthds-schema-sync` skill.
+### Removed
+
+- **`make_storage_provider_from_config` (Breaking):** The module-level storage-provider factory (`pipelex/tools/storage/storage_provider_factory.py`) is removed — storage is now resolved through the config-selected `StorageProviderRegistry` at boot. Downstream code that imported this helper (notably `pipelex-temporal`'s payload-codec factory) must select through the registry instead.
+
## [v0.37.0] - 2026-07-04
### Highlights
diff --git a/TODOS.md b/TODOS.md
index 3438ef2fc..9d20e33e2 100644
--- a/TODOS.md
+++ b/TODOS.md
@@ -1,72 +1,234 @@
-# TOML pipeline inputs — feature/Inputs
+# Master plan — Provider plugins: storage & secrets
-Status: **CHECKPOINT 3 CLEARED (2026-07-06)** — all four phases done and committed. Phases 1+2 (`9041c6986`), Phase 3 (`1fdfd26d0`), Phase 4 docs + changelog (`5ebd9bef2`), with full `make agent-check` + `make agent-test` green. Feature is ready for PR to dev — nothing pushed. This tracker replaces the retired hello-plugin tracker (that track completed and merged to dev via PR #1015).
+Status: **TRACK COMPLETE. All 5 phases DONE + all 5 Checkpoints CLEARED.** Both provider verticals (storage + secrets) shipped as config-selected plugin seams; Phase 5 (release gating / cross-repo bookkeeping) done and Checkpoint 5 (final whole-branch clean-room review) cleared with its triage applied. Branch: `feature/More-plugins-2` (worktree `/Users/lchoquel/repos/Pipelex/_plugins`). **Nothing pushed** (no upstream) — the PR to the intended base is Louis's call. This file is the **conductor**; the granular per-seam procedures live in linked docs and should not be duplicated here.
-## Goal
+Commits on the branch (newest substantive first; run `git log` for the exact tip — later `docs(plugins)` bookkeeping commits may sit on top):
-Accept pipeline inputs as TOML in addition to JSON, discriminated by file extension, on both CLI surfaces (`pipelex run pipe|bundle|method` and `pipelex-agent run ...`). Also let the input-template generators (`pipelex build inputs ...`, `pipelex-agent inputs ...`) emit TOML via `--format toml`. TOML's multi-line strings make text-heavy inputs much more pleasant to author than JSON.
+- `0bbc39474` — fix: Checkpoint-5 whole-branch review triage (**Phase 5 / Checkpoint 5 — the review-triage commit**)
+- `65603e912` — test: secrets-provider seam tests + docs (**Phase 4 — the review target**)
+- `fdf3e114b` — fix: Phase-3 review triage (kit-config sync + stale boot comment)
+- `8a0b32668` — feat: secrets provider → config-selected plugin seam (**Phase 3 code — the review target**)
+- `3940eb7a2` — docs: Phase 2 checkpoint bookkeeping — storage vertical done
+- `37a3b72f1` — fix: Phase-2 review triage (doc tense + stronger gcp secret-wiring assert)
+- `f08193e81` — test: storage-provider seam tests + docs (**Phase 2**)
+- `977c73811` — docs: this plan + Phase 1 checkpoint bookkeeping
+- `58961848a` — fix: contract.py v3 comment (Checkpoint-1 review triage)
+- `8268ff08f` — feat: storage provider → config-selected plugin seam (**Phase 1 code**)
+- `04434f785` — branch base (release v0.37.0 merge). Whole-branch diff for the final Checkpoint 5 = `git diff 04434f785..HEAD`.
-## Current state (verified against this tree)
+**Cold-start (resume here):** **The whole track is COMPLETE — nothing left to build.** Both provider verticals shipped (storage Phases 1–2, secrets Phases 3–4) and Phase 5 (release gating) + Checkpoint 5 (final whole-branch clean-room review + triage) are done. All gates green on the final tip: `agent-check`, `tb`, `check-config-sync`, full `agent-test`. The **only remaining action is Louis's**: open the PR `feature/More-plugins-2` → intended base, and (when the pipelex version carrying this lands) execute the cross-repo obligations recorded in the CHANGELOG — `pipelex-mistralai-workflows` bumps `targets_api` to 3 (bump-only); `pipelex-temporal` bumps `targets_api` to 3 **and** migrates its payload-codec factory off the removed `make_storage_provider_from_config` to `get_storage_provider_registry().get_required(...)`. Deferred design follow-ups (D3 external config passthrough incl. the external-method `uri_format` gap, first real external provider, cookbook example) are captured in `wip/plugins/`. `PLUGIN_API_VERSION` is 3.
-- Both surfaces take `--inputs` and discriminate by content prefix, not extension: `startswith("{")` → inline JSON, else → JSON file via `load_json_dict_from_path` + `resolve_inputs_paths`. Main CLI: `pipelex/cli/commands/run/_run_core.py:140-164`. Agent CLI: `pipelex/cli/agent_cli/commands/run/stdin_resolver.py` (`parse_cli_inputs` → `_parse_inputs_arg`).
-- The agent CLI additionally reads JSON from stdin (flat dict or `working_memory` envelope) — `stdin_resolver.py:_read_stdin_inputs`.
-- `run bundle <dir>` auto-detects `inputs.json` (`DEFAULT_INPUTS_FILE_NAME` in `pipelex/builder/conventions.py`) — main `cli/commands/run/bundle_cmd.py:149`, agent `cli/agent_cli/commands/run/bundle_cmd.py:118`.
-- TOML loading machinery already exists and is unused on this path: `pipelex/tools/misc/toml_utils.py` (`load_toml_from_path` via `tomli`, raising `TomlError`; `tomlkit` for writing).
-- Template generation: main CLI `build inputs` uses `render_inputs` (`pipelex/core/pipes/inputs/input_renderer.py`) → `InputStuffSpecs.render_inputs` (`input_stuff_specs.py:153`) which builds a dict then `json.dumps` — the dict is trivially extractable for a TOML serializer. Agent CLI `inputs` uses `build_inputs_for_pipe` (`pipelex/builder/operations/inputs_ops.py`) which already returns the dict.
-- The API runner receives the already-parsed dict, so this feature is purely client-side parsing — no wire or server change.
+> ⚠️ **Phase-4 heads-up (learned in Phase 2):** the s3/gcp "MissingDependencyError when *selected*" phrasing in the plan is imprecise — the SDK guard is deferred to *use*, not *selection*. Expect the secrets detail doc to carry the same imprecision for SDK-backed secrets providers (Vault/AWS) and treat it the same way (pin *import-light registration*, not a select-time raise). See "Phase 2 — as-built".
-## Decisions (settled with Louis, 2026-07-06)
+**Environment notes for the next session (avoid rediscovering these):**
-- **D1 — extension discrimination.** `.toml` suffix → TOML loader; every other value (including `.json` and extensionless) keeps today's JSON behavior. No content sniffing.
-- **D2 — inline and stdin stay JSON-only.** The `{`-prefix inline path is untouched; inline TOML is ambiguous with file paths. Agent-CLI stdin (flat dict + `working_memory` envelope) stays JSON-only.
-- **D3 — auto-detect both, error on ambiguity.** `run bundle <dir>` auto-detects `inputs.toml` alongside `inputs.json`; if both exist, hard error telling the user to pass `--inputs` explicitly.
-- **D4 — TOML datetimes rejected for now.** `tomli` parses TOML datetime/date/time into Python objects with no JSON equivalent. Adding `DATETIME` to `NativeConceptCode` is deferred to its own track; until then the loader rejects any datetime-typed value with an explicit not-implemented error.
-- **D5 — template generation gets `--format json|toml`** (default `json`) on both surfaces; when `toml` is selected and no explicit output path is given, the default filename becomes `inputs.toml`.
-- **D6 — everything lands in pipelex first.** No `docs/specs/` or `conformance/` edits ride this branch: conformance pins the released pipelex, so its tests can't go green until this ships in a release. Spec + conformance + skills + mthds-js sweep is an explicitly deferred follow-up wave (see "Deferred" below).
+- **Targeted tests for the storage seam:** `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" -o log_level=WARNING --tb=short -q tests/unit/pipelex/plugins/ tests/unit/pipelex/tools/storage/ tests/integration/pipelex/tools/storage/` (Phase 2 also needs `tests/integration/pipelex/plugins/` for the external-plugin harness). But Checkpoint 2 requires the **full** `make agent-test`.
+- **This machine's global `~/.pipelex/pipelex.toml` sets `storage_config.method = "s3"`**, so a plain `Pipelex.make(needs_inference=False)` boot smoke test yields an `S3StorageProvider`, *not* `local` (base + repo `.pipelex/` both say `local`; the global override wins). To smoke a specific method faithfully, pass `config_overrides={"pipelex": {"storage_config": {"method": "local"}}}` in a fresh interpreter. This is a machine/global-config condition, **not** a bug in the seam — don't chase it.
+- **Empirically confirmed** (so Phase 2 boot tests can rely on it): `StorageProviderConfig.method: str = Field(strict=False)` coerces a `StrEnum` input to a plain `str`, accepts an external token (e.g. `"azure"`) at parse, and `StrEnum` registry keys resolve against a plain-str `config.method` in dict lookup.
-## Phases
+> Replaces the retired cookbook hello-plugin tracker (complete; recoverable in git history).
-### Phase 1 — Shared inputs-file loader — DONE
+## What & why (cold-start summary)
-- [x] New module `pipelex/cli/commands/run/_inputs_file_loader.py`: `load_inputs_dict_from_path(path: Path) -> dict[str, Any]`. Suffix match: `.toml` → `load_toml_from_path`; else → `load_json_dict_from_path`. Also hosts the shared auto-detect probe `find_default_inputs_file(directory)` (D3).
-- [x] Datetime guard: recursive walk rejecting `datetime.datetime` / `date` / `time` instances, run on both formats. New `pipelex/cli/commands/run/exceptions.py` with `InputsDatetimeNotSupportedError` and `AmbiguousInputsFilesError` — both carry class-level `error_domain = INPUT` + `user_action` (CHANGE_INPUT), so the agent envelope self-describes and they must NOT get `AGENT_ERROR_HINTS`/`AGENT_ERROR_DOMAINS` entries (enforced by `test_agent_output_drift.py`). Messages include the offending key path (e.g. `record.entries[0].when`) and the quote-as-string workaround.
-- [x] `pipelex-dev generate-error-pages` run — new pages `docs/errors/inputs-datetime-not-supported-error.md`, `docs/errors/ambiguous-inputs-files-error.md` (+ index update).
-- [x] Unit tests: `tests/unit/pipelex/cli/commands/run/test_inputs_file_loader.py` (loads, extension rule, TomlError, datetime top-level/nested/all four TOML datetime flavors, JsonTypeError, multi-line string) and `test_inputs_file_probe.py` (json-only / toml-only / neither / both-ambiguous).
+Promote the two dependency-injection seams that pass all plugin criteria — **storage provider** and **secrets provider** — to formal `pipelex.plugins` entry-point plugins, so third parties can ship `pipelex-storage-<backend>` / `pipelex-secrets-<backend>` packages selected at deploy time. Chosen after a full boot-audit of DI seams; everything else is already a plugin, a boot-orchestrator-owned hub slot, or a population/config point that should stay one.
-### Phase 2 — Wire into both run surfaces — DONE
+**Read these three before starting** (they hold the detailed steps this master plan sequences):
-- [x] Main CLI `_run_core.py`: swapped to the shared loader; new except branches for `TomlError` and `InputsDatetimeNotSupportedError`; also fixed a pre-existing gap — invalid JSON in an inputs *file* (`json.JSONDecodeError`) previously escaped to the generic traceback handler, now gets a friendly message on both surfaces. The `JsonTypeError` wording stays JSON-specific (it can only fire on the JSON branch).
-- [x] Agent CLI `stdin_resolver.py::_parse_inputs_arg`: same swap; `agent_error` branches for `TomlError` / `InputsDatetimeNotSupportedError` / `json.JSONDecodeError`. `AGENT_ERROR_HINTS` + `AGENT_ERROR_DOMAINS` got `TomlError` entries (it has no class-level metadata — kept generic since TomlError also fires outside inputs loading); the two new errors self-describe (see Phase 1).
-- [x] Auto-detect + ambiguity (D3): `DEFAULT_INPUTS_TOML_FILE_NAME` in `conventions.py`; both `bundle_cmd.py` sites probe via the shared `find_default_inputs_file`. The probe is skipped entirely when `--inputs` is passed, so the ambiguity rule only bites on auto-detect (both surfaces).
-- [x] Verified `run method --inputs relative/path.toml` comes free: main-CLI `method_cmd.py` joins `method_dir / inputs_path` preserving the suffix. Also fixed a pre-existing asymmetry (Louis's call): the *agent* CLI `run method` used to treat `--inputs` as cwd-relative; both surfaces now resolve relative `--inputs` file paths against the method dir via the shared `resolve_inputs_arg_against_dir` helper in `_inputs_file_loader.py` (inline JSON and absolute paths pass through).
-- [x] Tests: e2e subprocess dry run with `inputs.toml` + relative `url` resolution (`tests/e2e/pipelex/cli/test_toml_inputs_run.py`, staged csv_demo fixture); main-CLI auto-detect/ambiguity/explicit-bypass (`tests/unit/pipelex/cli/commands/run/test_bundle_cmd_auto_inputs.py`); agent-CLI auto-detect + ambiguity envelope + bypass (`tests/unit/pipelex/cli/agent_cli/test_run_bundle_auto_inputs.py`); `.toml` branch + error envelopes in `test_stdin_resolver.py`; TOML load/resolve + error exits in `test_run_core_execution.py`; method-dir `--inputs` resolution helper (`test_inputs_arg_resolution.py`) + agent `run method` relative-path resolution (`test_run_method_inputs_resolution.py`).
+- `wip/plugins/README.md` — the track's shared decisions + boot-audit rationale + the new mechanism.
+- `wip/plugins/storage-provider-plugin.md` — storage detail (defines the shared mechanism). **Lands first.**
+- `wip/plugins/secrets-provider-plugin.md` — secrets detail (reuses the mechanism; **Decision W already RESOLVED = W-A**).
-**CHECKPOINT 1 — CLEARED 2026-07-06.** TOML inputs load end-to-end on both surfaces; `make agent-check` fully green (pyright/mypy/ruff/keyword-only); the full CLI unit, integration, and e2e test paths pass, including the new subprocess TOML dry run against the real binary. Work is uncommitted — commit before starting Phase 3.
+**The new mechanism (both seams share it):** a *keyed registry + config-selected singleton*. Plugins register N provider factories keyed by an open method token (mirroring `OrchestrationMode`); at boot, core reads one config field (`storage_config.method` / `secrets_config.method`), looks the token up in the registry, and calls the factory to produce the one provider set on the hub. Deliberately **not** a `HubSlot` (those are orchestrator-coupled).
-### Phase 3 — Template generation `--format toml` — DONE
+## Locked decisions (do not re-litigate)
-- [x] `InputStuffSpecs.build_inputs_template() -> dict` exposed; `render_inputs` (both the method and the module-level function in `input_renderer.py`) is now the JSON serializer over it.
-- [x] TOML serializer `serialize_inputs_template_to_toml` in `pipelex/core/pipes/inputs/input_renderer.py` (tomlkit), plus `render_inputs_toml(pipe)` and the `InputsTemplateFormat` StrEnum (`json|toml`). Verified the generator never emits `None` placeholders (Optionals are unwrapped to their inner type), but the pinned defensive policy is: None → `""` recursively, so keys stay visible in the template (`test_inputs_template_toml.py`).
-- [x] Main CLI `pipelex build inputs pipe|bundle|method`: `--format json|toml` (param `template_format`, default `json`); `toml` defaults the output filename to `inputs.toml` (next to bundle / `results/` / method `results/` — all three default-path sites).
-- [x] Agent CLI `pipelex-agent inputs pipe|bundle|method`: same `--format json|toml`; `toml` prints the raw TOML template to stdout via shared `emit_inputs_result`/`emit_no_inputs_result` helpers in agent `_inputs_core.py` (no-inputs case prints a TOML comment line — valid TOML, loads as `{}`). Deviation documented in `pipelex/cli/agent_cli/CLAUDE.md` (output-format section + commands table); `test_inputs_format_unaffected.py` reworked to pin the new contract (InputsTemplateFormat, still no `--error-format`).
-- [x] Round-trip tests: unit (`test_inputs_template_roundtrip.py` — TOML template loads through the Phase-1 loader identically to its JSON twin) and e2e (`test_toml_inputs_build.py` — real binary generates `inputs.toml` from the csv_demo fixture, then a dry run consumes it).
+- **DX-1 — one API bump.** `PLUGIN_API_VERSION` 2→3, done once on this branch. Both seams land before any release, so external plugins (`pipelex-temporal`, `pipelex-mistralai-workflows`) re-declare `targets_api=3` exactly once. Discovery uses strict-equality, so any menu addition is a breaking bump.
+- **DX-2 — unconditional builtins.** `StoragePlugin` and `SecretsPlugin` join `CORE_UNCONDITIONAL_PLUGIN_NAMES` — required infra can't be disabled into a broken boot.
+- **DX-3 — external-provider config surface is a follow-up**, not built here (fixed typed sub-configs only in Phase scope).
+- **D1 / S1 — open method token.** `method` config fields accept an open `str`, validated at registry lookup (unknown → `UnknownStorageMethodError` / `UnknownSecretsMethodError`), not at parse.
+- **W-A (secrets boot ordering) — RESOLVED, refined in Phase 3.** The `# needed for gateway check` comment was stale; the gateway path is secrets-free, so the hardcoded `EnvSecretsProvider()` is gone and secrets is now config-selected from the registry. **Placement deviation (deliberate, see "Phase 3 — as-built"):** the plan's literal W-A said move the `build_registrar` call *above the gateway block* (after line 209 / before old 212). Phase 3 instead placed it *after* the gateway service/terms-check block and *just before the telemetry factory* (secrets' true first consumer). Both satisfy W-A's only hard constraint (secrets resolved before telemetry); the after-the-gate placement additionally preserves precondition-gate-first semantics (an unaccepted-terms / first-run boot fails fast before any discovery work) — a contract two `__new__`-based gateway-terms unit tests encode, which the above-the-gate placement broke.
-**CHECKPOINT 2 — CLEARED 2026-07-06.** Template generation done on both surfaces; `make agent-check` fully green; targeted CLI + core + builder unit/integration/e2e suites pass (incl. the generate→dry-run e2e chain and a manual raw-TOML smoke of the agent CLI). Committed at this boundary.
+## Sequencing
-### Phase 4 — Docs, changelog, final gates — DONE
+Storage vertical (Phases 1–2) → Secrets vertical (Phases 3–4) → Release gating (Phase 5). One coherent commit per phase; a mandatory checkpoint after each.
-- [x] `docs/tools/cli/run.md`: reworked "Input JSON Format" → "Input File Formats" section (extension rule, JSON + TOML examples with multi-line strings, datetime-limitation warning, auto-detect/ambiguity note); all three `--inputs` option lines and the `run bundle` intro updated to mention JSON-or-TOML + inputs-file auto-detect.
-- [x] `docs/building-methods/pipes/provide-inputs.md`: new "Input Files: JSON or TOML" section (extension rule, equivalent JSON/TOML example, datetime-limitation warning, pointer to run CLI reference); `build inputs` tip mentions `--format toml`.
-- [x] `docs/tools/cli/build/inputs.md`: `--format json|toml` on all three subcommands, `--output` default-filename behavior (`inputs.json`/`inputs.toml`), `--format toml` example, TOML output + multiplicity examples; frontmatter/intro reworded.
-- [x] `docs/tools/cli/agent-cli.md`: `run --inputs` JSON-or-TOML + stdin-stays-JSON/auto-detect note; `inputs --format json|toml` deviation note; Output Contract updated for inputs' dual output. (`pipelex/cli/agent_cli/CLAUDE.md` was already updated in Phase 3.)
-- [x] `CHANGELOG.md` under `[Unreleased]`: Added — TOML inputs (extension-discriminated) + `--format toml` template generation; TOML-datetime limitation noted.
-- [x] Full `make agent-check` (pyright/mypy/ruff/plxt/keyword-only all green) + `make agent-test` (full suite green).
+---
-**CHECKPOINT 3 (final) — CLEARED 2026-07-06.** All phases done; gates fully green; docs + changelog landed. Phase 4 changes uncommitted at this boundary — commit, then open the PR to dev. Nothing pushed.
+## ⛔ Checkpoint protocol (MANDATORY at every checkpoint — do not skip, do not merge phases)
-## Deferred — follow-up wave AFTER this ships in a pipelex release (NOT this branch)
+At each `CHECKPOINT`, the agent **must stop** and do all three, in order:
-- **Spec + conformance sync (D6).** Grep `docs/specs/` (workspace root) for the `--inputs` surface (command-surface-map, protocol spec); add the extension rule + `--format toml` to the spec prose and matching conformance tests with the bidirectional `> Verified by:` ↔ `pytest.mark.spec` links. Gated on the conformance venv's pipelex pin reaching the release that carries this feature (same de-gate pattern as the Optionals track).
-- **Skills sweep.** `mthds-plugins` skills (`mthds-run`, `mthds-inputs`) document `inputs.json`; update after release (published plugin, separate repo).
-- **mthds-js check.** Verify `mthds-agent` passes `--inputs` through opaquely (no client-side `.json` assumption) — expected no-op.
-- **DATETIME native concept.** Add `DATETIME` to `NativeConceptCode`, then replace the Phase-1 datetime guard with real support. Own track.
+1. **Verify progress.** Run the gates for the phase and paste real results:
+ - Always: `make agent-check` (pyright/ruff/mypy/plxt/keyword-only) + `make tb` (boot sequence — critical whenever config/registry wiring changed).
+ - Test phases (2, 4) and the final phase: full `make agent-test`. Intermediate phases may scope to the phase's tests but must still pass `agent-check` + `tb`.
+ - If a gate fails: fix before proceeding; a checkpoint is not cleared with a red gate.
+
+2. **Commit, then update this file for cold start.** Commit the phase as one commit. Then edit `TODOS.md` (and/or the linked wip doc) so a brand-new session could resume with no lost context: tick the phase's checkboxes, record the **commit SHA**, any decisions taken or deviations from the linked plan, the current state of the code, and the exact next action. Treat this as a handoff you won't be present to explain.
+
+3. **Fan-out a Sonnet-5 `/code-review` sub-agent — with NO inherited context.** Spawn a **fresh** sub-agent (never a fork) whose entire input is a *pointer to the phase's changes* — the commit SHA / `git diff <base>..HEAD` / the working-tree file list — and nothing else. Do **not** hand it the plan, the rationale, the decisions, or your conclusions; a clean-room review is the point (we want clean solid software, not over-engineering). Spawn template:
+
+ ```
+ Agent(
+ subagent_type: "general-purpose", # fresh context — NOT "fork"
+ model: "sonnet", # Sonnet-5
+ description: "code-review phase N",
+ prompt: "Run the /code-review skill on the changes in commit <SHA> "
+ "(inspect via `git diff <SHA>^..<SHA>` in /Users/lchoquel/repos/Pipelex/_plugins). "
+ "Review ONLY those changes. You have no prior context and should assume none. "
+ "Focus: correctness bugs, and over-engineering / unnecessary abstraction / "
+ "speculative generality. Report findings ranked most-severe first; "
+ "if nothing substantive, say so plainly."
+ )
+ ```
+
+ Then **triage** the findings: apply genuine bug/simplification fixes in a follow-up commit; capture design-tradeoff findings (not silent bugs) as a deferred note under `wip/plugins/` rather than reflexively applying the convenient fix. Record the triage outcome in this file. Only then move to the next phase.
+
+---
+
+## Phase 1 — Storage: mechanism + builtin plugin + boot wiring
+*(Detail: `wip/plugins/storage-provider-plugin.md` Phases 1–3. Delivers a working config-selected storage plugin.)*
+
+- [x] `PLUGIN_API_VERSION` 2→3 in `pipelex/plugins/contract.py` (the one batched bump — DX-1).
+- [x] Registrar (`pipelex/plugins/registrar.py`): `storage_providers` dict field + `add_storage_provider(*, method, factory)` via the `_add` helper; `StorageProviderFactoryFn` alias.
+- [x] New `pipelex/plugins/storage_provider_registry.py`: `StorageProviderRegistry` (mirror `OrchestratorRegistry`: `get_optional`/`get_required`/`has`/`methods`).
+- [x] New `UnknownStorageMethodError` in `pipelex/plugins/exceptions.py`.
+- [x] Hub (`pipelex/hub.py`): `set_/get_storage_provider_registry` + module accessor.
+- [x] New `pipelex/plugins/storage/storage_plugin.py`: `StoragePlugin` (`name="storage"`) + module-level factory closures (`_make_local/_in_memory/_s3/_gcp_storage_provider`) holding the exact bodies moved from the deleted `make_storage_provider_from_config` `match` arms (keep `lazy_validate()`; keep GCP's `get_secrets_provider()` read — legal at the boot apply-point).
+- [x] `pipelex/plugins/builtins.py`: register `StoragePlugin()`; add `"storage"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES`.
+- [x] Boot (`pipelex.py`): build + `set_storage_provider_registry(...)` alongside the other registries; replace the `make_storage_provider_from_config` block with registry selection (explicit param > `registry.get_required(method=config.method)(config)`).
+- [x] Delete `pipelex/tools/storage/storage_provider_factory.py` (grep for other importers first — and re-home them).
+- [x] D1: relax `StorageProviderConfig.method` to open `str`.
+
+### Phase 1 — as-built (deviations & decisions, for cold start)
+
+- **Boot ordering (resolved a plan ambiguity).** The old factory ran at `pipelex.py:307-310`, *before* `build_registrar` (~393), but the registry only exists after `build_registrar`. So storage selection **moved down** to sit right beside the other registry constructions (after the 4 `set_*_registry` calls): build `StorageProviderRegistry(plugin_registrar.storage_providers)` → `set_storage_provider_registry` → `if storage_provider is None: select via registry` → `set_storage_provider`. Verified safe: no `get_storage_provider()` consumer runs during `setup()` before that point (all consumers are run-time), and secrets is on the hub (line 306) so the GCP factory's secret read works. This is byte-equivalent to the old factory and forward-compatible with the Phase 3 W-A move (which relocates only the `build_registrar` *call*, not the storage registry construction).
+- **`DuplicateStorageProviderError` added** (not in the plan checklist but required by the `_add` `on_duplicate` contract; mirrors `DuplicateOrchestratorError`).
+- **D1 blast radius.** `method: StorageMethod` → `method: str = Field(strict=False)` forced a `case _` in `storage_path` + `uri_format` (and the model-validator) because `reportMatchNotExhaustive` fires on open `str`. Empirically confirmed the field coerces a `StrEnum` input to a plain `str` and accepts an external token (e.g. `"azure"`) at parse. Same `case _` added to the two test-side matches (`test_storage_config.py`, `generator_fixtures.py`).
+- **Test re-homing.** `test_storage_provider_factory.py` **deleted** (its SUT is gone; Phase 2 adds the registry + parametrized-boot seam tests that supersede it). `generator_fixtures.py` now selects through `get_storage_provider_registry().get_required(...)`. `test_storage_provider_config.py`'s strict-coercion test rewritten to the D1 open-token behavior (`method == "local"` as plain str; `"azure"` accepted at parse).
+- **Built-in tokens** registered as the `StorageMethod` enum values (`StrEnum` keys are interchangeable with their plain-str form in dict lookup, so boot's plain-str `config.method` resolves them).
+
+### ⛔ CHECKPOINT 1 — run the protocol above (verify → commit + update → clean-room /code-review → triage)
+- [x] Gates green: `make agent-check` (ruff/plxt/pyright/mypy/keyword-only all pass), `make tb` (9 passed), `pipelex plugins list` shows `storage` (builtin, API 3, 4 contributions), boot `method="local"`→`LocalStorageProvider` / `method="in_memory"`→`InMemoryStorageProvider`, unknown token → `UnknownStorageMethodError`. Targeted `tests/unit/pipelex/plugins/ tests/unit+integration/.../tools/storage/` = 1036 passed; re-homed fixture consumers = 4 passed.
+- [x] Commit SHA recorded here: `8268ff08f`
+- [x] Cold-start state updated in this file.
+- [x] Sonnet-5 clean-room `/code-review` fanned out on `8268ff08f`; findings triaged. **Outcome:** review verified correctness (byte-identical factory bodies, safe `StorageMethod→str` D1 relaxation, correct boot ordering, keyword-only compliant, no bare `except`). 3 findings:
+ 1. *contract.py v3 comment implied secrets registry already exists* → **FIXED** in follow-up `58961848a` (reworded to "pre-reserved / lands in a follow-up").
+ 2. *storage-selection path has no direct automated test after deleting the factory test* → **deferred to Phase 2 as already planned** (Phase 2 checklist adds registry hit/miss + duplicate + parametrized-boot). Manually smoke-verified this phase.
+ 3. *`StorageProviderRegistry.get_optional`/`has` have no caller* → **kept (no change).** Confirmed by grep that all sibling read-views (orchestrator/bundle_validator/model_lister) expose the identical shape; `has` is family-wide unused and `get_optional` used by exactly one. It's an established, plan-mandated registry-family convention, not speculative surface — trimming would make storage the odd one out.
+ Reviewer also noted a benign boot-ordering nuance: a bad storage config now fails *after* `models_manager.setup()` rather than immediately after secrets (necessary consequence of needing the registrar built first; behavior otherwise byte-equivalent).
+
+---
+
+## Phase 2 — Storage: tests + docs
+*(Detail: `wip/plugins/storage-provider-plugin.md` Phases 4–5.)*
+
+- [x] Unit: `StorageProviderRegistry` hit/miss (`UnknownStorageMethodError`) + duplicate-registration fail-loud via `_add`. → `test_storage_provider_registry.py`.
+- [x] Built-in factories: each built-in `method` yields the right provider (registry selection); gcp reads its credentials from the hub secrets provider. → `test_storage_plugin.py`. *(Reframed from "parametrized boot" to direct factory-selection — see as-built; the boot-select line is covered end-to-end by the integration test.)*
+- [x] Integration: an external test plugin registering a fake `method` (entry-point discovered) is selectable via config, and lands on the hub. → `test_storage_external_plugin.py`.
+- [x] Import-light registration pinned: s3/gcp factories register without importing their SDK. → `test_import_light_boot.py` (added `google.cloud.storage` to BLOCKED + assert `storage_providers`).
+- [x] New `docs/under-the-hood/storage-provider-plugins.md` (mirrors `orchestrator-plugins.md`); mkdocs nav (both blocks); cross-links from `inference-backend-plugins.md` Related. *(No "the plugin seams are …" enumeration exists to update — grep-verified.)*
+- [x] CHANGELOG `[Unreleased]`: "breaking" `PLUGIN_API_VERSION` 2→3; storage is now a plugin seam. *(Added a fresh `[Unreleased]` — none existed, branch base was v0.37.0.)*
+
+### Phase 2 — as-built (deviations & decisions, for cold start)
+
+- **The s3/gcp SDK guard is deferred to *use*, not *selection* (plan text was imprecise).** The plan/detail-doc said "s3/gcp arms raise `MissingDependencyError` only when *selected* with the SDK absent." Not so: `S3StorageProvider.__init__` / `GcpStorageProvider.__init__` only store fields; the `find_spec` guard lives in `_get_session` / `_get_bucket`, invoked inside the I/O methods. So *selecting* (constructing) s3/gcp NEVER raises `MissingDependencyError` even with the SDK absent — the error is deferred to first load/store. The correct invariant Phase 2 pins is therefore **import-light registration** (the s3/gcp factories register without importing their SDK), done by extending `test_import_light_boot.py`. The `MissingDependencyError`-on-use path is already covered by the dedicated `test_{s3,gcp}_storage_provider.py`, so it is not re-tested in the seam tests (avoids redundancy). Both aioboto3 and google-cloud-storage happen to be installed in this dev venv, so an in-process "SDK absent" test would need to mock `find_spec` anyway.
+- **Phase-1 latent breakage found + fixed (Phase-1 escapee).** `tests/integration/pipelex/system/test_hub_slot_injection_precedence.py` booted with an empty fake registrar and no explicit `storage_provider`; Phase 1's new selection path (`get_required(method=config.method)`) now rejects that with `UnknownStorageMethodError`. Checkpoint 1 only ran the plugins/storage dirs, not `system/`, so it escaped — the full `make agent-test` at Checkpoint 2 caught it. Fixed by injecting a dependency-free `InMemoryStorageProvider()` in the four boots (this suite tests hub slots, not storage). Grep-verified it was the *only* test with the empty-registrar-boot pattern (`test_plugin_discovery` / `test_import_light_boot` call `build_registrar` directly, never `Pipelex.make`, so they're unaffected).
+- **Test layout** (one TestClass per module): registry read-view + duplicate (`test_storage_provider_registry.py`, mirrors `test_bundle_validator_registry.py`); built-in factories + gcp secret-wiring (`test_storage_plugin.py`); integration discovery→selection→hub (`test_storage_external_plugin.py`). Config helpers reused from `tests/unit/pipelex/tools/storage/test_storage_provider_config.py` (`make_{local,in_memory,s3,gcp}_config`) — local construction touches no filesystem (no tmp_path needed).
+- **Integration harness** = `mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[SimpleNamespace(name=..., load=lambda: FakePluginClass)])` (discovery instantiates a callable/class), keep the real `BUILTIN_PLUGINS`, boot `Pipelex.make(needs_inference=False, config_overrides={"pipelex": {"storage_config": {"method": "test_mem"}}})`, assert `get_storage_provider()` is the fake + registry `.has(method="test_mem")`. An external method token loads fine (the config validator's `case _: pass`). Per-module autouse `reset_pipelex_config_fixture` (teardown-first) overrides the global module fixture (mirror of the two existing `system/` boot-test modules). This is also the harness template for the Phase 4 secrets external-plugin test.
+- **GCP secret-wiring assertion (Phase-2 review triage).** The first cut only asserted the secret was *requested*; the clean-room reviewer flagged it didn't assert the *value* flowed. Strengthened via `mocker.patch(..., wraps=GcpStorageProvider)` + `assert_called_once_with(credentials_file_path=fake_secrets.credentials_path, …)` — the repo-idiomatic way (`reportPrivateUsage`/`SLF001` both hard-block a `provider._credentials_file_path` read, so no private access / suppressions).
+
+### ⛔ CHECKPOINT 2 — CLEARED
+- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), **full `make agent-test` (exit 0, "All tests passed")**. Targeted storage-seam + at-risk boot test also green in isolation.
+- [x] Commit SHAs: `f08193e81` (Phase 2 code/tests/docs/changelog) + `37a3b72f1` (review triage).
+- [x] Cold-start state updated in this file.
+- [x] Sonnet clean-room `/code-review` fanned out on `f08193e81`; findings triaged. **Outcome:** 2 findings, **both genuine and applied** in `37a3b72f1`. (1) *High — doc/code contradiction:* the new page reasserted "the same mechanism backs the secrets provider seam" as present tense, but secrets is not yet registry-selected (and commit `58961848a` had just retracted the same claim in `contract.py`) → reworded to planned/future. (2) *Medium — test gap:* the gcp test checked the secret was requested but not that its value reached the provider → added the wraps-ctor `assert_called_once_with` above. Reviewer otherwise verified: no correctness bugs, no over-engineering, mocks faithful to the real contracts, no tautological assertions, no redundancy across the three new files, all doc snippets match source, MkDocs/pytest conventions respected.
+
+---
+
+## Phase 3 — Secrets: mechanism + config + builtin plugin + boot wiring (W-A)
+*(Detail: `wip/plugins/secrets-provider-plugin.md` Phases 1–3. W-A is locked — this is a build instruction, not an investigation.)*
+
+- [x] Registrar: `secrets_providers` dict + `add_secrets_provider(*, method, factory)` via `_add`; `SecretsProviderFactoryFn` alias. *(API already at 3 from Phase 1 — no second bump.)*
+- [x] New `pipelex/plugins/secrets_provider_registry.py`: `SecretsProviderRegistry`.
+- [x] New `UnknownSecretsMethodError` in `pipelex/plugins/exceptions.py`. *(Also added `DuplicateSecretsProviderError` — required by the `_add` `on_duplicate` contract, mirroring storage.)*
+- [x] Hub: `set_/get_secrets_provider_registry` + module accessor.
+- [x] New config `SecretsProviderConfig` (`pipelex/tools/secrets/secrets_config.py`), `method: str = Field(strict=False)`, **no class default**. Placed as sibling of `storage_config` under `pipelex.*` (single flat model, no `SecretsConfig` subclass needed — secrets has no is_fetch/is_upload analogue).
+- [x] `pipelex/pipelex.toml`: added `[pipelex.secrets_config] method = "env"`; wired `secrets_config: SecretsProviderConfig` into the `Pipelex` config model. Same real-valued block added to `.pipelex/pipelex.toml` override. `make tb` green.
+- [x] New `pipelex/plugins/secrets/secrets_plugin.py`: `SecretsPlugin` + `_make_env_secrets_provider(config)` → `EnvSecretsProvider()` (unused `config` param carries `# noqa: ARG001`, the plugins-dir convention for a callback-signature-conforming factory).
+- [x] `builtins.py`: registered `SecretsPlugin()` in `BUILTIN_PLUGINS`; added `"secrets"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES`.
+- [x] **W-A boot edit** (`pipelex.py`): `build_registrar` + boot-orchestrator gate + secrets registry/selection now sit **just before the telemetry factory** (not above the gateway block — see as-built). Downstream registry constructions still reference the same `plugin_registrar`.
+- [x] **Deleted** the stale `# needed for gateway check` + hardcoded `EnvSecretsProvider()`; replaced with registry selection (explicit `setup(secrets_provider=...)` still wins). Dropped the now-unused `EnvSecretsProvider` import from `pipelex.py`.
+
+### Phase 3 — as-built (deviations & decisions, for cold start)
+
+- **W-A boot placement refined — after the precondition gate, not above the gateway block (the one real deviation).** The plan's literal W-A said move `build_registrar(config=get_config())` up to *before the gateway service block* (after line 209 / old line 212). Doing exactly that broke two unit tests — `tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py::{test_first_run_raises_inference_setup_required, test_needs_inference_true_raises_when_terms_not_accepted}`. Those tests exercise the gateway terms/first-run gate in isolation via `Pipelex.__new__(Pipelex)` (skipping `__init__`, so no config/hub is set) and rely on `setup()` reaching the terms-check `raise` **before** touching `get_config()` / `self.pipelex_hub`. Moving `build_registrar(config=get_config())` above the gate made `setup()` call `get_config()` first → `RuntimeError: Config instance is not set`. **Resolution:** placed the whole plugin-discovery + boot-orchestrator gate + secrets-registry/selection block **after** the gateway service/terms-check block and **immediately before the telemetry factory** — telemetry (`make_telemetry_manager`) is the *true first consumer* of `secrets_provider`, which is W-A's only hard constraint. This is strictly better than the plan's literal placement: the terms/first-run precondition gate fails fast before any discovery work, and no `__new__` test needed changing for the gate. (The boot-orchestrator gate also moved to this spot — still far earlier than its original post-`models_manager.setup` position, so fail-fast is preserved.)
+- **`DuplicateSecretsProviderError` added** (not in the plan checklist) — required by the `_add` `on_duplicate` contract; mirrors `DuplicateStorageProviderError` / `DuplicateOrchestratorError`.
+- **`SecretsProviderConfig` is a single flat model**, not a base+subclass like storage's `StorageProviderConfig`/`StorageConfig`. Storage's subclass exists only to add `is_fetch_remote_content_enabled`/`is_upload_local_content_enabled`; secrets has no such extra fields, so a lone `SecretsProviderConfig(ConfigModel)` with just `method` is the clean-solid shape (no speculative base).
+- **Phase-1 escapee proactively found + fixed (same class as storage's).** `test_hub_slot_injection_precedence.py` boots a fake **empty** registrar (patched into `build_registrar`) and passed an explicit `storage_provider` but no `secrets_provider`; the new secrets selection path (`get_required(method="env")`) rejects the empty registry with `UnknownSecretsMethodError`. Fixed by injecting `EnvSecretsProvider()` in the four boots (+ docstring updated). Grep-verified this was the only `Pipelex.make` boot with a mocked empty/partial registrar (the plugins-dir tests call `build_registrar` directly; `test_storage_external_plugin` keeps the real `BUILTIN_PLUGINS`, so its `env` secrets provider is present). Caught here via the full `make agent-test` rather than escaping to a later checkpoint.
+- **The env factory ignores config.** `_make_env_secrets_provider(config)` returns a bare `EnvSecretsProvider()`; the `config` param exists only to conform to `SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], ...]` (an SDK-backed external provider would read its own settings from it). Marked `# noqa: ARG001`, the established plugins-dir convention (see linkup/bedrock/azure_rest workers).
+
+### ⛔ CHECKPOINT 3 — CLEARED
+- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), **full `make agent-test` (exit 0, "All tests passed")**, `check-config-sync` PASSED (after the triage fix). Manual smokes: `pipelex plugins list` shows `secrets` (builtin, API 3, `secrets provider env`); default `method="env"`→`EnvSecretsProvider`; unknown token→`UnknownSecretsMethodError`.
+- [x] Commit SHAs: `8a0b32668` (Phase 3 code) + `fdf3e114b` (review triage).
+- [x] Cold-start state updated in this file.
+- [x] Sonnet clean-room `/code-review` fanned out on `8a0b32668` (fresh general-purpose agent, no context, pointed only at the commit); findings triaged. **Outcome:** reviewer confirmed **no correctness bug in the boot-order move** (traced every consumer: `build_registrar` is pure, runs after the gateway/terms gate and before the telemetry factory, its only pre-hub consumer; `UnknownBootOrchestratorError` moving earlier is a correct fail-fast improvement). 4 findings:
+ 1. *High — kit-config drift:* `pipelex/kit/configs/pipelex.toml` (the `pipelex init` template) missing the new `[pipelex.secrets_config]` block ⇒ `make check-config-sync` fails (CI + `make check` gate; **not** in `agent-check`, which is why Checkpoint-3 gates missed it). → **FIXED** in `fdf3e114b` via `make up-kit-configs`. *(Reconciled against the `worker_scopes` "not in kit copies" false-positive memory: that gate compares kit-configs ↔ `.pipelex` only, not the base `pipelex.toml`; a net-new section in `.pipelex` genuinely diverges — real, not a false positive.)*
+ 2. *Medium — stale comment:* the secrets-hub-set comment still said the registrar is "built below," but the reorder moved that build above it. → **FIXED** in `fdf3e114b` (reworded).
+ 3. *Over-engineering flag (registry+plugin+config+exceptions for a one-impl seam)* → **kept, no change.** Reviewer explicitly called it "consistent architecture applied deliberately across provider verticals, not a bug" — it is the plan-mandated byte-for-byte mirror of the accepted storage vertical.
+ 4. *Docs + dedicated tests missing* → **deferred to Phase 4 as planned** (Phase 3 = mechanism/boot; Phase 4 = tests + docs, exactly as storage split 1↔2).
+
+---
+
+## Phase 4 — Secrets: tests + docs
+*(Detail: `wip/plugins/secrets-provider-plugin.md` Phases 4–5.)*
+
+- [x] Unit: `SecretsProviderRegistry` hit/miss (`UnknownSecretsMethodError`) + duplicate fail-loud. → `test_secrets_provider_registry.py` (byte-for-byte mirror of `test_storage_provider_registry.py`).
+- [x] Boot: default config yields `EnvSecretsProvider` on the hub; explicit `setup(secrets_provider=...)` overrides. → `test_secrets_external_plugin.py` (first two methods). Plus a `test_secrets_plugin.py` unit (plugin registers the `env` factory; factory constructs `EnvSecretsProvider`), mirroring `test_storage_plugin.py` but lean (single builtin, no optional-dep matrix).
+- [x] Integration: external test plugin registering a fake secrets `method` (`"test_secret"`, entry-point discovered) is selectable via config. → `test_secrets_external_plugin.py`.
+- [x] Ordering guard: a boot with a non-env secrets method has secrets on the hub before storage's GCP arm reads it. → `test_secrets_external_plugin.py::test_config_selected_secrets_provider_is_read_by_storage_gcp_arm_at_boot`.
+- [x] New `docs/under-the-hood/secrets-provider-plugins.md` (emphasizes the lazy optional-dep closure for SDK-backed Vault/AWS providers + the boot-ordering relationship); mkdocs nav (both blocks); cross-links (storage↔secrets both directions, inference-backend Related, secrets-provider-injection); updated the storage page's stale "planned" forward-ref.
+- [x] CHANGELOG `[Unreleased]`: secrets is now a plugin seam; folded `add_secrets_provider` into the existing `PLUGIN_API_VERSION` 2→3 note.
+
+### Phase 4 — as-built (deviations & decisions, for cold start)
+
+- **The ordering guard is stronger than "assert boot order" implied — telemetry ALSO consumes the config-selected secrets provider, earlier.** First cut asserted `requested_secret_ids == ["GCP_CREDENTIALS_FILE_PATH"]` (exact). It failed: the recorded list led with `POSTHOG_ENDPOINT`, `POSTHOG_API_KEY`, … — the **telemetry factory** reads secrets from the same provider *before* storage is selected (secrets resolves right before the telemetry factory; storage much later). This is extra confirmation of correct ordering, not a bug. Fixed to a **membership** check: `"GCP_CREDENTIALS_FILE_PATH" in requested_secret_ids`. That token is requested **only** by storage's gcp factory (grep-confirmed), so its presence uniquely proves the gcp arm read the external provider from the hub. (Telemetry reads secrets even with `needs_inference=False` — it resolves the POSTHOG keys to build the manager, then the disabled path ignores them; the fake returning a canned gcp path for POSTHOG_* doesn't break boot.)
+- **The gcp override needs BOTH `bucket_name` and `project_id`.** Base `pipelex.toml` ships `[pipelex.storage_config.gcp]` with `bucket_name = ""` **and** `project_id = ""`; `StorageGcpConfig.lazy_validate()` (run inside the gcp factory) rejects either empty. So the ordering-guard boot passes `config_overrides={"pipelex": {"storage_config": {"method": "gcp", "gcp": {"bucket_name": "test-bucket", "project_id": "test-project"}}}}` (deep-merge fills the rest). `GcpStorageProvider.__init__` only stores fields (the `find_spec`/creds-file check lives in `_get_bucket`), so constructing it at boot with a fake creds path is import-light and does no I/O — safe even though `google-cloud-storage` is installed here.
+- **Default-env boot is deterministic on this branch.** `test_default_config_yields_the_env_provider_on_the_hub` boots with no `secrets_config` override and asserts `env`. Verified the machine global `~/.pipelex/pipelex.toml` has **no** `secrets_config` block (it's net-new here), so unlike storage's `method=s3` global-override gotcha, secrets has no machine override to fight — default→`env` holds locally and in CI.
+- **Import-light test NOT extended for secrets (deliberate, unlike storage Phase 2).** `test_import_light_boot.py` gained a storage assert because s3/gcp are SDK-backed and BLOCKED there — a real import-light risk. The only builtin secrets method is `env` (no SDK), so an added `assert registrar.secrets_providers` would restate the registry unit test, not pin an import-light invariant. Left untouched to avoid over-engineering; an external SDK-backed secrets plugin would carry its own import-light guarantee.
+- **Test module layout** (one TestClass per module): `test_secrets_provider_registry.py` (registry read-view + duplicate), `test_secrets_plugin.py` (builtin plugin + env factory), `test_secrets_external_plugin.py` (integration: default/override/external/ordering — one class, four methods). Integration harness identical to `test_storage_external_plugin.py`: `mocker.patch("pipelex.plugins.discovery._external_entry_points", …)` keeps the real `BUILTIN_PLUGINS`, per-module autouse teardown-first `reset_pipelex_config_fixture`. The fake external provider is a `SecretsProviderAbstract` subclass recording requested secret ids.
+- **The Phase-3 escapee stayed fixed.** `test_hub_slot_injection_precedence.py` (fixed in Phase 3 to inject `EnvSecretsProvider()` in its four boots) passed under the full suite — no new escapee surfaced.
+
+### ⛔ CHECKPOINT 4 — CLEARED
+- [x] Gates green: `make agent-check` (ruff/plxt/pyright 0 errors/mypy/keyword-only), `make tb` (9 passed), `check-config-sync` PASSED, **full `make agent-test` (exit 0, "All tests passed")**. Targeted secrets-seam trio green in isolation (11 passed).
+- [x] Commit SHA recorded here: `65603e912`
+- [x] Cold-start state updated in this file.
+- [x] Sonnet-5 clean-room `/code-review` fanned out on `65603e912` (fresh general-purpose agent, no context, pointed only at the commit). **Outcome: no substantive findings — nothing to fix, nothing to defer.** The reviewer ran all three test files against the tree (11 passed) and traced every factual claim to source: confirmed the secrets-before-telemetry-before-storage boot ordering in `pipelex.py`, the `gcp` factory's lazy credential check (fake creds path never touches disk / needs no GCP SDK at boot), telemetry's always-on `${POSTHOG_*}` secret resolution (validating the ordering-guard's membership-check rationale), `PLUGIN_API_VERSION` still `3` (changelog correctly folds in, no second bump), and identical nav placement in both mkdocs blocks. Verdict: assertions are concrete and non-tautological (the ordering guard is a genuine regression guard — reordering secrets-after-storage would raise or read the wrong provider inside the gcp factory), no over-engineering (secrets tests deliberately leaner than storage's — no optional-dep matrix, accurate to the single-impl `env` seam), no doc/code contradictions, all cross-links resolve.
+
+---
+
+## Phase 5 — Release gating & cross-repo (documentation in this branch; execution is release-gated)
+
+- [x] Record in the CHANGELOG / release notes the cross-repo `targets_api=3` obligation. **Corrected during Checkpoint-5 triage (F2):** the original wording ("they register no provider; the bump is the only change") was wrong for `pipelex-temporal`. `pipelex-mistralai-workflows` is bump-only, but `pipelex-temporal`'s payload-codec factory imports the now-removed `make_storage_provider_from_config`, so it needs the `targets_api` bump **and** a code migration to `get_storage_provider_registry().get_required(...)`. CHANGELOG now says this + carries a `### Removed` bullet for the deleted factory. Did NOT touch those repos here.
+- [x] Confirmed the whole branch is one clean per-phase commit sequence (feat/test → triage → bookkeeping per phase; `git log 04434f785..HEAD`). Opening the PR to the intended base is **left to Louis** (nothing pushed; no upstream).
+
+### ⛔ CHECKPOINT 5 (final) — CLEARED
+- [x] Gates green on the final tip: `make agent-check` (ruff/plxt/pyright 0 errors/mypy 2084 files/keyword-only), `make tb` (9 passed), `make check-config-sync` (passed), **full `make agent-test` (exit 0, "All tests passed")** — run both before the triage (tip `65603e912`+bookkeeping) and again after the triage commit `0bbc39474`.
+- [x] Final Sonnet-5 clean-room `/code-review` over the **whole-branch diff** (`git diff 04434f785..HEAD`) — fresh general-purpose agent, no inherited context, 10 finder angles + line-by-line + a test-quality pass. **Outcome: migration verified largely clean and faithful** (behavior-preserving factory→plugin split, correct secrets→hub→telemetry→storage→gcp-reads-secrets boot ordering, safe `StorageMethod`→`str` widening, conventions honored). 9 findings, all triaged in `0bbc39474`:
+ - **F2 [MED-HIGH] — FIXED (genuine, was actively misleading):** deleting `make_storage_provider_from_config` breaks `pipelex-temporal` at import (its codec factory calls it); CHANGELOG claimed a `targets_api` bump was temporal's only change. Corrected CHANGELOG + wip doc + added Removed bullet.
+ - **F6 [LOW] — FIXED (simplification):** removed dead `get_optional` from both registries (no production consumer; `get_required` doesn't delegate to it; only its own unit tests used it). This reverses Checkpoint-1's keep-decision, which rested on a factual misread — siblings expose `get_optional`+`has`, while storage/secrets hard-fail via `get_required`+`methods`, so `get_optional` was a vestigial mechanical copy, not a convention match.
+ - **F3 [MED] — FIXED:** added symmetric storage injection-precedence test (was tested for secrets only).
+ - **F5 [LOW] — FIXED:** parametrized `test_disabling_core_unconditional_plugin_raises` over `direct/storage/secrets/openai`.
+ - **F8 [LOW] — FIXED:** `contract.py` v3 comment no longer says `add_secrets_provider` is "in a follow-up commit" (it landed in Phase 3).
+ - **F9 [LOW] — FIXED:** `orchestrator-plugins.md` no longer claims `PLUGIN_API_VERSION` "is now 2".
+ - **F1 [HIGH per reviewer] — DEFERRED to D3 (design tradeoff, not a silent bug):** external storage method boots/selects cleanly but crashes on the first *generated-content* store because `uri_format` is defined only for the built-in methods (raises loudly on `case _:`). External config surface is explicitly the D3 follow-up; strengthened the D3 note + added a docs-page caveat. External providers are usable today for their own storage API, not yet as the generated-content backing store.
+ - **F7 [LOW] — KEPT:** `Field(strict=False)` on the `method` fields is cosmetically vestigial but removal risks a config-parse regression at enum-member call sites for zero benefit.
+ - **F4 [MED] — SKIPPED:** a `lazy_validate` negative-path factory test (trivial 3-line wiring; per-config validation already covered) — avoided test bloat.
+- [x] This file updated to DONE with all SHAs; deferred follow-ups (D3 external config passthrough incl. the `uri_format` gap, first real external provider, cookbook example) captured under `wip/plugins/`.
+
+---
+
+## Deferred follow-ups (not in this plan — see wip docs)
+
+- DX-3 external-provider config passthrough (storage D3 / secrets S4).
+- First real external provider (e.g. Vault / AWS Secrets Manager) as proof-of-seam — likely a private package if hosted-platform-targeted.
+- Cookbook `pipelex-storage-hello` external-plugin example, mirroring the hello-inference-plugin track.
diff --git a/docs/under-the-hood/inference-backend-plugins.md b/docs/under-the-hood/inference-backend-plugins.md
index 0e641a325..57c2b053d 100644
--- a/docs/under-the-hood/inference-backend-plugins.md
+++ b/docs/under-the-hood/inference-backend-plugins.md
@@ -241,6 +241,9 @@ The SPI is a documented, versioned **module/symbol list** gated by `PLUGIN_API_V
## Related
+- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, riding the same discovery/denylist machinery
+- [Storage Provider Plugins](storage-provider-plugins.md) — the config-selected-singleton seam (storage backend by `storage_config.method`)
+- [Secrets Provider Plugins](secrets-provider-plugins.md) — the config-selected-singleton seam (secrets backend by `secrets_config.method`)
- [Pipe Routing & Execution](pipe-routing-and-execution.md) — where worker construction sits in the run path
- [Error Model](error-model.md) — how these errors render and dereference
- [Architecture Overview](architecture-overview.md)
diff --git a/docs/under-the-hood/orchestrator-plugins.md b/docs/under-the-hood/orchestrator-plugins.md
index 879beb20f..ff65234f4 100644
--- a/docs/under-the-hood/orchestrator-plugins.md
+++ b/docs/under-the-hood/orchestrator-plugins.md
@@ -119,7 +119,7 @@ The contract is deliberately split so no layer overreaches:
- **Core** owns *transport* — `registrar.get_http_error_mappers()` runs every provider, builds the `{exc_type: mapper}` dict, and is fail-loud on a duplicate *resolved* exception type (naming both plugins). `ErrorReport` is a core type, so the seam carries **no** web-framework import.
- **The host runtime** owns *presentation* — at app construction it iterates the resolved mappers and wraps each into one framework error handler (FastAPI, …) that runs the mapper, then renders the `ErrorReport` through its own RFC 7807 + `DisclosureMode` path. FastAPI / Starlette stays only in the host; core and the plugin import neither.
-This is what lets the public `pipelex-api` base be orchestrator-agnostic and still render a Temporal (or Mistral) transport fault correctly: install the flavor's plugin and its mapper rides in; install none and there is simply nothing to wrap. The capability is optional, so it grew the plugin contract by one method → `PLUGIN_API_VERSION` is now **2**.
+This is what lets the public `pipelex-api` base be orchestrator-agnostic and still render a Temporal (or Mistral) transport fault correctly: install the flavor's plugin and its mapper rides in; install none and there is simply nothing to wrap. The capability is optional, contributing one method to the plugin contract.
---
diff --git a/docs/under-the-hood/secrets-provider-plugins.md b/docs/under-the-hood/secrets-provider-plugins.md
new file mode 100644
index 000000000..06e49ebfb
--- /dev/null
+++ b/docs/under-the-hood/secrets-provider-plugins.md
@@ -0,0 +1,169 @@
+---
+title: "Secrets Provider Plugins"
+description: "How Pipelex selects a secrets backend by config, the keyed-registry-plus-config-selected-singleton seam a secrets plugin rides, and how to author an out-of-tree pipelex-secrets-<backend> package."
+---
+
+# Secrets Provider Plugins
+
+Every credential Pipelex reads — an API key, a service-account file path, a signing secret — is resolved through a single **secrets provider** set on the hub at boot. Which provider that is comes entirely from data: one config field, `secrets_config.method`, names a backend, and a **secrets plugin** is what teaches Pipelex how to build the provider for that method.
+
+Core names no secrets backend by import or by string. The built-in `env` provider (reads secrets from environment variables) is a plugin too — the always-on `SecretsPlugin` — riding the exact same seam an out-of-tree `pipelex-secrets-<backend>` package (Vault, AWS Secrets Manager, GCP Secret Manager, …) would. This page documents that seam, the factory contract a plugin registers, and how to write one.
+
+This is the second application of the mechanism the [storage provider](storage-provider-plugins.md) seam introduced; secrets reuses it wholesale, so the two pages describe the same shape with the nouns swapped.
+
+---
+
+## A different shape from the inference/orchestrator seams
+
+The [inference-backend](inference-backend-plugins.md) and [orchestrator](orchestrator-plugins.md) registries are selected **per call** — a model's `sdk`, a request's `orchestration_mode`. Secrets is not: it is a **process-global singleton selected by its own config key**, independent of the orchestrator. So it rides the same **keyed registry + config-selected singleton** mechanism storage uses:
+
+> Plugins register N provider factories into a registry keyed by an open `method` token. At boot, core reads `secrets_config.method`, looks that token up in the registry, and calls the factory to produce the one provider set on the hub.
+
+This is deliberately **not** a hub slot (those are orchestrator-coupled, claimed only when `boot_orchestrator == plugin.name`). Secrets selection has nothing to do with the orchestrator, so it gets its own registry and its own config key — the sibling of `storage_config.method`.
+
+---
+
+## The seam in one view
+
+```
+boot (Pipelex.setup)
+ └─ build_registrar(config) # pure, import-light
+ ├─ for each plugin in BUILTIN_PLUGINS (SecretsPlugin is one)
+ └─ for each installed "pipelex.plugins" entry point
+ └─ plugin.register(registrar) # side-effect-free
+ └─ registrar.add_secrets_provider(method=…, factory=…)
+ └─ SecretsProviderRegistry(registrar.secrets_providers) # stored on the hub
+ └─ secrets_provider = registry.get_required(method=secrets_config.method)(secrets_config)
+ └─ set_secrets_provider(secrets_provider) # the one provider every consumer reads
+```
+
+There is no `match secrets_config.method:` anywhere in boot — the token set is open, so validation *is* the registry lookup. Adding a backend means registering a factory for its token; nothing in core changes. Every downstream consumer (`get_secret(...)`, telemetry credentials, the storage `gcp` arm, search/extract workers) keeps calling `get_secrets_provider()` and is unaffected by which method was selected.
+
+### Where in boot secrets is resolved
+
+Secrets is resolved **early** — right after plugin discovery, after the gateway/terms precondition gate but before the telemetry factory (its first in-process consumer) and well before storage selection. Two consequences that this ordering guarantees:
+
+- **Telemetry** receives the config-selected secrets provider when it is constructed.
+- **Storage's `gcp` factory** reads the config-selected secrets provider from the hub at *its* apply-point (it resolves `GCP_CREDENTIALS_FILE_PATH`) — secrets is already on the hub by then. This ordering is pinned by an integration test that boots a non-`env` secrets method alongside `gcp` storage and asserts the gcp arm read the external provider.
+
+### Injection precedence
+
+`Pipelex.setup` resolves the secrets provider in this order:
+
+1. an explicit `setup(secrets_provider=...)` parameter (test/host injection) — always wins;
+2. the config-selected registry factory (`get_required(method=secrets_config.method)(config)`).
+
+There is no separate core default: the built-in `SecretsPlugin` *is* the default supplier of the `env` method, so an ordinary boot always resolves through step 2.
+
+---
+
+## The contract: `SecretsProviderFactoryFn`
+
+A secrets backend is a typed **callable** — whole secrets config in, provider out (`pipelex/plugins/secrets_provider_registry.py`):
+
+```python
+SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], SecretsProviderAbstract]
+```
+
+Passing the whole `SecretsProviderConfig` (not a pre-resolved sub-config) is deliberate: it lets a factory read whatever it needs — an SDK-backed provider (Vault, AWS Secrets Manager) reads its own settings — at the boot apply-point, never at registration. The built-in `env` factory ignores the config entirely.
+
+A plugin contributes one factory per method it serves by calling the registrar menu in its `register`, passing the token as a raw string:
+
+```python
+registrar.add_secrets_provider(method="vault", factory=_make_vault_secrets_provider)
+```
+
+`register` is **side-effect-free** — it may call the registrar menu and nothing else (no hub access, no I/O, no SDK import). That is what keeps `build_registrar` safe to run more than once (at boot, and again for the `pipelex plugins list` diagnostic).
+
+### Import-light registration, deferred dependency
+
+The factory is a plain callable stored at registration and **invoked only at the boot apply-point**. Two invariants follow, and they are what let an SDK-backed provider ship without weighing down every boot:
+
+- **Import-light.** The plugin module must import no backend SDK at module load. The built-in `SecretsPlugin` needs none (the `env` provider reads environment variables); an out-of-tree `pipelex-secrets-vault` plugin must keep its `hvac`/`boto3`/etc. import *inside* the provider's methods (or its factory), never at module top-level, so discovery stays import-light even when the extra is installed.
+- **Fail at use, not at boot.** An optional dependency raises `MissingDependencyError` (naming the package and the `pipelex[<extra>]` install hint) when the backend is *used*, not when it is registered or selected.
+
+---
+
+## The built-in `SecretsPlugin`
+
+`pipelex/plugins/secrets/secrets_plugin.py` is the reference secrets plugin. It is **core-unconditional** — a secrets provider is required infra, so it joins `CORE_UNCONDITIONAL_PLUGIN_NAMES` and cannot be disabled into a boot with no secrets provider:
+
+```python
+class SecretsPlugin:
+ name = "secrets"
+ targets_api = PLUGIN_API_VERSION
+
+ def register(self, registrar: PluginRegistrar) -> None:
+ registrar.add_secrets_provider(method="env", factory=_make_env_secrets_provider)
+```
+
+The `env` factory ignores its config and returns a bare `EnvSecretsProvider` — the `config` parameter exists only to conform to `SecretsProviderFactoryFn`; an SDK-backed external provider is where reading the config earns its keep:
+
+```python
+def _make_env_secrets_provider(config: SecretsProviderConfig) -> SecretsProviderAbstract:
+ return EnvSecretsProvider()
+```
+
+`env` is a single-impl backend: `EnvSecretsProvider` resolves `get_required_secret` / `get_optional_secret` from environment variables, and its versioned reads (`*_specific_version`) raise `NotImplementedError` because environment variables have no version history. A full external provider (Vault, AWS Secrets Manager) implements those versioned reads.
+
+---
+
+## Selecting a method by config
+
+`secrets_config.method` is an **open `str` token** (Decision S1), not a closed enum. The built-in uses `"env"`; an external `pipelex-secrets-<backend>` plugin registers its own (e.g. `"vault"`). A config naming an external method **parses fine** — the token is stored verbatim and its installability is validated later, at registry lookup:
+
+```toml
+# .pipelex/pipelex.toml
+[pipelex.secrets_config]
+method = "vault" # an out-of-tree provider — selected iff its plugin is installed
+```
+
+Whether that token names an *installed* provider is validated at **registry lookup**, not at parse: an unknown method surfaces as `UnknownSecretsMethodError` at boot, which is the right layer — it lists the registered methods so the fix is obvious.
+
+!!! note "External-provider config surface is a scoped follow-up"
+ `SecretsProviderConfig` today carries only `method` — the built-in `env` backend needs no per-method sub-config. An out-of-tree `vault` provider has nowhere to read *its* structured config yet; a generic passthrough for external providers is a captured follow-up (Decision S4), not built speculatively. Until it lands, an external provider reads its own config from the environment or its own file.
+
+---
+
+## Fail-loud guarantees
+
+| Condition | Error |
+|-----------|-------|
+| `secrets_config.method` names no registered provider | `UnknownSecretsMethodError` (lists the registered methods) |
+| two plugins register the same `method` | `DuplicateSecretsProviderError` (names both plugins) |
+| `name` (`"secrets"`) in `plugins.disabled` | `CoreUnconditionalPluginDisabledError` |
+| entry point raises while loading/registering | `BrokenPluginError` |
+| optional SDK missing at use | `MissingDependencyError` (package + `pipelex[<extra>]` hint) |
+
+The duplicate detection is the same fail-loud `_add` helper the inference/orchestrator/storage menus use — a conflict names both contributing plugins, so a collision between two installed secrets packages is unambiguous.
+
+---
+
+## Authoring an out-of-tree secrets plugin
+
+A third-party secrets plugin is a distribution that:
+
+1. implements a `SecretsProviderAbstract` subclass — `get_required_secret` / `get_optional_secret`, the versioned `*_specific_version` reads, and `set_secret_as_env_var` — deferring any SDK import into those methods (import-light);
+2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_secrets_provider(method="<token>", factory=...)` — and nothing else;
+3. advertises itself under the `pipelex.plugins` entry-point group:
+
+```toml
+# pyproject.toml of your plugin package
+[project.entry-points."pipelex.plugins"]
+vault_secrets = "pipelex_secrets_vault.plugin:VaultSecretsPlugin"
+```
+
+Installing the distribution makes the method selectable (`secrets_config.method = "vault"`); uninstalling removes it. No core change, no central registration list — *presence* is the source of truth. A discovered plugin can be quarantined without uninstalling via the `plugins.disabled` denylist (matched against the entry-point name *before* load, so a broken install can still be disabled to recover startup — see [Inference Backend Plugins](inference-backend-plugins.md) for the shared discovery/denylist machinery).
+
+Use `pipelex plugins list` to see every discovered plugin, what each contributed, and its denylist state.
+
+---
+
+## Related
+
+- [Storage Provider Plugins](storage-provider-plugins.md) — the sibling config-selected-singleton seam this one mirrors; its `gcp` arm reads the secrets provider at boot
+- [Secrets Provider Injection](../advanced/secrets-provider-injection.md) — supplying a custom secrets provider from your own vault or secrets manager
+- [Inference Backend Plugins](inference-backend-plugins.md) — the per-call sibling seam and the shared discovery/denylist machinery
+- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, plus the hub-slot mechanism these config-selected seams deliberately avoid
+- [Error Model](error-model.md) — how these errors render and dereference
+- [Architecture Overview](architecture-overview.md)
diff --git a/docs/under-the-hood/storage-provider-plugins.md b/docs/under-the-hood/storage-provider-plugins.md
new file mode 100644
index 000000000..c5a1a5212
--- /dev/null
+++ b/docs/under-the-hood/storage-provider-plugins.md
@@ -0,0 +1,170 @@
+---
+title: "Storage Provider Plugins"
+description: "How Pipelex selects a storage backend by config, the keyed-registry-plus-config-selected-singleton seam a storage plugin rides, and how to author an out-of-tree pipelex-storage-<backend> package."
+---
+
+# Storage Provider Plugins
+
+Every asset Pipelex persists — a generated image, an uploaded document, a payload blob — is written through a single **storage provider** set on the hub at boot. Which provider that is comes entirely from data: one config field, `storage_config.method`, names a backend, and a **storage plugin** is what teaches Pipelex how to build the provider for that method.
+
+Core names no storage backend by import or by string. The built-in providers (`local`, `in_memory`, `s3`, `gcp`) are a plugin too — the always-on `StoragePlugin` — riding the exact same seam an out-of-tree `pipelex-storage-<backend>` package would. This page documents that seam, the factory contract a plugin registers, and how to write one.
+
+---
+
+## A different shape from the inference/orchestrator seams
+
+The [inference-backend](inference-backend-plugins.md) and [orchestrator](orchestrator-plugins.md) registries are selected **per call** — a model's `sdk`, a request's `orchestration_mode`. Storage is not: it is a **process-global singleton selected by its own config key**, independent of the orchestrator. So it rides a net-new mechanism the track calls a **keyed registry + config-selected singleton**:
+
+> Plugins register N provider factories into a registry keyed by an open `method` token. At boot, core reads `storage_config.method`, looks that token up in the registry, and calls the factory to produce the one provider set on the hub.
+
+This is deliberately **not** a hub slot (those are orchestrator-coupled, claimed only when `boot_orchestrator == plugin.name`). Storage selection has nothing to do with the orchestrator, so it gets its own registry and its own config key. The same mechanism also backs the [secrets provider](secrets-provider-plugins.md) seam (`secrets_config.method`).
+
+---
+
+## The seam in one view
+
+```
+boot (Pipelex.setup)
+ └─ build_registrar(config) # pure, import-light
+ ├─ for each plugin in BUILTIN_PLUGINS (StoragePlugin is one)
+ └─ for each installed "pipelex.plugins" entry point
+ └─ plugin.register(registrar) # side-effect-free
+ └─ registrar.add_storage_provider(method=…, factory=…)
+ └─ StorageProviderRegistry(registrar.storage_providers) # stored on the hub
+ └─ storage_provider = registry.get_required(method=storage_config.method)(storage_config)
+ └─ set_storage_provider(storage_provider) # the one provider every consumer reads
+```
+
+There is no `match storage_config.method:` anywhere in boot — the token set is open, so validation *is* the registry lookup. Adding a backend means registering a factory for its token; nothing in core changes. Every downstream consumer (input normalizer, content generator, PDF/image utilities, delivery executor) keeps calling `get_storage_provider()` and is unaffected by which method was selected.
+
+The provider is resolved **after** the secrets provider is on the hub, so a factory may perform a hub secrets read at its apply-point (the `gcp` factory does — see below).
+
+### Injection precedence
+
+`Pipelex.setup` resolves the storage provider in this order:
+
+1. an explicit `setup(storage_provider=...)` parameter (test/host injection) — always wins;
+2. the config-selected registry factory (`get_required(method=storage_config.method)(config)`).
+
+There is no separate core default: the built-in `StoragePlugin` *is* the default supplier of every method, so an ordinary boot always resolves through step 2.
+
+---
+
+## The contract: `StorageProviderFactoryFn`
+
+A storage backend is a typed **callable** — whole storage config in, provider out (`pipelex/plugins/storage_provider_registry.py`):
+
+```python
+StorageProviderFactoryFn = Callable[[StorageProviderConfig], StorageProviderAbstract]
+```
+
+Passing the whole `StorageProviderConfig` (not a pre-resolved sub-config) is deliberate: it lets a factory read whatever it needs — including a hub secrets read — at the boot apply-point, never at registration.
+
+A plugin contributes one factory per method it serves by calling the registrar menu in its `register`, passing the token as a raw string:
+
+```python
+registrar.add_storage_provider(method="azure", factory=_make_azure_storage_provider)
+```
+
+`register` is **side-effect-free** — it may call the registrar menu and nothing else (no hub access, no I/O, no SDK import). That is what keeps `build_registrar` safe to run more than once (at boot, and again for the `pipelex plugins list` diagnostic).
+
+### Import-light registration, deferred dependency
+
+The factory is a plain callable stored at registration and **invoked only at the boot apply-point**. Two invariants follow, and they are what let an SDK-backed provider ship without weighing down every boot:
+
+- **Import-light.** The plugin module must import no backend SDK at module load. Registering the built-in `s3`/`gcp` factories pulls in neither `aioboto3` nor `google-cloud-storage` — the SDK guard lives *inside* the provider's I/O methods (`_get_session` / `_get_bucket`), not its `__init__`. Selecting `s3` therefore *constructs* an `S3StorageProvider` even with the extra absent; the guard fires only on the first actual load/store.
+- **Fail at use, not at boot.** An optional dependency raises `MissingDependencyError` (naming the package and the `pipelex[<extra>]` install hint) when the backend is *used*, not when it is registered or selected.
+
+---
+
+## The built-in `StoragePlugin`
+
+`pipelex/plugins/storage/storage_plugin.py` is the reference storage plugin. It is **core-unconditional** — storage is required infra, so it joins `CORE_UNCONDITIONAL_PLUGIN_NAMES` and cannot be disabled into a boot with no storage:
+
+```python
+class StoragePlugin:
+ name = "storage"
+ targets_api = PLUGIN_API_VERSION
+
+ def register(self, registrar: PluginRegistrar) -> None:
+ registrar.add_storage_provider(method=StorageMethod.LOCAL, factory=_make_local_storage_provider)
+ registrar.add_storage_provider(method=StorageMethod.IN_MEMORY, factory=_make_in_memory_storage_provider)
+ registrar.add_storage_provider(method=StorageMethod.S3, factory=_make_s3_storage_provider)
+ registrar.add_storage_provider(method=StorageMethod.GCP, factory=_make_gcp_storage_provider)
+```
+
+Each factory is a module-level closure that None-checks its sub-config, calls `lazy_validate()`, and constructs the provider. The built-in tokens are the `StorageMethod` enum values (`"local"`, `"in_memory"`, `"s3"`, `"gcp"`); because a `StrEnum` key compares and hashes equal to its plain-`str` form, boot's plain-`str` `config.method` resolves them.
+
+The `gcp` factory is the one that reads a secret from the hub — legal because the factory runs at the boot apply-point, after the secrets provider is set, never inside `register`:
+
+```python
+def _make_gcp_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:
+ ...
+ return GcpStorageProvider(
+ bucket_name=config.gcp.bucket_name,
+ project_id=config.gcp.project_id,
+ credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),
+ signed_urls_lifespan=config.gcp.signed_urls_lifespan,
+ )
+```
+
+---
+
+## Selecting a method by config
+
+`storage_config.method` is an **open `str` token** (Decision D1), not a closed enum. The built-ins use the `StorageMethod` values; an external `pipelex-storage-<backend>` plugin registers its own (e.g. `"azure"`). A config naming an external method **parses fine** — the per-method sub-config validator requires a matching sub-config only for the four built-in tokens and lets any other token through:
+
+```toml
+# .pipelex/pipelex.toml
+[pipelex.storage_config]
+method = "azure" # an out-of-tree provider — no built-in sub-config required
+```
+
+Whether that token names an *installed* provider is validated at **registry lookup**, not at parse: an unknown method surfaces as `UnknownStorageMethodError` at boot, which is the right layer — it lists the registered methods so the fix is obvious.
+
+!!! note "External-provider config surface is a scoped follow-up"
+ `StorageProviderConfig` has fixed, typed per-method sub-models, so an out-of-tree `azure` provider has nowhere to read *its* structured config yet. The seam for built-in methods lands first; a generic passthrough sub-config for external providers is a captured follow-up (Decision D3), not built speculatively. Until it lands, an external provider reads its own config from the environment or its own file. **One live gap:** `StorageProviderConfig.uri_format` (read by `GeneratedContentFactory._build_storage_key` on every content store) is only defined for the four built-in methods and raises `StorageConfigError` for any other token — so an external provider selects and boots cleanly but a *generated-content store* through it fails until D3 gives external methods a `uri_format`. External providers are therefore usable today for their own direct storage API, not yet as the backing store for generated content.
+
+---
+
+## Fail-loud guarantees
+
+| Condition | Error |
+|-----------|-------|
+| `storage_config.method` names no registered provider | `UnknownStorageMethodError` (lists the registered methods) |
+| two plugins register the same `method` | `DuplicateStorageProviderError` (names both plugins) |
+| `name` (`"storage"`) in `plugins.disabled` | `CoreUnconditionalPluginDisabledError` |
+| entry point raises while loading/registering | `BrokenPluginError` |
+| optional SDK missing at use | `MissingDependencyError` (package + `pipelex[<extra>]` hint) |
+
+The duplicate detection is the same fail-loud `_add` helper the inference/orchestrator menus use — a conflict names both contributing plugins, so a collision between two installed storage packages is unambiguous.
+
+---
+
+## Authoring an out-of-tree storage plugin
+
+A third-party storage plugin is a distribution that:
+
+1. implements a `StorageProviderAbstract` subclass — the three hooks `_load_with_metadata`, `_store`, and `public_url` — deferring any SDK import into those methods (import-light);
+2. defines a plugin class (`name`, `targets_api`, `register`) whose `register` calls `add_storage_provider(method="<token>", factory=...)` — and nothing else;
+3. advertises itself under the `pipelex.plugins` entry-point group:
+
+```toml
+# pyproject.toml of your plugin package
+[project.entry-points."pipelex.plugins"]
+azure_storage = "pipelex_storage_azure.plugin:AzureStoragePlugin"
+```
+
+Installing the distribution makes the method selectable (`storage_config.method = "azure"`); uninstalling removes it. No core change, no central registration list — *presence* is the source of truth. A discovered plugin can be quarantined without uninstalling via the `plugins.disabled` denylist (matched against the entry-point name *before* load, so a broken install can still be disabled to recover startup — see [Inference Backend Plugins](inference-backend-plugins.md) for the shared discovery/denylist machinery).
+
+Use `pipelex plugins list` to see every discovered plugin, what each contributed, and its denylist state.
+
+---
+
+## Related
+
+- [Secrets Provider Plugins](secrets-provider-plugins.md) — the sibling config-selected-singleton seam riding the same mechanism; the `gcp` factory above reads its provider
+- [Inference Backend Plugins](inference-backend-plugins.md) — the per-call sibling seam and the shared discovery/denylist machinery
+- [Orchestrator Plugins](orchestrator-plugins.md) — the other per-call seam, plus the hub-slot mechanism storage deliberately avoids
+- [Error Model](error-model.md) — how these errors render and dereference
+- [Architecture Overview](architecture-overview.md)
diff --git a/mkdocs.yml b/mkdocs.yml
index 0db0e4f8b..d84b6140e 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -316,6 +316,8 @@ plugins:
- under-the-hood/pipe-routing-and-execution.md: "Pipe Routing & Execution"
- under-the-hood/inference-backend-plugins.md: "Inference Backend Plugins"
- under-the-hood/orchestrator-plugins.md: "Orchestrator Plugins"
+ - under-the-hood/storage-provider-plugins.md: "Storage Provider Plugins"
+ - under-the-hood/secrets-provider-plugins.md: "Secrets Provider Plugins"
- under-the-hood/error-model.md: "Error Model"
- under-the-hood/runtime-bridge-and-transport.md: "Runtime Bridge & Transport"
- under-the-hood/distributed-content-generation.md: "Content Generation Across Boundaries"
@@ -547,6 +549,8 @@ nav:
- Pipe Routing & Execution: under-the-hood/pipe-routing-and-execution.md
- Inference Backend Plugins: under-the-hood/inference-backend-plugins.md
- Orchestrator Plugins: under-the-hood/orchestrator-plugins.md
+ - Storage Provider Plugins: under-the-hood/storage-provider-plugins.md
+ - Secrets Provider Plugins: under-the-hood/secrets-provider-plugins.md
- Error Model: under-the-hood/error-model.md
- Runtime Bridge & Transport: under-the-hood/runtime-bridge-and-transport.md
- Content Generation Across Boundaries: under-the-hood/distributed-content-generation.md
diff --git a/pipelex/hub.py b/pipelex/hub.py
index d998ddef7..1d5bf8fea 100644
--- a/pipelex/hub.py
+++ b/pipelex/hub.py
@@ -53,6 +53,8 @@ if TYPE_CHECKING:
from pipelex.plugins.inference_backend_registry import InferenceBackendRegistry
from pipelex.plugins.model_lister_registry import ModelListerRegistry
from pipelex.plugins.orchestrator_registry import OrchestratorRegistry
+ from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry
+ from pipelex.plugins.storage_provider_registry import StorageProviderRegistry
from pipelex.tracing.event_log_protocol import EventLogProtocol
@@ -89,6 +91,8 @@ class PipelexHub:
self._model_lister_registry: ModelListerRegistry | None = None
self._orchestrator_registry: OrchestratorRegistry | None = None
self._bundle_validator_registry: BundleValidatorRegistry | None = None
+ self._storage_provider_registry: StorageProviderRegistry | None = None
+ self._secrets_provider_registry: SecretsProviderRegistry | None = None
self._inference_manager: InferenceManagerProtocol
self._report_delegate: ReportingProtocol
self._content_generator: ContentGeneratorProtocol | None = None
@@ -212,6 +216,12 @@ class PipelexHub:
def set_bundle_validator_registry(self, bundle_validator_registry: "BundleValidatorRegistry"):
self._bundle_validator_registry = bundle_validator_registry
+ def set_storage_provider_registry(self, storage_provider_registry: "StorageProviderRegistry"):
+ self._storage_provider_registry = storage_provider_registry
+
+ def set_secrets_provider_registry(self, secrets_provider_registry: "SecretsProviderRegistry"):
+ self._secrets_provider_registry = secrets_provider_registry
+
def set_inference_manager(self, inference_manager: InferenceManagerProtocol):
self._inference_manager = inference_manager
@@ -353,6 +363,18 @@ class PipelexHub:
raise RuntimeError(msg)
return self._bundle_validator_registry
+ def get_storage_provider_registry(self) -> "StorageProviderRegistry":
+ if self._storage_provider_registry is None:
+ msg = "StorageProviderRegistry is not initialized"
+ raise RuntimeError(msg)
+ return self._storage_provider_registry
+
+ def get_secrets_provider_registry(self) -> "SecretsProviderRegistry":
+ if self._secrets_provider_registry is None:
+ msg = "SecretsProviderRegistry is not initialized"
+ raise RuntimeError(msg)
+ return self._secrets_provider_registry
+
def get_inference_manager(self) -> InferenceManagerProtocol:
return self._inference_manager
@@ -534,6 +556,14 @@ def get_bundle_validator_registry() -> "BundleValidatorRegistry":
return get_pipelex_hub().get_bundle_validator_registry()
+def get_storage_provider_registry() -> "StorageProviderRegistry":
+ return get_pipelex_hub().get_storage_provider_registry()
+
+
+def get_secrets_provider_registry() -> "SecretsProviderRegistry":
+ return get_pipelex_hub().get_secrets_provider_registry()
+
+
def get_inference_manager() -> InferenceManagerProtocol:
return get_pipelex_hub().get_inference_manager()
diff --git a/pipelex/kit/configs/pipelex.toml b/pipelex/kit/configs/pipelex.toml
index 25d68f91d..a508997f6 100644
--- a/pipelex/kit/configs/pipelex.toml
+++ b/pipelex/kit/configs/pipelex.toml
@@ -119,6 +119,15 @@ bucket_name = ""
project_id = ""
signed_urls_lifespan_seconds = 3600 # Set to "disabled" for public URLs
+####################################################################################################
+# Secrets Config
+####################################################################################################
+
+[pipelex.secrets_config]
+# Secrets provider method: "env" (built-in, reads secrets from environment variables).
+# An external pipelex-secrets-<backend> plugin can add its own method (e.g. "vault").
+method = "env"
+
####################################################################################################
# Scan Config
####################################################################################################
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index 9c6fa44bb..a70c2b884 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -56,6 +56,8 @@ from pipelex.plugins.model_lister_registry import ModelListerRegistry
from pipelex.plugins.orchestrator_registry import OrchestratorRegistry
from pipelex.plugins.registrar import HubSlot, PluginRegistrar
from pipelex.plugins.sdk_client_manager import SdkClientManager
+from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry
+from pipelex.plugins.storage_provider_registry import StorageProviderRegistry
from pipelex.reporting.reporting_manager import ReportingManager
from pipelex.reporting.reporting_protocol import ReportingProtocol
from pipelex.system.configuration.config_loader import config_manager
@@ -87,10 +89,8 @@ from pipelex.test_extras.registry_test_models import TestRegistryModels
from pipelex.tools.jinja2.jinja2_template_loader import TemplateLoader
from pipelex.tools.jinja2.jinja2_template_registry import TemplateRegistry
from pipelex.tools.misc.package_utils import get_package_info
-from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
-from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_config
from pipelex.types import Self
from pipelex.urls import URLs
@@ -208,9 +208,6 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
if boot_orchestrator is not None:
get_config().plugins.boot_orchestrator = boot_orchestrator
- # Initialize secrets provider early - needed for gateway check
- secrets_provider = secrets_provider or EnvSecretsProvider()
-
# --- Pipelex Service and Telemetry --------------------------------------------------
# Check if Pipelex Gateway is enabled
@@ -278,6 +275,38 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
stacklevel=2,
)
+ # --- Plugin discovery -----------------------------------------------------------------
+ # Build the plugin registrar from the fully-resolved config (pure and import-light:
+ # registering the built-ins imports no backend SDK, constructs no client, touches no hub).
+ # Built here — after the gateway service/terms precondition gate above (so an unaccepted-terms or
+ # first-run boot fails fast before any discovery work) and before the telemetry factory below,
+ # which is the first consumer of the secrets provider. Secrets is now a config-selected plugin
+ # seam: the built-in SecretsPlugin's factory (and any external pipelex-secrets-<backend>) is
+ # looked up from the registrar-derived SecretsProviderRegistry just below. The other registries
+ # (inference, storage, …) are still built later at their own hub-set points, all referencing this
+ # same already-built registrar; the slot-claim thunks / teardown callbacks it also accumulates are
+ # applied at their ordered apply-points in later phases.
+ plugin_registrar = build_registrar(config=get_config())
+ self._plugin_registrar = plugin_registrar
+ # Reject an unknown boot orchestrator before falling through to the core defaults. The requested
+ # name (CLI --orchestrator / setup(boot_orchestrator=...) / config) is matched against registered
+ # plugin names — the same namespace the slot-claim gate uses (boot_orchestrator == plugin.name).
+ # When no plugin of that name registered (not installed, disabled, or a typo) nothing claims the
+ # hub slots, so without this guard the run would silently execute in-process instead of under the
+ # requested runtime. Checked here to fail fast, before the telemetry/model work.
+ requested_boot_orchestrator = get_config().plugins.boot_orchestrator
+ if requested_boot_orchestrator is not None and requested_boot_orchestrator not in plugin_registrar.registered_plugin_names:
+ raise UnknownBootOrchestratorError(requested=requested_boot_orchestrator)
+
+ # Secrets provider precedence: explicit setup() param > config-selected registry factory.
+ # The built-in SecretsPlugin supplies the "env" method, so there is no separate core default.
+ # Resolved here because the telemetry factory just below (and the model setup further down) consume it.
+ secrets_provider_registry = SecretsProviderRegistry(plugin_registrar.secrets_providers)
+ self.pipelex_hub.set_secrets_provider_registry(secrets_provider_registry)
+ if secrets_provider is None:
+ secrets_config = get_config().pipelex.secrets_config
+ secrets_provider = secrets_provider_registry.get_required(method=secrets_config.method)(secrets_config)
+
# Disable Pipelex telemetry when:
# - inference is not needed (no live runs to track), OR
# - the gateway config came from the cache (stale specs imply potentially stale model
@@ -304,10 +333,10 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.func_registry = func_registry or FuncRegistry()
self.pipelex_hub.set_func_registry(func_registry=self.func_registry)
self.pipelex_hub.set_secrets_provider(secrets_provider=secrets_provider)
- if storage_provider is None:
- storage_config = get_config().pipelex.storage_config
- storage_provider = make_storage_provider_from_config(storage_config)
- self.pipelex_hub.set_storage_provider(storage_provider)
+ # Storage is selected from the config-driven StorageProviderRegistry, built from the plugin
+ # registrar (constructed above, just before the telemetry factory). Its resolution and hub-set
+ # still happen later at the plugin-derived-registries block — after secrets is on the hub here,
+ # so the GCP factory's secret read works.
# Register stuff templates first (used by mermaid, reactflow, and stuff_viewer)
stuff_name, stuff_package, stuff_templates = STUFF_TEMPLATE_SET
@@ -383,28 +412,25 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
# backend-keyed unconditionally (eng review D4) — a keyless Temporal submitter must still
# dispatch activities and mock inside them, so `needs_inference` plays no role in picking the
# generator. Its other boot roles (gateway/model setup, credentials, telemetry) are unchanged.
- # --- Plugin discovery -----------------------------------------------------------------
- # Build the plugin registrar from the fully-resolved config (pure and import-light:
- # registering the built-ins imports no backend SDK). Runs after the gateway/model setup
- # checks and before the hub setup points below — the family worker factories look their
- # backends up on the hub registries at run time. The slot-claim thunks, CLI commands and
- # teardown callbacks the registrar also accumulates are applied at their ordered
- # apply-points in a later phase (no orchestrator plugin contributes any of them yet).
- plugin_registrar = build_registrar(config=get_config())
- self._plugin_registrar = plugin_registrar
- # Reject an unknown boot orchestrator before falling through to the core defaults. The
- # requested name (CLI --orchestrator / setup(boot_orchestrator=...) / config) is matched
- # against registered plugin names — the same namespace the slot-claim gate uses
- # (boot_orchestrator == plugin.name). When no plugin of that name registered (not installed,
- # disabled, or a typo) nothing claims the hub slots, so without this guard the run would
- # silently execute in-process instead of under the requested runtime.
- requested_boot_orchestrator = get_config().plugins.boot_orchestrator
- if requested_boot_orchestrator is not None and requested_boot_orchestrator not in plugin_registrar.registered_plugin_names:
- raise UnknownBootOrchestratorError(requested=requested_boot_orchestrator)
+ # --- Plugin-derived registries --------------------------------------------------------
+ # The plugin registrar was built earlier (with the boot-orchestrator gate checked and the
+ # config-selected secrets provider resolved) just before the telemetry factory. Turn its
+ # accumulated contributions into the hub registries here — after the gateway/model setup checks
+ # and before the hub setup points below — the family worker factories look their backends up on
+ # these at run time.
self.pipelex_hub.set_inference_backend_registry(InferenceBackendRegistry(plugin_registrar.inference_backends))
self.pipelex_hub.set_model_lister_registry(ModelListerRegistry(plugin_registrar.model_listers))
self.pipelex_hub.set_orchestrator_registry(OrchestratorRegistry(plugin_registrar.orchestrators))
self.pipelex_hub.set_bundle_validator_registry(BundleValidatorRegistry(plugin_registrar.bundle_validators))
+ storage_provider_registry = StorageProviderRegistry(plugin_registrar.storage_providers)
+ self.pipelex_hub.set_storage_provider_registry(storage_provider_registry)
+ # Storage provider precedence: explicit setup() param > config-selected registry factory.
+ # The built-in StoragePlugin supplies every method, so there is no separate core default.
+ # Resolves here (after secrets is on the hub) so the GCP factory's secret read works.
+ if storage_provider is None:
+ storage_config = get_config().pipelex.storage_config
+ storage_provider = storage_provider_registry.get_required(method=storage_config.method)(storage_config)
+ self.pipelex_hub.set_storage_provider(storage_provider)
self.pipelex_hub.set_dry_run_forced(not needs_inference)
# Injection precedence (codex C8): explicit setup() param > plugin slot-claim thunk > core default.
diff --git a/pipelex/pipelex.toml b/pipelex/pipelex.toml
index ddb1c75f9..8952bf3f4 100644
--- a/pipelex/pipelex.toml
+++ b/pipelex/pipelex.toml
@@ -26,6 +26,9 @@ bucket_name = ""
project_id = ""
signed_urls_lifespan_seconds = 3600
+[pipelex.secrets_config]
+method = "env" # env (built-in); an external plugin may add e.g. vault
+
[pipelex.observer_config]
observer_dir = "results/observer"
diff --git a/pipelex/plugins/builtins.py b/pipelex/plugins/builtins.py
index 690178120..c00627b63 100644
--- a/pipelex/plugins/builtins.py
+++ b/pipelex/plugins/builtins.py
@@ -15,12 +15,16 @@ from pipelex.plugins.openai.openai_plugin import OpenAIPlugin
from pipelex.plugins.openrouter.openrouter_plugin import OpenRouterPlugin
from pipelex.plugins.portkey.portkey_plugin import PortkeyPlugin
from pipelex.plugins.pypdfium2.pypdfium2_plugin import Pypdfium2Plugin
+from pipelex.plugins.secrets.secrets_plugin import SecretsPlugin
+from pipelex.plugins.storage.storage_plugin import StoragePlugin
# The plugins Pipelex ships with — discovered at boot ahead of any external
# entry-point plugin. Each is import-light: importing this module imports no
# backend SDK (the SDKs load lazily inside the make_worker closures).
BUILTIN_PLUGINS: list[PipelexPlugin] = [
DirectOrchestratorPlugin(),
+ StoragePlugin(),
+ SecretsPlugin(),
OpenAIPlugin(),
GatewayPlugin(),
PortkeyPlugin(),
@@ -41,6 +45,9 @@ BUILTIN_PLUGINS: list[PipelexPlugin] = [
# Built-in plugins that core requires unconditionally — naming one in
# ``plugins.disabled`` is a configuration error, not a no-op. ``direct`` owns the
# DIRECT orchestrator (you cannot boot without an in-process execution mode);
-# ``openai`` is the always-on default inference driver (no optional SDK to avoid),
-# so disabling it would only break the out-of-the-box experience.
-CORE_UNCONDITIONAL_PLUGIN_NAMES: frozenset[str] = frozenset({"direct", "openai"})
+# ``storage`` supplies every built-in storage backend (``storage_config.method`` must
+# resolve to a registered factory or boot fails loud); ``secrets`` supplies the built-in
+# ``env`` secrets backend (``secrets_config.method`` must likewise resolve or boot fails loud);
+# ``openai`` is the always-on default inference driver (no optional SDK to avoid), so disabling it
+# would only break the out-of-the-box experience.
+CORE_UNCONDITIONAL_PLUGIN_NAMES: frozenset[str] = frozenset({"direct", "storage", "secrets", "openai"})
diff --git a/pipelex/plugins/contract.py b/pipelex/plugins/contract.py
index 2ea89c730..5acf8c7d7 100644
--- a/pipelex/plugins/contract.py
+++ b/pipelex/plugins/contract.py
@@ -11,7 +11,12 @@ if TYPE_CHECKING:
# v2 added the optional ``add_http_error_mapper`` capability (a framework-agnostic
# transport-fault → ``ErrorReport`` mapping a host runtime renders into its own
# HTTP error response).
-PLUGIN_API_VERSION: int = 2
+#
+# v3 added ``add_storage_provider`` and ``add_secrets_provider`` — two config-selected,
+# process-global provider registries (``storage_config.method`` / ``secrets_config.method`` pick
+# the factory at boot). DX-1 batches both menu additions under this single bump so external plugins
+# re-declare ``targets_api`` only once.
+PLUGIN_API_VERSION: int = 3
@runtime_checkable
diff --git a/pipelex/plugins/exceptions.py b/pipelex/plugins/exceptions.py
index 57ec3416e..65a34fa0a 100644
--- a/pipelex/plugins/exceptions.py
+++ b/pipelex/plugins/exceptions.py
@@ -95,6 +95,34 @@ class DuplicateBundleValidatorError(PluginError):
super().__init__(message)
+class DuplicateStorageProviderError(PluginError):
+ """Two plugins registered a storage provider for the same method."""
+
+ def __init__(self, *, method: str, first_plugin: str, second_plugin: str):
+ self.method = method
+ self.first_plugin = first_plugin
+ self.second_plugin = second_plugin
+ message = (
+ f"Storage provider for method '{method}' is registered by both plugin "
+ f"'{first_plugin}' and plugin '{second_plugin}'. Each method must have a single provider."
+ )
+ super().__init__(message)
+
+
+class DuplicateSecretsProviderError(PluginError):
+ """Two plugins registered a secrets provider for the same method."""
+
+ def __init__(self, *, method: str, first_plugin: str, second_plugin: str):
+ self.method = method
+ self.first_plugin = first_plugin
+ self.second_plugin = second_plugin
+ message = (
+ f"Secrets provider for method '{method}' is registered by both plugin "
+ f"'{first_plugin}' and plugin '{second_plugin}'. Each method must have a single provider."
+ )
+ super().__init__(message)
+
+
class DuplicateHttpErrorMapperError(PluginError):
"""Two plugins registered an HTTP-error mapper for the same exception type."""
@@ -167,3 +195,45 @@ class UnknownBootOrchestratorError(PluginError):
"Core provides only in-process execution; booting under a distributed orchestrator requires installing its plugin."
)
super().__init__(message)
+
+
+class UnknownStorageMethodError(PluginError):
+ """A configured storage method has no registered provider factory.
+
+ ``storage_config.method`` selects a storage provider from the registry the built-in
+ ``StoragePlugin`` (and any external ``pipelex-storage-<backend>`` plugin) populates. When
+ the token names no registered factory — a typo, or an external provider plugin that is not
+ installed or was disabled via ``plugins.disabled`` — boot fails loud here rather than
+ starting with no storage. The message lists the registered methods so the fix is obvious.
+ """
+
+ def __init__(self, *, method: str, registered_methods: list[str]):
+ self.method = method
+ self.registered_methods = registered_methods
+ available = ", ".join(sorted(registered_methods)) or "(none)"
+ message = (
+ f"No storage provider is registered for method '{method}'. Registered methods: {available}. "
+ "Check storage_config.method, or install/enable the plugin that provides that method."
+ )
+ super().__init__(message)
+
+
+class UnknownSecretsMethodError(PluginError):
+ """A configured secrets method has no registered provider factory.
+
+ ``secrets_config.method`` selects a secrets provider from the registry the built-in
+ ``SecretsPlugin`` (and any external ``pipelex-secrets-<backend>`` plugin) populates. When
+ the token names no registered factory — a typo, or an external provider plugin that is not
+ installed or was disabled via ``plugins.disabled`` — boot fails loud here rather than
+ starting with no secrets provider. The message lists the registered methods so the fix is obvious.
+ """
+
+ def __init__(self, *, method: str, registered_methods: list[str]):
+ self.method = method
+ self.registered_methods = registered_methods
+ available = ", ".join(sorted(registered_methods)) or "(none)"
+ message = (
+ f"No secrets provider is registered for method '{method}'. Registered methods: {available}. "
+ "Check secrets_config.method, or install/enable the plugin that provides that method."
+ )
+ super().__init__(message)
diff --git a/pipelex/plugins/registrar.py b/pipelex/plugins/registrar.py
index 7321c3872..5ac094957 100644
--- a/pipelex/plugins/registrar.py
+++ b/pipelex/plugins/registrar.py
@@ -11,11 +11,15 @@ from pipelex.plugins.exceptions import (
DuplicateInferenceBackendError,
DuplicateModelListerError,
DuplicateOrchestratorError,
+ DuplicateSecretsProviderError,
+ DuplicateStorageProviderError,
HubSlotAlreadyClaimedError,
)
from pipelex.plugins.inference_backend_registry import InferenceFamily, MakeWorkerFn
from pipelex.plugins.model_lister_registry import ListModelsFn
from pipelex.plugins.orchestrator_registry import OrchestratorProtocol
+from pipelex.plugins.secrets_provider_registry import SecretsProviderFactoryFn
+from pipelex.plugins.storage_provider_registry import StorageProviderFactoryFn
from pipelex.runtime_bridge.orchestration_mode import OrchestrationMode
from pipelex.types import StrEnum
@@ -115,6 +119,8 @@ class PluginRegistrar:
self.model_listers: dict[str, ListModelsFn] = {}
self.orchestrators: dict[OrchestrationMode, OrchestratorProtocol] = {}
self.bundle_validators: dict[OrchestrationMode, BundleValidatorProtocol] = {}
+ self.storage_providers: dict[str, StorageProviderFactoryFn] = {}
+ self.secrets_providers: dict[str, SecretsProviderFactoryFn] = {}
# Ordered list (not a type-keyed dict) because the exception types are
# resolved lazily — only ``get_http_error_mappers`` invokes the providers,
# so duplicate-by-type detection is deferred to resolution time too.
@@ -126,6 +132,8 @@ class PluginRegistrar:
self._model_lister_sources: dict[str, str] = {}
self._orchestrator_sources: dict[OrchestrationMode, str] = {}
self._bundle_validator_sources: dict[OrchestrationMode, str] = {}
+ self._storage_provider_sources: dict[str, str] = {}
+ self._secrets_provider_sources: dict[str, str] = {}
self._slot_sources: dict[HubSlot, str] = {}
# Reassigned per plugin by build_registrar; the floating default keeps the
# menu methods safe to call outside a registration loop (e.g. a focused unit test).
@@ -193,6 +201,48 @@ class PluginRegistrar:
),
)
+ def add_storage_provider(self, *, method: str, factory: StorageProviderFactoryFn) -> None:
+ """Contribute a factory for one storage backend, keyed by an open ``method`` token.
+
+ The built-in ``StoragePlugin`` registers the ``local`` / ``in_memory`` / ``s3`` / ``gcp``
+ methods; an external ``pipelex-storage-<backend>`` plugin registers its own token (e.g.
+ ``"azure"``). Boot reads ``storage_config.method`` and calls the looked-up factory to
+ produce the one storage provider set on the hub. ``factory`` is invoked at that boot
+ apply-point, never here — so a factory may do heavy work (SDK import, a hub secrets read)
+ while ``register`` stays import-light. Fail-loud on a duplicate method, naming both plugins.
+ """
+ self._add(
+ store=self.storage_providers,
+ sources=self._storage_provider_sources,
+ key=method,
+ value=factory,
+ contribution=f"storage provider {method}",
+ on_duplicate=lambda first_plugin, second_plugin: DuplicateStorageProviderError(
+ method=method, first_plugin=first_plugin, second_plugin=second_plugin
+ ),
+ )
+
+ def add_secrets_provider(self, *, method: str, factory: SecretsProviderFactoryFn) -> None:
+ """Contribute a factory for one secrets backend, keyed by an open ``method`` token.
+
+ The built-in ``SecretsPlugin`` registers the ``env`` method; an external
+ ``pipelex-secrets-<backend>`` plugin registers its own token (e.g. ``"vault"``). Boot reads
+ ``secrets_config.method`` and calls the looked-up factory to produce the one secrets provider
+ set on the hub. ``factory`` is invoked at that boot apply-point, never here — so a factory may
+ do heavy work (SDK import) while ``register`` stays import-light. Fail-loud on a duplicate
+ method, naming both plugins.
+ """
+ self._add(
+ store=self.secrets_providers,
+ sources=self._secrets_provider_sources,
+ key=method,
+ value=factory,
+ contribution=f"secrets provider {method}",
+ on_duplicate=lambda first_plugin, second_plugin: DuplicateSecretsProviderError(
+ method=method, first_plugin=first_plugin, second_plugin=second_plugin
+ ),
+ )
+
def add_http_error_mapper(self, *, exc_type_provider: HttpErrorTypeProviderFn, to_error_report: HttpErrorMapperFn) -> None:
"""Contribute a mapping from a transport/runtime exception to a structured ``ErrorReport``.
diff --git a/pipelex/plugins/secrets/__init__.py b/pipelex/plugins/secrets/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/plugins/secrets/secrets_plugin.py b/pipelex/plugins/secrets/secrets_plugin.py
new file mode 100644
index 000000000..82d4b5b31
--- /dev/null
+++ b/pipelex/plugins/secrets/secrets_plugin.py
@@ -0,0 +1,26 @@
+from pipelex.plugins.contract import PLUGIN_API_VERSION
+from pipelex.plugins.registrar import PluginRegistrar
+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
+from pipelex.tools.secrets.secrets_config import SecretsProviderConfig
+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
+
+
+def _make_env_secrets_provider(config: SecretsProviderConfig) -> SecretsProviderAbstract: # noqa: ARG001 - env provider reads no config; uniform SecretsProviderFactoryFn signature
+ return EnvSecretsProvider()
+
+
+class SecretsPlugin:
+ """Always-on built-in provider of the ``env`` secrets backend (reads secrets from env vars).
+
+ Core-unconditional: secrets is required infra, so this plugin cannot be disabled into a boot
+ with no secrets provider (see ``CORE_UNCONDITIONAL_PLUGIN_NAMES``). It registers the one built-in
+ ``env`` method; ``secrets_config.method`` selects which factory boot invokes. Importing this module
+ is import-light — no SDK loads at register (the env provider needs none, and an external
+ ``pipelex-secrets-<backend>`` plugin defers its SDK import to its own factory).
+ """
+
+ name = "secrets"
+ targets_api = PLUGIN_API_VERSION
+
+ def register(self, registrar: PluginRegistrar) -> None:
+ registrar.add_secrets_provider(method="env", factory=_make_env_secrets_provider)
diff --git a/pipelex/plugins/secrets_provider_registry.py b/pipelex/plugins/secrets_provider_registry.py
new file mode 100644
index 000000000..16f39ecdc
--- /dev/null
+++ b/pipelex/plugins/secrets_provider_registry.py
@@ -0,0 +1,37 @@
+from collections.abc import Callable
+
+from pipelex.plugins.exceptions import UnknownSecretsMethodError
+from pipelex.tools.secrets.secrets_config import SecretsProviderConfig
+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
+
+# A plugin's factory for one secrets backend: whole secrets config in, provider out. The whole
+# config (not a pre-resolved sub-config) is passed so a factory can read whatever it needs — an
+# SDK-backed provider (Vault, AWS Secrets Manager) reads its own settings — at the boot apply-point,
+# never at registration. The built-in ``env`` factory ignores the config entirely.
+SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], SecretsProviderAbstract]
+
+
+class SecretsProviderRegistry:
+ """Read view over the secrets-provider factories contributed by discovered plugins.
+
+ Keyed by the open secrets ``method`` token (a ``str``; the built-in ``SecretsPlugin`` registers
+ ``"env"``, an external plugin registers e.g. ``"vault"``). Built once at boot from the registrar's
+ accumulated ``secrets_providers`` and stored on the hub; core reads ``secrets_config.method`` and
+ calls the looked-up factory to produce the one provider. Mirrors ``StorageProviderRegistry``.
+ """
+
+ def __init__(self, secrets_providers: dict[str, SecretsProviderFactoryFn]):
+ self._secrets_providers: dict[str, SecretsProviderFactoryFn] = dict(secrets_providers)
+
+ def get_required(self, *, method: str) -> SecretsProviderFactoryFn:
+ factory = self._secrets_providers.get(method)
+ if factory is None:
+ raise UnknownSecretsMethodError(method=method, registered_methods=self.methods)
+ return factory
+
+ def has(self, *, method: str) -> bool:
+ return method in self._secrets_providers
+
+ @property
+ def methods(self) -> list[str]:
+ return list(self._secrets_providers)
diff --git a/pipelex/plugins/storage/__init__.py b/pipelex/plugins/storage/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/pipelex/plugins/storage/storage_plugin.py b/pipelex/plugins/storage/storage_plugin.py
new file mode 100644
index 000000000..bd7141088
--- /dev/null
+++ b/pipelex/plugins/storage/storage_plugin.py
@@ -0,0 +1,79 @@
+from pathlib import Path
+
+from pipelex import log
+from pipelex.hub import get_secrets_provider
+from pipelex.plugins.contract import PLUGIN_API_VERSION
+from pipelex.plugins.registrar import PluginRegistrar
+from pipelex.tools.storage.exceptions import StorageConfigError
+from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider
+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider
+from pipelex.tools.storage.storage_config import StorageMethod, StorageProviderConfig
+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+
+
+def _make_local_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:
+ if config.local is None:
+ msg = "local config is required when method is local"
+ raise StorageConfigError(msg)
+ config.local.lazy_validate()
+ log.verbose(f"Using local storage at: {config.local.local_storage_path}")
+ return LocalStorageProvider(root_path=Path(config.local.local_storage_path))
+
+
+def _make_in_memory_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:
+ if config.in_memory is None:
+ msg = "in_memory config is required when method is in_memory"
+ raise StorageConfigError(msg)
+ config.in_memory.lazy_validate()
+ log.verbose("Using in-memory storage")
+ return InMemoryStorageProvider()
+
+
+def _make_s3_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:
+ if config.s3 is None:
+ msg = "S3 config is required when method is s3"
+ raise StorageConfigError(msg)
+ config.s3.lazy_validate()
+ log.verbose(f"Using S3 storage: bucket={config.s3.bucket_name}, region={config.s3.region}")
+ return S3StorageProvider(
+ bucket_name=config.s3.bucket_name,
+ region=config.s3.region,
+ signed_urls_lifespan=config.s3.signed_urls_lifespan,
+ )
+
+
+def _make_gcp_storage_provider(config: StorageProviderConfig) -> StorageProviderAbstract:
+ if config.gcp is None:
+ msg = "GCP config is required when method is gcp"
+ raise StorageConfigError(msg)
+ config.gcp.lazy_validate()
+ log.verbose(f"Using GCP storage: bucket={config.gcp.bucket_name}, project={config.gcp.project_id}")
+ return GcpStorageProvider(
+ bucket_name=config.gcp.bucket_name,
+ project_id=config.gcp.project_id,
+ # Hub secrets read is legal here: the factory runs at the boot apply-point, after the
+ # secrets provider is on the hub — never in register().
+ credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),
+ signed_urls_lifespan=config.gcp.signed_urls_lifespan,
+ )
+
+
+class StoragePlugin:
+ """Always-on built-in provider of the local / in_memory / s3 / gcp storage backends.
+
+ Core-unconditional: storage is required infra, so this plugin cannot be disabled into a
+ boot with no storage (see ``CORE_UNCONDITIONAL_PLUGIN_NAMES``). It registers one factory per
+ built-in method; ``storage_config.method`` selects which one boot invokes. Importing this
+ module is import-light — the s3/gcp SDKs load lazily inside their providers, not at register.
+ """
+
+ name = "storage"
+ targets_api = PLUGIN_API_VERSION
+
+ def register(self, registrar: PluginRegistrar) -> None:
+ registrar.add_storage_provider(method=StorageMethod.LOCAL, factory=_make_local_storage_provider)
+ registrar.add_storage_provider(method=StorageMethod.IN_MEMORY, factory=_make_in_memory_storage_provider)
+ registrar.add_storage_provider(method=StorageMethod.S3, factory=_make_s3_storage_provider)
+ registrar.add_storage_provider(method=StorageMethod.GCP, factory=_make_gcp_storage_provider)
diff --git a/pipelex/plugins/storage_provider_registry.py b/pipelex/plugins/storage_provider_registry.py
new file mode 100644
index 000000000..d49150875
--- /dev/null
+++ b/pipelex/plugins/storage_provider_registry.py
@@ -0,0 +1,36 @@
+from collections.abc import Callable
+
+from pipelex.plugins.exceptions import UnknownStorageMethodError
+from pipelex.tools.storage.storage_config import StorageProviderConfig
+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+
+# A plugin's factory for one storage backend: whole storage config in, provider out. The whole
+# config (not a pre-resolved sub-config) is passed so a factory can read whatever it needs —
+# including a hub secrets read (GCP) — at the boot apply-point, never at registration.
+StorageProviderFactoryFn = Callable[[StorageProviderConfig], StorageProviderAbstract]
+
+
+class StorageProviderRegistry:
+ """Read view over the storage-provider factories contributed by discovered plugins.
+
+ Keyed by the open storage ``method`` token (a ``str``; the built-ins keep the
+ ``StorageMethod`` values, an external plugin registers e.g. ``"azure"``). Built once at
+ boot from the registrar's accumulated ``storage_providers`` and stored on the hub; core
+ reads ``storage_config.method`` and calls the looked-up factory to produce the one provider.
+ """
+
+ def __init__(self, storage_providers: dict[str, StorageProviderFactoryFn]):
+ self._storage_providers: dict[str, StorageProviderFactoryFn] = dict(storage_providers)
+
+ def get_required(self, *, method: str) -> StorageProviderFactoryFn:
+ factory = self._storage_providers.get(method)
+ if factory is None:
+ raise UnknownStorageMethodError(method=method, registered_methods=self.methods)
+ return factory
+
+ def has(self, *, method: str) -> bool:
+ return method in self._storage_providers
+
+ @property
+ def methods(self) -> list[str]:
+ return list(self._storage_providers)
diff --git a/pipelex/system/configuration/configs.py b/pipelex/system/configuration/configs.py
index a4cd63827..52768c88a 100644
--- a/pipelex/system/configuration/configs.py
+++ b/pipelex/system/configuration/configs.py
@@ -13,6 +13,7 @@ from pipelex.system.configuration.config_model import ConfigModel
from pipelex.system.configuration.config_root import ConfigRoot
from pipelex.tools.aws.aws_config import AwsConfig
from pipelex.tools.log.log_config import LogConfig
+from pipelex.tools.secrets.secrets_config import SecretsProviderConfig
from pipelex.tools.storage.storage_config import StorageConfig
from pipelex.types import Self, StrEnum
@@ -198,6 +199,7 @@ class PipelineExecutionConfig(ConfigModel):
class Pipelex(ConfigModel):
storage_config: StorageConfig
+ secrets_config: SecretsProviderConfig
log_config: LogConfig
aws_config: AwsConfig
diff --git a/pipelex/tools/secrets/secrets_config.py b/pipelex/tools/secrets/secrets_config.py
new file mode 100644
index 000000000..5f23e754e
--- /dev/null
+++ b/pipelex/tools/secrets/secrets_config.py
@@ -0,0 +1,17 @@
+from pydantic import Field
+
+from pipelex.system.configuration.config_model import ConfigModel
+
+
+class SecretsProviderConfig(ConfigModel):
+ """Provider-selection config for the secrets-provider seam.
+
+ Mirrors ``StorageProviderConfig``'s open-token selection (S1): the built-in ``SecretsPlugin``
+ registers the ``env`` method; an external ``pipelex-secrets-<backend>`` plugin registers its own
+ (e.g. ``"vault"``). An unknown token is validated at registry lookup (``UnknownSecretsMethodError``
+ at boot), not at parse — so a config naming an external method still loads, and only the registry
+ decides what is installable. The built-in ``env`` method needs no per-method sub-config; a future
+ external-provider config passthrough is a scoped follow-up (S4).
+ """
+
+ method: str = Field(strict=False)
diff --git a/pipelex/tools/storage/storage_config.py b/pipelex/tools/storage/storage_config.py
index 98ddc61f5..95ccaa62a 100644
--- a/pipelex/tools/storage/storage_config.py
+++ b/pipelex/tools/storage/storage_config.py
@@ -171,7 +171,11 @@ class StorageGcpConfig(StorageBucketConfig):
class StorageProviderConfig(ConfigModel):
"""Provider-selection config shared by asset storage and payload codec storage."""
- method: StorageMethod = Field(strict=False)
+ # Open str token (D1): the built-ins use the StorageMethod values; an external
+ # ``pipelex-storage-<backend>`` plugin registers its own (e.g. "azure"). An unknown token is
+ # validated at registry lookup (UnknownStorageMethodError at boot), not at parse — so a config
+ # naming an external method still loads, and only the registry decides what is installable.
+ method: str = Field(strict=False)
local: StorageLocalConfig | None = None
in_memory: StorageInMemoryConfig | None = None
s3: StorageS3Config | None = None
@@ -196,6 +200,10 @@ class StorageProviderConfig(ConfigModel):
if not self.gcp:
msg = "gcp config is required when method is gcp"
raise StorageConfigError(msg)
+ case _:
+ # External method token (D1): no built-in sub-config is required. Whether the token
+ # names an installed provider is validated at registry lookup, not here.
+ pass
return self
@property
@@ -206,7 +214,7 @@ class StorageProviderConfig(ConfigModel):
msg = "local config is required to access storage_path"
raise StorageConfigError(msg)
return self.local.local_storage_path
- case StorageMethod.IN_MEMORY | StorageMethod.S3 | StorageMethod.GCP:
+ case _:
msg = f"storage_path is only available when method is local, but method is '{self.method}'"
raise StorageConfigError(msg)
@@ -233,6 +241,9 @@ class StorageProviderConfig(ConfigModel):
msg = "gcp config is required to access uri_format"
raise StorageConfigError(msg)
return self.gcp.uri_format
+ case _:
+ msg = f"uri_format is not available for method '{self.method}' (no built-in sub-config)"
+ raise StorageConfigError(msg)
class StorageConfig(StorageProviderConfig):
diff --git a/pipelex/tools/storage/storage_provider_factory.py b/pipelex/tools/storage/storage_provider_factory.py
deleted file mode 100644
index 4fff5088f..000000000
--- a/pipelex/tools/storage/storage_provider_factory.py
+++ /dev/null
@@ -1,64 +0,0 @@
-from pathlib import Path
-
-from pipelex import log
-from pipelex.hub import get_secrets_provider
-from pipelex.tools.storage.exceptions import StorageConfigError
-from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider
-from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
-from pipelex.tools.storage.local_storage_provider import LocalStorageProvider
-from pipelex.tools.storage.s3_storage_provider import S3StorageProvider
-from pipelex.tools.storage.storage_config import StorageMethod, StorageProviderConfig
-from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
-
-
-def make_storage_provider_from_config(storage_provider_config: StorageProviderConfig) -> StorageProviderAbstract:
- """Create a storage provider based on the provided storage configuration.
-
- Args:
- storage_provider_config: The storage configuration specifying method and provider-specific settings.
-
- Returns:
- A configured storage provider instance.
-
- Raises:
- StorageConfigError: If the required provider-specific config is missing or fails its lazy_validate checks.
- """
- match storage_provider_config.method:
- case StorageMethod.LOCAL:
- if storage_provider_config.local is None:
- msg = "local config is required when method is local"
- raise StorageConfigError(msg)
- storage_provider_config.local.lazy_validate()
- log.verbose(f"Using local storage at: {storage_provider_config.local.local_storage_path}")
- return LocalStorageProvider(root_path=Path(storage_provider_config.local.local_storage_path))
- case StorageMethod.IN_MEMORY:
- if storage_provider_config.in_memory is None:
- msg = "in_memory config is required when method is in_memory"
- raise StorageConfigError(msg)
- storage_provider_config.in_memory.lazy_validate()
- log.verbose("Using in-memory storage")
- return InMemoryStorageProvider()
- case StorageMethod.S3:
- if storage_provider_config.s3 is None:
- msg = "S3 config is required when method is s3"
- raise StorageConfigError(msg)
-
- storage_provider_config.s3.lazy_validate()
- log.verbose(f"Using S3 storage: bucket={storage_provider_config.s3.bucket_name}, region={storage_provider_config.s3.region}")
- return S3StorageProvider(
- bucket_name=storage_provider_config.s3.bucket_name,
- region=storage_provider_config.s3.region,
- signed_urls_lifespan=storage_provider_config.s3.signed_urls_lifespan,
- )
- case StorageMethod.GCP:
- if storage_provider_config.gcp is None:
- msg = "GCP config is required when method is gcp"
- raise StorageConfigError(msg)
- storage_provider_config.gcp.lazy_validate()
- log.verbose(f"Using GCP storage: bucket={storage_provider_config.gcp.bucket_name}, project={storage_provider_config.gcp.project_id}")
- return GcpStorageProvider(
- bucket_name=storage_provider_config.gcp.bucket_name,
- project_id=storage_provider_config.gcp.project_id,
- credentials_file_path=get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH"),
- signed_urls_lifespan=storage_provider_config.gcp.signed_urls_lifespan,
- )
diff --git a/tests/integration/pipelex/fixtures/generator_fixtures.py b/tests/integration/pipelex/fixtures/generator_fixtures.py
index 5f6d53d27..352a864d5 100644
--- a/tests/integration/pipelex/fixtures/generator_fixtures.py
+++ b/tests/integration/pipelex/fixtures/generator_fixtures.py
@@ -6,8 +6,8 @@ from pipelex.cogt.content_generation.content_generator import ContentGenerator
from pipelex.cogt.content_generation.content_generator_protocol import ContentGeneratorProtocol
from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
from pipelex.config import get_config
+from pipelex.hub import get_storage_provider_registry
from pipelex.tools.storage.storage_config import StorageMethod
-from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_config
S3_TEST_BUCKET = "pipelex-storage-test"
GCP_TEST_BUCKET = "pipelex-storage-test"
@@ -50,8 +50,11 @@ def generated_content_factory(tmp_path: Path) -> GeneratedContentFactory:
patched_provider_config = storage_provider_config.model_copy(update={"local": patched_local_config})
case StorageMethod.IN_MEMORY:
patched_provider_config = storage_provider_config
+ case _:
+ # External method token: no built-in sub-config to patch; select it as configured.
+ patched_provider_config = storage_provider_config
- storage_provider = make_storage_provider_from_config(patched_provider_config)
+ storage_provider = get_storage_provider_registry().get_required(method=patched_provider_config.method)(patched_provider_config)
return GeneratedContentFactory(storage_provider=storage_provider)
diff --git a/tests/integration/pipelex/plugins/test_secrets_external_plugin.py b/tests/integration/pipelex/plugins/test_secrets_external_plugin.py
new file mode 100644
index 000000000..aca8949a2
--- /dev/null
+++ b/tests/integration/pipelex/plugins/test_secrets_external_plugin.py
@@ -0,0 +1,158 @@
+"""End-to-end secrets-provider selection at boot: default env, explicit override, external discovery,
+and the storage↔secrets ordering guard.
+
+The built-in ``env`` method is the default and is selected onto the hub by an ordinary boot; an
+explicit ``setup(secrets_provider=...)`` still wins ahead of registry selection; a fake external
+method token discovered through a ``pipelex.plugins`` entry point is selectable via
+``secrets_config.method``. The final test is the ordering guard: with a non-env secrets method AND
+``gcp`` storage, storage's gcp factory reads the config-selected secrets provider from the hub —
+proving secrets is resolved and set on the hub before storage selection runs.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import TYPE_CHECKING
+
+import pytest
+from typing_extensions import override
+
+from pipelex.hub import get_secrets_provider, get_secrets_provider_registry, get_storage_provider
+from pipelex.pipelex import Pipelex
+from pipelex.plugins.contract import PLUGIN_API_VERSION
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
+from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from pytest_mock import MockerFixture
+
+ from pipelex.plugins.registrar import PluginRegistrar
+ from pipelex.tools.secrets.secrets_config import SecretsProviderConfig
+
+EXTERNAL_SECRETS_METHOD = "test_secret"
+EXTERNAL_PLUGIN_NAME = "test_secrets_ext"
+CANNED_CREDENTIALS_PATH = "/secrets/gcp-creds-from-external.json"
+
+
+class _RecordingSecretsProvider(SecretsProviderAbstract):
+ """An out-of-tree provider — its distinct type proves config-selection, and it records which
+ secrets were asked of it so the ordering guard can prove storage's gcp arm read *this* provider.
+ """
+
+ def __init__(self) -> None:
+ self.requested_secret_ids: list[str] = []
+
+ @override
+ def get_required_secret(self, secret_id: str) -> str:
+ self.requested_secret_ids.append(secret_id)
+ return CANNED_CREDENTIALS_PATH
+
+ @override
+ def get_optional_secret(self, secret_id: str) -> str | None:
+ return None
+
+ @override
+ def get_required_secret_specific_version(self, secret_id: str, *, version_id: str) -> str:
+ raise NotImplementedError
+
+ @override
+ def get_optional_secret_specific_version(self, secret_id: str, *, version_id: str) -> str | None:
+ raise NotImplementedError
+
+ @override
+ def set_secret_as_env_var(self, secret_id: str, *, version_id: str = "latest") -> None:
+ pass
+
+
+def _make_recording_secrets_provider(_config: SecretsProviderConfig) -> SecretsProviderAbstract:
+ return _RecordingSecretsProvider()
+
+
+class _FakeSecretsPlugin:
+ """An external plugin contributing one secrets method under the ``test_secret`` token."""
+
+ name = EXTERNAL_PLUGIN_NAME
+ targets_api = PLUGIN_API_VERSION
+
+ def register(self, registrar: PluginRegistrar) -> None:
+ registrar.add_secrets_provider(method=EXTERNAL_SECRETS_METHOD, factory=_make_recording_secrets_provider)
+
+
[email protected](autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Override the global module fixture: this module boots per test and tears down."""
+ Pipelex.teardown_if_needed()
+ yield
+ Pipelex.teardown_if_needed()
+
+
+def _test_integration_mode() -> IntegrationMode:
+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST
+
+
+def _patch_external_secrets_plugin(mocker: MockerFixture) -> None:
+ # The entry point resolves to the plugin class (a zero-arg factory); discovery instantiates it.
+ fake_entry_point = SimpleNamespace(name=EXTERNAL_PLUGIN_NAME, load=lambda: _FakeSecretsPlugin)
+ mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[fake_entry_point])
+
+
+class TestSecretsExternalPlugin:
+ def test_default_config_yields_the_env_provider_on_the_hub(self) -> None:
+ """An ordinary boot selects the built-in ``env`` method: an EnvSecretsProvider lands on the hub."""
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)
+
+ assert get_secrets_provider_registry().has(method="env")
+ assert isinstance(get_secrets_provider(), EnvSecretsProvider)
+
+ def test_explicit_secrets_provider_wins_over_config_selection(self) -> None:
+ """An explicit setup() param is the exact provider on the hub, ahead of registry selection."""
+ explicit = EnvSecretsProvider()
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, secrets_provider=explicit)
+
+ assert get_secrets_provider() is explicit
+
+ def test_external_secrets_method_is_discovered_and_selected_onto_the_hub(self, mocker: MockerFixture) -> None:
+ """A fake entry point + a config naming its token boots with that external provider on the hub."""
+ _patch_external_secrets_plugin(mocker)
+
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ config_overrides={"pipelex": {"secrets_config": {"method": EXTERNAL_SECRETS_METHOD}}},
+ )
+
+ assert get_secrets_provider_registry().has(method=EXTERNAL_SECRETS_METHOD)
+ assert isinstance(get_secrets_provider(), _RecordingSecretsProvider)
+
+ def test_config_selected_secrets_provider_is_read_by_storage_gcp_arm_at_boot(self, mocker: MockerFixture) -> None:
+ """Ordering guard: with a non-env secrets method AND gcp storage, storage's gcp factory reads the
+ config-selected external secrets provider from the hub. Secrets must be resolved and set on the hub
+ before storage selection — a regression that reordered them would leave the gcp arm without its
+ provider (or reading a stale env default), and this recorded request would not be the external one.
+ """
+ _patch_external_secrets_plugin(mocker)
+
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ config_overrides={
+ "pipelex": {
+ "secrets_config": {"method": EXTERNAL_SECRETS_METHOD},
+ "storage_config": {"method": "gcp", "gcp": {"bucket_name": "test-bucket", "project_id": "test-project"}},
+ }
+ },
+ )
+
+ storage_provider = get_storage_provider()
+ assert isinstance(storage_provider, GcpStorageProvider)
+ secrets_provider = get_secrets_provider()
+ assert isinstance(secrets_provider, _RecordingSecretsProvider)
+ # GCP_CREDENTIALS_FILE_PATH is requested only by storage's gcp factory, so its presence proves the
+ # gcp arm read *this* config-selected external provider from the hub at the boot apply-point. (The
+ # recorded list also carries the telemetry factory's POSTHOG_* reads — telemetry resolves secrets
+ # even earlier — which is why this is a membership check, not an exact-match.)
+ assert "GCP_CREDENTIALS_FILE_PATH" in secrets_provider.requested_secret_ids
diff --git a/tests/integration/pipelex/plugins/test_storage_external_plugin.py b/tests/integration/pipelex/plugins/test_storage_external_plugin.py
new file mode 100644
index 000000000..94fd2c36b
--- /dev/null
+++ b/tests/integration/pipelex/plugins/test_storage_external_plugin.py
@@ -0,0 +1,98 @@
+"""End-to-end: an out-of-tree storage plugin discovered through a ``pipelex.plugins`` entry point is
+selectable via ``storage_config.method`` and lands on the hub.
+
+This exercises the whole discovery → selection → hub chain the built-in providers ride, with a fake
+external method token: a fake entry point loads a plugin registering ``method="test_mem"``, the boot
+config selects that token, and the resolved provider is the one set on the hub.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import TYPE_CHECKING
+
+import pytest
+from typing_extensions import override
+
+from pipelex.hub import get_storage_provider, get_storage_provider_registry
+from pipelex.pipelex import Pipelex
+from pipelex.plugins.contract import PLUGIN_API_VERSION
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract, StoredData
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from pytest_mock import MockerFixture
+
+ from pipelex.plugins.registrar import PluginRegistrar
+ from pipelex.tools.storage.storage_config import StorageProviderConfig
+
+EXTERNAL_STORAGE_METHOD = "test_mem"
+EXTERNAL_PLUGIN_NAME = "test_storage_ext"
+
+
+class _FakeExternalStorageProvider(StorageProviderAbstract):
+ """A minimal out-of-tree provider — only its distinct type matters to the selection assertion."""
+
+ @override
+ async def _load_with_metadata(self, key: str) -> StoredData:
+ raise NotImplementedError
+
+ @override
+ async def _store(self, data: bytes, *, key: str, content_type: str | None) -> None:
+ raise NotImplementedError
+
+ @override
+ async def public_url(self, uri: str) -> str | None:
+ return None
+
+
+def _make_fake_external_provider(_config: StorageProviderConfig) -> StorageProviderAbstract:
+ return _FakeExternalStorageProvider()
+
+
+class _FakeStoragePlugin:
+ """An external plugin contributing one storage method under the ``test_mem`` token."""
+
+ name = EXTERNAL_PLUGIN_NAME
+ targets_api = PLUGIN_API_VERSION
+
+ def register(self, registrar: PluginRegistrar) -> None:
+ registrar.add_storage_provider(method=EXTERNAL_STORAGE_METHOD, factory=_make_fake_external_provider)
+
+
[email protected](autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Override the global module fixture: this module boots per test and tears down."""
+ Pipelex.teardown_if_needed()
+ yield
+ Pipelex.teardown_if_needed()
+
+
+def _test_integration_mode() -> IntegrationMode:
+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST
+
+
+class TestStorageExternalPlugin:
+ def test_explicit_storage_provider_wins_over_config_selection(self) -> None:
+ """An explicit setup() param is the exact provider on the hub, ahead of registry selection."""
+ explicit = _FakeExternalStorageProvider()
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, storage_provider=explicit)
+
+ assert get_storage_provider() is explicit
+
+ def test_external_storage_method_is_discovered_and_selected_onto_the_hub(self, mocker: MockerFixture) -> None:
+ """A fake entry point + a config naming its token boots with that external provider on the hub."""
+ # The entry point resolves to the plugin class (a zero-arg factory); discovery instantiates it.
+ fake_entry_point = SimpleNamespace(name=EXTERNAL_PLUGIN_NAME, load=lambda: _FakeStoragePlugin)
+ mocker.patch("pipelex.plugins.discovery._external_entry_points", return_value=[fake_entry_point])
+
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ config_overrides={"pipelex": {"storage_config": {"method": EXTERNAL_STORAGE_METHOD}}},
+ )
+
+ assert get_storage_provider_registry().has(method=EXTERNAL_STORAGE_METHOD)
+ assert isinstance(get_storage_provider(), _FakeExternalStorageProvider)
diff --git a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py
index 2d929bcb7..44fd94a2d 100644
--- a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py
+++ b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py
@@ -18,6 +18,8 @@ from pipelex.hub import get_content_generator, get_pipe_router
from pipelex.pipelex import Pipelex
from pipelex.plugins.registrar import HubSlot, PluginRegistrar
from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
if TYPE_CHECKING:
from pipelex.system.configuration.configs import PipelexConfig
@@ -36,7 +38,13 @@ def _test_integration_mode() -> IntegrationMode:
def _fake_registrar(mocker: MockerFixture) -> PluginRegistrar:
- """A registrar with empty registries (boot stores them but doesn't consume them during setup)."""
+ """A registrar with empty registries (boot stores them but doesn't consume them during setup).
+
+ Storage and secrets are the exceptions: boot *does* select each provider from the registrar during
+ setup, so every boot below passes an explicit ``storage_provider`` and ``secrets_provider`` — each
+ wins ahead of the (empty) registry — keeping this suite focused on hub slots rather than
+ storage/secrets selection.
+ """
registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(temporal=SimpleNamespace(is_enabled=False))))
mocker.patch("pipelex.pipelex.build_registrar", return_value=registrar)
return registrar
@@ -49,7 +57,12 @@ class TestHubSlotInjectionPrecedence:
registrar = _fake_registrar(mocker)
registrar.slot_claims[HubSlot.CONTENT_GENERATOR] = lambda: claimed
- Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ storage_provider=InMemoryStorageProvider(),
+ secrets_provider=EnvSecretsProvider(),
+ )
assert get_content_generator() is claimed
@@ -60,7 +73,13 @@ class TestHubSlotInjectionPrecedence:
registrar = _fake_registrar(mocker)
registrar.slot_claims[HubSlot.CONTENT_GENERATOR] = lambda: claimed
- Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, content_generator=explicit) # pyright: ignore[reportArgumentType]
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ content_generator=explicit, # pyright: ignore[reportArgumentType]
+ storage_provider=InMemoryStorageProvider(),
+ secrets_provider=EnvSecretsProvider(),
+ )
assert get_content_generator() is explicit
@@ -73,7 +92,13 @@ class TestHubSlotInjectionPrecedence:
# Claim PIPE_RUN too so boot does not build a default PipeRun from the sentinel router.
registrar.slot_claims[HubSlot.PIPE_RUN] = lambda: mocker.sentinel.claimed_run
- Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False, pipe_router=explicit_router) # pyright: ignore[reportArgumentType]
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ pipe_router=explicit_router, # pyright: ignore[reportArgumentType]
+ storage_provider=InMemoryStorageProvider(),
+ secrets_provider=EnvSecretsProvider(),
+ )
assert get_pipe_router() is explicit_router
@@ -89,7 +114,12 @@ class TestHubSlotInjectionPrecedence:
]
)
- Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ storage_provider=InMemoryStorageProvider(),
+ secrets_provider=EnvSecretsProvider(),
+ )
Pipelex.teardown_if_needed()
assert order == ["third", "second", "first"]
diff --git a/tests/integration/pipelex/tools/storage/test_storage_config.py b/tests/integration/pipelex/tools/storage/test_storage_config.py
index 1a2e58096..e76b666fb 100644
--- a/tests/integration/pipelex/tools/storage/test_storage_config.py
+++ b/tests/integration/pipelex/tools/storage/test_storage_config.py
@@ -41,3 +41,6 @@ class TestStorageConfigIntegration:
case StorageMethod.GCP:
assert storage_config.gcp is not None
assert storage_config.uri_format == storage_config.gcp.uri_format
+ case _:
+ # External method token (D1): no built-in sub-config to compare against.
+ pass
diff --git a/tests/unit/pipelex/plugins/test_import_light_boot.py b/tests/unit/pipelex/plugins/test_import_light_boot.py
index ce4c44fe1..38e70ded3 100644
--- a/tests/unit/pipelex/plugins/test_import_light_boot.py
+++ b/tests/unit/pipelex/plugins/test_import_light_boot.py
@@ -18,6 +18,7 @@ _GUARD_SCRIPT = textwrap.dedent(
"anthropic",
"mistralai",
"google.genai",
+ "google.cloud.storage",
"boto3",
"aioboto3",
"fal_client",
@@ -58,6 +59,10 @@ _GUARD_SCRIPT = textwrap.dedent(
assert registrar.inference_backends, "expected the built-in LLM backends to be registered"
assert registrar.model_listers, "expected the built-in model listers to be registered import-light"
assert registrar.orchestrators, "expected the built-in orchestrators to be registered"
+ # The built-in StoragePlugin registers all four methods without importing aioboto3 /
+ # google-cloud-storage: the s3/gcp SDK guards live inside the providers' I/O methods, so
+ # registration stays import-light even though those SDKs are BLOCKED above.
+ assert registrar.storage_providers, "expected the built-in storage providers to be registered import-light"
print("import-light OK")
"""
)
diff --git a/tests/unit/pipelex/plugins/test_plugin_discovery.py b/tests/unit/pipelex/plugins/test_plugin_discovery.py
index a6884e1a4..5dc04a1ce 100644
--- a/tests/unit/pipelex/plugins/test_plugin_discovery.py
+++ b/tests/unit/pipelex/plugins/test_plugin_discovery.py
@@ -191,11 +191,12 @@ class TestPluginDiscovery:
disabled_discovery = next(discovery for discovery in registrar.discoveries if discovery.name == "optional")
assert disabled_discovery.status == PluginStatus.DISABLED
- def test_disabling_core_unconditional_plugin_raises(self) -> None:
- """Denylisting a plugin core requires unconditionally (e.g. openai) is a startup error."""
+ @pytest.mark.parametrize("plugin_name", ["direct", "storage", "secrets", "openai"])
+ def test_disabling_core_unconditional_plugin_raises(self, plugin_name: str) -> None:
+ """Denylisting any plugin core requires unconditionally (storage/secrets included) is a startup error."""
with pytest.raises(CoreUnconditionalPluginDisabledError) as exc_info:
- build_registrar(config=_fake_config(["openai"]))
- assert "openai" in str(exc_info.value)
+ build_registrar(config=_fake_config([plugin_name]))
+ assert plugin_name in str(exc_info.value)
def test_discoveries_describe_builtins(self) -> None:
"""build_registrar records each built-in with origin/status/contributions for `plugins list`."""
diff --git a/tests/unit/pipelex/plugins/test_secrets_plugin.py b/tests/unit/pipelex/plugins/test_secrets_plugin.py
new file mode 100644
index 000000000..cd4c1fa2e
--- /dev/null
+++ b/tests/unit/pipelex/plugins/test_secrets_plugin.py
@@ -0,0 +1,49 @@
+"""The built-in SecretsPlugin: registers the one ``env`` factory, which constructs an EnvSecretsProvider.
+
+Secrets is a single-impl seam today — the ``env`` provider reads secrets from env vars and needs no
+config, so its factory ignores the config it is handed. Registration is import-light (no SDK); an
+out-of-tree ``pipelex-secrets-<backend>`` plugin would defer its SDK import into its own factory.
+Mirrors ``test_storage_plugin`` (kept lean: one built-in method, no optional-dep matrix).
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import TYPE_CHECKING, cast
+
+from pipelex.plugins.contract import PLUGIN_API_VERSION
+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar
+from pipelex.plugins.secrets.secrets_plugin import SecretsPlugin
+from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry
+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
+from pipelex.tools.secrets.secrets_config import SecretsProviderConfig
+
+if TYPE_CHECKING:
+ from pipelex.system.configuration.configs import PipelexConfig
+
+
+def _build_secrets_registry() -> SecretsProviderRegistry:
+ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))
+ SecretsPlugin().register(registrar)
+ return SecretsProviderRegistry(registrar.secrets_providers)
+
+
+class TestSecretsPlugin:
+ def test_registers_the_env_factory(self) -> None:
+ """SecretsPlugin is a named, API-versioned builtin registering exactly the built-in ``env`` method."""
+ plugin = SecretsPlugin()
+ assert plugin.name == "secrets"
+ assert plugin.targets_api == PLUGIN_API_VERSION
+
+ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))
+ discovery = registrar.begin_plugin(name="secrets", origin=PluginOrigin.BUILTIN, targets_api=PLUGIN_API_VERSION)
+ plugin.register(registrar)
+
+ assert set(registrar.secrets_providers) == {"env"}
+ assert "secrets provider env" in discovery.contributions
+
+ def test_env_factory_constructs_an_env_secrets_provider(self) -> None:
+ """Selecting ``env`` through the registry constructs an EnvSecretsProvider (the factory ignores its config)."""
+ registry = _build_secrets_registry()
+ provider = registry.get_required(method="env")(SecretsProviderConfig(method="env"))
+ assert isinstance(provider, EnvSecretsProvider)
diff --git a/tests/unit/pipelex/plugins/test_secrets_provider_registry.py b/tests/unit/pipelex/plugins/test_secrets_provider_registry.py
new file mode 100644
index 000000000..e0af20edc
--- /dev/null
+++ b/tests/unit/pipelex/plugins/test_secrets_provider_registry.py
@@ -0,0 +1,96 @@
+"""The secrets-provider seam: the registrar collects per-method provider factories into a registry.
+
+Pins the seam independent of the real built-in ``env`` provider: a factory registered for a method is
+the one the registry hands back; a miss raises ``UnknownSecretsMethodError`` listing the registered
+methods; a second registration for the same method fails loud naming both plugins; an empty registry
+misses every method. Mirrors ``test_storage_provider_registry``.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import TYPE_CHECKING, cast
+
+import pytest
+
+from pipelex.plugins.contract import PLUGIN_API_VERSION
+from pipelex.plugins.exceptions import DuplicateSecretsProviderError, UnknownSecretsMethodError
+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar
+from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry
+from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
+
+if TYPE_CHECKING:
+ from pipelex.system.configuration.configs import PipelexConfig
+ from pipelex.tools.secrets.secrets_config import SecretsProviderConfig
+ from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
+
+
+def _make_registrar() -> PluginRegistrar:
+ return PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))
+
+
+def _fake_factory(_config: SecretsProviderConfig) -> SecretsProviderAbstract:
+ """Stand-in factory: identity is all the registry tests assert (never actually invoked here)."""
+ return EnvSecretsProvider()
+
+
+class TestSecretsProviderRegistry:
+ def test_registered_factory_is_retrievable_by_method(self) -> None:
+ """A factory registered for a method is the exact callable the built registry returns for it."""
+ registrar = _make_registrar()
+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)
+
+ registry = SecretsProviderRegistry(registrar.secrets_providers)
+ assert registry.get_required(method="vault") is _fake_factory
+ assert registry.has(method="vault")
+ assert registry.methods == ["vault"]
+
+ def test_contribution_recorded_on_the_active_plugin(self) -> None:
+ """Registering a provider records a ``secrets provider <method>`` contribution on the plugin's discovery."""
+ registrar = _make_registrar()
+ discovery = registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+
+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)
+
+ assert "secrets provider vault" in discovery.contributions
+
+ def test_get_required_miss_raises_listing_registered_methods(self) -> None:
+ """A miss names the requested method and lists the registered ones (the boot-time actionable error)."""
+ registrar = _make_registrar()
+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+ registrar.add_secrets_provider(method="env", factory=_fake_factory)
+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)
+ registry = SecretsProviderRegistry(registrar.secrets_providers)
+
+ with pytest.raises(UnknownSecretsMethodError) as exc_info:
+ registry.get_required(method="aws")
+
+ assert exc_info.value.method == "aws"
+ assert set(exc_info.value.registered_methods) == {"env", "vault"}
+ message = str(exc_info.value)
+ assert "env" in message
+ assert "vault" in message
+
+ def test_empty_registry_misses_every_method(self) -> None:
+ """A registry with no factories misses every method — soft via has, loud via get_required."""
+ registry = SecretsProviderRegistry({})
+
+ assert not registry.has(method="env")
+ assert registry.methods == []
+ with pytest.raises(UnknownSecretsMethodError):
+ registry.get_required(method="env")
+
+ def test_duplicate_method_fails_loud_naming_both_plugins(self) -> None:
+ """Two plugins registering a provider for the same method is a fail-loud conflict naming both."""
+ registrar = _make_registrar()
+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)
+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+
+ with pytest.raises(DuplicateSecretsProviderError) as exc_info:
+ registrar.add_secrets_provider(method="vault", factory=_fake_factory)
+
+ assert exc_info.value.first_plugin == "alpha"
+ assert exc_info.value.second_plugin == "beta"
+ assert exc_info.value.method == "vault"
diff --git a/tests/unit/pipelex/plugins/test_storage_plugin.py b/tests/unit/pipelex/plugins/test_storage_plugin.py
new file mode 100644
index 000000000..c25eb8129
--- /dev/null
+++ b/tests/unit/pipelex/plugins/test_storage_plugin.py
@@ -0,0 +1,115 @@
+"""The built-in StoragePlugin: registers a factory for every built-in method, and each factory
+constructs the matching provider at the boot apply-point.
+
+The construction is deferred — selecting ``s3``/``gcp`` builds the provider without importing its
+optional SDK (the SDK guard lives inside the provider's I/O methods, not its ``__init__``), and the
+``gcp`` factory reads its credentials from the hub secrets provider at that apply-point. Registration
+itself never touches an SDK; the import-light guarantee is pinned by ``test_import_light_boot``.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import TYPE_CHECKING, Any, Callable, cast
+
+import pytest
+
+from pipelex.plugins.contract import PLUGIN_API_VERSION
+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar
+from pipelex.plugins.storage.storage_plugin import StoragePlugin
+from pipelex.plugins.storage_provider_registry import StorageProviderRegistry
+from pipelex.tools.storage.gcp_storage_provider import GcpStorageProvider
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
+from pipelex.tools.storage.local_storage_provider import LocalStorageProvider
+from pipelex.tools.storage.s3_storage_provider import S3StorageProvider
+from pipelex.tools.storage.storage_config import StorageMethod, StorageProviderConfig
+from tests.unit.pipelex.tools.storage.test_storage_provider_config import (
+ make_gcp_config,
+ make_in_memory_config,
+ make_local_config,
+ make_s3_config,
+)
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+ from pipelex.system.configuration.configs import PipelexConfig
+ from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+
+
+class _FakeSecretsProvider:
+ """Records which secrets the gcp factory asks for and returns a canned credentials path."""
+
+ def __init__(self, credentials_path: str) -> None:
+ self.credentials_path = credentials_path
+ self.requested_secret_ids: list[str] = []
+
+ def get_required_secret(self, *, secret_id: str) -> str:
+ self.requested_secret_ids.append(secret_id)
+ return self.credentials_path
+
+
+def _build_storage_registry() -> StorageProviderRegistry:
+ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))
+ StoragePlugin().register(registrar)
+ return StorageProviderRegistry(registrar.storage_providers)
+
+
+class TestStoragePlugin:
+ def test_registers_a_factory_for_every_builtin_method(self) -> None:
+ """StoragePlugin is a named, API-versioned builtin registering one factory per StorageMethod value."""
+ plugin = StoragePlugin()
+ assert plugin.name == "storage"
+ assert plugin.targets_api == PLUGIN_API_VERSION
+
+ registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))
+ discovery = registrar.begin_plugin(name="storage", origin=PluginOrigin.BUILTIN, targets_api=PLUGIN_API_VERSION)
+ plugin.register(registrar)
+
+ # StrEnum keys compare equal to their plain-str form, so boot's plain-str config.method resolves them.
+ assert set(registrar.storage_providers) == {"local", "in_memory", "s3", "gcp"}
+ assert set(registrar.storage_providers) == set(StorageMethod)
+ for method in StorageMethod:
+ assert f"storage provider {method}" in discovery.contributions
+
+ @pytest.mark.parametrize(
+ ("method", "field_name", "sub_config_factory", "expected_type"),
+ [
+ pytest.param(StorageMethod.LOCAL, "local", make_local_config, LocalStorageProvider, id="local"),
+ pytest.param(StorageMethod.IN_MEMORY, "in_memory", make_in_memory_config, InMemoryStorageProvider, id="in-memory"),
+ pytest.param(StorageMethod.S3, "s3", make_s3_config, S3StorageProvider, id="s3"),
+ ],
+ )
+ def test_factory_constructs_the_matching_provider(
+ self,
+ method: StorageMethod,
+ field_name: str,
+ sub_config_factory: Callable[[], Any],
+ expected_type: type[StorageProviderAbstract],
+ ) -> None:
+ """Selecting a method through the registry constructs its provider (s3 needs no aioboto3 to build)."""
+ registry = _build_storage_registry()
+ kwargs: dict[str, Any] = {"method": method, field_name: sub_config_factory()}
+ provider = registry.get_required(method=method)(StorageProviderConfig(**kwargs))
+ assert isinstance(provider, expected_type)
+
+ def test_gcp_factory_reads_credentials_from_the_hub_secrets_provider(self, mocker: MockerFixture) -> None:
+ """The gcp factory resolves GCP_CREDENTIALS_FILE_PATH from the hub secrets provider and passes it through."""
+ fake_secrets = _FakeSecretsProvider(credentials_path="/secrets/gcp-creds.json")
+ mocker.patch("pipelex.plugins.storage.storage_plugin.get_secrets_provider", return_value=fake_secrets)
+ # Spy the constructor (wraps → the real provider is still built and returned) to assert the
+ # resolved secret value flows through as credentials_file_path, not just that it was requested.
+ gcp_ctor = mocker.patch("pipelex.plugins.storage.storage_plugin.GcpStorageProvider", wraps=GcpStorageProvider)
+
+ gcp_config = make_gcp_config()
+ registry = _build_storage_registry()
+ provider = registry.get_required(method=StorageMethod.GCP)(StorageProviderConfig(method=StorageMethod.GCP, gcp=gcp_config))
+
+ assert isinstance(provider, GcpStorageProvider)
+ assert fake_secrets.requested_secret_ids == ["GCP_CREDENTIALS_FILE_PATH"]
+ gcp_ctor.assert_called_once_with(
+ bucket_name=gcp_config.bucket_name,
+ project_id=gcp_config.project_id,
+ credentials_file_path=fake_secrets.credentials_path,
+ signed_urls_lifespan=gcp_config.signed_urls_lifespan,
+ )
diff --git a/tests/unit/pipelex/plugins/test_storage_provider_registry.py b/tests/unit/pipelex/plugins/test_storage_provider_registry.py
new file mode 100644
index 000000000..2139967b3
--- /dev/null
+++ b/tests/unit/pipelex/plugins/test_storage_provider_registry.py
@@ -0,0 +1,96 @@
+"""The storage-provider seam: the registrar collects per-method provider factories into a registry.
+
+Pins the seam independent of the real built-in providers: a factory registered for a method is the
+one the registry hands back; a miss raises ``UnknownStorageMethodError`` listing the registered
+methods; a second registration for the same method fails loud naming both plugins; an empty registry
+misses every method.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import TYPE_CHECKING, cast
+
+import pytest
+
+from pipelex.plugins.contract import PLUGIN_API_VERSION
+from pipelex.plugins.exceptions import DuplicateStorageProviderError, UnknownStorageMethodError
+from pipelex.plugins.registrar import PluginOrigin, PluginRegistrar
+from pipelex.plugins.storage_provider_registry import StorageProviderRegistry
+from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
+
+if TYPE_CHECKING:
+ from pipelex.system.configuration.configs import PipelexConfig
+ from pipelex.tools.storage.storage_config import StorageProviderConfig
+ from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+
+
+def _make_registrar() -> PluginRegistrar:
+ return PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(plugins=SimpleNamespace(disabled=[]))))
+
+
+def _fake_factory(_config: StorageProviderConfig) -> StorageProviderAbstract:
+ """Stand-in factory: identity is all the registry tests assert (never actually invoked here)."""
+ return InMemoryStorageProvider()
+
+
+class TestStorageProviderRegistry:
+ def test_registered_factory_is_retrievable_by_method(self) -> None:
+ """A factory registered for a method is the exact callable the built registry returns for it."""
+ registrar = _make_registrar()
+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+ registrar.add_storage_provider(method="azure", factory=_fake_factory)
+
+ registry = StorageProviderRegistry(registrar.storage_providers)
+ assert registry.get_required(method="azure") is _fake_factory
+ assert registry.has(method="azure")
+ assert registry.methods == ["azure"]
+
+ def test_contribution_recorded_on_the_active_plugin(self) -> None:
+ """Registering a provider records a ``storage provider <method>`` contribution on the plugin's discovery."""
+ registrar = _make_registrar()
+ discovery = registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+
+ registrar.add_storage_provider(method="azure", factory=_fake_factory)
+
+ assert "storage provider azure" in discovery.contributions
+
+ def test_get_required_miss_raises_listing_registered_methods(self) -> None:
+ """A miss names the requested method and lists the registered ones (the boot-time actionable error)."""
+ registrar = _make_registrar()
+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+ registrar.add_storage_provider(method="local", factory=_fake_factory)
+ registrar.add_storage_provider(method="s3", factory=_fake_factory)
+ registry = StorageProviderRegistry(registrar.storage_providers)
+
+ with pytest.raises(UnknownStorageMethodError) as exc_info:
+ registry.get_required(method="azure")
+
+ assert exc_info.value.method == "azure"
+ assert set(exc_info.value.registered_methods) == {"local", "s3"}
+ message = str(exc_info.value)
+ assert "local" in message
+ assert "s3" in message
+
+ def test_empty_registry_misses_every_method(self) -> None:
+ """A registry with no factories misses every method — soft via has, loud via get_required."""
+ registry = StorageProviderRegistry({})
+
+ assert not registry.has(method="local")
+ assert registry.methods == []
+ with pytest.raises(UnknownStorageMethodError):
+ registry.get_required(method="local")
+
+ def test_duplicate_method_fails_loud_naming_both_plugins(self) -> None:
+ """Two plugins registering a provider for the same method is a fail-loud conflict naming both."""
+ registrar = _make_registrar()
+ registrar.begin_plugin(name="alpha", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+ registrar.add_storage_provider(method="azure", factory=_fake_factory)
+ registrar.begin_plugin(name="beta", origin=PluginOrigin.EXTERNAL, targets_api=PLUGIN_API_VERSION)
+
+ with pytest.raises(DuplicateStorageProviderError) as exc_info:
+ registrar.add_storage_provider(method="azure", factory=_fake_factory)
+
+ assert exc_info.value.first_plugin == "alpha"
+ assert exc_info.value.second_plugin == "beta"
+ assert exc_info.value.method == "azure"
diff --git a/tests/unit/pipelex/tools/storage/test_storage_provider_config.py b/tests/unit/pipelex/tools/storage/test_storage_provider_config.py
index 37fda6344..bb52dee33 100644
--- a/tests/unit/pipelex/tools/storage/test_storage_provider_config.py
+++ b/tests/unit/pipelex/tools/storage/test_storage_provider_config.py
@@ -88,11 +88,16 @@ class TestStorageProviderConfig:
StorageProviderConfig(method=method)
assert expected_message in str(exc_info.value)
- def test_method_string_coercion_despite_strict_mode(self):
- """The method field is Field(strict=False), so a plain string coerces to StorageMethod under strict config."""
+ def test_method_is_open_str_token(self):
+ """D1: method is an open str token stored verbatim. A built-in name is a plain string (equal to its
+ StorageMethod value), and an unknown/external name is accepted at parse — its installability is
+ validated later at registry lookup, not at parse — so a config naming an out-of-tree provider loads.
+ """
provider_config = StorageProviderConfig.model_validate({"method": "local", "local": make_local_config()})
- assert isinstance(provider_config.method, StorageMethod)
+ assert provider_config.method == "local"
assert provider_config.method == StorageMethod.LOCAL
+ external_config = StorageProviderConfig.model_validate({"method": "azure"})
+ assert external_config.method == "azure"
@pytest.mark.parametrize(
("method", "field_name", "sub_config_factory", "expected_uri_format"),
diff --git a/tests/unit/pipelex/tools/storage/test_storage_provider_factory.py b/tests/unit/pipelex/tools/storage/test_storage_provider_factory.py
deleted file mode 100644
index 051abc73f..000000000
--- a/tests/unit/pipelex/tools/storage/test_storage_provider_factory.py
+++ /dev/null
@@ -1,52 +0,0 @@
-from pathlib import Path
-
-import pytest
-
-from pipelex.tools.storage.exceptions import StorageConfigError
-from pipelex.tools.storage.in_memory_storage_provider import InMemoryStorageProvider
-from pipelex.tools.storage.local_storage_provider import LocalStorageProvider
-from pipelex.tools.storage.storage_config import (
- StorageInMemoryConfig,
- StorageLocalConfig,
- StorageMethod,
- StorageProviderConfig,
-)
-from pipelex.tools.storage.storage_provider_factory import make_storage_provider_from_config
-
-
-class TestStorageProviderFactory:
- def test_local_method_returns_local_provider(self, tmp_path: Path):
- """A valid local config yields a LocalStorageProvider."""
- provider_config = StorageProviderConfig(
- method=StorageMethod.LOCAL,
- local=StorageLocalConfig(uri_format="assets/{hash}", local_storage_path=str(tmp_path / "store")),
- )
- provider = make_storage_provider_from_config(provider_config)
- assert isinstance(provider, LocalStorageProvider)
-
- def test_in_memory_method_returns_in_memory_provider(self):
- """A valid in-memory config yields an InMemoryStorageProvider."""
- provider_config = StorageProviderConfig(
- method=StorageMethod.IN_MEMORY,
- in_memory=StorageInMemoryConfig(uri_format="assets/{hash}"),
- )
- provider = make_storage_provider_from_config(provider_config)
- assert isinstance(provider, InMemoryStorageProvider)
-
- def test_local_config_is_validated_before_provider_construction(self, tmp_path: Path):
- """A local uri_format without a {hash} placeholder is rejected when the provider is built."""
- provider_config = StorageProviderConfig(
- method=StorageMethod.LOCAL,
- local=StorageLocalConfig(uri_format="assets/constant", local_storage_path=str(tmp_path / "store")),
- )
- with pytest.raises(StorageConfigError, match="uri_format must contain a"):
- make_storage_provider_from_config(provider_config)
-
- def test_in_memory_config_is_validated_before_provider_construction(self):
- """An in-memory uri_format without a {hash} placeholder is rejected when the provider is built."""
- provider_config = StorageProviderConfig(
- method=StorageMethod.IN_MEMORY,
- in_memory=StorageInMemoryConfig(uri_format="assets/constant"),
- )
- with pytest.raises(StorageConfigError, match="uri_format must contain a"):
- make_storage_provider_from_config(provider_config)
diff --git a/wip/plugins/README.md b/wip/plugins/README.md
new file mode 100644
index 000000000..19e8654d2
--- /dev/null
+++ b/wip/plugins/README.md
@@ -0,0 +1,34 @@
+# Provider plugins track — storage & secrets
+
+Two features that promote existing dependency-injection seams to formal `pipelex.plugins` entry-point plugins, so third parties can ship `pipelex-storage-azure` / `pipelex-secrets-vault` packages selected at deploy time.
+
+- **[storage-provider-plugin.md](storage-provider-plugin.md)** — the storage provider seam (local / in_memory / s3 / gcp today). **Land this first** — it has the real multi-implementation need and exercises the new pattern end to end.
+- **[secrets-provider-plugin.md](secrets-provider-plugin.md)** — the secrets provider seam (env only today). Lands second, mirroring the storage pattern; validates that the mechanism generalizes to a second seam.
+
+Both were selected after a full audit of the boot-time DI seams (see the classification below). They are the only two seams that pass all three plugin criteria; everything else is either already a plugin, a boot-orchestrator-owned hub slot, or a population/config point that should stay one.
+
+## The shared new pattern these introduce
+
+Neither seam maps onto an existing plugin pattern. The current keyed registries (inference backends, model listers, orchestrators, bundle validators) are selected **per-request/per-model** by a wire or model field. The hub slots (content generator, pipe router, pipe run, task manager, isolated-execution probe) are singletons but selected by **one name gate** — `plugins.boot_orchestrator` — because they move as a coherent set with the runtime flavor.
+
+Storage and secrets are neither. Each is a **process-global singleton selected by its *own* config key**, independent of the orchestrator. So they need a net-new mechanism:
+
+> **Keyed registry + config-selected singleton.** Plugins register N provider factories into a registry keyed by an open method token (mirroring `OrchestrationMode`). At boot, core reads a single config field (`storage_config.method` / `secrets_config.method`), looks that token up in the registry, and calls the factory to produce the one provider set on the hub.
+
+This is deliberately **not** a `HubSlot` — hub slots are orchestrator-coupled (claimed inside `register` iff `boot_orchestrator == plugin.name`). Storage/secrets selection has nothing to do with the orchestrator, so reusing the slot machinery would wrongly couple them. The storage plan defines this pattern in full (§ "The new mechanism"); the secrets plan references it.
+
+## Cross-cutting decisions (apply to both)
+
+- **DX-1 — Batch the plugin-API version bump.** Discovery version-checks `targets_api` against `PLUGIN_API_VERSION` with **strict equality** (`pipelex/plugins/discovery.py`, `_register_plugin`), so *any* additive menu growth is a breaking bump that forces every external plugin to re-declare `targets_api`. Introduce **both** new registrar menu methods (`add_storage_provider` and `add_secrets_provider`) under a **single** `PLUGIN_API_VERSION` bump (2 → 3), even though the two features land in sequence, to avoid double-breaking the external plugins. The known external plugins that must bump their `targets_api` to 3: `pipelex-temporal`, `pipelex-mistralai-workflows`. (Per the workspace "no backward compatibility" principle, that's acceptable — note it in changelogs; there is no transition shim.)
+ - *Option not taken:* relax discovery to accept `targets_api <= PLUGIN_API_VERSION` for additive growth. That would make menu additions non-breaking but changes the plugin-system contract semantics — out of scope for this track; raise separately if the coarse-bump churn becomes painful.
+- **DX-2 — The builtin provider plugin is unconditional.** Storage and secrets are required infra: if the config-selected method has no registered factory, boot must fail loud. The builtin `StoragePlugin` / `SecretsPlugin` therefore join `CORE_UNCONDITIONAL_PLUGIN_NAMES` (`pipelex/plugins/builtins.py`) so they can't be disabled into a broken boot.
+- **DX-3 — External-provider config surface is a scoped follow-up.** `StorageProviderConfig` / the new `SecretsProviderConfig` have fixed, typed per-method sub-models. An out-of-tree provider (`azure`, `vault`) needs somewhere to read *its* config. Phase 1 of each plan lands the seam for the built-in methods only; a generic passthrough sub-config for external providers is captured as a follow-up in each plan, not built speculatively.
+- **DX-4 — Parallel keyed registries stay un-abstracted (declined in this track; possible whole-pattern cleanup later).** `SecretsProviderRegistry` and `StorageProviderRegistry` are near-identical (a PR-review finding on [#1024](https://github.com/Pipelex/pipelex/pull/1024) proposed extracting a shared generic base so lookup/error behavior can't silently diverge). **Decision: not done here.** The genuinely-shared logic is ~5 trivial lines (`.get()` → None-check → raise → return); against that, six sibling plugin registries already exist (`storage`, `secrets`, `orchestrator`, `bundle_validator`, `model_lister`, `inference_backend`) and **none share a base** — parallel registries are the established, tolerated convention. Their behavior deliberately diverges in load-bearing ways: miss-semantics differ (hard-raise via `get_required` vs soft-`None` via `get_optional`), key types differ (`str` / `tuple[InferenceFamily, str]` / `OrchestrationMode`), and each raises a distinct, per-domain exception with its own `type_uri` doc page. Abstracting only storage+secrets would leave 2-of-6 sharing a base and 4 not — an asymmetry more confusing than uniform duplication. **Escape hatch:** if the coarse duplication ever bites, do it as a *separate whole-pattern* cleanup — a `KeyedRegistry[K, V]` generic (dict copy + `has` + keys accessor + `get_optional`) with a per-domain `on_missing` hook so each concrete registry keeps its own exception and keys-property name, touching all six `pipelex/plugins/*_registry.py`. Not this PR; matches the "smallest correct surface, no speculative additions" principle.
+
+## Boot-audit classification (why only these two)
+
+- **Already plugins:** inference backends (LLM/img-gen/extract/search, incl. OCR & web search), model listers, orchestrators, bundle validators, HTTP error mappers.
+- **Boot-orchestrator-owned hub slots (leave as-is):** content generator, pipe router, pipe run, task manager, isolated-execution probe — they move as a set with the runtime flavor.
+- **Population / config points (NOT plugins):** class registry, func registry, model deck, template sets. Where under-modular (templates especially, which lack even an injection param today), the fix is a normal injection seam, not entry-point discovery.
+- **Prove-a-second-impl-first:** telemetry (don't double-wrap OTel's own exporter plugins), reporting/cost delegate (looks host-runtime-owned, not shelf-installable).
+- **Promote (this track):** **secrets** and **storage**.
diff --git a/wip/plugins/secrets-provider-plugin.md b/wip/plugins/secrets-provider-plugin.md
new file mode 100644
index 000000000..cdb382236
--- /dev/null
+++ b/wip/plugins/secrets-provider-plugin.md
@@ -0,0 +1,135 @@
+# Secrets provider → `pipelex.plugins` plugin
+
+Status: **COMPLETE.** All phases delivered — this vertical (Phases 3–4 of the master plan) mirrors the storage pattern ([storage-provider-plugin.md](storage-provider-plugin.md), which defines the shared "keyed registry + config-selected singleton" mechanism this plan reuses wholesale). See [TODOS.md](../../TODOS.md) for the execution record (Phases 3–4 code + tests + docs, Phase 5 release gating). This plan reads as "mirror storage, swap the nouns."
+
+Goal: turn the secrets provider into a formal plugin seam so third parties can ship `pipelex-secrets-<backend>` packages (Vault, AWS Secrets Manager, GCP Secret Manager, …) discovered via `pipelex.plugins` and selected at deploy time. The built-in `EnvSecretsProvider` becomes an unconditional builtin plugin.
+
+Read [README.md](README.md) for the track's shared decisions (DX-1 API bump, DX-2 unconditional builtin, DX-3 external config follow-up).
+
+---
+
+## Cold-start context (the seam as it is today)
+
+**Abstract base** — `pipelex/tools/secrets/secrets_provider_abstract.py`
+- `SecretsProviderAbstract(ABC)` (line 6). Abstract: `get_required_secret(secret_id)`, `get_optional_secret(secret_id)`, `get_required_secret_specific_version(secret_id, *, version_id)`, `get_optional_secret_specific_version(secret_id, *, version_id)`, `set_secret_as_env_var(secret_id, *, version_id=LATEST_SECRET_VERSION_NAME)`. Concrete convenience `get_secret(secret_id) -> str` delegates to `get_required_secret`.
+- `LATEST_SECRET_VERSION_NAME = "latest"` (line 3).
+
+**The one implementation** — `pipelex/tools/secrets/env_secrets_provider.py`
+- `EnvSecretsProvider` (line 9). Reads env vars via `get_required_env` / `get_optional_env`. The two `*_specific_version` methods `raise NotImplementedError`; `set_secret_as_env_var` is a no-op `pass`. **This is a single-impl seam** — the whole point of the plan is to open it.
+
+**Boot injection** — `pipelex/pipelex.py`
+- Import line 90; `setup()` param `secrets_provider: SecretsProviderAbstract | None = None` (line 186).
+- **Hardcoded default:** line 212 `secrets_provider = secrets_provider or EnvSecretsProvider()` — constructed **early**, before the gateway check (comment: "needed for gateway check"), and before config-driven subsystems. This early construction is the main structural wrinkle (see § The wrinkle).
+- Passed to telemetry factory (288) and to two setup helpers (345, 375). Set on hub line 306. `make()` mirror: param 560, forwarded 636.
+
+**Config** — **none today.** No field selects the provider; it is hardcoded to Env via the `or EnvSecretsProvider()` fallback. There is no `secrets_config` in `pipelex.toml` or `configs.py`. This plan introduces one.
+
+**Hub accessors** — `pipelex/hub.py`: `set_secrets_provider` (183), `get_required_secrets_provider()` (294), module-level `get_secrets_provider()` (472) and `get_secret(secret_id)` (607). Consumers (~4 files): `storage_provider_factory.py:62` (GCP creds — see the storage↔secrets coupling in the storage plan), `linkup_search_worker.py`, `linkup_extract_worker.py`, `show_cmd.py`. `secrets_utils.py` takes a `secrets_provider` param explicitly rather than pulling from the hub. **None of these change.**
+
+**Pattern to mirror:** identical to the storage plan — `PipelexPlugin` contract, the registrar `_add` helper, `OrchestratorRegistry` as the read-view template, `docling_plugin.py` for the lazy optional-dep closure shape, `CORE_UNCONDITIONAL_PLUGIN_NAMES`. See [storage-provider-plugin.md](storage-provider-plugin.md) § Cold-start context for the exact refs.
+
+---
+
+## The wrinkle: secrets is constructed before config-driven subsystems — but this is fine (Decision W RESOLVED → W-A)
+
+Storage is selected *after* config load and *after* secrets is on the hub — clean. Secrets looked different: `EnvSecretsProvider()` is built at `pipelex.py:212`, **early**, with the comment `# needed for gateway check`. That comment turned out to be **stale**, which is what makes W-A clean.
+
+**Verified 2026-07-06 (locks Decision W):**
+
+- **The gateway-check path is entirely secrets-free.** Neither `pipelex/system/pipelex_service/remote_config_fetcher.py` (`fetch_remote_config`, `make_dummy_remote_config`) nor `pipelex/system/pipelex_service/pipelex_service_config.py` (`is_pipelex_gateway_enabled`) references secrets in any form — no param, no hub read. Grep for `get_secret|secrets_provider|get_secrets_provider` in both is empty. The line-211 comment is wrong; nothing between line 212 and the telemetry factory needs secrets.
+- **The true first consumer of `secrets_provider` in `setup()` is `make_telemetry_manager` at line 288** (passed as a param). The hub secrets slot isn't set until line 306, so nothing in between could be reading it from the hub either.
+- **`build_registrar` is pure over config.** `PluginRegistrar.__init__(*, config)` takes only config; `build_registrar(config=get_config())` touches no hub, no models, no telemetry, constructs no SDK. `get_config()` is valid at `setup()` entry (config is loaded in `__init__` via `setup_config`, before `setup` runs). It's called exactly once today (line 393) and only re-read later by `_resolve_hub_slot` (line 505) and teardown (line 516).
+
+**⇒ Decision W = W-A, confirmed, zero ordering risk.** Move the single `build_registrar(config=get_config())` call (currently `pipelex.py:393-394`) up to **right after `boot_orchestrator` is set (after line 209), before line 212**. Then:
+
+1. Build `SecretsProviderRegistry(plugin_registrar.secrets_providers)`, set it on the hub.
+2. Resolve secrets: `if secrets_provider is None: secrets_provider = registry.get_required(method=get_config().pipelex.secrets_config.method)(get_config().pipelex.secrets_config)`.
+3. **Delete** the hardcoded `secrets_provider or EnvSecretsProvider()` line 212 and its stale comment.
+4. Everything downstream is untouched — telemetry (288), hub set (306), and the existing registry constructions / slot application at 404-418 keep referencing the already-built `self._plugin_registrar`. Only the *build call* moves up; its downstream consumers stay put.
+
+Bonus (recommended, not required): move the `boot_orchestrator` validation gate (`pipelex.py:401-403`, `UnknownBootOrchestratorError`) up alongside the build call — it's a pure check on the registrar and moving it fails fast on a typo'd orchestrator before any secrets/gateway work.
+
+**W-B is not needed** and is not the plan — recorded only so a future reader knows it was considered: keep Env hardcoded for a pre-gateway bootstrap, then re-resolve and overwrite the hub. Rejected because W-A is clean; W-B would leave a transient provider and a hub re-set for no benefit.
+
+---
+
+## The mechanism (reuse storage's)
+
+Same "keyed registry + config-selected singleton" as the storage plan:
+
+- New registrar menu method `add_secrets_provider(*, method: str, factory: SecretsProviderFactoryFn)` where `SecretsProviderFactoryFn = Callable[[SecretsProviderConfig], SecretsProviderAbstract]`, into `registrar.secrets_providers: dict[str, SecretsProviderFactoryFn]`, via `_add`.
+- New read-view `SecretsProviderRegistry` (`pipelex/plugins/secrets_provider_registry.py`) mirroring `OrchestratorRegistry` / the new `StorageProviderRegistry`.
+- New `UnknownSecretsMethodError` in `pipelex/plugins/exceptions.py`.
+- New config: `SecretsProviderConfig(ConfigModel)` with `method: str = Field(strict=False)` (default `"env"`); built-in token `"env"` needs no sub-config. Add `[pipelex.secrets_config]\nmethod = "env"` to `pipelex/pipelex.toml`, and wire `secrets_config` into the config model where storage_config lives (`pipelex/tools/secrets/secrets_config.py` new file + a field on the owning config — confirm placement with the config-owner; the storage one hangs under `pipelex.storage_config`, so `pipelex.secrets_config` is the natural sibling).
+- Builtin `SecretsPlugin` (`name="secrets"`, unconditional per DX-2) whose `register` calls `add_secrets_provider(method="env", factory=_make_env_secrets_provider)`; `_make_env_secrets_provider(config)` just returns `EnvSecretsProvider()`.
+- Boot selects via the registry at the point Decision W lands; the explicit `setup(secrets_provider=...)` param still wins ahead of registry selection.
+
+Because there's only one built-in method, introducing a config selector for a single impl looks like over-engineering — but the selector *is* the feature: it's what lets an external `vault` provider be chosen. `"env"` stays the default and the unconditional builtin, so out-of-the-box behavior is byte-identical.
+
+---
+
+## Decisions (recommended defaults — confirm before building)
+
+- **W — Selection point. ✅ RESOLVED = W-A (verified 2026-07-06).** Move the pure `build_registrar(config=get_config())` call up to after line 209 / before line 212; registry-select secrets there; delete the hardcoded `or EnvSecretsProvider()` and its stale `# needed for gateway check` comment. Confirmed zero ordering risk — the gateway path is secrets-free (see § The wrinkle for the evidence). No Phase-0 investigation needed; this is now a build instruction.
+- **S1 — Open method token + config selector.** Mirror storage D1: `SecretsProviderConfig.method` is an open `str`, validated at registry lookup, default `"env"`.
+- **S2 — One unconditional builtin `SecretsPlugin`.** `name="secrets"`, added to `CORE_UNCONDITIONAL_PLUGIN_NAMES` (DX-2).
+- **S3 — API bump batched with storage (DX-1).** `add_secrets_provider` ships in the same `PLUGIN_API_VERSION` 2→3 bump as `add_storage_provider`. If secrets lands in a much later PR than storage, that PR does its own bump and external plugins take a second `targets_api` update — acceptable but avoid if the two land close together.
+- **S4 — External config surface (follow-up).** Same as storage D3: `SecretsProviderConfig` gets a generic passthrough for out-of-tree providers as a scoped follow-up, not Phase 1. `"env"` needs no sub-config, so Phase 1 has nothing to add.
+
+---
+
+## Phased checklist
+
+### Phase 0 — Confirm design
+- [x] **Decision W RESOLVED = W-A** (verified 2026-07-06) — gateway path is secrets-free; `build_registrar` is pure over config; move the build call ahead of line 212. See § The wrinkle. No further tracing needed.
+- [ ] Confirm with the config owner where `secrets_config` hangs (sibling of `storage_config` under `pipelex.*`).
+- [ ] Confirm the `PLUGIN_API_VERSION` bump is shared with storage (DX-1 / S3) or standalone.
+
+### Phase 1 — The mechanism (core) — *skip the parts storage already landed*
+- [ ] If storage already bumped `PLUGIN_API_VERSION` to 3 **and** added `add_storage_provider`: only add `add_secrets_provider` + `secrets_providers` field to `PluginRegistrar`. If storage hasn't landed: do the 2→3 bump here too.
+- [ ] `add_secrets_provider(*, method, factory)` menu method via `_add`; `SecretsProviderFactoryFn` alias.
+- [ ] New `pipelex/plugins/secrets_provider_registry.py`: `SecretsProviderRegistry` (`get_optional` / `get_required` / `has` / `methods`).
+- [ ] New `UnknownSecretsMethodError` in `pipelex/plugins/exceptions.py`.
+- [ ] Hub: `set_secrets_provider_registry` / `get_secrets_provider_registry` + module accessor.
+
+### Phase 2 — Config
+- [ ] New `SecretsProviderConfig(ConfigModel)` (`pipelex/tools/secrets/secrets_config.py`), `method: str = Field(strict=False)`; no default in the class per config rules — default lives in TOML.
+- [ ] Add `[pipelex.secrets_config] method = "env"` to `pipelex/pipelex.toml`; wire `secrets_config` into the owning config model.
+- [ ] Add the same block to `.pipelex/pipelex.toml` override **only if** it's a genuine client-override case — secrets backend selection plausibly is (a deployer picks vault vs env), so include it with the real `method = "env"` value, never commented out (house rule).
+- [ ] `make tb` after config changes — config model ↔ TOML must stay in sync or boot fails.
+
+### Phase 3 — The builtin plugin + boot wiring (Decision W = W-A)
+- [ ] New `pipelex/plugins/secrets/secrets_plugin.py`: `SecretsPlugin` + `_make_env_secrets_provider(config)` returning `EnvSecretsProvider()`.
+- [ ] Register `SecretsPlugin()` in `BUILTIN_PLUGINS`; add `"secrets"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES`.
+- [ ] **Move the `build_registrar(config=get_config())` call** (`pipelex.py:393-394`) up to after line 209 / before line 212; keep `self._plugin_registrar = plugin_registrar`. Downstream registry constructions + slot application (404-418) stay put referencing the already-built registrar. Optionally move the `boot_orchestrator` gate (401-403) up with it for fail-fast.
+- [ ] Build + `set_secrets_provider_registry(...)` right after the moved build call.
+- [ ] **Delete** `pipelex.py:211-212` (the stale `# needed for gateway check` comment + `secrets_provider = secrets_provider or EnvSecretsProvider()`); replace with `if secrets_provider is None: secrets_provider = get_secrets_provider_registry().get_required(method=get_config().pipelex.secrets_config.method)(get_config().pipelex.secrets_config)` — explicit `setup(secrets_provider=...)` param still wins.
+- [ ] Drop the now-unused `EnvSecretsProvider` import from `pipelex.py` (it moves to the plugin module).
+
+### Phase 4 — Tests
+- [ ] Unit: `SecretsProviderRegistry` hit/miss (`UnknownSecretsMethodError`) + duplicate fail-loud.
+- [ ] Boot: default config yields `EnvSecretsProvider` on the hub; explicit `setup(secrets_provider=...)` still overrides.
+- [ ] Integration: external test plugin registering a fake `method="test_secret"` is selectable via config (mirror the inference external-plugin test).
+- [ ] Guard the storage↔secrets ordering: a boot with a non-env secrets method must still be on the hub before storage's GCP arm reads it (add/extend a boot-order assertion).
+- [ ] `pipelex plugins list` shows `secrets`.
+
+### Phase 5 — Docs & changelog
+- [ ] New `docs/under-the-hood/secrets-provider-plugins.md` (mirror the storage/orchestrator doc; emphasize the lazy optional-dep closure for SDK-backed providers like Vault/AWS). mkdocs nav.
+- [ ] Update any "the plugin seams are …" enumerations to include secrets.
+- [ ] CHANGELOG `[Unreleased]`: secrets provider is now a plugin seam; `PLUGIN_API_VERSION` bump note if not already recorded by the storage change.
+
+---
+
+## Verification / gates
+- [ ] `make agent-check`.
+- [ ] `make tb` — critical (new `secrets_config` ↔ TOML ↔ selection).
+- [ ] `make agent-test`.
+- [ ] Manual: `pipelex plugins list`; a run with default `method="env"`; confirm `get_secret(...)` still resolves env vars.
+
+## Cross-repo consequence (release-gated)
+- Same as storage: `pipelex-temporal` / `pipelex-mistralai-workflows` bump `targets_api` to 3 (only once, if batched with storage per DX-1).
+
+## Follow-ups (do not build now)
+- **S4 external config passthrough** for out-of-tree secrets providers.
+- **First real external provider** (Vault or AWS Secrets Manager) as the proof-of-seam, likely as a private/closed package if it targets the hosted platform.
+- **Retire the `*_specific_version` `NotImplementedError`s** in `EnvSecretsProvider` only if a consumer needs them — out of scope here.
diff --git a/wip/plugins/storage-provider-plugin.md b/wip/plugins/storage-provider-plugin.md
new file mode 100644
index 000000000..c0e6a0b4c
--- /dev/null
+++ b/wip/plugins/storage-provider-plugin.md
@@ -0,0 +1,130 @@
+# Storage provider → `pipelex.plugins` plugin
+
+Status: **COMPLETE.** All phases delivered — this vertical (Phases 1–2 of the master plan) defines the shared "keyed registry + config-selected singleton" mechanism (§ The new mechanism) that the secrets plan reuses. See [TODOS.md](../../TODOS.md) for the execution record (Phases 1–2 code + tests + docs, Phase 5 release gating).
+
+Goal: turn the storage provider into a formal plugin seam so third parties can ship `pipelex-storage-<backend>` packages discovered via the `pipelex.plugins` entry-point group and selected at deploy time by `storage_config.method`. The built-in providers (local / in_memory / s3 / gcp) become a single unconditional builtin plugin registering their factories — exactly as `OpenAIPlugin` registers several inference backends.
+
+Read [README.md](README.md) first for the track's shared decisions (DX-1 API bump, DX-2 unconditional builtin, DX-3 external config follow-up) and the boot-audit rationale.
+
+---
+
+## Cold-start context (the seam as it is today)
+
+Everything below is verified against the tree (worktree `_plugins`, branch is whatever's checked out — this is `pipelex/` code).
+
+**Abstract base** — `pipelex/tools/storage/storage_provider_abstract.py`
+- `StorageProviderAbstract` (line 16). Template methods `load` / `load_with_metadata` / `store`; subclass hooks `_load_with_metadata`, `_store`, `public_url` (abstract).
+- Also defines `PIPELEX_STORAGE_SCHEME = "pipelex-storage://"` and `StoredData(NamedTuple)` (`data: bytes`, `mime_type: str | None`).
+
+**Built-in implementations** — all under `pipelex/tools/storage/`
+- `LocalStorageProvider` (`local_storage_provider.py:12`) — ctor `(root_path: Path)`; hard-imports `aiofiles`; path-traversal guarded; `public_url` → `file://`.
+- `InMemoryStorageProvider` (`in_memory_storage_provider.py:13`) — a `RootModel[dict[str, bytes]]` **and** `StorageProviderAbstract`; no deps; `public_url` → `None`.
+- `S3StorageProvider` (`s3_storage_provider.py:15`) — ctor `(bucket_name, region, signed_urls_lifespan)`; optional dep **`aioboto3`** (extra `"s3"`) + botocore, lazy-imported inside methods, guarded by `find_spec` → `MissingDependencyError`.
+- `GcpStorageProvider` (`gcp_storage_provider.py:21`) — ctor `(bucket_name, project_id, credentials_file_path, signed_urls_lifespan)`; optional dep **`google-cloud-storage`** (extra `"gcp-storage"`), lazy-imported; sync GCS wrapped in `asyncio.to_thread`.
+
+**The factory being replaced** — `pipelex/tools/storage/storage_provider_factory.py`
+- `make_storage_provider_from_config(storage_provider_config)` (line 14) is a `match storage_provider_config.method:` over `StorageMethod`, each arm: `None`-check the sub-config → `sub.lazy_validate()` → construct the provider.
+- Imports all four concrete classes at module top (importing the factory imports all providers; optional SDKs still stay lazy inside the providers).
+- The **GCP arm** calls `get_secrets_provider().get_required_secret(secret_id="GCP_CREDENTIALS_FILE_PATH")` (line 62) — a storage→secrets→hub dependency. Safe at boot because storage is set on the hub (`pipelex.py:310`) *after* secrets (`pipelex.py:306`); **not** safe inside a plugin `register` (hub access is forbidden there). ⇒ construction must happen in a lazily-invoked factory closure, never in `register`.
+
+**Config** — `pipelex/tools/storage/storage_config.py`
+- `StorageMethod(StrEnum)` (line 13): `LOCAL`, `IN_MEMORY`, `S3`, `GCP`.
+- Per-method sub-models: `StorageLocalConfig`, `StorageInMemoryConfig`, `StorageS3Config`, `StorageGcpConfig`, each with `lazy_validate()` raising `StorageConfigError`.
+- `StorageProviderConfig` (line 171): `method: StorageMethod = Field(strict=False)` + one optional sub-field per method + a `model_validator(mode="after")` requiring the sub-config matching `method`.
+- `StorageConfig(StorageProviderConfig)` (line 238) adds `is_fetch_remote_content_enabled` / `is_upload_local_content_enabled`.
+- TOML `[pipelex.storage_config]` at `pipelex/pipelex.toml:5-27`; reached as `get_config().pipelex.storage_config`.
+
+**Boot injection** — `pipelex/pipelex.py`
+- `setup()` param `storage_provider: StorageProviderAbstract | None = None` (line 187).
+- Boot block (lines 307-310): `if storage_provider is None: … make_storage_provider_from_config(get_config().pipelex.storage_config)` then `set_storage_provider(...)`.
+- Also read at line 416 for the `CONTENT_GENERATOR` default thunk (`GeneratedContentFactory(storage_provider=...)`).
+- `make()` mirror: param line 561, forwarded line 637.
+
+**Hub accessors** — `pipelex/hub.py`: `set_storage_provider` (186), `get_storage_provider()` (306, raises if unset), module-level `get_storage_provider()` (476). Consumed by ~9 files / ~20 sites (input normalizer, pdf renderer, pypdfium2 worker, gateway extract worker, delivery executor, content generator, file/image/document prompt utils). **None of these change** — they keep calling `get_storage_provider()`.
+
+**The plugin pattern to mirror** (verified):
+- Contract `pipelex/plugins/contract.py` — `PipelexPlugin` protocol (`name`, `targets_api`, `register`); `PLUGIN_API_VERSION` (currently 2); `register` is **side-effect-free** (menu calls only; no hub, no I/O, no SDK import).
+- Registrar `pipelex/plugins/registrar.py` — menu methods like `add_orchestrator` / `add_bundle_validator` funnel through the generic `_add` helper (line 279) which fail-louds on duplicate keys naming both plugins. Keyed registries live as `dict` fields on the registrar.
+- Builtins `pipelex/plugins/builtins.py` — `BUILTIN_PLUGINS` is a list of instantiated plugin objects; `CORE_UNCONDITIONAL_PLUGIN_NAMES = frozenset({"direct", "openai"})`.
+- Discovery `pipelex/plugins/discovery.py` — `build_registrar(config=...)` iterates builtins then external `pipelex.plugins` entry points; strict-equality `targets_api` check; denylist via `config.plugins.disabled`.
+- Reference builtin registering several factories: `pipelex/plugins/openai/openai_plugin.py`. Reference optional-dep lazy closure: `pipelex/plugins/docling/docling_plugin.py` (`register` just calls `add_inference_backend`; the SDK import + `require_sdk` live in the module-level `_make_..._worker` closure).
+- Example registry read-view to copy: `pipelex/plugins/orchestrator_registry.py` (`OrchestratorRegistry`, `get_optional` / `has` / `modes`).
+
+---
+
+## The new mechanism (defines the shared pattern)
+
+A **keyed registry + config-selected singleton**. New, because no existing pattern selects a process-global singleton by its own config key (see [README.md](README.md) § The shared new pattern).
+
+- **Open method token.** Make the registry key an open `str` token, mirroring `OrchestrationMode` (an open string, not a closed enum). The built-in tokens stay the `StorageMethod` values (`"local"`, `"in_memory"`, `"s3"`, `"gcp"`); an external plugin registers e.g. `"azure"`. **Decision D1** below covers making `storage_config.method` accept arbitrary tokens.
+- **Registration side (in the plugin):** a new registrar menu method
+ `add_storage_provider(*, method: str, factory: StorageProviderFactoryFn)` where
+ `StorageProviderFactoryFn = Callable[[StorageProviderConfig], StorageProviderAbstract]`.
+ Accumulates into `registrar.storage_providers: dict[str, StorageProviderFactoryFn]`, fail-loud on duplicate via the existing `_add` helper.
+- **Read view:** `StorageProviderRegistry` (new, `pipelex/plugins/storage_provider_registry.py`) mirroring `OrchestratorRegistry` — `get_optional(method=...)` / `get_required(method=...)` (raising a new `UnknownStorageMethodError`) / `has` / `methods`.
+- **Selection side (in core boot):** replace the `make_storage_provider_from_config` call at `pipelex.py:307-310` with:
+ register-built `StorageProviderRegistry` on the hub → read `config.method` → `registry.get_required(method=config.method)(config)` → `set_storage_provider(...)`. The explicit `setup(storage_provider=...)` param still wins ahead of this (same three-tier precedence as content generator: explicit param > registry selection > — there is no separate core default; the builtin plugin *is* the default supplier).
+- **Where the per-method construction goes:** each `match` arm body from today's factory (None-check → `lazy_validate()` → construct) moves verbatim into a module-level factory closure in the builtin `StoragePlugin`, e.g. `_make_local_storage_provider(config)`, `_make_s3_storage_provider(config)`. The GCP closure keeps its `get_secrets_provider()` hub read — legal because the closure runs at the boot apply-point, not in `register`.
+
+Net effect: `make_storage_provider_from_config` and its `match` are deleted; the four arms become four registered closures; core boot does a registry lookup instead of a hardcoded factory call. `StorageProviderAbstract`, the four provider classes, the config models, and every consumer are unchanged.
+
+---
+
+## Decisions (recommended defaults — confirm before building)
+
+- **D1 — Open method token.** Make `StorageProviderConfig.method` accept an open `str` while keeping the `StorageMethod` enum for the built-ins (validate against the registry at boot, not against the enum at parse time — an unknown token surfaces as `UnknownStorageMethodError` from `get_required`, which is the right layer). *Recommended:* yes; without it external providers can be registered but never selected. *Alternative:* keep the closed enum and defer external selection — rejected, it guts the feature.
+- **D2 — One builtin `StoragePlugin`, unconditional.** A single builtin plugin (`name = "storage"`) registers all built-in methods, and joins `CORE_UNCONDITIONAL_PLUGIN_NAMES` (DX-2) so it can't be disabled into a broken boot. *Alternative:* one plugin per method — rejected as needless fragmentation; `OpenAIPlugin` already precedents "one plugin, many factories."
+- **D3 — External-provider config surface (follow-up, not Phase 1).** `StorageProviderConfig` has fixed typed sub-fields, so an `azure` provider has nowhere to read its config. Phase 1 lands the seam for built-in methods only. Follow-up: a generic passthrough (e.g. `extra: dict[str, dict[str, Any]]` on `StorageProviderConfig`, handed to the factory) so out-of-tree providers get structured config. Captured in § Follow-ups; do not build speculatively.
+- **D4 — Factory signature.** `Callable[[StorageProviderConfig], StorageProviderAbstract]` (whole config in, provider out) — lowest-churn move of the existing arm bodies, and keeps the GCP secrets read where it already is. *Alternative:* pass a pre-resolved typed sub-config + secrets provider explicitly (cleaner/more testable) — deferred; revisit if D3's passthrough makes "whole config in" awkward.
+- **D5 — API version bump batched with secrets (DX-1).** Introduce `add_storage_provider` **and** `add_secrets_provider` in the same `PLUGIN_API_VERSION` 2→3 bump even though storage lands first, so external plugins re-declare `targets_api` once. If secrets is genuinely far behind, fall back to bumping per-feature and accept the double external-plugin update — but default to batching.
+
+---
+
+## Phased checklist
+
+### Phase 0 — Confirm design
+- [ ] Walk D1–D5 with the user; lock the token-openness (D1) and the API-bump batching (D5) — those two shape everything downstream.
+
+### Phase 1 — The mechanism (core)
+- [ ] Add `PLUGIN_API_VERSION` bump to 3 in `pipelex/plugins/contract.py` (batched with secrets per DX-1; do the bump once).
+- [ ] Add `storage_providers: dict[str, StorageProviderFactoryFn]` field + `add_storage_provider(*, method, factory)` menu method to `PluginRegistrar` (`pipelex/plugins/registrar.py`), routed through `_add`. Add the `StorageProviderFactoryFn` type alias.
+- [ ] New `pipelex/plugins/storage_provider_registry.py`: `StorageProviderRegistry` (mirror `OrchestratorRegistry`) with `get_optional` / `get_required` / `has` / `methods`.
+- [ ] New `UnknownStorageMethodError` in `pipelex/plugins/exceptions.py` (message lists registered methods, mirroring `UnknownBootOrchestratorError`).
+- [ ] Hub: `set_storage_provider_registry` / `get_storage_provider_registry` on `PipelexHub` + module-level accessor (mirror the four existing registry accessors in `pipelex/hub.py`).
+
+### Phase 2 — The builtin plugin
+- [ ] New `pipelex/plugins/storage/storage_plugin.py`: `StoragePlugin` (`name="storage"`, `targets_api=PLUGIN_API_VERSION`, `register` calling `add_storage_provider` for each built-in method).
+- [ ] Module-level factory closures (`_make_local_storage_provider`, `_make_in_memory_storage_provider`, `_make_s3_storage_provider`, `_make_gcp_storage_provider`) holding the exact bodies moved from the deleted `match` arms (keep `lazy_validate()`; keep GCP's `get_secrets_provider()` read).
+- [ ] Register `StoragePlugin()` in `BUILTIN_PLUGINS` and add `"storage"` to `CORE_UNCONDITIONAL_PLUGIN_NAMES` (`pipelex/plugins/builtins.py`).
+
+### Phase 3 — Wire core boot to the registry
+- [ ] In `pipelex.py` setup: after `build_registrar(...)`, build + `set_storage_provider_registry(StorageProviderRegistry(plugin_registrar.storage_providers))` alongside the other four registry sets (near lines 404-407).
+- [ ] Replace the `make_storage_provider_from_config` block (lines 307-310) with the registry-selection path (explicit param > `registry.get_required(method=config.method)(config)`). Note ordering: storage still resolves after secrets is on the hub (keep the GCP secrets read working).
+- [ ] Delete `make_storage_provider_from_config` and `pipelex/tools/storage/storage_provider_factory.py` if nothing else imports it (grep first — `show_cmd`, tests). Move any residual import of the provider classes into the plugin module.
+- [ ] D1: relax `StorageProviderConfig.method` to open `str` (drop the strict enum bind at parse; validation happens at registry lookup).
+
+### Phase 4 — Tests
+- [ ] Unit: `StorageProviderRegistry` (`get_required` hit/miss → `UnknownStorageMethodError`; duplicate registration fail-loud through `_add`).
+- [ ] Unit/boot: booting with each built-in `method` yields the right provider on the hub (parametrized). Include the s3/gcp arms with SDK absent → the factory raising `MissingDependencyError` only when selected, not at registration.
+- [ ] Integration: an external test plugin registering a fake `method="test_mem"` (entry-point discovered) is selectable via config — mirror `tests/.../test_external_plugin.py` (the inference-backend external-plugin test) for the harness.
+- [ ] `pipelex plugins list` shows `storage` (origin builtin) with its contributions.
+
+### Phase 5 — Docs & changelog
+- [ ] New `docs/under-the-hood/storage-provider-plugins.md`, mirroring `docs/under-the-hood/orchestrator-plugins.md` structure (seam-in-one-view, the contract, the factory closure + lazy optional-dep guard, config selection, authoring an out-of-tree provider). Add to mkdocs nav.
+- [ ] Update `docs/under-the-hood/inference-backend-plugins.md` cross-links / any "the plugin seams are: inference, orchestrator" enumerations to include storage.
+- [ ] CHANGELOG `[Unreleased]`: "breaking" — `PLUGIN_API_VERSION` 2→3 (external plugins must re-declare `targets_api`); storage provider is now a plugin seam.
+
+---
+
+## Verification / gates
+- [ ] `make agent-check` (pyright, ruff, mypy, plxt, keyword-only guard).
+- [ ] `make tb` — **critical here:** it tests the boot sequence incl. config loading; the `storage_config` model ↔ TOML ↔ selection path must stay in sync.
+- [ ] `make agent-test` — full suite; catches the framework-positional and discovery edges the type checker can't.
+- [ ] Manual: `pipelex plugins list`; a real run with `method="local"`; flip to `method="in_memory"` and confirm selection.
+
+## Cross-repo consequence (release-gated, not this PR)
+- `pipelex-mistralai-workflows` must bump `targets_api` to 3 (DX-1); it registers no storage provider and imports none of the removed symbols, so the bump is its only change. `pipelex-temporal` needs the bump **and** a code migration: its payload-codec factory (`pipelex_temporal/codec/codec_factory.py`) imports the now-removed `make_storage_provider_from_config` and must switch to `get_storage_provider_registry().get_required(method=...)`. Do both when the pipelex version carrying this lands, not before.
+
+## Follow-ups (do not build now)
+- **D3 external config passthrough** — generic `extra` sub-config on `StorageProviderConfig` for out-of-tree providers. This must also fix the **`uri_format` gap**: `StorageProviderConfig.uri_format` (read by `GeneratedContentFactory._build_storage_key` on every content store) is defined only for the four built-in methods and raises `StorageConfigError` on the `case _:` arm for any external token — so an external provider selects and boots cleanly but crashes on the first *generated-content* store. D3 must give external methods a `uri_format` (e.g. promote it to a top-level field, or a passthrough default). Until then the docs page flags external providers as usable for their own storage API but not yet as the generated-content backing store.
+- **Cookbook example** — a `pipelex-storage-hello` external plugin mirroring the hello-inference-plugin cookbook track, once the seam is proven.