Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

refactor(boot): give the composition root the layer seam every other package has (#1073)

Louis Choquel8448c5ca2088
diff --git a/.drift/acks/config-docs.toml b/.drift/acks/config-docs.toml
index 5e986c4c4..5ecd26500 100644
--- a/.drift/acks/config-docs.toml
+++ b/.drift/acks/config-docs.toml
@@ -1,8 +1,8 @@
contract = "config-docs"
-digest = "sha256:084ae3c3afbaaa6568ef8eefab258059268c202f3fb5eb4c64d1ab96fcb67b1e"
+digest = "sha256:6559e8b12f68cc9b7edbb82cfcb78191add1fa8a555e58552a731d40da7ce4a1"
reviewed_by = "Louis Choquel"
-reviewed_at = "2026-07-27T17:00:55Z"
-rationale = "M2 mechanism/adapter split. Trigger diff on config_cogt.py is import-path-only: exactly four lines, pipelex.plugins.<vendor>.<vendor>_config -> pipelex.providers.<vendor>.<vendor>_config, from moving the seventeen built-in vendor adapters out of pipelex/plugins/ into pipelex/providers/. Verified by reading the staged diff in full — no config field, default, validator, TOML key, or section name changed, and the four config classes themselves were moved intact, not edited. The structural sync between configs.py and the pipelex.toml files is therefore untouched, which make tb confirms. Reviewed docs/configuration/ against the change: grepped the whole tree for pipelex.plugins, pipelex.providers, pipelex/plugins, pipelex/providers and each of the four config class names — zero hits. Those pages document TOML sections and keys ([pipelex.cogt], [pipelex.aws_config], etc.) and cite no Python import path anywhere, so an import-path move is invisible to them by construction. Nothing to update."
+reviewed_at = "2026-07-30T06:58:11Z"
+rationale = "Trigger is a docstring-only correction in config_loader.load_config: the config_dir Args entry still said 'project/global layering is bypassed and only this directory is read', which contradicts the same docstring's own summary six lines above and the body at line 313, where list_of_configs is seeded unconditionally with the package pipelex.toml. It is the exact wording this branch already corrected one file over in RuntimeHub.setup_config, and the sibling was missed. No config field, default, validator, TOML key, section name or loading behaviour changed — only prose. Reviewed docs/configuration/ against it: grepped every page for config_dir, 'only this directory', layering and --global. config_dir is not a user-facing concept and appears nowhere; the one layering hit (config-practical/telemetry-config.md:39) is about global-vs-project TOML precedence and is unaffected and correct. configuration/index.md:64-71 documents the default loading sequence and its step 1 ('Base configuration from the installed Pipelex package') is precisely the package-defaults-always-apply fact the corrected docstring now states, so the page already agreed with the truth the docstring was contradicting. Nothing to update."
[trigger_files]
"pipelex/cogt/config_cogt.py" = "blob:3c10f6a2b809e38b9a1112ce24ffb8ea4c32b04b"
@@ -10,7 +10,7 @@ rationale = "M2 mechanism/adapter split. Trigger diff on config_cogt.py is impor
"pipelex/pipelex.toml" = "blob:e646e431b5db2e467bc962b738be7bc9f3b3ffb8"
"pipelex/system/configuration/__init__.py" = "blob:8b137891791fe96927ad78e64b0aad7bded08bdc"
"pipelex/system/configuration/config_check.py" = "blob:efa78599c08e5e003d975cc54978ea5d267214a0"
-"pipelex/system/configuration/config_loader.py" = "blob:af3ba71ed4e589cec42cb197d7d11539cea50ecd"
+"pipelex/system/configuration/config_loader.py" = "blob:29fb321ef003c0fd725c87f560a791e8573b82f5"
"pipelex/system/configuration/config_model.py" = "blob:83b29dc6e93393c9b98c4353636caf0135c80339"
"pipelex/system/configuration/config_root.py" = "blob:c7d43851270912d17413cfc6a511247d3c7cc160"
"pipelex/system/configuration/configs.py" = "blob:99c936343aab7bcd4dee78631164ef10a32ebfe9"
diff --git a/.drift/acks/hub-layering-convention.toml b/.drift/acks/hub-layering-convention.toml
index 7eebd3ffd..b794f1345 100644
--- a/.drift/acks/hub-layering-convention.toml
+++ b/.drift/acks/hub-layering-convention.toml
@@ -1,10 +1,10 @@
contract = "hub-layering-convention"
-digest = "sha256:0d8be8d6f8ac57708dcda08ec0b835ed0dc507e321d9be071de4ffff96177f68"
-reviewed_by = "Claude (Opus 5)"
-reviewed_at = "2026-07-29T13:22:19Z"
-rationale = "Re-recorded after the final review round found a live counter-example to this page's central claim. Trigger is the guard's tuple and note again: pipelex.test_extras is added to RUNTIME_LAYER_PACKAGES, and the exclusion sentence is rewritten. Reviewed the target and edited three places. (1) The layer-rule enumeration under Enforcement gains pipelex.test_extras — sixth consecutive opening where an enumeration on this page was the rot, and it is the same enumeration as last time. (2) The exclusion sentence said only that kit is data files and language has interpreter edges, which reads exhaustive and was not: enumerating every top-level package and running measurement command 3 over each shows test_extras measures zero, ships, and is imported by pipelex.py at boot, while runtime_bridge and cli measure dirty and were simply never mentioned. The sentence now accounts for every top-level package rather than naming two. Considered and rejected declaring config, base_exceptions and urls, which are also clean and undeclared: the transitive rule already covers them through their 30, 32 and 5 declared runtime-layer importers, so declaring them buys nothing. test_extras had no such cover. (3) Known inversions gains the finding this round produced, which is a genuinely new shape for this page. Five built-in vendor adapters imported MissingDependencyError from pipelex.exceptions, the public all-errors aggregate, instead of from its definition site pipelex.system.exceptions. That aggregate re-exports every interpreter package's errors, so each of those five declared runtime-layer modules loaded 15 interpreter modules — inside pipelex.providers, a package this very tuple already declared clean. Both gates were structurally blind: the guard only chases interpreter_hub and the aggregate never reaches it, and the closure test's providers.builtins entry point does not cover them because the vendor plugins import the list modules inside a function, hiding them from the static graph and from the aggregator's closure at once. Fixed at the import site per the repo's direct-full-path-imports rule; providers.anthropic.anthropic_list added to RUNTIME_LAYER_ENTRY_POINTS, with a negative control run — reverting one import fails that entry point. The lesson recorded on the page is that a module re-exporting across layers is a layer boundary with the sign filed off. Ran the two prescribed mechanical checks: all thirteen declared packages named under The rule, 32 + 31 hub accessors covered. Both passed and both were blind to all three findings, eleventh consecutive time."
+digest = "sha256:2b509c7efe7ebaa99dcf1fce041dcf141b94e516dd91cda75b13deba982538f9"
+reviewed_by = "Louis Choquel"
+reviewed_at = "2026-07-30T05:25:03Z"
+rationale = "Trigger is a one-paragraph docstring accuracy fix on RuntimeHub.setup_config's config_dir parameter: it said 'project/global layering is bypassed and only this directory is read', which contradicted config_loader (the package defaults layer always applies) and the two make() docstrings that already said 'package defaults + this directory'. Reviewed docs/contribute/hub-layering.md: it does not describe setup_config, config_dir, or config layering anywhere — grep for config_dir/setup_config/'only this directory' returns nothing — so the page carries no copy of the claim that was wrong and nothing is stale. Re-ran the prescribed mechanical check anyway (every name in the guard's RUNTIME_LAYER_PACKAGES appears under 'The rule'): all fourteen present, including pipelex.runtime_boot from the previous ack, so the enumeration that has been the rot seven rounds running is clean this time. No doc change needed."
[trigger_files]
-"pipelex/cli/dev_cli/commands/hub_layering_guard.py" = "blob:755c74c5919486992961c36e7c2925c9425aebc2"
+"pipelex/cli/dev_cli/commands/hub_layering_guard.py" = "blob:a602008dabb5b1ce81496513c37253ef6f5c0765"
"pipelex/interpreter_hub.py" = "blob:b7ac885d348d32f1aa4b1e660da5b03533fb305b"
-"pipelex/runtime_hub.py" = "blob:8fffa04a52fb04ea89202fb4a24cf3d67793d83e"
+"pipelex/runtime_hub.py" = "blob:81fb34f097b440293b8b68f9eefb7289778c911b"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1628d6168..91f62a0c9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,8 @@
### Added
+- **The composition root now has the same layer seam as every other package: `pipelex/runtime_boot.py`.** `RuntimeBoot` stands up config, logging, secrets, telemetry, the class and func registries, the template sets, the model deck, storage, the content generator, the inference manager, the reporting delegate and the observers — and loads **zero interpreter modules** doing it (619 → 356 modules, 172 → **0** interpreter, measured). `pipelex.py` was the one module the layering work never reached: interpreter-layer by construction, therefore outside the declaration entirely, and an undeclared module is unpoliced rather than neutral — so nothing enforced which half of its boot sequence belonged to which layer, and there was no way to boot the runtime layer at all. This is also the first caller `RUNTIME_BUILTIN_PLUGINS` has ever had: a runtime-only boot discovers exactly the runtime half, leaving the two interpreter-touching built-ins absent. Declared in the hub-layering guard's `RUNTIME_LAYER_PACKAGES` as a module entry beside `pipelex.runtime_hub`, pinned by the import-closure test's `RUNTIME_LAYER_ENTRY_POINTS`, and pinned again by `tests/unit/pipelex/test_runtime_boot_closure.py`, which *boots* it in a subprocess rather than importing it — an import-time check cannot notice a boot step resolving out of an interpreter-contributed registry that is empty on that boot, and an in-process check would pass vacuously because both hubs are sticky singletons teardown never clears. Documented in [Hub layering convention](contribute/hub-layering.md#where-the-boot-splits).
+
- **`Concept` is pure data again, and the property is pinned.** `pipelex/core/concepts/concept.py` — a subclass of the MTHDS-protocol wire model `ConceptAbstract` — no longer reads the process-global class registry. `tests/unit/pipelex/core/concepts/test_concept_registry_boundary.py` asserts a golden set over an AST walk of `pipelex/core/concepts/**`: exactly two modules may touch the registry, and both are the **materialization write side** (`concept_factory` registering generated structure classes, `structure_generation/generator.py` looking up base classes to generate against). A golden set rather than a rule, for the reason the `cogt` boundary test gives — there is no predicate separating a sanctioned write-side read from a new read-side one, and no other gate can see the edge at all, since both the accessor and its users are runtime-layer. Documented in [Hub layering convention](contribute/hub-layering.md).
- **Renaming an error class is now a reviewable diff instead of a silent wire break.** `ErrorReport.error_type` is `type(self).__name__` with no indirection, and consumers outside this repo `switch` on that string — so a rename leaves their build green and their branch silently unmatched. Nothing in the repo flagged that, which is how `PipelexInterpreterError` → `MthdsParserError` reached four consumer sites unannounced. `tests/data/errors/error_identity.txt` now pins the `(error_type, title, type_uri)` triple of every discoverable `PipelexError` subclass, regenerated with `make generate-error-identity` (alias `make gei`, or `pipelex-dev generate-error-identity`) and gated by `tests/unit/pipelex/errors/test_error_identity_snapshot.py`. The rendering is pure — no Pipelex bootstrap — and the population comes from the same `iter_pipelex_error_subclasses()` helper the docs generator uses, so it cannot silently miss a deferred-import error module. Documented in [Error Model](under-the-hood/error-model.md).
@@ -16,6 +18,8 @@
### Changed
+- **Breaking: `Pipelex` is now the interpreter-layer boot composed on top of `RuntimeBoot`.** Same address, same public class name, and a `make()` that keeps every existing parameter and its docstring (it gains `config_dir`, and `pipeline_manager` widens to the abstract type — both non-breaking) — it subclasses the runtime boot, imports it downward, and appends the `InterpreterHub`, the composed plugin manifests, the `PipeFuncExecutorRegistry` and its executor, the `LibraryManager`, the `PipelineManager`, the `PipeRegistryModels` registration, the `PipeRouter` and the `PipeRun`. Inheritance preserves every attribute address, so no consumer changes. Two breaking details: `Pipelex.__init__` takes keyword-only `config_dir` in place of the inert positional `config_dir_path` (nothing in the workspace constructed `Pipelex` positionally), and the dead `is_pipelex_service_enabled` attribute is gone — it was assigned `False` and never updated, because `setup()` only ever wrote a local of the same name, so it always answered `False` and nothing read it. The class-level accessors (`get_optional_instance`, `is_fully_booted`, `get_instance`, `teardown_if_needed`) now resolve **by subclass**, so `RuntimeBoot.is_fully_booted()` answers for a `Pipelex` too.
+
- **Breaking: class resolution left the `Concept` wire model for the concept provider, fixing two live bugs on the way.** `ConceptProviderAbstract` gains `get_structure_class(concept)`, implemented by `ConceptLibrary` against the active registry with a `StuffContent` subclass bound. Four members of `Concept` change or go:
| was | is now |
@@ -59,6 +63,20 @@
### Fixed
+- **Fixed: an injected telemetry manager whose `teardown()` raises no longer aborts the failed-boot release suite.** `self.telemetry_manager` is whatever the telemetry factory produced, which includes a caller-supplied implementation — `telemetry_manager` is a public `make()` injection point typed only as `TelemetryManagerAbstract`. The built-in manager is written never to raise, but that guarantee does not extend to an injected one, and an unisolated call would skip every release after it (leaving logging configured and the class registry half-populated) and replace the boot error on the way out. The call is now isolated, logged, and followed by the rest of the suite.
+- **Fixed: a teardown that fails partway no longer wedges the process permanently.** The releases at the end of `_teardown_runtime` — the hub config, the class-registry scoping, the `MetaSingleton` de-registration — are what a later boot in the same process depends on, and they sat in a bare sequence below calls made through injectable abstract types (`telemetry_manager`, `reporting_delegate`, `class_registry`, and one level up `pipeline_manager`, all public `make()` injection points). A raiser therefore skipped them and left the instance registered, with **no recovery**: the next `make()` dies on "already initialized" and `teardown_if_needed` resolves that same instance and re-enters the same raiser. Both the runtime release block and `Pipelex.teardown`'s three-phase sequence now use `try`/`finally` — no `except`, so a half-failed teardown still surfaces rather than looking successful. This is the same flaw the failed-boot path already isolated one method away; the normal path's consequence was the worse of the two, since it did not even de-register the singleton.
+
+ The `finally` carries the **whole** un-poisoning set, not just the de-registration, and that distinction is the fix rather than a detail of it: releasing the singleton is what makes the next boot *succeed*, so anything poisoning left behind alongside it converts a loud failure into a silent one. `KajsonManager.teardown()`, `TemplateLoader.reset()` and `TemplateRegistry.clear()` therefore moved down beside it, exactly as `_release_after_failed_boot` already had them. `KajsonManager` is the one that bites — it is a singleton, so a surviving manager makes the next boot's `KajsonManager(class_registry=…)` hand back the old manager and silently discard the fresh registry, leaving `get_class_registry()` serving the previous boot's classes while that boot's own registrations land somewhere nothing resolves to, `is_ready` and all. The two release paths now guarantee the same poisoning set; the *dangling* set (`sdk_client_manager`, `reporting_delegate`, `func_registry`) still differs, which is the collapse tracked in `wip/boot-split/failed-boot-does-not-release-every-resource.md`. Pinned by `test_runtime_boot_teardown_resilience.py`, verified to fail when the move is reverted.
+
+ The plugin-callback phase moved inside the same `try` on both teardown paths, for the reason `_release_after_failed_boot` already gave at its own call to it: `_teardown_plugin_callbacks` swallows each callback's ordinary `Exception`, but `except Exception` does not cover `BaseException`, so a callback that calls `sys.exit()` — or a `KeyboardInterrupt` landing mid-teardown — would otherwise skip the release block entirely.
+- **Fixed: a second `make()` of the already-booted class is now refused instead of silently re-running `setup()`.** The exclusivity guard lived only in `__init__`, and `MetaSingleton.__call__` hands back the registered instance without re-running `__init__` — so a second `RuntimeBoot.make()` sailed past it and re-ran the entire setup on the live boot, returning the same object, while its docstring promised a `PipelexSetupError`. The two cross-class tests could not catch it, because a subclass is a different registry key and `__init__` does run there. The check now also runs *before* the construction in both `make()`s, single-sourced as `RuntimeBoot.raise_if_a_boot_already_holds_the_process_globals()`; the `__init__` site stays, since it is the only one that sees a direct construction.
+- **Fixed: the singleton lookup no longer iterates a live process-global dict.** `MetaSingleton.get_subclass_instance` walked `instances` directly while its sibling `clear_subclass_instances` already copied. The dict grows at run time (`GraphTracerManager` registers itself during a pipeline run) and this lookup now sits on the boot hot path `ensure_pipelex_booted` reaches from concurrent activities — where a key added mid-iteration is a `RuntimeError`. It iterates a copy.
+- **Fixed: a failed boot no longer leaves the telemetry manager singleton behind for the next boot to adopt.** `TelemetryManager` is an `ABCSingletonMeta` singleton, so discarding the boot instance did not release it — and because `MetaSingleton` short-circuits on an already-registered class, the next boot in the process got the *same object* with `__init__` never re-running, leaving `TelemetryManagerAbstract.get_instance()` resolving to a manager whose tracer provider had been shut down. Reachable in production through `ensure_pipelex_booted`'s per-call lazy boot, since the commonest boot failure (`models_manager.setup()` on a missing deck or credentials) fires after the telemetry factory. The failed-boot release path now calls the guarded `telemetry_manager.teardown()`, which flushes and clears the singleton.
+- **Fixed: plugin teardown is now best-effort per callback, and can no longer replace the exception that killed a boot.** Two defects on one path. A raising callback used to propagate in place of the setup error, so a failed boot reported a teardown failure and never its cause; the release path's state cleanup now runs in a `finally` so it cannot be skipped either. And the callbacks are now attempted **individually**: previously one raising callback ended the LIFO loop, so with two plugins registered the second one's runtime was left live — catching around the loop rather than inside it looks equivalent but the loop has already exited. Errors are logged rather than propagated, on both the failed-boot and the normal teardown path, because no caller of a teardown can act on "plugin B failed to release" while every caller is harmed by B never being asked. **Breaking**: `teardown()` no longer propagates a plugin callback's exception.
+- **Fixed: a failed boot no longer leaks a plugin runtime a boot-orchestrator already started.** `Pipelex.make()`'s failure handler released core's own process-global state but never ran the plugin-registered teardown callbacks, so a boot that died after the `TASK_MANAGER` slot thunk left whatever runtime that thunk stood up in place — and only the plugin knows what that is; our Temporal plugin's thunk registers its own process-global singletons and installs a sandbox predicate. The window is wider since the boot split (the thunk runs in the runtime half, so every interpreter-tail step now follows it, where three of them used to precede it), and the most reachable trigger is a plain config error: `pipe_func_config.execution_mode` naming an unregistered mode. The failure path now runs the plugin teardown callbacks first, mirroring the normal teardown's ordering, pinned by a test that asserts the thunk ran *and* the callback ran (so it cannot pass vacuously) and verified to fail without the fix.
+- **Fixed: `config_dir` at boot is no longer silently ignored.** `Pipelex(config_dir_path=…)` stored the path and never read it again — `setup_config` was called without a `config_dir`, even though the hub method accepts one — so an embedder passing an explicit config dir got the ordinary project/global layering and no error. It is now wired through to `setup_config(config_dir=…)`, with a test that boots against a temp dir and asserts the scoped load is what landed.
+- **Fixed: a second boot on top of an existing one now fails loud instead of serving a half-populated registry.** `set_runtime_hub`, `KajsonManager` and `log.configure` are all once-per-process, so a `Pipelex.make()` on top of a live `RuntimeBoot` (or vice versa) would quietly adopt the first boot's state. Each `make()` now asks the base class whether any boot holds the process globals and raises `PipelexSetupError` if one does.
+
- **Breaking: generated codegen artifacts were not lint-clean, so running your own formatter over them broke their stamp and got you accused of hand-editing.** A stamp's `content_hash` is a raw SHA-256 over the body bytes, and the emitters were not writing what a formatter wants — so the first `ruff check --fix` / `ruff format` (or `prettier --write`) over a generated tree rewrote the bytes and `pipelex codegen check` reported the file as `[hand-edited]`, which is the one thing the user did not do. It was silent until someone ran `codegen check`, and our own `pipelex-starter-python` template shipped a `piper/generated` lint exclusion to work around it — the tell that the emitter, not the consumer, was wrong.
**Emitted bytes change for all three targets, so a committed projection reports drift until it is regenerated** — `pipelex codegen types --target <target> -o <output_dir> <method_dir>`, or `pipelex build structures <dir>` for the python-structures alias. Both Python targets now emit isort-grouped imports (merged per module, stdlib separated from third-party) and double-quoted `Literal` members; `python-structures` additionally moves to builtin generics (`list[str]`, `dict[str, int]`) and `X | None`, dropping its `typing.List` / `Dict` / `Optional` imports. `ts-zod` drops the Python two-blank-line idiom Prettier always collapses, and wraps the binder import only when it exceeds Prettier's print width. Ruff's `TC003` is the one rule that cannot be answered in the bytes — moving `from datetime import date` into an `if TYPE_CHECKING:` block would break the generated models, since pydantic resolves annotations at runtime — so consumers set `runtime-evaluated-base-classes` instead; **the lint exclusion on generated paths is no longer needed and should be dropped.** Documented in [Codegen Projections](under-the-hood/codegen-projections.md). This repo now sets that option too, which retired two hand-written `# noqa: TC001` / `# noqa: TC003` suppressions that were working around the same rule one file at a time.
diff --git a/TODOS.md b/TODOS.md
new file mode 100644
index 000000000..b58e78f33
--- /dev/null
+++ b/TODOS.md
@@ -0,0 +1,257 @@
+# Boot split — the composition root gets the same layer seam as everything else
+
+**Status: complete.** Branch `refactor/Boot`, cut off `dev` at `96b992786`. This document describes the change for review: what it does, why, what was decided and why, and where a reviewer should push hardest.
+
+## ⏸ RESUME HERE — session paused 2026-07-30
+
+**PR:** https://github.com/Pipelex/pipelex/pull/1073 → `dev`.
+
+**State: bots are clean on `26b107d78`, and a `/review` pass has run on top of it. Its fixes are UNCOMMITTED in the working tree.**
+
+- CI on `26b107d78`: **21 pass / 0 fail** (3 conditional jobs skipped). Greptile Review check passed. **Codex reviewed `26b107d781` and found "no major issues".** **0 unresolved review threads.** So the open action from the previous pause — wait for CI + the bots on `62249bd6a`, then triage — is closed, and the triage found nothing live.
+- Local gates re-run green on the working tree, not just on the pushed one: `make agent-check` (pyright 0, mypy clean, keyword-only, hub-layering), full `make agent-test`, `make drift-check`.
+- The headline measurement was reproduced independently: `import pipelex.runtime_boot` → **356** modules / **0** interpreter; `import pipelex.pipelex` → 619.
+
+**What the `/review` pass changed (uncommitted):** one behavioural fix and a set of comment/doc accuracy corrections. See "Round 8" below.
+
+### The prescribed procedure is now complete
+
+Both procedural gaps from the previous pause are closed. Three sub-agents were fanned out, none with inherited context:
+
+1. **Opus bot-feedback triage** — polled CI to terminal, collected every thread (resolved and unresolved) plus both bots' review bodies, deduplicated, and verified each claim against source rather than trusting the bot. Verdict: CI all green; of 13 distinct items, 12 already addressed; **one live clear win.**
+2. **Adversarial review of the boot split** — nine findings, confirming the triage agent's one and adding accuracy defects.
+3. **Test-quality review** — traced every new behavioural assertion back to the `dev` code it pins. **Found no vacuously-passing test**; all eight fail if their fix is reverted, all five repointed `mocker.patch` targets resolve to real bindings, no order dependence under `-n 4`. Its findings are coverage gaps and one wrong comment.
+
+**What `62249bd6a` fixed from those reports:**
+
+- **The teardown wedge** (all three agents, independently). A raising step in `_teardown_runtime` skipped the release block at its end, leaving the boot registered with **no recovery** — `teardown_if_needed` resolves the same instance and re-enters the same raiser. Same for `Pipelex.teardown`'s middle phase. Both now `try`/`finally`, no `except`. Pre-existing on `dev`; this branch had fixed the identical flaw one method away and left its sibling bare.
+- **The same-class boot guard.** The guard sat only in `__init__`, which `MetaSingleton` skips for an already-registered class, so a second `RuntimeBoot.make()` silently re-ran the whole `setup()` on the live boot while its docstring promised a `PipelexSetupError`. Neither cross-class test could see it. Now checked before the construction in both `make()`s, single-sourced.
+- **`get_subclass_instance` iterating a live process-global dict** (its sibling already copied).
+- **Five claims that were wrong, not merely thin** — including a comment asserting a mutation result that is the *opposite* of what the mutation does (verified by re-running it).
+
+**Two new deferrals** in `wip/boot-split/`, per the no-over-engineering bar: `pipelex-setup-narrows-the-runtime-boot-contract.md` (an LSP narrowing whose every remedy trades one wart for another) and `boot-split-test-coverage-gaps.md` (four gaps whose fixes are decisions, not patches — notably that the closure subprocess boots outside the harness and reads the developer's real `~/.pipelex`).
+
+### ⚠ Read this before spawning any more sub-agents
+
+**A review sub-agent reverted the working tree twice mid-session** — `git stash` / restore to `d845f7dad` — to probe pre-fix behaviour. It destroyed a new test module, a CHANGELOG entry, and two source fixes, and it did so while a `make agent-test` run was executing against the tree. All of it was reconstructed and re-verified, and the agent was stopped.
+
+**Any future review fan-out must be told explicitly: read-only. No `git stash`, `git checkout --`, `git restore`, `git clean`, and no edits to tracked files. To compare against earlier behaviour, use `git show <rev>:<path>`.** The instruction to hand it "only a pointer to the changes under review" is not enough on its own — a pointer plus write access invites exactly this.
+
+**The immediate open action:** commit the working tree (see "Round 8"), push, and let CI + the bots run once more. The read-only instruction above was given to all four `/review` sub-agents and none of them touched the tree.
+
+### Round 8 — the `/review` pass, and the fix that traded loud for silent
+
+Four read-only sub-agents (testing, maintainability, security, performance) plus an adversarial pass. Performance found nothing. The two that matter both landed on the **same commit that fixed the round-7 wedge**, from opposite directions.
+
+**The one behavioural fix.** `62249bd6a` moved `_teardown_runtime`'s de-registration into a `finally` and stopped there. That is what makes the *next* boot succeed — so the releases still sitting in the `try` above a raising step stopped being a dangling-resource problem and became a **silent-corruption** one. `KajsonManager` is the sharp one: it is a singleton, so a surviving manager makes the next boot's `KajsonManager(class_registry=…)` hand back the old manager and discard the fresh registry. That boot then serves the previous boot's classes, registers its own into a registry nothing resolves to, and reports `is_ready`. On `dev` the identical raiser failed loudly on "already initialized". Verified end to end: `runtime_hub.reset_config()` calls `log.reset()` (so `log.configure` does not refuse), and `TemplateLoader.register_set` is idempotent (so nothing else fails loudly either) — the boot really does complete. `KajsonManager.teardown()`, `TemplateLoader.reset()` and `TemplateRegistry.clear()` moved into the `finally`, matching `_release_after_failed_boot` exactly; all three are idempotent. Pinned in `test_runtime_boot_teardown_resilience.py`, negative control run individually.
+
+**The lesson is the round-7 lesson again, one level up.** Round 7 recorded "fixing a bug class on one path and leaving its sibling". This was fixing *part of a release list* on one path and leaving the rest — and the partial fix was worse than none, because it removed the loud failure that had been covering the silent one. When a fix's whole justification is "these releases must not be skipped", the unit is the release **set**, not the statement that happened to be named in the report.
+
+**Also applied:** the plugin-callback phase moved inside the same `try` on both teardown paths (`except Exception` per callback does not cover `BaseException` — the argument `_release_after_failed_boot` already made and its two siblings had not). Plus seven comment/doc accuracy fixes, the sharpest being `runtime_boot.py`'s claim that `KajsonManager` "overwrites unconditionally" — it does the exact opposite, and line 721 of the same file said so. Two were the *same* sentence in two places (`config_loader.load_config`'s `config_dir` docstring still carried the wording this branch corrected in `RuntimeHub.setup_config`) and two were stale doc references the earlier sweeps missed because they say `pipelex.py`, not `pipelex/pipelex.py`.
+
+**Not applied, deliberately:** the `619 → 356` module counts stay in the CHANGELOG (a point-in-time measurement, not an enumeration count); the `make()` lifecycle duplication between the two halves stays (factoring it is the "no new abstraction over the two boots" non-goal). Three new coverage gaps are recorded in `wip/boot-split/boot-split-test-coverage-gaps.md` rather than fixed.
+
+**Cross-repo, re-verified with `git -C <repo> grep` over every repo enumerated from disk:** still exactly the two known breaks, `pipelex-temporal/tests/conftest.py:86` and `pipelex-transport/tests/conftest.py:89`. Nothing new.
+
+**How the review loop has been run** (worth continuing — every round but one produced at least one genuine defect):
+
+1. Read every unresolved thread; **verify the claim against the source before acting.** Several bot findings were right about the mechanism and wrong about severity; two were right about a mechanism I had reasoned about incorrectly; and one sub-agent finding, re-run by hand, was right where my own committed comment was wrong.
+2. Fix only clear wins. Defer genuine design tradeoffs as `.md` in `wip/boot-split/` with an analysis and a suggested shape.
+3. For every behavioural fix, **write the negative control**: revert the fix and confirm the new test fails. Do it per fix, not for the pair — reverting both at once proves only that the pair is needed. This caught a vacuous test and confirmed six real ones.
+4. Reply on the thread with the reasoning (including where my own earlier reasoning was wrong), resolve it, re-run gates, commit, push, ping both bots.
+
+**Three recurring mistakes of mine, if more findings arrive:**
+
+- **Cost misjudgement drove two wrong deferrals.** The telemetry singleton I deferred as "needs its own test matrix" was one guarded call; the `config_dir` propagation I priced at four lines needed a public abstract interface widened. *Write the patch to price it*, then decide.
+- **Reasoning from a concrete class's guarantees about a call through an injectable abstract type.** Bit three times now, on three different methods. Anything typed `…Abstract` or `…Protocol` that a caller can inject must be treated as unbounded — and that includes `teardown`.
+- **Fixing a bug class on one path and leaving its sibling.** The teardown wedge existed *next to* the failed-boot path whose 16-line comment explains precisely why it needs `try`/`finally`. When a fix is justified by a general argument, grep for every site the argument covers.
+
+**Five deferrals are recorded** in `wip/boot-split/`: `runtime-boot-external-interpreter-orchestrator.md`, `config-dir-does-not-scope-inference-paths.md`, `failed-boot-does-not-release-every-resource.md`, `pipelex-setup-narrows-the-runtime-boot-contract.md`, `boot-split-test-coverage-gaps.md`. Each has the analysis and a suggested remedy.
+
+**One required cross-repo follow-up** (separate repos, out of scope for this PR): `pipelex-temporal/tests/conftest.py:86` and `pipelex-transport/tests/conftest.py:89` patch `"pipelex.pipelex.load_pipelex_service_config_if_exists"` in autouse session fixtures; that symbol is now in `pipelex.runtime_boot`, so both suites break at session start once they pick this up. ⚠ Use `git -C <repo> grep` for any cross-repo sweep here — the environment's `grep` is a **shell function** that does not traverse sibling repos and silently returns zero.
+
+---
+
+## What this branch does
+
+`pipelex/pipelex.py` is split into two composition roots:
+
+- **`pipelex/runtime_boot.py`** (new) — `RuntimeBoot`: config, logging, the secrets provider, telemetry, the Kajson class registry + `CoreRegistryModels`, the template sets, `sdk_client_manager`, the model deck, the plugin-derived runtime registries, storage, the content generator, the inference manager, the reporting delegate, the observers, and the runtime slot claims. **Loads zero interpreter modules.**
+- **`pipelex/pipelex.py`** — `Pipelex`, unchanged address and unchanged public surface, now `class Pipelex(RuntimeBoot)`. It adds the `InterpreterHub`, the composed plugin manifests, the `PipeFuncExecutorRegistry` and its executor, the `LibraryManager` and default library dirs, the `PipelineManager`, the `PipeRegistryModels` registration, the `PipeRouter` and the `PipeRun`.
+
+## Why
+
+Pipelex has two layers and one hub each, enforced two ways: `check-hub-layering` owns the rule (no runtime-layer module imports `interpreter_hub`) and `test_runtime_layer_import_closure.py` owns the property it buys (importing the Pipelex runtime loads zero interpreter modules). Every package in the tree had been placed on one side of that line and pinned there.
+
+**The composition root never got the treatment.** It boots both layers in one interleaved sequence, so it is interpreter-layer by construction and sat entirely *outside* the declaration — and an undeclared module is not neutral, it is unpoliced. Two consequences:
+
+1. **The property was real for importing and vacuous for booting.** The entry point every embedder actually calls — `Pipelex.make()` — loaded the interpreter unconditionally. There was no way to boot the runtime layer at all.
+2. **The interleaving was undocumented and unenforced, so it would rot.** The interpreter constructions were sprinkled through `setup()` at four separate points with no marker saying so. Nothing told the next person which half a new boot step belonged in, and nothing caught a wrong answer. Now the wrong answer is an import in a declared runtime-layer module, and the guard names it.
+
+The shape has a worked in-tree precedent: the built-in plugin manifests were split exactly this way (runtime half in a declared runtime-layer home, interpreter half importing it downward and welding the two), which `docs/contribute/hub-layering.md` §"Placement, not coupling" prescribes and the layer-placement track applied three more times.
+
+## Measured payoff
+
+| | modules | interpreter modules | SLOC (non-blank) |
+|---|---|---|---|
+| `import pipelex.pipelex` | 619 | 172 | 60,026 |
+| `import pipelex.runtime_boot` | **356** | **0** | 27,630 |
+
+**The zero is the gate**, via a new closure entry point. The module and SLOC deltas are context, not gates. Reproduce — note the trailing dot in `'pipelex.'`, which is the convention the closure test uses; dropping it also counts the bare `pipelex` package and gives 357/620:
+
+```bash
+.venv/bin/python -c "import importlib, sys; importlib.import_module('pipelex.runtime_boot'); print(len([m for m in sys.modules if m.startswith('pipelex.')]))"
+```
+
+The projection made before any code was written was 356 / 0. It matched exactly, because the runtime half of the boot sequence already reached no interpreter module — so the split needed **no** deferred imports, no function-local `# noqa: PLC0415`, and no new indirection. Every import stays at module top level.
+
+## The load-bearing decisions
+
+**Inheritance, not composition.** `Pipelex` subclasses `RuntimeBoot` rather than holding one.
+
+- Every attribute address is preserved — `self.models_manager`, `self.class_registry`, `self.telemetry_manager`, `self.kajson_manager`, `self._plugin_registrar` all read the same from both halves — so no consumer inside or outside this repo changes. Composition would rename all of them for no architectural gain.
+- It reads as what it is: the interpreter boot *is* the runtime boot plus the interpreter constructions.
+
+**The singleton resolves by subclass.** `RuntimeBoot` carries `MetaSingleton` too, and the class-level accessors go through `MetaSingleton.get_subclass_instance` (in-tree precedent: `TelemetryManagerAbstract`, `GraphTracerManager`). Each `make()` asks the **base** class whether a boot exists — not `cls`. That is what makes the two mutually exclusive in *both* directions: asking `cls` would let `Pipelex.make()` boot on top of a live bare `RuntimeBoot`, because `get_subclass_instance(Pipelex)` cannot see one, and it would then quietly serve that boot's half-populated class registry. `set_runtime_hub`, `KajsonManager` and `log.configure` are all once-per-process.
+
+**Teardown ordering is explicit, not a template hook.** The plugin-contributed callbacks must run LIFO *before* `pipeline_manager.teardown()`, so a worker's in-flight resources release first. `RuntimeBoot` exposes `_teardown_plugin_callbacks()` and `_teardown_runtime()` as separate phases; `Pipelex.teardown` sequences them with its own step between. For a lifecycle this order-sensitive, explicit sequencing beats a base-calls-derived hook.
+
+**The failed-boot release path is shared, not duplicated.** Both `make()`s call `_release_after_failed_boot()`. That block is the subtle part — a partial boot must release process globals it acquired or the next boot raises "LogConfig is already set" and serves a stale registry — and it deliberately touches only entry points safe on a half-built instance. Its docstring records *why* it cannot just call `teardown()`: `teardown` reads `self.inference_manager` (and `self.pipeline_manager`) unguarded, and both are assigned partway through `setup()`. That is intentional — guarding them would let a half-built teardown look successful.
+
+## Deliberate behaviour changes
+
+Each has a test; each is in the changelog.
+
+- **A second boot on top of an existing one fails loud.** `Pipelex.make()` refuses when a bare `RuntimeBoot` holds the process globals, and vice versa. The error message names the class that actually holds them, not a hardcoded "Pipelex".
+- **`config_dir` at boot reaches `setup_config`.**
+- **Breaking:** `Pipelex.__init__` takes keyword-only `config_dir` in place of the inert positional `config_dir_path`. Nothing in the workspace constructed `Pipelex` positionally.
+- **Breaking:** `teardown()` no longer propagates a plugin teardown callback's exception. The callbacks are attempted individually and failures are logged, so one bad plugin cannot strand another's resources.
+
+## Pre-existing bugs fixed (flag-and-fix)
+
+- **`config_dir` was silently ignored.** Stored at `pipelex.py:123` as `config_dir_path` and never read again; `setup_config` was called without a `config_dir` even though `runtime_hub.setup_config` accepts one. An embedder passing an explicit config dir got ordinary project/global layering and no error.
+- **`is_pipelex_service_enabled` was permanently a lie.** Assigned `False` in `__init__` with the comment "Will be set during setup", but `setup()` only ever assigned a **local** of the same name. Nothing in the tree or any sibling repo read it. Deleted rather than wired — a public attribute that always answers `False` is worse than no attribute.
+- **`make()`'s `pipeline_manager` was annotated with the concrete `PipelineManager`** while `setup()`'s was `PipelineManagerAbstract`. Aligned on the abstract, as every other injected manager is.
+
+Also noted and deliberately left alone: `teardown()` reads `self.pipeline_manager` and `self.inference_manager` unguarded, so teardown on a half-built instance raises `AttributeError`. That is *why* `make()` carries its own release path; guards there would let a half-built teardown look successful. The reason is now recorded in the release helper's docstring so the next reader finds the reason rather than the symptom.
+
+## How it is pinned
+
+- **Declared:** `pipelex.runtime_boot` in `RUNTIME_LAYER_PACKAGES`, a **module** entry beside `pipelex.runtime_hub`. Declaring it is the entire reason the rule reaches it.
+- **Membership asserted both ways:** `test_the_boot_split_left_the_runtime_half_declared` requires the runtime half to stay declared *and* `pipelex.pipelex` to stay undeclared. Deletion is the silent regression — the neighbouring checks all *iterate* the tuple and say nothing about a name that is simply gone. And declaring `pipelex.pipelex` would make the layer rule fail by design, so "surely the boot should be runtime-layer" is a plausible future edit worth catching.
+- **Import closure:** `"pipelex.runtime_boot"` added to `RUNTIME_LAYER_ENTRY_POINTS`.
+- **Boot closure:** `tests/unit/pipelex/test_runtime_boot_closure.py` *runs* `RuntimeBoot.make()` in a subprocess and asserts zero interpreter modules loaded, no `InterpreterHub` installed, and `class_registry_scoping` left at its unscoped default.
+
+ **The subprocess is required, not preferred.** Both hubs are sticky class-attribute singletons (`RuntimeHub._instance` / `InterpreterHub._instance`) that teardown deliberately never clears, so once anything in a process has booted a `Pipelex`, an in-process `InterpreterHub.get_optional_instance()` answers with the stale hub forever and the assertion **passes vacuously**. And *booting* rather than importing is what catches a boot step resolving out of an interpreter-contributed registry — see the next section.
+
+## The one thing worth reviewing hardest
+
+**Empty registries on a runtime-only boot.** A runtime-only `build_registrar` sees `RUNTIME_BUILTIN_PLUGINS` alone, so the `direct` orchestrator, the direct bundle validator and the built-in PipeFunc executor modes are all absent: `OrchestratorRegistry` and `BundleValidatorRegistry` are constructed **empty**. That is harmless only because nothing in the runtime half *resolves* out of them at boot. Verified per-site rather than assumed:
+
+- The only two runtime-half sites that resolve out of a plugin-derived registry are secrets (`secrets_provider_registry.get_required`) and storage (`storage_provider_registry.get_required`). Both are runtime-contributed *and* core-unconditional — `StoragePlugin`/`SecretsPlugin` are in `RUNTIME_BUILTIN_PLUGINS`, and `storage`/`secrets`/`openai` are in `RUNTIME_CORE_UNCONDITIONAL_PLUGIN_NAMES`.
+- `OrchestratorRegistry` / `BundleValidatorRegistry` are only constructed and set on the hub, never resolved at boot; they are looked up at run time by the interpreter.
+- The `TASK_MANAGER`, `ISOLATED_EXECUTION_PROBE` and `CONTENT_GENERATOR` slots are `slot_claims` lookups that no-op or fall to a core default when unclaimed.
+- `build_registrar` uses `core_unconditional_plugin_names` **only** as a may-not-be-denylisted guard, so passing the `RUNTIME_*` pair requires nothing interpreter-side to be present.
+
+The closure test cannot catch a *runtime resolution* failure, which is exactly why the booted-runtime test boots instead of importing.
+
+**Boot-orchestrator gate on a runtime-only boot.** `plugins.boot_orchestrator` defaults to `None` (`configs.py:250`, absent from every TOML), so the gate does not fire on a default runtime-only boot. When config *does* name an interpreter-contributed orchestrator, the existing check raises `UnknownBootOrchestratorError` unchanged — the decided behaviour, since a process with no interpreter genuinely cannot honour that runtime. No code change; the reason is a comment at the gate.
+
+**The interleaving assumption.** The whole plan rested on the interpreter constructions being deferrable to the tail. They were, confirmed per-site and then by the full suite. Two specific confirmations: `PipelineManager.setup()` is a literal `pass`, and `LibraryManager.__init__` is pure dict initialisation — so neither depends on the class registry being populated, which is what makes it safe for them to move *after* the `CoreRegistryModels` registration they used to precede. The two values that cross the seam go the other way (`self._plugin_registrar` and `self.multi_observer`, built by the runtime half and read by the tail).
+
+**A registration-order consequence, verified inert.** Old order was Core → Pipe → Test; it is now Core → Test (runtime half) → Pipe (interpreter tail). `kajson`'s `register_classes` is name-keyed dict insertion, `TestRegistryModels` holds only `FictionCharacter`, and `test_registry_models_split.py` already pins the manifests pairwise name-disjoint *order-independently*. It is also not fixable without making the layering wrong: `test_extras` is declared runtime-layer, so its registration belongs beside `CoreRegistryModels`.
+
+## Deliberate non-goals
+
+- **No `boot/` package.** Two top-level modules mirroring `runtime_hub.py` / `interpreter_hub.py` — the tree's strongest naming precedent for a layered pair, and it needs no top-level-package accounting in the guard's docstring.
+- **`Pipelex` does not move and is not renamed.** It is the public entry point every consumer and sibling repo imports; the boundary never required moving it.
+- **No parameter-list refactor.** `setup()`/`make()` duplicate a long typed parameter list and the split adds a second copy of the runtime subset. That is the cost of two typed entry points; a kwargs-forwarding fix would trade typing for brevity.
+- **No new abstraction over the two boots.** No protocol, no factory, no registry. The interpreter boot subclasses the runtime boot; that is the whole mechanism.
+- **Doctor was assessed and deliberately not adopted.** `setup_doctor_runtime` is the tree's only other bare-`RuntimeHub` builder, but its needs are **not** a strict subset: it requires `log.configure_if_unset` rather than `configure` (pinned by `tests/unit/pipelex/tools/test_log_idempotence.py`), a `log_config_overrides` deep-merge the agent doctor uses to keep its JSON envelope clean, and specifically *not* an `SdkClientManager` — doctor exists to report on models/plugins/inference without standing them up. Two flag parameters and an unwanted construction to de-duplicate ~10 lines is a worse outcome than two short sequences that differ honestly.
+
+## Honest scoping
+
+This fixes no bug in the layering and unblocks no feature. The guard was green before and stays green — nothing runtime-layer imports `pipelex.pipelex`, so there was no violation to remove. What changes is that a capability becomes expressible and a boundary maintained by discipline becomes maintained by a test.
+
+The fair reviewer question is "who calls the runtime-only boot?" The honest answer: the pinned closure and booted-runtime entry points in this PR, and any embedder wanting inference + storage + models without the method machinery. That is the same shape as `RUNTIME_BUILTIN_PLUGINS`, which the plugin split created with zero importers — and which is load-bearing here, as its first caller.
+
+## Verification performed
+
+- `make agent-check` — ruff, `generate-mthds-schema`, plxt, pyright (0 errors), mypy (2385 files), `check-keyword-only`, `check-hub-layering`. All green.
+- `make agent-test` — full suite passes. This is the proof the interleaving was incidental.
+- `make tb` — green.
+- `make drift-check` — green; `hub-layering-convention` reviewed and acked, with a dogfood-log entry. The contract's prescribed mechanical check **caught a real gap**: `pipelex.runtime_boot` was missing from the doc's enumeration of the declared runtime layer. Fixed. Seven consecutive rounds where an enumeration on that page was the rot, and three in a row where it was that same enumeration — the log now names automating it as the clearest candidate the pilot has produced.
+- **Cross-repo**, redone with `git -C <repo> grep` over repos enumerated from disk. `pipelex-transport`'s `ALLOWED_SURFACE` names no boot/runtime module, and zero references anywhere outside this repo touch either removed surface (`config_dir_path`, `is_pipelex_service_enabled`). Most downstream references are `from pipelex.pipelex import Pipelex` or `pipelex.pipelex.Pipelex.make(...)` — the *class*, which does not move — and keep working.
+
+ ⚠️ **Two real downstream breaks, which the first sweep missed.** `pipelex-temporal/tests/conftest.py:86` and `pipelex-transport/tests/conftest.py:89` each `session_mocker.patch("pipelex.pipelex.load_pipelex_service_config_if_exists", …)` in an **autouse session fixture**. That symbol now lives in `pipelex.runtime_boot`, so both suites `AttributeError` at session start — total suite failure, not a subtle one. Both pin `pipelex>=0.40.0` as a floor with no `tool.uv.sources` override, so it arrives at the next routine lockfile refresh rather than only at a deliberate bump. **Required cross-repo follow-up** (separate repos, so out of scope for this PR): repoint both strings to `pipelex.runtime_boot.load_pipelex_service_config_if_exists`.
+
+ **Why the first sweep missed them — the methodology matters more than the result.** `grep` in this environment resolves to a **shell function**, not `/usr/bin/grep`, and it does not traverse sibling repos: a recursive `grep -r` from the workspace root returned zero and read as a clean bill of health. That is the same failure shape as a hardcoded repo list — a repo the instrument never reached is indistinguishable from one that came back clean. `git -C <repo> grep` per enumerated repo is both correct and fast, and is the instrument to use for any future cross-repo sweep here.
+
+**Test-side fallout was mechanical:** six `mocker.patch` targets named `pipelex.pipelex.<symbol>` for symbols that now live in `runtime_boot`, repointed, with the shared four-file constant renamed `RUNTIME_BOOT_MODULE`. Those string references are invisible to every import lint — the hazard the guard's own docstring calls out.
+
+## Review already applied
+
+An independent reviewer with no context on the plan raised four findings against the staged diff. Three were accepted and fixed:
+
+1. **Three `TestClass`es in one module** — violates `pytest_standards.md` ("NEVER EVER put more than one TestClass into a test module"), which is not lint-gated. Split into `test_runtime_boot_closure.py`, `test_runtime_boot_exclusivity.py`, `test_runtime_boot_config_dir.py`. The 2-line `_test_integration_mode()` helper is duplicated across the two files needing it rather than hoisted — that is what `test_hub_lifecycle.py` already does with the same helper, so a shared helper module would be the novel thing, not the duplication.
+2. **A fresh dead attribute `self.config_dir`** — set in `__init__`, read nowhere. Not a functional bug (the local parameter is what reaches `setup_config`), but it re-created the exact trap this PR removes twice over. Deleted, with a comment recording why the boot deliberately keeps no copy.
+3. **Hardcoded "Pipelex" in `RuntimeBoot`'s error messages** — a lie for an embedder that never touched `Pipelex`. Both messages now name the real class. The sharpest part of the finding was that *nothing caught it*, since the test asserted only the exception type; the exclusivity test now asserts the wording, and a second test pins the reverse direction.
+
+The fourth was the registration-order consequence above — verified inert, not fixable without misplacing `test_extras`, and explicitly not a requested fix. Recorded so it is known rather than rediscovered.
+
+Codex then raised a fifth on the PR, **verified correct and deliberately deferred**: a runtime-only boot rejects an interpreter-layer orchestrator contributed by a *built-in* (it is never registered, since `builtin_plugins` defaults to the runtime half) but **not** one contributed by an *external* entry-point plugin, because `build_registrar` discovers externals unconditionally. Such a boot would apply the plugin's runtime claims — including `TASK_MANAGER` — never apply its `PIPE_ROUTER` / `PIPE_RUN` / `PIPE_FUNC_EXECUTOR` claims, and still report ready.
+
+Not fixed here for three reasons: it needs an external interpreter-side orchestrator installed *and* configured *and* a caller of `RuntimeBoot.make()`, of which there are none; every remedy needs a layer signal the runtime layer deliberately does not have (a `PipelexPlugin` carries no layer field, by design); and the nearest clean remedy is a flag on the very class pair that exists to avoid flags — which would contradict the doctor-adoption decision taken twenty lines away in the same file. The gate's comment now states the hole precisely instead of overclaiming (it previously said the runtime-only boot "also rejects an interpreter-contributed orchestrator name, which is correct" — true for built-ins, false for externals, so the comment was the actual defect). Full analysis and two candidate remedies: [`wip/boot-split/runtime-boot-external-interpreter-orchestrator.md`](wip/boot-split/runtime-boot-external-interpreter-orchestrator.md).
+
+### Round 7 — the same mistake twice, named
+
+Codex found a third defect on the failed-boot path, and it is worth recording because it is the **same reasoning error** as the round-6 one, one line away.
+
+My telemetry release call sat as the first statement in the `finally`, justified by a comment citing the built-in `TelemetryManager`'s "telemetry teardown must never break the app" property. But `self.telemetry_manager` is whatever the *factory* produced, and `telemetry_manager` is a public `make()` injection point typed only as `TelemetryManagerAbstract` — so an injected implementation carries no such guarantee. Unisolated, a raising one skips every release below it and replaces the boot error going out.
+
+**The generalisable mistake: reasoning from a concrete class's guarantees about a call made through an injectable abstract type.** Round 6 was the granularity version of it (catching around a loop over unbounded callbacks); this is the identity version. Both are now isolated per-call, and the comments say why rather than asserting safety.
+
+Pinned by `test_an_injected_telemetry_manager_that_raises_does_not_abort_the_other_releases`. Two things about its construction are deliberate: it patches `TelemetryFactory.make_telemetry_manager` rather than passing `telemetry_manager=`, because the injection is only adopted in some telemetry/integration modes and the subject is the resulting *state*; and it proves the suite completed by asserting the **singleton de-registration**, which is the suite's last statement, rather than by re-booting — which also keeps the patched factory from leaking into a second boot. Verified as a real control: reverting the isolation fails it.
+
+### Round 6 — Greptile back to 5/5; Codex found the granularity bug in my round-4 fix
+
+Codex's last finding is a consequence of the fix I made two rounds earlier, and it is a good one: my `except Exception` sat around the **loop** rather than inside it, so the first raising callback (LIFO, i.e. the last registered) ended the loop and every remaining callback was skipped. With two plugins registered — Temporal plus a sandbox executor is the realistic pair — the second one's runtime stayed live while the log said only that one callback had failed.
+
+"Catch around the loop" and "catch per iteration" look equivalent and are not, which is exactly why this needed a test rather than a re-read. Fixed by moving the catch inside the loop; the outer one became dead and was removed, though the `try`/`finally` stays, because `except Exception` does not cover `BaseException` and the state releases must still run if a teardown is interrupted.
+
+Pinned by `test_one_raising_teardown_callback_does_not_skip_the_others`, which registers the raiser **last** so LIFO reaches it first — the ordering that strands the others. Verified as a real control: restoring the around-the-loop catch fails it with "a raising callback skipped the remaining teardown callbacks".
+
+**Third deliberate behaviour change, now declared:** `teardown()` no longer propagates a plugin callback's exception. That follows from making the loop best-effort, and it is the right semantics — no caller can act on "plugin B failed to release", while every caller is harmed by B never being asked. Same shape as `TelemetryManager.teardown`, which wraps each of its own shutdown steps for the same reason.
+
+### Round 5 — Greptile blocked on the deferred telemetry leak, and it was right to
+
+Greptile dropped from 5/5 to **3/5** and said the PR should not merge until the failed-boot cleanup clears the telemetry singleton — the item the pre-landing review had raised and I had deferred. Two independent reviewers on the same defect, one of them blocking, so I re-examined the deferral and **it was wrong on cost**.
+
+`TelemetryManager.teardown()` already ends with `TelemetryManagerAbstract.clear_instance()`, and its internals are each wrapped defensively with the comment "telemetry teardown must never break the app". So the fix is **one guarded call** on a path that already guards the same attribute — not the "change to failure-path semantics with its own test matrix" I had described. I had conflated the narrow fix (release the singleton) with the broad one (collapse the two teardown paths), and deferred both on the strength of the broad one's cost.
+
+Fixed, and pinned by `tests/unit/pipelex/test_runtime_boot_failed_boot_release.py`: an injected models manager whose `setup()` raises reproduces the real shape (telemetry is already live by then), and the test asserts no telemetry singleton survives. Verified as a real control — with the release reverted it fails on exactly that assertion.
+
+What remains deferred is genuinely narrower and of a different kind: `sdk_client_manager`, `reporting_delegate` and `func_registry` are still not released on a failed boot. Those leave resources dangling rather than corrupting the next boot, and adding three more calls would widen a second hand-maintained copy of the teardown list. The right fix is to collapse the two paths, which is a lifecycle decision — [`wip/boot-split/failed-boot-does-not-release-every-resource.md`](wip/boot-split/failed-boot-does-not-release-every-resource.md).
+
+### Codex round 4 — two more, one fixed and one deferred after being attempted
+
+1. **Fixed: the `try`/`finally` still let a teardown error replace the boot error.** The `finally` guaranteed the releases, but a raising plugin callback propagated out of `_release_after_failed_boot()`, so `make()`'s bare `raise` was never reached and the caller saw the *teardown* failure instead of the one that killed the boot. This refines a judgement I had made and stated on the previous round — I declined to catch, reasoning that "the five existing operations have the same property, so it's all six or none." That reasoning was wrong on its own terms: only the plugin callbacks are **unbounded third-party code**, everything in the `finally` is ours, so catching exactly one of the six is principled rather than inconsistent — and it is case (2) of this repo's own `except Exception` carve-out (a user-registered callback whose exception surface cannot be enumerated), which requires a naming comment and gets one.
+2. **Deferred after being attempted: `config_dir` scopes the main TOML load but not the inference file paths.** Real: `models_manager.setup()` is called without path arguments, so backends, routing profiles and the model deck still resolve through the layered `config_manager.*` properties — an alternate `config_dir` combines its own main settings with whatever tree is detected. `ModelManager.setup`'s comment already names the remedy, and doctor's `--global` path implements it.
+
+ The four-line propagation was written and tested, then **backed out**: those path overrides exist on the concrete `ModelManager`, not on `ModelManagerAbstract`, which is what the boot is typed against *and* a documented `make()` injection point. Pyright rejects the call, and this repo forbids the `cast`/`isinstance` shortcut — the honest fix is to widen the abstract interface, which is a change to a public injection contract and does not belong in a placement refactor claiming unchanged behaviour. Two things learned while attempting it and worth carrying: completing the scoping makes an explicit `config_dir` **require a complete config directory** (verified — a dir holding only `pipelex.toml` then fails with `InferenceBackendLibraryNotFoundError`), which is the correct contract but a deliberate behavioural decision; and doctor spells the four paths as inline literals while `config_loader` already owns them as constants, so a second copy would double a drift risk that should be collapsed instead.
+
+ What shipped instead: both `make()` docstrings now state the limit explicitly rather than promising "only this directory is read", the boot carries a `NOTE:` at the call site, and the analysis plus a four-step suggested shape is in [`wip/boot-split/config-dir-does-not-scope-inference-paths.md`](wip/boot-split/config-dir-does-not-scope-inference-paths.md) — including that the negative test for it was written on this branch and verified to fail on a reverted propagation, so it can be lifted from history.
+
+### Pre-landing review triage (second independent reviewer)
+
+A pre-landing review then found five more, **four of them in the two commits that fixed the earlier findings** — a reminder that a fix commit deserves the same scrutiny as the change it fixes. All five accepted:
+
+1. **`_release_after_failed_boot` ran unbounded plugin code before the un-poisoning, with no `try`/`finally`.** Introduced by the leak fix itself. If a plugin teardown callback raised, the state releases below it never ran — logging stayed configured (every later boot dies on "LogConfig is already set"), `KajsonManager` kept a half-populated registry, the singleton stayed registered — *and* the teardown error replaced the exception that actually killed the boot. The method's entire purpose is to leave the process re-bootable. Now `try: callbacks / finally: releases`, which is the shape `AGENTS.md` prescribes for required cleanup — and the same rule Codex had cited when reporting the leak.
+2. **The exclusivity guard and the release resolved at different classes.** `make()` asked the base (so it saw any boot) but `teardown_if_needed()` asked `cls`, so `Pipelex.teardown_if_needed()` silently no-opped against a live bare `RuntimeBoot` while `Pipelex.make()` kept refusing — a deadlock with no recovery path, reachable from every existing `teardown_if_needed()` call site. Both now resolve at the base: a release must be able to clear everything its matching guard refuses on.
+3. **`__init__` took `RuntimeHub._instance` before any exclusivity check.** `set_runtime_hub` overwrites unconditionally, so constructing a second boot directly clobbered a live boot's hub and only then failed at `log.configure`, orphaning it. `MetaSingleton` used to make this unreachable *by accident* — a second `Pipelex(...)` short-circuits on the registry and never re-runs `__init__` — but a subclass is a different registry key, so the accident stopped covering it. The exclusivity check moved into `__init__`, which holds on every path including direct construction; both `make()` guards were deleted as redundant rather than duplicated.
+4. **The `config_dir` test bypassed `make()`.** Construct-then-`setup()` re-implements what `make()` wraps, and `teardown()` on a half-built instance raises `AttributeError` — masking the real error and leaving the singleton registered for the rest of the xdist worker. Now goes through `RuntimeBoot.make(config_dir=…)`, which already accepts it.
+5. **The booted-runtime test had lost its negative control.** Its sweep lives in a `textwrap.dedent` string, so nothing type-checks the predicate; the sibling import-closure module carries a `DIRTY_ENTRY_POINT` for exactly this reason and the copy dropped it. Verified rather than assumed: changing the sweep's `split(".")[1]` to `[0]` made the module green forever. A control case now treats `cogt` — which the runtime boot loads by definition — as an interpreter package and requires a failure. Re-verified: with the predicate broken, the real case passes and **the control fails**.
+
+A sixth was **pre-existing and deferred**: a failed boot leaves the `TelemetryManager` singleton live, so the next boot in the process adopts the dead one (reachable via `ensure_pipelex_booted`'s per-call lazy boot, since the commonest boot failure — `models_manager.setup()` on a missing deck or credentials — fires *after* telemetry setup). Unchanged from `dev`, and widening the release path is a change to failure-path semantics with its own test matrix, not a comment fix — so it was written up as `wip/inputs/failed-boot-leaks-telemetry-singleton.md` with a suggested shape. (That deferral was **reversed two rounds later** — see "Round 5" below — and the doc was deleted in `760b4a9b3`; what survives of it is [`wip/boot-split/failed-boot-does-not-release-every-resource.md`](wip/boot-split/failed-boot-does-not-release-every-resource.md).) What *was* fixed is the docstring, which claimed this path "releases the same process-global state `teardown()` does": it releases a subset, and the "only safe entry points" rationale never explained the omission, since `reporting_delegate` and `telemetry_manager` are already guarded.
+
+Codex raised a **sixth** on the re-review, and this one was accepted and fixed: **a failed boot leaked a plugin runtime.** `_release_after_failed_boot()` released process-global state but never ran the plugin teardown callbacks, so a boot dying after the `TASK_MANAGER` thunk left a live runtime running. Verified that the boot split genuinely widened the window — pre-split the thunk ran at `pipelex.py:524`, *after* the pipe-func executor resolution (471), `pipeline_manager.setup()` (499) and the pipe-class registration (504); all three now follow it. The most reachable trigger is a plain config error (`pipe_func_config.execution_mode` naming an unregistered mode). Fixed by running `_teardown_plugin_callbacks()` first on that path, mirroring the normal teardown ordering. Pinned by `test_a_failed_interpreter_tail_still_runs_plugin_teardown_callbacks`, which asserts the thunk ran *and* the callback ran so it cannot pass vacuously — and which was confirmed to fail with the fix removed.
diff --git a/docs/contribute/hub-layering.md b/docs/contribute/hub-layering.md
index 8ef18de26..9aacc27e4 100644
--- a/docs/contribute/hub-layering.md
+++ b/docs/contribute/hub-layering.md
@@ -155,14 +155,31 @@ The second split *is* the layer boundary, and the same rule of thumb divides it.
External plugins are unaffected by the mechanism/adapter split: they import `pipelex.plugins.contract` and `pipelex.plugins.registrar`, both of which stayed put, and they are discovered through the `pipelex.plugins` entry-point group — a group identifier, not a module path, and unchanged.
-The registrar's slots are *ports* and they stay runtime-layer — `OrchestratorProtocol`, `BundleValidatorProtocol` and `PipeFuncExecutorFactoryFn` all live under `plugins/*_registry.py`. Only the *adapters* are interpreter-layer. That is ordinary ports-and-adapters, and it is what the layer rule's own remedy text prescribes: have the interpreter layer install it downward at boot. Where each derived registry is *installed* still follows the hub partition rather than the port's home: `Pipelex.setup` puts the orchestrator, bundle-validator, backend, lister, storage and secrets registries on the `RuntimeHub`, and the `PipeFuncExecutorRegistry` on the `InterpreterHub` — the one placement flagged [above](#which-hub-does-a-new-symbol-belong-to).
+The registrar's slots are *ports* and they stay runtime-layer — `OrchestratorProtocol`, `BundleValidatorProtocol` and `PipeFuncExecutorFactoryFn` all live under `plugins/*_registry.py`. Only the *adapters* are interpreter-layer. That is ordinary ports-and-adapters, and it is what the layer rule's own remedy text prescribes: have the interpreter layer install it downward at boot. Where each derived registry is *installed* still follows the hub partition rather than the port's home: `RuntimeBoot.setup` puts the orchestrator, bundle-validator, backend, lister, storage and secrets registries on the `RuntimeHub`, and `Pipelex.setup` puts the `PipeFuncExecutorRegistry` on the `InterpreterHub` — the one placement flagged [above](#which-hub-does-a-new-symbol-belong-to). Since [the boot split](#where-the-boot-splits) the two installs sit in the two halves that match their hubs, so the partition is now visible in the code rather than only in this sentence.
-Composition happens where welding is legal. `pipelex/providers/builtins.py` exports `RUNTIME_BUILTIN_PLUGINS`; `pipelex/interpreter_plugins/builtins.py` imports that half — downward, which the interpreter layer may do — and exports the composed `BUILTIN_PLUGINS` and `CORE_UNCONDITIONAL_PLUGIN_NAMES`. Both are **parameters** of `build_registrar`, never module globals it imports, so `plugins/discovery.py` stays runtime-layer: `pipelex.py` (boot) and the `pipelex plugins list` diagnostic pass the composed lists in. Exactly one place still answers "what are the built-in plugins" — it just lives in the layer permitted to say it.
+Composition happens where welding is legal. `pipelex/providers/builtins.py` exports `RUNTIME_BUILTIN_PLUGINS`; `pipelex/interpreter_plugins/builtins.py` imports that half — downward, which the interpreter layer may do — and exports the composed `BUILTIN_PLUGINS` and `CORE_UNCONDITIONAL_PLUGIN_NAMES`. Both are **parameters** of `build_registrar`, never module globals it imports, so `plugins/discovery.py` stays runtime-layer: `pipelex.py` (the interpreter boot) and the `pipelex plugins list` diagnostic pass the composed lists in, while `runtime_boot.py` defaults to the runtime half alone. Exactly one place still answers "what are the built-in plugins" — it just lives in the layer permitted to say it.
This is the seam external plugins already use. Our Temporal plugin contributes an orchestrator and our Daytona plugin a PipeFunc executor; both are interpreter-touching, both live outside this repo, and both arrive through the `pipelex.plugins` entry point without welding anything. The in-tree built-ins used to be the only ones that welded — [Known inversions](#known-inversions) records what that cost.
One consequence is worth stating as a contract, and it is written into `PipelexPlugin`'s docstring: **a plugin belongs to exactly one layer.** A capability needing adapters in both is two plugins. Nothing enforces it mechanically; a straddling plugin would simply put the interpreter back into every runtime closure, and the guard's transitive rule would say so.
+## Where the boot splits
+
+The composition root is split the same way, and it is the same move: a runtime half in a declared runtime-layer home, and an interpreter half that imports it downward and adds what only a method-running process needs.
+
+`pipelex/runtime_boot.py` holds `RuntimeBoot`. It stands up config, logging, the secrets provider, telemetry, the Kajson class registry with `CoreRegistryModels`, the template sets, the SDK client manager, the model deck, the plugin-derived runtime registries, storage, the content generator, the inference manager, the reporting delegate and the observers — and it loads **zero interpreter modules** doing it. `pipelex/pipelex.py` holds `Pipelex`, which subclasses it and appends the `InterpreterHub`, the composed plugin manifests, the `PipeFuncExecutorRegistry` and its executor, the `LibraryManager` and default library dirs, the `PipelineManager`, the `PipeRegistryModels` registration, the `PipeRouter` and the `PipeRun`.
+
+Two top-level modules, not a `boot/` package: they mirror `runtime_hub.py` / `interpreter_hub.py`, which is the tree's strongest naming precedent for a layered pair. `Pipelex` does not move and is not renamed — it is the public entry point every consumer and sibling repo imports, and the layer boundary never required moving it.
+
+**This is what finally gives `RUNTIME_BUILTIN_PLUGINS` a caller.** The plugin split created that constant with no importer in the tree; `RuntimeBoot.setup` defaults to it, so a runtime-only boot discovers exactly the runtime half and the two interpreter-touching built-ins are simply absent. The consequence is worth stating plainly, because it is the reason nothing needed redesigning to make this work: `OrchestratorRegistry`, `BundleValidatorRegistry` and the PipeFunc executor modes are all interpreter-contributed, and therefore **empty on a runtime-only boot** — which is harmless only because nothing in the runtime half *resolves* out of them at boot. They are constructed, set on the hub, and looked up at run time by the interpreter. The two registries the runtime half does resolve out of, secrets and storage, are runtime-contributed and core-unconditional.
+
+Inheritance rather than composition, deliberately: it preserves every attribute address (`self.models_manager`, `self.class_registry`, `self.telemetry_manager` and the rest read the same from both halves, so no consumer inside or outside this repo changes), and it reads as what it is — the interpreter boot *is* the runtime boot plus the interpreter constructions. Two consequences are load-bearing:
+
+- **The singleton resolves by subclass.** `RuntimeBoot` carries `MetaSingleton` too, and the class-level accessors go through `MetaSingleton.get_subclass_instance`, so `RuntimeBoot.is_fully_booted()` answers `True` while a `Pipelex` owns the process globals. Each `make()` asks the *base* class whether a boot exists, which is what makes the two mutually exclusive in both directions — `set_runtime_hub`, `KajsonManager` and `log.configure` are all once-per-process, so booting one on top of the other would serve a half-populated class registry.
+- **Teardown ordering is explicit, not a template hook.** The plugin-contributed teardown callbacks must run *before* `pipeline_manager.teardown()`, so a worker's in-flight resources release first. `RuntimeBoot` exposes `_teardown_plugin_callbacks()` and `_teardown_runtime()` as separate phases and `Pipelex.teardown` sequences them with its own step in between, because that order is the whole reason the split exists in this shape.
+
+Enforcement mirrors the packages': `pipelex.runtime_boot` is declared in `RUNTIME_LAYER_PACKAGES` (a *module* entry, beside `pipelex.runtime_hub`) and listed in the closure test's `RUNTIME_LAYER_ENTRY_POINTS`. `pipelex.pipelex` is deliberately **undeclared** — it is interpreter-side by construction, like `pipelex.cli`, so declaring it would fail the rule by design rather than enforce anything. `tests/unit/pipelex/cli/dev/test_hub_layering_guard.py` asserts both directions, because "surely the boot should be runtime-layer" is a plausible future edit.
+
## Placement, not coupling
Splitting the hub removes the *lookups* that crossed the boundary. It does not move a type that was simply filed in the wrong package — and a misfiled type drags its whole package into every closure that names it. Two were resolved this way; both are worth knowing as the pattern to apply to the next one.
@@ -201,9 +218,9 @@ so `concept.py` cannot import `runtime_hub` at module level. Hosting the accesso
### Library scoping crosses downward, at install time
-A run may pin a per-library `ClassRegistry` rather than the process-global Kajson one. Knowing *which* library is current is method-layer knowledge, so the runtime layer holds only a slot: `class_registry_scoping`, defaulting to a resolver that returns `None`. `set_interpreter_hub` installs the real resolver, which means scoping is live exactly when a `InterpreterHub` exists — an invariant a caller cannot forget to wire. `Pipelex.teardown` calls `class_registry_scoping.reset()` so a torn-down library manager is never reachable through a still-pinned library id.
+A run may pin a per-library `ClassRegistry` rather than the process-global Kajson one. Knowing *which* library is current is method-layer knowledge, so the runtime layer holds only a slot: `class_registry_scoping`, defaulting to a resolver that returns `None`. `set_interpreter_hub` installs the real resolver, which means scoping is live exactly when a `InterpreterHub` exists — an invariant a caller cannot forget to wire. The runtime teardown phase (`RuntimeBoot._teardown_runtime`, reached from every `teardown`) calls `class_registry_scoping.reset()` so a torn-down library manager is never reachable through a still-pinned library id. It lives on the runtime half because releasing the slot belongs to whichever teardown runs — on a runtime-only boot nothing installed a resolver and the reset is a no-op.
-This is the same shape as `HubSlot.ISOLATED_EXECUTION_PROBE`: a defaulted callable a higher layer replaces at boot, never an unset attribute. A process that only ever builds a `RuntimeHub` (the doctor path, most unit tests) therefore degrades to the process-global registry instead of raising.
+This is the same shape as `HubSlot.ISOLATED_EXECUTION_PROBE`: a defaulted callable a higher layer replaces at boot, never an unset attribute. A process that only ever builds a `RuntimeHub` therefore degrades to the process-global registry instead of raising. That used to describe only the doctor path and most unit tests; since [the boot split](#where-the-boot-splits) it is a first-class entry point — `RuntimeBoot.make()` installs no `InterpreterHub`, and a test pins that scoping stays at its unscoped default on that boot.
## Enforcement
@@ -217,9 +234,9 @@ make check-hub-layering # alias: make chl
An AST guard (`pipelex-dev check-hub-layering`, core in `pipelex/cli/dev_cli/commands/hub_layering_guard.py`) that runs in `make agent-check`, in the `make check` aggregate, and in CI. It checks three rules over `pipelex/` and `tests/`:
-1. **The layer rule.** A module in the declared runtime layer — `pipelex.cogt`, `pipelex.core`, `pipelex.errors`, `pipelex.graph`, `pipelex.observer`, `pipelex.plugins`, `pipelex.providers`, `pipelex.reporting`, `pipelex.system`, `pipelex.test_extras`, `pipelex.tools`, `pipelex.tracing`, plus the `pipelex.runtime_hub` module itself — may not import `pipelex.interpreter_hub`. The declaration is compliant with an empty exception list, so the guard hard-blocks on **any** violation. Entries are matched exact-or-dotted-prefix, which is why a bare *module* is a legal entry beside the packages: without `pipelex.runtime_hub` spelled out, the module at the centre of the rule would be exempt from it. `pipelex.core` was six entries until its interpreter tenants moved out — see [Where core splits](#where-core-splits) for what that enumeration was working around.
+1. **The layer rule.** A module in the declared runtime layer — `pipelex.cogt`, `pipelex.core`, `pipelex.errors`, `pipelex.graph`, `pipelex.observer`, `pipelex.plugins`, `pipelex.providers`, `pipelex.reporting`, `pipelex.system`, `pipelex.test_extras`, `pipelex.tools`, `pipelex.tracing`, plus the `pipelex.runtime_hub` and `pipelex.runtime_boot` modules themselves — may not import `pipelex.interpreter_hub`. The declaration is compliant with an empty exception list, so the guard hard-blocks on **any** violation. Entries are matched exact-or-dotted-prefix, which is why a bare *module* is a legal entry beside the packages: without `pipelex.runtime_hub` spelled out, the module at the centre of the rule would be exempt from it, and without `pipelex.runtime_boot` the runtime layer's own composition root would be — see [Where the boot splits](#where-the-boot-splits). `pipelex.core` was six entries until its interpreter tenants moved out — see [Where core splits](#where-core-splits) for what that enumeration was working around.
- `graph`, `tracing`, `observer` and `errors` joined the list late, and the delay cost something — see [Known inversions](#known-inversions). They are the run-graph data model with its tracer and renderers, the trace-event assembler feeding it, the run-observation hooks, and the error taxonomy: machinery present at execution time whatever is loaded, which is the runtime layer's own definition. `pipelex.test_extras` is on the list for the same reason: it ships, `pipelex.py` imports it at boot, and it measures clean. `pipelex.kit` is data files with no module to police; `pipelex.language`, `pipelex.runtime_bridge` and `pipelex.cli` all measure dirty and are interpreter-side by construction. That is every top-level package accounted for — which is the point, because the rule of thumb is that a measured-clean package should be *declared*, not left undeclared: **an undeclared package is not neutral, it is unpoliced.**
+ `graph`, `tracing`, `observer` and `errors` joined the list late, and the delay cost something — see [Known inversions](#known-inversions). They are the run-graph data model with its tracer and renderers, the trace-event assembler feeding it, the run-observation hooks, and the error taxonomy: machinery present at execution time whatever is loaded, which is the runtime layer's own definition. `pipelex.test_extras` is on the list for the same reason: it ships, `runtime_boot.py` imports it at boot, and it measures clean. `pipelex.kit` is data files with no module to police; `pipelex.language`, `pipelex.runtime_bridge` and `pipelex.cli` all measure dirty and are interpreter-side by construction. That is every top-level package accounted for — which is the point, because the rule of thumb is that a measured-clean package should be *declared*, not left undeclared: **an undeclared package is not neutral, it is unpoliced.**
2. **The dead-module rule.** *No* scanned module may reference `pipelex.hub`. It was deleted rather than aliased so a stale import fails loudly; this closes the one hole in that guarantee.
3. **The transitive rule.** No runtime-layer module may *reach* `pipelex.interpreter_hub` either, through any chain of module-level imports. Rules 1 and 2 are per-file and see one hop; this one resolves the whole module-level import graph of `pipelex/` and does reachability over it, reporting the shortest chain and the line of its first hop. It exists because the one-hop rules missed a live breach for a whole release: four modules of the declared runtime-layer `pipelex.plugins` package reached the interpreter hub through `runtime_bridge`, `pipeline` and `pipe_operators`, and both gates stayed green (see [Known inversions](#known-inversions)). A *direct* import is left to rule 1, which names the more specific remedy, rather than reported twice.
@@ -265,7 +282,7 @@ The detector runs in a subprocess, so its logic lives in a `textwrap.dedent` str
Each subprocess is time-bounded, because an unbounded one turns a deadlock into a hung suite instead of a failure.
-Alongside them, `tests/unit/pipelex/test_hub_lifecycle.py` pins that a boot installs both singletons, and that the production `Pipelex.teardown` — not a stand-in calling `class_registry_scoping.reset()` directly — really releases the scoping an `InterpreterHub` installed, and that the next boot re-installs it.
+Alongside them, `tests/unit/pipelex/test_hub_lifecycle.py` pins that a boot installs both singletons, and that the production `Pipelex.teardown` — not a stand-in calling `class_registry_scoping.reset()` directly — really releases the scoping an `InterpreterHub` installed, and that the next boot re-installs it. `tests/unit/pipelex/test_runtime_boot_closure.py` pins the boot-time half of the property: it *runs* `RuntimeBoot.make()` in a subprocess and asserts zero interpreter modules loaded, no `InterpreterHub` installed, and scoping left unscoped. A subprocess is required rather than preferred — both hubs are sticky class-attribute singletons that teardown never clears, so an in-process check would answer with a stale hub and pass vacuously. Booting rather than importing is what catches a boot step resolving out of an interpreter-contributed registry that is empty on that boot.
### This document — the `hub-layering-convention` drift contract
diff --git a/docs/contribute/registration-surface.md b/docs/contribute/registration-surface.md
index 6985434d4..ac48fd4ab 100644
--- a/docs/contribute/registration-surface.md
+++ b/docs/contribute/registration-surface.md
@@ -34,7 +34,7 @@ Each kind lives in its own package under `pipelex/pipe_operators/<kind>/` or `pi
`PipeRegistryModels` used to live in `pipelex/core/registry_models.py` alongside core's value model. Importing it loads every pipe operator, every pipe controller and their factories — so a module filed under `core/` pulled in the whole interpreter, and `core` read as a dependant of the pipe packages it is supposed to sit below. The manifest now lives in `pipelex/pipe_machinery/`; `core.registry_models` keeps `CoreRegistryModels` (the stuff-content classes) and imports nothing from `pipe_operators` / `pipe_controllers` / `pipe_signature`.
-Both manifests are registered side by side at boot (`pipelex/pipelex.py`), and they must stay **disjoint** — a class in both is a sign one half was edited without the other.
+The two manifests land in one registry from the two halves of the boot: `CoreRegistryModels` is registered by `RuntimeBoot.setup` in `pipelex/runtime_boot.py`, `PipeRegistryModels` by `Pipelex.setup` in `pipelex/pipelex.py` — see [Where the boot splits](hub-layering.md#where-the-boot-splits). They must stay **disjoint** — a class in both is a sign one half was edited without the other — and because registration is name-keyed dict insertion, which side registers first carries no meaning.
See [Hub Layering](hub-layering.md) for the layer boundary this split serves.
diff --git a/docs/under-the-hood/init-cli-flows.md b/docs/under-the-hood/init-cli-flows.md
index 0df8ebb54..8344237a3 100644
--- a/docs/under-the-hood/init-cli-flows.md
+++ b/docs/under-the-hood/init-cli-flows.md
@@ -366,7 +366,7 @@ All writers explicitly target `config_manager.global_config_dir`:
| Context | Module | How |
|---------|--------|-----|
| Doctor | `doctor_cmd.py` → `check_models()` | `load_pipelex_service_config_if_exists(config_dir=global_config_dir)` |
-| Runtime | `pipelex.py` setup | `load_pipelex_service_config_if_exists(config_dir=global_config_dir)` |
+| Runtime | `runtime_boot.py` setup | `load_pipelex_service_config_if_exists(config_dir=global_config_dir)` |
### Rules
diff --git a/pipelex/cli/dev_cli/commands/hub_layering_guard.py b/pipelex/cli/dev_cli/commands/hub_layering_guard.py
index 755c74c59..a602008da 100644
--- a/pipelex/cli/dev_cli/commands/hub_layering_guard.py
+++ b/pipelex/cli/dev_cli/commands/hub_layering_guard.py
@@ -94,6 +94,17 @@ SCAN_ROOTS: tuple[Path, ...] = (SOURCE_ROOT, TESTS_ROOT)
#: since nothing in the tuple is a prefix of it. Its closure *is* what the runtime layer means, so an
#: `interpreter_hub` import there is the one that would break the property outright.
#:
+#: `pipelex.runtime_boot` is the second module entry, and for the mirror-image reason: it is the runtime
+#: layer's *composition root*, so its closure is what "boot the runtime layer" costs. Declaring it is
+#: what puts it inside this rule's domain at all — an undeclared module is not neutral, it is unpoliced,
+#: and this one is the single most tempting place in the tree to reach for an interpreter type, since
+#: the class it defines is subclassed by the boot that does need them. Its interpreter half,
+#: `pipelex.pipelex`, is deliberately *not* declared: it is interpreter-side by construction, like
+#: `pipelex.cli`, and declaring it would fail the rule by design rather than enforce anything.
+#: A *module* entry is right for both because neither wants a package: they are two top-level modules
+#: mirroring `runtime_hub` / `interpreter_hub`, which is the tree's strongest naming precedent and needs
+#: no top-level-package accounting in the paragraph below.
+#:
#: `pipelex.core` is a single entry, and getting it there was the point of the modularity work. `core/`
#: used to be genuinely two layers, so this tuple had to name its data-model packages one by one —
#: concepts, domains, memory, stuffs, and the input/output *specs* under `pipes/` — while excluding the
@@ -143,8 +154,9 @@ RUNTIME_LAYER_PACKAGES: tuple[str, ...] = (
"pipelex.test_extras",
"pipelex.tools",
"pipelex.tracing",
- # the runtime layer's own hub — a module, not a package; see the note above
+ # the runtime layer's own hub and its composition root — modules, not packages; see the note above
"pipelex.runtime_hub",
+ "pipelex.runtime_boot",
)
#: The interpreter layer's hub, which no runtime-layer module may import — directly or transitively.
diff --git a/pipelex/interpreter_plugins/builtins.py b/pipelex/interpreter_plugins/builtins.py
index a8ce500c6..a968d1443 100644
--- a/pipelex/interpreter_plugins/builtins.py
+++ b/pipelex/interpreter_plugins/builtins.py
@@ -8,9 +8,12 @@ for them, and it is *allowed* to import downward, so composing the two halves he
doing it in the runtime adapters' own manifest was the weld that made "the adapters are a runtime
layer" false.
-There is still exactly one place that answers "what are the built-in plugins" — it just lives in the
-layer permitted to do the welding. Both callers of ``build_registrar`` (boot in ``pipelex.py``, and the
-``pipelex plugins list`` diagnostic) consume the lists below. See ``docs/contribute/hub-layering.md``.
+There is still exactly one place that answers "what are the built-in plugins, both layers" — it just
+lives in the layer permitted to do the welding. The lists below are what ``Pipelex.setup`` (the
+interpreter boot) and the ``pipelex plugins list`` diagnostic pass into ``build_registrar``. The third
+caller is the runtime layer's own composition root, ``pipelex/runtime_boot.py``, which calls the same
+function but defaults to ``RUNTIME_BUILTIN_PLUGINS`` alone — it may not name this module.
+See ``docs/contribute/hub-layering.md``.
"""
from pipelex.interpreter_plugins.direct.direct_plugin import DirectOrchestratorPlugin
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index db7369de7..0936c9822 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -1,49 +1,38 @@
-import types
-import warnings
-from collections.abc import Callable
+"""The interpreter layer's composition root: the runtime boot plus the method machinery.
+
+``Pipelex`` is the public entry point every consumer and sibling repo imports, and it is the
+interpreter half of the boot: it subclasses :class:`pipelex.runtime_boot.RuntimeBoot` — importing it
+downward, which the interpreter layer may do — and appends the constructions that only a process
+which will actually load and run a method needs: the ``InterpreterHub``, the composed plugin
+manifests, the ``PipeFuncExecutorRegistry`` and its executor, the ``LibraryManager``, the
+``PipelineManager``, the pipe-kind class registrations, the ``PipeRouter`` and the ``PipeRun``.
+
+Inheritance rather than composition is deliberate: every attribute address is preserved
+(``self.models_manager``, ``self.class_registry``, ``self.telemetry_manager`` and the rest keep
+working from both halves), the singleton stays keyed such that ``Pipelex`` and ``RuntimeBoot`` exclude
+each other, and it reads as what it is — the interpreter boot *is* the runtime boot plus the
+interpreter constructions. See ``docs/contribute/hub-layering.md``.
+"""
+
from pathlib import Path
-from typing import TYPE_CHECKING, Any, Self, TypeVar, cast
+from typing import Any, Self
-from kajson.class_registry import ClassRegistry
from kajson.class_registry_abstract import ClassRegistryAbstract
-from kajson.kajson_manager import KajsonManager
-from pydantic import ValidationError
+from typing_extensions import override
from pipelex import log
-from pipelex.base_exceptions import PipelexConfigError, PipelexSetupError
-from pipelex.cogt.content_generation.content_generator import ContentGenerator
+from pipelex.base_exceptions import PipelexSetupError
from pipelex.cogt.content_generation.content_generator_protocol import (
ContentGeneratorProtocol,
)
-from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
-from pipelex.cogt.exceptions import (
- InferenceBackendCredentialsError,
- InferenceBackendLibraryNotFoundError,
- InferenceBackendLibraryValidationError,
- ModelDeckNotFoundError,
- ModelDeckValidationError,
- RoutingProfileDisabledBackendError,
- RoutingProfileLibraryNotFoundError,
-)
from pipelex.cogt.inference.inference_manager import InferenceManager
-from pipelex.cogt.model_backends.backend_credentials import (
- BackendCredentialsErrorMsgFactory,
-)
-from pipelex.cogt.model_backends.gateway_config import GatewayConfig
-from pipelex.cogt.models.model_manager import ModelManager
from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
from pipelex.config import get_config, get_pipe_func_execution_mode
-from pipelex.core.registry_models import CoreRegistryModels
-from pipelex.core.stuffs.stuff_template_set import STUFF_TEMPLATE_SET
-from pipelex.core.validation import report_validation_error
-from pipelex.graph.mermaidflow.template_set import MERMAID_TEMPLATE_SET
-from pipelex.graph.reactflow.template_set import REACTFLOW_TEMPLATE_SET
from pipelex.interpreter_hub import InterpreterHub, set_interpreter_hub
from pipelex.interpreter_plugins.builtins import BUILTIN_PLUGINS, CORE_UNCONDITIONAL_PLUGIN_NAMES
from pipelex.libraries.library_manager import LibraryManager
from pipelex.libraries.library_manager_abstract import LibraryManagerAbstract
-from pipelex.observer.multi_observer import MultiObserver
-from pipelex.observer.observer_protocol import ObserverNoOp, ObserverProtocol
+from pipelex.observer.observer_protocol import ObserverProtocol
from pipelex.pipe_machinery.registry_models import PipeRegistryModels
from pipelex.pipe_operators.func.pipe_func_executor_protocol import PipeFuncExecutorProtocol
from pipelex.pipe_run.pipe_router import PipeRouter
@@ -51,141 +40,45 @@ from pipelex.pipe_run.pipe_router_protocol import PipeRouterProtocol
from pipelex.pipe_run.pipe_run import PipeRun
from pipelex.pipeline.pipeline_manager import PipelineManager
from pipelex.pipeline.pipeline_manager_abstract import PipelineManagerAbstract
-from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry
-from pipelex.plugins.discovery import build_registrar
-from pipelex.plugins.exceptions import UnknownBootOrchestratorError
-from pipelex.plugins.inference_backend_registry import InferenceBackendRegistry
-from pipelex.plugins.model_lister_registry import ModelListerRegistry
-from pipelex.plugins.orchestrator_registry import OrchestratorRegistry
from pipelex.plugins.pipe_func_executor_registry import PipeFuncExecutorRegistry
-from pipelex.plugins.registrar import HubSlot, PluginRegistrar
-from pipelex.plugins.sdk_client_manager import SdkClientManager
-from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry
-from pipelex.plugins.storage_provider_registry import StorageProviderRegistry
-from pipelex.reporting.reporting_manager import ReportingManager
+from pipelex.plugins.registrar import HubSlot
from pipelex.reporting.reporting_protocol import ReportingProtocol
-from pipelex.runtime_hub import RuntimeHub, set_runtime_hub
-from pipelex.system.configuration.config_loader import config_manager
+from pipelex.runtime_boot import PACKAGE_NAME, PACKAGE_VERSION, RuntimeBoot
from pipelex.system.configuration.config_root import ConfigRoot
-from pipelex.system.configuration.configs import PipelexConfig
from pipelex.system.environment import get_pipelexpath_dirs
-from pipelex.system.pipelex_service.exceptions import (
- GatewayTermsNotAcceptedError,
- InferenceSetupRequiredError,
- RemoteConfigStaleWarning,
-)
-from pipelex.system.pipelex_service.pipelex_service_config import (
- is_pipelex_gateway_enabled,
- load_pipelex_service_config_if_exists,
-)
-from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
-from pipelex.system.registries.class_registry_access import class_registry_scoping
-from pipelex.system.registries.func_registry import FuncRegistry, func_registry
-from pipelex.system.registries.singleton import MetaSingleton
-from pipelex.system.runtime import IntegrationMode, runtime_manager
-from pipelex.system.telemetry.observer_telemetry import ObserverTelemetry
+from pipelex.system.runtime import IntegrationMode
from pipelex.system.telemetry.telemetry_config import (
TelemetryConfig,
)
-from pipelex.system.telemetry.telemetry_factory import TelemetryFactory
from pipelex.system.telemetry.telemetry_manager_abstract import (
TelemetryManagerAbstract,
)
-from pipelex.test_extras.registry_test_models import TestRegistryModels
-from pipelex.tools.jinja2.jinja2_template_loader import TemplateLoader
-from pipelex.tools.jinja2.jinja2_template_registry import TemplateRegistry
-from pipelex.tools.misc.package_utils import get_package_info
from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
-from pipelex.urls import URLs
-
-if TYPE_CHECKING:
- from pipelex.system.pipelex_service.remote_config import RemoteConfig
- from pipelex.system.pipelex_service.types import RemoteConfigSource
-PACKAGE_NAME, PACKAGE_VERSION = get_package_info()
-_HubSlotImplT = TypeVar("_HubSlotImplT")
-
-
-class Pipelex(metaclass=MetaSingleton):
+class Pipelex(RuntimeBoot):
def __init__(
self,
- config_dir_path: Path | None = None,
+ *,
+ config_dir: Path | None = None,
config_cls: type[ConfigRoot] | None = None,
config_overrides: dict[str, Any] | None = None,
) -> None:
- # Readiness gate: flipped True only at the very end of make(), after setup() and the optional
- # validate_model_deck() both succeed. Readers (ensure_pipelex_booted) must gate on this, NOT on
- # mere registry presence -- MetaSingleton registers the instance before setup() configures the hub.
- self.is_ready: bool = False
- self.is_pipelex_service_enabled = False # Will be set during setup
- self.config_dir_path = config_dir_path or config_manager.pipelex_config_dir
# Two hubs, two lifecycles: RuntimeHub is process-scoped infrastructure, InterpreterHub is the
# library-scoped method machinery. Runtime is constructed first because it is the lower layer,
# so it reads first — not because installing the InterpreterHub needs it: that install only
# stores the class-registry scoping resolver, which resolves lazily at call time.
- self.runtime_hub = RuntimeHub()
- set_runtime_hub(self.runtime_hub)
+ super().__init__(config_dir=config_dir, config_cls=config_cls, config_overrides=config_overrides)
self.interpreter_hub = InterpreterHub()
set_interpreter_hub(self.interpreter_hub)
- # tools
- try:
- self.runtime_hub.setup_config(config_cls=config_cls or PipelexConfig, config_overrides=config_overrides)
- except ValidationError as validation_error:
- validation_error_msg = report_validation_error(category="config", validation_error=validation_error)
- msg = f"Could not setup config because of: {validation_error_msg}"
- raise PipelexConfigError(msg) from validation_error
-
- log_config = get_config().pipelex.log_config
- self.runtime_hub.set_console_print_target(target=log_config.console_print_target)
- log.configure(log_config=log_config)
- log.verbose("Logs are configured")
-
- # tools
- self.class_registry: ClassRegistryAbstract | None = None
- self.func_registry: FuncRegistry | None = None
- # plugins — the registrar is built in setup() (build_registrar); held for the slot-claim /
- # CLI-command / teardown apply-points. Declared here so teardown() can guard on it.
- self._plugin_registrar: PluginRegistrar | None = None
- # cogt
- self.sdk_client_manager = SdkClientManager()
- self.runtime_hub.set_sdk_client_manager(self.sdk_client_manager)
-
- self.reporting_delegate: ReportingProtocol | None = None
- self.telemetry_manager: TelemetryManagerAbstract | None = None
# pipeline
self.library_manager: LibraryManagerAbstract | None = None
log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} init done")
- @staticmethod
- def _get_config_file_not_found_error_msg(*, component_name: str) -> str:
- """Generate error message for missing config files."""
- return f"Config files are missing for the {component_name}. Run `pipelex init config` to generate the missing files."
-
- @staticmethod
- def _get_validation_error_msg(*, component_name: str, validation_exc: Exception) -> str:
- """Generate error message for invalid config files."""
- msg = ""
- cause_exc = validation_exc.__cause__
- if cause_exc is None:
- msg += f"\nUnexpexted error:{cause_exc}"
- raise PipelexSetupError(msg) from cause_exc
- if not isinstance(cause_exc, ValidationError):
- msg += f"\nUnexpexted cause:{cause_exc}"
- raise PipelexSetupError(msg) from cause_exc
- report = report_validation_error(category="config", validation_error=cause_exc)
- return f"""{msg}
-{report}
-
-Config files are invalid for the {component_name}.
-You can fix them manually, or run `pipelex init config` to regenerate them.
-Note that this command resets all config files to their default values.
-If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
-"""
-
+ @override
def setup(
self,
*,
@@ -209,253 +102,46 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
library_manager: LibraryManagerAbstract | None = None,
library_dirs: list[str] | list[Path] | None = None,
**kwargs: Any,
- ):
+ ) -> None:
if kwargs:
- msg = f"The base setup method does not support any additional arguments: {kwargs}"
+ msg = f"The Pipelex setup method does not support any additional arguments: {kwargs}"
raise PipelexSetupError(msg)
- # Boot this process under the named orchestrator plugin, when explicitly provided.
- # The matching boot-orchestrator plugin (e.g. Temporal) claims the hub slots in its
- # register() iff plugins.boot_orchestrator == its own name; any other value is in-process.
- if boot_orchestrator is not None:
- get_config().plugins.boot_orchestrator = boot_orchestrator
-
- # --- Pipelex Service and Telemetry --------------------------------------------------
-
- # Check if Pipelex Gateway is enabled
- # for now the only servic is the Pipelex Gateway
- is_pipelex_service_enabled = is_pipelex_gateway_enabled()
-
- effective_needs_model_specs = needs_model_specs if needs_model_specs is not None else needs_inference
-
- remote_config: RemoteConfig | None = None
- gateway_config: GatewayConfig | None = None
- gateway_config_source: RemoteConfigSource | None = None
- if is_pipelex_service_enabled:
- if not effective_needs_model_specs:
- # Use dummy config when inference is not needed (for testing without network access)
- remote_config = RemoteConfigFetcher.make_dummy_remote_config()
- gateway_model_specs = remote_config.backend_model_specs
- gateway_config = GatewayConfig(
- model_specs=gateway_model_specs,
- aws_region=remote_config.aws_region,
- )
- # Keep ``gateway_config_source`` as ``None``: the dummy specs are an empty
- # placeholder, not real Gateway data. ``ModelManager._enforce_gateway_model_membership``
- # treats ``source is None`` as "nothing to validate against," so the membership
- # check is skipped on this path — which is what we want for read-only flows like
- # ``pipelex-agent models`` without ``--backend``.
- log.verbose("Using dummy remote config (inference not needed)")
- else:
- # Terms acceptance is only required for actual inference usage, not for
- # read-only operations like fetching model specs for validation.
- # Also skip for CI mode — automated pipelines don't require human consent.
- if needs_inference and integration_mode.requires_terms_acceptance:
- pipelex_service_config = load_pipelex_service_config_if_exists(config_dir=config_manager.global_config_dir)
- # First-run check: fires if inference has never been configured
- # AND terms were never accepted (terms_accepted=true means existing
- # user who already completed gateway setup before this flag existed).
- if pipelex_service_config is None or (
- not pipelex_service_config.onboarding.inference_setup_completed and not pipelex_service_config.agreement.terms_accepted
- ):
- raise InferenceSetupRequiredError
- # Gateway terms check: this block only runs when gateway is
- # enabled (is_pipelex_service_enabled guard above). BYOK users
- # who disabled gateway via init skip this entire block.
- if not pipelex_service_config.agreement.terms_accepted:
- raise GatewayTermsNotAcceptedError
- # Fetch remote configuration (may fall back to on-disk cache when offline).
- remote_config_result = RemoteConfigFetcher.fetch_remote_config()
- remote_config = remote_config_result.config
- gateway_config_source = remote_config_result.source
- log.verbose(f"Successfully fetched Pipelex Gateway remote configuration (source={gateway_config_source})")
- gateway_model_specs = remote_config.backend_model_specs
- gateway_config = GatewayConfig(
- model_specs=gateway_model_specs,
- aws_region=remote_config.aws_region,
- )
- # Stale operation: warn loudly so machine consumers can re-surface the provenance.
- # Emission lives at this orchestration layer (not in the fetcher) so the fetcher
- # stays a pure data-returning function — and so test fixtures that swap in a
- # cached fetcher (tests/conftest.py) don't need to special-case warning replay.
- if gateway_config_source.is_cached:
- cached_at_iso = remote_config_result.cached_at.isoformat() if remote_config_result.cached_at else "unknown"
- warnings.warn(
- f"Pipelex Gateway is running off a cached remote config (snapshot: {cached_at_iso}). "
- "Run `pipelex init` while online to refresh.",
- RemoteConfigStaleWarning,
- stacklevel=2,
- )
-
- # --- Plugin discovery -----------------------------------------------------------------
- # Build the plugin registrar from the fully-resolved config (pure and import-light:
- # registering the built-ins imports no backend SDK, constructs no client, touches no hub).
- # Built here — after the gateway service/terms precondition gate above (so an unaccepted-terms or
- # first-run boot fails fast before any discovery work) and before the telemetry factory below,
- # which is the first consumer of the secrets provider. Secrets is now a config-selected plugin
- # seam: the built-in SecretsPlugin's factory (and any external pipelex-secrets-<backend>) is
- # looked up from the registrar-derived SecretsProviderRegistry just below. The other registries
- # (inference, storage, …) are still built later at their own hub-set points, all referencing this
- # same already-built registrar; the slot-claim thunks / teardown callbacks it also accumulates are
- # applied at their ordered apply-points in later phases.
- plugin_registrar = build_registrar(
- config=get_config(),
+ # The runtime layer first, with the *composed* plugin manifests: this process will run methods,
+ # so it needs the interpreter-touching built-ins (the `direct` orchestrator, the built-in
+ # PipeFunc executor modes) alongside the runtime half a bare RuntimeBoot discovers.
+ super().setup(
+ integration_mode=integration_mode,
+ needs_inference=needs_inference,
+ boot_orchestrator=boot_orchestrator,
+ needs_model_specs=needs_model_specs,
builtin_plugins=BUILTIN_PLUGINS,
core_unconditional_plugin_names=CORE_UNCONDITIONAL_PLUGIN_NAMES,
- )
- self._plugin_registrar = plugin_registrar
- # Reject an unknown boot orchestrator before falling through to the core defaults. The requested
- # name (CLI --orchestrator / setup(boot_orchestrator=...) / config) is matched against registered
- # plugin names — the same namespace the slot-claim gate uses (boot_orchestrator == plugin.name).
- # When no plugin of that name registered (not installed, disabled, or a typo) nothing claims the
- # hub slots, so without this guard the run would silently execute in-process instead of under the
- # requested runtime. Checked here to fail fast, before the telemetry/model work.
- requested_boot_orchestrator = get_config().plugins.boot_orchestrator
- if requested_boot_orchestrator is not None and requested_boot_orchestrator not in plugin_registrar.registered_plugin_names:
- raise UnknownBootOrchestratorError(requested=requested_boot_orchestrator)
-
- # Secrets provider precedence: explicit setup() param > config-selected registry factory.
- # The built-in SecretsPlugin supplies the "env" method, so there is no separate core default.
- # Resolved here because the telemetry factory just below (and the model setup further down) consume it.
- secrets_provider_registry = SecretsProviderRegistry(plugin_registrar.secrets_providers)
- self.runtime_hub.set_secrets_provider_registry(secrets_provider_registry)
- if secrets_provider is None:
- secrets_config = get_config().pipelex.secrets_config
- secrets_provider = secrets_provider_registry.get_required(method=secrets_config.method)(secrets_config)
-
- # Disable Pipelex telemetry when:
- # - inference is not needed (no live runs to track), OR
- # - the gateway config came from the cache (stale specs imply potentially stale model
- # identities; phoning home about pipe runs in that state would pollute metrics).
- gateway_source_is_cached = gateway_config_source is not None and gateway_config_source.is_cached
- is_pipelex_telemetry_enabled = is_pipelex_service_enabled and needs_inference and not gateway_source_is_cached
- self.telemetry_manager = TelemetryFactory.make_telemetry_manager(
+ class_registry=class_registry,
secrets_provider=secrets_provider,
- integration_mode=integration_mode,
- remote_config=remote_config,
- is_pipelex_telemetry_enabled=is_pipelex_telemetry_enabled,
+ storage_provider=storage_provider,
+ models_manager=models_manager,
+ inference_manager=inference_manager,
+ content_generator=content_generator,
+ reporting_delegate=reporting_delegate,
telemetry_config=telemetry_config,
- injected_telemetry_manager=telemetry_manager,
- )
- self.telemetry_manager.setup(integration_mode=integration_mode)
- self.runtime_hub.set_telemetry_manager(telemetry_manager=self.telemetry_manager)
-
- # --- Tools ----------------------------------------------------------------------------
-
- self.class_registry = class_registry or ClassRegistry()
- self.kajson_manager = KajsonManager(class_registry=self.class_registry)
-
- self.func_registry = func_registry or FuncRegistry()
- self.runtime_hub.set_func_registry(func_registry=self.func_registry)
- self.runtime_hub.set_secrets_provider(secrets_provider=secrets_provider)
- # Storage is selected from the config-driven StorageProviderRegistry, built from the plugin
- # registrar (constructed above, just before the telemetry factory). Its resolution and hub-set
- # still happen later at the plugin-derived-registries block — after secrets is on the hub here,
- # so the GCP factory's secret read works.
-
- # Register stuff templates first (used by mermaid, reactflow, and stuff_viewer)
- stuff_name, stuff_package, stuff_templates = STUFF_TEMPLATE_SET
- TemplateLoader.register_set(
- name=stuff_name,
- package=stuff_package,
- templates=stuff_templates,
- )
- reactflow_name, reactflow_package, reactflow_templates = REACTFLOW_TEMPLATE_SET
- TemplateLoader.register_set(
- name=reactflow_name,
- package=reactflow_package,
- templates=reactflow_templates,
+ telemetry_manager=telemetry_manager,
+ observers=observers,
)
- mermaid_name, mermaid_package, mermaid_templates = MERMAID_TEMPLATE_SET
- TemplateLoader.register_set(
- name=mermaid_name,
- package=mermaid_package,
- templates=mermaid_templates,
- )
- TemplateLoader.load_all()
-
- # --- AI Models and Inference Management ------------------------------------------------
- self.sdk_client_manager.setup()
+ # --- The interpreter layer ---------------------------------------------------------------
+ # Everything below needs a method to be loadable. Nothing the runtime setup above does
+ # consumes any of it, which is what lets these be a tail rather than an interleaving; the two
+ # values that cross the seam go the other way (``self._plugin_registrar`` and
+ # ``self.multi_observer``, both built by the runtime half and read here).
- self.models_manager: ModelManagerAbstract = models_manager or ModelManager()
- self.runtime_hub.set_models_manager(models_manager=self.models_manager)
+ plugin_registrar = self._plugin_registrar
+ if plugin_registrar is None:
+ msg = "The plugin registrar must be built by the runtime setup before the interpreter setup runs"
+ raise PipelexSetupError(msg)
- try:
- self.models_manager.setup(
- secrets_provider=secrets_provider,
- gateway_config=gateway_config,
- gateway_config_source=gateway_config_source,
- needs_inference=needs_inference,
- )
- except RoutingProfileLibraryNotFoundError as routing_not_found_exc:
- msg = self._get_config_file_not_found_error_msg(component_name="routing profile library")
- raise PipelexSetupError(msg) from routing_not_found_exc
- except InferenceBackendLibraryNotFoundError as backend_not_found_exc:
- msg = self._get_config_file_not_found_error_msg(component_name="inference backend library")
- raise PipelexSetupError(msg) from backend_not_found_exc
- except ModelDeckNotFoundError as deck_not_found_exc:
- msg = self._get_config_file_not_found_error_msg(component_name="model deck")
- raise PipelexSetupError(msg) from deck_not_found_exc
- except RoutingProfileDisabledBackendError as routing_profile_exc:
- msg = f"Some backend(s) required for a routing profile is not enabled: {routing_profile_exc}"
- raise PipelexSetupError(msg) from routing_profile_exc
-
- except InferenceBackendLibraryValidationError as backend_validation_exc:
- msg = self._get_validation_error_msg(component_name="inference backend library", validation_exc=backend_validation_exc)
- raise PipelexSetupError(msg) from backend_validation_exc
- except ModelDeckValidationError as deck_validation_exc:
- msg = self._get_validation_error_msg(component_name="model deck", validation_exc=deck_validation_exc)
- msg += "\n\nIf you added your own config files to the model deck then you'll have to change them manually."
- raise PipelexSetupError(msg) from deck_validation_exc
-
- except InferenceBackendCredentialsError as credentials_exc:
- backend_name = credentials_exc.backend_name
- var_name = credentials_exc.key_name
- error_msg = BackendCredentialsErrorMsgFactory.make_one_variable_missing_error_msg(
- secrets_provider=secrets_provider,
- backend_name=backend_name,
- var_name=var_name,
- )
- raise PipelexSetupError(error_msg) from credentials_exc
-
- # Keyless boot forces every run to DRY — consumed at PipeRunParamsFactory.make_run_params,
- # the single writer of run_mode, covering every entry point; generator selection is
- # backend-keyed unconditionally (eng review D4) — a keyless Temporal submitter must still
- # dispatch activities and mock inside them, so `needs_inference` plays no role in picking the
- # generator. Its other boot roles (gateway/model setup, credentials, telemetry) are unchanged.
- # --- Plugin-derived registries --------------------------------------------------------
- # The plugin registrar was built earlier (with the boot-orchestrator gate checked and the
- # config-selected secrets provider resolved) just before the telemetry factory. Turn its
- # accumulated contributions into the hub registries here — after the gateway/model setup checks
- # and before the hub setup points below — the family worker factories look their backends up on
- # these at run time.
- self.runtime_hub.set_inference_backend_registry(InferenceBackendRegistry(plugin_registrar.inference_backends))
- self.runtime_hub.set_model_lister_registry(ModelListerRegistry(plugin_registrar.model_listers))
- self.runtime_hub.set_orchestrator_registry(OrchestratorRegistry(plugin_registrar.orchestrators))
- self.runtime_hub.set_bundle_validator_registry(BundleValidatorRegistry(plugin_registrar.bundle_validators))
- storage_provider_registry = StorageProviderRegistry(plugin_registrar.storage_providers)
- self.runtime_hub.set_storage_provider_registry(storage_provider_registry)
pipe_func_executor_registry = PipeFuncExecutorRegistry(plugin_registrar.pipe_func_executors)
self.interpreter_hub.set_pipe_func_executor_registry(pipe_func_executor_registry)
- # Storage provider precedence: explicit setup() param > config-selected registry factory.
- # The built-in StoragePlugin supplies every method, so there is no separate core default.
- # Resolves here (after secrets is on the hub) so the GCP factory's secret read works.
- if storage_provider is None:
- storage_config = get_config().pipelex.storage_config
- storage_provider = storage_provider_registry.get_required(method=storage_config.method)(storage_config)
- self.runtime_hub.set_storage_provider(storage_provider)
-
- self.runtime_hub.set_dry_run_forced(is_forced=not needs_inference)
- # Injection precedence (codex C8): explicit setup() param > plugin slot-claim thunk > core default.
- # A boot-orchestrator plugin (Temporal worker) claims the CONTENT_GENERATOR slot; its thunk runs
- # only here, never at register, so booting a non-worker process imports no host-runtime SDK.
- if content_generator is None:
- content_generator = self._resolve_hub_slot(
- slot=HubSlot.CONTENT_GENERATOR,
- default=lambda: ContentGenerator(generated_content_factory=GeneratedContentFactory(storage_provider=storage_provider)),
- )
- self.runtime_hub.set_content_generator(content_generator)
# Injection precedence: explicit setup() param > plugin slot-claim thunk > config-selected
# registry factory. The PipeFunc execution axis is orthogonal to orchestration: a Temporal
@@ -472,14 +158,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
)
self.interpreter_hub.set_pipe_func_executor(pipe_func_executor)
- self.inference_manager = inference_manager or InferenceManager()
- self.runtime_hub.set_inference_manager(self.inference_manager)
-
- # --- Libraries & Registries -------------------------------------------------------------
-
- self.reporting_delegate = reporting_delegate or ReportingManager()
- self.runtime_hub.set_report_delegate(self.reporting_delegate)
- self.reporting_delegate.setup()
+ # --- Libraries -------------------------------------------------------------------------
self.library_manager = library_manager or LibraryManager()
self.interpreter_hub.set_library_manager(library_manager=self.library_manager)
@@ -498,41 +177,14 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.interpreter_hub.set_pipeline_manager(pipeline_manager=self.pipeline_manager)
self.pipeline_manager.setup()
- # Two manifests, one registry: core's value model and the pipe kinds. They are disjoint by
- # construction and pinned as such by tests/unit/pipelex/test_registry_models_split.py.
- self.class_registry.register_classes(CoreRegistryModels.get_all_models())
+ # Two manifests, one registry: core's value model (registered by the runtime setup above) and
+ # the pipe kinds. They are disjoint by construction and pinned as such by
+ # tests/unit/pipelex/test_registry_models_split.py, so the two calls landing in different
+ # halves carries no meaning beyond that.
+ if self.class_registry is None:
+ msg = "The class registry must be created by the runtime setup before the interpreter setup runs"
+ raise PipelexSetupError(msg)
self.class_registry.register_classes(PipeRegistryModels.get_all_models())
- if runtime_manager.is_unit_testing:
- log.verbose("Registering test models for unit testing")
- self.class_registry.register_classes(TestRegistryModels.get_all_models())
-
- # --- Observers -------------------------------------------------------------------------
-
- if not observers:
- no_op_observer = ObserverNoOp()
- observer_telemetry = ObserverTelemetry(telemetry_manager=self.telemetry_manager)
- observers = {"noop": no_op_observer, "telemetry": observer_telemetry}
- multi_observer = MultiObserver(observers=observers)
-
- # --- Task Manager ----------------------------------------------------------------------
- # The TASK_MANAGER slot is claimed only by a boot-orchestrator plugin running this process as
- # its runtime (a Temporal worker). The thunk does the full wiring on the plugin's own hub and
- # is torn down via a registered teardown callback (LIFO) — no core default, no explicit param.
-
- task_manager_factory = plugin_registrar.slot_claims.get(HubSlot.TASK_MANAGER)
- if task_manager_factory is not None:
- task_manager_factory()
-
- # --- Isolated-execution probe ----------------------------------------------------------
- # Claimed only by a boot-orchestrator plugin whose runtime has a replay/activity split (e.g. a
- # Temporal worker): the thunk resolves the ambient predicate that reports whether the current
- # call runs inside an isolated sub-execution (an activity). ReportingManager consults it to
- # route an activity-side usage emission to the per-process log instead of the workflow's
- # registered buffer (audit H1). Unclaimed (any in-process boot), the fresh hub's default
- # reports "never isolated", so no wiring is needed here.
- isolated_execution_probe_factory = plugin_registrar.slot_claims.get(HubSlot.ISOLATED_EXECUTION_PROBE)
- if isolated_execution_probe_factory is not None:
- self.runtime_hub.set_isolated_execution_probe(isolated_execution_probe_factory())
# --- Pipe Router -----------------------------------------------------------------------
# Injection precedence (codex C8): explicit setup() param > plugin slot-claim thunk > core default.
@@ -541,7 +193,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.interpreter_hub.set_pipe_router(pipe_router)
else:
self.interpreter_hub.set_pipe_router(
- self._resolve_hub_slot(slot=HubSlot.PIPE_ROUTER, default=lambda: PipeRouter(observer=multi_observer))
+ self._resolve_hub_slot(slot=HubSlot.PIPE_ROUTER, default=lambda: PipeRouter(observer=self.multi_observer))
)
# --- Pipe Run --------------------------------------------------------------------------
@@ -553,65 +205,28 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} setup done")
- def _resolve_hub_slot(self, *, slot: HubSlot, default: Callable[[], _HubSlotImplT]) -> _HubSlotImplT:
- """Resolve a process-global hub slot: a plugin's claimed thunk if present, else the core default.
-
- Call sites apply explicit-injection precedence first (an explicit ``setup()`` param wins),
- so this only arbitrates plugin-claim vs core-default. The claim is a thunk invoked here at the
- boot apply-point — never during ``register`` — so a non-worker boot constructs no plugin impl.
- The return type follows the core ``default`` (the claimed thunk is typed ``Any`` and adopts it).
- """
- if self._plugin_registrar is not None:
- factory = self._plugin_registrar.slot_claims.get(slot)
- if factory is not None:
- return cast("_HubSlotImplT", factory())
- return default()
-
- def teardown(self):
- # Plugin-contributed teardown callbacks (LIFO) — e.g. a Temporal worker tears down its task
- # manager + resets its hub. Names no integration: the callbacks were registered by whichever
- # boot-orchestrator plugin claimed the runtime, and run before core teardown so a worker's
- # in-flight Temporal resources release first.
- if self._plugin_registrar is not None:
- for teardown_callback in reversed(self._plugin_registrar.teardown_callbacks):
- teardown_callback()
-
- # pipelex
- self.pipeline_manager.teardown()
- if self.telemetry_manager:
- self.telemetry_manager.teardown()
-
- # cogt
- self.inference_manager.teardown()
- if self.reporting_delegate:
- self.reporting_delegate.teardown()
- self.sdk_client_manager.teardown()
-
- # tools
- self.kajson_manager.teardown()
- if self.class_registry:
- self.class_registry.teardown()
- func_registry.teardown()
- TemplateLoader.reset()
- TemplateRegistry.clear()
-
- log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} teardown done (except config & logs)")
- # Both hubs release their process-global state: the RuntimeHub drops its config, and the
- # class-registry scoping resolver the InterpreterHub installed at boot goes back to its unscoped
- # default so a torn-down library manager can never be reached through it.
- self.runtime_hub.reset_config()
- class_registry_scoping.reset()
- # Clear the singleton instance from metaclass
- if self.__class__ in MetaSingleton.instances:
- del MetaSingleton.instances[self.__class__]
-
- def __enter__(self) -> Self:
- return self
-
- def __exit__(self, exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: types.TracebackType | None) -> None:
- self.teardown()
+ @override
+ def teardown(self) -> None:
+ # The three phases in this order deliberately: the plugin-contributed callbacks (LIFO) run
+ # first — e.g. a Temporal worker tears down its task manager + resets its hub — so a worker's
+ # in-flight resources release before the pipeline manager drops the pipelines they may still
+ # be reporting on. Sequenced explicitly here rather than through a template hook, because that
+ # order is the whole reason the runtime teardown is split into phases.
+ # ``try``/``finally`` and not a bare sequence, covering *both* leading phases: ``pipeline_manager``
+ # is a public ``make()`` injection point typed as ``PipelineManagerAbstract``, so its ``teardown``
+ # can raise, and the plugin callbacks are unbounded third-party code whose per-callback
+ # ``except Exception`` does not cover ``BaseException``. ``_teardown_runtime`` is what leaves the
+ # process re-bootable, so skipping it would wedge the process for good — see its docstring. The
+ # order is preserved by keeping both statements in the ``try``. No ``except``: the failure still
+ # propagates.
+ try:
+ self._teardown_plugin_callbacks()
+ self.pipeline_manager.teardown()
+ finally:
+ self._teardown_runtime()
@classmethod
+ @override
def make(
cls,
*,
@@ -626,13 +241,14 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
inference_manager: InferenceManager | None = None,
content_generator: ContentGeneratorProtocol | None = None,
pipe_func_executor: PipeFuncExecutorProtocol | None = None,
- pipeline_manager: PipelineManager | None = None,
+ pipeline_manager: PipelineManagerAbstract | None = None,
pipe_router: PipeRouterProtocol | None = None,
reporting_delegate: ReportingProtocol | None = None,
telemetry_config: TelemetryConfig | None = None,
telemetry_manager: TelemetryManagerAbstract | None = None,
observers: dict[str, ObserverProtocol] | None = None,
library_dirs: list[str] | list[Path] | None = None,
+ config_dir: Path | None = None,
config_overrides: dict[str, Any] | None = None,
**kwargs: Any,
) -> Self:
@@ -675,6 +291,13 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
library_dirs: Default library directories for pipeline execution. If provided, these
directories will be used instead of the PIPELEXPATH environment variable.
Per-call library_dirs in execute/start will override this default.
+ config_dir: Optional explicit config dir. When provided, the **main TOML load** is scoped
+ to this directory (package defaults + this directory) instead of following
+ project/global layering. Note the limit: it scopes *that load* and nothing else. The
+ inference files — backends, routing profiles and the model deck — still resolve through
+ the layered paths, and the gateway consent/onboarding state is read from the global
+ config dir outright. So this does not fully isolate a boot from the surrounding
+ project. See ``wip/boot-split/config-dir-does-not-scope-inference-paths.md``.
config_overrides: Optional dict deep-merged on top of all TOML config layers
as the highest-priority override. Useful for tests that need specific
config without editing TOML files.
@@ -684,14 +307,16 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
Initialized Pipelex instance.
Raises:
- PipelexSetupError: If Pipelex is already initialized or setup fails
+ PipelexSetupError: If a boot already holds the process globals (a ``Pipelex`` or a bare
+ ``RuntimeBoot``), or if setup fails.
"""
- if cls.get_optional_instance() is not None:
- msg = "Pipelex is already initialized"
- raise PipelexSetupError(msg)
+ # Before the construction, not left to ``__init__``: a second ``make()`` on an already-registered
+ # class never re-runs ``__init__`` (see the guard's docstring). Asked of ``RuntimeBoot``, so a
+ # bare runtime boot blocks this one too.
+ cls.raise_if_a_boot_already_holds_the_process_globals()
- pipelex_instance = cls(config_overrides=config_overrides)
+ pipelex_instance = cls(config_dir=config_dir, config_overrides=config_overrides)
try:
pipelex_instance.setup(
integration_mode=integration_mode,
@@ -717,24 +342,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
if needs_inference:
pipelex_instance.models_manager.validate_model_deck()
except BaseException:
- # A failed boot must release the process-global singletons it acquired, not just the
- # singleton registration — otherwise they leak and poison the next boot in the same
- # process. setup() establishes these progressively (logging + hub config in __init__, the
- # KajsonManager class registry, the template registries) and may mutate config (e.g.
- # plugins.boot_orchestrator); failing partway skips teardown(), the normal release point.
- # We release the same process-global state teardown() does, but via its class-level entry
- # points so it is safe on a half-built instance (full teardown() would touch managers a
- # partial setup never assigned). Without this the next boot raises "LogConfig is already
- # set" and serves a stale, half-populated class registry (the KajsonManager singleton
- # ignores a fresh registry once created).
- pipelex_instance.runtime_hub.reset_config()
- class_registry_scoping.reset()
- KajsonManager.teardown()
- TemplateLoader.reset()
- TemplateRegistry.clear()
- # Cleanup the singleton instance if setup fails to avoid "already initialized" errors.
- if cls in MetaSingleton.instances:
- del MetaSingleton.instances[cls]
+ pipelex_instance._release_after_failed_boot()
raise
# Publish readiness only now: setup() AND the optional validate_model_deck() have both succeeded
# and the delete-on-failure handler above is behind us, so a reader can never adopt an instance
@@ -742,38 +350,3 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
pipelex_instance.is_ready = True
log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} ready")
return pipelex_instance
-
- @classmethod
- def get_optional_instance(cls) -> Self | None:
- instance = MetaSingleton.instances.get(cls)
- return cast("Self | None", instance)
-
- @classmethod
- def is_fully_booted(cls) -> bool:
- """True only when a singleton exists AND has completed make() (setup + validation).
-
- Distinct from ``get_optional_instance() is not None``: the metaclass registers the instance
- before ``setup()`` configures the hub. Callers that must not touch a half-built instance
- (``ensure_pipelex_booted``) gate on this.
- """
- instance = cls.get_optional_instance()
- return instance is not None and instance.is_ready
-
- @classmethod
- def get_instance(cls) -> Self:
- instance = MetaSingleton.instances.get(cls)
- if instance is None:
- msg = "Pipelex is not initialized"
- raise RuntimeError(msg)
- return cast("Self", instance)
-
- @classmethod
- def teardown_if_needed(cls) -> None:
- """Teardown the Pipelex singleton instance if it exists.
-
- This is useful for cleanup in finally blocks where the instance
- may or may not have been successfully created.
- """
- instance = cls.get_optional_instance()
- if instance is not None:
- instance.teardown()
diff --git a/pipelex/runtime_boot.py b/pipelex/runtime_boot.py
new file mode 100644
index 000000000..15b881340
--- /dev/null
+++ b/pipelex/runtime_boot.py
@@ -0,0 +1,955 @@
+"""The runtime layer's composition root: stand up inference without the method interpreter.
+
+Pipelex has two layers and one hub each (``docs/contribute/hub-layering.md``). Every package in the
+tree is placed on one side of that line and pinned there, but the *composition root* was the one
+module that never got the treatment: a single class that booted both layers in one interleaved
+sequence, so the only way into the process constructed an ``InterpreterHub``, a ``LibraryManager``, a
+``PipelineManager``, a ``PipeRouter`` and a ``PipeRun`` whether the caller would ever load a method or
+not. The layering property was real for *importing* and vacuous for *booting*.
+
+This module is the runtime half. It stands up config, logging, secrets, telemetry, the class and func
+registries, the template sets, the model deck, storage, the content generator, the inference manager,
+the reporting delegate and the observers — everything present at execution time whatever is loaded —
+and it loads **zero interpreter modules** doing it. ``pipelex.pipelex.Pipelex`` is the interpreter half:
+it subclasses this class, imports it downward (which the interpreter layer may do) and appends the
+method machinery.
+
+The split is the same move the built-in plugin manifests made, and it is what finally gives
+``RUNTIME_BUILTIN_PLUGINS`` a caller: a runtime-only boot discovers exactly that half, so the
+``direct`` orchestrator, the direct bundle validator and the built-in PipeFunc executor modes — all
+interpreter-contributed — are simply absent. Nothing here resolves out of those registries at boot;
+they are looked up at run time, by the interpreter.
+
+**Every import in this module stays at module top level.** That is not incidental: a function-local
+import is precisely what hides a breach from the static guard and the import-closure test at the same
+time, so if one ever seems necessary here the placement is wrong and the type should move instead.
+The module is declared in the hub-layering guard's ``RUNTIME_LAYER_PACKAGES`` and listed in the
+closure test's ``RUNTIME_LAYER_ENTRY_POINTS``; a booted-runtime test pins the same property through
+``make()`` rather than through an import.
+
+Three ``runtime_*`` names now live at the top of the package and they are different things:
+``runtime_hub`` is the runtime layer's service container, ``runtime_bridge`` is a transport, and
+``runtime_boot`` — this module — is the runtime layer's composition root.
+"""
+
+import types
+import warnings
+from collections.abc import Callable
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, Self, TypeVar, cast
+
+from kajson.class_registry import ClassRegistry
+from kajson.class_registry_abstract import ClassRegistryAbstract
+from kajson.kajson_manager import KajsonManager
+from pydantic import ValidationError
+
+from pipelex import log
+from pipelex.base_exceptions import PipelexConfigError, PipelexSetupError
+from pipelex.cogt.content_generation.content_generator import ContentGenerator
+from pipelex.cogt.content_generation.content_generator_protocol import (
+ ContentGeneratorProtocol,
+)
+from pipelex.cogt.content_generation.generated_content_factory import GeneratedContentFactory
+from pipelex.cogt.exceptions import (
+ InferenceBackendCredentialsError,
+ InferenceBackendLibraryNotFoundError,
+ InferenceBackendLibraryValidationError,
+ ModelDeckNotFoundError,
+ ModelDeckValidationError,
+ RoutingProfileDisabledBackendError,
+ RoutingProfileLibraryNotFoundError,
+)
+from pipelex.cogt.inference.inference_manager import InferenceManager
+from pipelex.cogt.model_backends.backend_credentials import (
+ BackendCredentialsErrorMsgFactory,
+)
+from pipelex.cogt.model_backends.gateway_config import GatewayConfig
+from pipelex.cogt.models.model_manager import ModelManager
+from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
+from pipelex.config import get_config
+from pipelex.core.registry_models import CoreRegistryModels
+from pipelex.core.stuffs.stuff_template_set import STUFF_TEMPLATE_SET
+from pipelex.core.validation import report_validation_error
+from pipelex.graph.mermaidflow.template_set import MERMAID_TEMPLATE_SET
+from pipelex.graph.reactflow.template_set import REACTFLOW_TEMPLATE_SET
+from pipelex.observer.multi_observer import MultiObserver
+from pipelex.observer.observer_protocol import ObserverNoOp, ObserverProtocol
+from pipelex.plugins.bundle_validator_registry import BundleValidatorRegistry
+from pipelex.plugins.discovery import build_registrar
+from pipelex.plugins.exceptions import UnknownBootOrchestratorError
+from pipelex.plugins.inference_backend_registry import InferenceBackendRegistry
+from pipelex.plugins.model_lister_registry import ModelListerRegistry
+from pipelex.plugins.orchestrator_registry import OrchestratorRegistry
+from pipelex.plugins.registrar import HubSlot, PluginRegistrar
+from pipelex.plugins.sdk_client_manager import SdkClientManager
+from pipelex.plugins.secrets_provider_registry import SecretsProviderRegistry
+from pipelex.plugins.storage_provider_registry import StorageProviderRegistry
+from pipelex.providers.builtins import RUNTIME_BUILTIN_PLUGINS, RUNTIME_CORE_UNCONDITIONAL_PLUGIN_NAMES
+from pipelex.reporting.reporting_manager import ReportingManager
+from pipelex.reporting.reporting_protocol import ReportingProtocol
+from pipelex.runtime_hub import RuntimeHub, set_runtime_hub
+from pipelex.system.configuration.config_loader import config_manager
+from pipelex.system.configuration.config_root import ConfigRoot
+from pipelex.system.configuration.configs import PipelexConfig
+from pipelex.system.pipelex_service.exceptions import (
+ GatewayTermsNotAcceptedError,
+ InferenceSetupRequiredError,
+ RemoteConfigStaleWarning,
+)
+from pipelex.system.pipelex_service.pipelex_service_config import (
+ is_pipelex_gateway_enabled,
+ load_pipelex_service_config_if_exists,
+)
+from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
+from pipelex.system.registries.class_registry_access import class_registry_scoping
+from pipelex.system.registries.func_registry import FuncRegistry, func_registry
+from pipelex.system.registries.singleton import MetaSingleton
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.system.telemetry.observer_telemetry import ObserverTelemetry
+from pipelex.system.telemetry.telemetry_config import (
+ TelemetryConfig,
+)
+from pipelex.system.telemetry.telemetry_factory import TelemetryFactory
+from pipelex.system.telemetry.telemetry_manager_abstract import (
+ TelemetryManagerAbstract,
+)
+from pipelex.test_extras.registry_test_models import TestRegistryModels
+from pipelex.tools.jinja2.jinja2_template_loader import TemplateLoader
+from pipelex.tools.jinja2.jinja2_template_registry import TemplateRegistry
+from pipelex.tools.misc.package_utils import get_package_info
+from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
+from pipelex.tools.storage.storage_provider_abstract import StorageProviderAbstract
+from pipelex.urls import URLs
+
+if TYPE_CHECKING:
+ from collections.abc import Sequence
+
+ from pipelex.plugins.contract import PipelexPlugin
+ from pipelex.system.pipelex_service.remote_config import RemoteConfig
+ from pipelex.system.pipelex_service.types import RemoteConfigSource
+
+PACKAGE_NAME, PACKAGE_VERSION = get_package_info()
+
+_HubSlotImplT = TypeVar("_HubSlotImplT")
+
+
+class RuntimeBoot(metaclass=MetaSingleton):
+ """Boot the runtime layer: inference, storage, models, telemetry — no method interpreter.
+
+ Subclassed by ``pipelex.pipelex.Pipelex``, which appends the interpreter constructions. The
+ singleton is therefore shared between the two: the class-level accessors below resolve **by
+ subclass**, so ``RuntimeBoot.is_fully_booted()`` answers ``True`` when a ``Pipelex`` owns the
+ process globals. Resolving by exact class instead would tell the embedder this module exists for
+ that nothing is booted while a ``Pipelex`` held the runtime hub.
+ """
+
+ def __init__(
+ self,
+ *,
+ config_dir: Path | None = None,
+ config_cls: type[ConfigRoot] | None = None,
+ config_overrides: dict[str, Any] | None = None,
+ ) -> None:
+ # Readiness gate: flipped True only at the very end of make(), after setup() and the optional
+ # validate_model_deck() both succeed. Readers (ensure_pipelex_booted) must gate on this, NOT on
+ # mere registry presence -- MetaSingleton registers the instance before setup() configures the hub.
+ # The process globals are single-owner, so exactly one boot may exist. Note *why* that needs a
+ # guard rather than falling out of the code: the three fail in three different ways and only one
+ # of them is loud. ``log.configure`` refuses a second call. ``set_runtime_hub`` overwrites
+ # unconditionally, leaving the *first* boot running against a hub nothing else resolves to.
+ # ``KajsonManager`` does the opposite and is the easiest to get backwards: it is a singleton, so
+ # ``KajsonManager(class_registry=…)`` hands back the existing manager and silently discards the
+ # fresh registry — the *second* boot then serves the first one's half-populated class registry
+ # while its own ``self.class_registry`` is what nothing resolves to.
+ #
+ # Checked here *as well as* in ``make()`` — see the guard's docstring for why neither site covers
+ # the other's case. Here it catches a direct construction, which is where the damage would be
+ # done: ``set_runtime_hub`` below overwrites ``RuntimeHub._instance`` unconditionally, so a direct
+ # construction while another boot is live would orphan that boot from the global hub and only then
+ # fail, at ``log.configure``.
+ self.raise_if_a_boot_already_holds_the_process_globals()
+
+ self.is_ready: bool = False
+ # ``config_dir`` is deliberately not stored on the instance. Its predecessor
+ # (``self.config_dir_path``) was, and nothing ever read it, which is precisely how it came to be
+ # dropped on the floor instead of reaching ``setup_config``. Keeping a copy beside the parameter
+ # that is actually authoritative would re-create that trap for the next reader.
+ # The runtime hub is process-scoped infrastructure, and it is constructed first because it is
+ # the lower layer, so it reads first. The interpreter half installs its own hub afterwards;
+ # that install does not need this one (it only stores the class-registry scoping resolver,
+ # which resolves lazily at call time).
+ self.runtime_hub = RuntimeHub()
+ set_runtime_hub(self.runtime_hub)
+
+ # tools
+ try:
+ self.runtime_hub.setup_config(
+ config_cls=config_cls or PipelexConfig,
+ config_overrides=config_overrides,
+ config_dir=config_dir,
+ )
+ except ValidationError as validation_error:
+ validation_error_msg = report_validation_error(category="config", validation_error=validation_error)
+ msg = f"Could not setup config because of: {validation_error_msg}"
+ raise PipelexConfigError(msg) from validation_error
+
+ log_config = get_config().pipelex.log_config
+ self.runtime_hub.set_console_print_target(target=log_config.console_print_target)
+ log.configure(log_config=log_config)
+ log.verbose("Logs are configured")
+
+ # tools
+ self.class_registry: ClassRegistryAbstract | None = None
+ self.func_registry: FuncRegistry | None = None
+ # plugins — the registrar is built in setup() (build_registrar); held for the slot-claim /
+ # CLI-command / teardown apply-points. Declared here so teardown() can guard on it.
+ self._plugin_registrar: PluginRegistrar | None = None
+ # cogt
+ self.sdk_client_manager = SdkClientManager()
+ self.runtime_hub.set_sdk_client_manager(self.sdk_client_manager)
+
+ self.reporting_delegate: ReportingProtocol | None = None
+ self.telemetry_manager: TelemetryManagerAbstract | None = None
+
+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} runtime init done")
+
+ @staticmethod
+ def _get_config_file_not_found_error_msg(*, component_name: str) -> str:
+ """Generate error message for missing config files."""
+ return f"Config files are missing for the {component_name}. Run `pipelex init config` to generate the missing files."
+
+ @staticmethod
+ def _get_validation_error_msg(*, component_name: str, validation_exc: Exception) -> str:
+ """Generate error message for invalid config files."""
+ msg = ""
+ cause_exc = validation_exc.__cause__
+ if cause_exc is None:
+ msg += f"\nUnexpexted error:{cause_exc}"
+ raise PipelexSetupError(msg) from cause_exc
+ if not isinstance(cause_exc, ValidationError):
+ msg += f"\nUnexpexted cause:{cause_exc}"
+ raise PipelexSetupError(msg) from cause_exc
+ report = report_validation_error(category="config", validation_error=cause_exc)
+ return f"""{msg}
+{report}
+
+Config files are invalid for the {component_name}.
+You can fix them manually, or run `pipelex init config` to regenerate them.
+Note that this command resets all config files to their default values.
+If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
+"""
+
+ def setup(
+ self,
+ *,
+ integration_mode: IntegrationMode,
+ needs_inference: bool = True,
+ boot_orchestrator: str | None = None,
+ needs_model_specs: bool | None = None,
+ builtin_plugins: "Sequence[PipelexPlugin] | None" = None,
+ core_unconditional_plugin_names: frozenset[str] | None = None,
+ class_registry: ClassRegistryAbstract | None = None,
+ secrets_provider: SecretsProviderAbstract | None = None,
+ storage_provider: StorageProviderAbstract | None = None,
+ models_manager: ModelManagerAbstract | None = None,
+ inference_manager: InferenceManager | None = None,
+ content_generator: ContentGeneratorProtocol | None = None,
+ reporting_delegate: ReportingProtocol | None = None,
+ telemetry_config: TelemetryConfig | None = None,
+ telemetry_manager: TelemetryManagerAbstract | None = None,
+ observers: dict[str, ObserverProtocol] | None = None,
+ **kwargs: Any,
+ ) -> None:
+ """Stand up the runtime layer.
+
+ ``builtin_plugins`` defaults to ``RUNTIME_BUILTIN_PLUGINS`` — the runtime-layer half — so a bare
+ runtime boot never loads the interpreter-touching built-ins; the interpreter boot passes the
+ composed list instead. ``core_unconditional_plugin_names`` defaults to the runtime-layer half for
+ the same reason, and must describe the same set as ``builtin_plugins``: requiring a name that was
+ never discovered fails boot.
+
+ Every other argument is the runtime subset of the same injections ``Pipelex.make`` documents.
+ """
+ if kwargs:
+ msg = f"The base setup method does not support any additional arguments: {kwargs}"
+ raise PipelexSetupError(msg)
+
+ # Boot this process under the named orchestrator plugin, when explicitly provided.
+ # The matching boot-orchestrator plugin (e.g. Temporal) claims the hub slots in its
+ # register() iff plugins.boot_orchestrator == its own name; any other value is in-process.
+ if boot_orchestrator is not None:
+ get_config().plugins.boot_orchestrator = boot_orchestrator
+
+ # --- Pipelex Service and Telemetry --------------------------------------------------
+
+ # Check if Pipelex Gateway is enabled
+ # for now the only servic is the Pipelex Gateway
+ is_pipelex_service_enabled = is_pipelex_gateway_enabled()
+
+ effective_needs_model_specs = needs_model_specs if needs_model_specs is not None else needs_inference
+
+ remote_config: RemoteConfig | None = None
+ gateway_config: GatewayConfig | None = None
+ gateway_config_source: RemoteConfigSource | None = None
+ if is_pipelex_service_enabled:
+ if not effective_needs_model_specs:
+ # Use dummy config when inference is not needed (for testing without network access)
+ remote_config = RemoteConfigFetcher.make_dummy_remote_config()
+ gateway_model_specs = remote_config.backend_model_specs
+ gateway_config = GatewayConfig(
+ model_specs=gateway_model_specs,
+ aws_region=remote_config.aws_region,
+ )
+ # Keep ``gateway_config_source`` as ``None``: the dummy specs are an empty
+ # placeholder, not real Gateway data. ``ModelManager._enforce_gateway_model_membership``
+ # treats ``source is None`` as "nothing to validate against," so the membership
+ # check is skipped on this path — which is what we want for read-only flows like
+ # ``pipelex-agent models`` without ``--backend``.
+ log.verbose("Using dummy remote config (inference not needed)")
+ else:
+ # Terms acceptance is only required for actual inference usage, not for
+ # read-only operations like fetching model specs for validation.
+ # Also skip for CI mode — automated pipelines don't require human consent.
+ if needs_inference and integration_mode.requires_terms_acceptance:
+ pipelex_service_config = load_pipelex_service_config_if_exists(config_dir=config_manager.global_config_dir)
+ # First-run check: fires if inference has never been configured
+ # AND terms were never accepted (terms_accepted=true means existing
+ # user who already completed gateway setup before this flag existed).
+ if pipelex_service_config is None or (
+ not pipelex_service_config.onboarding.inference_setup_completed and not pipelex_service_config.agreement.terms_accepted
+ ):
+ raise InferenceSetupRequiredError
+ # Gateway terms check: this block only runs when gateway is
+ # enabled (is_pipelex_service_enabled guard above). BYOK users
+ # who disabled gateway via init skip this entire block.
+ if not pipelex_service_config.agreement.terms_accepted:
+ raise GatewayTermsNotAcceptedError
+ # Fetch remote configuration (may fall back to on-disk cache when offline).
+ remote_config_result = RemoteConfigFetcher.fetch_remote_config()
+ remote_config = remote_config_result.config
+ gateway_config_source = remote_config_result.source
+ log.verbose(f"Successfully fetched Pipelex Gateway remote configuration (source={gateway_config_source})")
+ gateway_model_specs = remote_config.backend_model_specs
+ gateway_config = GatewayConfig(
+ model_specs=gateway_model_specs,
+ aws_region=remote_config.aws_region,
+ )
+ # Stale operation: warn loudly so machine consumers can re-surface the provenance.
+ # Emission lives at this orchestration layer (not in the fetcher) so the fetcher
+ # stays a pure data-returning function — and so test fixtures that swap in a
+ # cached fetcher (tests/conftest.py) don't need to special-case warning replay.
+ if gateway_config_source.is_cached:
+ cached_at_iso = remote_config_result.cached_at.isoformat() if remote_config_result.cached_at else "unknown"
+ warnings.warn(
+ f"Pipelex Gateway is running off a cached remote config (snapshot: {cached_at_iso}). "
+ "Run `pipelex init` while online to refresh.",
+ RemoteConfigStaleWarning,
+ stacklevel=2,
+ )
+
+ # --- Plugin discovery -----------------------------------------------------------------
+ # Build the plugin registrar from the fully-resolved config (pure and import-light:
+ # registering the built-ins imports no backend SDK, constructs no client, touches no hub).
+ # Built here — after the gateway service/terms precondition gate above (so an unaccepted-terms or
+ # first-run boot fails fast before any discovery work) and before the telemetry factory below,
+ # which is the first consumer of the secrets provider. Secrets is now a config-selected plugin
+ # seam: the built-in SecretsPlugin's factory (and any external pipelex-secrets-<backend>) is
+ # looked up from the registrar-derived SecretsProviderRegistry just below. The other registries
+ # (inference, storage, …) are still built later at their own hub-set points, all referencing this
+ # same already-built registrar; the slot-claim thunks / teardown callbacks it also accumulates are
+ # applied at their ordered apply-points in later phases.
+ plugin_registrar = build_registrar(
+ config=get_config(),
+ builtin_plugins=RUNTIME_BUILTIN_PLUGINS if builtin_plugins is None else builtin_plugins,
+ core_unconditional_plugin_names=(
+ RUNTIME_CORE_UNCONDITIONAL_PLUGIN_NAMES if core_unconditional_plugin_names is None else core_unconditional_plugin_names
+ ),
+ )
+ self._plugin_registrar = plugin_registrar
+ # Reject an unknown boot orchestrator before falling through to the core defaults. The requested
+ # name (CLI --orchestrator / setup(boot_orchestrator=...) / config) is matched against registered
+ # plugin names — the same namespace the slot-claim gate uses (boot_orchestrator == plugin.name).
+ # When no plugin of that name registered (not installed, disabled, or a typo) nothing claims the
+ # hub slots, so without this guard the run would silently execute in-process instead of under the
+ # requested runtime. Checked here to fail fast, before the telemetry/model work.
+ #
+ # On a runtime-only boot this rejects an orchestrator contributed by an interpreter-layer
+ # *built-in*, because ``builtin_plugins`` defaults to the runtime half and the name is therefore
+ # never registered. It does NOT reject an *external* entry-point plugin that is interpreter-side:
+ # ``build_registrar`` discovers external plugins unconditionally, so such a name satisfies this
+ # check, and a bare runtime boot would then apply the runtime slot claims while silently never
+ # applying the interpreter ones (``PIPE_ROUTER`` / ``PIPE_RUN`` / ``PIPE_FUNC_EXECUTOR``, all
+ # applied in ``Pipelex.setup``). Unreachable today — nothing calls ``RuntimeBoot.make`` yet — and
+ # deliberately not guarded here, because every remedy needs a layer signal this layer does not
+ # have. Analysed in ``wip/boot-split/runtime-boot-external-interpreter-orchestrator.md``; the first
+ # caller of a runtime-only boot is who has to settle it.
+ requested_boot_orchestrator = get_config().plugins.boot_orchestrator
+ if requested_boot_orchestrator is not None and requested_boot_orchestrator not in plugin_registrar.registered_plugin_names:
+ raise UnknownBootOrchestratorError(requested=requested_boot_orchestrator)
+
+ # Secrets provider precedence: explicit setup() param > config-selected registry factory.
+ # The built-in SecretsPlugin supplies the "env" method, so there is no separate core default.
+ # Resolved here because the telemetry factory just below (and the model setup further down) consume it.
+ secrets_provider_registry = SecretsProviderRegistry(plugin_registrar.secrets_providers)
+ self.runtime_hub.set_secrets_provider_registry(secrets_provider_registry)
+ if secrets_provider is None:
+ secrets_config = get_config().pipelex.secrets_config
+ secrets_provider = secrets_provider_registry.get_required(method=secrets_config.method)(secrets_config)
+
+ # Disable Pipelex telemetry when:
+ # - inference is not needed (no live runs to track), OR
+ # - the gateway config came from the cache (stale specs imply potentially stale model
+ # identities; phoning home about pipe runs in that state would pollute metrics).
+ gateway_source_is_cached = gateway_config_source is not None and gateway_config_source.is_cached
+ is_pipelex_telemetry_enabled = is_pipelex_service_enabled and needs_inference and not gateway_source_is_cached
+ self.telemetry_manager = TelemetryFactory.make_telemetry_manager(
+ secrets_provider=secrets_provider,
+ integration_mode=integration_mode,
+ remote_config=remote_config,
+ is_pipelex_telemetry_enabled=is_pipelex_telemetry_enabled,
+ telemetry_config=telemetry_config,
+ injected_telemetry_manager=telemetry_manager,
+ )
+ self.telemetry_manager.setup(integration_mode=integration_mode)
+ self.runtime_hub.set_telemetry_manager(telemetry_manager=self.telemetry_manager)
+
+ # --- Tools ----------------------------------------------------------------------------
+
+ self.class_registry = class_registry or ClassRegistry()
+ self.kajson_manager = KajsonManager(class_registry=self.class_registry)
+
+ self.func_registry = func_registry or FuncRegistry()
+ self.runtime_hub.set_func_registry(func_registry=self.func_registry)
+ self.runtime_hub.set_secrets_provider(secrets_provider=secrets_provider)
+ # Storage is selected from the config-driven StorageProviderRegistry, built from the plugin
+ # registrar (constructed above, just before the telemetry factory). Its resolution and hub-set
+ # still happen later at the plugin-derived-registries block — after secrets is on the hub here,
+ # so the GCP factory's secret read works.
+
+ # Register stuff templates first (used by mermaid, reactflow, and stuff_viewer)
+ stuff_name, stuff_package, stuff_templates = STUFF_TEMPLATE_SET
+ TemplateLoader.register_set(
+ name=stuff_name,
+ package=stuff_package,
+ templates=stuff_templates,
+ )
+ reactflow_name, reactflow_package, reactflow_templates = REACTFLOW_TEMPLATE_SET
+ TemplateLoader.register_set(
+ name=reactflow_name,
+ package=reactflow_package,
+ templates=reactflow_templates,
+ )
+ mermaid_name, mermaid_package, mermaid_templates = MERMAID_TEMPLATE_SET
+ TemplateLoader.register_set(
+ name=mermaid_name,
+ package=mermaid_package,
+ templates=mermaid_templates,
+ )
+ TemplateLoader.load_all()
+
+ # --- AI Models and Inference Management ------------------------------------------------
+
+ self.sdk_client_manager.setup()
+
+ self.models_manager: ModelManagerAbstract = models_manager or ModelManager()
+ self.runtime_hub.set_models_manager(models_manager=self.models_manager)
+
+ try:
+ # NOTE: ``config_dir`` scopes the main TOML load only; the inference files (backends,
+ # routing profiles, model deck) still resolve through the layered ``config_manager.*``
+ # properties, because pinning them requires the path overrides that exist on the *concrete*
+ # ``ModelManager`` and not on ``ModelManagerAbstract`` — which is what this attribute is
+ # typed as, and which is a public injection point. Widening that interface is a decision of
+ # its own, so the gap is documented rather than half-closed:
+ # ``wip/boot-split/config-dir-does-not-scope-inference-paths.md``. The docstrings say exactly
+ # this; do not read ``config_dir`` as "only this directory is read" for inference.
+ self.models_manager.setup(
+ secrets_provider=secrets_provider,
+ gateway_config=gateway_config,
+ gateway_config_source=gateway_config_source,
+ needs_inference=needs_inference,
+ )
+ except RoutingProfileLibraryNotFoundError as routing_not_found_exc:
+ msg = self._get_config_file_not_found_error_msg(component_name="routing profile library")
+ raise PipelexSetupError(msg) from routing_not_found_exc
+ except InferenceBackendLibraryNotFoundError as backend_not_found_exc:
+ msg = self._get_config_file_not_found_error_msg(component_name="inference backend library")
+ raise PipelexSetupError(msg) from backend_not_found_exc
+ except ModelDeckNotFoundError as deck_not_found_exc:
+ msg = self._get_config_file_not_found_error_msg(component_name="model deck")
+ raise PipelexSetupError(msg) from deck_not_found_exc
+ except RoutingProfileDisabledBackendError as routing_profile_exc:
+ msg = f"Some backend(s) required for a routing profile is not enabled: {routing_profile_exc}"
+ raise PipelexSetupError(msg) from routing_profile_exc
+
+ except InferenceBackendLibraryValidationError as backend_validation_exc:
+ msg = self._get_validation_error_msg(component_name="inference backend library", validation_exc=backend_validation_exc)
+ raise PipelexSetupError(msg) from backend_validation_exc
+ except ModelDeckValidationError as deck_validation_exc:
+ msg = self._get_validation_error_msg(component_name="model deck", validation_exc=deck_validation_exc)
+ msg += "\n\nIf you added your own config files to the model deck then you'll have to change them manually."
+ raise PipelexSetupError(msg) from deck_validation_exc
+
+ except InferenceBackendCredentialsError as credentials_exc:
+ backend_name = credentials_exc.backend_name
+ var_name = credentials_exc.key_name
+ error_msg = BackendCredentialsErrorMsgFactory.make_one_variable_missing_error_msg(
+ secrets_provider=secrets_provider,
+ backend_name=backend_name,
+ var_name=var_name,
+ )
+ raise PipelexSetupError(error_msg) from credentials_exc
+
+ # --- Plugin-derived registries --------------------------------------------------------
+ # The plugin registrar was built earlier (with the boot-orchestrator gate checked and the
+ # config-selected secrets provider resolved) just before the telemetry factory. Turn its
+ # accumulated contributions into the hub registries here — after the gateway/model setup checks
+ # and before the hub setup points below — the family worker factories look their backends up on
+ # these at run time.
+ #
+ # The orchestrator and bundle-validator registries are interpreter-contributed and therefore
+ # empty on a runtime-only boot. That is fine and deliberate: nothing here resolves out of them,
+ # they are looked up at run time by the interpreter.
+ self.runtime_hub.set_inference_backend_registry(InferenceBackendRegistry(plugin_registrar.inference_backends))
+ self.runtime_hub.set_model_lister_registry(ModelListerRegistry(plugin_registrar.model_listers))
+ self.runtime_hub.set_orchestrator_registry(OrchestratorRegistry(plugin_registrar.orchestrators))
+ self.runtime_hub.set_bundle_validator_registry(BundleValidatorRegistry(plugin_registrar.bundle_validators))
+ storage_provider_registry = StorageProviderRegistry(plugin_registrar.storage_providers)
+ self.runtime_hub.set_storage_provider_registry(storage_provider_registry)
+ # Storage provider precedence: explicit setup() param > config-selected registry factory.
+ # The built-in StoragePlugin supplies every method, so there is no separate core default.
+ # Resolves here (after secrets is on the hub) so the GCP factory's secret read works.
+ if storage_provider is None:
+ storage_config = get_config().pipelex.storage_config
+ storage_provider = storage_provider_registry.get_required(method=storage_config.method)(storage_config)
+ self.runtime_hub.set_storage_provider(storage_provider)
+
+ # Keyless boot forces every run to DRY — consumed at PipeRunParamsFactory.make_run_params,
+ # the single writer of run_mode, covering every entry point; generator selection is
+ # backend-keyed unconditionally (eng review D4) — a keyless Temporal submitter must still
+ # dispatch activities and mock inside them, so `needs_inference` plays no role in picking the
+ # generator. Its other boot roles (gateway/model setup, credentials, telemetry) are unchanged.
+ self.runtime_hub.set_dry_run_forced(is_forced=not needs_inference)
+ # Injection precedence (codex C8): explicit setup() param > plugin slot-claim thunk > core default.
+ # A boot-orchestrator plugin (Temporal worker) claims the CONTENT_GENERATOR slot; its thunk runs
+ # only here, never at register, so booting a non-worker process imports no host-runtime SDK.
+ if content_generator is None:
+ content_generator = self._resolve_hub_slot(
+ slot=HubSlot.CONTENT_GENERATOR,
+ default=lambda: ContentGenerator(generated_content_factory=GeneratedContentFactory(storage_provider=storage_provider)),
+ )
+ self.runtime_hub.set_content_generator(content_generator)
+
+ self.inference_manager = inference_manager or InferenceManager()
+ self.runtime_hub.set_inference_manager(self.inference_manager)
+
+ # --- Libraries & Registries -------------------------------------------------------------
+
+ self.reporting_delegate = reporting_delegate or ReportingManager()
+ self.runtime_hub.set_report_delegate(self.reporting_delegate)
+ self.reporting_delegate.setup()
+
+ # Core's value model. The pipe kinds are the interpreter half's own registration
+ # (``PipeRegistryModels``); the two manifests are disjoint by construction and pinned as such
+ # by tests/unit/pipelex/test_registry_models_split.py, so which side registers first carries
+ # no meaning.
+ self.class_registry.register_classes(CoreRegistryModels.get_all_models())
+ if runtime_manager.is_unit_testing:
+ log.verbose("Registering test models for unit testing")
+ self.class_registry.register_classes(TestRegistryModels.get_all_models())
+
+ # --- Observers -------------------------------------------------------------------------
+ # Built from runtime-layer parts and held on the instance because the interpreter half's
+ # PipeRouter consumes it — one of the two seams that lets the interpreter constructions be a
+ # tail rather than an interleaving.
+
+ if not observers:
+ no_op_observer = ObserverNoOp()
+ observer_telemetry = ObserverTelemetry(telemetry_manager=self.telemetry_manager)
+ observers = {"noop": no_op_observer, "telemetry": observer_telemetry}
+ self.multi_observer = MultiObserver(observers=observers)
+
+ # --- Task Manager ----------------------------------------------------------------------
+ # The TASK_MANAGER slot is claimed only by a boot-orchestrator plugin running this process as
+ # its runtime (a Temporal worker). The thunk does the full wiring on the plugin's own hub and
+ # is torn down via a registered teardown callback (LIFO) — no core default, no explicit param.
+
+ task_manager_factory = plugin_registrar.slot_claims.get(HubSlot.TASK_MANAGER)
+ if task_manager_factory is not None:
+ task_manager_factory()
+
+ # --- Isolated-execution probe ----------------------------------------------------------
+ # Claimed only by a boot-orchestrator plugin whose runtime has a replay/activity split (e.g. a
+ # Temporal worker): the thunk resolves the ambient predicate that reports whether the current
+ # call runs inside an isolated sub-execution (an activity). ReportingManager consults it to
+ # route an activity-side usage emission to the per-process log instead of the workflow's
+ # registered buffer (audit H1). Unclaimed (any in-process boot), the fresh hub's default
+ # reports "never isolated", so no wiring is needed here.
+ isolated_execution_probe_factory = plugin_registrar.slot_claims.get(HubSlot.ISOLATED_EXECUTION_PROBE)
+ if isolated_execution_probe_factory is not None:
+ self.runtime_hub.set_isolated_execution_probe(isolated_execution_probe_factory())
+
+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} runtime setup done")
+
+ def _resolve_hub_slot(self, *, slot: HubSlot, default: Callable[[], _HubSlotImplT]) -> _HubSlotImplT:
+ """Resolve a process-global hub slot: a plugin's claimed thunk if present, else the core default.
+
+ Call sites apply explicit-injection precedence first (an explicit ``setup()`` param wins),
+ so this only arbitrates plugin-claim vs core-default. The claim is a thunk invoked here at the
+ boot apply-point — never during ``register`` — so a non-worker boot constructs no plugin impl.
+ The return type follows the core ``default`` (the claimed thunk is typed ``Any`` and adopts it).
+ """
+ if self._plugin_registrar is not None:
+ factory = self._plugin_registrar.slot_claims.get(slot)
+ if factory is not None:
+ return cast("_HubSlotImplT", factory())
+ return default()
+
+ def _teardown_plugin_callbacks(self) -> None:
+ """Run the plugin-contributed teardown callbacks, LIFO.
+
+ Its own phase because the ordering is load-bearing: these must run *before* the interpreter
+ half's ``pipeline_manager.teardown()`` so a worker's in-flight Temporal resources release
+ first. ``Pipelex.teardown`` sequences the two explicitly rather than relying on a template
+ hook, which is what keeps that order legible.
+
+ Names no integration: the callbacks were registered by whichever boot-orchestrator plugin
+ claimed the runtime.
+
+ **Best-effort, per callback.** Each is unbounded third-party code, and a failure in one must not
+ skip the rest: with two plugins registered, a raising first callback would otherwise leave the
+ second plugin's runtime live. Catching around the *loop* instead of inside it looks equivalent
+ and is not — the loop has already exited by then. Errors are logged rather than propagated,
+ because no caller of a teardown can act on "plugin B failed to release", whereas every caller is
+ harmed by B never being asked. Same shape as ``TelemetryManager.teardown``, which wraps each of
+ its own shutdown steps for the same reason.
+ """
+ if self._plugin_registrar is None:
+ return
+ for teardown_callback in reversed(self._plugin_registrar.teardown_callbacks):
+ try:
+ teardown_callback()
+ except Exception as teardown_exc: # noqa: BLE001
+ # (2) a plugin-registered callback is unbounded third-party code; its exception surface
+ # cannot be enumerated, and the remaining callbacks still have resources to release.
+ log.error(f"A plugin teardown callback failed and was skipped: {teardown_exc}")
+
+ def _teardown_runtime(self) -> None:
+ """Release what the runtime boot acquired, and the process-global *state* with it.
+
+ Not the hub *instances*: ``set_runtime_hub`` (and, on the interpreter half,
+ ``set_interpreter_hub``) overwrite a ``ClassVar`` with no reset counterpart, so after this runs
+ ``get_runtime_hub()`` still hands out the torn-down hub rather than raising "not initialized".
+ Harmless in practice — the next boot replaces it — but worth not claiming otherwise.
+
+ The releases in the ``finally`` are what make the process *re-bootable*, so they must not be
+ skipped by a raising step above them — and several of those steps are reached through an
+ injectable abstract type (``telemetry_manager``, ``reporting_delegate``, ``class_registry`` are
+ all public ``make()`` injection points), so the built-in implementations' "teardown never
+ raises" guarantee does not cover them. Unguarded, a raiser would strand this instance in
+ ``MetaSingleton.instances`` and leave the process **permanently** unbootable: the next ``make()``
+ dies on "already initialized", and ``teardown_if_needed`` cannot rescue it because it resolves
+ the same instance and re-enters the same raiser. ``finally`` and no ``except``: the failure still
+ propagates — a teardown that half-failed must not look successful.
+
+ What a raising step *does* still skip is the releases that follow it in the ``try`` — the ones
+ before it have already run. That set is now only the *dangling* half: an SDK client left open, a
+ reporting buffer unflushed, the previous boot's ``func_registry`` entries carried forward.
+ Everything that would instead **poison** the next boot moved into the ``finally``, so the two
+ release paths guarantee the same set. Closing the dangling half means collapsing this path and
+ ``_release_after_failed_boot`` into one list —
+ ``wip/boot-split/failed-boot-does-not-release-every-resource.md``.
+ """
+ try:
+ if self.telemetry_manager:
+ self.telemetry_manager.teardown()
+
+ # cogt
+ self.inference_manager.teardown()
+ if self.reporting_delegate:
+ self.reporting_delegate.teardown()
+ self.sdk_client_manager.teardown()
+
+ # tools
+ if self.class_registry:
+ self.class_registry.teardown()
+ func_registry.teardown()
+
+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} teardown done (except config & logs)")
+ finally:
+ # The runtime hub releases its process-global config. ``class_registry_scoping`` is reset here
+ # too: the interpreter half installs the resolver at boot, so releasing it belongs to whichever
+ # teardown runs — and on a runtime-only boot nothing installed it, where the reset is a no-op.
+ self.runtime_hub.reset_config()
+ class_registry_scoping.reset()
+ # The same three ``_release_after_failed_boot`` releases, deliberately kept identical: these are
+ # the ones that *poison* the next boot rather than merely dangle, so they must not sit above a
+ # step that can raise. ``KajsonManager`` is the sharp one, and it is easy to get backwards — it
+ # is a singleton, so a surviving manager makes the next boot's ``KajsonManager(class_registry=…)``
+ # hand back the old one and silently discard the fresh registry. ``get_class_registry()`` would
+ # then serve the previous boot's contents while the new boot's own registrations land somewhere
+ # nothing resolves to, and that boot would still report ``is_ready``. Leaving these in the
+ # ``try`` traded the old *loud* failure — "already initialized", because a raiser skipped the
+ # de-registration below as well — for precisely that silent one. All three are idempotent.
+ KajsonManager.teardown()
+ TemplateLoader.reset()
+ TemplateRegistry.clear()
+ # Clear the singleton instance from metaclass
+ if self.__class__ in MetaSingleton.instances:
+ del MetaSingleton.instances[self.__class__]
+
+ def teardown(self) -> None:
+ # ``try``/``finally`` and not a bare sequence, for the reason ``_release_after_failed_boot``
+ # spells out at its own call to this method: ``_teardown_plugin_callbacks`` swallows each
+ # callback's ordinary ``Exception``, but ``except Exception`` does not cover ``BaseException`` —
+ # a plugin callback that calls ``sys.exit()``, or a ``KeyboardInterrupt`` landing mid-teardown,
+ # would otherwise skip ``_teardown_runtime`` and leave this instance registered with no way back.
+ # No ``except``: the failure still propagates.
+ try:
+ self._teardown_plugin_callbacks()
+ finally:
+ self._teardown_runtime()
+
+ def _release_after_failed_boot(self) -> None:
+ """Release the process globals a partial boot acquired. Called from every ``make()``'s handler.
+
+ Covers a failure in ``setup()`` only. Both ``make()``s construct *outside* their ``try``, so a
+ raise inside ``__init__`` itself — ``log.configure`` hitting "LogConfig is already set" because
+ an embedding host configured pipelex logging first is the realistic one — never reaches this
+ handler, and leaves nothing in ``MetaSingleton`` for ``teardown_if_needed`` to find either.
+ Widening the ``try`` is not the fix: this method reads attributes ``__init__`` assigns, so it
+ would fault on a half-constructed instance and hide the original error.
+
+ A failed boot must release the process-global singletons it acquired, not just the singleton
+ registration — otherwise they leak and poison the next boot in the same process. ``setup()``
+ establishes these progressively (logging + hub config in ``__init__``, the ``KajsonManager``
+ class registry, the template registries) and may mutate config (e.g.
+ ``plugins.boot_orchestrator``); failing partway skips ``teardown()``, the normal release point.
+
+ It exists rather than calling ``teardown()`` because that path reads ``self.inference_manager``
+ (and, on the interpreter half, ``self.pipeline_manager``) unguarded, and both are assigned
+ partway through ``setup()``, so a half-built ``teardown()`` raises ``AttributeError``. That is
+ deliberate: guarding them would let a half-built teardown look successful.
+
+ It releases a **subset** of what ``teardown()`` does, and the subset is chosen: everything that
+ would otherwise *poison the next boot* is here — the hub config, the class-registry scoping, the
+ ``KajsonManager``, the template registries, the telemetry singleton and the ``MetaSingleton``
+ registration. What is deliberately absent is ``sdk_client_manager``, ``reporting_delegate``,
+ ``func_registry``, ``inference_manager`` and ``class_registry``. The first three leave resources
+ dangling rather than corrupting the next boot; the last two are the very attributes a partial
+ ``setup()`` may not have assigned, which is the reason this path exists at all. Adding any of
+ them here would widen a second hand-maintained copy of the teardown list that is bound to
+ drift from the real one. Collapsing the two paths is the right fix and is a lifecycle decision of
+ its own — ``wip/boot-split/failed-boot-does-not-release-every-resource.md``.
+
+ Without this, the next boot raises "LogConfig is already set" and serves a stale, half-populated
+ class registry (the ``KajsonManager`` singleton ignores a fresh registry once created).
+
+ The plugin teardown callbacks run **first**, mirroring the ordering of the normal ``teardown``,
+ and they are what makes this path release more than the process-global state released below: by
+ the time the interpreter tail runs, a boot-orchestrator plugin's ``TASK_MANAGER`` thunk has
+ already stood its runtime up, and only the plugin knows what that cost. Our Temporal plugin's
+ thunk registers its own process-global singletons and installs a sandbox predicate — it does not
+ yet start a poller, which is a property of that thunk today and not of the slot. A failure
+ anywhere after that thunk — the ``pipe_func_config.execution_mode`` lookup raising on an unregistered mode is the
+ most reachable one, since it is a plain config error — would otherwise leak it, because nothing
+ else on this path calls the callbacks. Not a hypothetical widened by the boot split: the thunk
+ used to run *after* the pipe-func executor resolution, ``pipeline_manager.setup()`` and the
+ pipe-class registration, and now runs before all three.
+ """
+ # ``try``/``finally`` and not a bare sequence: the callbacks are plugin-supplied, so this is the
+ # one line here that runs unbounded third-party code. If one raises, the releases below are what
+ # the next boot in this process depends on — skipping them leaves logging configured (every later
+ # boot dies on "LogConfig is already set"), the ``KajsonManager`` holding a half-populated class
+ # registry, and this instance still registered as the singleton. It would also replace the
+ # exception that actually killed the boot with a teardown error. The releases must therefore
+ # happen on both paths, which is precisely what ``finally`` is for.
+ # ``try``/``finally`` with no ``except``: the callbacks swallow their own per-callback failures
+ # (see ``_teardown_plugin_callbacks``), so nothing ordinary escapes here — but ``except Exception``
+ # there does not cover ``BaseException``, and this method's whole job is to leave the process
+ # re-bootable. The releases below must therefore run even on a ``KeyboardInterrupt`` mid-teardown.
+ # They are also what a raising callback must never skip: without them, logging stays configured
+ # (every later boot dies on "LogConfig is already set") and the ``KajsonManager`` keeps a
+ # half-populated class registry.
+ try:
+ self._teardown_plugin_callbacks()
+ finally:
+ # The telemetry manager is a *process-global singleton* (``ABCSingletonMeta``), not just an
+ # attribute, so discarding this instance does not release it: the next boot in the process
+ # would adopt the dead manager and export spans through it, because ``__init__`` never
+ # re-runs for an already-registered singleton. Its ``teardown`` both flushes and calls
+ # ``clear_instance()``.
+ #
+ # Isolated in its own ``try`` for the same reason the plugin callbacks are, and it is worth
+ # being precise about why the obvious argument fails: the *built-in* ``TelemetryManager`` is
+ # written to never raise ("telemetry teardown must never break the app"), but
+ # ``telemetry_manager`` is a public ``make()`` injection point typed only as
+ # ``TelemetryManagerAbstract``, so an injected implementation carries no such guarantee.
+ # Unisolated, a raising one would skip every release below it — leaving logging configured
+ # and the class registry half-populated — and replace the boot error on the way out. Reasoning
+ # from the concrete class's guarantees about a call made through an injectable abstract type
+ # is the mistake to avoid here.
+ if self.telemetry_manager is not None:
+ try:
+ self.telemetry_manager.teardown()
+ except Exception as telemetry_exc: # noqa: BLE001
+ # (2) an injected telemetry manager is unbounded code; its exception surface cannot
+ # be enumerated, and the releases below must happen regardless.
+ log.error(f"Telemetry teardown failed while releasing a failed boot: {telemetry_exc}")
+ self.runtime_hub.reset_config()
+ class_registry_scoping.reset()
+ KajsonManager.teardown()
+ TemplateLoader.reset()
+ TemplateRegistry.clear()
+ # Cleanup the singleton instance if setup fails to avoid "already initialized" errors.
+ if self.__class__ in MetaSingleton.instances:
+ del MetaSingleton.instances[self.__class__]
+
+ def __enter__(self) -> Self:
+ return self
+
+ def __exit__(self, exc_type: type[BaseException] | None, exc_val: BaseException | None, exc_tb: types.TracebackType | None) -> None:
+ self.teardown()
+
+ @classmethod
+ def make(
+ cls,
+ *,
+ integration_mode: IntegrationMode = IntegrationMode.PYTHON,
+ needs_inference: bool = True,
+ boot_orchestrator: str | None = None,
+ needs_model_specs: bool | None = None,
+ class_registry: ClassRegistryAbstract | None = None,
+ secrets_provider: SecretsProviderAbstract | None = None,
+ storage_provider: StorageProviderAbstract | None = None,
+ models_manager: ModelManagerAbstract | None = None,
+ inference_manager: InferenceManager | None = None,
+ content_generator: ContentGeneratorProtocol | None = None,
+ reporting_delegate: ReportingProtocol | None = None,
+ telemetry_config: TelemetryConfig | None = None,
+ telemetry_manager: TelemetryManagerAbstract | None = None,
+ observers: dict[str, ObserverProtocol] | None = None,
+ config_dir: Path | None = None,
+ config_overrides: dict[str, Any] | None = None,
+ ) -> Self:
+ """Create and initialize a runtime-layer singleton: inference, storage, models, telemetry.
+
+ Loads no method interpreter. Use ``pipelex.pipelex.Pipelex.make`` to boot the full stack.
+ See that method for the shared arguments — including ``config_dir``, which scopes the *main
+ TOML* load to a single directory (and, per that docstring, not the inference file paths); the
+ ones absent here (``pipe_func_executor``, ``pipeline_manager``, ``pipe_router``,
+ ``library_dirs``) are interpreter-layer injections.
+
+ Returns the initialized runtime boot instance, and raises ``PipelexSetupError`` if a boot
+ already holds the process globals or if setup fails.
+ """
+ # Checked here and not left to ``__init__``: on a second ``make()`` for an already-registered
+ # class, ``MetaSingleton`` hands back the live instance without re-running ``__init__``, so the
+ # only guard that can see this case is one that runs *before* the construction.
+ cls.raise_if_a_boot_already_holds_the_process_globals()
+
+ runtime_boot = cls(config_dir=config_dir, config_overrides=config_overrides)
+ try:
+ runtime_boot.setup(
+ integration_mode=integration_mode,
+ needs_inference=needs_inference,
+ boot_orchestrator=boot_orchestrator,
+ needs_model_specs=needs_model_specs,
+ class_registry=class_registry,
+ secrets_provider=secrets_provider,
+ storage_provider=storage_provider,
+ models_manager=models_manager,
+ inference_manager=inference_manager,
+ content_generator=content_generator,
+ reporting_delegate=reporting_delegate,
+ telemetry_config=telemetry_config,
+ telemetry_manager=telemetry_manager,
+ observers=observers,
+ )
+ if needs_inference:
+ runtime_boot.models_manager.validate_model_deck()
+ except BaseException:
+ runtime_boot._release_after_failed_boot()
+ raise
+ # Publish readiness only now: setup() AND the optional validate_model_deck() have both succeeded
+ # and the delete-on-failure handler above is behind us, so a reader can never adopt an instance
+ # that is about to be removed from the registry.
+ runtime_boot.is_ready = True
+ log.verbose(f"{PACKAGE_NAME} version {PACKAGE_VERSION} runtime ready")
+ return runtime_boot
+
+ @classmethod
+ def raise_if_a_boot_already_holds_the_process_globals(cls) -> None:
+ """Refuse to stand up a second boot. Called from ``__init__`` **and** from every ``make()``.
+
+ Both call sites are load-bearing, and neither subsumes the other:
+
+ - ``__init__`` covers a **direct construction** of a class that is not yet registered — the one
+ that would do real damage, since ``set_runtime_hub`` overwrites ``RuntimeHub._instance``
+ unconditionally and would orphan the live boot before failing at ``log.configure``.
+ - ``make()`` covers a **second call on an already-registered class**, which ``__init__`` cannot
+ see: ``MetaSingleton.__call__`` returns the registered instance without re-running
+ ``__init__``, so ``make()`` would sail past the guard and re-run the whole ``setup()`` on the
+ live boot — silently, returning the same object, and rebinding ``inference_manager`` and
+ ``reporting_delegate`` so the previous ones are dropped without ever being torn down.
+
+ Asked of the *base* class, which is what makes a bare ``RuntimeBoot`` and a full ``Pipelex``
+ exclude each other in both directions; the message names whichever one actually holds them.
+ """
+ existing_boot = RuntimeBoot.get_optional_instance()
+ if existing_boot is not None:
+ msg = f"{type(existing_boot).__name__} is already initialized"
+ raise PipelexSetupError(msg)
+
+ @classmethod
+ def get_optional_instance(cls) -> Self | None:
+ """The booted instance, resolved **by subclass** rather than by exact class.
+
+ A ``Pipelex`` *is* a ``RuntimeBoot``, and it owns the same process globals
+ (``set_runtime_hub``, ``KajsonManager``, ``log.configure`` are all once-per-process). Keying
+ on the exact class would let ``RuntimeBoot.is_fully_booted()`` answer ``False`` while a
+ ``Pipelex`` held the runtime hub — and would let ``RuntimeBoot.make()`` boot on top of a live
+ ``Pipelex``, which is the direction that actually breaks: the guard asks
+ ``RuntimeBoot.get_optional_instance()`` outright, and an exact-class lookup keyed on
+ ``RuntimeBoot`` cannot see an instance registered under ``Pipelex``. (The mirror direction stays
+ blocked either way, because a bare runtime boot *is* registered under ``RuntimeBoot``; what
+ would open it is the guard asking ``cls`` instead of the base.) In-tree precedent:
+ ``TelemetryManagerAbstract`` and ``GraphTracerManager`` resolve their singletons the same way.
+ """
+ return MetaSingleton.get_subclass_instance(cls)
+
+ @classmethod
+ def is_fully_booted(cls) -> bool:
+ """True only when a singleton exists AND has completed make() (setup + validation).
+
+ Distinct from ``get_optional_instance() is not None``: the metaclass registers the instance
+ before ``setup()`` configures the hub. Callers that must not touch a half-built instance
+ (``ensure_pipelex_booted``) gate on this.
+ """
+ instance = cls.get_optional_instance()
+ return instance is not None and instance.is_ready
+
+ @classmethod
+ def get_instance(cls) -> Self:
+ instance = cls.get_optional_instance()
+ if instance is None:
+ msg = f"{cls.__name__} is not initialized"
+ raise RuntimeError(msg)
+ return instance
+
+ @classmethod
+ def teardown_if_needed(cls) -> None:
+ """Teardown whichever boot holds the process globals, if any.
+
+ This is useful for cleanup in finally blocks where the instance
+ may or may not have been successfully created.
+
+ Resolved at the **base** class, symmetric with the exclusivity guard in ``__init__``. Asking
+ ``cls`` instead would deadlock the process: ``Pipelex.teardown_if_needed()`` would silently
+ no-op against a live bare ``RuntimeBoot`` (``get_subclass_instance(Pipelex)`` cannot see one)
+ while ``Pipelex(...)`` kept refusing because one exists — with no way out. A release must be
+ able to clear everything its matching guard refuses on.
+ """
+ instance = RuntimeBoot.get_optional_instance()
+ if instance is not None:
+ instance.teardown()
diff --git a/pipelex/runtime_hub.py b/pipelex/runtime_hub.py
index 8fffa04a5..81fb34f09 100644
--- a/pipelex/runtime_hub.py
+++ b/pipelex/runtime_hub.py
@@ -147,9 +147,10 @@ class RuntimeHub:
as the highest-priority override. Useful for tests that need
specific config without editing TOML files.
config_dir: Optional explicit config dir. When provided, project/global
- layering is bypassed and only this directory is read. Used by the
- doctor ``--global`` path so the hub reflects exactly the directory
- being reported on.
+ layering is bypassed and the load becomes package defaults + this
+ directory (the package layer always applies — it is what the TOML
+ overrides *are* overrides of). Used by the doctor ``--global`` path
+ so the hub reflects exactly the directory being reported on.
"""
config_dict = config_manager.load_config(extra_overrides=config_overrides, config_dir=config_dir)
self.set_config(config=config_cls.model_validate(config_dict))
diff --git a/pipelex/system/configuration/config_loader.py b/pipelex/system/configuration/config_loader.py
index af3ba71ed..29fb321ef 100644
--- a/pipelex/system/configuration/config_loader.py
+++ b/pipelex/system/configuration/config_loader.py
@@ -303,7 +303,9 @@ class ConfigLoader:
Args:
extra_overrides: Optional dict deep-merged on top as the final layer.
config_dir: Optional explicit config dir. When given, project/global layering
- is bypassed and only this directory is read.
+ is bypassed and the load becomes package defaults + this directory (the
+ package layer always applies — it is what the TOML overrides *are*
+ overrides of), plus the unit-testing layer below when it applies.
Returns:
dict[str, Any]: The merged configuration dictionary
diff --git a/pipelex/system/registries/singleton.py b/pipelex/system/registries/singleton.py
index 97f0856e1..83626d0cf 100644
--- a/pipelex/system/registries/singleton.py
+++ b/pipelex/system/registries/singleton.py
@@ -30,8 +30,13 @@ class MetaSingleton(type):
This is useful when the base class is abstract and only concrete
subclasses are instantiated.
+
+ Iterates a *copy*, as ``clear_subclass_instances`` above already does: ``instances`` is a
+ process-global dict that grows at run time (``GraphTracerManager`` registers itself during a
+ pipeline run), and this lookup sits on the boot hot path ``ensure_pipelex_booted`` reaches from
+ concurrent activities. A dict that gains a key mid-iteration raises ``RuntimeError``.
"""
- for subclass, instance in cls.instances.items():
+ for subclass, instance in list(cls.instances.items()):
if issubclass(subclass, base_cls) and isinstance(instance, base_cls):
return instance
return None
diff --git a/tests/conftest.py b/tests/conftest.py
index 70bd9347e..74e03bb59 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -94,7 +94,7 @@ def cache_configs_for_session(session_mocker: MockerFixture):
session_mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _cached_fetch_remote_config)
# Cache pipelex service config to avoid flaky tests from concurrent file reads
session_mocker.patch(
- "pipelex.pipelex.load_pipelex_service_config_if_exists",
+ "pipelex.runtime_boot.load_pipelex_service_config_if_exists",
_cached_load_pipelex_service_config,
)
# Skip expensive telemetry shutdown (OTel + PostHog flush) during tests
diff --git a/tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py b/tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py
index db1b72463..ac05ab834 100644
--- a/tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py
+++ b/tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py
@@ -30,7 +30,7 @@ if TYPE_CHECKING:
from pytest_mock import MockerFixture
-PIPELEX_MODULE = "pipelex.pipelex"
+RUNTIME_BOOT_MODULE = "pipelex.runtime_boot"
def _empty_gateway_remote_config_result(source: RemoteConfigSource) -> RemoteConfigResult:
@@ -80,7 +80,7 @@ class TestGatewayUnknownModel:
``GatewayUnknownModelError(source=FRESH)`` with the missing model name surfaced.
"""
Pipelex.teardown_if_needed()
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
mocker.patch(
"pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
new_callable=mocker.PropertyMock,
@@ -115,7 +115,7 @@ class TestGatewayUnknownModel:
fetching specs.
"""
Pipelex.teardown_if_needed()
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
mocker.patch(
"pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
new_callable=mocker.PropertyMock,
@@ -141,7 +141,7 @@ class TestGatewayUnknownModel:
message must point at ``pipelex init`` (while online) to refresh the cache.
"""
Pipelex.teardown_if_needed()
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
mocker.patch(
"pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
new_callable=mocker.PropertyMock,
diff --git a/tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py b/tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py
index 9c8eaf93a..2cffb8048 100644
--- a/tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py
+++ b/tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py
@@ -32,7 +32,7 @@ if TYPE_CHECKING:
from pytest_mock import MockerFixture
-PIPELEX_MODULE = "pipelex.pipelex"
+RUNTIME_BOOT_MODULE = "pipelex.runtime_boot"
# Capture the original classmethod at module import — the session-scoped fixture in the root
# conftest replaces ``RemoteConfigFetcher.fetch_remote_config`` with a wrapper that caches the
@@ -60,7 +60,7 @@ class TestOfflineBaseline:
"""
Pipelex.teardown_if_needed()
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=False)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=False)
mocker.patch(
"pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
new_callable=mocker.PropertyMock,
@@ -108,9 +108,9 @@ class TestOfflineBaseline:
agreement=PipelexServiceAgreement(terms_accepted=True),
onboarding=PipelexServiceOnboarding(inference_setup_completed=True),
)
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
mocker.patch(
- f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists",
+ f"{RUNTIME_BOOT_MODULE}.load_pipelex_service_config_if_exists",
return_value=service_config,
)
mocker.patch(
diff --git a/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py b/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py
index 5af6f1526..50cb15d7e 100644
--- a/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py
+++ b/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py
@@ -49,7 +49,7 @@ if TYPE_CHECKING:
from pytest_mock import MockerFixture
-PIPELEX_MODULE = "pipelex.pipelex"
+RUNTIME_BOOT_MODULE = "pipelex.runtime_boot"
# Capture the unpatched classmethod at module import — the session conftest replaces
# ``fetch_remote_config`` with a cached shim. We need the real fetch path to exercise the
@@ -104,9 +104,9 @@ class TestSetupWithCache:
session_result = RemoteConfigFetcher.fetch_remote_config() # patched to session cache
RemoteConfigCache.store(session_result.config.model_dump(mode="json"))
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
mocker.patch(
- f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists",
+ f"{RUNTIME_BOOT_MODULE}.load_pipelex_service_config_if_exists",
return_value=_accepted_service_config(),
)
mocker.patch(
@@ -145,9 +145,9 @@ class TestSetupWithCache:
"""
Pipelex.teardown_if_needed()
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
mocker.patch(
- f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists",
+ f"{RUNTIME_BOOT_MODULE}.load_pipelex_service_config_if_exists",
return_value=_accepted_service_config(),
)
mocker.patch(
@@ -187,9 +187,9 @@ class TestSetupWithCache:
cached_at=datetime.now(tz=UTC),
)
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
mocker.patch(
- f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists",
+ f"{RUNTIME_BOOT_MODULE}.load_pipelex_service_config_if_exists",
return_value=_accepted_service_config(),
)
mocker.patch(
@@ -218,7 +218,7 @@ class TestSetupWithCache:
"""
Pipelex.teardown_if_needed()
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=False)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=False)
mocker.patch(
"pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
new_callable=mocker.PropertyMock,
diff --git a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py
index 543a2ea99..c5ce1a8e6 100644
--- a/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py
+++ b/tests/integration/pipelex/system/test_hub_slot_injection_precedence.py
@@ -52,7 +52,7 @@ def _fake_registrar(mocker: MockerFixture) -> PluginRegistrar:
registrar = PluginRegistrar(config=cast("PipelexConfig", SimpleNamespace(temporal=SimpleNamespace(is_enabled=False))))
registrar.begin_plugin(name="pipe_func", origin=PluginOrigin.BUILTIN, targets_api=PLUGIN_API_VERSION)
PipeFuncPlugin().register(registrar)
- mocker.patch("pipelex.pipelex.build_registrar", return_value=registrar)
+ mocker.patch("pipelex.runtime_boot.build_registrar", return_value=registrar)
return registrar
@@ -129,3 +129,67 @@ class TestHubSlotInjectionPrecedence:
Pipelex.teardown_if_needed()
assert order == ["third", "second", "first"]
+
+ def test_one_raising_teardown_callback_does_not_skip_the_others(self, mocker: MockerFixture) -> None:
+ """Teardown is best-effort *per callback*, so a bad plugin cannot strand a good one's resources.
+
+ Catching around the loop rather than inside it looks equivalent and is not: by the time an
+ exception reaches the caller the loop has already exited, so every remaining callback is
+ skipped — and with two plugins registered, the second one's runtime stays live. The raiser is
+ registered *last* so LIFO reaches it *first*, which is the ordering that strands the others.
+ """
+ order: list[str] = []
+ registrar = _fake_registrar(mocker)
+
+ def _boom() -> None:
+ order.append("raiser")
+ msg = "plugin teardown exploded"
+ raise RuntimeError(msg)
+
+ registrar.teardown_callbacks.extend([lambda: order.append("survivor"), _boom])
+
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ storage_provider=InMemoryStorageProvider(),
+ secrets_provider=EnvSecretsProvider(),
+ )
+ # Must not propagate: no caller of a teardown can act on "plugin B failed to release".
+ Pipelex.teardown_if_needed()
+
+ assert order == ["raiser", "survivor"], "a raising callback skipped the remaining teardown callbacks"
+
+ def test_a_failed_interpreter_tail_still_runs_plugin_teardown_callbacks(self, mocker: MockerFixture) -> None:
+ """A boot that dies after the TASK_MANAGER thunk must not leak the runtime it started.
+
+ The thunk starts a live plugin runtime — for a Temporal worker, threads and a client
+ connection — and it runs in the runtime half of the boot, so every interpreter-tail step
+ happens after it. `make()` handles a failure with `_release_after_failed_boot()`, which
+ releases process-global *state*; without the plugin callbacks it would leave the started
+ runtime running, and nothing else on that path calls them.
+
+ The boot split is what made this reachable in more than one place: the thunk used to run
+ *after* the pipe-func executor resolution, `pipeline_manager.setup()` and the pipe-class
+ registration, and now runs before all three. An injected `PipelineManager` whose `setup()`
+ raises stands in for any of them.
+ """
+ order: list[str] = []
+ registrar = _fake_registrar(mocker)
+ registrar.slot_claims[HubSlot.TASK_MANAGER] = lambda: order.append("task_manager_started")
+ registrar.teardown_callbacks.append(lambda: order.append("plugin_torn_down"))
+
+ exploding_pipeline_manager = mocker.Mock()
+ exploding_pipeline_manager.setup.side_effect = RuntimeError("interpreter tail failed")
+
+ with pytest.raises(RuntimeError, match="interpreter tail failed"):
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ pipeline_manager=exploding_pipeline_manager,
+ storage_provider=InMemoryStorageProvider(),
+ secrets_provider=EnvSecretsProvider(),
+ )
+
+ # Asserting the start too, so this cannot pass by the thunk simply never having run —
+ # which would make the leak window imaginary and the test vacuous.
+ assert order == ["task_manager_started", "plugin_torn_down"]
diff --git a/tests/unit/pipelex/cli/dev/test_hub_layering_guard.py b/tests/unit/pipelex/cli/dev/test_hub_layering_guard.py
index 3de9c6513..6a6c980ed 100644
--- a/tests/unit/pipelex/cli/dev/test_hub_layering_guard.py
+++ b/tests/unit/pipelex/cli/dev/test_hub_layering_guard.py
@@ -147,6 +147,34 @@ class TestHubLayeringGuard:
"whatever made it dirty to the layer it belongs to."
)
+ def test_the_boot_split_left_the_runtime_half_declared(self) -> None:
+ """The runtime composition root must stay declared — and its interpreter half must not be.
+
+ `pipelex.py` was carved in half the way the plugin manifests were: the runtime boot became
+ `pipelex/runtime_boot.py`, and `Pipelex` became the interpreter boot that subclasses it. Only
+ the runtime half is declarable, and declaring it is the entire reason the rule reaches it:
+ both halves were undeclared before the split, which made the composition root — the one module
+ that constructs *both* layers — the single largest unpoliced surface in the tree.
+
+ The negative half of the assertion matters just as much. `pipelex.pipelex` imports
+ `interpreter_hub` by construction, so declaring it would make the layer rule fail by design;
+ a well-meaning "the boot should be runtime-layer, surely" edit is a plausible future mistake
+ and this is what catches it.
+
+ Membership is the whole assertion, for the same reason as the two tests above: `is_runtime_layer`
+ matches by dotted prefix, so presence is the one bit that matters.
+ """
+ assert "pipelex.runtime_boot" in RUNTIME_LAYER_PACKAGES, (
+ "pipelex.runtime_boot is the runtime layer's composition root and is not declared "
+ "runtime-layer. Undeclared means unchecked, not flagged — restore the entry, or move "
+ "whatever made it dirty to the layer it belongs to."
+ )
+ assert "pipelex.pipelex" not in RUNTIME_LAYER_PACKAGES, (
+ "pipelex.pipelex is the *interpreter* half of the boot — it installs the InterpreterHub "
+ "and constructs the method machinery — so declaring it runtime-layer makes the layer rule "
+ "fail by design. Declare pipelex.runtime_boot instead; that is the half that stays clean."
+ )
+
def test_runtime_layer_may_import_runtime_hub(self) -> None:
"""The permitted direction is never flagged — the runtime layer lives on `runtime_hub`."""
violations = _violate(
diff --git a/tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py b/tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py
index 9b8ab2769..cd607f825 100644
--- a/tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py
+++ b/tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py
@@ -24,7 +24,7 @@ from pipelex.system.runtime import IntegrationMode
if TYPE_CHECKING:
from pytest_mock import MockerFixture
-PIPELEX_MODULE = "pipelex.pipelex"
+RUNTIME_BOOT_MODULE = "pipelex.runtime_boot"
class TestGatewayTermsCheck:
@@ -33,8 +33,8 @@ class TestGatewayTermsCheck:
@pytest.fixture
def _gateway_enabled_no_config(self, mocker: MockerFixture) -> None:
"""Configure mocks: gateway is enabled but no service config exists (first run)."""
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
- mocker.patch(f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists", return_value=None)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.load_pipelex_service_config_if_exists", return_value=None)
@pytest.fixture
def _gateway_enabled_setup_done_terms_not_accepted(self, mocker: MockerFixture) -> None:
@@ -43,8 +43,8 @@ class TestGatewayTermsCheck:
agreement=PipelexServiceAgreement(terms_accepted=False),
onboarding=PipelexServiceOnboarding(inference_setup_completed=True),
)
- mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
- mocker.patch(f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists", return_value=config)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.load_pipelex_service_config_if_exists", return_value=config)
@pytest.mark.usefixtures("_gateway_enabled_no_config")
def test_first_run_raises_inference_setup_required(self) -> None:
@@ -86,7 +86,7 @@ class TestGatewayTermsCheck:
mock_result = mocker.MagicMock()
mock_result.config = mock_remote_config
mock_result.source = "fresh"
- mocker.patch(f"{PIPELEX_MODULE}.RemoteConfigFetcher.fetch_remote_config", return_value=mock_result)
+ mocker.patch(f"{RUNTIME_BOOT_MODULE}.RemoteConfigFetcher.fetch_remote_config", return_value=mock_result)
pipelex_instance = Pipelex.__new__(Pipelex)
diff --git a/tests/unit/pipelex/test_runtime_boot_closure.py b/tests/unit/pipelex/test_runtime_boot_closure.py
new file mode 100644
index 000000000..eab14441c
--- /dev/null
+++ b/tests/unit/pipelex/test_runtime_boot_closure.py
@@ -0,0 +1,140 @@
+"""Booting the runtime layer loads zero interpreter modules — the boot-time half of the property.
+
+`pipelex-dev check-hub-layering` proves no runtime-layer *module* imports `interpreter_hub`, and
+`test_runtime_layer_import_closure.py` proves the declared runtime-layer entry points *import* clean.
+Neither says anything about *booting*: until `RuntimeBoot` existed, the only composition root in the
+tree built an `InterpreterHub`, a `LibraryManager`, a `PipelineManager`, a `PipeRouter` and a
+`PipeRun` whether the caller would ever load a method or not.
+
+This runs in a **subprocess**, and that is not a stylistic choice. Both hubs are sticky
+class-attribute singletons (`RuntimeHub._instance` / `InterpreterHub._instance`) that `teardown`
+deliberately does not clear — so once anything in this process has booted a `Pipelex`, an in-process
+`InterpreterHub.get_optional_instance()` answers with the stale hub forever and the assertion would
+pass vacuously. A fresh interpreter is the only place the question is answerable.
+
+It is also strictly stronger than the import-closure entry point it complements: that one *imports*
+`pipelex.runtime_boot`, this one *runs* `RuntimeBoot.make()`. A runtime-only boot builds its plugin
+registrar from `RUNTIME_BUILTIN_PLUGINS` alone, so `OrchestratorRegistry`, `BundleValidatorRegistry`
+and the PipeFunc executor modes are all empty on it — an import-time check cannot notice a boot step
+that tries to resolve out of one of them.
+"""
+
+import subprocess # noqa: S404
+import sys
+import textwrap
+
+from tests.unit.pipelex.test_runtime_layer_import_closure import INTERPRETER_PACKAGES
+
+#: Wall-clock bound on the booted-runtime subprocess, matching the import-closure harness: a boot
+#: that deadlocks must present as a failure, not as a hung suite
+#: (`docs/agents/debugging-hanging-pytest-runs.md`).
+SUBPROCESS_TIMEOUT_SECONDS = 300
+
+#: Boot the runtime layer in a fresh interpreter, then answer the three questions this module exists
+#: to ask. `needs_inference=False` keeps it offline: no gateway terms gate, no model-deck validation.
+#: The `sys.modules` sweep runs *before* `pipelex.interpreter_hub` is imported for the hub assertion,
+#: so importing it to ask the question cannot be what makes the answer wrong.
+_BOOTED_RUNTIME_SCRIPT = textwrap.dedent(
+ """
+ import sys
+
+ from pipelex.runtime_boot import RuntimeBoot
+ from pipelex.system.runtime import IntegrationMode
+
+ interpreter_packages = frozenset(sys.argv[1:])
+ # An empty set would make the sweep below flag nothing and the test pass vacuously — the same
+ # guard the import-closure harness carries, for the same reason.
+ if not interpreter_packages:
+ print("no interpreter packages passed — the sweep would match nothing")
+ raise SystemExit(2)
+
+ RuntimeBoot.make(integration_mode=IntegrationMode.PYTEST, needs_inference=False)
+
+ offenders = sorted(
+ name
+ for name in sys.modules
+ if name.startswith("pipelex.") and name.split(".")[1] in interpreter_packages
+ )
+ if offenders:
+ print(f"booting RuntimeBoot loaded {len(offenders)} interpreter module(s): {offenders}")
+ raise SystemExit(1)
+ if "pipelex.interpreter_hub" in sys.modules:
+ print("booting RuntimeBoot loaded pipelex.interpreter_hub")
+ raise SystemExit(1)
+
+ # Now it is safe to import the interpreter hub to ask whether the boot installed one.
+ from pipelex.interpreter_hub import InterpreterHub
+ from pipelex.runtime_hub import RuntimeHub
+ from pipelex.system.registries.class_registry_access import class_registry_scoping
+
+ if RuntimeHub.get_optional_instance() is None:
+ print("the runtime boot did not install a RuntimeHub")
+ raise SystemExit(1)
+ if InterpreterHub.get_optional_instance() is not None:
+ print("the runtime boot installed an InterpreterHub")
+ raise SystemExit(1)
+ # The documented degradation: with no InterpreterHub installed, class-registry scoping stays at
+ # its unscoped default rather than raising (docs/contribute/hub-layering.md).
+ if class_registry_scoping.resolve() is not None:
+ print("class_registry_scoping resolved a scoped registry on a runtime-only boot")
+ raise SystemExit(1)
+
+ print("runtime boot OK")
+ """
+)
+
+
+#: The negative control, and the reason it is needed: the sweep below lives inside a `textwrap.dedent`
+#: string, so nothing type-checks or lints the *logic* in it. A broken predicate would make the real
+#: case pass **forever** while flagging nothing — verified, not theorised: changing the sweep's
+#: `name.split(".")[1]` to `[0]` leaves the real case green (it matches nothing, so it finds nothing)
+#: and this control is the only thing that goes red. That asymmetry is the point — the real case cannot
+#: detect its own blindness, so deleting the control would make every other case in this module
+#: unfalsifiable. The `interpreter_hub in sys.modules` check is not a backstop either, since a runtime
+#: module importing `pipe_operators` directly never touches the hub — which is the whole case this test
+#: exists for.
+#:
+#: `cogt` is the control package: the runtime boot loads it by definition, so calling it an
+#: "interpreter package" must come back a failure, *reported as offending modules*. The sibling
+#: import-closure module carries the same control for the same reason (its `DIRTY_ENTRY_POINT`).
+CONTROL_PACKAGE_THE_RUNTIME_ALWAYS_LOADS = "cogt"
+
+
+def _run_booted_runtime(*, interpreter_packages: "tuple[str, ...]") -> subprocess.CompletedProcess[str]:
+ """Boot the runtime layer in a fresh interpreter and return the sweep's verdict."""
+ try:
+ return subprocess.run( # noqa: S603
+ [sys.executable, "-c", _BOOTED_RUNTIME_SCRIPT, *interpreter_packages],
+ capture_output=True,
+ text=True,
+ check=False,
+ timeout=SUBPROCESS_TIMEOUT_SECONDS,
+ )
+ except subprocess.TimeoutExpired as exc:
+ message = f"the booted-runtime subprocess did not finish within {SUBPROCESS_TIMEOUT_SECONDS}s"
+ raise AssertionError(message) from exc
+
+
+class TestBootedRuntimeLayer:
+ def test_the_sweep_still_detects_a_package_the_runtime_boot_really_loads(self) -> None:
+ """The control: same script, opposite verdict. Without this, every other case is vacuous."""
+ result = _run_booted_runtime(interpreter_packages=(CONTROL_PACKAGE_THE_RUNTIME_ALWAYS_LOADS,))
+
+ assert result.returncode == 1, (
+ f"treating {CONTROL_PACKAGE_THE_RUNTIME_ALWAYS_LOADS!r} as an interpreter package must fail — "
+ "the runtime boot loads it by definition. This passing means the sweep has stopped sweeping, "
+ "and the real case below now proves nothing.\n"
+ f"stdout={result.stdout}\nstderr={result.stderr}"
+ )
+ assert "interpreter module(s)" in result.stdout
+ assert "runtime boot OK" not in result.stdout
+
+ def test_booting_the_runtime_layer_loads_no_interpreter_module_and_installs_no_interpreter_hub(self) -> None:
+ result = _run_booted_runtime(interpreter_packages=INTERPRETER_PACKAGES)
+ assert result.returncode == 0, (
+ "booting the runtime layer must load zero interpreter modules and install no InterpreterHub.\n"
+ "This is the boot-time half of the hub-layering property — see docs/contribute/hub-layering.md "
+ "for how to find the shortest import path to an offender.\n"
+ f"stdout={result.stdout}\nstderr={result.stderr}"
+ )
+ assert "runtime boot OK" in result.stdout
diff --git a/tests/unit/pipelex/test_runtime_boot_config_dir.py b/tests/unit/pipelex/test_runtime_boot_config_dir.py
new file mode 100644
index 000000000..b1f82dd15
--- /dev/null
+++ b/tests/unit/pipelex/test_runtime_boot_config_dir.py
@@ -0,0 +1,53 @@
+"""`config_dir` at boot must actually scope the main config load.
+
+It scoped nothing at all at first: the parameter was stored as `self.config_dir_path` and never read
+again, while `setup_config` was called without a `config_dir`, so an embedder passing an explicit
+config dir got the ordinary project/global layering and no error. This is the test that would have
+caught it.
+
+A scoped load is *package defaults + this directory*, so one overridden leaf is enough to tell the
+two apart without writing a whole valid config tree.
+
+Scope note: `config_dir` scopes the main TOML load only — the inference file paths still resolve
+through the layered `config_manager.*` properties, because pinning them needs overrides that live on
+the concrete `ModelManager` rather than on the `ModelManagerAbstract` this boot is typed against. That
+gap is deliberate and documented (`wip/boot-split/config-dir-does-not-scope-inference-paths.md`), which is
+why this module asserts the main-config leaf and nothing about backends or decks.
+"""
+
+from pathlib import Path
+
+from pipelex.config import get_config
+from pipelex.pipelex import Pipelex
+from pipelex.runtime_boot import RuntimeBoot
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+
+
+def _test_integration_mode() -> IntegrationMode:
+ """The boot mode the session conftest uses, so a re-boot here matches the one it replaces."""
+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST
+
+
+class TestConfigDirReachesTheConfigLoader:
+ def test_an_explicit_config_dir_is_what_gets_loaded(self, tmp_path: Path) -> None:
+ # WARNING is quieter than the INFO default, never noisier — this boot configures logging.
+ (tmp_path / "pipelex.toml").write_text('[pipelex.log_config]\ndefault_log_level = "WARNING"\n')
+
+ Pipelex.teardown_if_needed()
+ try:
+ # Through `make()`, not a bare construct-then-setup: `make()` is the only entry point that
+ # releases the process globals when a boot dies partway (`_release_after_failed_boot`), and
+ # a hand-rolled `teardown()` on a half-built instance raises `AttributeError` instead —
+ # masking the real error and leaving the singleton registered for the rest of the worker.
+ runtime_boot = RuntimeBoot.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ config_dir=tmp_path,
+ )
+ try:
+ assert get_config().pipelex.log_config.default_log_level == "WARNING"
+ finally:
+ runtime_boot.teardown()
+ finally:
+ # Restore what the module fixture set up, so its teardown is sane.
+ Pipelex.make(integration_mode=_test_integration_mode())
diff --git a/tests/unit/pipelex/test_runtime_boot_exclusivity.py b/tests/unit/pipelex/test_runtime_boot_exclusivity.py
new file mode 100644
index 000000000..6a4ad9dfa
--- /dev/null
+++ b/tests/unit/pipelex/test_runtime_boot_exclusivity.py
@@ -0,0 +1,104 @@
+"""A runtime boot and a full boot own the same process globals, so they cannot coexist.
+
+`set_runtime_hub` and `KajsonManager` overwrite unconditionally — only `log.configure` refuses a second
+call — so a second boot on top of the first would silently replace them and serve a half-populated
+class registry, which is the failure mode the "already initialized" guard exists to prevent. The guard resolves the singleton *by subclass*, so it
+sees a bare `RuntimeBoot` too; keyed on the exact class instead, `Pipelex.make()` would happily boot
+on top of one, because `get_subclass_instance(Pipelex)` cannot see a `RuntimeBoot`.
+"""
+
+import pytest
+
+from pipelex.base_exceptions import PipelexSetupError
+from pipelex.pipelex import Pipelex
+from pipelex.runtime_boot import RuntimeBoot
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+
+
+def _test_integration_mode() -> IntegrationMode:
+ """The boot mode the session conftest uses, so a re-boot here matches the one it replaces."""
+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST
+
+
+class TestRuntimeBootAndPipelexAreOneProcessGlobal:
+ def test_a_full_boot_refuses_on_top_of_a_runtime_boot_and_succeeds_once_it_is_torn_down(self) -> None:
+ # The module-scoped conftest fixture booted a Pipelex; release it so this test owns the process.
+ Pipelex.teardown_if_needed()
+ try:
+ runtime_boot = RuntimeBoot.make(integration_mode=_test_integration_mode(), needs_inference=False)
+ try:
+ # The message must name what actually holds the globals. "Pipelex is already
+ # initialized" would be a lie here, and an embedder that never touched Pipelex
+ # deserves to be told what did — so the wording is asserted, not just the type.
+ with pytest.raises(PipelexSetupError, match="RuntimeBoot is already initialized"):
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)
+ finally:
+ runtime_boot.teardown()
+
+ # Same call, now that the runtime boot has released the globals.
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False).teardown()
+ finally:
+ # Restore what the fixture set up, so the rest of this module (and its teardown) is sane.
+ Pipelex.make(integration_mode=_test_integration_mode())
+
+ def test_a_runtime_boot_refuses_on_top_of_a_full_boot(self) -> None:
+ """The exclusion runs both ways, and here the booted class is the *sub*class."""
+ with pytest.raises(PipelexSetupError, match="Pipelex is already initialized"):
+ RuntimeBoot.make(integration_mode=_test_integration_mode(), needs_inference=False)
+
+ def test_a_second_make_of_the_already_booted_class_refuses_too(self) -> None:
+ """The same-class case, which the ``__init__`` guard alone cannot see.
+
+ ``MetaSingleton.__call__`` hands back the registered instance without re-running ``__init__``,
+ so a guard living only there is unreachable on a second ``make()`` for the *same* class — and
+ ``make()`` would then re-run the whole ``setup()`` on the live boot, silently, returning the
+ same object. The two cross-class tests above pass either way (a subclass is a different
+ registry key, so ``__init__`` does run), which is exactly why this one is needed: it is the
+ case both of them miss.
+ """
+ Pipelex.teardown_if_needed()
+ try:
+ runtime_boot = RuntimeBoot.make(integration_mode=_test_integration_mode(), needs_inference=False)
+ try:
+ with pytest.raises(PipelexSetupError, match="RuntimeBoot is already initialized"):
+ RuntimeBoot.make(integration_mode=_test_integration_mode(), needs_inference=False)
+ finally:
+ runtime_boot.teardown()
+
+ # And the subclass, which has its own guard for the same reason: the two must not drift,
+ # and "``__init__`` already covers it" is exactly the reasoning that left the base class
+ # unguarded.
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)
+ with pytest.raises(PipelexSetupError, match="Pipelex is already initialized"):
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)
+ finally:
+ Pipelex.teardown_if_needed()
+ Pipelex.make(integration_mode=_test_integration_mode())
+
+ def test_teardown_if_needed_releases_a_bare_runtime_boot_through_the_subclass(self) -> None:
+ """``Pipelex.teardown_if_needed()`` must release a live bare ``RuntimeBoot``, not no-op on it.
+
+ The one asymmetry with no way out. ``teardown_if_needed`` resolves at the *base* class; asking
+ ``cls`` instead would make this call silently no-op (``get_subclass_instance(Pipelex)`` cannot
+ see a ``RuntimeBoot``) while every ``Pipelex(...)`` kept refusing because one exists. Nothing
+ else in the suite exercises the classmethod against a bare runtime boot — the other tests
+ release theirs through the instance — so that "tidy-up" would stay green.
+ """
+ Pipelex.teardown_if_needed()
+ try:
+ RuntimeBoot.make(integration_mode=_test_integration_mode(), needs_inference=False)
+
+ Pipelex.teardown_if_needed()
+
+ assert RuntimeBoot.get_optional_instance() is None, (
+ "Pipelex.teardown_if_needed() no-oped on a live bare RuntimeBoot — the process is now unbootable and no teardown call can release it"
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+ Pipelex.make(integration_mode=_test_integration_mode())
+
+ def test_a_runtime_boot_is_visible_through_the_subclass_resolved_accessors(self) -> None:
+ """`RuntimeBoot.is_fully_booted()` must answer for a `Pipelex` too — it is one."""
+ assert Pipelex.is_fully_booted()
+ assert RuntimeBoot.is_fully_booted()
+ assert RuntimeBoot.get_optional_instance() is Pipelex.get_instance()
diff --git a/tests/unit/pipelex/test_runtime_boot_failed_boot_release.py b/tests/unit/pipelex/test_runtime_boot_failed_boot_release.py
new file mode 100644
index 000000000..2267317ea
--- /dev/null
+++ b/tests/unit/pipelex/test_runtime_boot_failed_boot_release.py
@@ -0,0 +1,115 @@
+"""A failed boot must not leave a process-global singleton behind for the next one to adopt.
+
+`_release_after_failed_boot` releases the process globals a partial boot acquired. The telemetry
+manager is the subtle member of that set: it is an `ABCSingletonMeta` singleton, not just an attribute
+on the boot instance, so discarding the instance does **not** release it. Because `MetaSingleton`
+short-circuits on an already-registered class, the next boot in the process gets the *same object* and
+its `__init__` never re-runs — so `TelemetryManagerAbstract.get_instance()` keeps resolving to a
+manager whose tracer provider has been shut down.
+
+That is reachable in production rather than theoretical: `ensure_pipelex_booted` is a per-call lazy
+boot on the runtime-bridge hot path, and the commonest boot failure — `models_manager.setup()` raising
+on a missing model deck or missing credentials — fires *after* the telemetry factory has already
+constructed and set up the manager.
+"""
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.pipelex import Pipelex
+from pipelex.runtime_boot import RuntimeBoot
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
+
+
+def _test_integration_mode() -> IntegrationMode:
+ """The boot mode the session conftest uses, so a re-boot here matches the one it replaces."""
+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST
+
+
+class TestFailedBootReleasesTheTelemetrySingleton:
+ def test_a_boot_failing_after_telemetry_setup_leaves_no_telemetry_singleton(
+ self,
+ mocker: MockerFixture,
+ ) -> None:
+ # An injected models manager whose setup raises reproduces the real shape: telemetry is already
+ # live by then, since the factory runs earlier in the runtime half.
+ exploding_models_manager = mocker.Mock()
+ exploding_models_manager.setup.side_effect = RuntimeError("model deck missing")
+
+ Pipelex.teardown_if_needed()
+ try:
+ assert TelemetryManagerAbstract.get_instance() is None, "the released boot left a telemetry singleton"
+
+ with pytest.raises(RuntimeError, match="model deck missing"):
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ models_manager=exploding_models_manager,
+ )
+
+ # The whole point: the failed boot got as far as standing telemetry up, and released it.
+ # Without that release the next boot adopts this object instead of building a fresh one.
+ assert TelemetryManagerAbstract.get_instance() is None, (
+ "a failed boot left the telemetry singleton registered — the next boot in this process "
+ "would adopt the dead manager, since MetaSingleton never re-runs __init__ for an "
+ "already-registered class"
+ )
+ finally:
+ # Restore what the module fixture set up, so its teardown is sane.
+ Pipelex.teardown_if_needed()
+ Pipelex.make(integration_mode=_test_integration_mode())
+
+ def test_an_injected_telemetry_manager_that_raises_does_not_abort_the_other_releases(
+ self,
+ mocker: MockerFixture,
+ ) -> None:
+ """The releases after the telemetry call must still run, and the boot error must still surface.
+
+ `self.telemetry_manager` is whatever the telemetry factory produced, and that includes a
+ caller-supplied implementation: `telemetry_manager` is a public `make()` injection point typed
+ only as `TelemetryManagerAbstract`, adopted whenever the integration mode and PostHog mode allow
+ custom telemetry. The *built-in* manager is written never to raise, so reasoning from the
+ concrete class would say this call is safe — and that reasoning does not survive an injected
+ implementation. If a raising teardown aborted the release suite, `log.configure` and the
+ `KajsonManager` registry would survive and every later boot in the process would die on
+ "LogConfig is already set".
+
+ The proof that the suite completed is the singleton de-registration, which is its **last**
+ statement — so its absence means every release before it ran too. Asserting that rather than
+ re-booting keeps the patched factory from leaking into a second boot.
+
+ The factory is patched rather than the parameter passed, because the injection is only adopted
+ in some telemetry modes and this test is about the state that results, not about how it arises.
+ """
+ exploding_telemetry = mocker.Mock()
+ exploding_telemetry.teardown.side_effect = RuntimeError("telemetry teardown exploded")
+ mocker.patch(
+ "pipelex.runtime_boot.TelemetryFactory.make_telemetry_manager",
+ return_value=exploding_telemetry,
+ )
+ exploding_models_manager = mocker.Mock()
+ exploding_models_manager.setup.side_effect = RuntimeError("model deck missing")
+
+ Pipelex.teardown_if_needed()
+ try:
+ # The *boot* error surfaces, not the teardown error that happened while handling it.
+ with pytest.raises(RuntimeError, match="model deck missing"):
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ models_manager=exploding_models_manager,
+ )
+
+ exploding_telemetry.teardown.assert_called_once()
+ assert RuntimeBoot.get_optional_instance() is None, (
+ "the raising telemetry teardown aborted the release suite — the boot is still registered, "
+ "so log.configure and the KajsonManager registry survived and the next boot in this "
+ 'process would die on "LogConfig is already set"'
+ )
+ finally:
+ # Stop the patched factory before re-booting, or the restore below adopts the exploding
+ # mock and its (unguarded, normal-path) teardown raises out of the fixture.
+ mocker.stopall()
+ Pipelex.teardown_if_needed()
+ Pipelex.make(integration_mode=_test_integration_mode())
diff --git a/tests/unit/pipelex/test_runtime_boot_teardown_resilience.py b/tests/unit/pipelex/test_runtime_boot_teardown_resilience.py
new file mode 100644
index 000000000..3e4d1a9c7
--- /dev/null
+++ b/tests/unit/pipelex/test_runtime_boot_teardown_resilience.py
@@ -0,0 +1,105 @@
+"""A teardown that fails partway must still leave the process re-bootable.
+
+The releases at the end of ``RuntimeBoot._teardown_runtime`` — the hub config, the class-registry
+scoping, and the ``MetaSingleton`` de-registration — are what a *later* boot in the same process
+depends on. If a raising step above them skipped them, the process would be wedged **permanently**:
+the next ``make()`` dies on "already initialized", and ``teardown_if_needed`` cannot rescue it,
+because it resolves the very same instance and re-enters the very same raiser.
+
+Reachable through injection rather than through a built-in defect, which is the point: the built-in
+``TelemetryManager`` and ``PipelineManager`` are written not to raise, but both are public ``make()``
+injection points typed as abstracts, so that guarantee belongs to the concrete classes and not to the
+calls made here. Both tests therefore inject, and both assert on the de-registration — the *last*
+statement of the release block — because its absence is exactly the wedge.
+
+De-registration alone is not the whole property, and asserting only it is how the first version of this
+module let a second defect through. Releasing the singleton makes the *next* boot succeed, so anything
+poisoning that still survives turns a loud "already initialized" into a silent wrong boot. The
+``KajsonManager`` is the one that bites: it is a singleton, so if it outlives the teardown the next
+boot's ``KajsonManager(class_registry=…)`` hands back the old manager and discards the fresh registry,
+and ``get_class_registry()`` keeps serving the previous boot's classes. The first test therefore also
+asserts no manager survives — a real control, since the raiser it injects is the *first* statement of
+the release block, so every later release in the ``try`` is skipped.
+"""
+
+import pytest
+from kajson.kajson_manager import KajsonManager
+from kajson.singleton import MetaSingleton as KajsonMetaSingleton
+from pytest_mock import MockerFixture
+
+from pipelex.pipelex import Pipelex
+from pipelex.system.runtime import IntegrationMode, runtime_manager
+
+
+def _test_integration_mode() -> IntegrationMode:
+ """The boot mode the session conftest uses, so a re-boot here matches the one it replaces."""
+ return IntegrationMode.CI if runtime_manager.is_ci_testing else IntegrationMode.PYTEST
+
+
+class TestAFailingTeardownLeavesTheProcessRebootable:
+ def test_a_raising_telemetry_teardown_still_releases_the_boot_singleton(self, mocker: MockerFixture) -> None:
+ """The first step of ``_teardown_runtime`` raises; the releases at its end must still run.
+
+ The factory is patched rather than the parameter passed, because an injected manager is only
+ adopted in some telemetry modes and this test is about the resulting state, not how it arises.
+ """
+ exploding_telemetry = mocker.Mock()
+ exploding_telemetry.teardown.side_effect = RuntimeError("telemetry teardown exploded")
+ mocker.patch("pipelex.runtime_boot.TelemetryFactory.make_telemetry_manager", return_value=exploding_telemetry)
+
+ Pipelex.teardown_if_needed()
+ try:
+ Pipelex.make(integration_mode=_test_integration_mode(), needs_inference=False)
+
+ # The teardown failure surfaces rather than being swallowed: a half-failed teardown must
+ # not look successful.
+ with pytest.raises(RuntimeError, match="telemetry teardown exploded"):
+ Pipelex.teardown_if_needed()
+
+ assert Pipelex.get_optional_instance() is None, (
+ "the raising telemetry teardown skipped the release block — this instance is still "
+ "registered, so the next make() in this process dies on 'already initialized' and "
+ "teardown_if_needed re-enters the same raiser: the process is wedged for good"
+ )
+ # The other half, and the one de-registration alone would hide. Releasing the boot singleton
+ # is what lets the next make() succeed, so a KajsonManager surviving alongside it does not
+ # fail loudly — it silently gives that boot the *previous* boot's class registry, because the
+ # singleton hands back the existing manager and discards the fresh registry passed to it.
+ assert KajsonManager not in KajsonMetaSingleton.instances, (
+ "the raising telemetry teardown skipped KajsonManager.teardown() — the next boot in this "
+ "process will succeed and silently serve this boot's class registry, with its own "
+ "registrations landing in a registry nothing resolves to"
+ )
+ finally:
+ # Stop the patched factory before re-booting, or the restore adopts the exploding mock.
+ mocker.stopall()
+ Pipelex.teardown_if_needed()
+ Pipelex.make(integration_mode=_test_integration_mode())
+
+ def test_a_raising_pipeline_manager_teardown_still_runs_the_runtime_teardown(self, mocker: MockerFixture) -> None:
+ """``Pipelex.teardown`` sequences three phases, so the interpreter one must not skip the last.
+
+ This is the same wedge reached one level up: the runtime teardown owns the release block, and
+ an unguarded ``pipeline_manager.teardown()`` between the two would skip the whole method.
+ """
+ exploding_pipeline_manager = mocker.Mock()
+ exploding_pipeline_manager.teardown.side_effect = RuntimeError("pipeline manager teardown exploded")
+
+ Pipelex.teardown_if_needed()
+ try:
+ Pipelex.make(
+ integration_mode=_test_integration_mode(),
+ needs_inference=False,
+ pipeline_manager=exploding_pipeline_manager,
+ )
+
+ with pytest.raises(RuntimeError, match="pipeline manager teardown exploded"):
+ Pipelex.teardown_if_needed()
+
+ assert Pipelex.get_optional_instance() is None, (
+ "the raising pipeline-manager teardown skipped _teardown_runtime entirely, leaving the "
+ "process wedged: no later boot can succeed and no teardown can undo it"
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+ Pipelex.make(integration_mode=_test_integration_mode())
diff --git a/tests/unit/pipelex/test_runtime_layer_import_closure.py b/tests/unit/pipelex/test_runtime_layer_import_closure.py
index 830a3a05c..6e71ca16d 100644
--- a/tests/unit/pipelex/test_runtime_layer_import_closure.py
+++ b/tests/unit/pipelex/test_runtime_layer_import_closure.py
@@ -66,6 +66,7 @@ import pytest
RUNTIME_LAYER_ENTRY_POINTS = [
"pipelex.cogt.content_generation.content_generator",
"pipelex.runtime_hub",
+ "pipelex.runtime_boot",
"pipelex.providers.builtins",
"pipelex.core.concepts.structure_generation.generator",
"pipelex.core.memory.input_shaper",
diff --git a/wip/boot-split/README.md b/wip/boot-split/README.md
new file mode 100644
index 000000000..d79d249dd
--- /dev/null
+++ b/wip/boot-split/README.md
@@ -0,0 +1,15 @@
+# Boot split — deferred items
+
+The composition root `pipelex/pipelex.py` was split into a runtime-layer `RuntimeBoot` (`pipelex/runtime_boot.py`) and the interpreter-layer `Pipelex` that subclasses it. The change itself is described in the branch's `TODOS.md` and in [Where the boot splits](../../docs/contribute/hub-layering.md#where-the-boot-splits); this folder holds only what was **deliberately not done**, each with the analysis and a suggested shape.
+
+Every note here was raised by a review pass, verified against the source, and deferred on a stated tradeoff rather than on effort. Production comments point at these files by path, so moving or renaming one means updating its citation.
+
+| Note | What it defers |
+|---|---|
+| [`runtime-boot-external-interpreter-orchestrator.md`](runtime-boot-external-interpreter-orchestrator.md) | A runtime-only boot rejects an interpreter-layer orchestrator contributed by a *built-in*, but not one contributed by an *external* entry-point plugin — `build_registrar` discovers externals unconditionally. Every remedy needs a layer signal the runtime layer deliberately does not have. Cited from `runtime_boot.py`'s boot-orchestrator gate. |
+| [`config-dir-does-not-scope-inference-paths.md`](config-dir-does-not-scope-inference-paths.md) | `config_dir` scopes the main TOML load but not the inference files (backends, routing profiles, model deck), because pinning those needs path overrides that live on the concrete `ModelManager` rather than on the `ModelManagerAbstract` the boot is typed against. Closing it means widening a public injection contract. Cited from both `make()` docstrings and the boot call site. |
+| [`failed-boot-does-not-release-every-resource.md`](failed-boot-does-not-release-every-resource.md) | `_teardown_runtime` and `_release_after_failed_boot` are two hand-maintained release lists. The half that *poisons* the next boot is now identical on both; the *dangling* half (`sdk_client_manager`, `reporting_delegate`, `func_registry`) is not. Collapsing them into one list is a lifecycle decision. |
+| [`pipelex-setup-narrows-the-runtime-boot-contract.md`](pipelex-setup-narrows-the-runtime-boot-contract.md) | `Pipelex.setup` is an `@override` that does not accept its base's `builtin_plugins` / `core_unconditional_plugin_names`, absorbs them in `**kwargs` and raises — so a caller written against the base type type-checks and fails at run time. Every candidate remedy trades one wart for another. |
+| [`boot-split-test-coverage-gaps.md`](boot-split-test-coverage-gaps.md) | The gaps *around* the new tests. A test-quality pass confirmed no new test passes vacuously; these are the coverage tradeoffs it found, each a decision rather than a patch. |
+
+None of these blocks the branch. The two that name a first caller — the external-orchestrator hole and the `config_dir` scoping — are the ones to settle before `RuntimeBoot.make()` becomes reachable from a real entry point.
diff --git a/wip/boot-split/boot-split-test-coverage-gaps.md b/wip/boot-split/boot-split-test-coverage-gaps.md
new file mode 100644
index 000000000..9fab33515
--- /dev/null
+++ b/wip/boot-split/boot-split-test-coverage-gaps.md
@@ -0,0 +1,57 @@
+# Boot-split test-coverage gaps left open deliberately
+
+**Status:** deferred. Found by a test-quality review pass on PR #1073, which confirmed no new test on that branch passes vacuously. These are the gaps it found *around* the new tests. Each is a coverage tradeoff rather than a defect, so each is recorded rather than patched — the two that were cheap and pinned a documented unrecoverable invariant (the same-class exclusivity guard, and `Pipelex.teardown_if_needed()` against a bare `RuntimeBoot`) were fixed on the branch instead.
+
+## 1. The `try`/`finally` around `_teardown_plugin_callbacks` is only distinguishable under `BaseException`
+
+`_teardown_plugin_callbacks` swallows every ordinary `Exception` per callback, so wrapping the call in `try`/`finally` differs from a bare sequence *only* when a `BaseException` passes through — and the 16-line justification on the failed-boot path says exactly that ("must run even on a `KeyboardInterrupt` mid-teardown"). No test raises one, so deleting the `try`/`finally` keeps the suite green.
+
+This now covers **three** sites, not one: `_release_after_failed_boot`, `RuntimeBoot.teardown` and `Pipelex.teardown` all wrap the call, the latter two added by the pre-landing review that found the sibling gap. Three unpinned copies of one argument is a stronger case for a test than one was.
+
+Still not fixed, for the same reason: the test would be pinning Python's `finally` semantics against a hand-raised `KeyboardInterrupt` from a plugin callback, contrived enough that the next reader would reasonably question why it exists. The rationale comments are the actual defence. Worth adding if any of the three paths ever grows a leading step that can raise ordinarily — at which point the test stops being contrived.
+
+## 2. The booted-runtime closure subprocess runs outside the test harness
+
+`tests/unit/pipelex/test_runtime_boot_closure.py` boots in a fresh interpreter via `subprocess.run`, and `RunMode` is set by an in-process session fixture rather than an env var. Measured inside the subprocess: `run_mode=normal`, `is_unit_testing=False`. Consequences:
+
+- `tests/pipelex_unit_test.toml` is **not** layered.
+- The developer's real global and project `.pipelex/` configs **are** (cwd is inherited).
+- `ensure_global_config_exists()` runs — a filesystem write to `~/.pipelex` from a unit test.
+
+So the verdict depends on the machine's config: a `.pipelex/pipelex.toml` enabling an external plugin that pulls an interpreter package would produce a spurious failure.
+
+Not fixed with the obvious one-liner (`env={**os.environ, "RUN_MODE": "unit_test"}`) because it cuts the other way too: the unit-test config layer may disable plugin groups, which would *weaken* the sweep it is meant to strengthen. Deciding which config a hermetic boot-closure test should run against — and whether the answer is a purpose-built minimal config dir rather than either existing one — is a decision worth making deliberately. Note that the sibling import-closure test does not have this problem, because it only imports.
+
+## 3. Untested new public surfaces
+
+- `RuntimeBoot.make(needs_inference=True)` — every call site passes `False`, so `validate_model_deck()` on the runtime-only path has no coverage.
+- `Pipelex.make(config_dir=...)` — `config_dir` is tested through `RuntimeBoot.make` only, though the parameter is documented at length on both.
+- The two `| None` narrowing guards in `Pipelex.setup` (plugin registrar, class registry). These are unreachable by construction — `super().setup()` assigns both unconditionally and any earlier failure propagates — and exist to satisfy the declarations. Testing an unreachable branch is not worth it; the note is here so nobody mistakes the absence for an oversight.
+
+## 4. Teardown *phase order* is asserted nowhere
+
+`Pipelex.teardown` sequences plugin callbacks → pipeline manager → runtime, and the comment calls the order load-bearing (a worker's in-flight resources must release before the pipeline manager drops the pipelines they may still be reporting on). `test_teardown_callbacks_run_lifo` pins LIFO *among* callbacks only. Replacing the three explicit lines with `self.pipeline_manager.teardown(); super().teardown()` reads cleaner, inverts the documented order, and passes everything.
+
+A test would need an ordering probe across all three phases — a plugin callback, an injected pipeline manager, and a runtime-teardown observer appending to one list. Worth doing when the Temporal worker's teardown path is next touched, since that is the integration the order exists for.
+
+## 5. The `MetaSingleton.get_subclass_instance` copy-fix is unpinned
+
+`pipelex/system/registries/singleton.py` changed `for subclass, instance in cls.instances.items()` to iterate `list(cls.instances.items())`, justified by a four-line comment: `instances` is a process-global dict that grows at run time (`GraphTracerManager` registers itself during a pipeline run) and this lookup sits on the boot hot path `ensure_pipelex_booted` reaches from concurrent threads.
+
+Reverting the `list(...)` leaves the whole suite green. The two concurrency tests in the tree — `tests/unit/pipelex/runtime_bridge/test_bootstrap_concurrency.py` — patch `Pipelex.get_optional_instance` and `Pipelex.make` out entirely, so they never reach the metaclass.
+
+Not fixed because a deterministic test needs a `base_cls` whose `__instancecheck__` or `__subclasshook__` inserts a key into `MetaSingleton.instances` mid-scan, plus a `finally` restoring the process-global dict. That is a test about CPython's dict-mutation semantics wearing a Pipelex costume, and the copy is a one-word defensive change with no downside. The honest position is that this is defence-in-depth rather than a pinned invariant; if it is ever removed as "unused", nothing will notice.
+
+## 6. `config_dir` is pinned as "reaches the loader", not as "bypasses layering"
+
+`tests/unit/pipelex/test_runtime_boot_config_dir.py` writes one overridden leaf into a temp dir and asserts it landed. That passes identically if `config_dir` were merely an *extra highest-priority layer* rather than a replacement for project/global layering — which is the half the docstrings actually promise.
+
+Two reasons it is left alone. First, the assertion belongs at `ConfigLoader.load_config`, not at the boot: it is a property of the layer list, and asserting absence at boot level needs a leaf that differs between `pipelex/pipelex.toml` and the project `.pipelex/pipelex.toml`, of which there is currently none committed. Second, the bypass is genuinely *not* total under pytest — `load_config` appends `tests/pipelex_{run_mode}.toml` on both branches — so a boot-level test would be pinning a statement that is true in production and false in the harness running it.
+
+## 7. `test_gateway_terms_check.py` stubs a `RemoteConfigSource` as a bare string
+
+`test_needs_inference_false_does_not_raise_when_terms_not_accepted` sets `mock_result.source = "fresh"` where the contract is a `RemoteConfigSource` (the session conftest uses the real enum). `setup()` therefore dies at `gateway_config_source.is_cached` with an `AttributeError` about ten lines past the gate the test exists for, and the trailing `except Exception: pass` swallows it.
+
+The test still does what its name says — the gate is reached and does not raise — but the swallow is wide enough that it cannot distinguish "gate skipped correctly" from "gate never reached", so a future reordering that moves an unrelated failure ahead of the gate would leave it green and blind. It also calls `setup()` on a `Pipelex.__new__(Pipelex)` with no `__init__`, which is what keeps the damage contained.
+
+Pre-existing (this branch only renamed the module constant it patches), and left alone because the one-line stub fix moves the swallowed `AttributeError` from one unasserted line to another without adding coverage. The real fix is to assert the specific expected failure instead of swallowing, which is a rewrite of the test's contract.
diff --git a/wip/boot-split/config-dir-does-not-scope-inference-paths.md b/wip/boot-split/config-dir-does-not-scope-inference-paths.md
new file mode 100644
index 000000000..6d147e10e
--- /dev/null
+++ b/wip/boot-split/config-dir-does-not-scope-inference-paths.md
@@ -0,0 +1,58 @@
+# `config_dir` at boot scopes the main TOML load but not the inference file paths
+
+**Status:** deferred, with the docstrings narrowed to say exactly what the parameter does. Found by Codex on PR #1073. Attempted on that PR and backed out — the fix needs a public abstract interface widened, which is a decision of its own.
+
+## What it does and does not scope
+
+`RuntimeBoot.__init__(config_dir=…)` passes the directory to `runtime_hub.setup_config(config_dir=…)`, so the **main TOML** load becomes `package defaults + this directory` instead of following project → global layering.
+
+It stops there. `models_manager.setup(...)` is called without path arguments, and `ModelManager.setup` falls back to the layered `config_manager.*` properties when they are `None`:
+
+- `config_manager.backends_file_path` → `inference/backends.toml`
+- `config_manager.backends_dir_path` → `inference/backends/`
+- `config_manager.routing_profiles_file_path` → `inference/routing_profiles.toml`
+- `config_manager.model_decks_dir_path` → `inference/deck/`
+
+Each resolves from the *detected* project or global config dir. So a boot pointed at an alternate `config_dir` combines that directory's main settings with whatever inference tree happens to be detected — cross-contamination between two config trees, silently.
+
+There is a **fifth** unscoped read, and it is not an inference path at all: the gateway consent/onboarding state.
+
+```python
+pipelex_service_config = load_pipelex_service_config_if_exists(config_dir=config_manager.global_config_dir)
+```
+
+That is hardcoded to the global dir, so a `config_dir`-scoped boot still reads `~/.pipelex/` for terms acceptance and `inference_setup_completed` — the two whose absence raises `GatewayTermsNotAcceptedError` / `InferenceSetupRequiredError`. Compounding it, the `config_dir` branch of `load_config` skips `ensure_global_config_exists()`, so a scoped boot never materialises the tree that this line then reads.
+
+Unlike the four inference paths, it is **not obvious that scoping this one is correct**: terms acceptance is per-user consent, and per-user global state is arguably where it belongs no matter which config directory a given boot reads. That is the question to settle first — the fix here may be "document it as intentional" rather than "propagate `config_dir`". It is listed with the others because the *documented limit* had omitted it, not because the behaviour is known to be wrong.
+
+`ModelManager.setup`'s own comment already says this is the intended remedy: *"Override paths let the doctor scope --global properly; default None falls back to layered config_manager paths for all other callers."* The doctor's `--global` path does exactly that, pinning all four with the comment *"so layered config_manager.X resolution doesn't silently fall back to the project-local files."*
+
+## Why it was not fixed on PR #1073
+
+The four-line propagation was written, tested, and backed out for one reason: **the path overrides exist on the concrete `ModelManager`, not on `ModelManagerAbstract`.**
+
+`RuntimeBoot.models_manager` is typed `ModelManagerAbstract` — and that is deliberate, because `models_manager` is a documented `make()` injection point, so an embedder may supply their own. Pyright rejects the call:
+
+```
+error: No parameter named "backends_library_path" (reportCallIssue)
+```
+
+The doctor avoids this only by constructing a concrete `ModelManager()` itself.
+
+This repo's own rule forbids the shortcut: *"When a getter returns a Protocol type, callers must only rely on methods declared on that Protocol. If you need to call a method that lives on a concrete implementation, extend the Protocol — do not work around it with `isinstance`, `cast`, or inline imports."* So the honest fix is to widen `ModelManagerAbstract.setup()` with the four optional path parameters and update every implementation — a change to a public injection contract, which does not belong inside a placement refactor whose whole claim is that behaviour is unchanged.
+
+Two smaller observations that argue for doing it properly rather than quickly:
+
+- Completing the scoping makes an explicit `config_dir` **require a complete config directory**. Verified: with the paths pinned, a directory holding only `pipelex.toml` fails boot with `InferenceBackendLibraryNotFoundError`. That is the correct contract — it is what "only this directory is read" means, and what `~/.pipelex` satisfies because `ensure_global_config_exists` materialises it from kit templates — but it is a real behavioural requirement that deserves to be introduced deliberately, not as a side effect.
+- The four path literals are currently spelled inline in `doctor_cmd.py` (`"inference"`, `"backends.toml"`, …) while `config_loader.py` already owns them as constants (`INFERENCE_DIR_NAME`, `BACKENDS_FILE_NAME`, `BACKENDS_DIR_NAME`, `ROUTING_PROFILES_FILE_NAME`, `MODEL_DECKS_DIR_NAME`). A second copy would double a drift risk that should instead be collapsed.
+
+## Suggested shape
+
+1. Widen `ModelManagerAbstract.setup()` with the four optional path parameters, and update every implementation (`@override` will surface any that drift).
+2. Add a `config_manager` helper that derives the scoped set from a directory — something like `inference_paths_for(config_dir)` returning the four — and use it from **both** `RuntimeBoot.setup` and `doctor_cmd`, replacing doctor's inline literals. That is the step that removes the drift risk rather than duplicating it.
+3. Pin it with the negative that matters: a `config_dir` holding only `pipelex.toml` must **fail** rather than quietly borrowing another tree's backends. That test was written on #1073 and verified to fail when the propagation is reverted, so it can be lifted straight from this branch's history.
+4. Document the "complete config directory required" contract on `config_dir` in both `make()` docstrings.
+
+## What exists today instead
+
+Both `make()` docstrings now state the limit explicitly and point here, so the parameter no longer promises isolation it does not deliver. The boot carries a `NOTE:` at the `models_manager.setup` call naming the same thing.
diff --git a/wip/boot-split/failed-boot-does-not-release-every-resource.md b/wip/boot-split/failed-boot-does-not-release-every-resource.md
new file mode 100644
index 000000000..e7daea634
--- /dev/null
+++ b/wip/boot-split/failed-boot-does-not-release-every-resource.md
@@ -0,0 +1,35 @@
+# A failed boot still leaves three non-singleton resources unreleased
+
+**Status:** deferred, narrowed. The headline item this file used to describe — the **telemetry manager singleton** surviving a failed boot so the next boot adopted the dead one — is **fixed** on PR #1073 (`_release_after_failed_boot` now calls the guarded `telemetry_manager.teardown()`, which flushes and calls `clear_instance()`), pinned by `tests/unit/pipelex/test_runtime_boot_failed_boot_release.py` and verified to fail when the release is reverted.
+
+A second item has since been fixed too, and it belonged to the *other* path — worth recording because this file used to scope itself to the failed-boot path alone, and that scoping is exactly what let the defect hide. `_teardown_runtime` kept `KajsonManager.teardown()`, `TemplateLoader.reset()` and `TemplateRegistry.clear()` in its `try`, above steps reachable through injectable abstract types, while its sibling had them in the `finally`. Since the same commit also moved the `MetaSingleton` de-registration into the `finally`, a raiser stopped failing loudly on the next boot ("already initialized") and started letting it **succeed against the previous boot's class registry** — the `KajsonManager` singleton hands back the surviving manager and discards the fresh registry. The three releases now sit in `_teardown_runtime`'s `finally` as well, so the *poisoning* half of the two lists is identical on both paths, and `test_runtime_boot_teardown_resilience.py` asserts no `KajsonManager` survives a raising teardown.
+
+**The scope of this note is therefore both paths, not just the failed-boot one.** What remains is smaller and of a different kind.
+
+## What is still not released on a failed boot
+
+`_release_after_failed_boot()` releases the process globals that would otherwise **poison the next boot**: the hub config, `class_registry_scoping`, `KajsonManager`, the template registries, the telemetry singleton, and the `MetaSingleton` registration. It does not release three things that `teardown()` does:
+
+| not released | what `teardown()` does | consequence |
+|---|---|---|
+| `sdk_client_manager` | `.teardown()` | vendor SDK clients constructed during `sdk_client_manager.setup()` are not closed |
+| `reporting_delegate` | `.teardown()` | the delegate's own resources are not released |
+| `func_registry` | `.teardown()` | registrations from this boot persist into the next one |
+
+None of these is a *singleton-identity* bug — that was the telemetry case, where a stale registration silently replaced a fresh construction, and the `KajsonManager` case above. These are ordinary "resources not closed on the error path": the next boot builds its own `sdk_client_manager` and `reporting_delegate`, so it does not inherit broken state, it merely leaves the old ones dangling. `func_registry` is the closest to a real problem, being a module-level global whose entries carry over — and note that it carries over on the **normal** teardown path too, whenever a step above it raises, since it stayed in `_teardown_runtime`'s `try`. That was left alone deliberately: moving it would break the symmetry with `_release_after_failed_boot`, which cannot release it at all, and the symmetry is what makes the two lists reviewable against each other. It is the first thing the collapse below should pick up.
+
+## Why it is deferred rather than patched
+
+Adding three more guarded calls to the failure path is easy but is the wrong shape. `_release_after_failed_boot` exists only because `teardown()` is unsafe on a half-built instance — it reads `self.inference_manager` (and, on the interpreter half, `self.pipeline_manager`) unguarded, deliberately, so that a half-built teardown cannot look successful. Every resource added to the failure path widens a **second, hand-maintained copy** of the teardown list, which will drift from the first.
+
+The better change is to collapse them:
+
+1. Guard `inference_manager` and `pipeline_manager` in `_teardown_runtime` / `Pipelex.teardown` the way `telemetry_manager` and `reporting_delegate` already are.
+2. Have `_release_after_failed_boot()` call the real teardown, keeping its own `try`/`finally` so the process-global un-poisoning still happens even if a manager teardown or a plugin callback raises.
+3. Delete the duplicated list.
+
+The risk to weigh in step 1 is the property the current code deliberately protects: guards let a half-built teardown *look* successful. The mitigation is that on the failure path the caller already knows the boot failed, so "looking successful" is not load-bearing there — unlike in `teardown()`, where it is. That asymmetry is the thing to decide, and it is a decision about lifecycle semantics rather than a missing line.
+
+## Why not on PR #1073
+
+That PR is a placement refactor whose claim is that a full `Pipelex` boot behaves identically. Collapsing the two teardown paths changes what *every* failed boot does for every existing caller and wants its own test matrix. The telemetry singleton was fixed there because it is a different class of defect — a stale singleton silently adopted by the next boot, production-reachable through `ensure_pipelex_booted`'s per-call lazy boot, and closed by one guarded call to an API written to never raise.
diff --git a/wip/boot-split/pipelex-setup-narrows-the-runtime-boot-contract.md b/wip/boot-split/pipelex-setup-narrows-the-runtime-boot-contract.md
new file mode 100644
index 000000000..e200aec28
--- /dev/null
+++ b/wip/boot-split/pipelex-setup-narrows-the-runtime-boot-contract.md
@@ -0,0 +1,32 @@
+# `Pipelex.setup` narrows its base class's parameter list, and `**kwargs` turns that into a runtime error
+
+**Status:** deferred as a design tradeoff, not applied. Found by an adversarial review pass on PR #1073. Nothing in the workspace hits it; the honest fix is a signature decision rather than a patch.
+
+## What happens
+
+`RuntimeBoot.setup()` documents `builtin_plugins` and `core_unconditional_plugin_names` as part of its contract — they are how a runtime-only boot discovers exactly the runtime half of the built-in plugins. `Pipelex.setup()` is an `@override` that deliberately does *not* accept them, because it supplies its own composed manifests (`BUILTIN_PLUGINS`, `CORE_UNCONDITIONAL_PLUGIN_NAMES`) to `super().setup()`.
+
+It absorbs the difference in `**kwargs: Any` and rejects it:
+
+```python
+if kwargs:
+ msg = f"The Pipelex setup method does not support any additional arguments: {kwargs}"
+ raise PipelexSetupError(msg)
+```
+
+So `pipelex_instance.setup(integration_mode=…, builtin_plugins=[])` type-checks — `**kwargs: Any` swallows the argument, and pyright sees nothing wrong — and raises at run time. A caller who wrote against the *base* type gets a green type check and a failure in production.
+
+## Why it was not "fixed"
+
+Every candidate remedy trades one wart for another, and the current behaviour is not the worst of them:
+
+- **Accept and ignore them.** Silently discarding a caller's plugin manifests is worse than refusing: they asked for a specific plugin set and would get a different one with no signal.
+- **Accept and honour them.** That contradicts what `Pipelex` *is* — the boot that runs methods, and therefore needs the interpreter-touching built-ins. A caller who wants a different set wants `RuntimeBoot`.
+- **Drop `**kwargs` and let Python raise `TypeError`.** Arguably the most honest, and it makes the narrowing visible to pyright at the call site. But `**kwargs` is doing a second job here — catching typo'd keyword arguments across a 17-parameter surface with a message that names them — and the `TypeError` wording would be worse for the common case.
+- **Split the base signature** so the manifest parameters live on a separate protected method the subclass overrides instead of on `setup()`. This is the actually-correct answer, and it is a change to a public injection contract that does not belong in a placement refactor.
+
+The narrowing is real (`Pipelex.setup` is not substitutable for `RuntimeBoot.setup`), and it is at least *loud* — the failure names the offending argument. Worth revisiting alongside any other work on the `setup()` signature; not worth a workaround on its own.
+
+## What exists today instead
+
+Nothing in the workspace passes either parameter to `Pipelex.setup`, and `Pipelex.make()` — the way essentially every consumer boots — does not expose them at all, so the reachable surface is a direct `setup()` call on a constructed instance.
diff --git a/wip/boot-split/runtime-boot-external-interpreter-orchestrator.md b/wip/boot-split/runtime-boot-external-interpreter-orchestrator.md
new file mode 100644
index 000000000..cb50a8290
--- /dev/null
+++ b/wip/boot-split/runtime-boot-external-interpreter-orchestrator.md
@@ -0,0 +1,52 @@
+# A runtime-only boot accepts an external interpreter-side orchestrator, then half-applies it
+
+**Status:** deferred, deliberately. Found by Codex on PR #1073 (the boot split), verified real, not fixed there.
+
+## The hole
+
+`RuntimeBoot.setup` gates the requested boot orchestrator like this:
+
+```python
+requested_boot_orchestrator = get_config().plugins.boot_orchestrator
+if requested_boot_orchestrator is not None and requested_boot_orchestrator not in plugin_registrar.registered_plugin_names:
+ raise UnknownBootOrchestratorError(requested=requested_boot_orchestrator)
+```
+
+On a bare runtime boot, `builtin_plugins` defaults to `RUNTIME_BUILTIN_PLUGINS`, so an orchestrator contributed by an interpreter-layer **built-in** (`direct`) is never registered and the gate correctly rejects it.
+
+But `build_registrar` discovers **external** `pipelex.plugins` entry points unconditionally — the `builtin_plugins` parameter only scopes the built-in half (`plugins/discovery.py`, the `for entry_point in _external_entry_points()` loop). So an installed external interpreter-side orchestrator — our Temporal plugin is the live example — registers its name on a runtime-only boot too, and satisfies the gate.
+
+What follows is silent and incoherent. The six `HubSlot`s split exactly 3/3 along the layer line:
+
+| slot | applied in |
+|---|---|
+| `CONTENT_GENERATOR` | `runtime_boot.py` |
+| `TASK_MANAGER` | `runtime_boot.py` |
+| `ISOLATED_EXECUTION_PROBE` | `runtime_boot.py` |
+| `PIPE_FUNC_EXECUTOR` | `pipelex.py` |
+| `PIPE_ROUTER` | `pipelex.py` |
+| `PIPE_RUN` | `pipelex.py` |
+
+So `RuntimeBoot.make(boot_orchestrator="temporal")` with the plugin installed applies the plugin's **runtime** claims — including `TASK_MANAGER`, which does the full worker wiring — never applies its `PIPE_ROUTER` / `PIPE_RUN` / `PIPE_FUNC_EXECUTOR` claims, and then sets `is_ready = True`. A process that believes it is running under an orchestrator, isn't.
+
+## Why it was not fixed on #1073
+
+- **Unreachable today.** It needs all three of: an external interpreter-side orchestrator plugin installed, config or an argument naming it, *and* a caller of `RuntimeBoot.make()`. There are no callers yet — the PR is explicit that the pinned test entry points are the only ones.
+- **Every remedy needs a layer signal the runtime layer deliberately does not have.** A `PipelexPlugin` carries no layer field, and `docs/contribute/hub-layering.md` says so on purpose ("a plugin belongs to exactly one layer… Nothing enforces it mechanically"). Asking the runtime layer to classify plugins by layer inverts the boundary the split exists to draw.
+- **The nearest clean remedy is a flag on the class pair that exists to avoid flags** — see below. On the same PR, the doctor-adoption question was declined for exactly this reason ("a flag argument added to make one caller fit is a worse outcome than two short sequences that differ honestly"), so adding one here for a path with no callers would contradict the decision taken twenty lines away.
+
+## Candidate remedy, for whoever adds the first runtime-only caller
+
+The invariant is expressible without classifying *plugins*, because the split is a property of the **slots**, which already live in the runtime layer:
+
+1. Give `HubSlot` an `is_interpreter_slot` property — a `match`/`case` over the six members. Exhaustive, so adding a seventh slot forces a placement decision at the type level (this repo's StrEnum idiom, per `.claude/rules/python-standards.md`).
+2. Let the boot say whether it honours those slots — a class attribute `RuntimeBoot.honours_interpreter_slots = False`, overridden `True` in `Pipelex`. This is polymorphism rather than a parameter, which is the mechanism the split already chose, so it is the least-bad shape available.
+3. In `RuntimeBoot.setup`, after `build_registrar`, raise when `honours_interpreter_slots` is false and any claimed slot is an interpreter slot. Name the plugin and the slots in the message.
+
+That check also subsumes the narrower orchestrator question: it catches *any* external plugin whose claims a runtime-only boot cannot honour, not just orchestrators, and it does not care whether the plugin is built-in or external.
+
+**Cheaper alternative worth weighing first:** have `build_registrar` take the set of slots the caller will apply and refuse a claim outside it. That pushes the check into the one place that already knows every claim, and removes the class attribute — but it widens `build_registrar`'s signature, and that function is currently a pure discover-and-register with no policy in it, which is a property worth protecting.
+
+## What exists today instead
+
+The gate's comment in `runtime_boot.py` states the hole precisely and points here, so the next reader meets the reasoning rather than rediscovering the trap. Nothing silently claims to handle it.
diff --git a/wip/drift-contracts/dogfood-log.md b/wip/drift-contracts/dogfood-log.md
index 754fd9226..5550600d7 100644
--- a/wip/drift-contracts/dogfood-log.md
+++ b/wip/drift-contracts/dogfood-log.md
@@ -37,3 +37,20 @@ One entry per ack. Verdicts: **real-catch** (the review found actual staleness),
- **2026-07-29 · cli-docs · friction** — fourth consecutive import-line-only opening, and the clearest one yet: two of the three triggers change a single `from` line each (`PipeRunMode` moving to `pipelex.system.pipe_run_mode`), and the third changes a *comment* — the guard's declaration rationale, which no CLI doc has ever described or could. Targets grepped for all four moved symbols and their old module paths: zero hits. Verified the affected surface live anyway (`run pipe --help` still lists `--dry-run` / `--mock-inputs` exactly as `agent-cli.md` lines 37-38 describe). The narrowing case is now overdetermined — the content-aware digest proposed on 2026-07-28 would have suppressed this, the two before it, and made the merge-induced reopening nearly free, while preserving every real-catch this contract has recorded (all of which touched a user-visible string). One further refinement this opening suggests: `pipelex/cli/dev_cli/**` arguably should not be a trigger for this contract at all — the dev CLI is not distributed with the package and neither review target covers it, so it can only ever produce friction here. It already has its own contract in `hub-layering-convention`, which is where its changes were genuinely reviewed.
- **2026-07-29 · hub-layering-convention · real-catch (twelfth)** — the opening that closes the thing this whole log has been watching rot. The caveat that two interpreter-named packages could not be named in the layer predicate was written into **six** places, and the leaf models that made it true moved this phase, so all six went false at once. Four were on the review target and all four were edited: the must-not-name list, the deliberately-absent paragraph, the `core.pipes.pipe_output` parenthetical under Where core splits, and "what the predicate still cannot name" under The property. **Sixth consecutive opening where an enumeration was the rot** — but this one carries the counter-example, and it is the most useful finding in the entry. The page's runnable `INTERPRETER = {…}` snippet is the one enumeration that *cannot* go stale, because `test_the_interpreter_package_set_matches_the_documented_one` compares it against `INTERPRETER_PACKAGES` by machine: it would have gone red on this change whether or not anyone reviewed the page. Every other enumeration on the page is prose, and every one of them has now rotted at least twice. That is a concrete, generalizable remedy for this contract's recurring failure — a doc enumeration that duplicates a code constant should either point at it (the `AGENT_ERROR_HINTS` pattern the 2026-07-29 `cli-docs` entry praised) or be machine-compared to it; prose duplication is the failure mode, not prose itself. The two prescribed mechanical checks passed and were blind to all six sites, **tenth consecutive time**, which is by now less a caveat than a measurement: they check *presence*, and every finding this contract produces is about *content*. Outcome verified rather than asserted — measurement across all nine runtime entry points returns zero `pipeline` and zero `pipe_run` modules, and the closure test is green with both names added. **Twelfth consecutive real-catch.**
- **2026-07-29 · hub-layering-convention · real-catch (thirteenth)** — third opening on this contract in one day, and the one that found a live counter-example to the page's own central claim rather than a stale sentence. Two of the three findings are the usual shape, now thoroughly established: the Enforcement enumeration was stale *again* (sixth consecutive opening, same enumeration as the previous two), and the exclusion sentence — "kit is data files and language has interpreter edges, so both stay out" — read exhaustive while accounting for two of the tree's twenty-seven top-level packages. Enumerating all of them and running the measurement over each turned up `test_extras`: clean, shipped, imported at boot, and undeclared. **The third finding is a new shape and the reason to read this entry.** Five built-in vendor adapters imported `MissingDependencyError` from `pipelex/exceptions.py` — the *public all-errors aggregate*, which re-exports every interpreter package's exceptions — instead of from its definition site. Each therefore loaded 15 interpreter modules, inside `pipelex.providers`, a package the guard's own tuple already declared clean. Both gates were blind by construction: the guard's rules only chase `interpreter_hub` and an aggregate of exception classes never reaches it, while the closure test's `providers.builtins` entry point misses these modules because the vendor plugins import them *inside a function* — a deferred import hides a module from the static graph and from the aggregator's runtime closure simultaneously. So the failure needed two independent blind spots to survive, and it had them. Fixed at the import site (the repo's existing "direct full-path imports everywhere" rule already forbade it), with one new entry point and a negative control. **The generalization worth carrying forward: a module that re-exports across layers is a layer boundary with the sign filed off.** `pipelex/exceptions.py` is a deliberate public convenience, so the answer is not to delete it — it is that runtime-layer code must import from definition sites, and that a lazily-imported module is invisible to every static gate in this repo. Two prescribed mechanical checks passed and were blind to all three findings, **eleventh consecutive time**; the counting is no longer interesting, the pattern is — they verify *presence*, and every finding this contract has ever produced concerns *content*. **Thirteenth consecutive real-catch.**
+- **2026-07-30 · hub-layering-convention · real-catch (fourteenth)** — opened by adding `pipelex.runtime_boot` to `RUNTIME_LAYER_PACKAGES`, the guard's tuple, when `pipelex.py` was split into a runtime-layer composition root and an interpreter-layer boot. The catch is the same enumeration for the third consecutive round: the prescribed mechanical check ("every declared package is named under 'The rule'") **failed** on `pipelex.runtime_boot`, so the page's own definition of the runtime layer would have shipped one entry short of the tuple it claims to describe. Seven consecutive rounds now where an enumeration on this page was the rot, and three in a row where it was *this* enumeration. Two observations worth carrying forward. First, the enumeration is the highest-value target on the page and the cheapest to check — the check is four lines of shell against the imported tuple, it has caught something every time it has been run, and it is still not automated; a test asserting the doc names every declared package would convert this recurring real-catch into a gate and retire the manual step. That is now the clearest candidate the pilot has produced. Second, the other prescribed check (hub accessor coverage) passed and was again blind to the finding — it has now been blind on every round it has been run, which is a signal about the check rather than about the page. Beyond the enumeration, six prose sites were genuinely stale because they named `Pipelex.setup` / `Pipelex.teardown` as doing things that now belong to the runtime half, and one — "a process that only ever builds a `RuntimeHub` (the doctor path, most unit tests)" — went from describing a degradation nobody could reach to describing a first-class, tested entry point. The contract opening was proportionate: the trigger was a one-line tuple edit, and the review turned up eight edits.
+
+## hub-layering-convention — fifteenth ack (boot-split review fixes)
+
+**Verdict: no-op (correct silence).** Trigger was a single docstring paragraph on `RuntimeHub.setup_config`'s `config_dir` — an accuracy fix, not a boundary change. The review target (`docs/contribute/hub-layering.md`) does not describe `setup_config`, `config_dir`, or config layering at all, so there was nothing on the page that could have gone stale.
+
+Worth recording precisely because it is the *opposite* of the previous fourteen: the contract fired on a file it watches for good reason (`runtime_hub.py` is one of the two hubs) but on a region of that file the doc has no stake in. The trigger is file-granular; the staleness is region-granular. That gap is the cost of the contract being cheap, and it is the right trade — a false positive costs one grep, and the alternative (region-scoped triggers) is a maintenance burden that would itself drift.
+
+The mechanical enumeration check ran clean for the first time in eight rounds — because the last ack fixed it and this change added no declared package. That is the automation candidate holding steady, not the problem being solved: the check still has to be remembered and run by hand.
+
+## config-docs — sibling of an already-applied fix, found by a review pass rather than by the contract
+
+**Verdict: no-op on the review target (correct silence), but the *opening* is the interesting part.** Trigger was a docstring-only correction to `ConfigLoader.load_config`'s `config_dir` parameter: it still read "project/global layering is bypassed and only this directory is read", which contradicts its own docstring summary six lines above and the body, where `list_of_configs` is seeded unconditionally with the package `pipelex.toml`. Reviewed `docs/configuration/**/*.md` in full — `config_dir` is not a user-facing concept and appears on no page; `index.md`'s documented loading sequence already states the package layer as step 1, i.e. the docs were right about the very fact the docstring was denying. Nothing to update.
+
+What this round adds to the pilot is a shape none of the previous entries has: **the contract opened on the second half of a fix whose first half had already shipped without opening it.** The identical wrong sentence lived in two places — `RuntimeHub.setup_config` and the `load_config` it delegates to. Correcting the first opened `hub-layering-convention` (fifteenth ack, "no-op, correct silence"); correcting the second opened `config-docs`, also a no-op. Two contracts fired, neither review target was stale, and neither contract could have pointed at the duplication, because a review contract watches code→doc drift and this was code→code drift. The sibling was found by a review pass reading both files, not by the manifest.
+
+The honest reading is not "add a contract for it" — the right instrument for two copies of one sentence is deduplication or a docstring that points at its source, not a review obligation. It is a data point on scope: `drift.toml`'s header says anything mechanizable belongs in a derived check, and this is the mirror case, something neither mechanizable *nor* expressible as a review target. Worth remembering the next time a no-op ack tempts a trigger-set widening.