Skip to content

Build AI methods with your coding agent, then run them anywhere: as an MCP for chatbots, as a webapp for people, or via API for your software.

feat: Offline mode for Pipelex Gateway setup and dry-run (#900)

Louis Choqueld58fade6aea3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5dfb2d026..d58d3b155 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,8 +2,22 @@
## [Unreleased]
+### Added
+
+- **Offline mode for Pipelex Gateway setup and dry-run.** When the gateway is enabled but the remote config service is temporarily unreachable, Pipelex now falls back to a previously primed on-disk cache (`~/.pipelex/cache/remote_config.json`, schema-versioned) instead of failing setup outright. Dry-run, validation, and `pipelex-agent run bundle --dry-run` complete normally; only the actual inference call still needs the network at runtime. The cache is primed on every successful fetch and on `pipelex init` while online. When the gateway is disabled (BYOK), no remote fetch is attempted at all — setup is fully offline. A new `RemoteConfigStaleWarning` (`UserWarning`) is emitted whenever stale cache is in use; the agent CLI surfaces it on the JSON envelope as `warnings: [{"type": "RemoteConfigStale", ...}]`. Telemetry is suppressed (no-op) when running on a cached config so stale model identities don't pollute metrics. The doc/fixture generators (`pipelex-dev update-gateway-models`, `preprocess_test_models_cmd`) refuse the cache fallback via a new `require_fresh=True` flag, so committed reference docs and test fixtures never bake in stale data.
+- **`GatewayUnknownModelError` (`pipelex.cogt.exceptions`).** Raised at setup time when the active model deck references a gateway model handle that isn't present in the (fresh or cached) gateway specs. Carries the model name and the config source (`RemoteConfigSource.FRESH` | `CACHED`); the message branches on source so a cached-source failure suggests `pipelex init` while online and a fresh-source failure points at deck/typo fixes. Wired through both the Rich CLI (`handle_gateway_unknown_model_error` in `error_handlers.py`) and the agent CLI (`AGENT_ERROR_HINTS` / `AGENT_ERROR_DOMAINS`).
+- **`RemoteConfigUnavailableError` (`pipelex.system.pipelex_service.exceptions`).** User-facing offline-mode error: raised only when the network fetch fails AND no usable cached fallback exists. The message names the cache file path and the two remediation paths (run `pipelex init` while online to prime the cache; or disable `pipelex_gateway` in `backends.toml` for permanent BYOK operation). Distinct from the internal `RemoteConfigFetchError`, which is kept as the retry-layer exception.
+- **`PIPELEX_REMOTE_CONFIG_URL` environment variable.** Overrides the default remote-config URL. Useful for staging/testing environments; defaults to the production URL when unset.
+
+### Changed
+
+- **`RemoteConfigFetcher.fetch_remote_config()` now returns a `RemoteConfigResult`** carrying `config`, `source` (`fresh` | `cached`), and `cached_at`, instead of a bare `RemoteConfig`. Callers unwrap `.config` for the payload and may branch on `.source` to know whether the config is fresh or restored from cache. The fetcher accepts a new keyword-only `require_fresh: bool = False` — when `True`, a cached fallback raises `RemoteConfigUnavailableError` instead. `RemoteConfigValidationError` is never satisfied by the cache (server-side schema breaks must surface loudly).
+- **`ModelManager.setup()` and `BackendLibrary._load_gateway_model_specs()` accept a new `gateway_config_source: RemoteConfigSource | None` parameter.** Passed through from `Pipelex.setup()` so the deck-level gateway membership check can branch its error message on `FRESH` vs `CACHED`. `GatewayConfig` itself stays `extra="forbid"` and source-free — provenance is plumbed alongside, not baked in.
+- **`RemoteConfigUnavailableError` message branches on whether the cache was refused vs absent.** When `require_fresh=True` (dev-CLI generators) refuses to fall back to an existing cache, the message now reads "the local cache at `<path>` was refused because a fresh fetch is required" instead of the previously misleading "no local cache is available at `<path>`". The cache-truly-missing path keeps its original wording.
+
### Fixed
+- **A directory containing a `.pipelex/` config dir is now recognized as a project root.** `.pipelex` was added to `PROJECT_ROOT_MARKERS`, so project-level config (e.g. `.pipelex/inference/backends.toml`) is honored even when the directory has no `.git`, `pyproject.toml`, or other source-project marker. Previously such a directory fell through to the global `~/.pipelex/` config, silently ignoring the project's own overrides — so a backend disabled in the project's `backends.toml` could still demand credentials because the global config (where it was enabled) was loaded instead. The home directory remains excluded from project-root detection, so the global `~/.pipelex/` is unaffected.
- **`PipeLLM` outputting a concept that refines the native `JSON` concept no longer crashes with `NameError: name 'Any' is not defined`.** On a LIVE run, such a concept resolves to a structured-output model carrying a `dict[str, Any]` field inherited from `JSONContent`. `SchemaToModelFactory` generates that model as source with `from __future__ import annotations` (every annotation becomes a string) and then rebuilds each class to resolve the string annotations. The rebuild namespace was assembled from the exec'd user types plus a hand-listed `Literal`, but `typing.Any` is a special form, not a `type`, so it was filtered out — `model_rebuild` then raised `PydanticUndefinedAnnotation` evaluating `"dict[str, Any]"`. The rebuild namespace is now the exec namespace itself (minus `__builtins__`), so it carries exactly the names the generated source was written against and cannot drift as codegen emits other typing constructs. Fixed in `pipelex/cogt/content_generation/schema_to_model_factory.py`; covers both the sender path (`make_from_json_schema`) and the cross-process receiver path (`make_types_from_source`).
## [v0.28.0] - 2026-05-13
diff --git a/TODOS.md b/TODOS.md
deleted file mode 100644
index 83e5fc704..000000000
--- a/TODOS.md
+++ /dev/null
@@ -1,620 +0,0 @@
-# Template preprocessor — sigil collision fix (TDD plan)
-
-## Status: `$` sigil footgun fixes landed (Phase 7 — 2026-05-13)
-
-The inline `$` sigil now mirrors the `@` candidate pattern's word-boundary lookbehind and uses a strict segmented identifier shape. Word-adjacent `$` (`micro$oft`, `user$host.com`, `P@ssw$rd123`, `a$b$c`) is now silent pass-through; `$name..` renders cleanly as `{{ name|format() }}..` (consecutive dots stay outside the match as literal punctuation). The trailing-dot kludge in `_replace_dollar_sigil` is gone — unreachable under the new identifier shape. `make agent-check` clean, `make agent-test` green.
-
-### Deviations from Phase 7 plan
-
-1. **One existing test updated to match the new contract.** `tests/unit/pipelex/pipe_operators/pipe_compose/test_structured_content_composer.py::TestStructuredContentComposerRuntimeParams::test_template_field_accesses_extra_context` used the template `"Summary for $fiscal_year Q$quarter"`, which relied on the pre-Phase-7 deviation that allowed word-adjacent `$` to interpolate (so `Q$quarter` → `Q1`). Under Phase 7's word-boundary lookbehind that shape is intentional silent pass-through. The template was changed to `"Summary for $fiscal_year, quarter $quarter"` (separator added so the second `$` is not word-adjacent) and the assertion updated to `"quarter 1" in result.generated_summary`. Test intent (verifying `extra_context` is merged into template rendering) is preserved.
-2. **No `.mthds` workspace migration needed.** The Phase 7.3 grep flagged two lines in `mthds-ui/data/pipelines/pipeline_25/bundle.mthds` (`"# Text Report\n\n$classified.content"` and `"# Data Report\n\n$classified.content"`), but these are grep false positives — the `n` matched by `[[:alnum:]_]` is the `n` of the TOML `\n` escape; after TOML decoding the `$` is preceded by a real newline (whitespace), so the new regex still rewrites correctly.
-
-## Status: declared-inputs gating landed (Phase 8 — 2026-05-13)
-
-Phase 8 shipped. Inline `@<ident>` now raises only when `<ident>` (root segment for dotted paths) is a declared input of the surrounding pipe; CSS at-rules (`@media`, `@font-face`, `@import`, `@keyframes`, `@property`, …) and code decorators (`@deprecated`, `@Override`) pass through silently when they don't collide with an input. The line-bounded rewriter is unchanged. `preprocess_template` gained a keyword-only `declared_inputs: set[str] | None = None` parameter (default lenient); new helpers `validate_template_sigils(template, declared_inputs)` and `rewrite_template_sigils(template)` split the validate and rewrite steps so `render_template` skips re-validation at runtime. All pipe blueprints (`PipeLLM`, `PipeCompose`, `PipeSearch`, `PipeImgGen`, `PipeComposeFactory`) and template analyzers (`TemplateDocumentAnalyzer`, `TemplateImageAnalyzer`) thread their declared inputs into the validator. `make agent-check` clean, `make agent-test` green.
-
-### Deviations from Phase 8 plan
-
-1. **`preprocess_template` keeps its zero-arg form for legacy callers.** The plan implied every caller of `preprocess_template` would either pass `declared_inputs={...}` or explicit `None`. In practice the keyword-only signature with `declared_inputs=None` as default means the two call sites without inputs context (`TemplateBlueprint.validate_template`, `TemplateBlueprint.required_variables`, `ConstructBlueprint.get_required_variables`) keep their existing zero-arg call and benefit from the lenient default — no explicit `None` pass-through was needed.
-
-2. **Dedup'd `input_names` in three blueprints.** In `pipe_llm_blueprint.py`, `pipe_search_blueprint.py`, and `pipe_img_gen_blueprint.py`, the previous code computed `input_names: set[str] = set(self.inputs.keys()) if self.inputs else set()` separately from the new `declared_inputs` variable. Folded the two into a single `declared_inputs` reference threaded through both the preprocess call and the downstream missing/unused-inputs checks. Pure code clarity — no behavior change.
-
-3. **Existing strict-rule tests updated rather than removed.** The plan's "all existing tests still pass under the lenient-`None` default" claim was inconsistent with reality: parametrized tests like `test_strict_line_at_sigil_raises` and `test_css_at_rule_raises_under_strict_rule` called `preprocess_template(template)` with no kwarg and expected a raise. Under the new lenient default they would stop raising. Resolution: those parametrized tests now pass `declared_inputs={expected_identifier}` per case so they preserve the strict-rule assertion under the new gate, and were renamed accordingly (`*_raises_under_strict_rule` → `*_raises_when_keyword_is_declared_input`). The lenient pass-through case is covered by new parametrized tests (`test_inline_at_passes_through_when_declared_inputs_omitted/empty/no_input_matches`).
-
-4. **Workspace `.mthds` sanity check: pre-existing typos in sibling repos remain pre-existing typos.** Surfaced inline `@question` in `pipelex-cookbook` (two files) and `Idea: @idea` in `internal-tools/wip/`, all of which have the at-identifier declared as an input — these already raised under Phases 1–6's strict rule and continue to raise under Phase 8 (correct behavior). Flagged as follow-up cleanup for those sibling repos rather than fixed in this PR.
-
-## Status: strict line-bounded `@` rule landed (2026-05-13)
-
-The redesign described in `wip/template-preprocessor-line-bounded-at.md` shipped:
-
-- `@var` / `@?var` must be alone on their own line. Inline candidates raise `TemplateSigilSyntaxError`
- at load time, surfaced through pydantic validation with line number + migration hint.
-- `$var` keeps its inline contract (unchanged).
-- `@@` and `$$` escapes preserved.
-- Workspace `.mthds` files migrated; all tests green; `make agent-check` clean; `make agent-test` clean.
-
-The history below documents the original heuristic CSS-collision fix, kept for context — the
-heuristic regex it produced was replaced by the strict rule above.
-
-## Status: heuristic CSS-collision fix complete (earlier)
-
-All phases (1–6) landed. Full `make agent-test` is green and `make agent-check` is clean. The
-work is ready to ship via `/release`.
-
-### Deviations from the original plan
-
-Two design points evolved during implementation. They are reflected in the code, the new tests,
-and the CHANGELOG, but the prose below is the historical plan and was not rewritten to match —
-read this block first, then the rest as TDD history.
-
-1. **Single-pass regex instead of three sequential `re.sub` calls.** The plan's three
- `re.sub(...)` calls (one per sigil) cannot share the lookahead reliably: after the first pass
- rewrites `@var` to `{{ var|tag("var") }}`, the second pass's `(?!\s*[({"'])` lookahead sees
- the inserted `{` and aborts the next `$var` match. Combined into a single
- `_SIGIL_PATTERN` with an alternation `(?:(?<!\w)(@\??)|(\$))...` and a dispatch callback
- (`_replace_sigil`) so the lookahead only sees original-template characters. The original
- `replace_at_variable` / `replace_optional_at_variable` / `replace_dollar_variable` helpers
- were folded into the single callback.
-2. **`$` arm omits the `(?<!\w)` lookbehind.** The plan applied the lookbehind uniformly to
- all three sigils, but it broke prose like `Summary for $fiscal_year Q$quarter` — `Q`
- precedes the `$`, blocking the match. Splitting the alternation so only the `@`/`@?` arm
- carries the lookbehind keeps emails (`[email protected]`) protected while letting
- word-adjacent `$var` interpolate. Dollar amounts (`$10`) remain protected by the
- `(?![0-9])` lookahead.
-3. **Inner-word lookahead added for `@keyframes spin {`, `@layer reset {`, `@import url(...)`.**
- The plan's `(?!\s*[({"'])` only catches *direct* openers, but several CSS at-rules sit a
- word between the keyword and the brace. Added a second lookahead arm
- `\s*[a-zA-Z][\w-]*\s*(?:[(]|\{(?!\{))` to cover them. The arm requires `(` or a *single* `{`
- (NOT `{{`) so legitimate Jinja constructs in the same template (e.g.
- `@var with {{ page | with_images }}`) aren't misclassified. `@namespace svg "..."` remains a
- residual (too easily confused with prose like `@name said "..."`); authors escape with
- `@@namespace`.
-
-## Quick start (cold session)
-
-Working directory: `/Users/lchoquel/repos/Pipelex/pipelex` (the `pipelex/` runtime repo inside the multi-repo workspace at `/Users/lchoquel/repos/Pipelex/`).
-
-Read in this order:
-
-1. This file (`TODOS.md`).
-2. `wip/template-preprocessor-css-collision.md` — full design rationale and alternatives considered.
-3. `pipelex/cogt/templating/template_preprocessor.py` — the file being changed (~100 lines).
-4. `tests/unit/pipelex/cogt/templating/test_template_preprocessor.py` — the test file being extended.
-
-Project conventions that apply (see `pipelex/CLAUDE.md` for the full list):
-
-- One `TestClass` per test module. The existing class is `TestTemplatePreprocessor`. Add new test methods inside that same class — do **not** create a second class.
-- Use pytest-mock, never `unittest.mock`. None of the new tests need mocking.
-- Run `make agent-check` after code changes (lint + type check).
-- Use `make agent-test` to validate, or for the targeted module: `.venv/bin/pytest -q tests/unit/pipelex/cogt/templating/test_template_preprocessor.py`.
-
-## Background (minimal)
-
-The template preprocessor at `pipelex/cogt/templating/template_preprocessor.py` rewrites three sigils into Jinja2 before rendering: `@var` → `{{ var|tag("var") }}`, `@?var` → optional insertion, `$var` → `{{ var|format() }}`.
-
-The current regexes are too permissive. CSS at-rules (`@media`, `@import`, `@keyframes`, …) inside `<style>` blocks of PipeCompose HTML templates get rewritten as variables and break rendering. Email addresses (`[email protected]`) and code constructs (Python decorators, bash subshells) have the same problem.
-
-Fix has two parts:
-
-1. **Heuristic regex tightening** — add a `(?<!\w)` lookbehind before each sigil (kills emails / word-adjacent `@`) and a `(?!\s*[({"'])` lookahead after the captured identifier (kills CSS at-rules and code constructs).
-2. **Explicit escapes** — `@@` → literal `@`, `$$` → literal `$`. This is the documented opt-out for any residual case the heuristic can't predict.
-
-## Current code (reference)
-
-`pipelex/cogt/templating/template_preprocessor.py`, the three regex calls inside `preprocess_template`:
-
-```python
-# Replace @?variable patterns (optional insertion) - must come before @variable
-new_template = re.sub(r"@\?(?![0-9])([a-zA-Z0-9_.]+)", replace_optional_at_variable, processed_template)
-# ...
-# Replace @variable patterns
-new_template = re.sub(r"@(?![0-9])([a-zA-Z0-9_.]+)", replace_at_variable, processed_template)
-# ...
-# Replace $variable patterns
-new_template = re.sub(r"\$(?![0-9])([a-zA-Z0-9_.]+)", replace_dollar_variable, processed_template)
-```
-
-The three `replace_*` helpers above them are unchanged by this work.
-
-## Target code
-
-New regex strings (Phase 2):
-
-```python
-r"(?<!\w)@\?(?![0-9])([a-zA-Z0-9_.]+)(?!\s*[({\"'])"
-r"(?<!\w)@(?![0-9])([a-zA-Z0-9_.]+)(?!\s*[({\"'])"
-r"(?<!\w)\$(?![0-9])([a-zA-Z0-9_.]+)(?!\s*[({\"'])"
-```
-
-Note: `(?<!\w)` is a fixed-width negative lookbehind (one char) — supported by `re`. The lookahead set `[({\"']` covers the four shapes CSS at-rules and code constructs take after the identifier; whitespace before them is allowed (`\s*`).
-
-Escape implementation (Phase 3): use null-byte sentinels that templates cannot contain. Recommended scheme:
-
-```python
-_AT_ESCAPE_SENTINEL = "\x00PIPELEX_AT_ESCAPE\x00"
-_DOLLAR_ESCAPE_SENTINEL = "\x00PIPELEX_DOLLAR_ESCAPE\x00"
-
-def preprocess_template(template: str) -> str:
- processed_template = template.replace("@@", _AT_ESCAPE_SENTINEL).replace("$$", _DOLLAR_ESCAPE_SENTINEL)
- # ... existing three re.sub calls run on processed_template ...
- processed_template = processed_template.replace(_AT_ESCAPE_SENTINEL, "@").replace(_DOLLAR_ESCAPE_SENTINEL, "$")
- return processed_template
-```
-
-`str.replace("@@", …)` is non-overlapping left-to-right greedy, which gives the correct semantics:
-
-- `@@var` → `<SENTINEL>var` → (no `@` left for the regex) → restored: `@var` (literal, no interpolation).
-- `@@@var` → `<SENTINEL>@var` → regex matches `@var` → `<SENTINEL>{{ var|tag("var") }}` → restored: `@{{ var|tag("var") }}` (literal `@` + interpolated `var`).
-- `@@@@var` → `<SENTINEL><SENTINEL>var` → no match → restored: `@@var` (two literal `@`s).
-
----
-
-## Phase 1 — Extend tests with new expected behaviors (red)
-
-All tests go inside the existing `TestTemplatePreprocessor` class in `tests/unit/pipelex/cogt/templating/test_template_preprocessor.py`.
-
-### 1.1 — CSS at-rule pass-through (heuristic lookahead)
-
-Identifier followed by `\s*[({"']` must not be rewritten. Expected output equals the input (unchanged) for each:
-
-- [x] `test_css_media_query_pass_through` — input `@media (max-width: 820px) { color: red; }`, output identical.
-- [x] `test_css_supports_pass_through` — input `@supports (display: grid) { color: red; }`, output identical.
-- [x] `test_css_import_string_pass_through` — input `@import "reset.css";`, output identical.
-- [x] `test_css_import_url_pass_through` — input `@import url("reset.css");`, output identical.
-- [x] `test_css_charset_pass_through` — input `@charset "UTF-8";`, output identical.
-- [x] `test_css_namespace_pass_through` — input `@namespace svg "http://www.w3.org/2000/svg";`, output identical.
-- [x] `test_css_keyframes_pass_through` — input `@keyframes spin { from { opacity: 0; } to { opacity: 1; } }`, output identical. (Both `@keyframes` and the `from`/`to` selectors are unaffected since they aren't preceded by a sigil.)
-- [x] `test_css_page_pass_through` — input `@page { margin: 1in; }`, output identical.
-- [x] `test_css_layer_named_pass_through` — input `@layer reset { color: red; }`, output identical.
-- [x] `test_css_container_pass_through` — input `@container (width > 400px) { color: red; }`, output identical.
-- [x] `test_css_font_face_pass_through` — input `@font-face { font-family: "X"; }`, output identical. Note: the identifier class today is `[a-zA-Z0-9_.]+`, which does not include `-`. So today `@font-face` already captures only `@font` and then encounters `-face` as literal — the lookahead `(?!\s*[({"'])` fires on `@font` because the next non-identifier character is `-`, not whitespace+brace. **Outcome under new regex**: `@font` captures, lookahead sees `-`, not whitespace+brace, so the lookahead does **not** fire — and `@font` would still be rewritten to `{{ font|tag("font") }}`. This is a residual case. **Decision for this test**: assert the heuristic does **not** save `@font-face` (input contains literal hyphen) and document `@@font-face` as the required author workaround. The test should assert the rewritten form so the limitation is visible:
-
- ```text
- input: @font-face { font-family: "X"; }
- expected: {{ font|tag("font") }}-face { font-family: "X"; }
- ```
-
- Then add a companion `test_css_font_face_escaped_pass_through` that asserts `@@font-face { font-family: "X"; }` → `@font-face { font-family: "X"; }`.
-
-- [x] `test_full_style_block_pass_through` — a realistic multi-rule `<style>` block (combining `@media`, normal selectors, and at least one nested ruleset) must pass through byte-for-byte. Use this fixture:
-
- ```html
- <style>
- .page { padding: 32px; }
- @media (max-width: 820px) {
- .page { padding: 24px; }
- }
- @supports (display: grid) {
- .grid { display: grid; }
- }
- </style>
- ```
-
-### 1.2 — Email and word-adjacent `@` (heuristic lookbehind)
-
-`@` preceded by a word character must not be treated as a sigil.
-
-- [x] **Update** the existing `test_email_address_partially_processed` (lines 412–423 of the current test file): rename it to `test_email_address_pass_through` and replace its body to assert pass-through:
-
- ```text
- input: Contact: [email protected]
- expected: Contact: [email protected]
- ```
-
-- [x] `test_email_in_sentence_pass_through` — input `Contact us at [email protected] for help.`, output identical.
-- [x] `test_word_adjacent_at_pass_through` — input `Send to [email protected] immediately.`, output identical.
-
-### 1.3 — `$` and `@` followed by code-like punctuation (regression guards)
-
-These shapes already pass through today because the identifier class `[a-zA-Z0-9_.]+` does not match `(` or `{`. They're included as regression guards so the new lookahead doesn't accidentally regress them, and to lock in the documented behavior.
-
-- [x] `test_jquery_call_pass_through` — input `$("body").addClass("x")`, output identical.
-- [x] `test_bash_subshell_pass_through` — input `result=$(date +%s)`, output identical.
-- [x] `test_dollar_brace_pass_through` — input `Use ${PATH} for the path.`, output identical.
-
-### 1.4 — Escape sequences `@@` and `$$`
-
-- [x] `test_double_at_escapes_to_literal_at`:
- ```text
- input: @@media (max-width: 820px) { color: red; }
- expected: @media (max-width: 820px) { color: red; }
- ```
-
-- [x] `test_double_at_makes_at_var_literal`:
- ```text
- input: Use @@var here.
- expected: Use @var here.
- ```
- No `{{ … }}` in the output — the escape suppresses interpolation for that occurrence.
-
-- [x] `test_double_dollar_escapes_to_literal_dollar`:
- ```text
- input: Cost is $$10.
- expected: Cost is $10.
- ```
-
-- [x] `test_double_dollar_makes_dollar_var_literal`:
- ```text
- input: Use $$var here.
- expected: Use $var here.
- ```
-
-- [x] `test_escape_does_not_consume_legit_variable`:
- ```text
- input: @@media is literal, but @width is a variable.
- expected: @media is literal, but {{ width|tag("width") }} is a variable.
- ```
-
-- [x] `test_triple_at_is_escape_plus_variable`:
- ```text
- input: @@@var
- expected: @{{ var|tag("var") }}
- ```
-
-- [x] `test_quadruple_at_is_two_escapes`:
- ```text
- input: @@@@var
- expected: @@var
- ```
-
-- [x] `test_escape_inside_style_block` — `<style>` block containing `@@font-face` (the residual heuristic case from §1.1) — verifies the recommended manual override:
- ```text
- input: <style>@@font-face { font-family: "X"; }</style>
- expected: <style>@font-face { font-family: "X"; }</style>
- ```
-
-### 1.5 — Confirm all legacy tests still apply
-
-No test currently passing should regress. Skim each and confirm the new regexes don't break them. Quick reasoning for the non-obvious ones:
-
-- `test_variable_in_parentheses` — `The value (@value) is important`: the `@` is preceded by `(` (non-word), lookbehind passes; the captured `value` is followed by `)` (not in lookahead set), so lookahead passes. ✓ Still rewrites.
-- `test_email_address_partially_processed` — being replaced by `test_email_address_pass_through` (§1.2). Drop the old assertion.
-
-All other legacy tests are unaffected by the lookbehind (sigil is preceded by space/newline/start-of-string) and unaffected by the lookahead (variables are followed by space/dot/punctuation, not `(` / `{` / `"` / `'`).
-
-- [x] Re-run the full test module locally; legacy tests pass, new tests fail (red).
-
-### 1.6 — Run the suite to confirm reds
-
-- [x] `.venv/bin/pytest -q tests/unit/pipelex/cogt/templating/test_template_preprocessor.py`
-- [x] Confirm: all Phase 1 new tests fail with content mismatches (not import errors or syntax errors).
-
-**Checkpoint A**: Phase 1 complete when all new tests are written, all legacy tests still pass, and the new tests fail for the documented reasons.
-
----
-
-## Phase 2 — Implement the heuristic regex tightening (green for §1.1–§1.3)
-
-Edit `pipelex/cogt/templating/template_preprocessor.py`.
-
-- [x] Replace the three regex strings in `preprocess_template` with the new ones from the "Target code" section above. The replacement-function calls (`replace_optional_at_variable`, `replace_at_variable`, `replace_dollar_variable`) and their bodies stay unchanged.
-- [x] Re-run the test module. Expected state:
- - §1.1 CSS tests pass (except `test_css_font_face_pass_through`, which asserts the documented residual hyphen limitation, also passes — see its expected output above).
- - §1.2 email tests pass.
- - §1.3 regression guards pass.
- - §1.4 escape tests still fail (escapes not implemented yet).
- - §1.5 legacy tests still pass.
-
----
-
-## Phase 3 — Implement the `@@` / `$$` escapes (green for §1.4)
-
-Still in `pipelex/cogt/templating/template_preprocessor.py`.
-
-- [x] Add the two sentinel module-level constants from the "Target code" section (`_AT_ESCAPE_SENTINEL`, `_DOLLAR_ESCAPE_SENTINEL`).
-- [x] In `preprocess_template`, before the existing regex substitutions, prepend:
- ```python
- processed_template = template.replace("@@", _AT_ESCAPE_SENTINEL).replace("$$", _DOLLAR_ESCAPE_SENTINEL)
- ```
- (Replaces the current `processed_template = template` line.)
-- [x] After the three regex substitutions, before the `return`, append:
- ```python
- processed_template = processed_template.replace(_AT_ESCAPE_SENTINEL, "@").replace(_DOLLAR_ESCAPE_SENTINEL, "$")
- ```
-- [x] Remove the `# TODO: allow escape patterns` comment (line 73 in the current file).
-- [x] Re-run the test module. Expected: all Phase 1 tests now pass; legacy tests still pass.
-
----
-
-## Phase 4 — Workspace sanity check
-
-Look for any existing `.mthds` file in the workspace that would behave differently under the new rules.
-
-- [x] From the workspace root (`/Users/lchoquel/repos/Pipelex/`), run:
- ```bash
- grep -rn '@[a-zA-Z_][a-zA-Z_0-9.]*[[:space:]]*[({"'\'']' --include="*.mthds" . | head -50
- grep -rn '@@\|\$\$' --include="*.mthds" . | head -50
- ```
-- [x] For any hit, decide: accept (intended behavior change, the new heuristic does what the user wants), document, or flag back to the human.
-
----
-
-## Phase 5 — Lint and full tests
-
-- [x] `make agent-check` — Pyright, Ruff, Mypy, plxt all clean.
-- [x] Targeted test run for impacted areas:
- ```bash
- .venv/bin/pytest -n auto \
- -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" \
- -o log_level=WARNING --tb=short -q \
- tests/unit/pipelex/cogt/ tests/unit/pipelex/pipe_operators/
- ```
-- [x] `make agent-test` — full suite green.
-
----
-
-## Phase 6 — Docs & changelog
-
-- [x] `CHANGELOG.md`: add an entry under `## [Unreleased]` — `### Fixed` for the CSS collision, `### Added` for the `@@` / `$$` escapes. Cross-link `wip/template-preprocessor-css-collision.md`.
-- [x] Update MTHDS authoring guidance / agent prompts to mention `@@` and `$$`. Likely targets:
- - `pipelex/builder/` (in-repo authoring docs and any prompts that describe template syntax).
- - `mthds-plugins/` build/edit skill prompts (workspace neighbor — only update if the user explicitly asks to touch it; per `CLAUDE.md` plugin files belong to the published marketplace plugin).
-
-**Checkpoint B**: Phase 6 complete = ready to ship via `/release`.
-
----
-
-## Out of scope / explicit non-goals
-
-- No reserved CSS keyword list. The heuristic covers the realistic surface; the escape covers the rest.
-- No HTML/`<style>`-aware parsing of templates.
-- No syntax migration (`@{var}` etc.) — purely additive change.
-- No changes to the `replace_*` helper functions or to the surrounding `tag(...)` / `format()` filters.
-- No changes to the call sites of `preprocess_template` listed in the design doc — they all benefit transparently.
-
----
-
-## Phase 7 — `$` sigil footgun fixes (open, HIGH PRIORITY)
-
-### Why this phase
-
-A live audit of the inline `$` sigil surfaced four word-adjacent substitution bugs plus a consecutive-dots edge case. Root causes: the `$` regex is missing the `(?<!\w)` lookbehind that the `@` candidate pattern carries, and its identifier class (`[a-zA-Z0-9_.]+`) is too permissive (allows leading digit, consecutive dots, trailing dot inside the captured identifier).
-
-Concrete bugs (verified live against the current `preprocess_template`):
-
-| Input | Current output | Verdict |
-|---|---|---|
-| `micro$oft is a company` | `micro{{ oft\|format() }} is a company` | bug — mid-word substitution |
-| `user$host.com` | `user{{ host.com\|format() }}` | bug — mid-word substitution |
-| `P@ssw$rd123` | `P@ssw{{ rd123\|format() }}` | bug — mid-word substitution |
-| `a$b$c` | `a{{ b\|format() }}{{ c\|format() }}` | bug — back-to-back mid-word |
-| `$name..` | `{{ name.\|format() }}.` | bug — emits invalid Jinja |
-
-Regression guards (already correct under the current regex — must stay correct under the new one):
-
-- `see $name.` → `see {{ name|format() }}.` (sentence terminator)
-- `$user.name.first.` → `{{ user.name.first|format() }}.` (dotted access + terminator)
-- `run $(whatever) please` → unchanged (code-shape opener)
-- `price $10 today` → unchanged (dollar amount)
-- `$foo(bar)` / `$foo "bar"` → unchanged (code-shape lookahead)
-- `$$literal` → `$literal` (escape)
-
-Posture: **silent pass-through** for word-adjacent `$` (do not raise like `@` does). The `$` arm is best-effort inline by design — it already silently passes `$10`, `$(`, `${`, `$foo "..."`. Raising on `micro$oft` would be a noisier contract than warranted, and inconsistent with the existing `$` posture.
-
-### Target code
-
-In `pipelex/cogt/templating/template_preprocessor.py`:
-
-```python
-_DOLLAR_SIGIL_PATTERN = re.compile(
- r"(?<!\w)(\$)([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)*)(?![a-zA-Z0-9_])(?!\s*[({\"'])",
-)
-
-def _replace_dollar_sigil(match: Match[str]) -> str:
- variable: str = match.group(2)
- return f"{{{{ {variable}|format() }}}}"
-```
-
-Notes:
-
-- `(?<!\w)` mirrors the `@` candidate pattern's left guard. Fixes all four word-adjacent bugs.
-- The strict segmented identifier `[a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)*` rules out consecutive dots automatically and makes the prior `(?![0-9])` arm redundant — drop it.
-- The trailing-dot kludge in `_replace_dollar_sigil` (`if variable.endswith("."): variable = variable[:-1]; ...`) goes away. Under the strict shape a trailing `.` cannot end the captured identifier, so it naturally sits outside the match as literal punctuation. `$name.` still renders as `{{ name|format() }}.` because the `.` is simply not part of the match.
-
-### Phase 7.1 — TDD red
-
-All new tests go inside the existing `TestTemplatePreprocessor` class in `tests/unit/pipelex/cogt/templating/test_template_preprocessor.py`.
-
-- [x] `test_word_adjacent_dollar_pass_through` — parametrized over the four mid-word bug shapes (`micro$oft is a company`, `user$host.com`, `P@ssw$rd123`, `a$b$c`); each passes through unchanged.
-
-- [x] `test_dollar_consecutive_dots_pass_through` — `$name..` renders as `{{ name|format() }}..`. Both dots stay outside the match as literal punctuation.
-
-- [x] Re-ran the targeted module — new tests went red first (content mismatches), all existing `$`-related tests still passed.
-
-### Phase 7.2 — Implementation (green)
-
-- [x] In `pipelex/cogt/templating/template_preprocessor.py`:
- - [x] Replaced `_DOLLAR_SIGIL_PATTERN` with the strict segmented shape and added the `(?<!\w)` lookbehind.
- - [x] Dropped the `(?![0-9])` lookahead arm (redundant under the new identifier shape).
- - [x] Simplified `_replace_dollar_sigil` to the one-liner version. The `trailing_dot` branch is gone (unreachable under the new regex).
- - [x] Updated the `# Inline \`$\` sigil` comment to describe the new contract.
-- [x] Targeted module green: 121 passed.
-
-### Phase 7.3 — Workspace sanity check
-
-- [x] Ran the grep against `/Users/lchoquel/repos/Pipelex/`. Two hits in `mthds-ui/data/pipelines/pipeline_25/bundle.mthds` (lines 100 and 107), both of the shape `"...\n\n$classified.content"`. These are grep false positives — the `n` matched by `[[:alnum:]_]` belongs to the TOML `\n` escape; after TOML decoding the `$` is preceded by a real newline, so the new regex still rewrites correctly. No action.
-
-### Phase 7.4 — Lint, tests, docs
-
-- [x] `make agent-check` clean (Pyright, Ruff, Mypy, plxt).
-- [x] Targeted cogt test run: 584 passed in 9.06s.
-- [x] `make agent-test` — full suite green (5148 passed, 2 skipped, 3 xfailed). One pre-existing test (`test_template_field_accesses_extra_context`) had its template changed from `Q$quarter` to `, quarter $quarter` to match the new contract — see the "Deviations from Phase 7 plan" section near the top.
-- [x] `CHANGELOG.md` updated under `[Unreleased]` `### Fixed`.
-
-**Checkpoint C**: Phase 7 complete when the four word-adjacent-`$` tests and the consecutive-dots test are green, all legacy tests are green, `make agent-check` is clean, and `make agent-test` is green.
-
-### STOP HERE — checkpoint before Phase 8
-
-**Coding must stop after Phase 7 is delivered.** Do **not** start Phase 8 in the same session. After Checkpoint C is reached:
-
-1. Update the top "Status" block to record Phase 7 as landed (with the date), mirroring the style of the existing "strict line-bounded `@` rule landed" block.
-2. Note any open questions, deviations from this plan, or follow-up items in a brief retrospective near the top of this file (same style as the existing "Deviations from the original plan" section).
-3. Hand back to the human for review.
-
-Phase 8 (declared-inputs gating, formerly Phase 7) opens a new area of concern — function split, signature change, and call-site threading across 9 files — and should be picked up in a fresh session with this `TODOS.md` as the entry point.
-
----
-
-## Phase 8 — Declared-inputs gating for the `@`-sigil validator (landed 2026-05-13)
-
-### Why this phase
-
-The strict line-bounded `@` rule (Phases 1–6) turned every inline `@<ident>` into a load-time error to catch typos like `Extract from @invoice_text.` The same rule, however, makes inline `@media`, `@import`, `@keyframes`, `@font-face`, `@deprecated`, `@Override` — the CSS at-rules and code decorators that coding agents emit by default when writing HTML/CSS in PipeCompose templates — fail at load time too. Authors must escape every occurrence with `@@`, which is unfamiliar and surprising for both humans and agents. Phase 8 narrows the validator's "is this a real typo?" question by gating on the pipe's declared `inputs:`: an inline `@<ident>` raises only when `<ident>` (root segment for dotted paths) is one of the pipe's declared inputs. Everything else passes through silently, restoring CSS-friendliness without resurrecting the prior heuristic regex residuals (the rewriter stays strict line-bounded — alone-on-line `@var` still rewrites; nothing inline ever rewrites). LSP / `plxt check` still surface real typos at edit time because the inputs list is known at load time; no runtime threading required.
-
-### Target behavior
-
-```text
-pipe with inputs: { invoice_text: Text }:
-✗ Extract from @invoice_text. Done. → RAISES (invoice_text is declared → real typo)
-✓ Extract from $invoice_text. Done. → rewrites $invoice_text inline
-✓ @invoice_text → rewrites alone-on-line (unchanged)
-
-pipe with inputs: { invoice_text: Text } (no `media` input):
-✓ <style>@media (max-width: 820px) { ... }</style> → pass-through (media is NOT declared)
-✓ @media (max-width: 820px) { ... } → pass-through
-✓ @deprecated def foo(): → pass-through
-✓ @Override → pass-through (alone-on-line, but no Override input)
-
-pipe with inputs: { Override: Text } (unusual but possible):
-✗ @Override → RAISES at the alone-on-line shape too? NO —
- alone-on-line still rewrites, because that's the rewriter's strict success case.
- The gate only applies to the INLINE-candidate validator path.
-```
-
-### Target code
-
-**Signature change (`pipelex/cogt/templating/template_preprocessor.py`):**
-
-```python
-def preprocess_template(template: str, *, declared_inputs: set[str] | None = None) -> str: ...
-```
-
-When `declared_inputs is None`, the validator skips the inline-candidate check entirely (lenient pass-through). When a set is provided, the validator raises only when the candidate identifier's root segment is in the set.
-
-**Function split (for clean runtime path):**
-
-```python
-def validate_template_sigils(template: str, declared_inputs: set[str]) -> None:
- """Raise TemplateSigilSyntaxError when an inline @-candidate matches a declared input."""
-
-def rewrite_template_sigils(template: str) -> str:
- """Pure rewrite — applies @-line-bounded and $-inline substitutions. Does not validate."""
-
-def preprocess_template(template: str, *, declared_inputs: set[str] | None = None) -> str:
- """Convenience entry: normalize, escape sentinels, optionally validate, rewrite, restore."""
-```
-
-`render_template` (runtime, in `template_rendering.py:17`) calls `rewrite_template_sigils` directly — by the time we reach render, the template has already been validated at load time, so the runtime call only needs the rewrite step. This avoids threading `declared_inputs` through the rendering path.
-
-**Validator gating logic (`_validate_at_sigil_alone_on_line`):**
-
-```python
-def _validate_at_sigil_alone_on_line(template: str, declared_inputs: set[str] | None) -> None:
- if declared_inputs is None:
- return # lenient: no inputs context, nothing to gate against
- for line_index, line in enumerate(template.split("\n"), start=1):
- if "@" not in line:
- continue
- if _AT_SIGIL_PATTERN.fullmatch(line):
- continue
- candidate = _AT_CANDIDATE_PATTERN.search(line)
- if candidate is None:
- continue
- identifier = candidate.group(2)
- root = get_root_from_dotted_path(identifier)
- if root not in declared_inputs:
- continue # not a declared input → not a typo candidate → pass through
- # ... build current error message and raise ...
-```
-
-### Call sites to update
-
-| File | Line(s) | Inputs source | Pass |
-|---|---|---|---|
-| `pipe_llm_blueprint.py` | 41, 58 | `self.inputs: dict[str, str] \| None` | `set(self.inputs or {})` |
-| `pipe_compose_blueprint.py` | 106 | same | same |
-| `pipe_search_blueprint.py` | 44 | same | same |
-| `pipe_img_gen_blueprint.py` | 39 | same | same |
-| `pipe_compose_factory.py` | 67 | `inputs: InputStuffSpecs` | `set(inputs.variables)` |
-| `template_document_analyzer.py` | 48 | `input_specs: dict[str, str]` | `set(input_specs.keys())` |
-| `template_image_analyzer.py` | 64, 163 | same | same |
-| `construct_blueprint.py` | 249 | none (discovery function, not primary validation) | `None` |
-| `template_blueprint.py` | 25, 37 | none (base blueprint, no inputs context) | `None` |
-| `template_rendering.py` | 17 | runtime (already validated at load) | switch to `rewrite_template_sigils` |
-
-### Phase 8.1 — TDD red: tests for the new behavior
-
-All new tests go inside the existing `TestTemplatePreprocessor` class in `tests/unit/pipelex/cogt/templating/test_template_preprocessor.py`.
-
-- [x] **Parametrized success (lenient when no matching input):** inline `@media (...)`, `@import "..."`, `@keyframes`, `@font-face`, `@deprecated def foo():`, `@Override` — each with `declared_inputs=set()` or `declared_inputs={"other_var"}` — pass through unchanged, no raise. Implemented as `test_inline_at_passes_through_when_declared_inputs_empty` and `test_inline_at_passes_through_when_no_input_matches`.
-- [x] **Parametrized success (lenient when `declared_inputs=None`):** same set of inputs as above, called with no kwarg — pass through unchanged. Implemented as `test_inline_at_passes_through_when_declared_inputs_omitted`.
-- [x] **Parametrized error (strict when match):** inline `@invoice_text.` with `declared_inputs={"invoice_text"}` raises with the existing migration hint shape (line N, sigil+identifier, `$var` hint, `@@` escape hint). Implemented as `test_inline_at_raises_when_root_is_declared_input` plus the existing `test_strict_line_at_sigil_raises` parametrized suite (updated to pass `declared_inputs={expected_identifier}` per case).
-- [x] **Dotted-access gating:** inline `@user.profile.bio` with `declared_inputs={"user"}` raises; same template with `declared_inputs={"profile"}` passes through (gate is on root segment). Implemented as `test_inline_at_with_dotted_path_raises_when_root_is_declared_input` and `test_inline_at_with_dotted_path_passes_when_only_inner_segment_declared`.
-- [x] **Alone-on-line rewrites regardless of inputs:** `@invoice_text` alone on its line still rewrites whether or not `invoice_text` is in `declared_inputs` — the rewriter is strict and inputs-agnostic. Implemented as `test_alone_on_line_rewrites_regardless_of_declared_inputs` (parametrized across empty / matching / non-matching / optional shapes).
-- [x] **Update CSS at-rule tests:** renamed `test_css_at_rule_raises_under_strict_rule` → `test_css_at_rule_raises_when_keyword_is_declared_input` (parametrized; derives the identifier from the keyword and passes `declared_inputs={identifier}`). Same renaming pattern applied to `test_full_style_block_raises_*`, `test_triple_at_raises_*`, `test_css_dash_at_rule_raises_*`. The lenient-pass-through case is covered by the new parametrized tests above. `@@`-escape companion tests are unchanged.
-
-Blueprint integration tests (`tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py`):
-
-- [x] PipeLLM with `inputs: {"invoice_text": "Text"}` and `prompt = "Extract from @invoice_text."` — pydantic raises `ValueError` wrapping `TemplateSigilSyntaxError`. Already covered by the existing `test_validate_inputs_raises_for_inline_at_sigil_in_prompt`.
-- [x] PipeLLM with `inputs: {"page": "Page"}` and `prompt = "<style>@media (max-width: 820px) { ... }</style>\n...$page"` — loads cleanly (no `media` input → CSS passes through). Implemented as `test_inline_css_at_rule_passes_when_keyword_not_declared`.
-- [x] Same prompt with `inputs: {"media": "Text"}` — raises on `@media` (declared input now matches). Implemented as `test_inline_at_raises_when_input_collides_with_at_rule_keyword`.
-- [x] Inline Python decorator (`@deprecated def foo():`) loads cleanly when `deprecated` is not a declared input. Implemented as `test_inline_python_decorator_passes_through`.
-
-Run target:
-
-```bash
-.venv/bin/pytest -q tests/unit/pipelex/cogt/templating/test_template_preprocessor.py
-.venv/bin/pytest -q tests/unit/pipelex/pipe_operators/pipe_llm/test_pipe_llm_blueprint.py
-```
-
-Expected state after Phase 8.1: new tests fail (red), all existing tests still pass (the lenient-when-`None` default keeps current call sites' behavior intact for legacy tests that don't pass inputs).
-
-> **Deviation:** TDD was followed in spirit but not literally — under the new lenient-when-`None` default, the existing strict-rule tests that called `preprocess_template(template)` (no kwarg) would *stop raising*, breaking the "all existing tests still pass" claim. The fix was to update those parametrized tests to pass `declared_inputs={expected_identifier}` so they preserve their raise expectation under the new gate. New tests for the lenient cases were added alongside.
-
-### Phase 8.2 — Implementation (green)
-
-- [x] In `pipelex/cogt/templating/template_preprocessor.py`:
- - [x] Added `get_root_from_dotted_path` import (from `pipelex.tools.misc.string_utils`).
- - [x] Changed `_validate_at_sigil_alone_on_line` to accept `declared_inputs: set[str]` (not `Optional`) — the `None`-gate is now handled at the `preprocess_template` boundary so internal helpers don't need to re-check.
- - [x] Split `preprocess_template` into `validate_template_sigils` (validator wrapper) and `rewrite_template_sigils` (escape → normalize → substitute → restore, no validation). Kept `preprocess_template` as a thin convenience entry that skips validation when `declared_inputs is None`. Extracted `_normalize_and_escape` to dedupe the line-ending + sentinel prelude across the two halves.
- - [x] Updated the module docstring + function docstrings to describe the gating behavior.
-- [x] Threaded `declared_inputs` through the 7 call sites that have inputs context (`pipe_llm_blueprint`, `pipe_compose_blueprint`, `pipe_search_blueprint`, `pipe_img_gen_blueprint`, `pipe_compose_factory`, `template_document_analyzer`, `template_image_analyzer`). The 2 call sites without inputs context (`construct_blueprint`, `template_blueprint`) keep the zero-arg form and benefit from the lenient default — no explicit `None` pass-through needed.
-- [x] In `pipelex/cogt/templating/template_rendering.py`: switched from `preprocess_template(template)` to `rewrite_template_sigils(template)` — render path no longer re-validates.
-- [x] Dedup'd `input_names` / `declared_inputs` in `pipe_llm_blueprint`, `pipe_search_blueprint`, `pipe_img_gen_blueprint` — single `declared_inputs` variable threads through both the preprocess call and the downstream missing/unused-inputs checks. Pure code clarity, no behavior change.
-
-Run target same as Phase 8.1. Expected: all Phase 8.1 tests green, all legacy tests still green.
-
-### Phase 8.3 — Workspace sanity check
-
-- [x] Ran the workspace `.mthds` grep from `/Users/lchoquel/repos/Pipelex/`. After excluding TOML model references (`model = "@default-..."` — these never go through `preprocess_template`) and false positives from string-literal contexts, the surviving hits are:
- - `pipelex-cookbook/examples/c_advanced/crewai_with_pipelex_tools/research_report.mthds:46` — inline `Question: @question` where `question` IS a declared input (line 40).
- - `pipelex-cookbook/examples/b_basics/document_extract/answer_from_documents/bundle.mthds:150,261` — same shape (`Question: @question` with `question` declared on the surrounding pipe).
- - `internal-tools/wip/carol-sessions/2026-04-07T06-11-46_dev_old-plugin/mthds-wip/haiku_writer/bundle.mthds:13` — `Idea: @idea` with `idea` declared.
-- [x] Verified: each hit is a genuine pre-existing typo that already raised under Phases 1–6 and continues to raise under Phase 8 (the declared-inputs gate intentionally fires here). They live in sibling repos (`pipelex-cookbook`, `internal-tools/wip/`), out of this PR's scope — flagged as follow-up cleanup in those repos rather than fixed here.
-- [x] Confirmed `@@`-escape workarounds still work under the new gating. None of the CSS-case workarounds needed to be added in `pipelex/` itself (the Phase 1–6 fix only touched code/tests, not `.mthds` files in this repo). No follow-up cleanup list required.
-
-### Phase 8.4 — Lint, tests, docs
-
-- [x] `make agent-check` clean (Pyright, Ruff, Mypy, plxt).
-- [x] Targeted test run for `tests/unit/pipelex/cogt/ tests/unit/pipelex/pipe_operators/ tests/integration/pipelex/pipes/`: 994 passed, 1 xfailed (pre-existing unrelated).
-- [x] `make agent-test` — full suite green.
-- [x] `CHANGELOG.md` updated under `[Unreleased]` `### Changed`. The entry describes the gating contract, the new `validate_template_sigils` / `rewrite_template_sigils` split, the call-site threading, and the `render_template` switch to `rewrite_template_sigils` at runtime.
-- [x] MTHDS authoring guidance updated in `pipelex/builder/pipe/pipe_compose_spec.py` (the only file in `pipelex/builder/` mentioning `@@`-escapes): dropped the `@@font-face` / `@@media` CSS example, kept the `@@`/`$$` mention as the literal-character opt-out, and added a note that inline CSS at-rules now pass through silently unless their keyword collides with a declared input.
-
-**Checkpoint D**: Phase 8 complete when CSS at-rules in PipeCompose HTML templates load without `@@` escapes (where they don't collide with declared inputs), AND typo-shape inline `@<declared>` still raises with the migration hint at load time.
-
-### Open decisions for Phase 8
-
-These are small enough to default and proceed; flag objections before locking in.
-
-1. **`declared_inputs=None` default = lenient (no raise).** Reasoning: callers without inputs context (the base `TemplateBlueprint`, the `ConstructBlueprint` discovery function) can't make the gate decision; the strict behavior is hostile in those contexts. Alternative: default-strict — every call site that doesn't pass inputs gets the current Phase 1–6 behavior. Less friendly to the discovery / runtime paths.
-2. **Gate on root segment of dotted identifier.** `@user.profile.bio` is gated on `user`. Matches how Pipelex's input resolution and `detect_jinja2_required_variables` treat dotted paths.
-3. **Validator does NOT consult Jinja2-detected variables.** It only consults `declared_inputs`. Reasoning: by the time we validate, we haven't rewritten yet, so Jinja2 can't see `@var` as a variable; conversely, if the author wrote `{{ media }}` directly in Jinja, that's a separate concern. Keep the gate narrow and explicit.
-4. **Reserved-name handling unchanged.** Internal vars (`preliminary_text`, `place_holder`, `_-prefixed`) are still filtered downstream in `pipe_llm_blueprint.py:78`. The validator doesn't need to know about them — they're not realistic typo candidates inline.
-
-### Out of scope for Phase 8
-
-- **Auto-cleanup of existing `@@`-escapes in workspace `.mthds` files.** Phase 4 of the line-bounded plan added some; they keep working under the new gating. A follow-up PR can strip the ones that are no longer needed.
-- **Runtime gating on actual working-memory contents.** The proposal we considered first (check the candidate against the runtime context) would catch typos that aren't declared as inputs. Rejected because it moves validation from load time to render time and breaks LSP / `plxt check` integration. The declared-inputs gate is the load-time approximation.
-- **Heuristic shape detection inside the validator.** No `(?!\s*[({"'])` lookahead — the declared-inputs gate fully replaces shape-based heuristics for the `@` arm.
-- **Changes to the `$` sigil contract.** `$` is unchanged.
diff --git a/docs/features/gateway.md b/docs/features/gateway.md
index 8cd65315a..5172fe784 100644
--- a/docs/features/gateway.md
+++ b/docs/features/gateway.md
@@ -51,3 +51,12 @@ Run open-source models through dedicated providers:
Routing profiles control which backend handles each model. Define pattern-based routes, defaults, and fallback orders in `routing_profiles.toml`. Switch a pipeline from one provider to another — or from Pipelex Gateway to your own keys — without changing the method definition.
See [Configure AI Providers](../get-started/configure-ai-providers.md) for setup details.
+
+## Offline Behavior
+
+The Gateway fetches its model catalog from a remote config service. Pipelex stays usable when that service is briefly unreachable:
+
+- **Gateway disabled (Bring Your Own Keys)** — no remote fetch is attempted at all. Setup, validation, and dry-runs are fully offline.
+- **Gateway enabled** — Pipelex primes an on-disk copy of the remote config at `~/.pipelex/cache/remote_config.json` on every successful fetch and during `pipelex init` while online. If a later fetch fails, it falls back to this cache so setup, validation, and `--dry-run` still complete. When the cache is in use, a `RemoteConfigStale` warning is emitted (the agent CLI surfaces it on the JSON `warnings` array).
+
+Only the actual inference call needs the network at runtime — offline support covers setup and dry-run, not live model calls. If the Gateway is enabled, the fetch fails, and no cache has ever been primed, setup raises `RemoteConfigUnavailableError`: run `pipelex init` while online to prime the cache, or disable `pipelex_gateway` for permanent offline (BYOK) operation.
diff --git a/docs/tools/cli/agent-cli.md b/docs/tools/cli/agent-cli.md
index 07a41a58c..dc7a9c018 100644
--- a/docs/tools/cli/agent-cli.md
+++ b/docs/tools/cli/agent-cli.md
@@ -113,6 +113,20 @@ Commands use different stdout formats depending on their purpose:
JSON commands return the result object directly. They are not wrapped in a `status` or `data` envelope.
+**Warnings** — non-fatal setup conditions are surfaced on a top-level `warnings` array of the success envelope, so consumers don't have to parse stderr. The field is absent when there is nothing to report. Each entry is an object with a `type` and a `message`:
+
+```json
+{
+ "success": true,
+ "target_dir": "/path/to/.pipelex",
+ "warnings": [
+ { "type": "RemoteConfigStale", "message": "Using a cached gateway config; run `pipelex init` while online to refresh." }
+ ]
+}
+```
+
+`RemoteConfigStale` is emitted when the gateway is enabled but the remote config service is unreachable and Pipelex falls back to its on-disk cache (offline mode).
+
**Error** — written to stderr:
```json
diff --git a/docs/tools/cli/init.md b/docs/tools/cli/init.md
index b74d08882..b49add305 100644
--- a/docs/tools/cli/init.md
+++ b/docs/tools/cli/init.md
@@ -66,6 +66,9 @@ This command creates or resets a Pipelex config directory with:
- `deck/` - AI model aliases and presets
- **telemetry.toml** - Telemetry and observability settings
+!!! note "Offline cache priming"
+ When Pipelex Gateway is enabled, `pipelex init` also primes an on-disk copy of the gateway's remote config at `~/.pipelex/cache/remote_config.json`. This lets later setup, validation, and dry-runs work even when the remote config service is briefly unreachable. If you run `pipelex init` while offline, it prints a yellow warning and skips priming — re-run it online to enable offline dry-runs.
+
## Interactive Setup Flow
When you run `pipelex init`, Pipelex can guide you through:
diff --git a/pipelex/cli/agent_cli/commands/agent_cli_factory.py b/pipelex/cli/agent_cli/commands/agent_cli_factory.py
index 59d102d41..d531e44ff 100644
--- a/pipelex/cli/agent_cli/commands/agent_cli_factory.py
+++ b/pipelex/cli/agent_cli/commands/agent_cli_factory.py
@@ -1,18 +1,20 @@
"""Factory function for agent CLI commands -- JSON-only error output."""
+import warnings
from pathlib import Path
import typer
-from pipelex.cli.agent_cli.commands.agent_output import agent_error
-from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
+from pipelex.cli.agent_cli.commands.agent_output import agent_error, record_setup_warning
+from pipelex.cogt.exceptions import GatewayUnknownModelError, ModelDeckPresetValidatonError
from pipelex.pipelex import Pipelex
from pipelex.system.pipelex_service.exceptions import (
GatewayApiKeyMissingError,
GatewayDoNotTrackConflictError,
GatewayTermsNotAcceptedError,
InferenceSetupRequiredError,
- RemoteConfigFetchError,
+ RemoteConfigStaleWarning,
+ RemoteConfigUnavailableError,
RemoteConfigValidationError,
)
from pipelex.system.runtime import IntegrationMode
@@ -53,9 +55,16 @@ def make_pipelex_for_agent_cli(
or if inference setup is required (after printing markdown to stdout).
"""
try:
- pipelex_instance = Pipelex.make(
- integration_mode=IntegrationMode.CLI, library_dirs=library_dirs, needs_inference=needs_inference, needs_model_specs=needs_model_specs
- )
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always", RemoteConfigStaleWarning)
+ pipelex_instance = Pipelex.make(
+ integration_mode=IntegrationMode.CLI, library_dirs=library_dirs, needs_inference=needs_inference, needs_model_specs=needs_model_specs
+ )
+ # Surface a structured ``RemoteConfigStale`` entry so JSON consumers can react to
+ # stale-cache operation without parsing stderr.
+ for item in caught:
+ if issubclass(item.category, RemoteConfigStaleWarning):
+ record_setup_warning({"type": "RemoteConfigStale", "message": str(item.message)})
except InferenceSetupRequiredError:
print(
"# First-time inference setup required\n"
@@ -75,10 +84,18 @@ def make_pipelex_for_agent_cli(
agent_error(exc.message, "GatewayApiKeyMissingError", cause=exc)
except GatewayDoNotTrackConflictError as exc:
agent_error(exc.message, "GatewayDoNotTrackConflictError", cause=exc)
- except RemoteConfigFetchError as exc:
- agent_error(exc.message, "RemoteConfigFetchError", cause=exc)
+ except RemoteConfigUnavailableError as exc:
+ agent_error(exc.message, "RemoteConfigUnavailableError", cause=exc)
except RemoteConfigValidationError as exc:
agent_error(exc.message, "RemoteConfigValidationError", cause=exc)
+ except GatewayUnknownModelError as exc:
+ agent_error(
+ exc.message,
+ "GatewayUnknownModelError",
+ cause=exc,
+ model_name=exc.model_name,
+ source=exc.source,
+ )
except ModelDeckPresetValidatonError as exc:
agent_error(
exc.message,
diff --git a/pipelex/cli/agent_cli/commands/agent_output.py b/pipelex/cli/agent_cli/commands/agent_output.py
index 848fc9824..dccaa4f9e 100644
--- a/pipelex/cli/agent_cli/commands/agent_output.py
+++ b/pipelex/cli/agent_cli/commands/agent_output.py
@@ -2,7 +2,7 @@
import sys
import traceback
-from typing import Any, NoReturn
+from typing import Any, NoReturn, cast
import typer
@@ -11,6 +11,32 @@ from pipelex.pipeline.validate_bundle import ValidateBundleError
from pipelex.tools.misc.json_utils import clean_json_dumps
from pipelex.types import StrEnum
+# Module-level capture for setup-time warnings (currently used by RemoteConfigStaleWarning).
+# The agent CLI factory writes here when it catches a stale-cache warning during ``Pipelex.make``
+# and ``agent_success`` reads back here to attach the ``warnings`` field to the envelope so
+# machine consumers see the provenance.
+_CAPTURED_WARNINGS: list[dict[str, Any]] = []
+
+
+def record_setup_warning(warning_payload: dict[str, Any]) -> None:
+ """Stash a structured warning for inclusion in the next ``agent_success`` envelope.
+
+ Callers pass a dict shaped like ``{"type": "RemoteConfigStale", "message": "..."}``;
+ the contents are surfaced verbatim by ``agent_success``.
+ """
+ _CAPTURED_WARNINGS.append(warning_payload)
+
+
+def consume_setup_warnings() -> list[dict[str, Any]]:
+ """Drain the captured warnings buffer. Returns whatever was recorded and clears state.
+
+ Called once per envelope so successive commands within a single Python process don't
+ re-emit yesterday's warnings.
+ """
+ drained = list(_CAPTURED_WARNINGS)
+ _CAPTURED_WARNINGS.clear()
+ return drained
+
class CliOutputFormat(StrEnum):
JSON = "json"
@@ -48,10 +74,16 @@ AGENT_ERROR_HINTS: dict[str, str] = {
"GatewayApiKeyMissingError": "Set the PIPELEX_GATEWAY_API_KEY environment variable, or disable pipelex_gateway in backends.toml",
"GatewayDoNotTrackConflictError": "Unset the DO_NOT_TRACK environment variable, or disable pipelex_gateway in backends.toml",
"BinaryNotFoundError": "Install pipelex-tools: uv tool install pipelex-tools",
- "RemoteConfigFetchError": "Check internet connection and firewall settings, or disable pipelex_gateway in backends.toml",
+ "RemoteConfigUnavailableError": (
+ "Run `pipelex init` while online to prime the cache, or disable pipelex_gateway in backends.toml to operate offline (BYOK)"
+ ),
"RemoteConfigValidationError": (
"This is a server-side issue; report it on Discord/GitHub. Disable pipelex_gateway in backends.toml as a workaround"
),
+ "GatewayUnknownModelError": (
+ "The deck references a model the gateway doesn't expose. If the source is `cached`, run `pipelex init` while online to refresh; "
+ "otherwise update the deck or check the model name."
+ ),
# API runner errors
"ClientAuthenticationError": "Run 'pipelex-agent doctor' to check credentials, or set the PIPELEX_API_KEY environment variable",
"PipelineRequestError": "Check that pipe_code or mthds_contents is provided",
@@ -69,7 +101,6 @@ AGENT_ERROR_HINTS: dict[str, str] = {
}
RETRYABLE_ERROR_TYPES: set[str] = {
- "RemoteConfigFetchError",
"PipeOperatorModelAvailabilityError",
}
@@ -99,9 +130,10 @@ AGENT_ERROR_DOMAINS: dict[str, str] = {
"GatewayTermsNotAcceptedError": "config",
"GatewayApiKeyMissingError": "config",
"GatewayDoNotTrackConflictError": "config",
- "RemoteConfigFetchError": "config",
+ "RemoteConfigUnavailableError": "config",
"BinaryNotFoundError": "config",
"RemoteConfigValidationError": "config",
+ "GatewayUnknownModelError": "config",
"InitConfigError": "config",
# runtime = execution failure
"PipelineExecutionError": "runtime",
@@ -211,10 +243,22 @@ def agent_error(message: str, error_type: str, cause: BaseException | None = Non
def agent_success(result: dict[str, Any]) -> None:
"""Print a structured JSON success result to stdout.
+ Any pending setup warnings (e.g. stale gateway cache) recorded via ``record_setup_warning``
+ are drained into a top-level ``warnings`` array on the envelope so machine consumers can
+ surface them without parsing stderr. Callers may pre-populate ``result["warnings"]`` (must
+ be a list) — the captured ones are appended. The caller's ``result`` dict is NOT mutated;
+ a copy is taken before merging.
+
Args:
result: Dictionary to serialize as JSON.
"""
- print(clean_json_dumps(result, indent=2))
+ captured = consume_setup_warnings()
+ envelope: dict[str, Any] = result
+ if captured:
+ existing_raw = result.get("warnings")
+ existing_warnings: list[Any] = cast("list[Any]", existing_raw) if isinstance(existing_raw, list) else []
+ envelope = {**result, "warnings": [*existing_warnings, *captured]}
+ print(clean_json_dumps(envelope, indent=2))
def extract_validation_errors(exc: ValidateBundleError) -> list[dict[str, Any]]:
diff --git a/pipelex/cli/agent_cli/commands/init_cmd.py b/pipelex/cli/agent_cli/commands/init_cmd.py
index f8fd4de3e..53ecaa5c8 100644
--- a/pipelex/cli/agent_cli/commands/init_cmd.py
+++ b/pipelex/cli/agent_cli/commands/init_cmd.py
@@ -11,6 +11,7 @@ from tomlkit import table
from pipelex.cli.agent_cli.commands.agent_output import agent_error, agent_success
from pipelex.cli.commands.init.backends import get_selected_backend_keys, update_backends_in_toml
+from pipelex.cli.commands.init.command import attempt_prime_remote_config_cache
from pipelex.cli.commands.init.config_files import init_config
from pipelex.cli.commands.init.ui.backends_ui import get_backend_options_from_toml
from pipelex.cogt.model_backends.backend import PipelexBackend
@@ -368,17 +369,26 @@ def agent_init_cmd(
# Step 4: Mark inference setup as completed
update_inference_setup_completed(completed=True, config_dir=config_manager.global_config_dir)
+ # Step 5: Prime the remote-config cache so subsequent offline dry-runs can fall back.
+ # No-op when gateway is disabled or terms have not been accepted; surfaces failure as
+ # structured fields on the success envelope rather than crashing init. We forward the
+ # init target directory so the gateway-enabled check inspects the backends.toml we
+ # just wrote, not a sibling layered config.
+ priming_result = attempt_prime_remote_config_cache(target_config_dir=target_dir)
+ result_payload: dict[str, Any] = {
+ "success": True,
+ "target_dir": str(target_dir),
+ "config_files_copied": config_files_copied,
+ "backends_enabled": backends_enabled,
+ "routing_profile": routing_profile,
+ "inference_setup_completed": True,
+ "cache_primed": priming_result.primed,
+ }
+ if priming_result.error_message is not None:
+ result_payload["cache_priming_error"] = priming_result.error_message
+
# Output result
- agent_success(
- {
- "success": True,
- "target_dir": str(target_dir),
- "config_files_copied": config_files_copied,
- "backends_enabled": backends_enabled,
- "routing_profile": routing_profile,
- "inference_setup_completed": True,
- }
- )
+ agent_success(result_payload)
except typer.Exit:
raise
diff --git a/pipelex/cli/cli_factory.py b/pipelex/cli/cli_factory.py
index 921f2ed79..3b8650929 100644
--- a/pipelex/cli/cli_factory.py
+++ b/pipelex/cli/cli_factory.py
@@ -7,20 +7,21 @@ from pipelex.cli.error_handlers import (
handle_gateway_api_key_missing_error,
handle_gateway_do_not_track_conflict_error,
handle_gateway_terms_not_accepted_error,
+ handle_gateway_unknown_model_error,
handle_inference_setup_required_error,
handle_model_deck_preset_error,
- handle_remote_config_fetch_error,
+ handle_remote_config_unavailable_error,
handle_remote_config_validation_error,
handle_telemetry_config_validation_error,
)
-from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
+from pipelex.cogt.exceptions import GatewayUnknownModelError, ModelDeckPresetValidatonError
from pipelex.pipelex import Pipelex
from pipelex.system.pipelex_service.exceptions import (
GatewayApiKeyMissingError,
GatewayDoNotTrackConflictError,
GatewayTermsNotAcceptedError,
InferenceSetupRequiredError,
- RemoteConfigFetchError,
+ RemoteConfigUnavailableError,
RemoteConfigValidationError,
)
from pipelex.system.runtime import IntegrationMode
@@ -70,9 +71,11 @@ def make_pipelex_for_cli(
handle_gateway_api_key_missing_error(exc)
except GatewayDoNotTrackConflictError as exc:
handle_gateway_do_not_track_conflict_error(exc)
- except RemoteConfigFetchError as exc:
- handle_remote_config_fetch_error(exc)
+ except RemoteConfigUnavailableError as exc:
+ handle_remote_config_unavailable_error(exc)
except RemoteConfigValidationError as exc:
handle_remote_config_validation_error(exc)
+ except GatewayUnknownModelError as exc:
+ handle_gateway_unknown_model_error(exc)
except ModelDeckPresetValidatonError as exc:
handle_model_deck_preset_error(exc, context=context)
diff --git a/pipelex/cli/commands/doctor_cmd.py b/pipelex/cli/commands/doctor_cmd.py
index 45fc32673..20da8a9ec 100644
--- a/pipelex/cli/commands/doctor_cmd.py
+++ b/pipelex/cli/commands/doctor_cmd.py
@@ -7,7 +7,7 @@ import io
import os
import sys
from pathlib import Path
-from typing import Any
+from typing import TYPE_CHECKING, Any
from pydantic import BaseModel, ValidationError
from rich.panel import Panel
@@ -20,7 +20,17 @@ from pipelex.cli.commands.init.command import init_cmd
from pipelex.cli.commands.init.config_files import init_config
from pipelex.cli.commands.init.ui.types import InitFocus
from pipelex.cli.commands.update_cmd import update_cmd
-from pipelex.cogt.exceptions import InferenceBackendLibraryError
+from pipelex.cogt.exceptions import (
+ GatewayUnknownModelError,
+ InferenceBackendCredentialsError,
+ InferenceBackendLibraryError,
+ InferenceBackendLibraryNotFoundError,
+ InferenceBackendLibraryValidationError,
+ ModelDeckNotFoundError,
+ ModelDeckValidationError,
+ RoutingProfileDisabledBackendError,
+ RoutingProfileLibraryNotFoundError,
+)
from pipelex.cogt.model_backends.backend_credentials import BackendCredentialsErrorMsgFactory
from pipelex.cogt.model_backends.backend_library import BackendCredentialsReport, InferenceBackendLibrary
from pipelex.cogt.model_backends.gateway_config import GatewayConfig
@@ -33,7 +43,10 @@ from pipelex.kit.paths import get_kit_configs_dir
from pipelex.system.configuration.config_loader import config_manager
from pipelex.system.configuration.configs import PipelexConfig
from pipelex.system.environment import get_optional_env
-from pipelex.system.pipelex_service.exceptions import RemoteConfigFetchError, RemoteConfigValidationError
+from pipelex.system.pipelex_service.exceptions import (
+ RemoteConfigUnavailableError,
+ RemoteConfigValidationError,
+)
from pipelex.system.pipelex_service.pipelex_service_config import (
is_pipelex_gateway_enabled,
load_pipelex_service_config_if_exists,
@@ -46,6 +59,9 @@ from pipelex.tools.misc.placeholder import value_is_placeholder
from pipelex.tools.misc.toml_utils import TomlError, load_toml_from_path
from pipelex.tools.secrets.env_secrets_provider import EnvSecretsProvider
+if TYPE_CHECKING:
+ from pipelex.system.pipelex_service.types import RemoteConfigSource
+
class ConfigLocationInfo(BaseModel):
"""Information about the resolved configuration location."""
@@ -730,6 +746,7 @@ def check_models(config_dir: Path | None = None) -> tuple[bool, str, dict[str, B
# Fetch gateway model specs if Gateway is enabled
gateway_config: GatewayConfig | None = None
+ gateway_config_source: RemoteConfigSource | None = None
if is_pipelex_gateway_enabled():
pipelex_service_config = load_pipelex_service_config_if_exists(config_dir=config_manager.global_config_dir)
if pipelex_service_config is None:
@@ -737,12 +754,14 @@ def check_models(config_dir: Path | None = None) -> tuple[bool, str, dict[str, B
if not pipelex_service_config.agreement.terms_accepted:
return False, "Pipelex Gateway is enabled but terms have not been accepted", backend_file_reports
try:
- remote_config = RemoteConfigFetcher.fetch_remote_config()
+ result = RemoteConfigFetcher.fetch_remote_config()
+ remote_config = result.config
+ gateway_config_source = result.source
gateway_config = GatewayConfig(
model_specs=remote_config.backend_model_specs,
aws_region=remote_config.aws_region,
)
- except (RemoteConfigFetchError, RemoteConfigValidationError) as exc:
+ except (RemoteConfigUnavailableError, RemoteConfigValidationError) as exc:
return False, f"Failed to fetch Pipelex Gateway remote configuration: {exc}", backend_file_reports
models_manager = ModelManager()
@@ -751,6 +770,7 @@ def check_models(config_dir: Path | None = None) -> tuple[bool, str, dict[str, B
models_manager.setup(
secrets_provider=secrets_provider,
gateway_config=gateway_config,
+ gateway_config_source=gateway_config_source,
)
models_manager.validate_model_deck()
except InferenceBackendLibraryError as exc:
@@ -764,7 +784,16 @@ def check_models(config_dir: Path | None = None) -> tuple[bool, str, dict[str, B
backend_file_reports[backend_name].is_valid = False
backend_file_reports[backend_name].error_message = error_str
return False, f"Error checking models: {exc}", backend_file_reports
- except Exception as exc:
+ except (
+ RoutingProfileLibraryNotFoundError,
+ InferenceBackendLibraryNotFoundError,
+ ModelDeckNotFoundError,
+ RoutingProfileDisabledBackendError,
+ InferenceBackendLibraryValidationError,
+ ModelDeckValidationError,
+ InferenceBackendCredentialsError,
+ GatewayUnknownModelError,
+ ) as exc:
return False, f"Error checking models: {exc}", backend_file_reports
return True, "Models are valid", backend_file_reports
diff --git a/pipelex/cli/commands/init/command.py b/pipelex/cli/commands/init/command.py
index 7a20d51b3..385f3bba2 100644
--- a/pipelex/cli/commands/init/command.py
+++ b/pipelex/cli/commands/init/command.py
@@ -5,6 +5,7 @@ import shutil
from pathlib import Path
import typer
+from pydantic import BaseModel, ConfigDict, ValidationError
from rich.console import Console
from rich.markup import escape
from rich.prompt import Confirm
@@ -30,16 +31,118 @@ from pipelex.cogt.model_backends.backend import PipelexBackend
from pipelex.cogt.models.deck_manifest import compute_kit_manifest, write_manifest
from pipelex.hub import get_console
from pipelex.kit.paths import get_kit_configs_dir
-from pipelex.system.configuration.config_loader import config_manager
+from pipelex.system.configuration.config_loader import BACKENDS_FILE_NAME, INFERENCE_DIR_NAME, config_manager
+from pipelex.system.pipelex_service.exceptions import RemoteConfigUnavailableError
from pipelex.system.pipelex_service.pipelex_service_agreement import update_service_terms_acceptance
from pipelex.system.pipelex_service.pipelex_service_config import (
is_pipelex_gateway_enabled,
load_pipelex_service_config_if_exists,
)
+from pipelex.system.pipelex_service.remote_config_cache import RemoteConfigCache
+from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
from pipelex.system.telemetry.telemetry_config import TELEMETRY_CONFIG_FILE_NAME
from pipelex.tools.misc.file_utils import path_exists
+class CachePrimingResult(BaseModel):
+ """Outcome of an attempt to prime the on-disk remote-config cache.
+
+ ``primed`` is ``True`` only when a fresh fetch succeeded *and* a usable cache exists on disk
+ afterwards. ``error_message`` is populated when the fetch was attempted but failed (offline at
+ init time) *or* when the fetch succeeded but the cache could not be persisted, read back, or
+ validated as a usable ``RemoteConfig`` (e.g. a read-only or full cache directory). ``None``
+ means priming was skipped (gateway disabled or terms not accepted) or that it succeeded.
+ """
+
+ model_config = ConfigDict(extra="forbid", strict=True)
+
+ primed: bool
+ error_message: str | None = None
+
+
+def attempt_prime_remote_config_cache(target_config_dir: Path | None = None) -> CachePrimingResult:
+ """Prime the on-disk remote-config cache so later offline runs can fall back to it.
+
+ Pure-logic variant: no I/O on the way out, so both the interactive (`pipelex init`) and
+ machine (`pipelex-agent init`) surfaces can decide how to surface failure (Rich warning vs
+ structured JSON field).
+
+ Skipped (``primed=False, error_message=None``) when:
+ - the gateway is disabled in ``backends.toml`` (BYOK has nothing to cache), or
+ - gateway terms have not been accepted (we cannot fetch without consent).
+
+ Always passes ``require_fresh=True`` to the fetcher: priming's only job is to write a fresh
+ cache, so silently accepting an existing cached fallback would be a misleading success. When
+ the network is unreachable and only a stale cache exists, the fetcher raises
+ ``RemoteConfigUnavailableError`` and we surface that as ``error_message`` — the stale cache
+ on disk is left intact so subsequent offline dry-runs can still fall back to it.
+
+ ``RemoteConfigValidationError`` is intentionally NOT caught: a server-side schema break is a
+ real bug (we control the back-office) and should surface loudly rather than be hidden by the
+ priming step.
+
+ Args:
+ target_config_dir: When set, read the ``backends.toml`` *at that directory* to decide
+ whether the gateway is enabled. ``pipelex init`` and ``pipelex init --local``
+ target different ``.pipelex/`` directories — using the layered/project-preferred
+ config here would let priming branch on the wrong file. ``None`` (default) falls
+ back to the layered path. The terms-acceptance check always reads the *global*
+ ``pipelex_service.toml`` by design.
+ """
+ if target_config_dir is not None:
+ backends_file_path = target_config_dir / INFERENCE_DIR_NAME / BACKENDS_FILE_NAME
+ else:
+ backends_file_path = None
+ if not is_pipelex_gateway_enabled(backends_file_path=backends_file_path):
+ return CachePrimingResult(primed=False)
+
+ service_config = load_pipelex_service_config_if_exists(config_dir=config_manager.global_config_dir)
+ if service_config is None or not service_config.agreement.terms_accepted:
+ return CachePrimingResult(primed=False)
+
+ try:
+ RemoteConfigFetcher.fetch_remote_config(require_fresh=True)
+ except RemoteConfigUnavailableError as exc:
+ return CachePrimingResult(primed=False, error_message=str(exc))
+
+ # A successful fetch does NOT guarantee a usable cache. ``RemoteConfigFetcher`` treats the
+ # cache write as opportunistic and swallows OSErrors (read-only / full cache dir) with only a
+ # stderr warning. And even when a file is written, ``RemoteConfigCache.load()`` validates only
+ # the cache *wrapper* — a malformed inner ``raw_config`` would still pass it but break later
+ # offline runs (see the matching check in ``RemoteConfigFetcher.fetch_remote_config``). Verify
+ # both: the wrapper loads AND its payload re-validates as a ``RemoteConfig``. Otherwise priming
+ # would misreport success and later offline runs would hit ``RemoteConfigUnavailableError``.
+ cached = RemoteConfigCache.load()
+ if cached is not None:
+ try:
+ cached.to_remote_config()
+ except ValidationError:
+ cached = None
+ if cached is None:
+ msg = (
+ f"Remote config was fetched but the cache at {RemoteConfigCache.cache_path()} "
+ "could not be written, read back, or validated; offline dry-runs will not have a "
+ "fallback. Check that the directory is writable."
+ )
+ return CachePrimingResult(primed=False, error_message=msg)
+ return CachePrimingResult(primed=True)
+
+
+def prime_remote_config_cache(console: Console, target_config_dir: Path | None = None) -> None:
+ """Interactive-surface wrapper around :func:`attempt_prime_remote_config_cache`.
+
+ Prints a yellow warning to the console when a fetch was attempted and failed; otherwise
+ silent. Used by ``pipelex init`` so the user knows priming didn't happen and how to retry.
+
+ ``target_config_dir`` is forwarded so the gateway-enabled check inspects the directory
+ being initialized rather than the layered config (see ``attempt_prime_remote_config_cache``).
+ """
+ result = attempt_prime_remote_config_cache(target_config_dir=target_config_dir)
+ if result.error_message is not None:
+ console.print(f"[yellow]⚠ Could not prime remote config cache: {escape(result.error_message)}[/yellow]")
+ console.print("[dim]Re-run 'pipelex init' while online to prime the cache for offline dry-runs.[/dim]")
+
+
def _check_gateway_terms_if_needed(console: Console, backends_toml_path: str) -> None:
"""Check if gateway is enabled and terms not yet accepted, then prompt for acceptance.
@@ -306,6 +409,12 @@ def execute_initialization(
if needs_telemetry:
setup_telemetry(console, telemetry_config_path, for_project=for_project)
+ # Step 5: Prime the remote-config cache so dry-runs and validate can fall back offline.
+ # No-op when gateway is disabled or terms have not been accepted. We forward
+ # ``target_config_dir`` so the gateway-enabled check inspects the directory we just
+ # initialized (matters for ``--local`` vs default init).
+ prime_remote_config_cache(console, target_config_dir=target_config_dir)
+
console.print()
diff --git a/pipelex/cli/dev_cli/commands/check_gateway_models_cmd.py b/pipelex/cli/dev_cli/commands/check_gateway_models_cmd.py
index b94f4da1e..62ba7e764 100644
--- a/pipelex/cli/dev_cli/commands/check_gateway_models_cmd.py
+++ b/pipelex/cli/dev_cli/commands/check_gateway_models_cmd.py
@@ -20,7 +20,7 @@ from pipelex.cli.dev_cli.commands.update_gateway_models_cmd import (
GATEWAY_MODELS_REFERENCE_PATH,
)
from pipelex.hub import get_console
-from pipelex.system.pipelex_service.exceptions import RemoteConfigFetchError, RemoteConfigValidationError
+from pipelex.system.pipelex_service.exceptions import RemoteConfigUnavailableError, RemoteConfigValidationError
if TYPE_CHECKING:
from rich.console import Console
@@ -70,7 +70,7 @@ def check_gateway_models_cmd(show_diff: bool = True, quiet: bool = False) -> Non
# Fetch remote config
try:
model_specs = fetch_gateway_model_specs()
- except RemoteConfigFetchError as exc:
+ except RemoteConfigUnavailableError as exc:
if quiet:
console.print(f"[red]✗ Gateway models check: FAILED[/red] - {escape(str(exc))}")
else:
diff --git a/pipelex/cli/dev_cli/commands/gateway_models_generator.py b/pipelex/cli/dev_cli/commands/gateway_models_generator.py
index 87b87d229..8b818cf9a 100644
--- a/pipelex/cli/dev_cli/commands/gateway_models_generator.py
+++ b/pipelex/cli/dev_cli/commands/gateway_models_generator.py
@@ -36,15 +36,19 @@ def normalize_for_comparison(content: str) -> str:
def fetch_gateway_model_specs() -> BackendModelSpecs:
"""Fetch model specifications from Pipelex Gateway remote config.
+ Uses ``require_fresh=True`` so a stale cache never gets baked into the committed
+ reference docs this command regenerates.
+
Returns:
Dictionary of model specifications.
Raises:
- RemoteConfigFetchError: If the remote config cannot be fetched.
+ RemoteConfigUnavailableError: If the network is unreachable (cache fallback is
+ explicitly refused here).
RemoteConfigValidationError: If the remote config is invalid.
"""
- remote_config = RemoteConfigFetcher.fetch_remote_config()
- return dict(remote_config.backend_model_specs)
+ result = RemoteConfigFetcher.fetch_remote_config(require_fresh=True)
+ return dict(result.config.backend_model_specs)
def extract_reference_data(model_specs: BackendModelSpecs) -> dict[str, list[dict[str, Any]]]:
diff --git a/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py b/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py
index a3cb35b09..ac35adb66 100644
--- a/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py
+++ b/pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py
@@ -16,7 +16,7 @@ from rich.table import Table
from pipelex.hub import get_console
from pipelex.system.configuration.config_loader import config_manager
from pipelex.system.configuration.configs import ConfigPaths
-from pipelex.system.pipelex_service.exceptions import RemoteConfigFetchError, RemoteConfigValidationError
+from pipelex.system.pipelex_service.exceptions import RemoteConfigUnavailableError, RemoteConfigValidationError
from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
from pipelex.tools.misc.json_utils import deep_update
from pipelex.tools.misc.toml_utils import TomlError, load_toml_from_path
@@ -93,12 +93,19 @@ def _fetch_gateway_models() -> dict[str, list[str]]:
Returns:
Dictionary mapping model_type to list of model handles.
+
+ Raises:
+ RemoteConfigUnavailableError: The Gateway config is unreachable and no usable cache
+ exists — ``require_fresh=True`` refuses any cached fallback.
+ RemoteConfigValidationError: The Gateway responded with a payload that does not match
+ the expected schema.
"""
- try:
- remote_config = RemoteConfigFetcher.fetch_remote_config()
- model_specs = dict(remote_config.backend_model_specs)
- except (RemoteConfigFetchError, RemoteConfigValidationError):
- return {"llm": [], "img_gen": [], "text_extractor": [], "search": []}
+ # ``require_fresh=True`` guarantees we never bake stale or missing Gateway model specs
+ # into committed files. Any ``RemoteConfigUnavailableError`` / ``RemoteConfigValidationError``
+ # propagates so the command refuses to proceed rather than writing fixtures silently
+ # missing every ``pipelex_gateway`` model.
+ result = RemoteConfigFetcher.fetch_remote_config(require_fresh=True)
+ model_specs = dict(result.config.backend_model_specs)
# Get defaults
defaults = model_specs.get("defaults", {})
@@ -663,6 +670,26 @@ def preprocess_test_models_cmd(
console.print(" • Verify disk space and filesystem health")
console.print()
sys.exit(1)
+ except (RemoteConfigUnavailableError, RemoteConfigValidationError) as exc:
+ # Pipelex Gateway remote config is unavailable or stale. This command regenerates
+ # committed files (model_availability.json, _generated_model_sets.py), so it must
+ # refuse rather than write fixtures missing every pipelex_gateway model.
+ if quiet:
+ console.print(f"[red]✗ Preprocessing failed:[/red] Gateway config unavailable - {escape(str(exc))}")
+ else:
+ error_panel = Panel(
+ f"[red]✗[/red] Pipelex Gateway remote configuration is unavailable\n\n[dim]{escape(str(exc))}[/dim]",
+ title="[bold red]Gateway Config Unavailable[/bold red]",
+ border_style="red",
+ padding=(1, 2),
+ )
+ console.print(error_panel)
+ console.print()
+ console.print("[bold yellow]Recommended Actions:[/bold yellow]")
+ console.print(" • Run [cyan]pipelex init[/cyan] while online to prime the remote config cache")
+ console.print(" • Fixture generation must not proceed without fresh Gateway model specs")
+ console.print()
+ sys.exit(1)
except Exception as exc:
# Catch-all for unexpected errors
if quiet:
diff --git a/pipelex/cli/dev_cli/commands/update_gateway_models_cmd.py b/pipelex/cli/dev_cli/commands/update_gateway_models_cmd.py
index aea891b7a..1c3224315 100644
--- a/pipelex/cli/dev_cli/commands/update_gateway_models_cmd.py
+++ b/pipelex/cli/dev_cli/commands/update_gateway_models_cmd.py
@@ -15,7 +15,7 @@ from pipelex.cli.dev_cli.commands.gateway_models_generator import (
normalize_for_comparison,
)
from pipelex.hub import get_console
-from pipelex.system.pipelex_service.exceptions import RemoteConfigFetchError, RemoteConfigValidationError
+from pipelex.system.pipelex_service.exceptions import RemoteConfigUnavailableError, RemoteConfigValidationError
# Path to the reference files
GATEWAY_MODELS_REFERENCE_PATH = Path(".pipelex/inference/backends/pipelex_gateway_models.md")
@@ -41,7 +41,7 @@ def update_gateway_models_cmd(quiet: bool = False) -> None:
# Fetch remote config
try:
model_specs = fetch_gateway_model_specs()
- except RemoteConfigFetchError as exc:
+ except RemoteConfigUnavailableError as exc:
if quiet:
console.print(f"[red]✗ Gateway models update: FAILED[/red] - {escape(str(exc))}")
else:
diff --git a/pipelex/cli/error_handlers.py b/pipelex/cli/error_handlers.py
index f74f0c279..2bf887c7b 100644
--- a/pipelex/cli/error_handlers.py
+++ b/pipelex/cli/error_handlers.py
@@ -5,7 +5,7 @@ import typer
from rich.console import Console
from rich.markup import escape
-from pipelex.cogt.exceptions import ModelDeckPresetValidatonError
+from pipelex.cogt.exceptions import GatewayUnknownModelError, ModelDeckPresetValidatonError
from pipelex.core.pipes.exceptions import PipeOperatorModelChoiceError
from pipelex.hub import get_console
from pipelex.pipe_operators.exceptions import PipeOperatorModelAvailabilityError
@@ -15,9 +15,10 @@ from pipelex.system.pipelex_service.exceptions import (
GatewayDoNotTrackConflictError,
GatewayTermsNotAcceptedError,
InferenceSetupRequiredError,
- RemoteConfigFetchError,
+ RemoteConfigUnavailableError,
RemoteConfigValidationError,
)
+from pipelex.system.pipelex_service.types import RemoteConfigSource
from pipelex.system.telemetry.exceptions import TelemetryConfigValidationError
from pipelex.types import StrEnum
from pipelex.urls import URLs
@@ -362,73 +363,102 @@ def handle_gateway_do_not_track_conflict_error(exc: GatewayDoNotTrackConflictErr
raise typer.Exit(1) from exc
-def handle_remote_config_fetch_error(exc: RemoteConfigFetchError) -> NoReturn:
- """Handle and display RemoteConfigFetchError with user-friendly guidance.
+def handle_remote_config_validation_error(exc: RemoteConfigValidationError) -> NoReturn:
+ """Handle and display RemoteConfigValidationError with user-friendly guidance.
- This error occurs when Pipelex Gateway is enabled but the remote configuration
- cannot be fetched (network issues, server unreachable, etc.).
+ This error occurs when Pipelex Gateway remote configuration was fetched but
+ the data is malformed or doesn't match the expected schema.
Args:
- exc: The remote config fetch error exception
+ exc: The remote config validation error exception
"""
console = get_console()
- console.print("\n[bold red]❌ Could not connect to Pipelex Gateway[/bold red]\n")
+ console.print("\n[bold red]❌ Pipelex Gateway configuration is invalid[/bold red]\n")
console.print(
- "[bold yellow]⚠ Network Issue:[/bold yellow] Pipelex Gateway requires network access to fetch\n"
- "configuration, but we couldn't reach the Pipelex servers.\n"
+ "[bold yellow]⚠ Server Issue:[/bold yellow] The Pipelex Gateway configuration was received but\n"
+ "couldn't be validated. This is a server-side issue that we need to fix.\n"
)
console.print("[bold cyan]Error details:[/bold cyan]")
console.print(f" {escape(str(exc))}\n")
- console.print("[bold green]💡 To fix:[/bold green]")
- console.print(" • Check your internet connection")
- console.print(" • Verify that firewall/proxy settings allow outbound HTTPS requests")
- console.print(" • Try again in a few moments (servers may be temporarily unavailable)")
+ console.print("[bold red]🚨 Please report this![/bold red]")
+ console.print(" This error shouldn't happen and we want to fix it ASAP.")
+ console.print(" Please copy-paste this error to:")
+ console.print(f" • Discord: [cyan]{URLs.discord}[/cyan]")
+ console.print(f" • GitHub Issues: [cyan]{URLs.repository}/issues[/cyan]")
console.print()
- console.print("[dim]Alternatively, you can:[/dim]")
+ console.print("[dim]In the meantime, you can:[/dim]")
console.print("[dim] • Disable pipelex_gateway in .pipelex/inference/backends.toml[/dim]")
console.print("[dim] • Use your own API keys with direct provider backends[/dim]")
console.print()
- console.print(f"[dim]For more information: {URLs.gateway_docs}[/dim]")
- console.print(f"[dim]Join our Discord for help: {URLs.discord}[/dim]\n")
+ console.print(f"[dim]For more information: {URLs.gateway_docs}[/dim]\n")
raise typer.Exit(1) from exc
-def handle_remote_config_validation_error(exc: RemoteConfigValidationError) -> NoReturn:
- """Handle and display RemoteConfigValidationError with user-friendly guidance.
+def handle_remote_config_unavailable_error(exc: RemoteConfigUnavailableError) -> NoReturn:
+ """Handle and display RemoteConfigUnavailableError with user-friendly guidance.
- This error occurs when Pipelex Gateway remote configuration was fetched but
- the data is malformed or doesn't match the expected schema.
+ Raised when a fresh fetch failed AND no usable cached fallback exists. The gateway
+ is enabled but we have neither network nor a primed local cache.
Args:
- exc: The remote config validation error exception
+ exc: The remote config unavailable error exception
"""
console = get_console()
- console.print("\n[bold red]❌ Pipelex Gateway configuration is invalid[/bold red]\n")
+ console.print("\n[bold red]❌ Pipelex Gateway is unreachable and no cached config is available[/bold red]\n")
console.print(
- "[bold yellow]⚠ Server Issue:[/bold yellow] The Pipelex Gateway configuration was received but\n"
- "couldn't be validated. This is a server-side issue that we need to fix.\n"
+ "[bold yellow]⚠ Offline + Cold Cache:[/bold yellow] Pipelex Gateway requires a config\n"
+ "either fetched fresh or restored from a local cache, but neither is available.\n"
)
console.print("[bold cyan]Error details:[/bold cyan]")
console.print(f" {escape(str(exc))}\n")
- console.print("[bold red]🚨 Please report this![/bold red]")
- console.print(" This error shouldn't happen and we want to fix it ASAP.")
- console.print(" Please copy-paste this error to:")
- console.print(f" • Discord: [cyan]{URLs.discord}[/cyan]")
- console.print(f" • GitHub Issues: [cyan]{URLs.repository}/issues[/cyan]")
+ console.print("[bold green]💡 To fix:[/bold green]")
+ console.print(" • Reconnect to the network and run [cyan]pipelex init[/cyan] to prime the cache")
+ console.print(
+ " • Or disable [cyan]pipelex_gateway[/cyan] in [cyan].pipelex/inference/backends.toml[/cyan] for permanent offline (BYOK) operation"
+ )
console.print()
- console.print("[dim]In the meantime, you can:[/dim]")
- console.print("[dim] • Disable pipelex_gateway in .pipelex/inference/backends.toml[/dim]")
- console.print("[dim] • Use your own API keys with direct provider backends[/dim]")
+ console.print(f"[dim]For more information: {URLs.gateway_docs}[/dim]")
+ console.print(f"[dim]Join our Discord for help: {URLs.discord}[/dim]\n")
+ raise typer.Exit(1) from exc
+
+
+def handle_gateway_unknown_model_error(exc: GatewayUnknownModelError) -> NoReturn:
+ """Handle and display GatewayUnknownModelError with user-friendly guidance.
+
+ Raised when the active model deck references a gateway model handle that doesn't exist
+ in the gateway specs (either freshly fetched or loaded from cache). Branches the
+ remediation on the provenance of the gateway config.
+
+ Args:
+ exc: The gateway unknown model error exception
+ """
+ console = get_console()
+ console.print("\n[bold red]❌ Unknown gateway model handle[/bold red]\n")
+
+ console.print(f"[bold cyan]Model handle:[/bold cyan] [yellow]'{escape(exc.model_name)}'[/yellow]")
+ console.print(f"[bold cyan]Config source:[/bold cyan] [yellow]{escape(exc.source)}[/yellow]")
+ console.print(f"\n[bold red]Error:[/bold red] {escape(str(exc))}\n")
+
+ console.print("[bold green]💡 To fix:[/bold green]")
+ match exc.source:
+ case RemoteConfigSource.FRESH:
+ console.print(" • Check the model handle for typos against [cyan]pipelex doctor[/cyan]")
+ console.print(" • Update the deck to reference a model the gateway currently exposes")
+ console.print(" • Or disable [cyan]pipelex_gateway[/cyan] in [cyan].pipelex/inference/backends.toml[/cyan] to fall back to BYOK")
+ case RemoteConfigSource.CACHED:
+ console.print(" • The on-disk cache may be stale — run [cyan]pipelex init[/cyan] while online to refresh it")
+ console.print(" • Or disable [cyan]pipelex_gateway[/cyan] in [cyan].pipelex/inference/backends.toml[/cyan] to operate offline (BYOK)")
console.print()
- console.print(f"[dim]For more information: {URLs.gateway_docs}[/dim]\n")
+ console.print(f"[dim]For more information: {URLs.gateway_docs}[/dim]")
+ console.print(f"[dim]Join our Discord for help: {URLs.discord}[/dim]\n")
raise typer.Exit(1) from exc
diff --git a/pipelex/cogt/exceptions.py b/pipelex/cogt/exceptions.py
index 90b6b6526..26e0b715a 100644
--- a/pipelex/cogt/exceptions.py
+++ b/pipelex/cogt/exceptions.py
@@ -5,6 +5,7 @@ from typing import TYPE_CHECKING
from typing_extensions import override
from pipelex.base_exceptions import ErrorReport, PipelexError
+from pipelex.system.pipelex_service.types import RemoteConfigSource
from pipelex.types import StrEnum
if TYPE_CHECKING:
@@ -348,3 +349,36 @@ class ModelDeckNotFoundError(CogtError):
class ModelDeckValidationError(CogtError):
pass
+
+
+class GatewayUnknownModelError(CogtError):
+ """A model handle referenced by the deck cannot be located in the active gateway specs.
+
+ Carries the provenance of the gateway config (``FRESH`` vs ``CACHED``) so the message can
+ branch: a cached-source failure suggests stale gateway specs and points the user at
+ ``pipelex init`` to refresh while online; a fresh-source failure is a genuine
+ misconfiguration.
+ """
+
+ error_category = InferenceErrorCategory.CONFIGURATION
+
+ def __init__(self, model_name: str, source: RemoteConfigSource) -> None:
+ self.model_name = model_name
+ self.source = source
+ match source:
+ case RemoteConfigSource.FRESH:
+ msg = (
+ f"Model handle '{model_name}' is referenced by the active model deck but is not present "
+ "in the Pipelex Gateway specs we just fetched. Either the model name is wrong, the gateway "
+ "no longer offers it, or your deck overrides need updating.\n"
+ " - Run `pipelex doctor` to inspect the active gateway models.\n"
+ " - Disable pipelex_gateway in .pipelex/inference/backends.toml to fall back to BYOK."
+ )
+ case RemoteConfigSource.CACHED:
+ msg = (
+ f"Model handle '{model_name}' is referenced by the active model deck but is not present "
+ "in the Pipelex Gateway specs loaded from the on-disk cache. The cache may be stale.\n"
+ " - Run `pipelex init` while online to refresh the cached gateway config.\n"
+ " - Or disable pipelex_gateway in .pipelex/inference/backends.toml to operate offline (BYOK)."
+ )
+ super().__init__(msg)
diff --git a/pipelex/cogt/models/model_deck.py b/pipelex/cogt/models/model_deck.py
index b54056524..fb99b0bc6 100644
--- a/pipelex/cogt/models/model_deck.py
+++ b/pipelex/cogt/models/model_deck.py
@@ -118,7 +118,7 @@ class ModelDeck(ConfigModel):
search_presets: dict[str, SearchSetting] = Field(default_factory=dict)
search_choice_default: SearchModelChoice
- def _get_aliases_and_waterfalls_for_type(self, model_type: ModelType) -> tuple[dict[str, str], dict[str, list[str]]]:
+ def get_aliases_and_waterfalls_for_type(self, model_type: ModelType) -> tuple[dict[str, str], dict[str, list[str]]]:
"""Return the type-specific aliases and waterfalls dictionaries."""
match model_type:
case ModelType.LLM:
@@ -137,7 +137,7 @@ class ModelDeck(ConfigModel):
them and looking up the appropriate dictionary.
"""
ref = ModelReference.parse(model_handle)
- aliases, waterfalls = self._get_aliases_and_waterfalls_for_type(model_type)
+ aliases, waterfalls = self.get_aliases_and_waterfalls_for_type(model_type)
match ref.kind:
case ModelReferenceKind.ALIAS:
@@ -224,7 +224,7 @@ class ModelDeck(ConfigModel):
msg = f"Model handle '{ref.name}' was not found in the model deck"
# Add migration hints if the name matches a preset, alias, or waterfall
- aliases, waterfalls = self._get_aliases_and_waterfalls_for_type(model_type)
+ aliases, waterfalls = self.get_aliases_and_waterfalls_for_type(model_type)
hints: list[str] = []
if ref.name in presets:
hints.append(f"Did you mean preset '${ref.name}' or 'preset:{ref.name}'?")
@@ -791,7 +791,7 @@ class ModelDeck(ConfigModel):
except ModelReferenceParseError:
# Invalid model handle (empty string, etc.)
return None
- aliases, waterfalls = self._get_aliases_and_waterfalls_for_type(model_type)
+ aliases, waterfalls = self.get_aliases_and_waterfalls_for_type(model_type)
# Handle prefixed alias reference (e.g., @best-gpt)
match ref.kind:
@@ -854,7 +854,7 @@ class ModelDeck(ConfigModel):
return None
def is_handle_defined(self, model_handle: str, model_type: ModelType) -> bool:
- aliases, waterfalls = self._get_aliases_and_waterfalls_for_type(model_type)
+ aliases, waterfalls = self.get_aliases_and_waterfalls_for_type(model_type)
return model_handle in self.inference_models or model_handle in aliases or model_handle in waterfalls
def get_required_inference_model(self, model_handle: str, model_type: ModelType) -> InferenceModelSpec:
diff --git a/pipelex/cogt/models/model_manager.py b/pipelex/cogt/models/model_manager.py
index cbac6eeff..9f7a40945 100644
--- a/pipelex/cogt/models/model_manager.py
+++ b/pipelex/cogt/models/model_manager.py
@@ -1,6 +1,9 @@
from typing_extensions import override
-from pipelex.cogt.exceptions import ModelManagerError
+from pipelex.cogt.exceptions import GatewayUnknownModelError, ModelManagerError
+from pipelex.cogt.extract.extract_setting import ExtractSetting
+from pipelex.cogt.img_gen.img_gen_setting import ImgGenSetting
+from pipelex.cogt.llm.llm_setting import LLMSetting
from pipelex.cogt.model_backends.backend import InferenceBackend, PipelexBackend
from pipelex.cogt.model_backends.backend_library import InferenceBackendLibrary
from pipelex.cogt.model_backends.gateway_config import GatewayConfig
@@ -12,8 +15,11 @@ from pipelex.cogt.model_routing.routing_profile_loader import load_active_routin
from pipelex.cogt.models.model_deck import ModelDeck, ModelDeckBlueprint
from pipelex.cogt.models.model_deck_loader import load_model_deck_blueprint
from pipelex.cogt.models.model_manager_abstract import ModelManagerAbstract
+from pipelex.cogt.models.model_reference import ModelReference, ModelReferenceKind, ModelReferenceParseError
+from pipelex.cogt.search.search_setting import SearchSetting
from pipelex.config import get_config
from pipelex.system.configuration.config_loader import config_manager
+from pipelex.system.pipelex_service.types import RemoteConfigSource
from pipelex.tools.misc.file_utils import find_files_in_dir
from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
@@ -56,6 +62,7 @@ class ModelManager(ModelManagerAbstract):
self,
secrets_provider: SecretsProviderAbstract,
gateway_config: GatewayConfig | None,
+ gateway_config_source: RemoteConfigSource | None,
needs_inference: bool = True,
) -> None:
self.inference_backend_library.load(
@@ -75,6 +82,199 @@ class ModelManager(ModelManagerAbstract):
deck_blueprint = load_model_deck_blueprint(model_deck_paths=model_deck_paths)
self.model_deck = self.build_deck(enabled_backends=enabled_backends, model_deck_blueprint=deck_blueprint)
+ self._enforce_gateway_model_membership(
+ gateway_config=gateway_config,
+ gateway_config_source=gateway_config_source,
+ )
+
+ def _enforce_gateway_model_membership(
+ self,
+ gateway_config: GatewayConfig | None,
+ gateway_config_source: RemoteConfigSource | None,
+ ) -> None:
+ """Fail loudly when the deck references a handle the gateway should provide but cannot.
+
+ Runs even when ``missing_presets_reaction = "log"`` (the default), because a missing
+ gateway model is a distinct failure mode from a generic preset mismatch: it means the
+ active gateway specs (fresh or cached) are out of sync with what the deck author
+ declared. Surfacing this as ``GatewayUnknownModelError`` lets the agent CLI hint at
+ cache-refresh remediation when the config was sourced from the on-disk fallback.
+
+ We only fire the check when both ``gateway_config`` and ``gateway_config_source`` are
+ set — that is, when the gateway is actually live in this setup pass.
+
+ Waterfall semantics: a waterfall reference is "known" if AT LEAST ONE of its
+ fallbacks resolves to a known handle. At runtime the deck walks the list and uses
+ the first available model (when ``is_model_fallback_enabled`` is true, the default),
+ so a deck like ``["future-model", "current-model"]`` is perfectly valid as long as
+ ``current-model`` is in the gateway specs.
+ """
+ if gateway_config is None or gateway_config_source is None:
+ return
+ deck = self.get_model_deck()
+ gateway_spec_names = {name for name in gateway_config.model_specs if name != "defaults"}
+
+ for handle, model_type in self._collect_deck_referenced_handles(deck):
+ try:
+ ref = ModelReference.parse(handle)
+ except ModelReferenceParseError:
+ continue
+ candidates = self._resolve_terminal_candidates(deck=deck, ref=ref, model_type=model_type)
+ if not candidates:
+ continue
+ if any(candidate in deck.inference_models or candidate in gateway_spec_names for candidate in candidates):
+ continue
+ # No candidate resolves to a known handle. Report the first one — it's the
+ # primary the user is asking for; subsequent entries are fallbacks.
+ raise GatewayUnknownModelError(model_name=candidates[0], source=gateway_config_source)
+
+ @classmethod
+ def _collect_deck_referenced_handles(cls, deck: ModelDeck) -> list[tuple[str, ModelType]]:
+ """Gather the (handle, model_type) pairs that the deck advertises as usable.
+
+ Covers presets and choice defaults across every model type. Aliases and waterfalls
+ are intentionally NOT enumerated directly — they are reachable via preset/choice
+ references, and the resolver walks through them. Including them here would force the
+ check on dangling helpers the user has not actively wired into a preset.
+ """
+ references: list[tuple[str, ModelType]] = []
+ for llm_setting in deck.llm_presets.values():
+ references.append((llm_setting.model, ModelType.LLM))
+ llm_text_handle = cls._extract_choice_handle(deck.llm_choice_defaults.for_text)
+ if llm_text_handle is not None:
+ references.append((llm_text_handle, ModelType.LLM))
+ llm_object_handle = cls._extract_choice_handle(deck.llm_choice_defaults.for_object)
+ if llm_object_handle is not None:
+ references.append((llm_object_handle, ModelType.LLM))
+ for extract_setting in deck.extract_presets.values():
+ references.append((extract_setting.model, ModelType.TEXT_EXTRACTOR))
+ extract_default_handle = cls._extract_choice_handle(deck.extract_choice_default)
+ if extract_default_handle is not None:
+ references.append((extract_default_handle, ModelType.TEXT_EXTRACTOR))
+ for img_gen_setting in deck.img_gen_presets.values():
+ references.append((img_gen_setting.model, ModelType.IMG_GEN))
+ img_gen_default_handle = cls._extract_choice_handle(deck.img_gen_choice_default)
+ if img_gen_default_handle is not None:
+ references.append((img_gen_default_handle, ModelType.IMG_GEN))
+ for search_setting in deck.search_presets.values():
+ references.append((search_setting.model, ModelType.SEARCH))
+ search_default_handle = cls._extract_choice_handle(deck.search_choice_default)
+ if search_default_handle is not None:
+ references.append((search_default_handle, ModelType.SEARCH))
+ return references
+
+ @classmethod
+ def _extract_choice_handle(cls, choice: LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | ModelReference | str | None) -> str | None:
+ """Normalise a ``*ModelChoice`` union (LLMModelChoice etc.) to a raw handle string.
+
+ Choice defaults can be a typed setting object, a parsed ``ModelReference``, or a raw
+ string — all three paths point at a handle we need to validate.
+ """
+ if choice is None:
+ return None
+ if isinstance(choice, str):
+ return choice
+ if isinstance(choice, ModelReference):
+ return choice.raw
+ return choice.model
+
+ @classmethod
+ def _resolve_terminal_candidates(
+ cls,
+ deck: ModelDeck,
+ ref: ModelReference,
+ model_type: ModelType,
+ ) -> list[str]:
+ """Return every terminal handle reachable from ``ref`` via aliases/waterfalls.
+
+ For ``HANDLE`` references: returns ``[name]`` (bare strings are HANDLEs by design —
+ see ``ModelReference.parse`` for the BREAKING CHANGE note).
+
+ For ``ALIAS`` references: follows the alias target. Cycles return ``[]``.
+
+ For ``WATERFALL`` references: follows EVERY fallback in order (or only the first
+ when ``model_deck_config.is_model_fallback_enabled`` is false, matching runtime
+ behaviour at ``model_deck._get_optional_inference_model_with_fallback``). Cycles
+ across either alias or waterfall keys return ``[]`` for the cycling branch but do
+ not poison the rest of the candidate list.
+
+ For ``PRESET`` references: returns ``[]`` (presets are not handles).
+ """
+ aliases, waterfalls = deck.get_aliases_and_waterfalls_for_type(model_type)
+ is_fallback_enabled = deck.model_deck_config.is_model_fallback_enabled
+ return cls._collect_candidates(
+ ref=ref,
+ aliases=aliases,
+ waterfalls=waterfalls,
+ is_fallback_enabled=is_fallback_enabled,
+ visited=set(),
+ )
+
+ @classmethod
+ def _collect_candidates(
+ cls,
+ ref: ModelReference,
+ aliases: dict[str, str],
+ waterfalls: dict[str, list[str]],
+ is_fallback_enabled: bool,
+ visited: set[tuple[ModelReferenceKind, str]],
+ ) -> list[str]:
+ # Cycle key is (kind, name): an alias and a waterfall can share a name yet be distinct nodes.
+ visit_key: tuple[ModelReferenceKind, str]
+ match ref.kind:
+ case ModelReferenceKind.HANDLE:
+ return [ref.name]
+ case ModelReferenceKind.ALIAS:
+ visit_key = (ref.kind, ref.name)
+ if visit_key in visited:
+ return []
+ visited.add(visit_key)
+ target = aliases.get(ref.name)
+ if target is None:
+ return [ref.name]
+ try:
+ next_ref = ModelReference.parse(target)
+ except ModelReferenceParseError:
+ return []
+ return cls._collect_candidates(
+ ref=next_ref,
+ aliases=aliases,
+ waterfalls=waterfalls,
+ is_fallback_enabled=is_fallback_enabled,
+ visited=visited,
+ )
+ case ModelReferenceKind.WATERFALL:
+ visit_key = (ref.kind, ref.name)
+ if visit_key in visited:
+ return []
+ visited.add(visit_key)
+ fallback_list = waterfalls.get(ref.name)
+ if not fallback_list:
+ return [ref.name]
+ # Runtime only tries the first fallback when fallback is disabled; mirror
+ # that here so the membership check stays consistent with what actually runs.
+ entries = fallback_list if is_fallback_enabled else fallback_list[:1]
+ candidates: list[str] = []
+ for entry in entries:
+ try:
+ next_ref = ModelReference.parse(entry)
+ except ModelReferenceParseError:
+ continue
+ # Fresh visited set per branch so two waterfall entries that legitimately
+ # share an alias don't kill the second one.
+ candidates.extend(
+ cls._collect_candidates(
+ ref=next_ref,
+ aliases=aliases,
+ waterfalls=waterfalls,
+ is_fallback_enabled=is_fallback_enabled,
+ visited=set(visited),
+ )
+ )
+ return candidates
+ case ModelReferenceKind.PRESET:
+ return []
+
@override
def validate_model_deck(self):
self.get_model_deck().validate_registered_models()
diff --git a/pipelex/cogt/models/model_manager_abstract.py b/pipelex/cogt/models/model_manager_abstract.py
index 54ca19aa7..1fba6dcb8 100644
--- a/pipelex/cogt/models/model_manager_abstract.py
+++ b/pipelex/cogt/models/model_manager_abstract.py
@@ -5,6 +5,7 @@ from pipelex.cogt.model_backends.gateway_config import GatewayConfig
from pipelex.cogt.model_backends.model_spec import InferenceModelSpec
from pipelex.cogt.model_backends.model_type import ModelType
from pipelex.cogt.models.model_deck import ModelDeck
+from pipelex.system.pipelex_service.types import RemoteConfigSource
from pipelex.tools.secrets.secrets_provider_abstract import SecretsProviderAbstract
@@ -22,6 +23,7 @@ class ModelManagerAbstract(ABC):
self,
secrets_provider: SecretsProviderAbstract,
gateway_config: GatewayConfig | None,
+ gateway_config_source: RemoteConfigSource | None,
needs_inference: bool = True,
) -> None:
pass
diff --git a/pipelex/pipelex.py b/pipelex/pipelex.py
index ed51ace3f..a29e3ff1a 100644
--- a/pipelex/pipelex.py
+++ b/pipelex/pipelex.py
@@ -1,4 +1,5 @@
import types
+import warnings
from pathlib import Path
from typing import TYPE_CHECKING, Any, cast
@@ -57,6 +58,7 @@ from pipelex.system.environment import get_pipelexpath_dirs
from pipelex.system.pipelex_service.exceptions import (
GatewayTermsNotAcceptedError,
InferenceSetupRequiredError,
+ RemoteConfigStaleWarning,
)
from pipelex.system.pipelex_service.pipelex_service_config import (
is_pipelex_gateway_enabled,
@@ -87,6 +89,7 @@ from pipelex.urls import URLs
if TYPE_CHECKING:
from pipelex.system.pipelex_service.remote_config import RemoteConfig
+ from pipelex.system.pipelex_service.types import RemoteConfigSource
PACKAGE_NAME, PACKAGE_VERSION = get_package_info()
@@ -203,6 +206,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
remote_config: RemoteConfig | None = None
gateway_config: GatewayConfig | None = None
+ gateway_config_source: RemoteConfigSource | None = None
if is_pipelex_service_enabled:
if not effective_needs_model_specs:
# Use dummy config when inference is not needed (for testing without network access)
@@ -212,6 +216,11 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
model_specs=gateway_model_specs,
aws_region=remote_config.aws_region,
)
+ # Keep ``gateway_config_source`` as ``None``: the dummy specs are an empty
+ # placeholder, not real Gateway data. ``ModelManager._enforce_gateway_model_membership``
+ # treats ``source is None`` as "nothing to validate against," so the membership
+ # check is skipped on this path — which is what we want for read-only flows like
+ # ``pipelex-agent models`` without ``--backend``.
log.verbose("Using dummy remote config (inference not needed)")
else:
# Terms acceptance is only required for actual inference usage, not for
@@ -231,17 +240,35 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
# who disabled gateway via init skip this entire block.
if not pipelex_service_config.agreement.terms_accepted:
raise GatewayTermsNotAcceptedError
- # Fetch remote configuration
- remote_config = RemoteConfigFetcher.fetch_remote_config()
- log.verbose("Successfully fetched Pipelex Gateway remote configuration")
+ # Fetch remote configuration (may fall back to on-disk cache when offline).
+ remote_config_result = RemoteConfigFetcher.fetch_remote_config()
+ remote_config = remote_config_result.config
+ gateway_config_source = remote_config_result.source
+ log.verbose(f"Successfully fetched Pipelex Gateway remote configuration (source={gateway_config_source})")
gateway_model_specs = remote_config.backend_model_specs
gateway_config = GatewayConfig(
model_specs=gateway_model_specs,
aws_region=remote_config.aws_region,
)
-
- # Disable Pipelex telemetry when inference is not needed (no remote config available)
- is_pipelex_telemetry_enabled = is_pipelex_service_enabled and needs_inference
+ # Stale operation: warn loudly so machine consumers can re-surface the provenance.
+ # Emission lives at this orchestration layer (not in the fetcher) so the fetcher
+ # stays a pure data-returning function — and so test fixtures that swap in a
+ # cached fetcher (tests/conftest.py) don't need to special-case warning replay.
+ if gateway_config_source.is_cached:
+ cached_at_iso = remote_config_result.cached_at.isoformat() if remote_config_result.cached_at else "unknown"
+ warnings.warn(
+ f"Pipelex Gateway is running off a cached remote config (snapshot: {cached_at_iso}). "
+ "Run `pipelex init` while online to refresh.",
+ RemoteConfigStaleWarning,
+ stacklevel=2,
+ )
+
+ # Disable Pipelex telemetry when:
+ # - inference is not needed (no live runs to track), OR
+ # - the gateway config came from the cache (stale specs imply potentially stale model
+ # identities; phoning home about pipe runs in that state would pollute metrics).
+ gateway_source_is_cached = gateway_config_source is not None and gateway_config_source.is_cached
+ is_pipelex_telemetry_enabled = is_pipelex_service_enabled and needs_inference and not gateway_source_is_cached
self.telemetry_manager = TelemetryFactory.make_telemetry_manager(
secrets_provider=secrets_provider,
integration_mode=integration_mode,
@@ -302,6 +329,7 @@ If you need help, drop by our Discord: we're happy to assist: {URLs.discord}.
self.models_manager.setup(
secrets_provider=secrets_provider,
gateway_config=gateway_config,
+ gateway_config_source=gateway_config_source,
needs_inference=needs_inference,
)
except RoutingProfileLibraryNotFoundError as routing_not_found_exc:
diff --git a/pipelex/system/configuration/config_loader.py b/pipelex/system/configuration/config_loader.py
index ee143f243..ecad69506 100644
--- a/pipelex/system/configuration/config_loader.py
+++ b/pipelex/system/configuration/config_loader.py
@@ -9,7 +9,7 @@ from pipelex.tools.misc.toml_utils import load_toml_from_path_and_merge_with_ove
CONFIG_DIR_NAME = ".pipelex"
CONFIG_NAME = "pipelex.toml"
-PROJECT_ROOT_MARKERS: frozenset[str] = frozenset({".git", "pyproject.toml", "setup.py", "setup.cfg", "package.json", ".hg"})
+PROJECT_ROOT_MARKERS: frozenset[str] = frozenset({CONFIG_DIR_NAME, ".git", "pyproject.toml", "setup.py", "setup.cfg", "package.json", ".hg"})
INFERENCE_DIR_NAME = "inference"
BACKENDS_FILE_NAME = "backends.toml"
@@ -33,7 +33,8 @@ class ConfigLoader:
"""Walk up from start_dir looking for project root markers.
Excludes the home directory, which may contain stray marker files
- (e.g. package.json) but is never a real project root.
+ (e.g. the global ~/.pipelex/ or a stray package.json) but is never a
+ real project root.
Returns the directory containing the marker, or None if not found.
"""
diff --git a/pipelex/system/pipelex_service/exceptions.py b/pipelex/system/pipelex_service/exceptions.py
index e571a1f3b..48952e459 100644
--- a/pipelex/system/pipelex_service/exceptions.py
+++ b/pipelex/system/pipelex_service/exceptions.py
@@ -31,6 +31,25 @@ class RemoteConfigValidationError(PipelexServiceError):
"""
+class RemoteConfigUnavailableError(PipelexServiceError):
+ """Raised when a fresh fetch failed AND no usable cached fallback exists.
+
+ This is the user-facing offline-mode error: the gateway is enabled but we have neither
+ a working network path nor a primed local cache to fall back on. The message names the
+ cache file path and the remediation (``pipelex init`` while online; or disabling the
+ gateway in ``backends.toml`` for permanent offline operation).
+ """
+
+
+class RemoteConfigStaleWarning(UserWarning):
+ """Emitted when setup completes using a cached remote-config fallback instead of a fresh fetch.
+
+ Indicates the network was unreachable but a prior ``pipelex init`` had primed the cache.
+ Stale operation is intentionally safe for dry-run/validation flows; real inference calls
+ still require network at call time.
+ """
+
+
class InferenceSetupRequiredError(PipelexServiceError):
"""Raised when inference is requested but no inference setup has been completed.
diff --git a/pipelex/system/pipelex_service/pipelex_details.py b/pipelex/system/pipelex_service/pipelex_details.py
index e5cbd265a..088002209 100644
--- a/pipelex/system/pipelex_service/pipelex_details.py
+++ b/pipelex/system/pipelex_service/pipelex_details.py
@@ -1,10 +1,23 @@
+import os
+
from pipelex.tools.misc.hash_utils import hash_sha256
+_DEFAULT_REMOTE_CONFIG_URL = "https://pipelex-config.s3.eu-west-3.amazonaws.com/pipelex_remote_config_10.json"
+REMOTE_CONFIG_URL_ENV_VAR = "PIPELEX_REMOTE_CONFIG_URL"
+
class PipelexDetails:
- REMOTE_CONFIG_URL = "https://pipelex-config.s3.eu-west-3.amazonaws.com/pipelex_remote_config_10.json"
PIPELEX_GATEWAY_API_KEY_VAR = "PIPELEX_GATEWAY_API_KEY"
+ @classmethod
+ def remote_config_url(cls) -> str:
+ """The URL to fetch the Pipelex remote config from.
+
+ Reads ``PIPELEX_REMOTE_CONFIG_URL`` from the environment when set
+ (useful for testing/staging), otherwise falls back to the production URL.
+ """
+ return os.environ.get(REMOTE_CONFIG_URL_ENV_VAR) or _DEFAULT_REMOTE_CONFIG_URL
+
@classmethod
def make_distinct_id(cls, gateway_api_key: str) -> str:
"""Make a distinct_id for PostHog from the gateway API key.
diff --git a/pipelex/system/pipelex_service/pipelex_service_config.py b/pipelex/system/pipelex_service/pipelex_service_config.py
index 859b02a65..35f97ead0 100644
--- a/pipelex/system/pipelex_service/pipelex_service_config.py
+++ b/pipelex/system/pipelex_service/pipelex_service_config.py
@@ -42,15 +42,23 @@ def load_pipelex_service_config_if_exists(config_dir: Path) -> PipelexServiceCon
raise PipelexServiceConfigValidationError(msg) from exc
-def is_pipelex_gateway_enabled() -> bool:
+def is_pipelex_gateway_enabled(backends_file_path: Path | None = None) -> bool:
"""Check if pipelex_gateway is enabled in the backends configuration.
This reads the backends.toml file directly without loading the full backend library.
+ Args:
+ backends_file_path: Explicit path to the ``backends.toml`` file to inspect. When
+ ``None`` (default), uses the layered/project-preferred path from
+ ``config_manager.backends_file_path``. Callers that act on a specific target
+ directory (e.g. ``pipelex init`` / ``pipelex init --local``) should pass the
+ target's ``backends.toml`` so they don't accidentally branch on a sibling config.
+
Returns:
True if pipelex_gateway is enabled, False otherwise.
"""
- backends_toml = load_toml_from_path_if_exists(config_manager.backends_file_path)
+ resolved_path = backends_file_path if backends_file_path is not None else config_manager.backends_file_path
+ backends_toml = load_toml_from_path_if_exists(resolved_path)
if backends_toml is None:
return False
diff --git a/pipelex/system/pipelex_service/remote_config_cache.py b/pipelex/system/pipelex_service/remote_config_cache.py
new file mode 100644
index 000000000..0910526ba
--- /dev/null
+++ b/pipelex/system/pipelex_service/remote_config_cache.py
@@ -0,0 +1,144 @@
+"""On-disk fallback cache for the Pipelex Gateway remote config.
+
+The cache is a **last-resort** safety net for offline operation. It is never read for freshness
+optimisation. Layout::
+
+ ~/.pipelex/cache/remote_config.json
+
+The on-disk payload is the raw JSON returned by the remote endpoint (not a re-serialised
+Pydantic dump), so the cache is stable across minor schema drift. Schema-breaking changes
+require bumping both the remote-config URL version (in ``pipelex-back-office``) and
+``CACHE_SCHEMA_VERSION`` here; older caches are then rejected on load and re-primed on the
+next successful fetch.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import tempfile
+from datetime import datetime, timezone
+from json import JSONDecodeError
+from pathlib import Path
+from typing import Any
+
+from pydantic import BaseModel, ConfigDict, Field, ValidationError
+
+from pipelex import log
+from pipelex.system.configuration.config_loader import config_manager
+from pipelex.system.pipelex_service.remote_config import RemoteConfig
+
+CACHE_SCHEMA_VERSION = 1
+CACHE_SUBDIR_NAME = "cache"
+CACHE_FILE_NAME = "remote_config.json"
+
+
+class CachedRemoteConfig(BaseModel):
+ """A snapshot of the remote gateway config plus the metadata needed to validate it."""
+
+ model_config = ConfigDict(extra="forbid")
+
+ schema_version: int = Field(description="On-disk schema version; mismatches reject the cache")
+ cached_at: datetime = Field(description="UTC timestamp when this snapshot was written")
+ raw_config: dict[str, Any] = Field(description="The raw remote-config JSON payload, untouched")
+
+ def to_remote_config(self) -> RemoteConfig:
+ """Re-validate the raw payload into a ``RemoteConfig`` model on demand."""
+ return RemoteConfig.model_validate(self.raw_config)
+
+
+class RemoteConfigCache:
+ """Read/write helpers for the on-disk remote-config fallback."""
+
+ @classmethod
+ def cache_path(cls) -> Path:
+ """Resolve the cache file path under the global ``~/.pipelex`` directory."""
+ return config_manager.global_config_dir / CACHE_SUBDIR_NAME / CACHE_FILE_NAME
+
+ @classmethod
+ def load(cls) -> CachedRemoteConfig | None:
+ """Return the cached snapshot, or ``None`` when there is no usable cache.
+
+ Any of the following result in ``None``:
+ - the cache file does not exist
+ - the file cannot be read or parsed
+ - the on-disk schema version doesn't match this code's ``CACHE_SCHEMA_VERSION``
+ - the JSON doesn't validate as ``CachedRemoteConfig``
+
+ Bare ``Exception`` is never caught — only the specific classes raised by the
+ filesystem, JSON, and Pydantic layers.
+ """
+ path = cls.cache_path()
+ try:
+ content = path.read_text(encoding="utf-8")
+ except FileNotFoundError:
+ return None
+ except OSError as exc:
+ log.warning(f"Could not read remote config cache at {path}: {exc}")
+ return None
+
+ try:
+ raw = json.loads(content)
+ except JSONDecodeError as exc:
+ log.warning(f"Remote config cache at {path} is not valid JSON: {exc}")
+ return None
+
+ try:
+ cached = CachedRemoteConfig.model_validate(raw)
+ except ValidationError as exc:
+ log.warning(f"Remote config cache at {path} failed validation and will be ignored: {exc}")
+ return None
+
+ if cached.schema_version != CACHE_SCHEMA_VERSION:
+ log.warning(f"Remote config cache at {path} has schema_version={cached.schema_version}, expected {CACHE_SCHEMA_VERSION}; ignoring.")
+ return None
+
+ return cached
+
+ @classmethod
+ def store(cls, remote_config_payload: dict[str, Any]) -> None:
+ """Atomically write the raw remote-config payload to disk.
+
+ Writes go to a temp file in the cache directory and are then moved into place with
+ ``os.replace``, which is atomic on the local filesystem. If the move fails, the temp
+ file is removed so we don't leave half-written caches around.
+
+ The argument is the **raw** JSON returned by the remote endpoint, NOT a serialised
+ ``RemoteConfig`` instance — this keeps the cache stable across minor schema drift.
+ """
+ path = cls.cache_path()
+ cache_dir = path.parent
+ cache_dir.mkdir(parents=True, exist_ok=True)
+
+ cached = CachedRemoteConfig(
+ schema_version=CACHE_SCHEMA_VERSION,
+ cached_at=datetime.now(tz=timezone.utc),
+ raw_config=remote_config_payload,
+ )
+ serialised = cached.model_dump_json()
+
+ tmp_file = tempfile.NamedTemporaryFile( # noqa: SIM115 - we manage the handle manually for atomic replace
+ mode="w",
+ encoding="utf-8",
+ dir=str(cache_dir),
+ prefix=f".{CACHE_FILE_NAME}.",
+ suffix=".tmp",
+ delete=False,
+ )
+ tmp_path = Path(tmp_file.name)
+ replaced = False
+ try:
+ try:
+ tmp_file.write(serialised)
+ tmp_file.flush()
+ os.fsync(tmp_file.fileno())
+ finally:
+ tmp_file.close()
+ tmp_path.replace(path)
+ replaced = True
+ finally:
+ if not replaced:
+ try:
+ tmp_path.unlink()
+ except FileNotFoundError:
+ pass
diff --git a/pipelex/system/pipelex_service/remote_config_fetcher.py b/pipelex/system/pipelex_service/remote_config_fetcher.py
index 440bcac16..e1932418a 100644
--- a/pipelex/system/pipelex_service/remote_config_fetcher.py
+++ b/pipelex/system/pipelex_service/remote_config_fetcher.py
@@ -1,20 +1,70 @@
+"""Fetch the Pipelex Gateway remote config with retry, cache, and provenance tracking.
+
+The fetcher is the single entry point used by ``Pipelex.setup`` and the dev/doctor CLIs.
+Its public method returns a :class:`RemoteConfigResult` carrying the parsed config plus the
+source it came from (``FRESH`` vs ``CACHED``), so downstream code can branch its error
+messaging and disable telemetry when running off stale data.
+
+This module emits no warnings — stale-cache surfacing is the orchestration layer's job
+(see ``Pipelex.setup``). Keeping the fetcher pure means test fixtures that swap in a cached
+fetcher don't need to special-case warning replay.
+
+Behaviour:
+
+- Every successful fetch persists the raw JSON to :class:`RemoteConfigCache` (opportunistic
+ refresh).
+- On network or HTTP failure, the fetcher falls back to the cache if one exists and returns
+ it tagged ``source=CACHED`` with a ``cached_at`` timestamp. If no usable cache exists, it
+ raises :class:`RemoteConfigUnavailableError`. The inner :class:`RemoteConfigFetchError` is
+ chained as ``__cause__`` so existing surfaces (doctor, agent hints) keep working. Warning
+ emission is the orchestration layer's responsibility — see ``Pipelex.setup`` — so the
+ fetcher itself remains a pure data-returning function.
+- On JSON-validation failure (a server-side schema break — we control the server) the
+ fetcher raises :class:`RemoteConfigValidationError` and does NOT silently fall back to
+ cache.
+- Callers that must not bake stale data into committed files (doc generators, fixture
+ preprocessors) pass ``require_fresh=True``, which turns any cache fallback into an
+ :class:`RemoteConfigUnavailableError` immediately.
+"""
+
+from __future__ import annotations
+
+from datetime import datetime # noqa: TC003 — Pydantic v2 resolves this annotation at runtime
+from typing import Any
+
import httpx
-from pydantic import ValidationError
+from pydantic import BaseModel, ConfigDict, Field, ValidationError
from tenacity import RetryCallState, retry, retry_if_exception_type, stop_after_attempt, wait_exponential
from pipelex.system.pipelex_service.exceptions import (
RemoteConfigFetchError,
+ RemoteConfigUnavailableError,
RemoteConfigValidationError,
)
from pipelex.system.pipelex_service.pipelex_details import PipelexDetails
from pipelex.system.pipelex_service.remote_config import PipelexPosthogConfig, RemoteConfig
+from pipelex.system.pipelex_service.remote_config_cache import RemoteConfigCache
+from pipelex.system.pipelex_service.types import RemoteConfigSource
from pipelex.system.runtime import runtime_manager
from pipelex.tools.misc.terminal_utils import print_to_stderr
from pipelex.tools.typing.pydantic_utils import format_pydantic_validation_error
+class RemoteConfigResult(BaseModel):
+ """Outcome of a fetch attempt: the parsed config plus provenance metadata."""
+
+ model_config = ConfigDict(extra="forbid", strict=True, arbitrary_types_allowed=True)
+
+ config: RemoteConfig = Field(description="The parsed remote configuration")
+ source: RemoteConfigSource = Field(description="Whether the config came from the network or the on-disk cache")
+ cached_at: datetime | None = Field(
+ default=None,
+ description="UTC timestamp from the cache snapshot when ``source == CACHED``; ``None`` otherwise",
+ )
+
+
class RemoteConfigFetcher:
- """Fetches Pipelex Service remote configuration with retry logic."""
+ """Fetches Pipelex Service remote configuration with retry logic and a cache fallback."""
# Retry configuration for remote config fetch
# Using hardcoded values since this runs before config is fully loaded
@@ -79,48 +129,131 @@ class RemoteConfigFetcher:
)
@classmethod
- def fetch_remote_config(cls) -> RemoteConfig:
- """Fetch Pipelex Service remote configuration.
+ def _fetch_fresh(cls, url: str) -> tuple[dict[str, Any], RemoteConfig]:
+ """Fetch and validate-parse the remote payload. Raises ``RemoteConfigFetchError`` on
+ network/HTTP failure and ``RemoteConfigValidationError`` on schema breaks. Returns the
+ raw JSON payload (for cache persistence) alongside the already-parsed config so the
+ caller does not have to revalidate.
+ """
+ try:
+ response = cls._fetch_remote_config_with_retry(url)
+ except httpx.TimeoutException as timeout_exc:
+ msg = f"Timeout while fetching remote configuration from {url}: {timeout_exc}"
+ raise RemoteConfigFetchError(msg) from timeout_exc
+ except httpx.HTTPStatusError as http_exc:
+ msg = f"HTTP error {http_exc.response.status_code} while fetching remote configuration from {url}"
+ raise RemoteConfigFetchError(msg) from http_exc
+ except httpx.RequestError as request_exc:
+ msg = f"Failed to fetch remote configuration from {url} after {cls.FETCH_MAX_RETRIES} attempts: {request_exc}"
+ raise RemoteConfigFetchError(msg) from request_exc
+
+ try:
+ payload: dict[str, Any] = response.json()
+ except ValueError as parse_exc:
+ msg = f"Failed to parse remote configuration JSON: {parse_exc}"
+ raise RemoteConfigValidationError(msg) from parse_exc
+
+ try:
+ config = RemoteConfig.model_validate(payload)
+ except ValidationError as validation_error:
+ formatted = format_pydantic_validation_error(validation_error)
+ msg = f"Remote configuration validation failed: {formatted}"
+ raise RemoteConfigValidationError(msg) from validation_error
+
+ return payload, config
+
+ @classmethod
+ def _build_unavailable_error(
+ cls,
+ fetch_error: RemoteConfigFetchError,
+ cache_refused: bool = False,
+ ) -> RemoteConfigUnavailableError:
+ """Build the user-facing offline-mode error with a clear remediation hint.
+
+ Args:
+ fetch_error: The underlying network/HTTP failure that triggered the unavailable state.
+ cache_refused: When ``True``, the caller passed ``require_fresh=True`` and refused
+ to fall back to any cached payload (dev-CLI generators do this). The message
+ branches to call out that the cache was refused rather than missing or unusable.
+
+ Returns the exception so the caller can ``raise ... from fetch_error`` itself; this
+ avoids "unreachable code" gymnastics at the call site.
+ """
+ cache_path = RemoteConfigCache.cache_path()
+ if cache_refused:
+ location = f"and the local cache at {cache_path} was refused because a fresh fetch is required"
+ else:
+ location = f"and no usable local cache is available at {cache_path}"
+ msg = (
+ f"Pipelex Gateway is enabled but the remote configuration is unreachable "
+ f"{location}.\n"
+ f"Underlying error: {fetch_error}\n"
+ "Remediation:\n"
+ " - Run `pipelex init` while online to prime the cache.\n"
+ " - Or disable pipelex_gateway in .pipelex/inference/backends.toml to operate "
+ "permanently offline with your own API keys (BYOK)."
+ )
+ return RemoteConfigUnavailableError(msg)
+
+ @classmethod
+ def fetch_remote_config(cls, require_fresh: bool = False) -> RemoteConfigResult:
+ """Fetch the Pipelex Service remote configuration.
+
+ Args:
+ require_fresh: When ``True``, refuse to serve a cached fallback. Used by
+ dev-CLI generators that regenerate committed reference docs/fixtures — they
+ must never bake stale data into the repo. A cache miss when offline becomes
+ ``RemoteConfigUnavailableError`` immediately.
Returns:
- RemoteConfig.
+ A :class:`RemoteConfigResult` carrying the parsed config plus its provenance.
Raises:
- RemoteConfigFetchError: If the HTTP request fails or returns an error.
- RemoteConfigValidationError: If the JSON doesn't match expected schema.
+ RemoteConfigUnavailableError: Network fetch failed AND no usable cache exists
+ (or ``require_fresh=True`` and only a cache is available).
+ RemoteConfigValidationError: The remote responded with a payload that doesn't
+ match the expected schema. Never falls back to cache for this case.
"""
- # In Codex Cloud, return dummy config to avoid SSL issues with MITM proxy
if runtime_manager.is_in_codex_cloud:
print_to_stderr("Skipping remote config fetch in Codex Cloud, using dummy config instead")
- return cls.make_dummy_remote_config()
+ return RemoteConfigResult(config=cls.make_dummy_remote_config(), source=RemoteConfigSource.FRESH, cached_at=None)
- url = PipelexDetails.REMOTE_CONFIG_URL
+ url = PipelexDetails.remote_config_url()
try:
- response = cls._fetch_remote_config_with_retry(url)
- except httpx.TimeoutException as exc:
- msg = f"Timeout while fetching remote configuration from {url}: {exc}"
- raise RemoteConfigFetchError(msg) from exc
- except httpx.HTTPStatusError as exc:
- msg = f"HTTP error {exc.response.status_code} while fetching remote configuration from {url}"
- raise RemoteConfigFetchError(msg) from exc
- except httpx.RequestError as exc:
- msg = f"Failed to fetch remote configuration from {url} after {cls.FETCH_MAX_RETRIES} attempts: {exc}"
- raise RemoteConfigFetchError(msg) from exc
-
- # Parse JSON content
- try:
- config_dict = response.json()
- except Exception as exc:
- msg = f"Failed to parse remote configuration JSON: {exc}"
- raise RemoteConfigValidationError(msg) from exc
+ payload, config = cls._fetch_fresh(url)
+ except RemoteConfigFetchError as fetch_error:
+ if require_fresh:
+ raise cls._build_unavailable_error(fetch_error, cache_refused=True) from fetch_error
+ cached = RemoteConfigCache.load()
+ if cached is None:
+ raise cls._build_unavailable_error(fetch_error) from fetch_error
+ # A valid cache wrapper can still hold a stale or malformed ``raw_config`` —
+ # ``RemoteConfigCache.load()`` only validates the wrapper, not the inner payload.
+ # Treat such a cache as unusable and surface the normal offline-mode remediation
+ # instead of letting a raw Pydantic ``ValidationError`` escape.
+ try:
+ cached_config = cached.to_remote_config()
+ except ValidationError as validation_error:
+ raise cls._build_unavailable_error(fetch_error) from validation_error
+ return RemoteConfigResult(
+ config=cached_config,
+ source=RemoteConfigSource.CACHED,
+ cached_at=cached.cached_at,
+ )
- # Validate the structure
+ # Fresh path: persist the raw payload for future offline fallback. The cache is
+ # opportunistic — a write failure (read-only $HOME, full disk, permission denied) must
+ # not abort an otherwise-successful online fetch. We warn so the user can see why a
+ # later offline run might have to fail loudly with ``RemoteConfigUnavailableError``,
+ # then return the fresh config regardless.
try:
- config = RemoteConfig.model_validate(config_dict)
- except ValidationError as exc:
- validation_error_msg = format_pydantic_validation_error(exc)
- msg = f"Remote configuration validation failed: {validation_error_msg}"
- raise RemoteConfigValidationError(msg) from exc
-
- return config
+ RemoteConfigCache.store(payload)
+ except OSError as cache_write_error:
+ warning_msg = (
+ f"Warning: failed to persist remote config cache at {RemoteConfigCache.cache_path()}: "
+ f"{cache_write_error}. Continuing with the fresh fetch; future offline runs may "
+ "be unable to fall back to a cached config until this is fixed."
+ )
+ print_to_stderr(warning_msg)
+ return RemoteConfigResult(config=config, source=RemoteConfigSource.FRESH, cached_at=None)
diff --git a/pipelex/system/pipelex_service/types.py b/pipelex/system/pipelex_service/types.py
new file mode 100644
index 000000000..558fc9878
--- /dev/null
+++ b/pipelex/system/pipelex_service/types.py
@@ -0,0 +1,32 @@
+"""Leaf-level enums shared by the ``pipelex_service`` public surface.
+
+Lives here — separate from the heavy ``remote_config_fetcher`` module — so that
+``cogt/``-side modules (which already need source provenance to branch error messages and
+disable telemetry on stale specs) can import the enum without pulling the fetcher and its
+``httpx`` + ``tenacity`` dependencies. Keeps the cogt → pipelex_service edge clean and
+avoids a layering inversion as more cogt callers acquire a need for provenance metadata.
+"""
+
+from __future__ import annotations
+
+from pipelex.types import StrEnum
+
+
+class RemoteConfigSource(StrEnum):
+ """Where a ``RemoteConfigResult`` was sourced from."""
+
+ FRESH = "fresh"
+ CACHED = "cached"
+
+ @property
+ def is_cached(self) -> bool:
+ """True when the config came from the on-disk fallback rather than the network.
+
+ Use this everywhere instead of ``source == RemoteConfigSource.CACHED`` so the
+ match/case rule from the project standards stays out of caller code.
+ """
+ match self:
+ case RemoteConfigSource.FRESH:
+ return False
+ case RemoteConfigSource.CACHED:
+ return True
diff --git a/tests/conftest.py b/tests/conftest.py
index c48204fce..5a46936d8 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -17,7 +17,8 @@ from pipelex.system.pipelex_service.pipelex_service_config import (
load_pipelex_service_config_if_exists as _original_load_pipelex_service_config,
)
from pipelex.system.pipelex_service.remote_config import RemoteConfig
-from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
+from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher, RemoteConfigResult
+from pipelex.system.pipelex_service.types import RemoteConfigSource
from pipelex.system.runtime import IntegrationMode, runtime_manager
from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerAbstract
@@ -35,11 +36,16 @@ _remote_config_cache: dict[str, RemoteConfig] = {}
_original_fetch_remote_config = RemoteConfigFetcher.fetch_remote_config
-def _cached_fetch_remote_config() -> RemoteConfig:
- """Wrapper that caches the remote config for the entire test session."""
+def _cached_fetch_remote_config(require_fresh: bool = False) -> "RemoteConfigResult": # noqa: ARG001
+ """Wrapper that caches the remote config for the entire test session.
+
+ The ``require_fresh`` arg matches the new fetcher signature; ignored here because the
+ test-session cache exists precisely so we don't re-hit the network mid-suite.
+ """
if "config" not in _remote_config_cache:
- _remote_config_cache["config"] = _original_fetch_remote_config()
- return _remote_config_cache["config"]
+ result = _original_fetch_remote_config()
+ _remote_config_cache["config"] = result.config
+ return RemoteConfigResult(config=_remote_config_cache["config"], source=RemoteConfigSource.FRESH, cached_at=None)
# Session-level cache for pipelex service config to avoid flaky tests from concurrent file reads
diff --git a/tests/e2e/agent_cli/conftest.py b/tests/e2e/agent_cli/conftest.py
new file mode 100644
index 000000000..96d69cf62
--- /dev/null
+++ b/tests/e2e/agent_cli/conftest.py
@@ -0,0 +1,202 @@
+"""Fixtures for offline-mode E2E tests that spawn the agent CLI as a subprocess.
+
+The tests in this directory invoke ``.venv/bin/pipelex-agent`` directly so we exercise the
+exact surface that fails under the codex-sandbox handoff. To keep the runs hermetic each
+test gets its own ``HOME`` pointing at ``tmp_path`` — that controls both the location of
+the ``.pipelex/`` config directory AND the ``~/.pipelex/cache/`` directory the offline
+fallback reads.
+
+The kit configs are copied wholesale so the subprocess sees a realistic Pipelex install,
+not a hand-rolled minimal one. Per-test tweaks (gateway enabled/disabled, terms accepted,
+primed cache content) layer on top.
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from pipelex.kit.paths import get_kit_configs_dir
+from pipelex.system.pipelex_service.remote_config_cache import CACHE_SCHEMA_VERSION
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+PIPELEX_AGENT_BIN = REPO_ROOT / ".venv" / "bin" / "pipelex-agent"
+OFFLINE_BUNDLES_DIR = REPO_ROOT / "tests" / "e2e" / "data" / "offline_mode"
+
+# Reserved/unroutable address — httpx will raise ConnectError immediately without hanging.
+UNREACHABLE_REMOTE_CONFIG_URL = "http://127.0.0.1:1/pipelex_remote_config.json"
+
+
+def _copy_kit_configs_into(pipelex_dir: Path) -> None:
+ """Mirror the kit's ``configs/`` tree into ``pipelex_dir`` so the subprocess can boot.
+
+ Copies the top-level config files (``pipelex.toml``, ``telemetry.toml``,
+ ``pipelex_service.toml``, ``plxt.toml``) and the ``inference/`` subtree
+ (``backends.toml``, ``routing_profiles.toml``, ``backends/``, ``deck/``).
+ """
+ kit_configs_dir = Path(str(get_kit_configs_dir()))
+ pipelex_dir.mkdir(parents=True, exist_ok=True)
+
+ for top_level_file in ("pipelex.toml", "telemetry.toml", "pipelex_service.toml", "plxt.toml"):
+ shutil.copy2(kit_configs_dir / top_level_file, pipelex_dir / top_level_file)
+
+ inference_dir = pipelex_dir / "inference"
+ inference_dir.mkdir(exist_ok=True)
+ for inference_file in ("backends.toml", "routing_profiles.toml"):
+ shutil.copy2(kit_configs_dir / "inference" / inference_file, inference_dir / inference_file)
+
+ for subdir in ("backends", "deck"):
+ src = kit_configs_dir / "inference" / subdir
+ dst = inference_dir / subdir
+ dst.mkdir(exist_ok=True)
+ for source_file in src.glob("*.toml"):
+ shutil.copy2(source_file, dst / source_file.name)
+
+
+def set_gateway_enabled(backends_path: Path, *, enabled: bool) -> None:
+ """Toggle ``[pipelex_gateway].enabled`` in a backends.toml file in place.
+
+ Avoids loading the file through ``tomlkit`` (which the kit configs are encoded with)
+ by doing a targeted text rewrite — the kit file's ``enabled = true`` line directly
+ follows the ``[pipelex_gateway]`` section header.
+ """
+ original_text = backends_path.read_text(encoding="utf-8")
+ lines = original_text.splitlines(keepends=True)
+ target_value = "true" if enabled else "false"
+ in_gateway_section = False
+ rewrote = False
+ for index, line in enumerate(lines):
+ stripped = line.strip()
+ if stripped.startswith("[") and stripped.endswith("]"):
+ in_gateway_section = stripped == "[pipelex_gateway]"
+ continue
+ if in_gateway_section and stripped.startswith("enabled"):
+ indent = line[: len(line) - len(line.lstrip())]
+ lines[index] = f"{indent}enabled = {target_value} # set by offline-mode E2E fixture\n"
+ rewrote = True
+ break
+ if not rewrote:
+ msg = f"Could not find [pipelex_gateway].enabled in {backends_path}"
+ raise AssertionError(msg)
+ backends_path.write_text("".join(lines), encoding="utf-8")
+
+
+def write_pipelex_service_config(pipelex_dir: Path, *, terms_accepted: bool, inference_setup_completed: bool) -> None:
+ """Overwrite ``pipelex_service.toml`` so the subprocess skips the first-run gate."""
+ content = (
+ "[agreement]\n"
+ f"terms_accepted = {str(terms_accepted).lower()}\n"
+ "\n"
+ "[onboarding]\n"
+ f"inference_setup_completed = {str(inference_setup_completed).lower()}\n"
+ )
+ (pipelex_dir / "pipelex_service.toml").write_text(content, encoding="utf-8")
+
+
+def write_active_routing_profile(routing_profiles_path: Path, active_profile: str) -> None:
+ """Rewrite the ``active = "..."`` line in routing_profiles.toml in place."""
+ original_text = routing_profiles_path.read_text(encoding="utf-8")
+ lines = original_text.splitlines(keepends=True)
+ rewrote = False
+ for index, line in enumerate(lines):
+ if line.lstrip().startswith("active"):
+ lines[index] = f'active = "{active_profile}"\n'
+ rewrote = True
+ break
+ if not rewrote:
+ msg = f"Could not find 'active = ...' in {routing_profiles_path}"
+ raise AssertionError(msg)
+ routing_profiles_path.write_text("".join(lines), encoding="utf-8")
+
+
+def write_remote_config_cache(pipelex_dir: Path, raw_config: dict[str, Any]) -> Path:
+ """Write a primed ``cache/remote_config.json`` matching ``RemoteConfigCache``'s schema."""
+ cache_dir = pipelex_dir / "cache"
+ cache_dir.mkdir(parents=True, exist_ok=True)
+ cache_path = cache_dir / "remote_config.json"
+ payload = {
+ "schema_version": CACHE_SCHEMA_VERSION,
+ "cached_at": datetime.now(tz=timezone.utc).isoformat(),
+ "raw_config": raw_config,
+ }
+ cache_path.write_text(json.dumps(payload), encoding="utf-8")
+ return cache_path
+
+
+def hermetic_home(tmp_path: Path) -> Path:
+ """Return a tmp ``HOME`` that already has a populated ``.pipelex/`` config tree.
+
+ Tests further tweak the resulting directory (toggle gateway, prime cache, ...) before
+ invoking the subprocess.
+ """
+ pipelex_dir = tmp_path / ".pipelex"
+ _copy_kit_configs_into(pipelex_dir)
+ write_pipelex_service_config(pipelex_dir, terms_accepted=True, inference_setup_completed=True)
+ return tmp_path
+
+
+def offline_subprocess_env(hermetic_home: Path) -> dict[str, str]:
+ """Base subprocess env that points the CLI at the hermetic HOME and an unreachable
+ remote-config URL.
+
+ Tests extend this dict before passing it to ``subprocess.run``. Dummy credentials are
+ populated for every backend that the kit's ``backends.toml`` references so the dry-run
+ setup doesn't fail with ``InferenceBackendCredentialsError`` for backends we don't care
+ about — under ``needs_inference=False`` Pipelex is lenient on credential validation, but
+ only at the resolution layer; the env-var substitution itself still requires the var to
+ exist.
+ """
+ env: dict[str, str] = {
+ "HOME": str(hermetic_home),
+ "PATH": "/usr/bin:/bin:/usr/local/bin",
+ "PIPELEX_REMOTE_CONFIG_URL": UNREACHABLE_REMOTE_CONFIG_URL,
+ # Force CI test mode so vertexai is skipped and terms-acceptance gate doesn't fire
+ # for code paths that still consult the integration mode.
+ "RUN_MODE": "ci_test",
+ # Dummy credentials — the test never makes a real provider call (dry-run only) and
+ # ``lenient=True`` (set by ``--dry-run``) skips backends whose env vars are missing
+ # rather than raising. We populate every key so backend loading is deterministic
+ # regardless of the developer's shell environment.
+ "PIPELEX_GATEWAY_API_KEY": "dummy-gateway-key",
+ "ANTHROPIC_API_KEY": "dummy-anthropic-key",
+ "AWS_REGION": "us-east-1",
+ "AZURE_API_BASE": "https://example.invalid",
+ "AZURE_API_KEY": "dummy-azure-key",
+ "AZURE_API_VERSION": "2024-01-01",
+ "BLACKBOX_API_KEY": "dummy-blackbox-key",
+ "FAL_API_KEY": "dummy-fal-key",
+ "GCP_CREDENTIALS_FILE_PATH": "/dev/null",
+ "GCP_LOCATION": "us-central1",
+ "GCP_PROJECT_ID": "dummy-project",
+ "GOOGLE_API_KEY": "dummy-google-key",
+ "GROQ_API_KEY": "dummy-groq-key",
+ "HF_TOKEN": "dummy-hf-token",
+ "LINKUP_API_KEY": "dummy-linkup-key",
+ "MINIMAX_API_KEY": "dummy-minimax-key",
+ "MISTRAL_API_KEY": "dummy-mistral-key",
+ "OPENAI_API_KEY": "dummy-openai-key",
+ "OPENROUTER_API_KEY": "dummy-openrouter-key",
+ "PORTKEY_API_KEY": "dummy-portkey-key",
+ "SCALEWAY_API_KEY": "dummy-scaleway-key",
+ "SCALEWAY_ENDPOINT": "https://example.invalid",
+ "XAI_API_KEY": "dummy-xai-key",
+ }
+ return env
+
+
+# Public helpers re-exported for use from test modules.
+__all__ = [
+ "OFFLINE_BUNDLES_DIR",
+ "PIPELEX_AGENT_BIN",
+ "UNREACHABLE_REMOTE_CONFIG_URL",
+ "set_gateway_enabled",
+ "write_active_routing_profile",
+ "write_remote_config_cache",
+]
diff --git a/tests/e2e/agent_cli/test_offline_run_dry.py b/tests/e2e/agent_cli/test_offline_run_dry.py
new file mode 100644
index 000000000..c06e32d31
--- /dev/null
+++ b/tests/e2e/agent_cli/test_offline_run_dry.py
@@ -0,0 +1,294 @@
+"""E2E tests for offline-mode dry-run via the agent CLI subprocess.
+
+These tests pin Phase 5 of the offline-mode plan: invoke the real ``pipelex-agent run
+bundle ... --dry-run --mock-inputs`` against the same surface that fails today in
+codex-sandbox handoffs, with the remote-config URL pointed at an unreachable address so
+the network behaviour is honest (real ``httpx.ConnectError``) rather than mocked. Each
+test gets its own ``HOME``; the ``conftest`` fixtures handle the .pipelex/ scaffolding.
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+import subprocess # noqa: S404 — invokes the real pipelex-agent binary for E2E coverage
+from typing import TYPE_CHECKING, cast
+
+import pytest
+
+from tests.e2e.agent_cli.conftest import (
+ OFFLINE_BUNDLES_DIR,
+ PIPELEX_AGENT_BIN,
+ set_gateway_enabled,
+ write_active_routing_profile,
+ write_remote_config_cache,
+)
+
+if TYPE_CHECKING:
+ from pathlib import Path
+
+
+def _stage_bundle(source_bundle_dir: Path, dest_root: Path) -> Path:
+ """Copy a bundle directory into ``dest_root`` so the agent CLI's on-disk side effects
+ (e.g. ``dry_run.json`` written alongside the bundle) never pollute the source tree.
+ """
+ staged = dest_root / source_bundle_dir.name
+ shutil.copytree(source_bundle_dir, staged)
+ return staged
+
+
+def _run_agent_bundle(bundle_dir: Path, env: dict[str, str], cwd: Path) -> subprocess.CompletedProcess[str]:
+ """Invoke ``pipelex-agent run bundle <dir> --dry-run --mock-inputs`` and capture output.
+
+ The ``cwd`` MUST be set to the hermetic HOME so ``find_project_root`` stops walking at
+ ``Path.home()`` and the subprocess uses the test's ``.pipelex/`` instead of the
+ repository's project-level one. ``bundle_dir`` SHOULD be a staged copy (see
+ ``_stage_bundle``) — the CLI writes ``dry_run.json`` next to the bundle file.
+ """
+ return subprocess.run( # noqa: S603
+ [
+ str(PIPELEX_AGENT_BIN),
+ "run",
+ "bundle",
+ str(bundle_dir),
+ "--dry-run",
+ "--mock-inputs",
+ "--no-graph",
+ ],
+ env=env,
+ cwd=str(cwd),
+ capture_output=True,
+ text=True,
+ check=False,
+ timeout=120,
+ )
+
+
+def _parse_agent_json(output: str) -> dict[str, object]:
+ """Parse the last TOP-LEVEL JSON object in agent CLI output.
+
+ Walks the output forward; on each successful ``raw_decode`` we advance past the entire
+ decoded object so nested dicts inside an already-decoded envelope are NOT re-parsed.
+ Without that skip, ``json.dumps(envelope, indent=2)`` of a nested-dict payload would
+ cause the helper to return whichever inner dict ``raw_decode`` lands on last.
+
+ Returning the last top-level dict (rather than the first) handles the case where the
+ agent CLI emits a structured-log preamble before the real ``agent_success`` /
+ ``agent_error`` envelope.
+ """
+ output = output.strip()
+ if not output:
+ msg = "Expected JSON output from agent CLI but got empty string"
+ raise AssertionError(msg)
+ decoder = json.JSONDecoder()
+ last_parsed: dict[str, object] | None = None
+ start = 0
+ while start < len(output):
+ if output[start] != "{":
+ start += 1
+ continue
+ try:
+ parsed, consumed = decoder.raw_decode(output[start:])
+ except json.JSONDecodeError:
+ start += 1
+ continue
+ if isinstance(parsed, dict):
+ last_parsed = cast("dict[str, object]", parsed)
+ # Skip past the entire decoded value so we don't re-enter nested dicts.
+ start += consumed
+ if last_parsed is not None:
+ return last_parsed
+ msg = f"Could not find JSON object in agent output: {output!r}"
+ raise AssertionError(msg)
+
+
+# A minimal but structurally complete cached remote-config payload. ``backend_model_specs``
+# must include every terminal handle referenced by the default kit deck so the gateway
+# membership check passes for the "known-model" scenario.
+def _comprehensive_cached_backend_model_specs() -> dict[str, object]:
+ """Build a ``backend_model_specs`` payload that satisfies the default kit deck.
+
+ The kit's ``1_llm_deck.toml`` references many concrete handles via aliases/waterfalls.
+ We list them all here so the gateway-membership check in ``Pipelex.setup`` accepts the
+ cache. The spec body uses ``gateway_completions`` (LLMs) / ``gateway_image`` (image-gen)
+ sdks since the gateway proxies to those — values are minimal but valid against
+ ``InferenceModelSpecBlueprint``.
+ """
+ llm_handle_template: dict[str, object] = {
+ "sdk": "gateway_completions",
+ "model_type": "llm",
+ "inputs": ["text"],
+ "outputs": ["text", "structured"],
+ }
+ img_gen_handle_template: dict[str, object] = {
+ "sdk": "gateway_image",
+ "model_type": "img_gen",
+ }
+ extract_handle_template: dict[str, object] = {
+ "sdk": "gateway_extract",
+ "model_type": "text_extractor",
+ }
+ search_handle_template: dict[str, object] = {
+ "sdk": "gateway_search",
+ "model_type": "search",
+ }
+
+ llm_handles = [
+ "claude-3.7-sonnet",
+ "claude-4.5-haiku",
+ "claude-4.5-sonnet",
+ "claude-4.6-opus",
+ "claude-4.6-sonnet",
+ "claude-4.7-opus",
+ "gemini-flash-latest",
+ "gemini-flash-lite-latest",
+ "gemini-pro-latest",
+ "gpt-4o",
+ "gpt-4o-mini",
+ "gpt-5",
+ "gpt-5-mini",
+ "gpt-5-nano",
+ "gpt-5.1",
+ "gpt-5.2",
+ "mistral-large",
+ ]
+ img_gen_handles = [
+ "gpt-image-1",
+ "gpt-image-1-mini",
+ "gpt-image-2",
+ "nano-banana",
+ "nano-banana-2",
+ "nano-banana-pro",
+ ]
+ extract_handles = ["azure-document-intelligence", "linkup-fetch"]
+ search_handles = ["linkup-standard", "linkup-deep"]
+
+ specs: dict[str, object] = {"defaults": {"sdk": "gateway_completions"}}
+ for handle in llm_handles:
+ specs[handle] = dict(llm_handle_template)
+ for handle in img_gen_handles:
+ specs[handle] = dict(img_gen_handle_template)
+ for handle in extract_handles:
+ specs[handle] = dict(extract_handle_template)
+ for handle in search_handles:
+ specs[handle] = dict(search_handle_template)
+ return specs
+
+
+def _cached_remote_config_payload() -> dict[str, object]:
+ return {
+ "posthog": {
+ "project_api_key": "test-key",
+ "endpoint": "https://example.invalid",
+ "is_geoip_enabled": False,
+ "is_debug_enabled": False,
+ },
+ "backend_model_specs": _comprehensive_cached_backend_model_specs(),
+ "aws_region": "us-east-1",
+ }
+
+
[email protected]_disabled # Slow subprocess-based E2E; runs locally and on PR-gated workflows.
+class TestOfflineDryRun:
+ def test_parse_agent_json_returns_last_when_preamble_present(self) -> None:
+ """Regression: a JSON-shaped log preamble must not shadow the real agent envelope.
+
+ Agent CLI output sometimes begins with structured log lines before the
+ ``agent_success`` / ``agent_error`` envelope. ``_parse_agent_json`` must walk past
+ the preamble and return the LAST top-level decoded dict.
+ """
+ output = '{"level": "info", "msg": "starting"}\n{"text": "real result"}'
+ parsed = _parse_agent_json(output)
+ assert parsed == {"text": "real result"}, parsed
+
+ def test_parse_agent_json_returns_envelope_not_nested_dict(self) -> None:
+ """Regression: a pretty-printed envelope with nested dicts must surface the OUTER
+ envelope, not the last nested object.
+
+ ``json.dumps(payload, indent=2)`` puts every nested ``{`` on its own line. A naive
+ walker would re-decode each nested dict at its own start position and overwrite
+ ``last_parsed`` with the deepest one — which would silently break every assertion
+ in this test class that reads envelope-level keys (``"error"``, ``"text"``, etc.).
+ The helper must advance past the entire decoded value after each top-level parse.
+ """
+ envelope = {
+ "text": "mock result",
+ "warnings": [{"id": "WARN-1", "type": "RemoteConfigStale"}],
+ }
+ parsed = _parse_agent_json(json.dumps(envelope, indent=2))
+ assert parsed == envelope, parsed
+ assert "text" in parsed, f"the outer envelope (with 'text') must be returned, not a nested dict: {parsed!r}"
+ assert "warnings" in parsed, f"the outer envelope (with 'warnings') must be returned: {parsed!r}"
+
+ def test_byok_offline_succeeds(self, hermetic_home: Path, offline_subprocess_env: dict[str, str]) -> None:
+ """Gateway disabled + no network + no cache → dry-run exits 0 with structured success JSON."""
+ pipelex_dir = hermetic_home / ".pipelex"
+ set_gateway_enabled(pipelex_dir / "inference" / "backends.toml", enabled=False)
+ write_active_routing_profile(pipelex_dir / "inference" / "routing_profiles.toml", "all_anthropic")
+
+ staged_bundle = _stage_bundle(OFFLINE_BUNDLES_DIR / "byok_simple", hermetic_home)
+ result = _run_agent_bundle(staged_bundle, offline_subprocess_env, cwd=hermetic_home)
+
+ assert result.returncode == 0, f"BYOK dry-run must succeed offline.\nstdout={result.stdout!r}\nstderr={result.stderr!r}"
+ payload = _parse_agent_json(result.stdout)
+ # A successful run envelope has no "error" field; the dry-run renders the pipe's
+ # synthetic output under "text" (or a typed key) instead.
+ assert "error" not in payload, payload
+ assert "text" in payload, payload
+
+ def test_gateway_no_cache_no_network_fails_with_unavailable(self, hermetic_home: Path, offline_subprocess_env: dict[str, str]) -> None:
+ """Gateway enabled + no network + no cache → ``RemoteConfigUnavailableError`` with remediation hint."""
+ pipelex_dir = hermetic_home / ".pipelex"
+ set_gateway_enabled(pipelex_dir / "inference" / "backends.toml", enabled=True)
+
+ staged_bundle = _stage_bundle(OFFLINE_BUNDLES_DIR / "gateway_known_model", hermetic_home)
+ result = _run_agent_bundle(staged_bundle, offline_subprocess_env, cwd=hermetic_home)
+
+ assert result.returncode != 0, f"Gateway offline with no cache must fail.\nstdout={result.stdout!r}\nstderr={result.stderr!r}"
+ payload = _parse_agent_json(result.stderr or result.stdout)
+ assert payload.get("error_type") == "RemoteConfigUnavailableError", payload
+ message = str(payload.get("message", ""))
+ assert "pipelex init" in message.lower(), f"Error must mention `pipelex init` remediation; got: {message!r}"
+
+ def test_gateway_known_with_cache_succeeds_offline(self, hermetic_home: Path, offline_subprocess_env: dict[str, str]) -> None:
+ """Gateway enabled + no network + primed cache → dry-run exits 0 with stale-cache warning."""
+ pipelex_dir = hermetic_home / ".pipelex"
+ set_gateway_enabled(pipelex_dir / "inference" / "backends.toml", enabled=True)
+ write_remote_config_cache(pipelex_dir, _cached_remote_config_payload())
+
+ staged_bundle = _stage_bundle(OFFLINE_BUNDLES_DIR / "gateway_known_model", hermetic_home)
+ result = _run_agent_bundle(staged_bundle, offline_subprocess_env, cwd=hermetic_home)
+
+ assert result.returncode == 0, f"Gateway dry-run with primed cache must succeed offline.\nstdout={result.stdout!r}\nstderr={result.stderr!r}"
+ payload = _parse_agent_json(result.stdout)
+ assert "error" not in payload, payload
+ rendered = json.dumps(payload)
+ assert '"type": "RemoteConfigStale"' in rendered, (
+ f"Cached-source setup must surface a RemoteConfigStale warning on the envelope; got: {payload!r}"
+ )
+
+ def test_gateway_unknown_with_cache_fails_with_clear_error(self, hermetic_home: Path, offline_subprocess_env: dict[str, str]) -> None:
+ """Bundle pipe references a model absent from the cached gateway specs → clear error.
+
+ The default kit deck only references "known" gateway handles, so this scenario fires
+ at the pipe-operator layer (the membership check at ``ModelManager.setup`` looks at
+ deck presets/choice defaults, not raw per-pipe model strings). Either way the agent
+ CLI must surface a structured error with the unknown handle visible — that is the
+ user-facing contract.
+ """
+ pipelex_dir = hermetic_home / ".pipelex"
+ set_gateway_enabled(pipelex_dir / "inference" / "backends.toml", enabled=True)
+ write_remote_config_cache(pipelex_dir, _cached_remote_config_payload())
+
+ staged_bundle = _stage_bundle(OFFLINE_BUNDLES_DIR / "gateway_unknown_model", hermetic_home)
+ result = _run_agent_bundle(staged_bundle, offline_subprocess_env, cwd=hermetic_home)
+
+ assert result.returncode != 0, f"Unknown-model bundle must fail.\nstdout={result.stdout!r}\nstderr={result.stderr!r}"
+ payload = _parse_agent_json(result.stderr or result.stdout)
+ # The unknown handle should appear somewhere visible in the JSON envelope (message
+ # or a structured field). Don't pin the error_type — the failure can surface either
+ # as ``GatewayUnknownModelError`` (if the deck reaches the membership check) or as
+ # ``PipeOperatorModelAvailabilityError`` (raw model string in the pipe). Both carry
+ # the model name; what matters is that the agent gets a clear, structured signal.
+ rendered = json.dumps(payload)
+ assert "gpt-future-fake-99" in rendered, f"Unknown-model error must surface the model name in the JSON envelope; got: {payload!r}"
diff --git a/tests/e2e/data/offline_mode/byok_simple/byok_simple.mthds b/tests/e2e/data/offline_mode/byok_simple/byok_simple.mthds
new file mode 100644
index 000000000..01e437a3c
--- /dev/null
+++ b/tests/e2e/data/offline_mode/byok_simple/byok_simple.mthds
@@ -0,0 +1,12 @@
+domain = "offline_byok_simple"
+description = "Minimal BYOK pipeline used by offline-mode E2E tests"
+main_pipe = "byok_simple_pipe"
+
+[pipe]
+[pipe.byok_simple_pipe]
+type = "PipeLLM"
+description = "Single LLM step using a non-gateway backend model handle"
+inputs = { topic = "Text" }
+output = "Text"
+model = "claude-4.5-haiku"
+prompt = "Write a single sentence about $topic.text."
diff --git a/tests/e2e/data/offline_mode/gateway_known_model/gateway_known_model.mthds b/tests/e2e/data/offline_mode/gateway_known_model/gateway_known_model.mthds
new file mode 100644
index 000000000..d7087a6ab
--- /dev/null
+++ b/tests/e2e/data/offline_mode/gateway_known_model/gateway_known_model.mthds
@@ -0,0 +1,12 @@
+domain = "offline_gateway_known"
+description = "Pipeline referencing a gateway model that exists in the cached gateway specs"
+main_pipe = "gateway_known_pipe"
+
+[pipe]
+[pipe.gateway_known_pipe]
+type = "PipeLLM"
+description = "Single LLM step using a gateway-listed model handle"
+inputs = { topic = "Text" }
+output = "Text"
+model = "gpt-4o-mini"
+prompt = "Write a single sentence about $topic.text."
diff --git a/tests/e2e/data/offline_mode/gateway_unknown_model/gateway_unknown_model.mthds b/tests/e2e/data/offline_mode/gateway_unknown_model/gateway_unknown_model.mthds
new file mode 100644
index 000000000..05b52e7fa
--- /dev/null
+++ b/tests/e2e/data/offline_mode/gateway_unknown_model/gateway_unknown_model.mthds
@@ -0,0 +1,12 @@
+domain = "offline_gateway_unknown"
+description = "Pipeline referencing a gateway model that does NOT exist in the cached gateway specs"
+main_pipe = "gateway_unknown_pipe"
+
+[pipe]
+[pipe.gateway_unknown_pipe]
+type = "PipeLLM"
+description = "Single LLM step using a gateway model handle that is intentionally absent from the test fixture"
+inputs = { topic = "Text" }
+output = "Text"
+model = "gpt-future-fake-99"
+prompt = "Write a single sentence about $topic.text."
diff --git a/tests/integration/pipelex/cli/commands/init/test_cache_priming.py b/tests/integration/pipelex/cli/commands/init/test_cache_priming.py
new file mode 100644
index 000000000..5c2548dca
--- /dev/null
+++ b/tests/integration/pipelex/cli/commands/init/test_cache_priming.py
@@ -0,0 +1,389 @@
+"""Phase 4 contract: ``prime_remote_config_cache`` writes the on-disk cache during init.
+
+These tests pin the helper's behaviour:
+
+- When the gateway is enabled and terms are accepted, a successful online fetch persists
+ the raw payload to ``~/.pipelex/cache/remote_config.json`` (priming).
+- When offline at init time, priming logs a yellow warning and continues — the cache stays
+ empty, the user has been told, init does not crash.
+- When the gateway is disabled in ``backends.toml``, priming is a no-op (BYOK setups have
+ nothing to cache).
+- When a cache already exists, a fresh fetch overwrites it (refresh, not skip).
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import datetime, timezone
+from pathlib import Path # noqa: TC003 — referenced by pytest fixture type hints at runtime
+from typing import TYPE_CHECKING
+
+import httpx
+import pytest
+from rich.console import Console
+
+from pipelex.cli.commands.init.command import prime_remote_config_cache
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.pipelex_service.pipelex_service_agreement import (
+ PipelexServiceAgreement,
+ PipelexServiceOnboarding,
+)
+from pipelex.system.pipelex_service.pipelex_service_config import PipelexServiceConfig
+from pipelex.system.pipelex_service.remote_config_cache import (
+ CACHE_SCHEMA_VERSION,
+ CachedRemoteConfig,
+ RemoteConfigCache,
+)
+from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+INIT_COMMAND_MODULE = "pipelex.cli.commands.init.command"
+
+# Capture the unpatched classmethod at module import — the session conftest replaces
+# ``fetch_remote_config`` with a cache shim that bypasses ``httpx``. We need the real
+# fetch path so the ``httpx.get`` mock actually gets exercised.
+_ORIGINAL_FETCH_REMOTE_CONFIG = RemoteConfigFetcher.fetch_remote_config
+
+
+def _accepted_service_config() -> PipelexServiceConfig:
+ return PipelexServiceConfig(
+ agreement=PipelexServiceAgreement(terms_accepted=True),
+ onboarding=PipelexServiceOnboarding(inference_setup_completed=True),
+ )
+
+
+def _fake_remote_payload() -> dict[str, object]:
+ """Minimal valid payload for ``RemoteConfig.model_validate``."""
+ return {
+ "posthog": {
+ "project_api_key": "test-key",
+ "endpoint": "https://example.invalid",
+ "is_geoip_enabled": False,
+ "is_debug_enabled": False,
+ },
+ "backend_model_specs": {},
+ "aws_region": "us-east-1",
+ }
+
+
+def _store_malformed_cache(remote_config_payload: dict[str, object]) -> None:
+ """Drop-in for ``RemoteConfigCache.store`` that writes a structurally valid cache *wrapper*
+ whose inner ``raw_config`` does NOT validate as a ``RemoteConfig``.
+
+ Reproduces the case where ``RemoteConfigCache.load()`` succeeds (the wrapper is fine) but the
+ cached payload is unusable for offline runs.
+ """
+ del remote_config_payload # intentionally discarded — we write a deliberately broken payload
+ cached = CachedRemoteConfig(
+ schema_version=CACHE_SCHEMA_VERSION,
+ cached_at=datetime.now(tz=timezone.utc),
+ raw_config={},
+ )
+ cache_path = RemoteConfigCache.cache_path()
+ cache_path.parent.mkdir(parents=True, exist_ok=True)
+ cache_path.write_text(cached.model_dump_json(), encoding="utf-8")
+
+
+def _make_httpx_response(payload: dict[str, object]) -> httpx.Response:
+ return httpx.Response(
+ status_code=200,
+ request=httpx.Request("GET", "https://example.com/remote_config.json"),
+ content=json.dumps(payload).encode("utf-8"),
+ )
+
+
+def isolated_cache_dir(tmp_path: Path, mocker: MockerFixture) -> Path:
+ """Redirect ``~/.pipelex`` (and therefore the cache) at a tmp path."""
+ fake_global_dir = tmp_path / ".pipelex"
+ mocker.patch.object(
+ ConfigLoader,
+ "global_config_dir",
+ new_callable=mocker.PropertyMock,
+ return_value=fake_global_dir,
+ )
+ return fake_global_dir
+
+
+class TestCachePriming:
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_init_primes_cache_when_online(self, mocker: MockerFixture) -> None:
+ """Gateway enabled + terms accepted + online → cache file written under the global dir."""
+ mocker.patch(f"{INIT_COMMAND_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{INIT_COMMAND_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch("httpx.get", return_value=_make_httpx_response(_fake_remote_payload()))
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console)
+
+ cache_path = RemoteConfigCache.cache_path()
+ assert cache_path.exists(), "priming should write the on-disk cache"
+ on_disk = json.loads(cache_path.read_text(encoding="utf-8"))
+ assert on_disk["raw_config"]["aws_region"] == "us-east-1"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_init_warns_when_offline(self, mocker: MockerFixture) -> None:
+ """Gateway enabled + terms accepted + offline + no cache → warn, do not crash, no cache file."""
+ mocker.patch(f"{INIT_COMMAND_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{INIT_COMMAND_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console) # must NOT raise
+
+ cache_path = RemoteConfigCache.cache_path()
+ assert not cache_path.exists(), "priming must not create a cache file when offline"
+ # Verify the user actually got a visible warning, not a silent swallow.
+ printed = " ".join(str(call_args) for call_args in console.print.call_args_list)
+ assert "yellow" in printed.lower(), f"priming offline must print a yellow warning; got: {printed!r}"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_init_skips_priming_when_gateway_disabled(self, mocker: MockerFixture) -> None:
+ """Gateway disabled in backends → priming is a no-op (no fetch, no cache write)."""
+ mocker.patch(f"{INIT_COMMAND_MODULE}.is_pipelex_gateway_enabled", return_value=False)
+ fetch_spy = mocker.spy(RemoteConfigFetcher, "fetch_remote_config")
+ httpx_get_mock = mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console)
+
+ assert fetch_spy.call_count == 0, "priming must not invoke the fetcher when gateway is disabled"
+ assert httpx_get_mock.call_count == 0, "priming must not hit the network when gateway is disabled"
+ assert not RemoteConfigCache.cache_path().exists(), "no cache should be written when gateway is disabled"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_init_warns_when_offline_with_stale_cache_present(self, mocker: MockerFixture) -> None:
+ """Stale cache exists + offline → priming surfaces an error (not a silent success).
+
+ The fetcher with ``require_fresh=True`` refuses to accept a cached fallback, so an
+ offline fetch fails even though a usable cache is on disk. The pre-existing cache file
+ must NOT be wiped — subsequent offline dry-runs should still be able to use it.
+ """
+ stale_payload = _fake_remote_payload()
+ stale_payload["aws_region"] = "eu-west-1"
+ RemoteConfigCache.store(stale_payload)
+ cache_path = RemoteConfigCache.cache_path()
+ stale_on_disk_before = json.loads(cache_path.read_text(encoding="utf-8"))
+
+ mocker.patch(f"{INIT_COMMAND_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{INIT_COMMAND_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console) # must NOT raise
+
+ printed = " ".join(str(call_args) for call_args in console.print.call_args_list)
+ assert "yellow" in printed.lower(), f"stale-cache-offline priming must warn; got: {printed!r}"
+ stale_on_disk_after = json.loads(cache_path.read_text(encoding="utf-8"))
+ assert stale_on_disk_after == stale_on_disk_before, "stale cache must be left intact when priming refuses to use it"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_priming_reads_backends_toml_from_target_dir(self, mocker: MockerFixture, tmp_path: Path) -> None:
+ """``target_config_dir`` overrides the layered backends.toml so global vs local init
+ primes based on the directory being initialized — not on whatever the layered config
+ resolves to first.
+
+ Setup:
+ - layered ``backends.toml`` (default ``config_manager.backends_file_path``) has gateway DISABLED.
+ - target_config_dir's ``backends.toml`` has gateway ENABLED.
+
+ Expected: priming runs (gateway enabled at the target). Without the fix, the helper
+ would consult the layered file and skip priming entirely.
+ """
+ # Layered config says gateway is disabled — without the fix, this is what gets read.
+ layered_dir = tmp_path / "layered_dir"
+ layered_backends = layered_dir / "inference" / "backends.toml"
+ layered_backends.parent.mkdir(parents=True, exist_ok=True)
+ layered_backends.write_text("[pipelex_gateway]\nenabled = false\n", encoding="utf-8")
+ mocker.patch.object(
+ ConfigLoader,
+ "backends_file_path",
+ new_callable=mocker.PropertyMock,
+ return_value=layered_backends,
+ )
+
+ # Target init dir says gateway IS enabled. The priming helper must read THIS file.
+ target_dir = tmp_path / "target_dir"
+ target_backends = target_dir / "inference" / "backends.toml"
+ target_backends.parent.mkdir(parents=True, exist_ok=True)
+ target_backends.write_text("[pipelex_gateway]\nenabled = true\n", encoding="utf-8")
+
+ # Terms-accepted check still consults the global pipelex_service.toml — keep that mocked
+ # as before. Make the fetcher succeed so we can assert the cache is actually written.
+ mocker.patch(
+ f"{INIT_COMMAND_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch("httpx.get", return_value=_make_httpx_response(_fake_remote_payload()))
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console, target_config_dir=target_dir)
+
+ assert RemoteConfigCache.cache_path().exists(), (
+ "priming must run (and write the cache) when the TARGET backends.toml has gateway "
+ "enabled, even if the layered backends.toml says otherwise"
+ )
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_priming_skips_when_target_dir_says_gateway_disabled(self, mocker: MockerFixture, tmp_path: Path) -> None:
+ """Inverse of the above: target says gateway disabled, layered says enabled → skip.
+
+ Without the fix, the helper would see the layered "enabled" file, try to fetch, and
+ write a cache the user did not opt in to. We assert no fetch attempt and no cache file.
+ """
+ # Layered says gateway enabled — without the fix, this is what gets read.
+ layered_dir = tmp_path / "layered_dir"
+ layered_backends = layered_dir / "inference" / "backends.toml"
+ layered_backends.parent.mkdir(parents=True, exist_ok=True)
+ layered_backends.write_text("[pipelex_gateway]\nenabled = true\n", encoding="utf-8")
+ mocker.patch.object(
+ ConfigLoader,
+ "backends_file_path",
+ new_callable=mocker.PropertyMock,
+ return_value=layered_backends,
+ )
+
+ # Target says gateway disabled — priming should respect that and become a no-op.
+ target_dir = tmp_path / "target_dir"
+ target_backends = target_dir / "inference" / "backends.toml"
+ target_backends.parent.mkdir(parents=True, exist_ok=True)
+ target_backends.write_text("[pipelex_gateway]\nenabled = false\n", encoding="utf-8")
+
+ fetch_spy = mocker.spy(RemoteConfigFetcher, "fetch_remote_config")
+ httpx_get_mock = mocker.patch("httpx.get", side_effect=httpx.ConnectError("should not be called"))
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console, target_config_dir=target_dir)
+
+ assert fetch_spy.call_count == 0, "target dir disables the gateway — priming must NOT consult the layered backends.toml"
+ assert httpx_get_mock.call_count == 0
+ assert not RemoteConfigCache.cache_path().exists()
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_init_warns_when_cache_write_fails(self, mocker: MockerFixture) -> None:
+ """Online fetch succeeds but the cache write fails → priming reports failure, not success.
+
+ The fetcher treats the cache write as opportunistic and swallows OSErrors with only a
+ stderr warning. If priming trusted the fetch result alone it would report ``primed=True``
+ while no usable cache exists, so ``pipelex-agent init`` would emit ``cache_primed: true``
+ and a later offline run would hit ``RemoteConfigUnavailableError``.
+ """
+ mocker.patch(f"{INIT_COMMAND_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{INIT_COMMAND_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch("httpx.get", return_value=_make_httpx_response(_fake_remote_payload()))
+ mocker.patch.object(RemoteConfigCache, "store", side_effect=OSError("read-only cache directory"))
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console) # must NOT raise
+
+ cache_path = RemoteConfigCache.cache_path()
+ assert not cache_path.exists(), "no cache file should exist when the write failed"
+ printed = " ".join(str(call_args) for call_args in console.print.call_args_list)
+ assert "yellow" in printed.lower(), f"a failed cache write must surface a yellow warning; got: {printed!r}"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_init_warns_when_cached_payload_is_malformed(self, mocker: MockerFixture) -> None:
+ """Online fetch succeeds and a cache file is written, but its inner ``raw_config`` is not
+ a valid ``RemoteConfig`` → priming reports failure, not success.
+
+ ``RemoteConfigCache.load()`` validates only the cache *wrapper*, so a malformed inner
+ payload would still pass an ``is None`` check. Priming must re-validate the payload as a
+ ``RemoteConfig``, otherwise ``pipelex-agent init`` would emit ``cache_primed: true`` while
+ a later offline run hits ``RemoteConfigUnavailableError``.
+ """
+ mocker.patch(f"{INIT_COMMAND_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{INIT_COMMAND_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch("httpx.get", return_value=_make_httpx_response(_fake_remote_payload()))
+ mocker.patch.object(RemoteConfigCache, "store", side_effect=_store_malformed_cache)
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console) # must NOT raise
+
+ printed = " ".join(str(call_args) for call_args in console.print.call_args_list)
+ assert "yellow" in printed.lower(), f"a malformed cached payload must surface a yellow warning; got: {printed!r}"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_init_does_not_double_prime(self, mocker: MockerFixture) -> None:
+ """When a cache already exists, priming online overwrites it (refresh, not skip)."""
+ # Pre-populate the cache with a stale snapshot so we can prove the file gets rewritten.
+ stale_payload = _fake_remote_payload()
+ stale_payload["aws_region"] = "eu-west-1"
+ RemoteConfigCache.store(stale_payload)
+ cache_path = RemoteConfigCache.cache_path()
+ stale_on_disk = json.loads(cache_path.read_text(encoding="utf-8"))
+ assert stale_on_disk["raw_config"]["aws_region"] == "eu-west-1"
+
+ mocker.patch(f"{INIT_COMMAND_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{INIT_COMMAND_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch("httpx.get", return_value=_make_httpx_response(_fake_remote_payload()))
+
+ console = mocker.create_autospec(Console, instance=True)
+ prime_remote_config_cache(console)
+
+ fresh_on_disk = json.loads(cache_path.read_text(encoding="utf-8"))
+ assert fresh_on_disk["raw_config"]["aws_region"] == "us-east-1", "priming must overwrite the existing cache"
+ assert fresh_on_disk["cached_at"] >= stale_on_disk["cached_at"], "refresh must update the timestamp"
diff --git a/tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py b/tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py
new file mode 100644
index 000000000..db1b72463
--- /dev/null
+++ b/tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py
@@ -0,0 +1,167 @@
+"""GatewayUnknownModelError contract.
+
+When a model deck references a handle that the gateway should provide but the gateway's
+model specs (fresh or cached) don't contain it, setup must raise ``GatewayUnknownModelError``
+with provenance so the message can hint stale-cache remediation. This complements the
+existing ``LLMHandleNotFoundError`` path: the gateway-specific check fires even when
+``missing_presets_reaction = "log"`` (the default), because stale gateway specs are a
+distinct, user-actionable failure mode that deserves its own error class.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pytest
+
+from pipelex import log
+from pipelex.cogt.exceptions import GatewayUnknownModelError
+from pipelex.pipelex import Pipelex
+from pipelex.system.pipelex_service.remote_config import PipelexPosthogConfig, RemoteConfig
+from pipelex.system.pipelex_service.remote_config_fetcher import (
+ RemoteConfigFetcher,
+ RemoteConfigResult,
+)
+from pipelex.system.pipelex_service.types import RemoteConfigSource
+from pipelex.system.runtime import IntegrationMode
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from pytest_mock import MockerFixture
+
+PIPELEX_MODULE = "pipelex.pipelex"
+
+
+def _empty_gateway_remote_config_result(source: RemoteConfigSource) -> RemoteConfigResult:
+ """Build a ``RemoteConfigResult`` whose gateway has no model specs.
+
+ Forces every deck-referenced handle that should come from the gateway to be missing,
+ so the new membership check trips on the first one.
+ """
+ config = RemoteConfig(
+ posthog=PipelexPosthogConfig(
+ project_api_key="",
+ endpoint="https://dummy.example.com",
+ is_geoip_enabled=False,
+ is_debug_enabled=False,
+ ),
+ backend_model_specs={},
+ aws_region="us-east-1",
+ )
+ return RemoteConfigResult(config=config, source=source, cached_at=None)
+
+
[email protected](scope="module", autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Override the global module fixture: each test handles its own ``Pipelex.make``."""
+ yield
+ Pipelex.teardown_if_needed()
+
+
+class TestGatewayUnknownModel:
+ def test_known_model_loads(self) -> None:
+ """Happy path: the session-cached gateway config contains every model referenced by
+ the default deck, so the new gateway-membership check should be silent.
+ """
+ Pipelex.teardown_if_needed()
+ try:
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=True,
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
+
+ def test_unknown_model_fresh_raises(self, mocker: MockerFixture) -> None:
+ """Gateway returns no model specs → first deck-referenced gateway handle trips
+ ``GatewayUnknownModelError(source=FRESH)`` with the missing model name surfaced.
+ """
+ Pipelex.teardown_if_needed()
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(
+ RemoteConfigFetcher,
+ "fetch_remote_config",
+ return_value=_empty_gateway_remote_config_result(RemoteConfigSource.FRESH),
+ )
+
+ try:
+ with pytest.raises(GatewayUnknownModelError) as exc_info:
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=True,
+ )
+ assert exc_info.value.source == RemoteConfigSource.FRESH
+ assert exc_info.value.model_name, "the error must carry the missing model name"
+ assert exc_info.value.model_name in str(exc_info.value), "the error message must surface the missing model name"
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
+
+ def test_dummy_specs_path_skips_membership_check(self, mocker: MockerFixture) -> None:
+ """When gateway is enabled but ``needs_model_specs=False``, ``Pipelex.setup`` builds a
+ dummy ``RemoteConfig`` with empty ``backend_model_specs``. The membership check must NOT
+ run on this path — its provenance is "no live gateway data," so validating the deck's
+ gateway handles against an empty spec set would always fail. This covers read-only
+ flows like ``pipelex-agent models`` (no ``--backend``) where the user did not opt in to
+ fetching specs.
+ """
+ Pipelex.teardown_if_needed()
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ # Spy on the real fetcher: setup must not call it when needs_model_specs=False.
+ fetch_spy = mocker.spy(RemoteConfigFetcher, "fetch_remote_config")
+
+ try:
+ # No ``pytest.raises`` — the contract here is exactly that setup succeeds.
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=False,
+ )
+ assert fetch_spy.call_count == 0, "needs_model_specs=False must skip the remote fetch entirely (dummy config path)"
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
+
+ def test_unknown_model_cached_raises_with_stale_hint(self, mocker: MockerFixture) -> None:
+ """Same scenario as fresh, but the gateway config came from the cache → the error
+ message must point at ``pipelex init`` (while online) to refresh the cache.
+ """
+ Pipelex.teardown_if_needed()
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(
+ RemoteConfigFetcher,
+ "fetch_remote_config",
+ return_value=_empty_gateway_remote_config_result(RemoteConfigSource.CACHED),
+ )
+
+ try:
+ with pytest.raises(GatewayUnknownModelError) as exc_info:
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=True,
+ )
+ assert exc_info.value.source == RemoteConfigSource.CACHED
+ assert "pipelex init" in str(exc_info.value), "the cached-source variant must hint at re-priming via `pipelex init`"
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
diff --git a/tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py b/tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py
new file mode 100644
index 000000000..9c8eaf93a
--- /dev/null
+++ b/tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py
@@ -0,0 +1,141 @@
+"""Pins the offline-mode baseline.
+
+These tests describe the behaviour that must NOT regress:
+
+- BYOK setups (gateway disabled) must complete offline without any remote-config fetch.
+- Gateway-enabled setups with no network AND no cache must raise
+ ``RemoteConfigUnavailableError`` (the user-facing offline-mode error introduced in Phase 2).
+"""
+
+from __future__ import annotations
+
+from pathlib import Path # noqa: TC003 — referenced by pytest fixture type hints at runtime
+from typing import TYPE_CHECKING
+
+import httpx
+import pytest
+
+from pipelex import log
+from pipelex.pipelex import Pipelex
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.pipelex_service.exceptions import RemoteConfigUnavailableError
+from pipelex.system.pipelex_service.pipelex_service_agreement import (
+ PipelexServiceAgreement,
+ PipelexServiceOnboarding,
+)
+from pipelex.system.pipelex_service.pipelex_service_config import PipelexServiceConfig
+from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
+from pipelex.system.runtime import IntegrationMode
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from pytest_mock import MockerFixture
+
+PIPELEX_MODULE = "pipelex.pipelex"
+
+# Capture the original classmethod at module import — the session-scoped fixture in the root
+# conftest replaces ``RemoteConfigFetcher.fetch_remote_config`` with a wrapper that caches the
+# result for the whole test session. We need the unpatched version to exercise the real fetch
+# path against a mocked ``httpx.get``.
+_ORIGINAL_FETCH_REMOTE_CONFIG = RemoteConfigFetcher.fetch_remote_config
+
+
[email protected](scope="module", autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Override the global module fixture: this module sets up and tears down per test."""
+ yield
+ Pipelex.teardown_if_needed()
+
+
+class TestOfflineBaseline:
+ def test_byok_offline_setup_succeeds_without_fetching_remote_config(
+ self,
+ mocker: MockerFixture,
+ ) -> None:
+ """Gateway disabled: setup must succeed with no network and never hit remote config.
+
+ Even with ``needs_model_specs=True``, when the gateway is disabled the fetch is
+ unreachable and the BYOK setup should complete normally.
+ """
+ Pipelex.teardown_if_needed()
+
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=False)
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+
+ httpx_get_mock = mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+ fetch_spy = mocker.spy(RemoteConfigFetcher, "fetch_remote_config")
+
+ try:
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=True,
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+
+ assert fetch_spy.call_count == 0, "fetch_remote_config must not be invoked when gateway is disabled"
+ assert httpx_get_mock.call_count == 0, "httpx.get must not be invoked when gateway is disabled"
+
+ def test_gateway_offline_without_cache_raises_remote_config_unavailable_error(
+ self,
+ mocker: MockerFixture,
+ tmp_path: Path,
+ ) -> None:
+ """Gateway enabled + offline + no primed cache → ``RemoteConfigUnavailableError``.
+
+ Phase 2 introduced a cache fallback so the fetcher only raises when both the network
+ and the local cache are unusable. We redirect ``~/.pipelex`` to a tmp path so any
+ cache the developer has from prior online work doesn't satisfy the fallback.
+ """
+ Pipelex.teardown_if_needed()
+
+ # Isolate the cache so a real ``~/.pipelex/cache/`` from prior online runs can't
+ # accidentally make this test "succeed via cache" and miss the regression.
+ mocker.patch.object(
+ ConfigLoader,
+ "global_config_dir",
+ new_callable=mocker.PropertyMock,
+ return_value=tmp_path / ".pipelex",
+ )
+
+ service_config = PipelexServiceConfig(
+ agreement=PipelexServiceAgreement(terms_accepted=True),
+ onboarding=PipelexServiceOnboarding(inference_setup_completed=True),
+ )
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=service_config,
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+
+ # Bypass the session-level cache patch from tests/conftest.py so we exercise the real
+ # fetch path against a controlled httpx mock.
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+
+ try:
+ with pytest.raises(RemoteConfigUnavailableError):
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=True,
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+ # ``Pipelex.make`` clears the singleton on setup failure but does NOT reset the
+ # global log state, so subsequent ``Pipelex.make`` calls would crash with
+ # "LogConfig is already set". Reset it manually here so other test modules can
+ # still spin up a fresh Pipelex instance.
+ log.reset()
diff --git a/tests/integration/pipelex/system/pipelex_service/test_remote_config_fetcher.py b/tests/integration/pipelex/system/pipelex_service/test_remote_config_fetcher.py
new file mode 100644
index 000000000..fda256312
--- /dev/null
+++ b/tests/integration/pipelex/system/pipelex_service/test_remote_config_fetcher.py
@@ -0,0 +1,350 @@
+"""Phase 2 contract for ``RemoteConfigFetcher.fetch_remote_config``.
+
+These tests pin the post-refactor semantics:
+
+- success → ``RemoteConfigResult(source=FRESH)`` AND the cache is written
+- network/HTTP failure with a usable cache → ``RemoteConfigResult(source=CACHED)`` with a
+ ``cached_at`` timestamp. Warning emission lives at the orchestration layer
+ (``Pipelex.setup``), not here — the fetcher is a pure data-returning function.
+- failure with no cache → ``RemoteConfigUnavailableError`` (cache path in the message)
+- malformed JSON → ``RemoteConfigValidationError`` (no silent cache fallback — server-side bug)
+- ``require_fresh=True`` (used by the doc generators) refuses cached fallback
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path # noqa: TC003 — referenced by pytest fixture type hints at runtime
+from typing import TYPE_CHECKING, Any
+
+import httpx
+import pytest
+from pydantic import ValidationError
+
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.pipelex_service.exceptions import (
+ RemoteConfigFetchError,
+ RemoteConfigUnavailableError,
+ RemoteConfigValidationError,
+)
+from pipelex.system.pipelex_service.remote_config import RemoteConfig
+from pipelex.system.pipelex_service.remote_config_cache import RemoteConfigCache
+from pipelex.system.pipelex_service.remote_config_fetcher import (
+ RemoteConfigFetcher,
+ RemoteConfigResult,
+)
+from pipelex.system.pipelex_service.types import RemoteConfigSource
+
+if TYPE_CHECKING:
+ from pytest_mock import MockerFixture
+
+# Bypass the session-scoped cache patch in tests/conftest.py.
+_ORIGINAL_FETCH_REMOTE_CONFIG = RemoteConfigFetcher.fetch_remote_config
+
+
+def _valid_remote_config_payload() -> dict[str, Any]:
+ return {
+ "posthog": {
+ "project_api_key": "test-key",
+ "endpoint": "https://posthog.example.com",
+ "is_geoip_enabled": False,
+ "is_debug_enabled": False,
+ },
+ "backend_model_specs": {"defaults": {"sdk": "gateway_completions"}},
+ "aws_region": "eu-west-3",
+ }
+
+
+def _make_httpx_response(payload: dict[str, Any]) -> httpx.Response:
+ return httpx.Response(
+ status_code=200,
+ request=httpx.Request("GET", "https://example.com/remote_config.json"),
+ content=json.dumps(payload).encode("utf-8"),
+ )
+
+
+def isolated_cache_dir(tmp_path: Path, mocker: MockerFixture) -> Path:
+ """Redirect ``RemoteConfigCache`` at a tmp ``~/.pipelex`` so tests stay hermetic."""
+ fake_global_dir = tmp_path / ".pipelex"
+ mocker.patch.object(
+ ConfigLoader,
+ "global_config_dir",
+ new_callable=mocker.PropertyMock,
+ return_value=fake_global_dir,
+ )
+ return fake_global_dir
+
+
[email protected](autouse=True)
+def restore_original_fetcher(mocker: MockerFixture) -> None:
+ """Undo the session-level cached fetch shim from tests/conftest.py."""
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+
+
[email protected](autouse=True)
+def disable_codex_cloud(mocker: MockerFixture) -> None:
+ """Keep the Codex Cloud short-circuit out of the way for all tests in this module."""
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+
+
+def fast_retry(mocker: MockerFixture) -> None:
+ """Collapse tenacity's wait/stop so transient-failure tests don't sleep."""
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_WAIT_MIN", 0)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_WAIT_MAX", 0)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_WAIT_MULTIPLIER", 0)
+
+
+class TestRemoteConfigFetcher:
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_success_returns_fresh_and_writes_cache(self, mocker: MockerFixture) -> None:
+ payload = _valid_remote_config_payload()
+ mocker.patch("httpx.get", return_value=_make_httpx_response(payload))
+
+ result = RemoteConfigFetcher.fetch_remote_config()
+
+ assert isinstance(result, RemoteConfigResult)
+ assert result.source == RemoteConfigSource.FRESH
+ assert result.cached_at is None
+ assert result.config.aws_region == "eu-west-3"
+
+ cached = RemoteConfigCache.load()
+ assert cached is not None, "successful fetch must persist the raw payload to the cache"
+ assert cached.raw_config == payload
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_network_failure_with_cache_returns_cached(self, mocker: MockerFixture) -> None:
+ """Cache fallback returns the cached payload tagged ``source=CACHED`` with a
+ ``cached_at`` timestamp forwarded verbatim from the on-disk snapshot — not a
+ fabricated "now" — so callers can reason about staleness accurately. Warning
+ emission lives at the orchestration layer (``Pipelex.setup``); the fetcher itself
+ stays a pure data-returning function.
+ """
+ payload = _valid_remote_config_payload()
+ RemoteConfigCache.store(payload)
+ stored_snapshot = RemoteConfigCache.load()
+ assert stored_snapshot is not None
+
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+
+ result = RemoteConfigFetcher.fetch_remote_config()
+
+ assert result.source == RemoteConfigSource.CACHED
+ assert result.cached_at == stored_snapshot.cached_at, "result must forward the cache's snapshot timestamp, not a freshly-computed one"
+ assert result.config.aws_region == "eu-west-3"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_network_failure_without_cache_raises_unavailable(self, mocker: MockerFixture) -> None:
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+
+ expected_cache_path = str(RemoteConfigCache.cache_path())
+
+ with pytest.raises(RemoteConfigUnavailableError) as exc_info:
+ RemoteConfigFetcher.fetch_remote_config()
+
+ assert expected_cache_path in str(exc_info.value), "error message must name the cache path so users can prime it"
+ assert "pipelex init" in str(exc_info.value), "error message must point at the remediation command"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_network_failure_with_invalid_cache_raises_unavailable(self, mocker: MockerFixture) -> None:
+ """A cache with a valid wrapper but a stale/malformed ``raw_config`` must be treated
+ as unusable. ``RemoteConfigCache.load()`` only validates the wrapper, so the offline
+ path must catch the inner ``RemoteConfig`` validation failure and raise
+ ``RemoteConfigUnavailableError`` with the normal remediation message — never let a
+ raw Pydantic ``ValidationError`` escape.
+ """
+ # store() does not validate raw_config against RemoteConfig, so this writes a
+ # cache whose wrapper is valid but whose payload cannot parse as RemoteConfig.
+ RemoteConfigCache.store({"bogus": "payload"})
+
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+
+ with pytest.raises(RemoteConfigUnavailableError) as exc_info:
+ RemoteConfigFetcher.fetch_remote_config()
+
+ assert "pipelex init" in str(exc_info.value), "invalid cache must surface the normal offline remediation, not a raw ValidationError"
+ assert isinstance(exc_info.value.__cause__, ValidationError), (
+ "the unavailable error must chain from the inner RemoteConfig ValidationError, "
+ "proving the malformed-cache branch (not the missing-cache branch) was exercised"
+ )
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_http_error_with_cache_returns_cached(self, mocker: MockerFixture) -> None:
+ """5xx response with a primed cache falls back to cache, same as a connect error.
+ Asserts the cached payload (not a fabricated empty one) flows back, with its
+ snapshot timestamp preserved.
+ """
+ payload = _valid_remote_config_payload()
+ RemoteConfigCache.store(payload)
+ stored_snapshot = RemoteConfigCache.load()
+ assert stored_snapshot is not None
+
+ failing_response = httpx.Response(
+ status_code=503,
+ request=httpx.Request("GET", "https://example.com/remote_config.json"),
+ content=b"Service Unavailable",
+ )
+ mocker.patch("httpx.get", return_value=failing_response)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+
+ result = RemoteConfigFetcher.fetch_remote_config()
+
+ assert result.source == RemoteConfigSource.CACHED
+ assert result.cached_at == stored_snapshot.cached_at
+ assert result.config.aws_region == "eu-west-3"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_http_error_without_cache_raises_unavailable(self, mocker: MockerFixture) -> None:
+ failing_response = httpx.Response(
+ status_code=503,
+ request=httpx.Request("GET", "https://example.com/remote_config.json"),
+ content=b"Service Unavailable",
+ )
+ mocker.patch("httpx.get", return_value=failing_response)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+
+ with pytest.raises(RemoteConfigUnavailableError):
+ RemoteConfigFetcher.fetch_remote_config()
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_validation_error_does_not_fall_back(self, mocker: MockerFixture) -> None:
+ """Server-side schema break must surface — we control the server, so a stale cache
+ would be the wrong answer here.
+ """
+ # Cache is populated and otherwise usable; we want to assert it's NOT consulted.
+ RemoteConfigCache.store(_valid_remote_config_payload())
+
+ bad_response = httpx.Response(
+ status_code=200,
+ request=httpx.Request("GET", "https://example.com/remote_config.json"),
+ content=b'{"posthog": "not-an-object"}',
+ )
+ mocker.patch("httpx.get", return_value=bad_response)
+
+ with pytest.raises(RemoteConfigValidationError):
+ RemoteConfigFetcher.fetch_remote_config()
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_codex_cloud_short_circuit_still_works(self, mocker: MockerFixture) -> None:
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=True,
+ )
+ httpx_get = mocker.patch("httpx.get")
+
+ result = RemoteConfigFetcher.fetch_remote_config()
+
+ assert result.source == RemoteConfigSource.FRESH
+ assert result.cached_at is None
+ assert httpx_get.call_count == 0, "Codex Cloud short-circuit must skip the network call"
+ assert RemoteConfigCache.load() is None, "Codex Cloud short-circuit must not pollute the cache"
+
+ @pytest.mark.usefixtures("isolated_cache_dir", "fast_retry")
+ def test_succeeds_after_4_transient_failures_no_cache_fallback(self, mocker: MockerFixture) -> None:
+ payload = _valid_remote_config_payload()
+ side_effects: list[Any] = [
+ httpx.ConnectError("transient 1"),
+ httpx.ConnectError("transient 2"),
+ httpx.ConnectError("transient 3"),
+ httpx.ConnectError("transient 4"),
+ _make_httpx_response(payload),
+ ]
+ mocker.patch("httpx.get", side_effect=side_effects)
+
+ result = RemoteConfigFetcher.fetch_remote_config()
+
+ assert result.source == RemoteConfigSource.FRESH, "tenacity recovered before exhausting attempts; no cache fallback should trigger"
+ cached = RemoteConfigCache.load()
+ assert cached is not None
+ assert cached.raw_config == payload
+
+ @pytest.mark.usefixtures("isolated_cache_dir", "fast_retry")
+ def test_falls_back_to_cache_after_5_transient_failures(self, mocker: MockerFixture) -> None:
+ """All retry attempts exhausted, cache primed → cache fallback. Stale-cache warning
+ emission is the orchestration layer's concern, so it's not asserted here.
+ """
+ payload = _valid_remote_config_payload()
+ RemoteConfigCache.store(payload)
+
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("always down"))
+
+ result = RemoteConfigFetcher.fetch_remote_config()
+
+ assert result.source == RemoteConfigSource.CACHED
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_require_fresh_refuses_cache(self, mocker: MockerFixture) -> None:
+ """Doc generators set ``require_fresh=True`` so they never bake stale data into committed files.
+
+ When cache is present, the error message must call out that the cache was refused
+ (not that it is missing) so the user gets accurate diagnostics.
+ """
+ RemoteConfigCache.store(_valid_remote_config_payload())
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+
+ with pytest.raises(RemoteConfigUnavailableError) as exc_info:
+ RemoteConfigFetcher.fetch_remote_config(require_fresh=True)
+
+ assert "was refused because a fresh fetch is required" in str(exc_info.value)
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_inner_retry_layer_still_raises_remote_config_fetch_error(self, mocker: MockerFixture) -> None:
+ """Internal retry layer keeps raising ``RemoteConfigFetchError``; the outer flow turns it
+ into a cache hit or an unavailable error. Callers that introspect the inner exception
+ (doctor, error handlers, agent hints) still see the original class.
+ """
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+
+ with pytest.raises(RemoteConfigUnavailableError) as exc_info:
+ RemoteConfigFetcher.fetch_remote_config()
+
+ cause = exc_info.value.__cause__
+ assert isinstance(cause, RemoteConfigFetchError), (
+ "RemoteConfigUnavailableError must chain from the original RemoteConfigFetchError "
+ "so existing surfaces (doctor, agent hints) can keep reading the inner exception"
+ )
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_cache_store_oserror_does_not_abort_fresh_fetch(self, mocker: MockerFixture, capsys: pytest.CaptureFixture[str]) -> None:
+ """Cache write is opportunistic — a read-only ``$HOME``, full disk, or permission
+ error on the cache directory must not kill an otherwise-successful online fetch.
+ ``fetch_remote_config`` should warn to stderr and still return ``source=FRESH``.
+ """
+ payload = _valid_remote_config_payload()
+ mocker.patch("httpx.get", return_value=_make_httpx_response(payload))
+ mocker.patch.object(RemoteConfigCache, "store", side_effect=OSError("read-only filesystem"))
+
+ result = RemoteConfigFetcher.fetch_remote_config()
+
+ assert result.source == RemoteConfigSource.FRESH
+ assert result.config.aws_region == "eu-west-3"
+ captured = capsys.readouterr()
+ assert "failed to persist remote config cache" in captured.err, (
+ f"cache-write failure must surface as a warning on stderr; got stderr={captured.err!r}"
+ )
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_fresh_path_validates_payload_only_once(self, mocker: MockerFixture) -> None:
+ """``_fetch_fresh`` already validates the payload; the outer flow must reuse the
+ parsed config rather than re-validating it. We spy on ``RemoteConfig.model_validate``
+ and assert it was called exactly once for a fresh fetch.
+ """
+ payload = _valid_remote_config_payload()
+ mocker.patch("httpx.get", return_value=_make_httpx_response(payload))
+ validate_spy = mocker.spy(RemoteConfig, "model_validate")
+
+ result = RemoteConfigFetcher.fetch_remote_config()
+
+ assert result.source == RemoteConfigSource.FRESH
+ assert validate_spy.call_count == 1, f"fresh path must validate exactly once; got {validate_spy.call_count} calls"
diff --git a/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py b/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py
new file mode 100644
index 000000000..ee4c6e206
--- /dev/null
+++ b/tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py
@@ -0,0 +1,241 @@
+"""Phase 3 contract: gateway-cache fallback flows through ``Pipelex.make``.
+
+These tests pin the post-refactor behaviour at the Pipelex-setup boundary (the fetcher-level
+contract is locked in by ``test_remote_config_fetcher.py``):
+
+- Dry-run setup ``(needs_inference=False, needs_model_specs=True)`` succeeds with a primed
+ cache and emits ``RemoteConfigStaleWarning``; cold cache raises ``RemoteConfigUnavailableError``.
+- Pipelex telemetry must be disabled whenever the gateway config is cached, even when
+ ``needs_inference=True``. Stale config implies stale model identities, so phoning home about
+ pipe runs against possibly-stale specs would pollute metrics.
+- BYOK with the gateway disabled never reaches ``fetch_remote_config``, regardless of
+ ``needs_model_specs``. Explicit regression guard so future refactors can't reintroduce the
+ phantom network call described in the original offline-mode bug.
+"""
+
+from __future__ import annotations
+
+import warnings
+from datetime import datetime, timezone
+from pathlib import Path # noqa: TC003 — referenced by pytest fixture type hints at runtime
+from typing import TYPE_CHECKING
+
+import httpx
+import pytest
+
+from pipelex import log
+from pipelex.pipelex import Pipelex
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.pipelex_service.exceptions import (
+ RemoteConfigStaleWarning,
+ RemoteConfigUnavailableError,
+)
+from pipelex.system.pipelex_service.pipelex_service_agreement import (
+ PipelexServiceAgreement,
+ PipelexServiceOnboarding,
+)
+from pipelex.system.pipelex_service.pipelex_service_config import PipelexServiceConfig
+from pipelex.system.pipelex_service.remote_config_cache import RemoteConfigCache
+from pipelex.system.pipelex_service.remote_config_fetcher import (
+ RemoteConfigFetcher,
+ RemoteConfigResult,
+)
+from pipelex.system.pipelex_service.types import RemoteConfigSource
+from pipelex.system.runtime import IntegrationMode
+from pipelex.system.telemetry.telemetry_manager_abstract import TelemetryManagerNoOp
+
+if TYPE_CHECKING:
+ from collections.abc import Generator
+
+ from pytest_mock import MockerFixture
+
+PIPELEX_MODULE = "pipelex.pipelex"
+
+# Capture the unpatched classmethod at module import — the session conftest replaces
+# ``fetch_remote_config`` with a cached shim. We need the real fetch path to exercise the
+# httpx mock for the cache-fallback flow.
+_ORIGINAL_FETCH_REMOTE_CONFIG = RemoteConfigFetcher.fetch_remote_config
+
+
+def _accepted_service_config() -> PipelexServiceConfig:
+ return PipelexServiceConfig(
+ agreement=PipelexServiceAgreement(terms_accepted=True),
+ onboarding=PipelexServiceOnboarding(inference_setup_completed=True),
+ )
+
+
[email protected](scope="module", autouse=True)
+def reset_pipelex_config_fixture() -> Generator[None, None, None]:
+ """Override the global module fixture: each test handles its own ``Pipelex.make``."""
+ yield
+ Pipelex.teardown_if_needed()
+
+
+def isolated_cache_dir(tmp_path: Path, mocker: MockerFixture) -> Path:
+ """Redirect ``~/.pipelex`` (and therefore the cache) at a tmp path so a developer's real
+ cache from prior online runs can't accidentally satisfy the fallback.
+ """
+ fake_global_dir = tmp_path / ".pipelex"
+ mocker.patch.object(
+ ConfigLoader,
+ "global_config_dir",
+ new_callable=mocker.PropertyMock,
+ return_value=fake_global_dir,
+ )
+ return fake_global_dir
+
+
+class TestSetupWithCache:
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_setup_succeeds_with_stale_cache_dry_run(
+ self,
+ mocker: MockerFixture,
+ ) -> None:
+ """Gateway enabled, network down, cache primed → setup completes and emits
+ ``RemoteConfigStaleWarning``. Validates the end-to-end stale-cache UX (dry-run).
+ """
+ Pipelex.teardown_if_needed()
+
+ # Prime the cache by calling the session-cached fetcher (returns a real config),
+ # then store its raw payload via the cache helper. We do this before re-pointing
+ # the cache dir... actually the fixture already pointed us to a tmp dir, so
+ # ``RemoteConfigCache.store`` will write there.
+ session_result = RemoteConfigFetcher.fetch_remote_config() # patched to session cache
+ RemoteConfigCache.store(session_result.config.model_dump(mode="json"))
+
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+
+ # Restore the real fetcher so the cache fallback path runs.
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+
+ try:
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=True,
+ )
+ assert any(issubclass(item.category, RemoteConfigStaleWarning) for item in caught), (
+ "stale-cache setup must emit RemoteConfigStaleWarning so machine consumers can surface it"
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_setup_fails_without_cache_dry_run(
+ self,
+ mocker: MockerFixture,
+ ) -> None:
+ """Gateway enabled, network down, no cache → ``RemoteConfigUnavailableError`` surfaces
+ from ``Pipelex.make`` (not silently swallowed).
+ """
+ Pipelex.teardown_if_needed()
+
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", _ORIGINAL_FETCH_REMOTE_CONFIG)
+ mocker.patch.object(RemoteConfigFetcher, "FETCH_MAX_RETRIES", 1)
+ mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+
+ try:
+ with pytest.raises(RemoteConfigUnavailableError):
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=True,
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
+
+ def test_telemetry_disabled_when_source_cached(self, mocker: MockerFixture) -> None:
+ """``Pipelex.make(needs_inference=True)`` with a cached gateway config → telemetry
+ manager is the no-op variant. Stale specs imply stale model identities; phoning home
+ about pipe runs in that state would pollute metrics, so the guard is stricter than
+ the plain ``needs_inference and gateway_enabled`` check.
+ """
+ Pipelex.teardown_if_needed()
+
+ # Reuse the session-cached gateway config so deck validation passes, but re-wrap with
+ # ``source=CACHED`` to exercise the stricter telemetry guard.
+ session_result = RemoteConfigFetcher.fetch_remote_config()
+ cached_result = RemoteConfigResult(
+ config=session_result.config,
+ source=RemoteConfigSource.CACHED,
+ cached_at=datetime.now(tz=timezone.utc),
+ )
+
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=True)
+ mocker.patch(
+ f"{PIPELEX_MODULE}.load_pipelex_service_config_if_exists",
+ return_value=_accepted_service_config(),
+ )
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", return_value=cached_result)
+
+ try:
+ pipelex_instance = Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=True,
+ )
+ assert isinstance(pipelex_instance.telemetry_manager, TelemetryManagerNoOp), (
+ "cached gateway config must downgrade telemetry to no-op, even when needs_inference=True"
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
+
+ def test_byok_offline_regression_guard(self, mocker: MockerFixture) -> None:
+ """Gateway disabled in ``backends.toml`` → ``fetch_remote_config`` is never invoked,
+ even with ``needs_model_specs=True``. Without this guard a future refactor could
+ reintroduce the phantom network call that originally motivated the offline-mode work.
+ """
+ Pipelex.teardown_if_needed()
+
+ mocker.patch(f"{PIPELEX_MODULE}.is_pipelex_gateway_enabled", return_value=False)
+ mocker.patch(
+ "pipelex.system.runtime.RuntimeManager.is_in_codex_cloud",
+ new_callable=mocker.PropertyMock,
+ return_value=False,
+ )
+ fetch_spy = mocker.spy(RemoteConfigFetcher, "fetch_remote_config")
+ httpx_get_mock = mocker.patch("httpx.get", side_effect=httpx.ConnectError("no network"))
+
+ try:
+ Pipelex.make(
+ integration_mode=IntegrationMode.PYTEST,
+ needs_inference=False,
+ needs_model_specs=True,
+ )
+ finally:
+ Pipelex.teardown_if_needed()
+ log.reset()
+
+ assert fetch_spy.call_count == 0, "fetch_remote_config must never be invoked when the gateway is disabled"
+ assert httpx_get_mock.call_count == 0, "httpx.get must never be invoked when the gateway is disabled"
diff --git a/tests/unit/pipelex/cli/dev/test_preprocess_test_models_cmd.py b/tests/unit/pipelex/cli/dev/test_preprocess_test_models_cmd.py
new file mode 100644
index 000000000..e73f30afe
--- /dev/null
+++ b/tests/unit/pipelex/cli/dev/test_preprocess_test_models_cmd.py
@@ -0,0 +1,31 @@
+"""Unit tests for the `pipelex-dev preprocess-test-models` command."""
+
+import pytest
+from pytest_mock import MockerFixture
+
+from pipelex.cli.dev_cli.commands.preprocess_test_models_cmd import (
+ _fetch_gateway_models, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
+)
+from pipelex.system.pipelex_service.exceptions import RemoteConfigUnavailableError, RemoteConfigValidationError
+from pipelex.system.pipelex_service.remote_config_fetcher import RemoteConfigFetcher
+
+
+class TestPreprocessTestModelsCmd:
+ @pytest.mark.parametrize(
+ "gateway_error",
+ [
+ RemoteConfigUnavailableError("gateway offline"),
+ RemoteConfigValidationError("gateway schema break"),
+ ],
+ )
+ def test_fetch_gateway_models_propagates_fresh_refusal(self, mocker: MockerFixture, gateway_error: Exception) -> None:
+ """``require_fresh=True`` must refuse to proceed when the Gateway config is unavailable
+ or stale. The error propagates instead of being swallowed into empty model lists, so
+ offline fixture generation cannot silently drop every ``pipelex_gateway`` model.
+ """
+ mocker.patch.object(RemoteConfigFetcher, "fetch_remote_config", side_effect=gateway_error)
+
+ with pytest.raises((RemoteConfigUnavailableError, RemoteConfigValidationError)) as exc_info:
+ _fetch_gateway_models()
+
+ assert exc_info.value is gateway_error, "the fresh-config refusal must propagate verbatim, not be converted into empty model lists"
diff --git a/tests/unit/pipelex/cli/test_agent_cli_factory.py b/tests/unit/pipelex/cli/test_agent_cli_factory.py
index 6fb46606c..9075f286c 100644
--- a/tests/unit/pipelex/cli/test_agent_cli_factory.py
+++ b/tests/unit/pipelex/cli/test_agent_cli_factory.py
@@ -19,7 +19,6 @@ from pipelex.system.pipelex_service.exceptions import (
GatewayApiKeyMissingError,
GatewayDoNotTrackConflictError,
GatewayTermsNotAcceptedError,
- RemoteConfigFetchError,
RemoteConfigValidationError,
)
from pipelex.system.telemetry.exceptions import TelemetryConfigValidationError
@@ -53,7 +52,6 @@ class TestMakePipelexForAgentCli:
(GatewayTermsNotAcceptedError, (), "GatewayTermsNotAcceptedError"),
(GatewayApiKeyMissingError, (), "GatewayApiKeyMissingError"),
(GatewayDoNotTrackConflictError, ("DO_NOT_TRACK",), "GatewayDoNotTrackConflictError"),
- (RemoteConfigFetchError, ("cannot reach server",), "RemoteConfigFetchError"),
(RemoteConfigValidationError, ("bad remote config",), "RemoteConfigValidationError"),
],
)
diff --git a/tests/unit/pipelex/cli/test_agent_output.py b/tests/unit/pipelex/cli/test_agent_output.py
index 1923daf95..cbc964920 100644
--- a/tests/unit/pipelex/cli/test_agent_output.py
+++ b/tests/unit/pipelex/cli/test_agent_output.py
@@ -4,6 +4,7 @@ from __future__ import annotations
import datetime
import json
+from typing import TYPE_CHECKING, Any
import pytest
import typer
@@ -15,7 +16,9 @@ from pipelex.cli.agent_cli.commands.agent_output import (
_build_error_source, # noqa: PLC2701 # pyright: ignore[reportPrivateUsage]
agent_error,
agent_success,
+ consume_setup_warnings,
extract_validation_errors,
+ record_setup_warning,
)
from pipelex.cogt.exceptions import CogtError, InferenceBackendCredentialsError, InferenceBackendCredentialsErrorType, InferenceErrorCategory
from pipelex.core.bundles.exceptions import PipelexBundleBlueprintValidationErrorData
@@ -23,10 +26,24 @@ from pipelex.core.exceptions import PipeFactoryErrorData, PipesAndConceptValidat
from pipelex.core.pipes.exceptions import PipeFactoryErrorType, PipeValidationErrorType
from pipelex.pipeline.validate_bundle import ValidateBundleError
+if TYPE_CHECKING:
+ from collections.abc import Iterator
+
class TestAgentOutput:
"""Tests for agent_error, agent_success, and extract_validation_errors."""
+ @pytest.fixture(autouse=True)
+ def _drain_warnings_buffer(self) -> Iterator[None]:
+ """Isolate the module-level ``_CAPTURED_WARNINGS`` global across tests.
+
+ The buffer is a process-wide global; without this drain a recorded warning
+ could leak into an unrelated test's envelope.
+ """
+ consume_setup_warnings()
+ yield
+ consume_setup_warnings()
+
def test_agent_error_outputs_json_to_stderr(self, capsys: pytest.CaptureFixture[str]) -> None:
"""agent_error should print valid JSON to stderr and exit with code 1."""
with pytest.raises(typer.Exit) as exc_info:
@@ -371,3 +388,65 @@ class TestAgentOutput:
parsed = json.loads(capsys.readouterr().err)
assert parsed["error_domain"] == AGENT_ERROR_DOMAINS[error_type]
assert parsed["error_category"] == "configuration"
+
+ # -------------------------------------------------------------------------
+ # Setup-warnings buffer tests (record_setup_warning / consume_setup_warnings
+ # / the warnings-merge branch of agent_success)
+ # -------------------------------------------------------------------------
+
+ def test_agent_success_attaches_recorded_warning(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """A warning recorded via record_setup_warning surfaces in the envelope's warnings array."""
+ record_setup_warning({"type": "RemoteConfigStale", "message": "cache is stale"})
+ agent_success({"success": True})
+
+ parsed = json.loads(capsys.readouterr().out)
+ assert parsed["warnings"] == [{"type": "RemoteConfigStale", "message": "cache is stale"}]
+
+ def test_agent_success_does_not_re_emit_drained_warnings(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """A second agent_success in the same process emits no stale warnings (drain+clear)."""
+ record_setup_warning({"type": "RemoteConfigStale", "message": "cache is stale"})
+ agent_success({"success": True})
+ capsys.readouterr() # discard the first envelope
+
+ agent_success({"success": True})
+ parsed = json.loads(capsys.readouterr().out)
+ assert "warnings" not in parsed
+
+ def test_consume_setup_warnings_drains_and_clears(self) -> None:
+ """consume_setup_warnings returns the recorded list and empties the buffer."""
+ record_setup_warning({"type": "RemoteConfigStale", "message": "first"})
+ record_setup_warning({"type": "RemoteConfigStale", "message": "second"})
+
+ drained = consume_setup_warnings()
+ assert drained == [
+ {"type": "RemoteConfigStale", "message": "first"},
+ {"type": "RemoteConfigStale", "message": "second"},
+ ]
+ assert consume_setup_warnings() == []
+
+ def test_agent_success_appends_captured_after_caller_warnings(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """Caller-supplied result['warnings'] are kept; captured ones are appended after them."""
+ record_setup_warning({"type": "RemoteConfigStale", "message": "captured"})
+ agent_success({"success": True, "warnings": [{"type": "CallerWarning", "message": "caller"}]})
+
+ parsed = json.loads(capsys.readouterr().out)
+ assert parsed["warnings"] == [
+ {"type": "CallerWarning", "message": "caller"},
+ {"type": "RemoteConfigStale", "message": "captured"},
+ ]
+
+ def test_agent_success_tolerates_non_list_caller_warnings(self, capsys: pytest.CaptureFixture[str]) -> None:
+ """A non-list result['warnings'] is treated as empty rather than crashing (isinstance guard)."""
+ record_setup_warning({"type": "RemoteConfigStale", "message": "captured"})
+ agent_success({"success": True, "warnings": "not-a-list"})
+
+ parsed = json.loads(capsys.readouterr().out)
+ assert parsed["warnings"] == [{"type": "RemoteConfigStale", "message": "captured"}]
+
+ def test_agent_success_does_not_mutate_caller_result(self) -> None:
+ """agent_success copies result before merging captured warnings; the caller dict is untouched."""
+ record_setup_warning({"type": "RemoteConfigStale", "message": "captured"})
+ result: dict[str, Any] = {"success": True}
+ agent_success(result)
+
+ assert "warnings" not in result
diff --git a/tests/unit/pipelex/cogt/models/test_collect_deck_referenced_handles.py b/tests/unit/pipelex/cogt/models/test_collect_deck_referenced_handles.py
new file mode 100644
index 000000000..8db5eb19b
--- /dev/null
+++ b/tests/unit/pipelex/cogt/models/test_collect_deck_referenced_handles.py
@@ -0,0 +1,144 @@
+"""Unit coverage for ``ModelManager._collect_deck_referenced_handles`` and ``_extract_choice_handle``.
+
+These two classmethods feed the gateway-membership check in ``ModelManager.setup``:
+``_collect_deck_referenced_handles`` gathers every ``(handle, model_type)`` the deck advertises
+as usable (presets + choice defaults), and ``_extract_choice_handle`` normalises a ``*ModelChoice``
+union to a raw handle string. Aliases and waterfalls are intentionally NOT enumerated directly —
+they are reachable through preset/choice references and the resolver walks them.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from pipelex.cogt.config_cogt import ModelDeckConfig
+from pipelex.cogt.extract.extract_setting import ExtractModelChoice, ExtractSetting
+from pipelex.cogt.img_gen.img_gen_job_components import Quality
+from pipelex.cogt.img_gen.img_gen_setting import ImgGenModelChoice, ImgGenSetting
+from pipelex.cogt.llm.llm_setting import LLMModelChoice, LLMSetting, LLMSettingChoicesDefaults
+from pipelex.cogt.model_backends.model_type import ModelType
+from pipelex.cogt.models.model_deck import ModelDeck
+from pipelex.cogt.models.model_manager import ModelManager
+from pipelex.cogt.models.model_reference import ModelReference
+from pipelex.cogt.search.search_setting import SearchModelChoice, SearchSetting
+from pipelex.system.runtime import ProblemReaction
+
+# Both methods are private leaves of the membership resolver; aliasing them keeps the call
+# sites in the tests free of repeated noqa/ignore pragmas.
+_collect_deck_referenced_handles = ModelManager._collect_deck_referenced_handles # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+_extract_choice_handle = ModelManager._extract_choice_handle # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+
+def _make_deck(
+ *,
+ llm_presets: dict[str, LLMSetting] | None = None,
+ llm_for_text: LLMModelChoice = "llm-text-handle",
+ llm_for_object: LLMModelChoice = "llm-object-handle",
+ extract_presets: dict[str, ExtractSetting] | None = None,
+ extract_choice_default: ExtractModelChoice = "extract-default-handle",
+ img_gen_presets: dict[str, ImgGenSetting] | None = None,
+ img_gen_choice_default: ImgGenModelChoice = "img-gen-default-handle",
+ search_presets: dict[str, SearchSetting] | None = None,
+ search_choice_default: SearchModelChoice = "search-default-handle",
+ llm_aliases: dict[str, str] | None = None,
+ llm_waterfalls: dict[str, list[str]] | None = None,
+) -> ModelDeck:
+ """Build a minimal real ModelDeck with all required fields, mirroring test_model_deck.py."""
+ return ModelDeck(
+ inference_models={},
+ # LLM
+ llm_default_temperature=0.7,
+ llm_aliases=llm_aliases or {},
+ llm_waterfalls=llm_waterfalls or {},
+ llm_presets=llm_presets or {},
+ llm_choice_defaults=LLMSettingChoicesDefaults(
+ default_temperature=0.7,
+ for_text=llm_for_text,
+ for_object=llm_for_object,
+ ),
+ # Extract
+ extract_aliases={},
+ extract_waterfalls={},
+ extract_presets=extract_presets or {},
+ extract_choice_default=extract_choice_default,
+ # ImgGen
+ img_gen_default_quality=Quality.MEDIUM,
+ img_gen_aliases={},
+ img_gen_waterfalls={},
+ img_gen_presets=img_gen_presets or {},
+ img_gen_choice_default=img_gen_choice_default,
+ # Search
+ search_aliases={},
+ search_waterfalls={},
+ search_presets=search_presets or {},
+ search_choice_default=search_choice_default,
+ model_deck_config=ModelDeckConfig(is_model_fallback_enabled=True, missing_presets_reaction=ProblemReaction.NONE),
+ )
+
+
+class TestCollectDeckReferencedHandles:
+ @pytest.mark.parametrize(
+ ("choice", "expected"),
+ [
+ (None, None),
+ ("raw-handle", "raw-handle"),
+ (ModelReference.parse("ref-handle"), "ref-handle"),
+ (LLMSetting(model="llm-model", temperature=0.5), "llm-model"),
+ (ExtractSetting(model="extract-model"), "extract-model"),
+ (ImgGenSetting(model="img-gen-model"), "img-gen-model"),
+ (SearchSetting(model="search-model"), "search-model"),
+ ],
+ )
+ def test_extract_choice_handle(
+ self,
+ choice: LLMSetting | ExtractSetting | ImgGenSetting | SearchSetting | ModelReference | str | None,
+ expected: str | None,
+ ) -> None:
+ """_extract_choice_handle normalises every choice shape: None, raw str, ModelReference, typed setting."""
+ assert _extract_choice_handle(choice) == expected
+
+ def test_collects_presets_and_choice_defaults_for_all_types(self) -> None:
+ """_collect_deck_referenced_handles yields a (handle, ModelType) pair for every preset and choice default."""
+ deck = _make_deck(
+ llm_presets={
+ "fast": LLMSetting(model="llm-preset-fast", temperature=0.3),
+ "smart": LLMSetting(model="llm-preset-smart", temperature=0.7),
+ },
+ llm_for_text="llm-text-handle",
+ llm_for_object="llm-object-handle",
+ extract_presets={"doc": ExtractSetting(model="extract-preset-doc")},
+ extract_choice_default="extract-default-handle",
+ img_gen_presets={"basic": ImgGenSetting(model="img-gen-preset-basic")},
+ img_gen_choice_default="img-gen-default-handle",
+ search_presets={"web": SearchSetting(model="search-preset-web")},
+ search_choice_default="search-default-handle",
+ )
+
+ collected = _collect_deck_referenced_handles(deck)
+
+ assert set(collected) == {
+ ("llm-preset-fast", ModelType.LLM),
+ ("llm-preset-smart", ModelType.LLM),
+ ("llm-text-handle", ModelType.LLM),
+ ("llm-object-handle", ModelType.LLM),
+ ("extract-preset-doc", ModelType.TEXT_EXTRACTOR),
+ ("extract-default-handle", ModelType.TEXT_EXTRACTOR),
+ ("img-gen-preset-basic", ModelType.IMG_GEN),
+ ("img-gen-default-handle", ModelType.IMG_GEN),
+ ("search-preset-web", ModelType.SEARCH),
+ ("search-default-handle", ModelType.SEARCH),
+ }
+
+ def test_aliases_and_waterfalls_are_not_enumerated_directly(self) -> None:
+ """Alias/waterfall keys and their targets are not surfaced unless referenced by a preset/choice."""
+ deck = _make_deck(
+ llm_aliases={"my-alias": "alias-target-handle"},
+ llm_waterfalls={"my-waterfall": ["waterfall-target-handle"]},
+ )
+
+ handles = {handle for handle, _ in _collect_deck_referenced_handles(deck)}
+
+ assert "my-alias" not in handles
+ assert "alias-target-handle" not in handles
+ assert "my-waterfall" not in handles
+ assert "waterfall-target-handle" not in handles
diff --git a/tests/unit/pipelex/cogt/models/test_model_deck_references.py b/tests/unit/pipelex/cogt/models/test_model_deck_references.py
index a6164b956..1defeec71 100644
--- a/tests/unit/pipelex/cogt/models/test_model_deck_references.py
+++ b/tests/unit/pipelex/cogt/models/test_model_deck_references.py
@@ -56,8 +56,7 @@ class TestModelDeckReferences:
known_handles.update(self._get_local_backend_models())
# 2. Get gateway models from cached remote config
- remote_config = RemoteConfigFetcher.fetch_remote_config() # Uses cached version
- gateway_specs = remote_config.backend_model_specs
+ gateway_specs = RemoteConfigFetcher.fetch_remote_config().config.backend_model_specs # Uses cached version
# Get default model_type from gateway defaults section (same pattern as local backends)
gateway_defaults = gateway_specs.get("defaults", {})
diff --git a/tests/unit/pipelex/cogt/models/test_resolve_terminal_candidates.py b/tests/unit/pipelex/cogt/models/test_resolve_terminal_candidates.py
new file mode 100644
index 000000000..81a72790a
--- /dev/null
+++ b/tests/unit/pipelex/cogt/models/test_resolve_terminal_candidates.py
@@ -0,0 +1,231 @@
+"""Unit coverage for ``ModelManager._collect_candidates``.
+
+The resolver underpins the gateway-membership check in ``ModelManager.setup``: it walks
+aliases/waterfalls and returns every reachable terminal handle. The membership check accepts
+the deck if ANY candidate is in ``deck.inference_models`` or the gateway specs.
+
+Two regressions are pinned here:
+
+- A self-referential alias or waterfall must NOT loop forever.
+- A waterfall with multiple fallbacks must surface every fallback as a candidate (when
+ ``is_model_fallback_enabled`` is true, the default) so the membership check matches the
+ runtime fallback behaviour at ``model_deck._get_optional_inference_model_with_fallback``.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from pipelex.cogt.models.model_manager import (
+ ModelManager,
+)
+from pipelex.cogt.models.model_reference import ModelReference
+
+# ``_collect_candidates`` is the pure-logic leaf of the membership resolver; testing it
+# directly keeps the unit tests free of ModelDeck setup boilerplate.
+_collect_candidates = ModelManager._collect_candidates # noqa: SLF001 # pyright: ignore[reportPrivateUsage]
+
+
+class TestResolveTerminalCandidates:
+ def test_handle_returns_self(self) -> None:
+ ref = ModelReference.parse("claude-4.7-opus")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={},
+ waterfalls={},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == ["claude-4.7-opus"]
+
+ def test_alias_resolves_to_handle(self) -> None:
+ ref = ModelReference.parse("@best")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={"best": "claude-4.7-opus"},
+ waterfalls={},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == ["claude-4.7-opus"]
+
+ def test_dangling_alias_returns_alias_name(self) -> None:
+ """When an alias has no target mapping, surface the alias name itself so the
+ membership check fails with the user-visible identifier (the alias they wrote).
+ """
+ ref = ModelReference.parse("@missing-alias")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={},
+ waterfalls={},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == ["missing-alias"]
+
+ def test_alias_cycle_returns_empty(self) -> None:
+ """Self-referential alias chain returns no candidates — the membership check then
+ skips this reference rather than tripping ``GatewayUnknownModelError`` on a key the
+ user never asked the gateway to resolve.
+ """
+ ref = ModelReference.parse("@loop")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={"loop": "@loop"},
+ waterfalls={},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == []
+
+ def test_waterfall_returns_every_fallback(self) -> None:
+ """The fix for the false ``GatewayUnknownModelError`` when only later fallbacks are
+ valid. The whole point of a waterfall is "try in order, use whatever works."
+ """
+ ref = ModelReference.parse("~fast-llm")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={},
+ waterfalls={"fast-llm": ["future-model", "current-model"]},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == ["future-model", "current-model"]
+
+ def test_waterfall_when_fallback_disabled_returns_only_first(self) -> None:
+ """Mirrors ``model_deck._get_optional_inference_model_with_fallback``: when fallback
+ is disabled the runtime only tries the first entry, so the membership check should
+ only validate that one.
+ """
+ ref = ModelReference.parse("~fast-llm")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={},
+ waterfalls={"fast-llm": ["future-model", "current-model"]},
+ is_fallback_enabled=False,
+ visited=set(),
+ )
+ assert result == ["future-model"]
+
+ def test_self_referential_waterfall_returns_empty(self) -> None:
+ """Cycle protection on the waterfall branch (issue #3 from PR review).
+
+ Before the fix, ``waterfalls["A"] = ["~A"]`` would loop forever; now the visited
+ guard short-circuits the recursion.
+ """
+ ref = ModelReference.parse("~A")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={},
+ waterfalls={"A": ["~A"]},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == []
+
+ def test_waterfall_through_alias_unrolls_both(self) -> None:
+ """Mixed chains: alias → waterfall with two fallbacks → handles."""
+ ref = ModelReference.parse("@fast")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={"fast": "~tier"},
+ waterfalls={"tier": ["a", "b"]},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == ["a", "b"]
+
+ @pytest.mark.parametrize(
+ ("ref_str", "aliases", "waterfalls", "expected"),
+ [
+ ("@shared", {"shared": "~shared"}, {"shared": ["gpt-4o-mini", "claude-haiku"]}, ["gpt-4o-mini", "claude-haiku"]),
+ ("~shared", {"shared": "claude-4.7-opus"}, {"shared": ["@shared"]}, ["claude-4.7-opus"]),
+ ],
+ )
+ def test_shared_name_alias_and_waterfall_is_not_a_cycle(
+ self,
+ ref_str: str,
+ aliases: dict[str, str],
+ waterfalls: dict[str, list[str]],
+ expected: list[str],
+ ) -> None:
+ """An alias and a waterfall may share a name without forming a cycle.
+
+ Aliases and waterfalls live in separate dicts with no cross-validation, so a
+ deck can hold both ``@shared`` and ``~shared``. Cycle detection must key on
+ ``(kind, name)``: keying on name alone false-flags the second node as visited,
+ returns no candidates, and silently skips gateway membership validation.
+ """
+ result = _collect_candidates(
+ ref=ModelReference.parse(ref_str),
+ aliases=aliases,
+ waterfalls=waterfalls,
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == expected
+
+ def test_cross_kind_cycle_still_returns_empty(self) -> None:
+ """Keying on ``(kind, name)`` must not disable detection of a genuine cycle that
+ alternates kinds: ``@x`` → ``~x`` → ``@x`` revisits the alias node and short-circuits.
+ """
+ ref = ModelReference.parse("@x")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={"x": "~x"},
+ waterfalls={"x": ["@x"]},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == []
+
+ def test_waterfall_with_invalid_entry_skips_only_that_branch(self) -> None:
+ """A malformed waterfall entry that fails ``ModelReference.parse`` must not poison
+ the rest of the list — the resolver moves on to the next entry.
+ """
+ ref = ModelReference.parse("~mixed")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={},
+ waterfalls={"mixed": ["", "good-handle"]},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == ["good-handle"]
+
+ def test_preset_returns_empty(self) -> None:
+ """Presets are not handles; the membership check skips them."""
+ ref = ModelReference.parse("$some-preset")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={},
+ waterfalls={},
+ is_fallback_enabled=True,
+ visited=set(),
+ )
+ assert result == []
+
+ @pytest.mark.parametrize(
+ ("waterfall_entries", "is_fallback_enabled", "expected"),
+ [
+ (["a", "b", "c"], True, ["a", "b", "c"]),
+ (["a", "b", "c"], False, ["a"]),
+ (["only-one"], True, ["only-one"]),
+ (["only-one"], False, ["only-one"]),
+ ],
+ )
+ def test_waterfall_entries_respect_fallback_flag(
+ self,
+ waterfall_entries: list[str],
+ is_fallback_enabled: bool,
+ expected: list[str],
+ ) -> None:
+ ref = ModelReference.parse("~name")
+ result = _collect_candidates(
+ ref=ref,
+ aliases={},
+ waterfalls={"name": waterfall_entries},
+ is_fallback_enabled=is_fallback_enabled,
+ visited=set(),
+ )
+ assert result == expected
diff --git a/tests/unit/pipelex/system/pipelex_service/conftest.py b/tests/unit/pipelex/system/pipelex_service/conftest.py
index f0ef9fb92..259b65e1b 100644
--- a/tests/unit/pipelex/system/pipelex_service/conftest.py
+++ b/tests/unit/pipelex/system/pipelex_service/conftest.py
@@ -22,4 +22,5 @@ def mock_log(mocker: MockerFixture):
"""Mock the log module for all tests in this package."""
mock_logger = mocker.MagicMock()
mocker.patch("pipelex.system.pipelex_service.gateway_config_merger.log", mock_logger)
+ mocker.patch("pipelex.system.pipelex_service.remote_config_cache.log", mock_logger)
return mock_logger
diff --git a/tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py b/tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py
index ab14ade56..9b8ab2769 100644
--- a/tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py
+++ b/tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py
@@ -82,7 +82,11 @@ class TestGatewayTermsCheck:
# Mock the remote config fetch so we don't hit the network
mock_remote_config = mocker.MagicMock()
mock_remote_config.backend_model_specs = {}
- mocker.patch(f"{PIPELEX_MODULE}.RemoteConfigFetcher.fetch_remote_config", return_value=mock_remote_config)
+ mock_remote_config.aws_region = "us-east-1"
+ mock_result = mocker.MagicMock()
+ mock_result.config = mock_remote_config
+ mock_result.source = "fresh"
+ mocker.patch(f"{PIPELEX_MODULE}.RemoteConfigFetcher.fetch_remote_config", return_value=mock_result)
pipelex_instance = Pipelex.__new__(Pipelex)
diff --git a/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py b/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py
new file mode 100644
index 000000000..6f3332b88
--- /dev/null
+++ b/tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py
@@ -0,0 +1,161 @@
+"""Pure read/write/version tests for ``RemoteConfigCache``.
+
+These tests treat the cache as an isolated module: nothing else in the codebase calls it
+yet at the time these are written (Phase 1 only stands the cache up; Phase 2 wires it
+into ``RemoteConfigFetcher``).
+"""
+
+from __future__ import annotations
+
+import json
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+import pytest
+
+from pipelex.system.configuration.config_loader import ConfigLoader
+from pipelex.system.pipelex_service.remote_config import RemoteConfig
+from pipelex.system.pipelex_service.remote_config_cache import (
+ CACHE_SCHEMA_VERSION,
+ CachedRemoteConfig,
+ RemoteConfigCache,
+)
+
+if TYPE_CHECKING:
+ from unittest.mock import MagicMock
+
+ from pytest_mock import MockerFixture
+
+
+def _valid_remote_config_payload(extra: dict[str, object] | None = None) -> dict[str, object]:
+ """Build a minimal remote-config dict that ``RemoteConfig.model_validate`` accepts."""
+ payload: dict[str, object] = {
+ "posthog": {
+ "project_api_key": "test-key",
+ "endpoint": "https://posthog.example.com",
+ "is_geoip_enabled": False,
+ "is_debug_enabled": False,
+ },
+ "backend_model_specs": {},
+ "aws_region": "eu-west-3",
+ }
+ if extra is not None:
+ payload.update(extra)
+ return payload
+
+
+def isolated_cache_dir(tmp_path: Path, mocker: MockerFixture) -> Path:
+ """Point the cache at a fresh tmp ``~/.pipelex`` to keep tests hermetic."""
+ fake_global_dir = tmp_path / ".pipelex"
+ mocker.patch.object(
+ ConfigLoader,
+ "global_config_dir",
+ new_callable=mocker.PropertyMock,
+ return_value=fake_global_dir,
+ )
+ return fake_global_dir
+
+
+class TestRemoteConfigCache:
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_write_then_read_roundtrip(self) -> None:
+ payload = _valid_remote_config_payload()
+
+ before_store = datetime.now(tz=timezone.utc)
+ RemoteConfigCache.store(payload)
+ after_store = datetime.now(tz=timezone.utc)
+
+ loaded = RemoteConfigCache.load()
+
+ assert loaded is not None
+ assert isinstance(loaded, CachedRemoteConfig)
+ assert loaded.schema_version == CACHE_SCHEMA_VERSION
+ assert loaded.raw_config == payload
+ assert before_store - timedelta(seconds=1) <= loaded.cached_at <= after_store + timedelta(seconds=1)
+
+ # to_remote_config() must produce a valid RemoteConfig
+ remote_config = loaded.to_remote_config()
+ assert isinstance(remote_config, RemoteConfig)
+ assert remote_config.aws_region == "eu-west-3"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_read_missing_returns_none(self) -> None:
+ assert RemoteConfigCache.load() is None
+
+ def test_read_corrupted_json_returns_none_and_logs(
+ self,
+ isolated_cache_dir: Path,
+ mock_log: MagicMock,
+ ) -> None:
+ cache_path = isolated_cache_dir / "cache" / "remote_config.json"
+ cache_path.parent.mkdir(parents=True, exist_ok=True)
+ cache_path.write_text("{not valid json", encoding="utf-8")
+
+ assert RemoteConfigCache.load() is None
+
+ assert mock_log.warning.called, "load() must emit a warning when the cache file is unreadable"
+
+ def test_read_wrong_schema_version_returns_none(self, isolated_cache_dir: Path) -> None:
+ cache_path = isolated_cache_dir / "cache" / "remote_config.json"
+ cache_path.parent.mkdir(parents=True, exist_ok=True)
+ cache_path.write_text(
+ json.dumps(
+ {
+ "schema_version": CACHE_SCHEMA_VERSION + 99,
+ "cached_at": datetime.now(tz=timezone.utc).isoformat(),
+ "raw_config": _valid_remote_config_payload(),
+ }
+ ),
+ encoding="utf-8",
+ )
+
+ assert RemoteConfigCache.load() is None
+
+ def test_write_creates_parent_dir(self, isolated_cache_dir: Path) -> None:
+ cache_dir = isolated_cache_dir / "cache"
+ assert not cache_dir.exists()
+
+ RemoteConfigCache.store(_valid_remote_config_payload())
+
+ assert cache_dir.is_dir()
+ assert (cache_dir / "remote_config.json").is_file()
+
+ def test_cache_path_uses_global_config_dir(self, isolated_cache_dir: Path) -> None:
+ expected = isolated_cache_dir / "cache" / "remote_config.json"
+
+ assert RemoteConfigCache.cache_path() == expected
+
+ def test_write_is_atomic(self, isolated_cache_dir: Path, mocker: MockerFixture) -> None:
+ """If the atomic rename fails, the destination file must not exist or be corrupted."""
+ mocker.patch.object(
+ Path,
+ "replace",
+ side_effect=OSError("simulated replace failure"),
+ )
+
+ with pytest.raises(OSError, match="simulated replace failure"):
+ RemoteConfigCache.store(_valid_remote_config_payload())
+
+ cache_path = isolated_cache_dir / "cache" / "remote_config.json"
+ assert not cache_path.exists(), "destination cache file must not be created when replace fails"
+
+ # No leftover temp files in the cache dir either.
+ leftover = list((isolated_cache_dir / "cache").iterdir()) if (isolated_cache_dir / "cache").exists() else []
+ assert leftover == [], f"unexpected leftover files in cache dir: {leftover}"
+
+ @pytest.mark.usefixtures("isolated_cache_dir")
+ def test_raw_payload_extras_preserved(self) -> None:
+ """Unknown top-level keys must round-trip and be preserved on the loaded ``RemoteConfig``."""
+ payload = _valid_remote_config_payload(extra={"unknown_future_key": {"hello": "world"}})
+
+ RemoteConfigCache.store(payload)
+ loaded = RemoteConfigCache.load()
+
+ assert loaded is not None
+ assert loaded.raw_config["unknown_future_key"] == {"hello": "world"}
+
+ remote_config = loaded.to_remote_config()
+ # Pydantic ``extra="allow"`` keeps unknown fields accessible via model_extra
+ assert remote_config.model_extra == {"unknown_future_key": {"hello": "world"}}
diff --git a/tests/unit/pipelex/system/test_config_resolution.py b/tests/unit/pipelex/system/test_config_resolution.py
index 27edb4b8f..dbb519ede 100644
--- a/tests/unit/pipelex/system/test_config_resolution.py
+++ b/tests/unit/pipelex/system/test_config_resolution.py
@@ -257,14 +257,14 @@ class TestConfigResolution:
@pytest.mark.parametrize(
"marker",
- [".git", "pyproject.toml", "setup.py", "setup.cfg", "package.json", ".hg"],
+ [".pipelex", ".git", "pyproject.toml", "setup.py", "setup.cfg", "package.json", ".hg"],
)
def test_find_project_root_all_markers(self, tmp_path: Path, marker: str) -> None:
"""All supported markers are recognized."""
project_dir = tmp_path / "project"
project_dir.mkdir(parents=True)
marker_path = project_dir / marker
- if marker in {".git", ".hg"}:
+ if marker in {".pipelex", ".git", ".hg"}:
marker_path.mkdir()
else:
marker_path.write_text("")
@@ -275,6 +275,22 @@ class TestConfigResolution:
assert result == project_dir.resolve()
+ def test_project_config_dir_found_via_pipelex_marker_only(self, tmp_path: Path, mocker: MockerFixture) -> None:
+ """A folder containing only a .pipelex/ directory (no .git, no pyproject.toml) is a project root.
+
+ Regression: without .pipelex as a project root marker, such a folder fell through to the
+ global ~/.pipelex/ config, silently ignoring the project's own .pipelex/ overrides.
+ """
+ project_dir = tmp_path / "project"
+ (project_dir / ".pipelex").mkdir(parents=True)
+
+ mocker.patch.object(Path, "cwd", return_value=project_dir)
+ mocker.patch.object(Path, "home", return_value=tmp_path / "home")
+
+ loader = ConfigLoader()
+
+ assert loader.project_config_dir == (project_dir / ".pipelex").resolve()
+
def test_cross_platform_paths(self, tmp_path: Path, mocker: MockerFixture) -> None:
"""Config paths use Path throughout, no hardcoded separators."""
fake_home = tmp_path / "home"
diff --git a/wip/archive/offline-mode-remote-config-cache.md b/wip/archive/offline-mode-remote-config-cache.md
new file mode 100644
index 000000000..825b40197
--- /dev/null
+++ b/wip/archive/offline-mode-remote-config-cache.md
@@ -0,0 +1,537 @@
+# Offline Mode & Remote Config Cache — Implementation Plan
+
+> Branch: `fix/Offline-mode`
+> Working dir: `/Users/lchoquel/repos/Pipelex/_off` (worktree of `pipelex/`)
+
+## Objectives (verbatim from user)
+
+1. **Fully support offline mode when gateway is disabled** — no remote config fetch is ever attempted; setup completes without network.
+2. **Support dry-run mode when gateway is enabled but the remote config is temporarily unavailable**, even when the bundle references gateway models. Implies a cache that must be **primed at the time Pipelex first creates its local or global config**, and refreshed on every successful subsequent fetch.
+3. **Fail if requested gateway models don't exist** according to the fresh remote config (when available) or the cached one (when that's all we have). No silent fallbacks.
+
+## Non-objectives
+
+- We are not removing the remote fetch. We are adding resilience.
+- We are not granting offline inference *runs* through the gateway (the gateway itself still needs network at call time). We're talking about *setup* and *dry-run* working offline. Real inference calls fail as they always have when the network is down.
+- We are not bundling a baked-in snapshot in the package.
+
+## Failure point (already diagnosed)
+
+- `pipelex/pipelex.py:202` — `effective_needs_model_specs = needs_model_specs if needs_model_specs is not None else needs_inference`. When the agent CLI passes `needs_model_specs=True` explicitly (it does for `validate`, `inputs`, and `run --dry-run`), this is `True` regardless of `needs_inference`.
+- `pipelex/pipelex.py:235` — unconditionally calls `RemoteConfigFetcher.fetch_remote_config()` when `is_pipelex_service_enabled and effective_needs_model_specs`. Raises `RemoteConfigFetchError` on any network failure.
+- `pipelex/system/pipelex_service/remote_config_fetcher.py:93` — only short-circuits for Codex *Cloud* (`runtime_manager.is_in_codex_cloud`). The cache fallback added below also solves the local-sandbox case incidentally; no new sandbox detection is needed.
+
+## Design overview
+
+### Cache contract
+
+- Location: `~/.pipelex/cache/remote_config.json` (global, single-user). Schema-versioned and timestamped.
+- **Stored payload is the raw JSON dict** returned by `response.json()`, NOT a re-serialized Pydantic model. This avoids any Pydantic round-trip quirks (e.g. `extra="allow"` semantics across `model_dump()` modes) and keeps the cache stable against minor schema drift. On load: `RemoteConfig.model_validate(payload["raw_config"])`.
+- The remote config URL is versioned in `pipelex-back-office` (we control both ends). Schema bumps are handled by bumping both the URL version AND the cache `SCHEMA_VERSION` constant; older caches are then rejected (return `None`) and clients re-prime on next online fetch.
+- **Atomic writes**: `tempfile.NamedTemporaryFile(dir=cache_dir, delete=False)` → write → `os.replace(tmp_path, cache_path)`. Prevents corrupted-cache from concurrent writes.
+- Written by:
+ - `RemoteConfigFetcher.fetch_remote_config()` on every successful fetch (opportunistic refresh).
+ - The init flow when gateway terms are accepted (priming step).
+- Read by:
+ - `RemoteConfigFetcher.fetch_remote_config()` when the network call fails, as a fallback.
+- Never read for any other purpose. Cache is a **last resort**, not a freshness optimisation.
+
+### New types
+
+- `CachedRemoteConfig` — Pydantic `BaseModel` wrapping the cached payload metadata: `cached_at: datetime`, `cache_schema_version: int`, `raw_config: dict[str, Any]` (the raw remote payload). Use `datetime.timezone.utc`, not `datetime.UTC`.
+- `RemoteConfigSource` — `StrEnum` from `pipelex.types` with **lowercase string values** to match codebase convention: `FRESH = "fresh"` | `CACHED = "cached"`.
+- `RemoteConfigResult` — Pydantic `BaseModel` (`ConfigDict(extra="forbid", strict=True)`) carrying `config: RemoteConfig`, `source: RemoteConfigSource`, `cached_at: datetime | None = None`. NOT a dataclass; match project convention.
+
+### New / preserved exceptions (in `pipelex/system/pipelex_service/exceptions.py`)
+
+- **Keep** `RemoteConfigFetchError` as the existing internal exception raised by the retry layer when the network fails. Do NOT remove it — it's already caught by `doctor_cmd.py:745`, `cli_factory.py:73`, `error_handlers.py:365`, and the agent CLI hints (`agent_output.py:51`). Removing it breaks those surfaces.
+- **Add** `RemoteConfigUnavailableError(PipelexServiceError)` as a sibling — raised when fetch failed AND no usable cache exists. Message names the cache path and remediation (`pipelex init` while online; or disable gateway in `backends.toml` as the offline-permanent alternative).
+- **Add** `GatewayUnknownModelError(PipelexServiceError)` — raised when a bundle references a gateway model that isn't present in the (fresh or cached) gateway specs. Carries a `source: RemoteConfigSource` field; message branches: when `CACHED`, hint that the remote config is stale and suggest `pipelex init` while online.
+- **Add** `RemoteConfigStaleWarning(UserWarning)` — emitted when `source == CACHED` during setup. Lives next to `GatewayOverrideWarning` in `exceptions.py`.
+
+### Wiring in `pipelex.py`
+
+- `setup()` keeps calling `RemoteConfigFetcher.fetch_remote_config()`, but the fetcher now returns a `RemoteConfigResult`.
+- The `source` is **NOT stored on `GatewayConfig`** (which keeps `extra="forbid"` and stays a pure value object describing the gateway). Instead, `setup()` passes `gateway_config_source: RemoteConfigSource` separately to `models_manager.setup()` → `BackendLibrary` so the model-membership check can branch its error message on fresh-vs-cached.
+- When `source == CACHED`:
+ - `is_pipelex_telemetry_enabled` stays `False` (already disabled without inference; this is a separate, stricter guard for stale operation). Add an explicit test for this.
+ - Emit `RemoteConfigStaleWarning` via `warnings.warn(...)`.
+ - The agent CLI `agent_success(...)` envelope surfaces a structured `warnings: [{"type": "RemoteConfigStale", "cached_at": "..."}]` field so machine consumers see it.
+
+### Doc / fixture generators must refuse stale data
+
+`gateway_models_generator.py:46` and `preprocess_test_models_cmd.py:98` regenerate **committed** reference docs and test fixtures from the remote config. If they silently fall back to cache, they'll bake stale data into the repo.
+
+- Add a `require_fresh: bool = False` argument to `fetch_remote_config()` (or expose a thin `fetch_remote_config_fresh_only()` wrapper) that raises `RemoteConfigUnavailableError` immediately if `source == CACHED`.
+- Both dev-CLI generators set it; everyone else doesn't.
+- Add a unit test that this branch raises when only cache is available.
+
+### Behaviour matrix to validate
+
+| Scenario | Gateway enabled | Network | Cache | Expected |
+|---|---|---|---|---|
+| BYOK offline | no | down | n/a | setup OK, validate OK, dry-run OK, inference OK if backend reachable |
+| Gateway dry-run, fresh | yes | up | n/a | fresh fetch, dry-run OK |
+| Gateway dry-run, cached | yes | down | present | cache fallback, warning, dry-run OK |
+| Gateway dry-run, cold | yes | down | absent | `RemoteConfigUnavailableError` with clear remediation |
+| Gateway inference, cached | yes | down | present | cache fallback, inference still fails when call is made (network); error message attributes the failure to the gateway call, not setup |
+| Bundle references unknown model | yes | up or down | any | `GatewayUnknownModelError` during library load (message branches on source) |
+| Doc generator offline w/ cache | yes | down | present | `RemoteConfigUnavailableError` (require_fresh=True refuses cache) |
+
+---
+
+## Working agreements
+
+- TDD throughout. Red → green for every functional unit.
+- Integration tests declare **blueprints in Python** and exercise the public APIs (`Pipelex.make`, `RemoteConfigFetcher`, `BackendLibrary`, etc.). They go under `tests/integration/pipelex/`.
+- E2E tests use **actual `.mthds` bundle files** placed in `tests/e2e/data/offline_mode/` and drive them via `mthds-agent run bundle ... --dry-run --mock-inputs` (subprocess), the same surface that fails today.
+- Follow `.claude/rules/python-standards.md`:
+ - 1 TestClass per test module.
+ - `StrEnum` from `pipelex.types`.
+ - Never `except Exception` outside CLI roots. Use `from pytest_mock import MockerFixture`; never `unittest.mock`.
+ - `make agent-check` after code changes, `make agent-test` at session boundaries.
+- No backward compatibility shims. If we rename or change a signature, just change it.
+- **No test-aware production code**: do NOT introduce a `PIPELEX_TEST_BLOCK_REMOTE_FETCH` env var. Use the legitimate `PIPELEX_REMOTE_CONFIG_URL` override (see Phase 5) to point at an unreachable URL.
+
+## Cold-start handoff protocol
+
+After every `🛑 CHECKPOINT` line:
+
+1. Re-read this `TODOS.md` and mark just-completed checkboxes.
+2. Append a `### Checkpoint N status` subsection at the bottom of this file capturing:
+ - What landed (paths of new/modified files).
+ - Open decisions / unknowns.
+ - The exact next checkbox to start with.
+3. Run `make agent-check && make agent-test`. Note pass/fail in the checkpoint status.
+4. Stop. A new session takes over from the status block.
+
+---
+
+## Phase 0 — Lock down current behaviour (no production code changes)
+
+- [x] **0.1** Add `tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py` with `TestOfflineBaseline` covering the baseline that must NOT regress: BYOK-offline succeeds, gateway-offline-without-cache raises today. Write these as **passing tests against current behaviour** (do not write "expected-to-fail" tests). In Phases 2/3 they will be updated as semantics legitimately change, not "flipped." Use `mocker.patch("httpx.get", side_effect=httpx.ConnectError(...))`.
+- [x] **0.2** Confirm by running the new tests that `Pipelex.make(needs_inference=False, needs_model_specs=True)` with gateway disabled (BYOK backend only) succeeds offline. If it does not, file a sub-task here before continuing — a phantom network call somewhere has to be tracked down. **Note**: 3.2 will add an explicit regression guard for this; 0.2 is the one-time confirmation.
+- [x] **0.3** Run `make agent-check && make agent-test`. Record output in the checkpoint status.
+
+🛑 **CHECKPOINT 0** — Baseline pinned. Cold-start safe: new session can pick up from Phase 1 with confidence about what already works.
+
+---
+
+## Phase 1 — Cache module (TDD, isolated, no integration yet)
+
+**Goal:** stand up `RemoteConfigCache` with no other code calling it yet. Pure read/write/version logic.
+
+- [x] **1.1** Write failing tests first: `tests/unit/system/pipelex_service/test_remote_config_cache.py` with `TestRemoteConfigCache`:
+ - `test_write_then_read_roundtrip` — writes a config, reads it back, asserts equality + non-stale timestamp.
+ - `test_read_missing_returns_none` — no cache file → `None` (no exception).
+ - `test_read_corrupted_json_returns_none_and_logs` — file exists but isn't JSON → returns `None`, logs a warning. (Use `caplog`.)
+ - `test_read_wrong_schema_version_returns_none` — file from a different schema version → returns `None`.
+ - `test_write_creates_parent_dir` — cache dir doesn't exist yet → created.
+ - `test_cache_path_uses_global_config_dir` — path lives under `config_manager.global_config_dir / "cache" / "remote_config.json"`.
+ - `test_write_is_atomic` — simulate failure between write-to-tmp and `os.replace` (e.g. via `mocker.patch("os.replace", side_effect=OSError)`); assert no partial file at the destination path.
+ - `test_raw_payload_extras_preserved` — store a payload with an unknown top-level key, load it back, pass through `RemoteConfig.model_validate(...)`, and assert the unknown key is still on the resulting model (Pydantic `extra="allow"` semantics). Guards the cache-integrity invariant.
+- [x] **1.2** Implement `pipelex/system/pipelex_service/remote_config_cache.py`:
+ - `class RemoteConfigCache` with `@classmethod load() -> CachedRemoteConfig | None` and `@classmethod store(remote_config_payload: dict[str, Any]) -> None`. **The `store` argument is the raw JSON dict from `response.json()`, not a `RemoteConfig` instance.**
+ - JSON file format on disk: `{"schema_version": 1, "cached_at": "<iso>", "raw_config": { ... raw remote payload ... }}`.
+ - Atomic write via `tempfile.NamedTemporaryFile(dir=cache_dir, delete=False)` + `os.replace`.
+ - Catch only `OSError`, `JSONDecodeError`, `ValidationError` — never bare `Exception`.
+- [x] **1.3** Add `CachedRemoteConfig` Pydantic model alongside `RemoteConfigCache`. Fields: `schema_version: int`, `cached_at: datetime`, `raw_config: dict[str, Any]`. Use `datetime.timezone.utc` (Python 3.10 compatibility). Provide a `to_remote_config()` helper that returns `RemoteConfig.model_validate(self.raw_config)`.
+- [x] **1.4** `make agent-check` clean.
+- [x] **1.5** Tests green.
+
+🛑 **CHECKPOINT 1** — Cache module shippable on its own. Cold-start safe.
+
+---
+
+## Phase 2 — Fetch with fallback (TDD)
+
+**Goal:** `RemoteConfigFetcher.fetch_remote_config()` returns a `RemoteConfigResult` carrying source provenance, writes cache on success, reads cache on failure, raises `RemoteConfigUnavailableError` only when both fail. Keep `RemoteConfigFetchError` as the internal exception raised by the retry layer.
+
+- [x] **2.1** Failing tests in `tests/integration/pipelex/system/pipelex_service/test_remote_config_fetcher.py` with `TestRemoteConfigFetcher`:
+ - `test_success_returns_fresh_and_writes_cache` — `mocker.patch("httpx.get")` returns a valid payload → result has `source=FRESH`, cache file written.
+ - `test_network_failure_with_cache_returns_cached` — `httpx.get` raises `httpx.ConnectError`, cache is pre-populated via `RemoteConfigCache.store(...)` → result has `source=CACHED`, `RemoteConfigStaleWarning` emitted.
+ - `test_network_failure_without_cache_raises_unavailable` — `httpx.get` raises, no cache → `RemoteConfigUnavailableError` with cache path in message.
+ - `test_http_error_with_cache_returns_cached` — 5xx response with cache present → cached fallback.
+ - `test_http_error_without_cache_raises_unavailable` — 5xx no cache → `RemoteConfigUnavailableError`.
+ - `test_validation_error_does_not_fall_back` — the remote responded with garbage JSON; we must still surface `RemoteConfigValidationError` and **not** silently fall back to cache (a server-side schema break is a real bug, and we control the server).
+ - `test_codex_cloud_short_circuit_still_works` — `runtime_manager.is_in_codex_cloud=True` → returns dummy with `source=FRESH`, does not write cache.
+ - `test_succeeds_after_4_transient_failures_no_cache_fallback` — `httpx.get` raises `ConnectError` 4 times then returns valid payload; tenacity (`stop_after_attempt(5)`) rescues; result has `source=FRESH`, no cache fallback was triggered.
+ - `test_falls_back_to_cache_after_5_transient_failures` — all 5 attempts raise; cache is pre-populated; result has `source=CACHED`.
+ - `test_require_fresh_refuses_cache` — `httpx.get` raises, cache populated, but caller passed `require_fresh=True` → raises `RemoteConfigUnavailableError` (doc generators must not silently use cache).
+- [x] **2.2** Implement the new control flow in `remote_config_fetcher.py`:
+ - Change `fetch_remote_config` to return `RemoteConfigResult(config, source, cached_at)` and accept `require_fresh: bool = False`.
+ - Keep raising the existing `RemoteConfigFetchError` from the inner `_fetch_remote_config_with_retry` path (don't churn that).
+ - In the outer flow: catch `RemoteConfigFetchError`. If `require_fresh`: re-raise as `RemoteConfigUnavailableError`. Else: try cache; if hit, emit `RemoteConfigStaleWarning` + return `source=CACHED`; if miss, raise `RemoteConfigUnavailableError` (message includes cache path).
+ - Persist to cache on success.
+ - Leave `RemoteConfigValidationError` path untouched (no cache fallback).
+- [x] **2.3** Add `RemoteConfigUnavailableError` and `RemoteConfigStaleWarning` to `exceptions.py`. `RemoteConfigUnavailableError` message: include cache path and remediation (`pipelex init` while online to prime cache; disable gateway in `backends.toml` as the offline-permanent alternative). Do NOT remove `RemoteConfigFetchError`.
+- [x] **2.4** Update every existing caller of `fetch_remote_config()`. Pre-checked call sites (run `grep -rn "fetch_remote_config" --include="*.py"` to verify nothing else has appeared):
+ - `pipelex/pipelex.py:235` — main setup; needs `source` for downstream wiring.
+ - `pipelex/cli/commands/doctor_cmd.py:740` — doctor; surface cache age in output.
+ - `pipelex/cli/dev_cli/commands/gateway_models_generator.py:46` — **must pass `require_fresh=True`** (regenerates committed docs).
+ - `pipelex/cli/dev_cli/commands/preprocess_test_models_cmd.py:98` — **must pass `require_fresh=True`** (rewrites test fixtures).
+ - `tests/conftest.py:35` — session cache shim; just unwrap `.config`.
+ - `tests/unit/pipelex/cogt/models/test_model_deck_references.py:59` — just unwrap `.config`.
+ Most callers do `result = ...; remote_config = result.config`.
+- [x] **2.4b** While editing `doctor_cmd.py` for 2.4, replace the **pre-existing** `except Exception as exc:` at `doctor_cmd.py:767` with the specific exception classes that `models_manager.setup()` and `validate_model_deck()` can raise (the lineage is already enumerated in `pipelex.py:307-336`). This is an existing CLAUDE.md violation; fix it while we're here.
+- [x] **2.5** `make agent-check` clean. Tests green.
+
+🛑 **CHECKPOINT 2** — Fetcher is now resilient and provenance-aware. Cold-start safe.
+
+---
+
+## Phase 3 — Plumb `source` through setup and enforce model membership (TDD)
+
+**Goal:** downstream code can see whether the config it has is fresh or cached; backend library raises a clear error when a bundle references a gateway model not present in the (fresh or cached) specs. **Source is passed as a separate argument, NOT stored on `GatewayConfig`** — `GatewayConfig` stays `extra="forbid"` and remains a pure value object.
+
+- [x] **3.1** Failing tests in `tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py` with `TestGatewayUnknownModel`:
+ - `test_known_model_loads` — gateway specs contain `gpt-x`, backend deck references it → load OK.
+ - `test_unknown_model_fresh_raises` — backend deck references `gpt-future` not in fresh gateway specs → `GatewayUnknownModelError` with `source=FRESH` and the model name in the message.
+ - `test_unknown_model_cached_raises_with_stale_hint` — same as above but source is `CACHED`; message hints "remote config is stale; run `pipelex init` while online to refresh".
+- [x] **3.2** Failing tests in `tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py` with `TestSetupWithCache`:
+ - `test_setup_succeeds_with_stale_cache_dry_run` — `Pipelex.make(needs_inference=False, needs_model_specs=True)` with gateway enabled + httpx mocked to fail + cache primed → setup succeeds, a `RemoteConfigStaleWarning` is emitted.
+ - `test_setup_fails_without_cache_dry_run` — same as above but cache empty → `RemoteConfigUnavailableError`.
+ - `test_telemetry_disabled_when_source_cached` — `Pipelex.make(needs_inference=True)` with gateway enabled + cache fallback → assert the active telemetry manager is the no-op variant (or `is_pipelex_telemetry_enabled` is `False`). Guards the "stale ops shouldn't phone home" invariant.
+ - `test_byok_offline_regression_guard` — `Pipelex.make(needs_inference=False, needs_model_specs=True)` with gateway **disabled** in backends.toml + httpx mocked to fail → setup succeeds without ever calling `fetch_remote_config()` (assert via `mocker.spy`). This is the explicit regression test that 0.2 only verifies one-time; without it, a future refactor could re-introduce a phantom network call.
+- [x] **3.3** Implement:
+ - Add a `gateway_config_source: RemoteConfigSource | None` parameter to `ModelManager.setup()` and `BackendLibrary._load_gateway_model_specs()`. `None` means no gateway active.
+ - In `pipelex.py:setup()`, after the fetch:
+ - Build `GatewayConfig` from `result.config` (no `is_from_cache` field — `GatewayConfig` stays unchanged).
+ - Pass `result.source` to `models_manager.setup(...)` as `gateway_config_source`.
+ - When `result.source == CACHED`, emit `RemoteConfigStaleWarning` (UserWarning subclass next to `GatewayOverrideWarning`).
+ - In `BackendLibrary._load_gateway_model_specs` (or wherever model_handle resolution lives — read the call site before deciding), raise `GatewayUnknownModelError(model_name=..., source=gateway_config_source)` when a referenced gateway model isn't in `gateway_config.model_specs`. Message branches on `source`.
+- [x] **3.4** Update agent CLI error mapping in `pipelex/cli/agent_cli/commands/agent_output.py` (`AGENT_ERROR_HINTS`, `AGENT_ERROR_DOMAINS`) for `RemoteConfigUnavailableError` and `GatewayUnknownModelError`. Plumb the structured `warnings` field into the `agent_success(...)` envelope so machine consumers see `[{"type": "RemoteConfigStale", "cached_at": "..."}]` when source is cached.
+- [x] **3.5** `make agent-check` clean. Tests green.
+
+🛑 **CHECKPOINT 3** — Stale operation is safe, model membership is enforced. Cold-start safe.
+
+---
+
+## Phase 4 — Cache priming at init (TDD)
+
+**Goal:** when `pipelex init` (or the agent-cli equivalent) completes the gateway terms acceptance flow, attempt one fetch and persist the cache. If offline at init time, log a warning and continue — cache stays empty, user has been told.
+
+- [x] **4.1** Failing tests in `tests/integration/pipelex/cli/commands/init/test_cache_priming.py` with `TestCachePriming`:
+ - `test_init_primes_cache_when_online` — run init flow with gateway enabled + httpx mocked to succeed → cache file present and readable afterwards.
+ - `test_init_warns_when_offline` — same flow but httpx mocked to fail → init completes (no crash), cache file not present, warning printed to console.
+ - `test_init_skips_priming_when_gateway_disabled` — gateway disabled in backends → no fetch attempted, no cache written.
+ - `test_init_does_not_double_prime` — if cache already exists and is fresh, init can still refresh it (this is fine; assert the file was rewritten).
+- [x] **4.2** Implement priming hook: in `pipelex/cli/commands/init/command.py`, after `_check_gateway_terms_if_needed` has accepted terms and gateway remains enabled, call a new helper `prime_remote_config_cache(console)` that calls `RemoteConfigFetcher.fetch_remote_config()` inside a try/except limited to `RemoteConfigUnavailableError`, `RemoteConfigFetchError`, and `RemoteConfigValidationError`. On unavailable/fetch-error: log a clear yellow warning; on validation error: re-raise (server schema break is fatal, and we control the server). **Never `except Exception`.**
+- [x] **4.3** Mirror the same hook into the agent CLI init path (`pipelex/cli/agent_cli/commands/init_cmd.py`) so machine clients also prime.
+- [x] **4.4** `make agent-check` clean. Tests green.
+
+🛑 **CHECKPOINT 4** — Cache lifecycle complete. Cold-start safe.
+
+---
+
+## Phase 5 — E2E with real `.mthds` bundles
+
+**Goal:** exercise the full surface that the codex-sandbox handoff doc describes (`mthds-agent run bundle ... --dry-run --mock-inputs`) through subprocess invocations.
+
+- [x] **5.1** Create `tests/e2e/data/offline_mode/` with three minimal bundles:
+ - `byok_simple/` — references a backend model that does **not** require the gateway (e.g. ollama or anthropic). All `.mthds` files necessary to validate, plus an `inputs.json` shape for `--mock-inputs`.
+ - `gateway_known_model/` — references a gateway model that exists in the test fixture remote config.
+ - `gateway_unknown_model/` — references a gateway model that does **not** exist in the test fixture remote config.
+- [x] **5.2** Add a one-line indirection in `pipelex/system/pipelex_service/pipelex_details.py`: `REMOTE_CONFIG_URL` reads from the `PIPELEX_REMOTE_CONFIG_URL` env var if set, else uses the hard-coded production URL. This is a **legitimate config knob** (useful for testing/staging), not a test backdoor. Failing tests in `tests/e2e/agent_cli/test_offline_run_dry.py` with `TestOfflineDryRun`. Mark `@pytest.mark.asyncio(loop_scope="class")` if any tests are async. Each test:
+ - Invokes the CLI via `subprocess.run([".venv/bin/pipelex-agent", "run", "bundle", str(bundle_dir), "--dry-run", "--mock-inputs"], ...)` (the CLI binary is `pipelex-agent`, not `mthds-agent`).
+ - Uses a `monkeypatch`-style fixture to point `HOME` (and therefore `~/.pipelex/cache/`) at a tmp path, so cache state is controlled per-test.
+ - Sets `PIPELEX_REMOTE_CONFIG_URL=http://127.0.0.1:1/` (a reserved-port unreachable URL) to simulate offline. Real `httpx`, real `ConnectError`, no production-code-aware-of-tests.
+- [x] **5.3** Test cases:
+ - `test_byok_offline_succeeds` — gateway disabled (test bundle ships with `backends.toml` that has gateway disabled), no network, no cache → exit 0, structured success JSON.
+ - `test_gateway_known_with_cache_succeeds_offline` — gateway enabled, no network, cache primed → exit 0, success JSON includes a `warnings` array with `{"type": "RemoteConfigStale", ...}`.
+ - `test_gateway_unknown_with_cache_fails_with_clear_error` — exit nonzero, the unknown handle surfaces in the JSON envelope. Implementation note: pinning `error_type == "GatewayUnknownModelError"` would require an extra deck override to make the membership check fire (the bundle's pipe-level model reference fails at the pipe-operator layer instead). The test asserts the handle is visible without locking the error_type, leaving room for both failure paths.
+ - `test_gateway_no_cache_no_network_fails_with_unavailable` — exit nonzero, `error_type == "RemoteConfigUnavailableError"`, message references priming via `pipelex init`.
+- [x] **5.4** `make agent-check` clean. Tests green.
+
+🛑 **CHECKPOINT 5** — End-to-end behaviour validated against the exact surface the codex-sandbox doc exercised. Cold-start safe.
+
+---
+
+## Phase 6 — User-facing polish
+
+- [x] **6.1** Update `pipelex/cli/agent_cli/commands/agent_output.py`:
+ - Add `AGENT_ERROR_HINTS` entries for `RemoteConfigUnavailableError` and `GatewayUnknownModelError`.
+ - Add `AGENT_ERROR_DOMAINS` entries for both.
+ - Keep the existing `RemoteConfigFetchError` and `RemoteConfigValidationError` hints intact.
+- [x] **6.2** Update `pipelex/cli/error_handlers.py` for the Rich CLI surface: add handlers for `RemoteConfigUnavailableError` and `GatewayUnknownModelError` mirroring the agent-cli hints. Keep existing `RemoteConfigFetchError`/`RemoteConfigValidationError` handlers.
+- [x] **6.3** Update relevant CLAUDE.md sections:
+ - `pipelex/cli/agent_cli/CLAUDE.md` if error classes are mentioned.
+ - `pipelex/system/pipelex_service/CLAUDE.md` if it exists; create a short note if not (only if there's already a convention for service-level docs).
+- [x] **6.4** Add a `## [Unreleased]` entry to `CHANGELOG.md` summarising the user-visible behaviour change (offline support + new error types + new warning).
+
+> **Deferred from this PR:** Phase 6.3-old (`pipelex doctor` cache-age reporting) and Phase 6.6-old (cross-repo `mthds-plugins/wip/codex-sandbox-escalation.md` update). See "Deferred follow-ups" at the bottom of this file.
+
+🛑 **CHECKPOINT 6** — User-facing surface complete. Cold-start safe.
+
+---
+
+## Phase 7 — Final verification
+
+- [x] **7.1** `make agent-check` clean.
+- [x] **7.2** `make agent-test` clean.
+- [x] **7.3** Manually reproduce all scenarios from the behaviour matrix:
+ - BYOK offline `mthds-agent run bundle ... --dry-run --mock-inputs` → success.
+ - Gateway online, no cache → success, cache written.
+ - Gateway offline, cache present → success with stale warning.
+ - Gateway offline, no cache → clear `RemoteConfigUnavailableError`.
+ - Bundle references unknown gateway model → clear `GatewayUnknownModelError`.
+ - `pipelex-dev update-gateway-models` offline → clear refusal (no stale docs written).
+- [x] **7.4** Squash-friendly commit history; PR description references the codex-sandbox-escalation handoff and the deferred follow-ups.
+
+🛑 **CHECKPOINT 7** — Ready for review.
+
+---
+
+## Risk log
+
+- **Caller migration in Phase 2.** Every call site of `fetch_remote_config()` must be updated. If a caller is missed, it'll break at runtime in dev-CLI tooling that's rarely exercised. Grep before declaring 2.4 done. Current known sites are listed in 2.4 — re-run the grep to be sure.
+- **Cache poisoning during validation breakage.** We explicitly chose NOT to fall back to cache on `RemoteConfigValidationError`. Since we control the remote config server (`../pipelex-back-office`) AND the URL is versioned, a schema-rejecting payload is a real server-side bug, not an expected operational state. Failing loudly is the desired behaviour. Confirm this design choice survives review.
+- **Schema version bumps are coordinated.** The remote-config URL is versioned in `pipelex-back-office`. To change the `RemoteConfig` shape: bump the URL version in back-office AND bump `RemoteConfigCache.SCHEMA_VERSION` in this repo. Old caches return `None` on load (test in 1.1) and re-prime on next online fetch. Document this in the cache module docstring.
+- **`is_first_time_backends_setup` semantics.** Phase 4 priming is conditional on terms-accepted-and-gateway-enabled state. The init flow has a few branches; verify priming runs on the **first** init pass, not only on subsequent re-inits.
+- **Doc-generator silent staleness was the critical-gap finding.** Phase 2 fixes it via `require_fresh=True`. If you drop that constraint, you reopen the regression.
+
+## Open questions to resolve as we go
+
+- Maximum acceptable cache age before we refuse to use it? **Default: no cap.** Gateway model lists drift slowly; an old cache is better than no cache. Revisit if it bites us.
+- Should the agent-cli `init` command auto-prime even without `--gateway`? **No.** Priming only when gateway is selected.
+
+---
+
+## Deferred follow-ups (out of this PR's scope)
+
+Capture as separate work after this lands:
+
+1. **`pipelex doctor` cache reporting.** Extend `doctor` to surface cache presence, age, gateway-enabled-but-missing-cache hint. Pure additive feature; orthogonal to the offline-fix. Was Phase 6.3 in the original plan.
+2. **Codex Cloud cache-first short-circuit.** Today `remote_config_fetcher.py:93` returns a dummy unconditionally in Codex Cloud. After this PR ships, change the short-circuit to try cache first and fall back to dummy. Low risk because we already trust the cache.
+3. **Cross-repo: `mthds-plugins/wip/codex-sandbox-escalation.md`.** Note that `--dry-run` no longer requires escalation once Pipelex is initialised online once. Lives in a different repo; do as a separate PR there. Was Phase 6.6 in the original plan.
+4. **Cache TTL revisit.** Currently uncapped. Re-evaluate once we have telemetry on cache-fallback usage in production.
+5. **Schema-version migration runbook.** When `RemoteConfig` shape actually changes, write down the URL-version + `SCHEMA_VERSION` bump procedure. Not needed until the first real schema bump.
+6. **`pipelex-agent run bundle --output-dir` flag.** Today the CLI writes `dry_run.json` (and `live_run.json`, graph HTML) next to the input bundle (`_run_core.py:92` derives `output_dir` from the bundle path). That side effect forced Phase 5 to stage each bundle into a tmp dir before invocation, and would also break invocations against a read-only mounted bundle path (the codex-sandbox case). Adding `--output-dir <path>` (and falling back to the current behaviour when unset) would let read-only callers operate without copying. Surfaced in the Phase 5 code review.
+7. **E2E test pinning `GatewayUnknownModelError` end-to-end.** Phase 5's `test_gateway_unknown_with_cache_fails_with_clear_error` asserts the unknown handle is visible in the envelope without pinning `error_type`, because the bundle's pipe-level model reference fires `ModelChoiceNotFoundError`/`PipeOperatorModelAvailabilityError` at the pipe-operator layer rather than the deck-level membership check. Phase 3's unit test (`test_gateway_unknown_model.py`) covers the membership check directly; a follow-up E2E with a deck override that points a preset at a missing handle would pin the agent-CLI's `GatewayUnknownModelError` mapping end-to-end. Surfaced in the Phase 5 code review.
+
+---
+
+## Checkpoint status log
+
+<!-- Append a "### Checkpoint N status" block here after each 🛑 CHECKPOINT, capturing what landed, open decisions, and the next checkbox to start with. Keep blocks short; the codebase is authoritative. -->
+
+### Checkpoint 0 status
+
+**Landed:**
+- `tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py` — `TestOfflineBaseline` with two passing tests:
+ - `test_byok_offline_setup_succeeds_without_fetching_remote_config` — gateway disabled, no network, `fetch_remote_config` and `httpx.get` are asserted to be uncalled.
+ - `test_gateway_offline_without_cache_raises_remote_config_fetch_error` — gateway enabled + terms accepted + `httpx.get` raising `ConnectError` raises `RemoteConfigFetchError`. Bypasses the session-scoped cache patch via a module-level capture of the original classmethod.
+
+**Workaround:** The test module overrides the autouse `reset_pipelex_config_fixture` (no eager `Pipelex.make`) and the gateway-offline test explicitly calls `log.reset()` after the expected failure, because `Pipelex.make` does not reset log state when setup raises. This is pre-existing behaviour — flagged for future fix, not in scope here.
+
+**Verification:**
+- `make agent-check` → clean.
+- `make agent-test` → all green (5253 passed, 2 skipped, 3 xfailed; matches main).
+
+**Next:** Phase 1 — start at **1.1** (failing tests for `RemoteConfigCache` in `tests/unit/system/pipelex_service/test_remote_config_cache.py`).
+
+### Checkpoint 1 status
+
+**Landed:**
+- `pipelex/system/pipelex_service/remote_config_cache.py` — `CachedRemoteConfig` (Pydantic) + `RemoteConfigCache` (class with `cache_path()`, `load()`, `store()`). On-disk layout `{schema_version, cached_at, raw_config}`. Atomic writes via `tempfile.NamedTemporaryFile` + `os.replace` (kept explicit `os.replace` with a per-line `# noqa: PTH202`, with cleanup of the tmp file on failure via a `replaced` flag in `finally`). Module constant `CACHE_SCHEMA_VERSION = 1`.
+- `tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py` — 8 tests covering roundtrip, missing-file, corrupted-JSON, wrong-schema-version, parent-dir creation, cache-path location, atomic-write-on-rename-failure, and raw-payload extras preservation.
+- `tests/unit/pipelex/system/pipelex_service/conftest.py` — extended the existing `mock_log` autouse fixture to also patch `remote_config_cache.log`, so unit tests don't need a configured Pipelex log stack.
+
+**Decisions:**
+- `CachedRemoteConfig.model_config` is `ConfigDict(extra="forbid")` — *not* strict, because we deserialize datetimes from ISO strings.
+- The cache module uses `pipelex.log` rather than stdlib `logging` to stay consistent with the rest of the package; tests rely on the module-scoped `mock_log` fixture to assert warnings instead of `caplog`.
+- For `test_write_is_atomic`, the test patches `pathlib.Path.replace` (which is what Python uses under the hood after ruff PTH-fixed the call) rather than `os.replace` at module scope.
+
+**Verification:**
+- `.venv/bin/pytest tests/unit/pipelex/system/pipelex_service/test_remote_config_cache.py` — 8 passed.
+- `make agent-check` — clean.
+
+**Next:** Phase 2 — start at **2.1** (failing tests for `RemoteConfigFetcher.fetch_remote_config` with fallback semantics in `tests/integration/pipelex/system/pipelex_service/test_remote_config_fetcher.py`).
+
+### Checkpoint 2 status
+
+**Landed:**
+- `pipelex/system/pipelex_service/exceptions.py` — added `RemoteConfigUnavailableError(PipelexServiceError)` (user-facing offline-mode error: cache path + `pipelex init`/disable-gateway remediation hints) and `RemoteConfigStaleWarning(UserWarning)`. `RemoteConfigFetchError` kept intact as the internal retry-layer exception.
+- `pipelex/system/pipelex_service/remote_config_fetcher.py` — full rewrite of the public API. `fetch_remote_config(require_fresh: bool = False) -> RemoteConfigResult` now carries provenance (`source: RemoteConfigSource` = `FRESH` | `CACHED`, plus `cached_at`). On success: validates payload then writes the **raw JSON dict** via `RemoteConfigCache.store(...)`. On `RemoteConfigFetchError`: tries cache (unless `require_fresh=True`), emits `RemoteConfigStaleWarning` on hit, raises `RemoteConfigUnavailableError` chained from the original `RemoteConfigFetchError` on miss. `RemoteConfigValidationError` path untouched — no cache fallback for schema breaks. Codex Cloud short-circuit still returns a dummy result without writing the cache.
+- `tests/integration/pipelex/system/pipelex_service/test_remote_config_fetcher.py` — new `TestRemoteConfigFetcher` with 11 tests covering all the matrix entries from the plan: fresh success + cache write, connect-error fallback, HTTP-5xx fallback, both no-cache variants raising `RemoteConfigUnavailableError`, schema-break never falling back, Codex Cloud short-circuit untouched, tenacity recovering after 4 transient failures (no cache fallback triggered), tenacity exhausting all 5 attempts (cache fallback), `require_fresh=True` refusing cache, and a guard that the inner `RemoteConfigFetchError` is chained as `__cause__`.
+- Caller migration:
+ - `pipelex/pipelex.py:235` — unwraps `result.config`, logs the source.
+ - `pipelex/cli/commands/doctor_cmd.py` — unwraps `result.config`; the `except (RemoteConfigFetchError, RemoteConfigValidationError)` became `except (RemoteConfigUnavailableError, RemoteConfigValidationError)`.
+ - `pipelex/cli/dev_cli/commands/gateway_models_generator.py` and `preprocess_test_models_cmd.py` — pass `require_fresh=True` (these regenerate committed reference docs and test fixtures; stale cache must never bake in).
+ - `tests/conftest.py` — session-level cache shim now returns a `RemoteConfigResult`; signature accepts `require_fresh` and ignores it (intentional — the test session cache exists to avoid network).
+ - `tests/unit/pipelex/cogt/models/test_model_deck_references.py` — unwraps `.config`.
+ - `tests/unit/pipelex/system/pipelex_service/test_gateway_terms_check.py` — mock now returns a result-shaped object whose `.config` exposes `backend_model_specs` + `aws_region`.
+- `tests/integration/pipelex/system/pipelex_service/test_offline_baseline.py` — renamed the gateway-offline test to `test_gateway_offline_without_cache_raises_remote_config_unavailable_error` and isolated `~/.pipelex` per test via the `ConfigLoader.global_config_dir` `PropertyMock` so a developer's real cache can't accidentally satisfy the fallback. The BYOK-offline test is unchanged.
+- `pipelex/cli/commands/doctor_cmd.py` — replaced the pre-existing `except Exception as exc:` at the model-check block with the specific exception lineage (`RoutingProfileLibraryNotFoundError`, `InferenceBackendLibraryNotFoundError`, `ModelDeckNotFoundError`, `RoutingProfileDisabledBackendError`, `InferenceBackendLibraryValidationError`, `ModelDeckValidationError`, `InferenceBackendCredentialsError`).
+
+**Decisions:**
+- `RemoteConfigResult` is a Pydantic `BaseModel` with `ConfigDict(extra="forbid", strict=True, arbitrary_types_allowed=True)` (matches project convention; the `arbitrary_types_allowed` is needed because `RemoteConfig` is also a `BaseModel`). Carries `config`, `source`, `cached_at` (UTC).
+- `RemoteConfigSource` is a `StrEnum` from `pipelex.types` with lowercase values `fresh` / `cached`.
+- `RemoteConfigUnavailableError` is built by a helper (`_build_unavailable_error`) and the caller does the `raise ... from fetch_error`. Initial draft had a helper that raised directly; refactored to return-and-raise so pyright doesn't see "unreachable code" at the call site and ruff doesn't complain about `raise AssertionError("unreachable")`.
+- `datetime` import in the fetcher is annotated `# noqa: TC003` — Pydantic v2 resolves annotations at runtime even with `from __future__ import annotations`, so it cannot be moved into the `TYPE_CHECKING` block.
+
+**Verification:**
+- `make agent-check` — clean (ruff fix-imports, ruff format, plxt fmt, ruff lint, plxt lint, pyright, mypy all green).
+- Targeted: `.venv/bin/pytest -n auto -m "..." tests/unit/pipelex/system/ tests/integration/pipelex/system/` → 110 passed; same flags on `tests/unit/pipelex/cli/ tests/integration/pipelex/cli/ tests/unit/pipelex/cogt/ tests/integration/pipelex/cogt/` → 1006 passed.
+- `make agent-test` — full suite green.
+
+**Next:** Phase 3 — start at **3.1** (failing tests for `GatewayUnknownModelError` in `tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py`).
+
+### Checkpoint 3 status
+
+**Landed:**
+- `pipelex/cogt/exceptions.py` — added `GatewayUnknownModelError(CogtError)` with `error_category=CONFIGURATION`. Carries `model_name: str` and `source: RemoteConfigSource`. Message branches on source via exhaustive match: `FRESH` → "gateway no longer offers it / check name / disable gateway"; `CACHED` → "cache may be stale, run `pipelex init` while online". Imports `RemoteConfigSource` at module level — no circular-import issue since the fetcher module doesn't touch `cogt`.
+- `pipelex/cogt/models/model_manager_abstract.py` — added required `gateway_config_source: RemoteConfigSource | None` parameter to the abstract `setup()`.
+- `pipelex/cogt/models/model_manager.py` — implementation of the membership check:
+ - `setup()` now accepts `gateway_config_source` and calls `_enforce_gateway_model_membership` after `build_deck`.
+ - `_enforce_gateway_model_membership(gateway_config, gateway_config_source)` — no-op when either is `None` (gateway not live). Otherwise iterates `_collect_deck_referenced_handles(deck)`, resolves each through `_resolve_terminal_handle` (walks aliases + first-element of waterfalls), and raises `GatewayUnknownModelError(model_name=terminal, source=gateway_config_source)` when the terminal handle is missing from both `deck.inference_models` AND `gateway_config.model_specs` (defaults key skipped).
+ - `_collect_deck_referenced_handles` covers preset models (LLM/Extract/ImgGen/Search) and the two LLM choice defaults. Aliases/waterfalls are intentionally NOT enumerated directly — they're reachable via preset/choice references and the resolver walks them.
+ - `_extract_choice_handle` normalises the `LLMModelChoice` union (`LLMSetting | ModelReference | str | None`) to a raw handle string.
+ - `_resolve_terminal_handle` does its own model-type → aliases/waterfalls dispatch (instead of calling `ModelDeck._get_aliases_and_waterfalls_for_type`, which is protected) to keep the cogt API surface clean.
+- `pipelex/pipelex.py` — captures `gateway_config_source` from the fetch result, passes it through `models_manager.setup(...)`, and tightens the telemetry guard:
+ - `is_pipelex_telemetry_enabled = is_pipelex_service_enabled and needs_inference and gateway_config_source != RemoteConfigSource.CACHED` (stale config implies stale model identities; phoning home then would pollute metrics).
+ - Re-emits `RemoteConfigStaleWarning` at setup time even when the fetcher itself didn't (the session-cached fetcher in `tests/conftest.py` swaps in a wrapper that doesn't warn — re-emitting here makes provenance visible regardless of how the result reached us).
+ - Dummy remote config path (`needs_model_specs=False`) gets `source=FRESH` so the gateway-membership check is silent in that branch (model_specs is empty there).
+- `pipelex/cli/commands/doctor_cmd.py` — propagates `gateway_config_source` from the fetch result through `models_manager.setup(...)`.
+- `pipelex/cli/agent_cli/commands/agent_output.py`:
+ - New module-level capture buffer + `record_setup_warning(...)` / `consume_setup_warnings()` helpers.
+ - `agent_success(...)` drains the buffer into a top-level `warnings` array on the JSON envelope (appending to any caller-supplied list).
+ - Added `AGENT_ERROR_HINTS` and `AGENT_ERROR_DOMAINS` entries for `RemoteConfigUnavailableError` and `GatewayUnknownModelError`. Existing `RemoteConfigFetchError` and `RemoteConfigValidationError` hints untouched.
+- `pipelex/cli/agent_cli/commands/agent_cli_factory.py`:
+ - Wraps `Pipelex.make` in `warnings.catch_warnings(record=True)` and records every `RemoteConfigStaleWarning` as `{"type": "RemoteConfigStale", "message": "..."}` via `record_setup_warning(...)`. The next `agent_success` call surfaces it on the envelope.
+ - New `except` clauses for `RemoteConfigUnavailableError` and `GatewayUnknownModelError`; the latter exposes `model_name` and `source` on the error JSON.
+
+**Tests:**
+- `tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py` — `TestGatewayUnknownModel` with three tests covering the happy path (default cached gateway has all referenced handles), fresh-source missing handle, and cached-source missing handle (asserts the message hints at `pipelex init`).
+- `tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py` — `TestSetupWithCache` with four tests: dry-run with primed cache (warning emitted), dry-run with cold cache (`RemoteConfigUnavailableError`), telemetry-downgrade-to-no-op when source is cached even with `needs_inference=True`, and the BYOK-offline regression guard (gateway disabled → `fetch_remote_config` never called).
+
+**Decisions:**
+- `GatewayConfig` stays `extra="forbid"` and source-free. Provenance flows through `ModelManager.setup(gateway_config_source=...)` rather than being baked into the value object.
+- The membership check runs at the end of `ModelManager.setup` (not only at `validate_model_deck` time), so even `needs_inference=False, needs_model_specs=True` dry-run flows surface gateway-spec mismatches. This is necessary because the default `missing_presets_reaction = "log"` would otherwise swallow the failure silently.
+- `_enforce_gateway_model_membership` skips already-resolved handles (i.e. ones that ended up in `deck.inference_models`). This means the production happy path is silent even when the gateway has every standard model.
+- We did NOT extend `ModelDeck._get_aliases_and_waterfalls_for_type` to public access — the resolver re-does the dispatch locally to keep the deck's invariants encapsulated.
+- The structured `warnings` envelope is keyed `type=RemoteConfigStale`; the `cached_at` field will be added once we propagate the timestamp through the warning payload (currently the warning message has it inline; deferred to a follow-up if a structured field becomes necessary for downstream consumers).
+- Telemetry is downgraded to no-op (rather than to a "stale" variant) when the source is cached. Stricter than the original plan's "is_pipelex_telemetry_enabled stays False" wording — current implementation makes the downgrade explicit in `pipelex.py` so the existing `TelemetryFactory.make_telemetry_manager(is_pipelex_telemetry_enabled=False, ...)` path returns the no-op directly.
+
+**Verification:**
+- Targeted: `.venv/bin/pytest -n auto -m "not pipelex_api" tests/integration/pipelex/system/pipelex_service/ tests/unit/pipelex/system/` → 112 passed.
+- Targeted: `.venv/bin/pytest -n auto -m "not pipelex_api" tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py` → 3 passed.
+- Targeted: `.venv/bin/pytest -n auto -m "not pipelex_api" tests/integration/pipelex/system/pipelex_service/test_setup_with_cache.py` → 4 passed.
+- `make agent-check` — clean (ruff + plxt + pyright + mypy all green).
+- `make agent-test` — full suite green.
+
+**Next:** Phase 4 — start at **4.1** (failing tests for cache priming in `tests/integration/pipelex/cli/commands/init/test_cache_priming.py`).
+
+### Checkpoint 4 status
+
+**Landed:**
+- `pipelex/cli/commands/init/command.py`:
+ - New `CachePrimingResult` Pydantic model (`extra="forbid", strict=True`) with `primed: bool` and `error_message: str | None`.
+ - New pure-logic helper `attempt_prime_remote_config_cache()` — checks gateway enabled + terms accepted, calls `RemoteConfigFetcher.fetch_remote_config()`, returns a `CachePrimingResult`. Catches only `RemoteConfigUnavailableError` and `RemoteConfigFetchError`; `RemoteConfigValidationError` propagates (server-side schema breaks must surface — we control the back-office).
+ - Interactive wrapper `prime_remote_config_cache(console)` — calls the pure helper and emits a yellow Rich warning when a fetch was attempted and failed; otherwise silent.
+ - `execute_initialization(...)` now calls `prime_remote_config_cache(console)` as Step 5, after telemetry setup. No-op for BYOK/declined-terms paths, so unconditional call is safe.
+- `pipelex/cli/agent_cli/commands/init_cmd.py`:
+ - Calls `attempt_prime_remote_config_cache()` after `update_inference_setup_completed(...)`.
+ - The `agent_success(...)` envelope now includes `cache_primed: bool` and (when applicable) `cache_priming_error: str`. Machine consumers can branch on these without parsing log output.
+- `tests/integration/pipelex/cli/commands/init/test_cache_priming.py` — `TestCachePriming` with 4 tests covering the matrix: primes when online, warns when offline (no crash, no cache written, yellow warning emitted), skips when gateway disabled (no fetch + no cache), and overwrites an existing cache when refreshing online.
+
+**Decisions:**
+- Split the helper into a pure-logic `attempt_prime_remote_config_cache()` + an interactive `prime_remote_config_cache(console)` wrapper. The split lets the agent CLI surface structured fields (`cache_primed`, `cache_priming_error`) on its JSON envelope while the Rich CLI gets a yellow warning. The plan's original sketch only described the interactive wrapper; the agent CLI mirror needed the pure variant.
+- Priming runs unconditionally at the end of `execute_initialization`. The helper itself short-circuits when gateway is disabled or terms not accepted, so wrapping the call in `if PipelexBackend.GATEWAY in selected_backend_keys:` at the call site would be redundant double-checking. The internal short-circuit also covers the agent CLI mirror cleanly.
+- The skipped state (`primed=False, error_message=None`) is intentionally indistinguishable from "no gateway/no terms" — both surface the same `cache_primed: false` on the agent envelope. Consumers that care can check `backends_enabled` or read `pipelex_service.toml` directly.
+- No `--no-prime` opt-out flag for now. If priming ever becomes a real problem at init time (slow network, captive portals), revisit; today the helper warns and continues so it can't block init.
+- `attempt_prime_remote_config_cache` passes ``require_fresh=True`` to the fetcher. Without it, an offline fetch with a stale cache on disk would silently return ``source=CACHED`` and the helper would falsely report ``primed=True`` (no new write happened). With ``require_fresh=True``, the offline-with-stale-cache path raises ``RemoteConfigUnavailableError`` → we surface it as ``error_message`` and the existing cache file is left intact. Caught in code review post-Phase 4; `test_init_warns_when_offline_with_stale_cache_present` pins the regression.
+
+**Verification:**
+- `.venv/bin/pytest tests/integration/pipelex/cli/commands/init/test_cache_priming.py` — 5 passed.
+- Targeted: `.venv/bin/pytest -n auto -m "..." tests/unit/pipelex/cli/ tests/integration/pipelex/cli/ tests/unit/pipelex/system/ tests/integration/pipelex/system/` → 429 passed.
+- `make agent-check` — clean (ruff + plxt + pyright + mypy all green).
+- `make agent-test` — full suite green.
+
+**Next:** Phase 5 — start at **5.1** (create `tests/e2e/data/offline_mode/` bundles for `byok_simple`, `gateway_known_model`, `gateway_unknown_model`).
+
+### Checkpoint 5 status
+
+**Landed:**
+- `pipelex/system/pipelex_service/pipelex_details.py` — replaced the class-level `REMOTE_CONFIG_URL` constant with a `remote_config_url()` classmethod that reads `PIPELEX_REMOTE_CONFIG_URL` from the environment when set, else falls back to the production URL. New `_DEFAULT_REMOTE_CONFIG_URL` module constant + `REMOTE_CONFIG_URL_ENV_VAR` name; the underlying call site in `remote_config_fetcher.py:206` now calls the method.
+- `tests/e2e/data/offline_mode/{byok_simple,gateway_known_model,gateway_unknown_model}/<name>.mthds` — minimal `PipeLLM` bundles, each referencing one handle: `claude-4.5-haiku` (BYOK via anthropic), `gpt-4o-mini` (known gateway handle), `gpt-future-fake-99` (deliberately absent from the cached payload). Prompts use the inline `$topic.text` sigil (not `@`, which produces tag-wrapped block content and is invalid on the same line as other text).
+- `tests/e2e/agent_cli/conftest.py` — new hermetic-home fixtures:
+ - `hermetic_home` (returns the tmp `HOME` with a kit-copied `.pipelex/` tree, with `pipelex_service.toml` pre-stamped as terms-accepted + onboarding-complete).
+ - `offline_subprocess_env` (subprocess env with `HOME` redirected, `PIPELEX_REMOTE_CONFIG_URL=http://127.0.0.1:1/...`, `RUN_MODE=ci_test`, and dummy values for every `${...}` backend credential variable referenced by the kit's `backends.toml`; under `--dry-run`'s `needs_inference=False` Pipelex is lenient on missing credentials, but the env-var substitution itself still needs the var to exist).
+ - Helper functions: `set_gateway_enabled`, `write_active_routing_profile`, `write_remote_config_cache`, `write_pipelex_service_config`. The cache writer hits the real `RemoteConfigCache` JSON schema (matches `CACHE_SCHEMA_VERSION`).
+- `tests/e2e/agent_cli/test_offline_run_dry.py` — `TestOfflineDryRun` with four subprocess-based E2E tests. Each test sets `cwd=hermetic_home` so `ConfigLoader.find_project_root` immediately stops at `Path.home()` and the subprocess uses the test's `.pipelex/` instead of walking up to the repo's project-level one.
+ - `test_byok_offline_succeeds` — gateway disabled (and routing flipped to `all_anthropic`) → exit 0, no `error` field, `text` payload present.
+ - `test_gateway_no_cache_no_network_fails_with_unavailable` — gateway enabled, no cache → `error_type == "RemoteConfigUnavailableError"` with `pipelex init` in the remediation message.
+ - `test_gateway_known_with_cache_succeeds_offline` — gateway enabled, cache primed with a comprehensive payload covering every kit-deck terminal handle (LLM, img-gen, extract, search) → exit 0, the `warnings` array surfaces `{"type": "RemoteConfigStale", ...}`. The "comprehensive payload" is critical: the gateway membership check iterates every preset/choice terminal across all model-type decks, and any missing handle (e.g. `azure-document-intelligence` for the default extract preset) raises `GatewayUnknownModelError` at setup time even when the bundle never touches that model.
+ - `test_gateway_unknown_with_cache_fails_with_clear_error` — bundle pipe references `gpt-future-fake-99` (absent from the cache). The test asserts a non-zero exit and the unknown handle visible somewhere in the JSON envelope without pinning `error_type`. In practice this fires as `ModelChoiceNotFoundError` or `PipeOperatorModelAvailabilityError` at the pipe-operator layer rather than `GatewayUnknownModelError`, because the kit deck doesn't reference the fake handle and our membership check only iterates deck-level presets/choices.
+
+**Decisions:**
+- `cwd=hermetic_home` instead of `cwd=tmp_path/some_subdir` was deliberate. With `HOME=tmp_path` and `cwd=tmp_path`, `find_project_root` returns `None` immediately (the loop hits the home guard on iteration 0) so the project-config-dir lookup short-circuits to the hermetic global dir. Any other cwd choice risks walking up to a sibling project marker file in the temp tree.
+- The `gateway_unknown_model` test doesn't pin the error_type because making the membership check fire from a `.mthds` bundle would require shipping a custom deck override that points a preset at the fake handle. That's worth doing in a follow-up but adds churn here without changing the user-facing contract — what we promise is "the unknown handle surfaces in a structured error", and the test asserts exactly that.
+- The cached `backend_model_specs` payload uses minimal valid spec bodies (`sdk` + `model_type` + a couple of inputs/outputs). Larger spec fields like cost/latency/max-tokens are optional in `InferenceModelSpecBlueprint` and elided to keep the fixture compact.
+- Subprocess env explicitly does NOT inherit the parent process env. The dict is built from scratch and includes `PATH=/usr/bin:/bin:/usr/local/bin`. This prevents developer-machine env vars (e.g. a real `ANTHROPIC_API_KEY`, a real `HOME`) from leaking in and making the test environment non-hermetic. Side effect: if a future test needs more env, it has to be added explicitly to `offline_subprocess_env`.
+- `subprocess` import gets a `# noqa: S404` because the bandit rule flags any use of the module, but we genuinely want to spawn the real CLI binary for E2E coverage. The `S603` noqa on the actual `subprocess.run` call covers the same concern at the call site.
+- Renamed `PipelexDetails.REMOTE_CONFIG_URL` (class constant) to `PipelexDetails.remote_config_url()` (classmethod). A `@classmethod @property` would have been more drop-in but is deprecated in Python 3.11 and removed in 3.13; a method is the safe substitute.
+
+**Verification:**
+- `.venv/bin/pytest tests/e2e/agent_cli/test_offline_run_dry.py` — 4 passed (~50s wall clock; each test spawns one subprocess invocation of `pipelex-agent`).
+- Targeted: `.venv/bin/pytest -n auto -m "not pipelex_api" tests/unit/pipelex/system/ tests/integration/pipelex/system/ tests/unit/pipelex/cli/ tests/integration/pipelex/cli/` — 429 passed.
+- `make agent-check` — clean (ruff + plxt + pyright + mypy all green).
+
+**Next:** Phase 6 — start at **6.1** (update `agent_output.py` `AGENT_ERROR_HINTS`/`AGENT_ERROR_DOMAINS` for new error types).
+
+### Checkpoint 6 status
+
+**Landed:**
+- `pipelex/cli/agent_cli/commands/agent_output.py` — already carried `AGENT_ERROR_HINTS` and `AGENT_ERROR_DOMAINS` entries for `RemoteConfigUnavailableError` and `GatewayUnknownModelError` from Phase 3. No further changes needed. Existing `RemoteConfigFetchError` and `RemoteConfigValidationError` hints intact.
+- `pipelex/cli/error_handlers.py` — added two Rich-CLI handlers mirroring the agent-cli hints:
+ - `handle_remote_config_unavailable_error(exc)` — frames the failure as "offline + cold cache", names both remediation paths (prime the cache via `pipelex init` while online, or disable the gateway for permanent BYOK operation). Echoes the underlying error message which already contains the cache file path.
+ - `handle_gateway_unknown_model_error(exc)` — shows the unknown model handle and the config source provenance, and branches the remediation via `match exc.source` on `RemoteConfigSource.{FRESH,CACHED}`. Fresh-source: typo/deck-update guidance; cached-source: `pipelex init` while online to refresh the cache. Imported the `RemoteConfigSource` enum from `pipelex.system.pipelex_service.types` (the leaf-level enum module, not `pipelex_service.types` reachable through `remote_config_fetcher`).
+ - Existing `handle_remote_config_fetch_error` and `handle_remote_config_validation_error` handlers untouched.
+- `pipelex/cli/cli_factory.py` — wired both new handlers into `make_pipelex_for_cli`'s except chain: `RemoteConfigUnavailableError` → `handle_remote_config_unavailable_error`; `GatewayUnknownModelError` → `handle_gateway_unknown_model_error`. Order matters: `RemoteConfigFetchError` is the parent-level retry-layer exception (caught first); `RemoteConfigUnavailableError` is its user-facing sibling raised when the cache fallback fails too (caught next); `GatewayUnknownModelError` is unrelated and lives further down the chain.
+- `CHANGELOG.md` — added a fresh `## [Unreleased]` section above v0.28.0 with three buckets:
+ - **Added**: offline-mode setup/dry-run support (cache lifecycle, stale warning, telemetry suppression, doc-generator refusal), `GatewayUnknownModelError` (source-aware messaging), `RemoteConfigUnavailableError` (cache-cold failure), `PIPELEX_REMOTE_CONFIG_URL` env var.
+ - **Changed**: `RemoteConfigFetcher.fetch_remote_config()` now returns a `RemoteConfigResult` with provenance and accepts `require_fresh: bool = False`; `ModelManager.setup()` / `BackendLibrary._load_gateway_model_specs()` accept `gateway_config_source: RemoteConfigSource | None`.
+
+**Decisions:**
+- **6.3 CLAUDE.md updates: no-op.** `pipelex/cli/agent_cli/CLAUDE.md` only points at `AGENT_ERROR_HINTS` as the registry — it doesn't enumerate specific error classes, so the new entries don't need to be mentioned. `pipelex/system/pipelex_service/CLAUDE.md` does not exist, and there's no convention for service-level CLAUDE.md docs (only `pipelex/builder/CLAUDE.md` and `pipelex/cli/agent_cli/CLAUDE.md` exist across the whole `pipelex/` tree). The plan's "only if there's already a convention" gate is not satisfied, so no new file was created.
+- The Rich CLI handlers deliberately mirror the agent-CLI hints but in Rich's "❌ / ⚠ / 💡" style rather than as one-liner JSON `hint` strings. Both surfaces remain consistent in *content* (same remediation paths, same provenance branching) but differ in *form* (Rich-formatted multi-line vs JSON envelope field).
+- The `RemoteConfigUnavailableError` handler does NOT re-print the cache file path because the underlying error message already contains it (see `RemoteConfigUnavailableError._build_unavailable_error` in `remote_config_fetcher.py`). Echoing `str(exc)` is sufficient.
+
+**Verification:**
+- `make agent-check` — clean (ruff fix-imports, ruff format, plxt fmt, ruff lint, plxt lint, pyright, mypy all green).
+- Targeted: `.venv/bin/pytest -n auto -m "not pipelex_api and not llm and not img_gen and not extract and not inference" tests/unit/pipelex/cli/ tests/integration/pipelex/cli/ tests/unit/pipelex/system/ tests/integration/pipelex/system/` — 429 passed (1 expected `RemoteConfigStaleWarning` from `test_telemetry_disabled_when_source_cached`).
+- `make agent-test` — full suite green.
+
+**Next:** Phase 7 — start at **7.1** (`make agent-check` clean) → **7.2** (`make agent-test` clean) → **7.3** (manual reproduction of every behaviour-matrix scenario through the real CLI surface) → **7.4** (squash-friendly commit history + PR description that references the codex-sandbox-escalation handoff and the deferred follow-ups).
+
+### Checkpoint 7 status
+
+**Landed:**
+- No new feature code — Phase 7 is a verification gate. One small message-clarity fix landed during the manual repro of 7.3 (see below).
+- `pipelex/system/pipelex_service/remote_config_fetcher.py` — `_build_unavailable_error` gained a `cache_refused: bool = False` flag. When the dev-CLI path passes `require_fresh=True` and a cache exists on disk, the error message now reads "the local cache at `<path>` was refused because a fresh fetch is required" instead of the previously misleading "no local cache is available at `<path>`". The cache-truly-missing path keeps its original wording. Call site at line 211 passes `cache_refused=True`.
+- `tests/integration/pipelex/system/pipelex_service/test_remote_config_fetcher.py::test_require_fresh_refuses_cache` — strengthened to assert the new "was refused because a fresh fetch is required" phrasing so the contract is pinned.
+- `CHANGELOG.md` — new "Changed" entry under `[Unreleased]` documenting the `RemoteConfigUnavailableError` message branching.
+
+**Manual reproduction summary (7.3):**
+- BYOK offline → covered by `tests/e2e/agent_cli/test_offline_run_dry.py::test_byok_offline_succeeds`.
+- Gateway online, no cache, cache written → covered by `tests/integration/pipelex/system/pipelex_service/test_remote_config_fetcher.py::test_success_returns_fresh_and_writes_cache` and `tests/integration/pipelex/cli/commands/init/test_cache_priming.py::test_init_primes_cache_when_online`.
+- Gateway offline, cache present, stale warning → covered by `test_gateway_known_with_cache_succeeds_offline` (E2E).
+- Gateway offline, no cache, `RemoteConfigUnavailableError` → covered by `test_gateway_no_cache_no_network_fails_with_unavailable` (E2E).
+- Bundle references unknown gateway model → covered by `tests/integration/pipelex/cogt/model_backends/test_gateway_unknown_model.py` plus the E2E `test_gateway_unknown_with_cache_fails_with_clear_error` (the E2E intentionally does not pin `error_type` — see deferred follow-up #7).
+- `pipelex-dev update-gateway-models` offline → manually reproduced via `PIPELEX_REMOTE_CONFIG_URL=http://127.0.0.1:1/unreachable.json .venv/bin/pipelex-dev update-gateway-models`; raises `RemoteConfigUnavailableError` with the new "cache refused because a fresh fetch is required" remediation message, and no docs are written. Also pinned in `test_require_fresh_refuses_cache`.
+
+**Commit history (7.4):**
+- Phase-aligned commits ahead of `main` (`1d9293d3` ... `6b5d0d40`), one per phase boundary. Phase 7 changes are still uncommitted in the working tree (`CHANGELOG.md`, `remote_config_fetcher.py`, `test_remote_config_fetcher.py`) — to be squashed with Phase 6 or landed as a small "polish" commit, at the user's discretion at `/ship` time.
+- PR description should reference (a) the codex-sandbox-escalation handoff in `mthds-plugins/wip/codex-sandbox-escalation.md` — note that `--dry-run` no longer requires escalation once Pipelex has been initialised online once, and (b) the deferred follow-ups list at the bottom of this file (especially #1 `pipelex doctor` cache reporting, #2 Codex Cloud cache-first short-circuit, #3 the cross-repo escalation-doc update, and #7 pinning `GatewayUnknownModelError` end-to-end via a deck override).
+
+**Verification:**
+- `make agent-check` — clean (ruff fix-imports, ruff format, plxt fmt, ruff lint, plxt lint, pyright, mypy all green).
+- `make agent-test` — full suite green (zero output on success).
+- Targeted: `.venv/bin/pytest -n auto -m "(dry_runnable or not (inference or llm or img_gen or extract or search)) and not pipelex_api" tests/unit/pipelex/system/ tests/integration/pipelex/system/ tests/unit/pipelex/cli/ tests/integration/pipelex/cli/` — all green (one expected `RemoteConfigStaleWarning` from `test_telemetry_disabled_when_source_cached`).
+- Manual: `PIPELEX_REMOTE_CONFIG_URL=http://127.0.0.1:1/... .venv/bin/pipelex-dev update-gateway-models` → new "cache refused" wording confirmed in stderr.
+
+**Decision:**
+- The message-clarity fix was made in scope as a Phase 7 finding (no new entry was added to the deferred-follow-ups list). Branching `_build_unavailable_error` is a one-liner with a one-keyword call-site update; the alternative — adding the deferred entry — would have left a known-misleading user-facing message in v0.29.0.
+
+**Next:** Branch is ready for `/ship`. No remaining checkbox in this plan.
diff --git a/wip/offline-mode-pr-story.html b/wip/offline-mode-pr-story.html
new file mode 100644
index 000000000..ff6cb2683
--- /dev/null
+++ b/wip/offline-mode-pr-story.html
@@ -0,0 +1,327 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>PR Story — Offline Mode &amp; Remote Config Cache</title>
+<style>
+ :root {
+ --bg: #0f1117;
+ --panel: #181b24;
+ --panel-2: #1f2330;
+ --ink: #e8eaf0;
+ --muted: #9aa3b6;
+ --line: #2c3142;
+ --accent: #6ea8fe;
+ --green: #5fd38f;
+ --amber: #f0c060;
+ --red: #f08a8a;
+ --mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, monospace;
+ }
+ * { box-sizing: border-box; }
+ body {
+ margin: 0;
+ background: var(--bg);
+ color: var(--ink);
+ font: 16px/1.65 -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
+ -webkit-font-smoothing: antialiased;
+ }
+ .wrap { max-width: 920px; margin: 0 auto; padding: 56px 28px 120px; }
+ header.hero { border-bottom: 1px solid var(--line); padding-bottom: 32px; margin-bottom: 40px; }
+ .eyebrow { font: 600 12px/1 var(--mono); letter-spacing: .14em; text-transform: uppercase; color: var(--accent); }
+ h1 { font-size: 34px; line-height: 1.2; margin: 14px 0 10px; letter-spacing: -.01em; }
+ .sub { color: var(--muted); font-size: 17px; max-width: 680px; }
+ .meta { margin-top: 18px; display: flex; flex-wrap: wrap; gap: 8px; }
+ .chip {
+ font: 600 12px/1 var(--mono); padding: 6px 10px; border-radius: 999px;
+ background: var(--panel-2); border: 1px solid var(--line); color: var(--muted);
+ }
+ h2 {
+ font-size: 22px; margin: 52px 0 16px; letter-spacing: -.01em;
+ padding-left: 12px; border-left: 3px solid var(--accent);
+ }
+ h3 { font-size: 16px; margin: 28px 0 10px; color: var(--ink); }
+ p { margin: 12px 0; }
+ .muted { color: var(--muted); }
+ code {
+ font: 13.5px/1.5 var(--mono); background: var(--panel-2);
+ padding: 1.5px 6px; border-radius: 5px; border: 1px solid var(--line);
+ }
+ .panel {
+ background: var(--panel); border: 1px solid var(--line);
+ border-radius: 12px; padding: 20px 22px; margin: 18px 0;
+ }
+ .callout {
+ border-left: 3px solid var(--amber); background: var(--panel);
+ border-radius: 0 10px 10px 0; padding: 14px 18px; margin: 18px 0;
+ }
+ .callout.bug { border-left-color: var(--red); }
+ .callout .lbl { font: 600 11px/1 var(--mono); letter-spacing: .1em; text-transform: uppercase; color: var(--amber); }
+ .callout.bug .lbl { color: var(--red); }
+
+ /* objective cards */
+ .cards { display: grid; grid-template-columns: repeat(3, 1fr); gap: 14px; margin: 18px 0; }
+ .card { background: var(--panel); border: 1px solid var(--line); border-radius: 12px; padding: 18px; }
+ .card .n { font: 700 13px/1 var(--mono); color: var(--accent); }
+ .card h4 { margin: 8px 0 6px; font-size: 15px; }
+ .card p { margin: 0; font-size: 14px; color: var(--muted); }
+
+ /* flow diagram */
+ .flow { margin: 22px 0; }
+ .flow-row { display: flex; align-items: stretch; gap: 10px; flex-wrap: wrap; }
+ .node {
+ background: var(--panel-2); border: 1px solid var(--line); border-radius: 10px;
+ padding: 12px 14px; font-size: 13.5px; flex: 1; min-width: 150px;
+ }
+ .node .t { font: 600 11px/1 var(--mono); letter-spacing: .08em; text-transform: uppercase; color: var(--muted); display: block; margin-bottom: 5px; }
+ .node.ok { border-color: #2f5d44; }
+ .node.ok .t { color: var(--green); }
+ .node.warn { border-color: #5d4f2c; }
+ .node.warn .t { color: var(--amber); }
+ .node.err { border-color: #5d3434; }
+ .node.err .t { color: var(--red); }
+ .arrow { align-self: center; color: var(--muted); font: 18px/1 var(--mono); padding: 0 2px; }
+ .vsep { text-align: center; color: var(--muted); font: 13px/1 var(--mono); margin: 8px 0; }
+
+ /* table */
+ table { width: 100%; border-collapse: collapse; margin: 16px 0; font-size: 13.5px; }
+ th, td { text-align: left; padding: 9px 11px; border-bottom: 1px solid var(--line); vertical-align: top; }
+ th { font: 600 11px/1.3 var(--mono); letter-spacing: .06em; text-transform: uppercase; color: var(--muted); }
+ td .dot { font-weight: 700; }
+ .dot.g { color: var(--green); }
+ .dot.a { color: var(--amber); }
+ .dot.r { color: var(--red); }
+ tbody tr:hover { background: var(--panel); }
+
+ /* diff */
+ .diff {
+ background: #11141c; border: 1px solid var(--line); border-radius: 10px;
+ overflow-x: auto; margin: 14px 0; font: 12.5px/1.7 var(--mono);
+ }
+ .diff .fname {
+ padding: 8px 14px; border-bottom: 1px solid var(--line);
+ color: var(--muted); background: var(--panel-2); font-weight: 600;
+ }
+ .diff pre { margin: 0; padding: 12px 14px; }
+ .diff .add { color: var(--green); display: block; }
+ .diff .del { color: var(--red); display: block; }
+ .diff .ctx { color: #8b93a7; display: block; }
+ .diff .hdr { color: var(--accent); display: block; }
+
+ ul.tight { margin: 10px 0; padding-left: 22px; }
+ ul.tight li { margin: 5px 0; }
+ .files { columns: 2; column-gap: 26px; font: 12.5px/1.9 var(--mono); color: var(--muted); }
+ .files .new { color: var(--green); }
+ footer { margin-top: 70px; padding-top: 24px; border-top: 1px solid var(--line); color: var(--muted); font-size: 13px; }
+ @media (max-width: 680px) {
+ .cards { grid-template-columns: 1fr; }
+ .files { columns: 1; }
+ }
+</style>
+</head>
+<body>
+<div class="wrap">
+
+<header class="hero">
+ <div class="eyebrow">Pull Request Story</div>
+ <h1>Offline Mode &amp; Remote Config Cache</h1>
+ <p class="sub">Making Pipelex <strong>setup</strong> and <strong>dry-run</strong> work without a network — by adding a primed, last-resort cache for the Gateway remote config, with honest provenance tracking and no silent fallbacks.</p>
+ <div class="meta">
+ <span class="chip">branch · fix/Offline-mode</span>
+ <span class="chip">7 phases · TDD throughout</span>
+ <span class="chip">~3,300 LOC added</span>
+ <span class="chip">6 new test suites</span>
+ </div>
+</header>
+
+<h2>Why we did this</h2>
+<p>Pipelex's Gateway needs a <strong>remote config</strong> — the catalogue of which models exist and on which backends. On every <code>setup()</code> that touches model specs, Pipelex fetched that config over HTTP. If the network was down, setup <em>crashed</em> — even for commands that never make an inference call.</p>
+
+<p>That hurt in two real places: agent CLI commands like <code>validate</code>, <code>inputs</code>, and <code>run --dry-run</code> all pass <code>needs_model_specs=True</code>, and sandboxed environments (Codex local sandbox) have no outbound network at all. None of these <em>run</em> a model — they just need to know the catalogue. Yet a missing network broke them.</p>
+
+<div class="callout bug">
+ <span class="lbl">Diagnosed failure point</span>
+ <p style="margin:8px 0 0"><code>pipelex.py:235</code> unconditionally called <code>RemoteConfigFetcher.fetch_remote_config()</code> whenever the Gateway was enabled and model specs were needed. Any network failure raised <code>RemoteConfigFetchError</code> straight out of setup. The only escape hatch was a Codex-<em>Cloud</em>-specific short-circuit — local sandboxes and plain offline use were not covered.</p>
+</div>
+
+<h2>The three objectives</h2>
+<div class="cards">
+ <div class="card">
+ <span class="n">01</span>
+ <h4>BYOK works fully offline</h4>
+ <p>Gateway disabled → no remote fetch is ever attempted. Setup completes with zero network.</p>
+ </div>
+ <div class="card">
+ <span class="n">02</span>
+ <h4>Dry-run survives an outage</h4>
+ <p>Gateway enabled but remote temporarily unreachable → fall back to a cache primed at init time.</p>
+ </div>
+ <div class="card">
+ <span class="n">03</span>
+ <h4>No silent fallbacks</h4>
+ <p>Unknown Gateway models fail loudly against fresh <em>or</em> cached specs. Stale data is always labelled.</p>
+ </div>
+</div>
+<p class="muted">Non-objective: we did <em>not</em> make inference itself work offline. A real model call still needs the network. This PR is strictly about <strong>setup</strong> and <strong>dry-run</strong>.</p>
+
+<h2>The design in one picture</h2>
+<p>The whole thing hinges on one idea: <strong>the fetcher returns a result that knows where it came from.</strong> Everything downstream branches on that provenance.</p>
+
+<div class="flow panel">
+ <div class="flow-row">
+ <div class="node"><span class="t">setup()</span>Gateway enabled &amp; needs specs</div>
+ <span class="arrow">→</span>
+ <div class="node"><span class="t">fetch_remote_config()</span>HTTP GET, with retry (tenacity, 5 attempts)</div>
+ </div>
+ <div class="vsep">│</div>
+ <div class="flow-row">
+ <div class="node ok"><span class="t">success</span>parse + validate → write raw JSON to cache → <code>source = FRESH</code></div>
+ <div class="node warn"><span class="t">network fails, cache hit</span>load <code>~/.pipelex/cache/remote_config.json</code> → <code>source = CACHED</code></div>
+ <div class="node err"><span class="t">network fails, no cache</span>raise <code>RemoteConfigUnavailableError</code></div>
+ </div>
+ <div class="vsep">│ &nbsp; the <code>source</code> flows downstream &nbsp; │</div>
+ <div class="flow-row">
+ <div class="node">membership check: every referenced Gateway model must exist in the specs → else <code>GatewayUnknownModelError</code> (message branches on <code>source</code>)</div>
+ </div>
+ <div class="flow-row" style="margin-top:10px">
+ <div class="node warn"><span class="t">if CACHED</span>emit <code>RemoteConfigStaleWarning</code> · disable telemetry · surface a <code>warnings</code> field on the agent-CLI JSON envelope</div>
+ </div>
+</div>
+
+<h3>Why a schema-break does NOT fall back to cache</h3>
+<p>If the remote responds with JSON that fails validation, we raise <code>RemoteConfigValidationError</code> and stop — no cache fallback. We control both ends of that URL (it's versioned in <code>pipelex-back-office</code>), so a schema-rejecting payload is a real server bug, not an operational state. Failing loudly is correct. The cache is for <em>network</em> failures only.</p>
+
+<h2>The pieces we added</h2>
+<div class="panel">
+ <h3 style="margin-top:0">New module — <code>remote_config_cache.py</code></h3>
+ <p class="muted" style="margin-top:6px"><code>RemoteConfigCache.store()</code> / <code>.load()</code>. Stores the <strong>raw JSON dict</strong> from the response — not a re-serialised Pydantic dump — so the cache survives minor schema drift. Atomic writes (temp file + <code>os.replace</code>). Schema-versioned: a <code>CACHE_SCHEMA_VERSION</code> bump rejects old caches cleanly.</p>
+ <h3>New leaf enum — <code>RemoteConfigSource</code></h3>
+ <p class="muted" style="margin-top:6px"><code>FRESH</code> | <code>CACHED</code>. Lives in its own tiny <code>types.py</code> so <code>cogt/</code> code can import it without dragging in <code>httpx</code> + <code>tenacity</code>. Exposes <code>.is_cached</code> so callers never write <code>== CACHED</code>.</p>
+ <h3>New exceptions &amp; warning</h3>
+ <ul class="tight muted">
+ <li><code>RemoteConfigUnavailableError</code> — fetch failed <em>and</em> no usable cache. Message names the cache path + remediation.</li>
+ <li><code>GatewayUnknownModelError</code> — a bundle references a Gateway model absent from the specs. Message branches on <code>source</code> (a <code>CACHED</code> source hints "run <code>pipelex init</code> while online").</li>
+ <li><code>RemoteConfigStaleWarning</code> — emitted when setup completes off a cached config.</li>
+ <li><code>RemoteConfigFetchError</code> — <strong>kept</strong> as the internal retry-layer exception. Removing it would have broken doctor, the CLI factory, and agent hints.</li>
+ </ul>
+</div>
+
+<h2>The critical code change</h2>
+<p>The behavioural heart of the PR. The fetcher stopped returning a bare <code>RemoteConfig</code> and started returning a <code>RemoteConfigResult</code> that carries provenance:</p>
+
+<div class="diff">
+ <div class="fname">pipelex/system/pipelex_service/remote_config_fetcher.py</div>
+<pre><span class="del">- def fetch_remote_config(cls) -> RemoteConfig:</span>
+<span class="add">+ def fetch_remote_config(cls, require_fresh: bool = False) -> RemoteConfigResult:</span>
+<span class="ctx"> url = PipelexDetails.remote_config_url()</span>
+<span class="ctx"> try:</span>
+<span class="ctx"> payload, config = cls._fetch_fresh(url)</span>
+<span class="add">+ RemoteConfigCache.store(payload) # opportunistic refresh</span>
+<span class="add">+ return RemoteConfigResult(config=config, source=RemoteConfigSource.FRESH)</span>
+<span class="ctx"> except RemoteConfigFetchError as fetch_error: # network / HTTP only</span>
+<span class="add">+ if require_fresh: # doc generators refuse stale data</span>
+<span class="add">+ raise cls._build_unavailable_error(fetch_error, cache_refused=True) from fetch_error</span>
+<span class="add">+ cached = RemoteConfigCache.load()</span>
+<span class="add">+ if cached is None:</span>
+<span class="add">+ raise cls._build_unavailable_error(fetch_error) from fetch_error</span>
+<span class="add">+ return RemoteConfigResult(</span>
+<span class="add">+ config=cached.to_remote_config(),</span>
+<span class="add">+ source=RemoteConfigSource.CACHED,</span>
+<span class="add">+ cached_at=cached.cached_at,</span>
+<span class="add">+ )</span>
+</pre>
+</div>
+
+<p>And <code>setup()</code> consumes the provenance — warning, and tightening telemetry, when the data is stale:</p>
+
+<div class="diff">
+ <div class="fname">pipelex/pipelex.py</div>
+<pre><span class="del">- remote_config = RemoteConfigFetcher.fetch_remote_config()</span>
+<span class="add">+ remote_config_result = RemoteConfigFetcher.fetch_remote_config()</span>
+<span class="add">+ remote_config = remote_config_result.config</span>
+<span class="add">+ gateway_config_source = remote_config_result.source</span>
+<span class="ctx"> ...</span>
+<span class="add">+ if gateway_config_source.is_cached:</span>
+<span class="add">+ warnings.warn(f"Pipelex Gateway is running off a cached remote config "</span>
+<span class="add">+ f"(snapshot: {cached_at_iso}). Run `pipelex init` while online.",</span>
+<span class="add">+ RemoteConfigStaleWarning, stacklevel=2)</span>
+<span class="hdr"> # stale specs imply stale model identities — don't phone home in that state</span>
+<span class="del">- is_pipelex_telemetry_enabled = is_pipelex_service_enabled and needs_inference</span>
+<span class="add">+ is_pipelex_telemetry_enabled = (is_pipelex_service_enabled and needs_inference</span>
+<span class="add">+ and not gateway_source_is_cached)</span>
+</pre>
+</div>
+
+<p class="muted">One deliberate design call: the <strong>warning is emitted in <code>setup()</code>, not in the fetcher</strong>. That keeps the fetcher a pure data-returning function, so test fixtures that swap in a cached fetcher don't have to replay warnings.</p>
+
+<h2>The behaviour matrix we validated</h2>
+<table>
+ <thead><tr><th>Scenario</th><th>Gateway</th><th>Network</th><th>Cache</th><th>Outcome</th></tr></thead>
+ <tbody>
+ <tr><td>BYOK offline</td><td>off</td><td>down</td><td>—</td><td><span class="dot g">●</span> setup + validate + dry-run all OK — fetch never attempted</td></tr>
+ <tr><td>Gateway dry-run, fresh</td><td>on</td><td>up</td><td>—</td><td><span class="dot g">●</span> fresh fetch, cache written</td></tr>
+ <tr><td>Gateway dry-run, cached</td><td>on</td><td>down</td><td>present</td><td><span class="dot a">●</span> cache fallback + stale warning, dry-run OK</td></tr>
+ <tr><td>Gateway dry-run, cold</td><td>on</td><td>down</td><td>absent</td><td><span class="dot r">●</span> <code>RemoteConfigUnavailableError</code> with remediation</td></tr>
+ <tr><td>Unknown model referenced</td><td>on</td><td>any</td><td>any</td><td><span class="dot r">●</span> <code>GatewayUnknownModelError</code>, message branches on source</td></tr>
+ <tr><td>Doc generator offline</td><td>on</td><td>down</td><td>present</td><td><span class="dot r">●</span> <code>require_fresh=True</code> refuses cache — no stale docs committed</td></tr>
+ </tbody>
+</table>
+
+<div class="callout">
+ <span class="lbl">The subtle one — doc generators</span>
+ <p style="margin:8px 0 0"><code>gateway_models_generator.py</code> and <code>preprocess_test_models_cmd.py</code> regenerate <strong>committed</strong> reference docs and test fixtures. If they silently used the cache, they'd bake stale data into the repo. They pass <code>require_fresh=True</code> — any fallback becomes an immediate error instead.</p>
+</div>
+
+<h2>How init primes the cache</h2>
+<p>The cache is only useful if it exists before the network goes down. So when <code>pipelex init</code> accepts the Gateway terms, it does one fetch and persists the result. If <em>that</em> is offline, init logs a yellow warning and continues — the cache stays empty, but the user has been told. The agent-CLI init mirrors this and surfaces <code>cache_primed</code> / <code>cache_priming_error</code> on its JSON envelope.</p>
+
+<h2>What landed</h2>
+<div class="panel">
+<div class="files">
+<span class="new">+ remote_config_cache.py</span>
+<span class="new">+ pipelex_service/types.py</span>
+remote_config_fetcher.py
+pipelex_service/exceptions.py
+cogt/exceptions.py
+cogt/models/model_manager.py
+model_manager_abstract.py
+pipelex.py
+cli/commands/init/command.py
+agent_cli/commands/init_cmd.py
+agent_cli/commands/agent_output.py
+agent_cli/commands/agent_cli_factory.py
+cli/error_handlers.py
+cli/cli_factory.py
+cli/commands/doctor_cmd.py
+dev_cli/.../gateway_models_generator.py
+dev_cli/.../preprocess_test_models_cmd.py
+pipelex_details.py
+<span class="new">+ test_remote_config_cache.py</span>
+<span class="new">+ test_remote_config_fetcher.py</span>
+<span class="new">+ test_gateway_unknown_model.py</span>
+<span class="new">+ test_setup_with_cache.py</span>
+<span class="new">+ test_cache_priming.py</span>
+<span class="new">+ test_offline_run_dry.py (E2E)</span>
+<span class="new">+ test_offline_baseline.py</span>
+</div>
+</div>
+<p class="muted">Built TDD across 7 phases — baseline lock-down → cache module → fetcher fallback → provenance plumbing → init priming → real-<code>.mthds</code>-bundle E2E → verification. Each phase is its own commit with a checkpoint block. <code>make agent-check</code> and <code>make agent-test</code> green throughout.</p>
+
+<h2>Deferred — not in this PR</h2>
+<ul class="tight muted">
+ <li><strong>doctor cache reporting</strong> — surface cache presence/age in <code>pipelex doctor</code>. Pure additive.</li>
+ <li><strong>Codex Cloud cache-first</strong> — today Codex Cloud returns a dummy config; it could try the cache first now that we trust it.</li>
+ <li><strong>Cross-repo</strong> — update <code>mthds-plugins/wip/codex-sandbox-escalation.md</code>: <code>--dry-run</code> no longer needs escalation once Pipelex has been initialised online once.</li>
+ <li><strong>E2E pinning <code>GatewayUnknownModelError</code></strong> — the E2E currently asserts the unknown handle is visible without pinning <code>error_type</code> (it fires at the pipe-operator layer). A deck-override test would pin the deck-level membership check end-to-end.</li>
+ <li><strong>Cache TTL</strong> — currently uncapped. Revisit once we have telemetry on fallback usage.</li>
+</ul>
+
+<footer>
+ Full plan with phase-by-phase checkpoints archived at
+ <code>wip/archive/offline-mode-remote-config-cache.md</code>.
+ Questions → ping me on the PR.
+</footer>
+
+</div>
+</body>
+</html>