Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(llm): propose transient provider fallback chains (#648)

fcoalcantarajrecd0986c9991
diff --git a/README.md b/README.md
index 331e441b..0e85971c 100644
--- a/README.md
+++ b/README.md
@@ -334,6 +334,7 @@ diagram, crate breakdown, schema notes, and invariants.
| [`docs/auto-improvement-loop.md`](docs/auto-improvement-loop.md) | Auto-improvement design notes: Hermes-inspired scheduled review, auto-approval default, manual review opt-in, pending proposal storage, and curator work. |
| [`docs/companion-crates.md`](docs/companion-crates.md) | Boundary and implementation plan for optional companion projects, including the standalone importer at [`companions/ai-memory-importer`](companions/ai-memory-importer), without widening core ai-memory. |
| [`docs/llm-provider-comparison.md`](docs/llm-provider-comparison.md) | Empirical notes behind the recommended LLM defaults. |
+| [`docs/llm-provider-fallback.md`](docs/llm-provider-fallback.md) | Proposed opt-in fallback-chain design for transient LLM-provider failures; not yet a supported configuration surface. |
| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | Operational summary: data flow, crate layout, cross-cutting invariants, schema. |
| [`docs/design-decisions.md`](docs/design-decisions.md) | The full v1 spec. |
| Research docs under `docs/` | Karpathy LLM Wiki notes, Hermes Agent, agentmemory / basic-memory / cognee deep-dives, lessons-learned from upstream issues. |
diff --git a/docs/llm-provider-fallback.md b/docs/llm-provider-fallback.md
new file mode 100644
index 00000000..4271898d
--- /dev/null
+++ b/docs/llm-provider-fallback.md
@@ -0,0 +1,160 @@
+# Proposed LLM provider fallback chains
+
+Issue: #648
+
+## Problem
+
+The server currently constructs one `Arc<dyn LlmProvider>` for every LLM-backed
+operation. Existing retry loops can retry a transient failure against that same
+provider, but cannot use a second configured provider. A temporary rate limit
+or upstream outage can therefore degrade a bootstrap, consolidation, lint, or
+review even when another provider is healthy.
+
+This proposal introduces an opt-in ordered chain behind the existing
+`LlmProvider` trait. It deliberately leaves the zero-LLM and existing
+single-provider paths unchanged.
+
+## Goals
+
+- Fail over only after a transient provider failure, using the established
+ `LlmError::is_transient()` policy.
+- Preserve the original request, JSON schema, and logical operation id on every
+ candidate attempt.
+- Keep credentials in the existing one-time configuration load and out of logs,
+ status payloads, and persisted state.
+- Avoid a global timeout policy. Each candidate uses its own configured request
+ timeout.
+- Make the selected candidate and failover outcome observable through passive
+ provider-health reporting.
+
+## Non-goals
+
+- A router for arbitrary third-party APIs or a Command Code integration.
+- Reading credentials from another application's database or installing a
+ provider CLI.
+- Retrying deterministic failures such as authentication, invalid requests,
+ unsupported schema, or malformed responses.
+- Changing embeddings, hook latency, wiki/storage behavior, or MCP schemas in
+ the first implementation.
+
+## Configuration shape
+
+The current top-level LLM fields remain the primary profile. Fallbacks live in
+`config.toml` so a profile can carry an explicit base URL and a credential
+*environment-variable name* without putting credential values in config.
+
+```toml
+llm_provider = "opencode"
+llm_model = "mimo-v2.5-free"
+
+[[llm_fallbacks]]
+provider = "openai-compat"
+model = "poolside/laguna-s-2.1-free"
+base_url = "http://127.0.0.1:49375/v1"
+api_key_env = "AI_MEMORY_LOCAL_ROUTER_TOKEN"
+
+[[llm_fallbacks]]
+provider = "gemini"
+model = "gemini-3.5-flash"
+api_key_env = "GEMINI_API_KEY"
+```
+
+`api_key_env` is optional when the provider already has a native credential
+source, such as OpenAI OAuth or Copilot. The loader validates every profile and
+resolves all credential material once. A missing credential, invalid provider,
+or malformed profile fails startup rather than leaving a latent fallback that
+only fails under an outage.
+
+The proposed initial semantics are append-only: the primary runs first, then
+the fallbacks in declaration order. No environment-variable shorthand is added
+until it can encode profile boundaries and credential names unambiguously.
+
+## Implementation boundary
+
+Add `FallbackLlmProvider` in `crates/ai-memory-llm/src/fallback.rs`:
+
+```text
+Config::load
+ -> primary ProviderConfig + fallback ProviderConfig values
+ -> build_provider for each value
+ -> FallbackLlmProvider(Vec<Candidate>)
+ -> existing Arc<dyn LlmProvider> consumers
+```
+
+Each `Candidate` records only a provider name and model for diagnostics plus its
+`Arc<dyn LlmProvider>`. It does not retain raw configuration or credentials.
+The wrapper implements all four trait entry points:
+
+- `complete`
+- `complete_with_operation_id`
+- `complete_structured_raw`
+- `complete_structured_raw_with_operation_id`
+
+Each method calls the corresponding candidate method. A structured call passes
+the unchanged schema to every eligible candidate; an operation-aware call passes
+the same `LlmOperationId` to every eligible candidate. This keeps caller retry
+and idempotency semantics intact.
+
+`build_provider` remains the sole dialect/auth construction path. The CLI stays
+thin: `Config::llm_provider_chain()` can return either `None`, one provider, or
+the wrapper, while callers still receive `Option<Arc<dyn LlmProvider>>`.
+
+## Failure policy and circuit state
+
+| Failure | Try next candidate? | Rationale |
+| --- | --- | --- |
+| 429 | Yes | Existing transient policy. |
+| 5xx | Yes | Existing transient policy. |
+| timeout / connection error | Yes | Existing transient policy. |
+| 400 / 401 / 403 / 404 / 422 | No | Usually request, capability, model, or credential configuration. |
+| schema / response-shape / deserialize error | No | Same input would deterministically fail again. |
+
+Before a candidate is called, the wrapper checks an in-memory circuit keyed by
+`(provider, model)`. A transient error opens that candidate's circuit for a
+bounded cooldown; other candidates remain eligible. A successful response
+closes that candidate's circuit. Restarting the server clears circuits. The
+initial implementation uses no durable circuit state and no separate forced
+request deadline.
+
+If every eligible candidate fails, the wrapper returns a bounded aggregate
+error containing provider/model labels and error classes, never response bodies
+or secrets.
+
+## Health reporting
+
+`ProviderHealth` currently represents one LLM role. Extend its LLM snapshot
+with a candidate list while preserving the existing top-level active
+provider/model fields for compatibility. Per-candidate state contains:
+
+- provider and model;
+- configured / last-selected state;
+- last success or error timestamp;
+- redacted error class and HTTP status when available;
+- circuit-open-until timestamp.
+
+Status records only calls the server already made. It does not probe fallbacks
+or trigger background recovery traffic.
+
+## Tests
+
+1. Unit-test chain order with fake `LlmProvider` instances.
+2. Verify all four trait methods preserve request/schema/operation id.
+3. Verify `429`, `5xx`, timeout, and connection failures advance; verify
+ deterministic errors stop on the first candidate.
+4. Verify an open circuit skips only its `(provider, model)` candidate and a
+ success closes it.
+5. Test configuration validation, including missing/empty profiles and missing
+ credentials, through `Config::load()` without reading a real home directory.
+6. Test health snapshots for candidate labels and redaction.
+7. Keep existing single-provider config tests byte-for-byte compatible.
+
+## Delivery sequence
+
+1. Add profile parsing, validation, and tests with no behavior change when no
+ fallback profile exists.
+2. Add the `FallbackLlmProvider` wrapper and focused fake-provider tests.
+3. Wire it into `serve`, extend passive health, and add the configuration and
+ status documentation plus `CHANGELOG.md` entry.
+4. Run the workspace gates and manual `llm-test` cases against a local
+OpenAI-compatible endpoint that first returns a transient failure and then a
+valid structured completion.