Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs(llm): propose transient provider fallback chains (#648)
fcoalcantarajrecd0986c9991
diff --git a/README.md b/README.mdindex 331e441b..0e85971c 100644--- a/README.md+++ b/README.md@@ -334,6 +334,7 @@ diagram, crate breakdown, schema notes, and invariants.| [`docs/auto-improvement-loop.md`](docs/auto-improvement-loop.md) | Auto-improvement design notes: Hermes-inspired scheduled review, auto-approval default, manual review opt-in, pending proposal storage, and curator work. || [`docs/companion-crates.md`](docs/companion-crates.md) | Boundary and implementation plan for optional companion projects, including the standalone importer at [`companions/ai-memory-importer`](companions/ai-memory-importer), without widening core ai-memory. || [`docs/llm-provider-comparison.md`](docs/llm-provider-comparison.md) | Empirical notes behind the recommended LLM defaults. |+| [`docs/llm-provider-fallback.md`](docs/llm-provider-fallback.md) | Proposed opt-in fallback-chain design for transient LLM-provider failures; not yet a supported configuration surface. || [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | Operational summary: data flow, crate layout, cross-cutting invariants, schema. || [`docs/design-decisions.md`](docs/design-decisions.md) | The full v1 spec. || Research docs under `docs/` | Karpathy LLM Wiki notes, Hermes Agent, agentmemory / basic-memory / cognee deep-dives, lessons-learned from upstream issues. |diff --git a/docs/llm-provider-fallback.md b/docs/llm-provider-fallback.mdnew file mode 100644index 00000000..4271898d--- /dev/null+++ b/docs/llm-provider-fallback.md@@ -0,0 +1,160 @@+# Proposed LLM provider fallback chains++Issue: #648++## Problem++The server currently constructs one `Arc<dyn LlmProvider>` for every LLM-backed+operation. Existing retry loops can retry a transient failure against that same+provider, but cannot use a second configured provider. A temporary rate limit+or upstream outage can therefore degrade a bootstrap, consolidation, lint, or+review even when another provider is healthy.++This proposal introduces an opt-in ordered chain behind the existing+`LlmProvider` trait. It deliberately leaves the zero-LLM and existing+single-provider paths unchanged.++## Goals++- Fail over only after a transient provider failure, using the established+ `LlmError::is_transient()` policy.+- Preserve the original request, JSON schema, and logical operation id on every+ candidate attempt.+- Keep credentials in the existing one-time configuration load and out of logs,+ status payloads, and persisted state.+- Avoid a global timeout policy. Each candidate uses its own configured request+ timeout.+- Make the selected candidate and failover outcome observable through passive+ provider-health reporting.++## Non-goals++- A router for arbitrary third-party APIs or a Command Code integration.+- Reading credentials from another application's database or installing a+ provider CLI.+- Retrying deterministic failures such as authentication, invalid requests,+ unsupported schema, or malformed responses.+- Changing embeddings, hook latency, wiki/storage behavior, or MCP schemas in+ the first implementation.++## Configuration shape++The current top-level LLM fields remain the primary profile. Fallbacks live in+`config.toml` so a profile can carry an explicit base URL and a credential+*environment-variable name* without putting credential values in config.++```toml+llm_provider = "opencode"+llm_model = "mimo-v2.5-free"++[[llm_fallbacks]]+provider = "openai-compat"+model = "poolside/laguna-s-2.1-free"+base_url = "http://127.0.0.1:49375/v1"+api_key_env = "AI_MEMORY_LOCAL_ROUTER_TOKEN"++[[llm_fallbacks]]+provider = "gemini"+model = "gemini-3.5-flash"+api_key_env = "GEMINI_API_KEY"+```++`api_key_env` is optional when the provider already has a native credential+source, such as OpenAI OAuth or Copilot. The loader validates every profile and+resolves all credential material once. A missing credential, invalid provider,+or malformed profile fails startup rather than leaving a latent fallback that+only fails under an outage.++The proposed initial semantics are append-only: the primary runs first, then+the fallbacks in declaration order. No environment-variable shorthand is added+until it can encode profile boundaries and credential names unambiguously.++## Implementation boundary++Add `FallbackLlmProvider` in `crates/ai-memory-llm/src/fallback.rs`:++```text+Config::load+ -> primary ProviderConfig + fallback ProviderConfig values+ -> build_provider for each value+ -> FallbackLlmProvider(Vec<Candidate>)+ -> existing Arc<dyn LlmProvider> consumers+```++Each `Candidate` records only a provider name and model for diagnostics plus its+`Arc<dyn LlmProvider>`. It does not retain raw configuration or credentials.+The wrapper implements all four trait entry points:++- `complete`+- `complete_with_operation_id`+- `complete_structured_raw`+- `complete_structured_raw_with_operation_id`++Each method calls the corresponding candidate method. A structured call passes+the unchanged schema to every eligible candidate; an operation-aware call passes+the same `LlmOperationId` to every eligible candidate. This keeps caller retry+and idempotency semantics intact.++`build_provider` remains the sole dialect/auth construction path. The CLI stays+thin: `Config::llm_provider_chain()` can return either `None`, one provider, or+the wrapper, while callers still receive `Option<Arc<dyn LlmProvider>>`.++## Failure policy and circuit state++| Failure | Try next candidate? | Rationale |+| --- | --- | --- |+| 429 | Yes | Existing transient policy. |+| 5xx | Yes | Existing transient policy. |+| timeout / connection error | Yes | Existing transient policy. |+| 400 / 401 / 403 / 404 / 422 | No | Usually request, capability, model, or credential configuration. |+| schema / response-shape / deserialize error | No | Same input would deterministically fail again. |++Before a candidate is called, the wrapper checks an in-memory circuit keyed by+`(provider, model)`. A transient error opens that candidate's circuit for a+bounded cooldown; other candidates remain eligible. A successful response+closes that candidate's circuit. Restarting the server clears circuits. The+initial implementation uses no durable circuit state and no separate forced+request deadline.++If every eligible candidate fails, the wrapper returns a bounded aggregate+error containing provider/model labels and error classes, never response bodies+or secrets.++## Health reporting++`ProviderHealth` currently represents one LLM role. Extend its LLM snapshot+with a candidate list while preserving the existing top-level active+provider/model fields for compatibility. Per-candidate state contains:++- provider and model;+- configured / last-selected state;+- last success or error timestamp;+- redacted error class and HTTP status when available;+- circuit-open-until timestamp.++Status records only calls the server already made. It does not probe fallbacks+or trigger background recovery traffic.++## Tests++1. Unit-test chain order with fake `LlmProvider` instances.+2. Verify all four trait methods preserve request/schema/operation id.+3. Verify `429`, `5xx`, timeout, and connection failures advance; verify+ deterministic errors stop on the first candidate.+4. Verify an open circuit skips only its `(provider, model)` candidate and a+ success closes it.+5. Test configuration validation, including missing/empty profiles and missing+ credentials, through `Config::load()` without reading a real home directory.+6. Test health snapshots for candidate labels and redaction.+7. Keep existing single-provider config tests byte-for-byte compatible.++## Delivery sequence++1. Add profile parsing, validation, and tests with no behavior change when no+ fallback profile exists.+2. Add the `FallbackLlmProvider` wrapper and focused fake-provider tests.+3. Wire it into `serve`, extend passive health, and add the configuration and+ status documentation plus `CHANGELOG.md` entry.+4. Run the workspace gates and manual `llm-test` cases against a local+OpenAI-compatible endpoint that first returns a transient failure and then a+valid structured completion.