Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(llm): ordered provider fallback chains for transient failures (#648)

AkitaOnRails14c7235e22b2
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2a8800ce..00a0f277 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- Ordered LLM provider fallback chains (#648). `[[llm_fallbacks]]` in
+ `config.toml` configures one or more additional providers, each with its
+ own `provider`, `model`, optional `base_url`, and optional `api_key_env`
+ (the environment-variable *name* holding that profile's key — never the
+ key itself in config). The primary provider (`llm_provider`) always runs
+ first; fallbacks run only after a transient failure
+ (`LlmError::is_transient()`: 429, 5xx, timeout, connection error) advances
+ past the current candidate, in declaration order, with the original
+ request, JSON schema, and logical operation id preserved on every
+ attempt. A deterministic failure (400/401/403/404/422, an unsupported
+ schema, or a malformed response) still stops immediately — the same
+ policy a single provider already had. Each candidate carries a 30s
+ in-memory circuit: a transient failure opens it, a success closes it,
+ and a restart clears all circuit state. `Config::load` validates every
+ profile and resolves its credential once, at startup — a missing/empty
+ provider or model, an unknown provider, or a `RequiredApiKey` provider
+ with no resolved `api_key_env` fails startup rather than leaving a
+ latent fallback that only fails once the primary is already down; a
+ provider with a native credential source (OpenAI OAuth, Copilot) needs
+ no `api_key_env`. `GET /admin/status` (and `ai-memory status`) now
+ reports each candidate's provider/model label, last-selected/last
+ success/error state, and circuit-open-until timestamp alongside the
+ existing top-level LLM provider/model fields, which are unchanged for
+ every existing single-provider deployment.
- `bootstrap --resume` recovers an interrupted run from durable per-chunk
progress instead of re-paying for every LLM call (#621). Each chunk's pages
are recorded (keyed by a fingerprint of the pruned sources + chunk budget, so
diff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rs
index 91e2122d..f2b01fed 100644
--- a/crates/ai-memory-cli/src/commands/serve.rs
+++ b/crates/ai-memory-cli/src/commands/serve.rs
@@ -16,7 +16,7 @@ use ai_memory_hooks::{
DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, HookState, ProjectCacheStore, WorkstreamState, hook_router,
workstream_router,
};
-use ai_memory_llm::{Embedder, LlmProvider, ProviderHealth, build_embedder, build_provider};
+use ai_memory_llm::{Embedder, LlmProvider, ProviderHealth, build_embedder};
use ai_memory_mcp::human_auth::{Cidr, HumanAuthRuntime, LoginLimiter};
use ai_memory_mcp::{
AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_sweep_tuning,
@@ -1985,7 +1985,14 @@ fn configure_consolidator(
let provider_name = cfg.provider.name().to_string();
let model = cfg.model.clone();
let retry_hint = llm_retry_hint(&provider_name, &model, cfg.base_url.as_deref());
- let llm = build_provider(cfg).context("building LLM provider from config")?;
+ // Routes through the same construction `llm_provider_config` just
+ // confirmed is `Some`: wraps `[primary, fallbacks...]` in a
+ // `FallbackLlmProvider` when `llm_fallbacks` is non-empty, else returns
+ // the plain provider unchanged (existing single-provider behavior).
+ let llm = config
+ .llm_provider_chain()
+ .context("building LLM provider chain from config")?
+ .context("LLM provider configured but chain construction returned none")?;
let llm = provider_health.wrap_llm_provider(llm, provider_name, model, Some(retry_hint));
info!(
provider = llm.name(),
diff --git a/crates/ai-memory-cli/src/commands/status.rs b/crates/ai-memory-cli/src/commands/status.rs
index 607ff3bf..7a499e2d 100644
--- a/crates/ai-memory-cli/src/commands/status.rs
+++ b/crates/ai-memory-cli/src/commands/status.rs
@@ -379,10 +379,36 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {
{
println!(" retry: {hint}");
}
+ // Only present when `llm_fallbacks` is configured — an older server
+ // or a plain single-provider setup reports an empty list.
+ if !report.providers.llm_candidates.is_empty() {
+ println!(" fallback chain:");
+ for candidate in &report.providers.llm_candidates {
+ println!(" {}", fallback_candidate_line(candidate));
+ }
+ }
}
Ok(())
}
+/// One line per `llm_candidates` entry: label, whether it answered the last
+/// completed call, and its most notable state (an open circuit takes
+/// priority over a stale last-error status).
+fn fallback_candidate_line(candidate: &ai_memory_llm::CandidateHealth) -> String {
+ let mut line = format!("{}/{}", candidate.provider, candidate.model);
+ if candidate.last_selected {
+ line.push_str(" (last selected)");
+ }
+ if let Some(until) = &candidate.circuit_open_until {
+ line.push_str(&format!(" — circuit open until {until}"));
+ } else if let Some(status) = candidate.last_error_status {
+ line.push_str(&format!(" — last error: status {status}"));
+ } else if let Some(class) = &candidate.last_error_class {
+ line.push_str(&format!(" — last error: {class}"));
+ }
+ line
+}
+
/// Render a spool age (ms) as a compact human duration, or `-` when the spool
/// holds no events. Allowed to saturate: an operator reading a stuck-spool
/// diagnosis wants the magnitude, not sub-second precision.
@@ -576,6 +602,42 @@ mod tests {
assert!(provider_health_line(&role).contains("bad token"));
}
+ #[test]
+ fn fallback_candidate_line_marks_the_last_selected_candidate() {
+ let candidate = ai_memory_llm::CandidateHealth {
+ provider: "gemini".to_string(),
+ model: "gemini-3.5-flash".to_string(),
+ last_selected: true,
+ last_success_at: Some("2026-05-28T12:00:00Z".parse::<Timestamp>().unwrap()),
+ last_error_at: None,
+ last_error_status: None,
+ last_error_class: None,
+ circuit_open_until: None,
+ };
+ let line = fallback_candidate_line(&candidate);
+ assert!(line.contains("gemini/gemini-3.5-flash"));
+ assert!(line.contains("last selected"));
+ }
+
+ #[test]
+ fn fallback_candidate_line_surfaces_an_open_circuit_over_a_stale_error() {
+ let when = "2026-05-28T12:00:30Z".parse::<Timestamp>().unwrap();
+ let candidate = ai_memory_llm::CandidateHealth {
+ provider: "openai-compat".to_string(),
+ model: "local-router".to_string(),
+ last_selected: false,
+ last_success_at: None,
+ last_error_at: Some(when),
+ last_error_status: Some(503),
+ last_error_class: Some("provider".to_string()),
+ circuit_open_until: Some(when),
+ };
+ let line = fallback_candidate_line(&candidate);
+ assert!(line.contains("openai-compat/local-router"));
+ assert!(line.contains("circuit open until"));
+ assert!(!line.contains("last error"));
+ }
+
#[test]
fn spool_age_line_renders_durations_and_empty() {
assert_eq!(spool_age_line(None), "-");
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index aac387e3..4b330060 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -7,10 +7,12 @@
//! `getMergedEnv()`, masking the bug for weeks).
use std::path::{Path, PathBuf};
+use std::sync::Arc;
use ai_memory_llm::{
- AuthRequirement, EmbedderChoice, EmbedderConfig, ExtraHeaders, LlmError, LlmResult,
- OPENCODE_DEFAULT_MODEL, ProviderAuth, ProviderChoice, ProviderConfig, ReasoningEffort,
+ AuthRequirement, Candidate, EmbedderChoice, EmbedderConfig, ExtraHeaders, FallbackLlmProvider,
+ LlmError, LlmProvider, LlmResult, OPENCODE_DEFAULT_MODEL, ProviderAuth, ProviderChoice,
+ ProviderConfig, ReasoningEffort, build_provider,
};
use anyhow::{Context, Result};
use figment::{
@@ -113,6 +115,31 @@ impl DecaySettings {
}
}
+/// One `[[llm_fallbacks]]` entry: an ordered LLM provider tried only after
+/// the primary (`llm_provider`) fails a transient call
+/// (`LlmError::is_transient()`). See `docs/llm-provider-fallback.md`.
+///
+/// `Config::load` validates every profile and resolves its credential once,
+/// at startup — a missing/empty provider or model, an unknown provider, or
+/// a missing credential fails startup rather than leaving a latent fallback
+/// that only fails under an outage.
+#[derive(Debug, Clone, Default, Serialize, Deserialize)]
+#[serde(default)]
+pub struct FallbackProfile {
+ /// Provider selection (same wire names as `llm_provider`).
+ pub provider: String,
+ /// Model id for this candidate.
+ pub model: String,
+ /// Optional base URL override (required for `openai-compat`, same as
+ /// the top-level `llm_base_url`).
+ pub base_url: Option<String>,
+ /// Environment-variable name holding this profile's API key. Optional
+ /// only for a provider with a native credential source (OpenAI OAuth,
+ /// Copilot); every other provider must set it — it is never inherited
+ /// from the primary provider's own env var.
+ pub api_key_env: Option<String>,
+}
+
/// Top-level runtime configuration.
///
/// `deny_unknown_fields` is intentionally NOT set: figment's
@@ -194,6 +221,32 @@ pub struct Config {
/// replaces, so a duplicate would break the request instead of
/// overriding it. Values are never logged.
pub llm_headers: Vec<String>,
+ /// Ordered LLM fallback chain, tried after the primary provider only on
+ /// a transient failure (`LlmError::is_transient()`); empty by default
+ /// (no behavior change). See [`FallbackProfile`] and
+ /// `docs/llm-provider-fallback.md`. Configure via TOML:
+ /// ```toml
+ /// [[llm_fallbacks]]
+ /// provider = "openai-compat"
+ /// model = "poolside/laguna-s-2.1-free"
+ /// base_url = "http://127.0.0.1:49375/v1"
+ /// api_key_env = "AI_MEMORY_LOCAL_ROUTER_TOKEN"
+ /// ```
+ /// No environment-variable shorthand: figment cannot round-trip
+ /// `Vec<Struct>` from `AI_MEMORY_*` env vars (see
+ /// `AI_MEMORY_ADMISSION_WEBHOOKS_JSON`'s comment below), and encoding
+ /// per-profile credential names into one env var would be ambiguous.
+ #[serde(default)]
+ pub llm_fallbacks: Vec<FallbackProfile>,
+ /// Validated [`ProviderConfig`] for each `llm_fallbacks` entry
+ /// (credentials included), resolved once by `Config::load` in
+ /// declaration order. Not a config-file key — see
+ /// [`Self::llm_provider_chain`]. `pub` (like [`Self::runtime_env`])
+ /// only so struct-update syntax (`..Config::default()`) keeps working
+ /// from every call site; construct it through `Config::load`, not by
+ /// hand.
+ #[serde(skip)]
+ pub llm_fallback_configs: Vec<ProviderConfig>,
/// Opt-in: run LLM consolidation on SessionEnd (in addition to the
/// always-written heuristic session page), when an LLM provider is
/// configured. Off by default. Provider work is durably queued after the
@@ -651,6 +704,8 @@ impl Default for Config {
llm_timeout_secs: ai_memory_llm::DEFAULT_REQUEST_TIMEOUT_SECS,
llm_reasoning_effort: None,
llm_headers: Vec::new(),
+ llm_fallbacks: Vec::new(),
+ llm_fallback_configs: Vec::new(),
consolidate_on_session_end: false,
capture_assistant: false,
strip_root_combinators: false,
@@ -1058,6 +1113,34 @@ impl Config {
.context("parsing AI_MEMORY_LLM_HEADERS / llm_headers")?;
validate_auth_secrets(&config.auth)?;
+ // Validated and resolved eagerly, unlike the primary provider (which
+ // stays lazily checked at `serve` startup): a fallback can otherwise
+ // sit unused for months and only fail once the primary is already
+ // down. The environment is read here, once, per invariant #1 (no
+ // `std::env::var` outside `load`).
+ let mut fallback_configs = Vec::with_capacity(config.llm_fallbacks.len());
+ for (i, profile) in config.llm_fallbacks.iter().enumerate() {
+ let resolved_key = match non_empty(profile.api_key_env.as_deref()) {
+ Some(name) => Some(SecretString::from(env_string(name).ok_or_else(|| {
+ anyhow::anyhow!(
+ "llm_fallbacks[{i}].api_key_env={name} is set but the environment \
+ variable is missing or empty"
+ )
+ })?)),
+ None => None,
+ };
+ let provider_cfg = config
+ .fallback_provider_config(i, profile, resolved_key)
+ .with_context(|| format!("validating llm_fallbacks[{i}]"))?;
+ // Constructed and discarded so a malformed profile (missing
+ // base_url, bad credential shape) fails startup instead of
+ // silently sitting unused until the primary has an outage.
+ build_provider(provider_cfg.clone())
+ .with_context(|| format!("building llm_fallbacks[{i}]"))?;
+ fallback_configs.push(provider_cfg);
+ }
+ config.llm_fallback_configs = fallback_configs;
+
Ok(config)
}
@@ -1087,22 +1170,12 @@ impl Config {
let Some(provider_raw) = non_empty(self.llm_provider.as_deref()) else {
return Ok(None);
};
- let provider = match provider_raw {
- "anthropic" => ProviderChoice::Anthropic,
- "openai" => ProviderChoice::OpenAi,
- "gemini" | "google" => ProviderChoice::Gemini,
- "openai-compat" | "openai_compat" => ProviderChoice::OpenAiCompat,
- "openai-oauth" | "openai_oauth" => ProviderChoice::OpenAiOAuth,
- "copilot" | "github-copilot" | "github_copilot" => ProviderChoice::Copilot,
- "anthropic-oauth" | "anthropic_oauth" => ProviderChoice::AnthropicOAuth,
- "opencode" | "opencode-zen" | "opencode_zen" => ProviderChoice::OpenCode,
- other => {
- return Err(LlmError::NotConfigured(format!(
- "AI_MEMORY_LLM_PROVIDER={other} is not one of \
- anthropic|openai|gemini|openai-compat|openai-oauth|copilot|anthropic-oauth|opencode"
- )));
- }
- };
+ let provider = provider_choice_from_str(provider_raw).ok_or_else(|| {
+ LlmError::NotConfigured(format!(
+ "AI_MEMORY_LLM_PROVIDER={provider_raw} is not one of \
+ anthropic|openai|gemini|openai-compat|openai-oauth|copilot|anthropic-oauth|opencode"
+ ))
+ })?;
let model = match non_empty(self.llm_model.as_deref()) {
Some(s) => s.to_string(),
None => match provider {
@@ -1140,6 +1213,133 @@ impl Config {
}))
}
+ /// Resolve one `[[llm_fallbacks]]` profile into a validated
+ /// [`ProviderConfig`].
+ ///
+ /// `resolved_key` is the value already read from the profile's
+ /// `api_key_env` (or `None` when it sets none) — the environment is
+ /// read once, in [`Self::load`], and passed down as data so this stays
+ /// directly testable without mutating process env (`std::env::set_var`
+ /// is `unsafe` under edition 2024, forbidden workspace-wide).
+ ///
+ /// # Errors
+ /// Returns [`LlmError::NotConfigured`] for an empty provider/model, an
+ /// unknown provider, or a `RequiredApiKey` provider with no resolved
+ /// credential.
+ fn fallback_provider_config(
+ &self,
+ index: usize,
+ profile: &FallbackProfile,
+ resolved_key: Option<SecretString>,
+ ) -> LlmResult<ProviderConfig> {
+ let provider_raw = non_empty(Some(profile.provider.as_str())).ok_or_else(|| {
+ LlmError::NotConfigured(format!("llm_fallbacks[{index}].provider must not be empty"))
+ })?;
+ let provider = provider_choice_from_str(provider_raw).ok_or_else(|| {
+ LlmError::NotConfigured(format!(
+ "llm_fallbacks[{index}].provider={provider_raw} is not one of \
+ anthropic|openai|gemini|openai-compat|openai-oauth|copilot|anthropic-oauth|opencode"
+ ))
+ })?;
+ let model = non_empty(Some(profile.model.as_str()))
+ .ok_or_else(|| {
+ LlmError::NotConfigured(format!("llm_fallbacks[{index}].model must not be empty"))
+ })?
+ .to_string();
+ if resolved_key.is_none()
+ && matches!(
+ provider.auth_requirement(),
+ AuthRequirement::RequiredApiKey { .. }
+ )
+ {
+ return Err(LlmError::NotConfigured(format!(
+ "llm_fallbacks[{index}] provider={provider_raw} requires api_key_env \
+ (no native credential source for this provider)"
+ )));
+ }
+ Ok(ProviderConfig {
+ provider,
+ model,
+ auth: self.fallback_provider_auth(provider, resolved_key),
+ base_url: non_empty(profile.base_url.as_deref()).map(str::to_string),
+ compat_strict: self.llm_compat_strict,
+ request_timeout_secs: self.llm_timeout_secs,
+ reasoning_effort: self.llm_reasoning_effort,
+ extra_headers: self.llm_extra_headers()?,
+ })
+ }
+
+ /// Auth resolution for a fallback profile. Deliberately distinct from
+ /// [`Self::provider_auth`]: that method falls back to the *primary*
+ /// provider's own fixed env var
+ /// (e.g. `ANTHROPIC_API_KEY`) when no override is given, which would let
+ /// a fallback profile silently reuse the primary's credential even
+ /// though it omitted `api_key_env` — defeating the "fails startup"
+ /// validation in [`Self::fallback_provider_config`]. Native credential
+ /// sources (OpenAI OAuth, Copilot, Anthropic OAuth) are process-wide by
+ /// nature and are still shared with the primary provider.
+ fn fallback_provider_auth(
+ &self,
+ provider: ProviderChoice,
+ resolved_key: Option<SecretString>,
+ ) -> ProviderAuth {
+ match provider.auth_requirement() {
+ AuthRequirement::RequiredApiKey { env_var } => {
+ ProviderAuth::required_api_key_from_env(env_var, resolved_key)
+ }
+ AuthRequirement::OptionalApiKey { env_var } => {
+ ProviderAuth::optional_api_key_from_env(env_var, resolved_key)
+ }
+ AuthRequirement::OpenAiOAuthToken => {
+ ProviderAuth::openai_oauth_token_file(self.openai_oauth_token_path())
+ }
+ AuthRequirement::CopilotToken => ProviderAuth::copilot(
+ self.copilot_token_path(),
+ self.runtime_env.copilot_github_token.clone(),
+ self.runtime_env.github_copilot_api_token.clone(),
+ self.runtime_env.copilot_api_url.clone(),
+ ),
+ AuthRequirement::AnthropicOAuthToken => {
+ ProviderAuth::anthropic_oauth_token(self.runtime_env.anthropic_oauth_token.clone())
+ }
+ }
+ }
+
+ /// Build the configured LLM provider, including any ordered
+ /// `llm_fallbacks` chain.
+ ///
+ /// `None` when no LLM is configured; the plain provider when no
+ /// fallback is configured (existing single-provider callers are
+ /// unaffected); otherwise a [`FallbackLlmProvider`] wrapping the
+ /// primary and its fallbacks in declaration order. See
+ /// `docs/llm-provider-fallback.md`.
+ ///
+ /// # Errors
+ /// Propagates any error from constructing the primary or a fallback
+ /// provider (`build_provider` is the sole construction path for both).
+ pub fn llm_provider_chain(&self) -> LlmResult<Option<Arc<dyn LlmProvider>>> {
+ let Some(primary_cfg) = self.llm_provider_config()? else {
+ return Ok(None);
+ };
+ if self.llm_fallback_configs.is_empty() {
+ return Ok(Some(build_provider(primary_cfg)?));
+ }
+ let mut candidates = Vec::with_capacity(1 + self.llm_fallback_configs.len());
+ candidates.push(Candidate::new(
+ primary_cfg.provider.name(),
+ primary_cfg.model.clone(),
+ build_provider(primary_cfg)?,
+ ));
+ for cfg in &self.llm_fallback_configs {
+ candidates.push(Candidate::new(
+ cfg.provider.name(),
+ cfg.model.clone(),
+ build_provider(cfg.clone())?,
+ ));
+ }
+ Ok(Some(Arc::new(FallbackLlmProvider::new(candidates))))
+ }
+
/// OpenAI-compatible embedding key. `EMBEDDING_API_KEY` is checked first
/// so an operator can point the embedder at one provider while the LLM
/// uses another; without it, direct OpenAI keeps requiring
@@ -1387,6 +1587,21 @@ impl Config {
}
}
+/// Wire-name parsing shared by `llm_provider` and `llm_fallbacks[].provider`.
+fn provider_choice_from_str(raw: &str) -> Option<ProviderChoice> {
+ Some(match raw {
+ "anthropic" => ProviderChoice::Anthropic,
+ "openai" => ProviderChoice::OpenAi,
+ "gemini" | "google" => ProviderChoice::Gemini,
+ "openai-compat" | "openai_compat" => ProviderChoice::OpenAiCompat,
+ "openai-oauth" | "openai_oauth" => ProviderChoice::OpenAiOAuth,
+ "copilot" | "github-copilot" | "github_copilot" => ProviderChoice::Copilot,
+ "anthropic-oauth" | "anthropic_oauth" => ProviderChoice::AnthropicOAuth,
+ "opencode" | "opencode-zen" | "opencode_zen" => ProviderChoice::OpenCode,
+ _ => return None,
+ })
+}
+
fn env_string(name: &str) -> Option<String> {
std::env::var(name).ok().and_then(|s| {
let trimmed = s.trim();
@@ -2680,6 +2895,213 @@ mod tests {
}
}
+ fn load_with_toml(toml: &str) -> anyhow::Result<Config> {
+ let tmp = TempDir::new().unwrap();
+ let config_path = tmp.path().join("config.toml");
+ std::fs::write(&config_path, toml).unwrap();
+ Config::load(Some(&config_path), Some(tmp.path().to_path_buf()))
+ }
+
+ #[test]
+ fn llm_fallbacks_default_to_empty_and_no_behavior_change() {
+ let cfg = Config::default();
+ assert!(cfg.llm_fallbacks.is_empty());
+ assert!(cfg.llm_fallback_configs.is_empty());
+ }
+
+ #[test]
+ fn load_rejects_an_empty_fallback_provider() {
+ let error = load_with_toml("[[llm_fallbacks]]\nprovider = \"\"\nmodel = \"m\"\n")
+ .expect_err("an empty provider must fail closed");
+ assert!(
+ format!("{error:#}").contains("llm_fallbacks[0].provider must not be empty"),
+ "unexpected error: {error:#}"
+ );
+ }
+
+ #[test]
+ fn load_rejects_an_unknown_fallback_provider() {
+ let error = load_with_toml(
+ "[[llm_fallbacks]]\nprovider = \"not-a-real-provider\"\nmodel = \"m\"\n",
+ )
+ .expect_err("an unknown provider must fail closed");
+ assert!(
+ format!("{error:#}").contains("is not one of"),
+ "unexpected error: {error:#}"
+ );
+ }
+
+ #[test]
+ fn load_rejects_an_empty_fallback_model() {
+ let error = load_with_toml("[[llm_fallbacks]]\nprovider = \"gemini\"\nmodel = \"\"\n")
+ .expect_err("an empty model must fail closed");
+ assert!(
+ format!("{error:#}").contains("llm_fallbacks[0].model must not be empty"),
+ "unexpected error: {error:#}"
+ );
+ }
+
+ #[test]
+ fn load_rejects_a_required_api_key_provider_with_no_api_key_env() {
+ let error = load_with_toml("[[llm_fallbacks]]\nprovider = \"gemini\"\nmodel = \"m\"\n")
+ .expect_err("a RequiredApiKey provider needs api_key_env");
+ assert!(
+ format!("{error:#}").contains("requires api_key_env"),
+ "unexpected error: {error:#}"
+ );
+ }
+
+ /// The fallback never silently inherits the *primary*'s fixed env var
+ /// (`GEMINI_API_KEY`) — it must fail closed instead of looking healthy
+ /// until the primary has an outage.
+ #[test]
+ fn load_rejects_a_fallback_that_would_otherwise_inherit_the_primary_credential() {
+ let error = load_with_toml(
+ "llm_provider = \"gemini\"\n[[llm_fallbacks]]\nprovider = \"gemini\"\nmodel = \"m\"\n",
+ )
+ .expect_err("omitting api_key_env must not fall back to GEMINI_API_KEY");
+ assert!(
+ format!("{error:#}").contains("requires api_key_env"),
+ "unexpected error: {error:#}"
+ );
+ }
+
+ #[test]
+ fn load_rejects_a_missing_or_empty_api_key_env_value() {
+ let error = load_with_toml(
+ "[[llm_fallbacks]]\nprovider = \"gemini\"\nmodel = \"m\"\n\
+ api_key_env = \"AI_MEMORY_TEST_FALLBACK_UNSET_KEY_648\"\n",
+ )
+ .expect_err("an unresolved api_key_env must fail closed");
+ assert!(
+ format!("{error:#}").contains(
+ "llm_fallbacks[0].api_key_env=AI_MEMORY_TEST_FALLBACK_UNSET_KEY_648 is set but \
+ the environment variable is missing or empty"
+ ),
+ "unexpected error: {error:#}"
+ );
+ }
+
+ #[test]
+ fn load_rejects_an_openai_compat_fallback_with_no_base_url() {
+ let error =
+ load_with_toml("[[llm_fallbacks]]\nprovider = \"openai-compat\"\nmodel = \"m\"\n")
+ .expect_err("openai-compat needs a base_url");
+ assert!(
+ format!("{error:#}").contains("LLM_BASE_URL"),
+ "unexpected error: {error:#}"
+ );
+ }
+
+ #[test]
+ fn load_accepts_a_well_formed_fallback_profile_with_no_primary() {
+ // `llm_fallbacks` is meaningful even without a primary configured
+ // (it is only ever consulted through `llm_provider_chain`, which
+ // returns `None` when there is no primary) — the profile itself
+ // must still validate independently.
+ let cfg = load_with_toml(
+ "[[llm_fallbacks]]\nprovider = \"openai-compat\"\nmodel = \"m\"\n\
+ base_url = \"http://127.0.0.1:9\"\n",
+ )
+ .unwrap();
+ let fallbacks = &cfg.llm_fallback_configs;
+ assert_eq!(fallbacks.len(), 1);
+ assert_eq!(fallbacks[0].provider, ProviderChoice::OpenAiCompat);
+ assert_eq!(fallbacks[0].model, "m");
+ assert_eq!(fallbacks[0].base_url.as_deref(), Some("http://127.0.0.1:9"));
+ assert!(cfg.llm_provider_chain().unwrap().is_none());
+ }
+
+ /// `fallback_provider_config` takes the resolved credential as a plain
+ /// parameter (see its doc comment): the environment is read once, in
+ /// `Config::load`, and passed down as data so the happy path stays
+ /// directly testable without mutating process env.
+ #[test]
+ fn fallback_provider_config_resolves_a_directly_supplied_credential() {
+ let cfg = Config::default();
+ let profile = FallbackProfile {
+ provider: "gemini".into(),
+ model: "gemini-3.5-flash".into(),
+ base_url: None,
+ api_key_env: Some("GEMINI_FALLBACK_KEY".into()),
+ };
+ let provider_cfg = cfg
+ .fallback_provider_config(0, &profile, Some(SecretString::from("sk-fallback-test")))
+ .unwrap();
+ assert_eq!(provider_cfg.provider, ProviderChoice::Gemini);
+ assert_eq!(provider_cfg.model, "gemini-3.5-flash");
+ assert_eq!(
+ provider_cfg.auth.require_api_key().unwrap().expose_secret(),
+ "sk-fallback-test"
+ );
+ }
+
+ /// `openai-oauth`/`copilot`/`anthropic-oauth` are native credential
+ /// sources: no `api_key_env` is required, and the fallback shares the
+ /// same process-wide token material as the primary.
+ #[test]
+ fn fallback_provider_config_allows_no_api_key_env_for_a_native_credential_provider() {
+ let cfg = Config::default();
+ let profile = FallbackProfile {
+ provider: "anthropic-oauth".into(),
+ model: "claude-sonnet-4-6".into(),
+ base_url: None,
+ api_key_env: None,
+ };
+ let provider_cfg = cfg.fallback_provider_config(0, &profile, None).unwrap();
+ assert_eq!(provider_cfg.provider, ProviderChoice::AnthropicOAuth);
+ }
+
+ #[test]
+ fn llm_provider_chain_is_the_plain_provider_when_no_fallback_is_configured() {
+ let cfg = Config {
+ llm_provider: Some("openai".into()),
+ runtime_env: RuntimeEnv {
+ openai_api_key: Some(SecretString::from("sk-test-key")),
+ ..RuntimeEnv::default()
+ },
+ ..Config::default()
+ };
+ let chain = cfg.llm_provider_chain().unwrap().unwrap();
+ assert_eq!(chain.name(), "openai");
+ // A plain provider reports no candidates — only a fallback chain does.
+ assert!(chain.candidate_health().is_empty());
+ }
+
+ /// End-to-end happy path through `Config::load` -> `llm_provider_chain`:
+ /// a keyless `openai-compat` primary plus one keyless `openai-compat`
+ /// fallback (distinct base URLs, as `docs/llm-provider-fallback.md`'s
+ /// example does with distinct providers) produces a `FallbackLlmProvider`
+ /// exposing both candidates, in declaration order.
+ #[test]
+ fn llm_provider_chain_wraps_the_primary_and_every_fallback_in_order() {
+ let cfg = load_with_toml(
+ "llm_provider = \"openai-compat\"\n\
+ llm_model = \"primary-model\"\n\
+ llm_base_url = \"http://127.0.0.1:9\"\n\
+ [[llm_fallbacks]]\n\
+ provider = \"openai-compat\"\n\
+ model = \"fallback-model\"\n\
+ base_url = \"http://127.0.0.1:9\"\n",
+ )
+ .unwrap();
+
+ let chain = cfg.llm_provider_chain().unwrap().unwrap();
+ assert_eq!(chain.name(), "openai-compat");
+ assert_eq!(chain.model(), "primary-model");
+ let candidates = chain.candidate_health();
+ assert_eq!(candidates.len(), 2);
+ assert_eq!(candidates[0].provider, "openai-compat");
+ assert_eq!(candidates[0].model, "primary-model");
+ assert_eq!(candidates[1].provider, "openai-compat");
+ assert_eq!(candidates[1].model, "fallback-model");
+ }
+
+ #[test]
+ fn llm_provider_chain_none_when_no_llm_is_configured() {
+ assert!(Config::default().llm_provider_chain().unwrap().is_none());
+ }
+
#[test]
fn anthropic_oauth_provider_underscore_alias_also_resolves() {
let cfg = Config {
diff --git a/crates/ai-memory-llm/src/error.rs b/crates/ai-memory-llm/src/error.rs
index 9f9bf918..e78732da 100644
--- a/crates/ai-memory-llm/src/error.rs
+++ b/crates/ai-memory-llm/src/error.rs
@@ -42,6 +42,20 @@ pub enum LlmError {
/// JSON schema for structured output could not be derived.
#[error("schema: {0}")]
Schema(String),
+
+ /// Every eligible candidate in an ordered LLM fallback chain failed, or
+ /// every candidate's circuit was open. Carries only provider/model
+ /// labels and error classes/status — never response bodies or secrets.
+ /// See `docs/llm-provider-fallback.md`.
+ #[error("llm fallback chain exhausted after {attempted} candidate(s): {summary}")]
+ AllCandidatesFailed {
+ /// Number of candidates actually attempted (skipped open circuits
+ /// are excluded from this count).
+ attempted: usize,
+ /// Bounded `provider/model: class[ status]` summary, one entry per
+ /// attempt, joined with `"; "`.
+ summary: String,
+ },
}
impl LlmError {
@@ -63,6 +77,32 @@ impl LlmError {
_ => false,
}
}
+
+ /// HTTP status captured from this error, when the failure carries one.
+ #[must_use]
+ pub fn http_status(&self) -> Option<u16> {
+ match self {
+ Self::Http(e) => e.status().map(|status| status.as_u16()),
+ Self::Provider { status, .. } => Some(*status),
+ _ => None,
+ }
+ }
+
+ /// Redacted error category used for health reporting and the fallback
+ /// chain's aggregate error — never a response body or secret.
+ #[must_use]
+ pub const fn class(&self) -> &'static str {
+ match self {
+ Self::Http(_) => "http",
+ Self::Provider { .. } => "provider",
+ Self::Serde(_) => "serde",
+ Self::UnexpectedShape(_) => "unexpected-shape",
+ Self::NotConfigured(_) => "not-configured",
+ Self::Auth(_) => "auth",
+ Self::Schema(_) => "schema",
+ Self::AllCandidatesFailed { .. } => "all-candidates-failed",
+ }
+ }
}
impl From<serde_json::Error> for LlmError {
@@ -115,4 +155,61 @@ mod tests {
assert!(!LlmError::UnexpectedShape("no tool block".into()).is_transient());
assert!(!LlmError::NotConfigured("no key".into()).is_transient());
}
+
+ #[test]
+ fn all_candidates_failed_is_not_transient() {
+ // The aggregate error is terminal: nothing is left to advance to.
+ assert!(
+ !LlmError::AllCandidatesFailed {
+ attempted: 2,
+ summary: String::new(),
+ }
+ .is_transient()
+ );
+ }
+
+ #[test]
+ fn http_status_reads_provider_status_only() {
+ assert_eq!(
+ LlmError::Provider {
+ status: 503,
+ body: "boom".into()
+ }
+ .http_status(),
+ Some(503)
+ );
+ assert_eq!(LlmError::Schema("bad".into()).http_status(), None);
+ }
+
+ #[test]
+ fn class_is_a_stable_redacted_label_never_the_message() {
+ let cases: &[(LlmError, &str)] = &[
+ (
+ LlmError::Provider {
+ status: 500,
+ body: "secret upstream body".into(),
+ },
+ "provider",
+ ),
+ (LlmError::Serde("nope".into()), "serde"),
+ (
+ LlmError::UnexpectedShape("no tool block".into()),
+ "unexpected-shape",
+ ),
+ (LlmError::NotConfigured("no key".into()), "not-configured"),
+ (LlmError::Auth("expired".into()), "auth"),
+ (LlmError::Schema("bad".into()), "schema"),
+ (
+ LlmError::AllCandidatesFailed {
+ attempted: 1,
+ summary: String::new(),
+ },
+ "all-candidates-failed",
+ ),
+ ];
+ for (err, expected) in cases {
+ assert_eq!(err.class(), *expected, "unexpected class for {err}");
+ assert!(!err.class().contains("secret upstream body"));
+ }
+ }
}
diff --git a/crates/ai-memory-llm/src/fallback.rs b/crates/ai-memory-llm/src/fallback.rs
new file mode 100644
index 00000000..4186d6d1
--- /dev/null
+++ b/crates/ai-memory-llm/src/fallback.rs
@@ -0,0 +1,753 @@
+//! Ordered LLM provider fallback chain.
+//!
+//! See `docs/llm-provider-fallback.md` (issue #648) for the full design.
+//! `FallbackLlmProvider` wraps an ordered list of already-constructed
+//! providers behind the same [`LlmProvider`] trait: on a transient failure
+//! (`LlmError::is_transient()`) it advances to the next eligible candidate,
+//! preserving the original request, schema, and operation id; a
+//! deterministic failure (bad request, auth, unsupported schema, malformed
+//! response) returns immediately without trying the rest of the chain.
+//! `Config::llm_provider_chain` in `ai-memory-cli` is the sole place that
+//! builds one; `build_provider` remains the sole construction path for each
+//! candidate.
+
+use std::sync::{Arc, Mutex};
+use std::time::{Duration, Instant};
+
+use async_trait::async_trait;
+use jiff::Timestamp;
+
+use crate::error::{LlmError, LlmResult};
+use crate::health::CandidateHealth;
+use crate::provider::LlmProvider;
+use crate::types::{ChatRequest, ChatResponse, LlmOperationId};
+
+/// How long a candidate's circuit stays open after a transient failure
+/// before it is eligible again. Restarting the process clears all circuit
+/// state; there is no durable or cross-process cooldown.
+pub const CIRCUIT_COOLDOWN: Duration = Duration::from_secs(30);
+
+/// One provider in an ordered fallback chain.
+///
+/// Carries only the diagnostic labels used by health reporting and circuit
+/// tracking — never the raw configuration or credentials that built
+/// `inner`. `build_provider` remains the sole construction path; this type
+/// only wraps its result.
+pub struct Candidate {
+ provider: &'static str,
+ model: String,
+ inner: Arc<dyn LlmProvider>,
+}
+
+impl Candidate {
+ /// Wrap an already-constructed provider for use in a fallback chain.
+ #[must_use]
+ pub fn new(
+ provider: &'static str,
+ model: impl Into<String>,
+ inner: Arc<dyn LlmProvider>,
+ ) -> Self {
+ Self {
+ provider,
+ model: model.into(),
+ inner,
+ }
+ }
+}
+
+/// Per-candidate mutable state: circuit + last-call bookkeeping, shared
+/// between the call path and health reporting.
+#[derive(Default)]
+struct CandidateState {
+ last_success_at: Option<Timestamp>,
+ last_error_at: Option<Timestamp>,
+ last_error_status: Option<u16>,
+ last_error_class: Option<&'static str>,
+ circuit_open_until: Option<Instant>,
+}
+
+struct CandidateEntry {
+ candidate: Candidate,
+ state: Mutex<CandidateState>,
+}
+
+/// Ordered LLM provider fallback chain.
+///
+/// Tries each eligible candidate in declaration order, advancing only on a
+/// transient failure; a deterministic failure returns immediately. See the
+/// module docs and `docs/llm-provider-fallback.md`.
+pub struct FallbackLlmProvider {
+ entries: Vec<CandidateEntry>,
+ /// Index of the candidate that answered the most recently completed
+ /// call, for health reporting.
+ last_selected: Mutex<Option<usize>>,
+}
+
+impl FallbackLlmProvider {
+ /// Build a chain from an ordered list of candidates (primary first,
+ /// then fallbacks in declaration order).
+ #[must_use]
+ pub fn new(candidates: Vec<Candidate>) -> Self {
+ Self {
+ entries: candidates
+ .into_iter()
+ .map(|candidate| CandidateEntry {
+ candidate,
+ state: Mutex::new(CandidateState::default()),
+ })
+ .collect(),
+ last_selected: Mutex::new(None),
+ }
+ }
+
+ fn circuit_open(&self, index: usize) -> bool {
+ let state = self.entries[index]
+ .state
+ .lock()
+ .unwrap_or_else(|e| e.into_inner());
+ matches!(state.circuit_open_until, Some(until) if Instant::now() < until)
+ }
+
+ fn record_success(&self, index: usize) {
+ {
+ let mut state = self.entries[index]
+ .state
+ .lock()
+ .unwrap_or_else(|e| e.into_inner());
+ state.last_success_at = Some(Timestamp::now());
+ state.circuit_open_until = None;
+ }
+ *self.last_selected.lock().unwrap_or_else(|e| e.into_inner()) = Some(index);
+ }
+
+ fn record_error(&self, index: usize, err: &LlmError) {
+ let mut state = self.entries[index]
+ .state
+ .lock()
+ .unwrap_or_else(|e| e.into_inner());
+ state.last_error_at = Some(Timestamp::now());
+ state.last_error_status = err.http_status();
+ state.last_error_class = Some(err.class());
+ if err.is_transient() {
+ state.circuit_open_until = Some(Instant::now() + CIRCUIT_COOLDOWN);
+ }
+ }
+
+ fn describe_failure(&self, index: usize, err: &LlmError) -> String {
+ let candidate = &self.entries[index].candidate;
+ match err.http_status() {
+ Some(status) => format!(
+ "{}/{}: {} {status}",
+ candidate.provider,
+ candidate.model,
+ err.class()
+ ),
+ None => format!(
+ "{}/{}: {}",
+ candidate.provider,
+ candidate.model,
+ err.class()
+ ),
+ }
+ }
+
+ fn exhausted(&self, attempted: usize, summary: Vec<String>) -> LlmError {
+ LlmError::AllCandidatesFailed {
+ attempted,
+ summary: summary.join("; "),
+ }
+ }
+}
+
+#[async_trait]
+impl LlmProvider for FallbackLlmProvider {
+ fn name(&self) -> &'static str {
+ self.entries
+ .first()
+ .map_or("fallback-chain-empty", |entry| entry.candidate.provider)
+ }
+
+ fn model(&self) -> &str {
+ self.entries
+ .first()
+ .map_or("", |entry| entry.candidate.model.as_str())
+ }
+
+ async fn complete(&self, request: ChatRequest) -> LlmResult<ChatResponse> {
+ let mut attempted = 0usize;
+ let mut summary = Vec::new();
+ for (i, entry) in self.entries.iter().enumerate() {
+ if self.circuit_open(i) {
+ continue;
+ }
+ attempted += 1;
+ match entry.candidate.inner.complete(request.clone()).await {
+ Ok(response) => {
+ self.record_success(i);
+ return Ok(response);
+ }
+ Err(err) => {
+ let transient = err.is_transient();
+ summary.push(self.describe_failure(i, &err));
+ self.record_error(i, &err);
+ if !transient {
+ return Err(err);
+ }
+ }
+ }
+ }
+ Err(self.exhausted(attempted, summary))
+ }
+
+ async fn complete_with_operation_id(
+ &self,
+ request: ChatRequest,
+ operation_id: LlmOperationId,
+ ) -> LlmResult<ChatResponse> {
+ let mut attempted = 0usize;
+ let mut summary = Vec::new();
+ for (i, entry) in self.entries.iter().enumerate() {
+ if self.circuit_open(i) {
+ continue;
+ }
+ attempted += 1;
+ match entry
+ .candidate
+ .inner
+ .complete_with_operation_id(request.clone(), operation_id)
+ .await
+ {
+ Ok(response) => {
+ self.record_success(i);
+ return Ok(response);
+ }
+ Err(err) => {
+ let transient = err.is_transient();
+ summary.push(self.describe_failure(i, &err));
+ self.record_error(i, &err);
+ if !transient {
+ return Err(err);
+ }
+ }
+ }
+ }
+ Err(self.exhausted(attempted, summary))
+ }
+
+ async fn complete_structured_raw(
+ &self,
+ request: ChatRequest,
+ schema: serde_json::Value,
+ ) -> LlmResult<serde_json::Value> {
+ let mut attempted = 0usize;
+ let mut summary = Vec::new();
+ for (i, entry) in self.entries.iter().enumerate() {
+ if self.circuit_open(i) {
+ continue;
+ }
+ attempted += 1;
+ match entry
+ .candidate
+ .inner
+ .complete_structured_raw(request.clone(), schema.clone())
+ .await
+ {
+ Ok(value) => {
+ self.record_success(i);
+ return Ok(value);
+ }
+ Err(err) => {
+ let transient = err.is_transient();
+ summary.push(self.describe_failure(i, &err));
+ self.record_error(i, &err);
+ if !transient {
+ return Err(err);
+ }
+ }
+ }
+ }
+ Err(self.exhausted(attempted, summary))
+ }
+
+ async fn complete_structured_raw_with_operation_id(
+ &self,
+ request: ChatRequest,
+ schema: serde_json::Value,
+ operation_id: LlmOperationId,
+ ) -> LlmResult<serde_json::Value> {
+ let mut attempted = 0usize;
+ let mut summary = Vec::new();
+ for (i, entry) in self.entries.iter().enumerate() {
+ if self.circuit_open(i) {
+ continue;
+ }
+ attempted += 1;
+ match entry
+ .candidate
+ .inner
+ .complete_structured_raw_with_operation_id(
+ request.clone(),
+ schema.clone(),
+ operation_id,
+ )
+ .await
+ {
+ Ok(value) => {
+ self.record_success(i);
+ return Ok(value);
+ }
+ Err(err) => {
+ let transient = err.is_transient();
+ summary.push(self.describe_failure(i, &err));
+ self.record_error(i, &err);
+ if !transient {
+ return Err(err);
+ }
+ }
+ }
+ }
+ Err(self.exhausted(attempted, summary))
+ }
+
+ fn candidate_health(&self) -> Vec<CandidateHealth> {
+ let selected = *self.last_selected.lock().unwrap_or_else(|e| e.into_inner());
+ self.entries
+ .iter()
+ .enumerate()
+ .map(|(i, entry)| {
+ let state = entry.state.lock().unwrap_or_else(|e| e.into_inner());
+ let now = Instant::now();
+ CandidateHealth {
+ provider: entry.candidate.provider.to_string(),
+ model: entry.candidate.model.clone(),
+ last_selected: selected == Some(i),
+ last_success_at: state.last_success_at,
+ last_error_at: state.last_error_at,
+ last_error_status: state.last_error_status,
+ last_error_class: state.last_error_class.map(str::to_string),
+ // Only report while the cooldown genuinely has not
+ // elapsed yet — a stale past instant (no later call
+ // resets it) would otherwise read as "still open".
+ circuit_open_until: state.circuit_open_until.and_then(|until| {
+ (until > now)
+ .then(|| Timestamp::now() + until.saturating_duration_since(now))
+ }),
+ }
+ })
+ .collect()
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use std::sync::atomic::{AtomicUsize, Ordering};
+
+ use super::*;
+ use crate::types::{ChatMessage, Usage};
+
+ /// A scripted provider double: returns each configured outcome in
+ /// order (repeating the last one once exhausted), and records every
+ /// request/schema/operation id it was called with.
+ struct ScriptedLlm {
+ name: &'static str,
+ model: String,
+ outcomes: Mutex<Vec<LlmResult<ChatResponse>>>,
+ calls: AtomicUsize,
+ seen_requests: Mutex<Vec<ChatRequest>>,
+ seen_schemas: Mutex<Vec<serde_json::Value>>,
+ seen_operation_ids: Mutex<Vec<Option<LlmOperationId>>>,
+ }
+
+ impl ScriptedLlm {
+ fn new(name: &'static str, model: &str, outcomes: Vec<LlmResult<ChatResponse>>) -> Self {
+ Self {
+ name,
+ model: model.to_string(),
+ outcomes: Mutex::new(outcomes),
+ calls: AtomicUsize::new(0),
+ seen_requests: Mutex::new(Vec::new()),
+ seen_schemas: Mutex::new(Vec::new()),
+ seen_operation_ids: Mutex::new(Vec::new()),
+ }
+ }
+
+ fn ok(name: &'static str, model: &str, text: &str) -> Self {
+ Self::new(
+ name,
+ model,
+ vec![Ok(ChatResponse {
+ text: text.to_string(),
+ usage: Some(Usage::default()),
+ model: model.to_string(),
+ })],
+ )
+ }
+
+ fn err(name: &'static str, model: &str, err: LlmError) -> Self {
+ Self::new(name, model, vec![Err(err)])
+ }
+
+ fn calls(&self) -> usize {
+ self.calls.load(Ordering::SeqCst)
+ }
+
+ fn next_outcome(&self) -> LlmResult<ChatResponse> {
+ self.calls.fetch_add(1, Ordering::SeqCst);
+ let mut outcomes = self.outcomes.lock().unwrap();
+ if outcomes.len() > 1 {
+ outcomes.remove(0)
+ } else {
+ clone_outcome(&outcomes[0])
+ }
+ }
+
+ fn record(&self, request: &ChatRequest, operation_id: Option<LlmOperationId>) {
+ self.seen_requests.lock().unwrap().push(request.clone());
+ self.seen_operation_ids.lock().unwrap().push(operation_id);
+ }
+ }
+
+ fn clone_outcome(outcome: &LlmResult<ChatResponse>) -> LlmResult<ChatResponse> {
+ match outcome {
+ Ok(response) => Ok(response.clone()),
+ Err(err) => Err(clone_err(err)),
+ }
+ }
+
+ fn clone_err(err: &LlmError) -> LlmError {
+ match err {
+ LlmError::Provider { status, body } => LlmError::Provider {
+ status: *status,
+ body: body.clone(),
+ },
+ LlmError::Serde(msg) => LlmError::Serde(msg.clone()),
+ LlmError::UnexpectedShape(msg) => LlmError::UnexpectedShape(msg.clone()),
+ LlmError::NotConfigured(msg) => LlmError::NotConfigured(msg.clone()),
+ LlmError::Auth(msg) => LlmError::Auth(msg.clone()),
+ LlmError::Schema(msg) => LlmError::Schema(msg.clone()),
+ other => panic!("clone_err: unsupported variant in test double: {other}"),
+ }
+ }
+
+ #[async_trait]
+ impl LlmProvider for ScriptedLlm {
+ fn name(&self) -> &'static str {
+ self.name
+ }
+
+ fn model(&self) -> &str {
+ &self.model
+ }
+
+ async fn complete(&self, request: ChatRequest) -> LlmResult<ChatResponse> {
+ self.record(&request, None);
+ self.next_outcome()
+ }
+
+ async fn complete_with_operation_id(
+ &self,
+ request: ChatRequest,
+ operation_id: LlmOperationId,
+ ) -> LlmResult<ChatResponse> {
+ self.record(&request, Some(operation_id));
+ self.next_outcome()
+ }
+
+ async fn complete_structured_raw(
+ &self,
+ request: ChatRequest,
+ schema: serde_json::Value,
+ ) -> LlmResult<serde_json::Value> {
+ self.record(&request, None);
+ self.seen_schemas.lock().unwrap().push(schema);
+ self.next_outcome()
+ .map(|response| serde_json::json!({ "text": response.text }))
+ }
+
+ async fn complete_structured_raw_with_operation_id(
+ &self,
+ request: ChatRequest,
+ schema: serde_json::Value,
+ operation_id: LlmOperationId,
+ ) -> LlmResult<serde_json::Value> {
+ self.record(&request, Some(operation_id));
+ self.seen_schemas.lock().unwrap().push(schema);
+ self.next_outcome()
+ .map(|response| serde_json::json!({ "text": response.text }))
+ }
+ }
+
+ fn transient(status: u16) -> LlmError {
+ LlmError::Provider {
+ status,
+ body: "upstream body that must never leak".to_string(),
+ }
+ }
+
+ #[tokio::test]
+ async fn chain_tries_candidates_in_order_and_stops_at_first_success() {
+ let first = Arc::new(ScriptedLlm::err("anthropic", "claude", transient(503)));
+ let second = Arc::new(ScriptedLlm::ok("gemini", "gemini-flash", "second answer"));
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("anthropic", "claude", first.clone()),
+ Candidate::new("gemini", "gemini-flash", second.clone()),
+ ]);
+
+ let response = chain
+ .complete(ChatRequest::user_prompt("hi"))
+ .await
+ .unwrap();
+
+ assert_eq!(response.text, "second answer");
+ assert_eq!(first.calls(), 1);
+ assert_eq!(second.calls(), 1);
+ }
+
+ #[tokio::test]
+ async fn all_four_methods_preserve_request_schema_and_operation_id() {
+ let first = Arc::new(ScriptedLlm::err("primary", "m1", transient(500)));
+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "ok"));
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("primary", "m1", first.clone()),
+ Candidate::new("secondary", "m2", second.clone()),
+ ]);
+ let request = ChatRequest::user_prompt("preserve me").with_max_tokens(42);
+ let schema = serde_json::json!({ "type": "object" });
+ let op_id = LlmOperationId::new();
+
+ chain.complete(request.clone()).await.unwrap();
+ chain
+ .complete_with_operation_id(request.clone(), op_id)
+ .await
+ .unwrap();
+ chain
+ .complete_structured_raw(request.clone(), schema.clone())
+ .await
+ .unwrap();
+ chain
+ .complete_structured_raw_with_operation_id(request.clone(), schema.clone(), op_id)
+ .await
+ .unwrap();
+
+ // Every call advanced past `first` (transient) onto `second`, which
+ // must have seen the exact same request/schema/operation id on
+ // each of the four entry points.
+ assert_eq!(second.calls(), 4);
+ let seen_requests = second.seen_requests.lock().unwrap();
+ assert!(seen_requests.iter().all(
+ |r| matches!(&r.messages[..], [ChatMessage { content, .. }] if content == "preserve me")
+ && r.max_tokens == 42
+ ));
+ let seen_schemas = second.seen_schemas.lock().unwrap();
+ assert_eq!(seen_schemas.len(), 2);
+ assert!(seen_schemas.iter().all(|s| *s == schema));
+ let seen_ops = second.seen_operation_ids.lock().unwrap();
+ // complete/complete_structured_raw pass None; the *_with_operation_id
+ // calls pass the same op_id every time.
+ assert_eq!(*seen_ops, vec![None, Some(op_id), None, Some(op_id)]);
+ }
+
+ #[tokio::test]
+ async fn transient_statuses_advance_to_the_next_candidate() {
+ for status in [429, 500, 502, 503, 504] {
+ let first = Arc::new(ScriptedLlm::err("primary", "m1", transient(status)));
+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "ok"));
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("primary", "m1", first.clone()),
+ Candidate::new("secondary", "m2", second.clone()),
+ ]);
+
+ let result = chain.complete(ChatRequest::user_prompt("hi")).await;
+ assert!(result.is_ok(), "status {status} should have advanced");
+ assert_eq!(second.calls(), 1, "status {status} should have advanced");
+ }
+ }
+
+ #[tokio::test]
+ async fn deterministic_errors_stop_on_the_first_candidate() {
+ let deterministic = [
+ LlmError::Provider {
+ status: 400,
+ body: String::new(),
+ },
+ LlmError::Provider {
+ status: 401,
+ body: String::new(),
+ },
+ LlmError::Provider {
+ status: 403,
+ body: String::new(),
+ },
+ LlmError::Provider {
+ status: 404,
+ body: String::new(),
+ },
+ LlmError::Provider {
+ status: 422,
+ body: String::new(),
+ },
+ LlmError::Schema("bad schema".into()),
+ LlmError::Serde("deserialize failed".into()),
+ LlmError::UnexpectedShape("no tool block".into()),
+ ];
+ for err in deterministic {
+ let expected_class = err.class();
+ let first = Arc::new(ScriptedLlm::err("primary", "m1", err));
+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "unreachable"));
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("primary", "m1", first.clone()),
+ Candidate::new("secondary", "m2", second.clone()),
+ ]);
+
+ let result = chain.complete(ChatRequest::user_prompt("hi")).await;
+ let returned = result.expect_err("deterministic error must propagate, not succeed");
+ assert_eq!(returned.class(), expected_class);
+ assert_eq!(first.calls(), 1);
+ assert_eq!(
+ second.calls(),
+ 0,
+ "class {expected_class} must not try the next candidate"
+ );
+ }
+ }
+
+ #[tokio::test]
+ async fn all_candidates_failing_returns_a_bounded_aggregate_error() {
+ let first = Arc::new(ScriptedLlm::err(
+ "primary",
+ "m1",
+ transient_with_body(500, "primary secret body"),
+ ));
+ let second = Arc::new(ScriptedLlm::err(
+ "secondary",
+ "m2",
+ transient_with_body(503, "secondary secret body"),
+ ));
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("primary", "m1", first.clone()),
+ Candidate::new("secondary", "m2", second.clone()),
+ ]);
+
+ let err = chain
+ .complete(ChatRequest::user_prompt("hi"))
+ .await
+ .expect_err("every candidate failed");
+ match &err {
+ LlmError::AllCandidatesFailed { attempted, summary } => {
+ assert_eq!(*attempted, 2);
+ assert!(summary.contains("primary/m1"));
+ assert!(summary.contains("secondary/m2"));
+ assert!(summary.contains("500"));
+ assert!(summary.contains("503"));
+ assert!(!summary.contains("secret body"));
+ }
+ other => panic!("expected AllCandidatesFailed, got {other}"),
+ }
+ }
+
+ fn transient_with_body(status: u16, body: &str) -> LlmError {
+ LlmError::Provider {
+ status,
+ body: body.to_string(),
+ }
+ }
+
+ #[tokio::test]
+ async fn an_open_circuit_skips_only_its_candidate() {
+ let first = Arc::new(ScriptedLlm::err("primary", "m1", transient(500)));
+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "ok"));
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("primary", "m1", first.clone()),
+ Candidate::new("secondary", "m2", second.clone()),
+ ]);
+
+ // First call: primary fails transiently and opens its circuit;
+ // secondary answers.
+ chain
+ .complete(ChatRequest::user_prompt("hi"))
+ .await
+ .unwrap();
+ assert_eq!(first.calls(), 1);
+ assert_eq!(second.calls(), 1);
+
+ // Second call, still inside the cooldown: primary must be skipped
+ // outright (not attempted again), secondary answers again.
+ chain
+ .complete(ChatRequest::user_prompt("hi"))
+ .await
+ .unwrap();
+ assert_eq!(first.calls(), 1, "an open circuit must not be attempted");
+ assert_eq!(second.calls(), 2);
+ }
+
+ #[test]
+ fn a_success_closes_an_open_circuit_immediately() {
+ let chain = FallbackLlmProvider::new(vec![Candidate::new(
+ "primary",
+ "m1",
+ Arc::new(ScriptedLlm::ok("primary", "m1", "ok")),
+ )]);
+ // Simulate an open circuit far in the future — a real cooldown, not
+ // yet elapsed.
+ {
+ let mut state = chain.entries[0].state.lock().unwrap();
+ state.circuit_open_until = Some(Instant::now() + Duration::from_secs(600));
+ }
+ assert!(chain.circuit_open(0));
+
+ chain.record_success(0);
+
+ assert!(
+ !chain.circuit_open(0),
+ "a success must close the circuit immediately, not wait out the cooldown"
+ );
+ }
+
+ #[test]
+ fn an_elapsed_cooldown_closes_the_circuit_without_an_explicit_success() {
+ let chain = FallbackLlmProvider::new(vec![Candidate::new(
+ "primary",
+ "m1",
+ Arc::new(ScriptedLlm::ok("primary", "m1", "ok")),
+ )]);
+ {
+ let mut state = chain.entries[0].state.lock().unwrap();
+ state.circuit_open_until = Some(Instant::now() - Duration::from_secs(1));
+ }
+ assert!(!chain.circuit_open(0));
+ }
+
+ #[tokio::test]
+ async fn candidate_health_reports_labels_last_selected_and_redacted_errors() {
+ let first = Arc::new(ScriptedLlm::err(
+ "primary",
+ "m1",
+ transient_with_body(500, "must not leak"),
+ ));
+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "ok"));
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("primary", "m1", first),
+ Candidate::new("secondary", "m2", second),
+ ]);
+
+ chain
+ .complete(ChatRequest::user_prompt("hi"))
+ .await
+ .unwrap();
+
+ let health = chain.candidate_health();
+ assert_eq!(health.len(), 2);
+ assert_eq!(health[0].provider, "primary");
+ assert_eq!(health[0].model, "m1");
+ assert!(!health[0].last_selected);
+ assert_eq!(health[0].last_error_status, Some(500));
+ assert_eq!(health[0].last_error_class.as_deref(), Some("provider"));
+ assert!(health[0].circuit_open_until.is_some());
+ assert!(health[1].last_selected);
+ assert!(health[1].last_success_at.is_some());
+ assert!(health[1].circuit_open_until.is_none());
+
+ let rendered = format!("{health:?}");
+ assert!(!rendered.contains("must not leak"));
+ }
+}
diff --git a/crates/ai-memory-llm/src/health.rs b/crates/ai-memory-llm/src/health.rs
index 0860da57..6fd7ceb3 100644
--- a/crates/ai-memory-llm/src/health.rs
+++ b/crates/ai-memory-llm/src/health.rs
@@ -21,14 +21,28 @@ const MAX_ERROR_MESSAGE_CHARS: usize = 1024;
pub struct ProviderHealth {
llm: ProviderRoleHealth,
embedding: ProviderRoleHealth,
+ /// The configured LLM provider, held only so `snapshot()` can read its
+ /// [`LlmProvider::candidate_health`] — a plain provider always returns
+ /// empty here, a [`crate::fallback::FallbackLlmProvider`] returns its
+ /// ordered candidate list. Never dereferenced for anything but that
+ /// read, so this adds no behavior beyond passive reporting.
+ llm_provider: Arc<Mutex<Option<Arc<dyn LlmProvider>>>>,
}
impl ProviderHealth {
/// Return a serializable snapshot of the latest recorded state.
#[must_use]
pub fn snapshot(&self) -> ProviderHealthSnapshot {
+ let llm_candidates = self
+ .llm_provider
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .as_ref()
+ .map(|provider| provider.candidate_health())
+ .unwrap_or_default();
ProviderHealthSnapshot {
llm: self.llm.snapshot(),
+ llm_candidates,
embedding: self.embedding.snapshot(),
}
}
@@ -44,6 +58,7 @@ impl ProviderHealth {
) -> Arc<dyn LlmProvider> {
let health = self.llm.clone();
health.configure(provider.into(), model.into(), None, retry_hint);
+ *self.llm_provider.lock().unwrap_or_else(|e| e.into_inner()) = Some(inner.clone());
Arc::new(HealthRecordingLlmProvider { inner, health })
}
@@ -95,10 +110,46 @@ impl ProviderHealthStatus {
pub struct ProviderHealthSnapshot {
/// LLM provider role.
pub llm: ProviderRoleHealthSnapshot,
+ /// Ordered LLM fallback-chain candidate state (`llm_fallbacks`), in
+ /// declaration order (index 0 is the primary provider). Empty for a
+ /// single configured provider or no provider at all.
+ /// `#[serde(default)]` so a response from a server built before this
+ /// field existed still deserializes for a newer CLI.
+ #[serde(default)]
+ pub llm_candidates: Vec<CandidateHealth>,
/// Embedding provider role.
pub embedding: ProviderRoleHealthSnapshot,
}
+/// One candidate's passive health state in an ordered LLM fallback chain.
+/// See `docs/llm-provider-fallback.md`. Every entry here is by construction
+/// a configured chain member; redaction is total — only labels, timestamps,
+/// an HTTP status, and a short error class are recorded, never a response
+/// body or credential.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+pub struct CandidateHealth {
+ /// Candidate provider label.
+ pub provider: String,
+ /// Candidate model id.
+ pub model: String,
+ /// Whether this candidate answered the most recently completed chain
+ /// call.
+ pub last_selected: bool,
+ /// Timestamp of the last call this candidate answered successfully.
+ pub last_success_at: Option<Timestamp>,
+ /// Timestamp of the last call this candidate failed.
+ pub last_error_at: Option<Timestamp>,
+ /// HTTP status captured from the last error, when available.
+ pub last_error_status: Option<u16>,
+ /// Redacted error class captured from the last error
+ /// ([`LlmError::class`]), never a response body or secret.
+ pub last_error_class: Option<String>,
+ /// When this candidate's circuit reopens, if a transient failure has it
+ /// currently open. `None` once the cooldown has elapsed, even if no
+ /// later call has explicitly closed it.
+ pub circuit_open_until: Option<Timestamp>,
+}
+
/// Wire-format health snapshot for one provider role.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct ProviderRoleHealthSnapshot {
@@ -303,6 +354,10 @@ impl LlmProvider for HealthRecordingLlmProvider {
self.health.record_result(&result);
result
}
+
+ fn candidate_health(&self) -> Vec<CandidateHealth> {
+ self.inner.candidate_health()
+ }
}
struct HealthRecordingEmbedder {
diff --git a/crates/ai-memory-llm/src/lib.rs b/crates/ai-memory-llm/src/lib.rs
index 5e4d4f32..1ba6307d 100644
--- a/crates/ai-memory-llm/src/lib.rs
+++ b/crates/ai-memory-llm/src/lib.rs
@@ -60,6 +60,7 @@ pub mod copilot;
pub mod embedding;
pub mod error;
pub mod factory;
+pub mod fallback;
pub mod gemini;
pub mod google;
pub mod health;
@@ -94,10 +95,12 @@ pub use factory::{
EmbedderChoice, EmbedderConfig, ProviderChoice, ProviderConfig, build_embedder, build_provider,
default_embedding_dim, try_default_embedding_dim,
};
+pub use fallback::{CIRCUIT_COOLDOWN, Candidate, FallbackLlmProvider};
pub use gemini::GeminiProvider;
pub use google::{DEFAULT_MODEL as GOOGLE_DEFAULT_EMBED_MODEL, GoogleEmbedder};
pub use health::{
- ProviderHealth, ProviderHealthSnapshot, ProviderHealthStatus, ProviderRoleHealthSnapshot,
+ CandidateHealth, ProviderHealth, ProviderHealthSnapshot, ProviderHealthStatus,
+ ProviderRoleHealthSnapshot,
};
#[cfg(feature = "local-embeddings")]
pub use local::{LOCAL_DIM, LOCAL_MODEL, LocalEmbedder, fetch_model, model_present};
diff --git a/crates/ai-memory-llm/src/provider.rs b/crates/ai-memory-llm/src/provider.rs
index 63336b69..7ba2929b 100644
--- a/crates/ai-memory-llm/src/provider.rs
+++ b/crates/ai-memory-llm/src/provider.rs
@@ -65,6 +65,19 @@ pub trait LlmProvider: Send + Sync {
) -> LlmResult<serde_json::Value> {
self.complete_structured_raw(request, schema).await
}
+
+ /// Per-candidate health entries for an ordered LLM fallback chain.
+ ///
+ /// Empty for a plain, non-chained provider (the default). A wrapping
+ /// decorator (health recording) must forward this to its inner
+ /// provider; [`FallbackLlmProvider`](crate::fallback::FallbackLlmProvider)
+ /// is the only implementation that returns a non-empty list.
+ /// [`ProviderHealth`](crate::health::ProviderHealth) reads this at
+ /// snapshot time — it never probes providers or generates traffic on
+ /// its own.
+ fn candidate_health(&self) -> Vec<crate::health::CandidateHealth> {
+ Vec::new()
+ }
}
/// Typed wrapper around [`LlmProvider::complete_structured_raw`].
diff --git a/crates/ai-memory-llm/tests/suite/fallback_provider.rs b/crates/ai-memory-llm/tests/suite/fallback_provider.rs
new file mode 100644
index 00000000..b310abdc
--- /dev/null
+++ b/crates/ai-memory-llm/tests/suite/fallback_provider.rs
@@ -0,0 +1,111 @@
+//! Integration coverage for `FallbackLlmProvider` against real network
+//! failures: a genuine connection-refused and a genuine client-side
+//! timeout, each produced by a real `reqwest::Error` rather than a
+//! hand-built `LlmError`. `fallback.rs`'s own unit tests cover the pure
+//! chain-order/circuit/redaction logic with fake providers; this file
+//! proves `LlmError::is_transient()` sees those two real failure classes
+//! the same way when they arrive through an actual `OpenAiCompatProvider`
+//! HTTP call, matching `docs/llm-provider-fallback.md`'s delivery-sequence
+//! scenario (a candidate that first fails transiently, then a working one).
+
+use std::time::Duration;
+
+use ai_memory_llm::types::ChatRequest;
+use ai_memory_llm::{Candidate, FallbackLlmProvider, LlmProvider, OpenAiCompatProvider};
+use wiremock::matchers::{method, path};
+use wiremock::{Mock, MockServer, ResponseTemplate};
+
+fn compat_provider(
+ base_url: impl Into<String>,
+ model: &str,
+ timeout_secs: u64,
+) -> OpenAiCompatProvider {
+ OpenAiCompatProvider::new(base_url.into(), None, model)
+ .expect("provider builds")
+ .with_timeout_secs(timeout_secs)
+}
+
+fn ok_response_body(content: &str) -> serde_json::Value {
+ serde_json::json!({
+ "id": "id",
+ "object": "chat.completion",
+ "created": 0,
+ "model": "second-model",
+ "choices": [{
+ "index": 0,
+ "message": { "role": "assistant", "content": content },
+ "finish_reason": "stop",
+ }],
+ "usage": { "prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2 },
+ })
+}
+
+/// A closed loopback port refuses the connection immediately, producing a
+/// real `reqwest::Error` with `.is_connect() == true` — no wiremock or
+/// external network access needed.
+async fn closed_port_url() -> String {
+ let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind an ephemeral port");
+ let addr = listener.local_addr().expect("local addr");
+ drop(listener); // nothing listens on `addr` from here on
+ format!("http://{addr}/v1")
+}
+
+#[tokio::test]
+async fn a_real_connection_error_advances_to_the_next_candidate() {
+ let unreachable_base = closed_port_url().await;
+ let working = MockServer::start().await;
+ Mock::given(method("POST"))
+ .and(path("/v1/chat/completions"))
+ .respond_with(ResponseTemplate::new(200).set_body_json(ok_response_body("second answer")))
+ .mount(&working)
+ .await;
+
+ let first = compat_provider(unreachable_base, "first-model", 5);
+ let second = compat_provider(working.uri(), "second-model", 5);
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("openai-compat", "first-model", std::sync::Arc::new(first)),
+ Candidate::new("openai-compat", "second-model", std::sync::Arc::new(second)),
+ ]);
+
+ let response = chain
+ .complete(ChatRequest::user_prompt("hi"))
+ .await
+ .expect("a real connection error must advance to the next candidate");
+ assert_eq!(response.text, "second answer");
+}
+
+#[tokio::test]
+async fn a_real_client_timeout_advances_to_the_next_candidate() {
+ let slow = MockServer::start().await;
+ Mock::given(method("POST"))
+ .and(path("/v1/chat/completions"))
+ .respond_with(
+ ResponseTemplate::new(200)
+ .set_body_json(ok_response_body("too slow"))
+ .set_delay(Duration::from_secs(3)),
+ )
+ .mount(&slow)
+ .await;
+ let working = MockServer::start().await;
+ Mock::given(method("POST"))
+ .and(path("/v1/chat/completions"))
+ .respond_with(ResponseTemplate::new(200).set_body_json(ok_response_body("second answer")))
+ .mount(&working)
+ .await;
+
+ // `with_timeout_secs` only accepts whole seconds, so the mock's
+ // artificial delay is set well beyond the 1s client timeout — a real
+ // `reqwest::Error::is_timeout() == true`, not a race.
+ let first = compat_provider(slow.uri(), "first-model", 1);
+ let second = compat_provider(working.uri(), "second-model", 5);
+ let chain = FallbackLlmProvider::new(vec![
+ Candidate::new("openai-compat", "first-model", std::sync::Arc::new(first)),
+ Candidate::new("openai-compat", "second-model", std::sync::Arc::new(second)),
+ ]);
+
+ let response = chain
+ .complete(ChatRequest::user_prompt("hi"))
+ .await
+ .expect("a real client timeout must advance to the next candidate");
+ assert_eq!(response.text, "second answer");
+}
diff --git a/crates/ai-memory-llm/tests/suite/mod.rs b/crates/ai-memory-llm/tests/suite/mod.rs
index 0337bf86..5a50f954 100644
--- a/crates/ai-memory-llm/tests/suite/mod.rs
+++ b/crates/ai-memory-llm/tests/suite/mod.rs
@@ -3,5 +3,6 @@
//! no extra binary; a new file must be declared below.
mod extra_headers_on_the_wire;
+mod fallback_provider;
mod openai_compat_embedder;
mod openai_compat_strict;
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 2c581cbc..124fb98a 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -626,6 +626,55 @@ GITHUB_COPILOT_API_TOKEN optional pre-minted Copilot API token
COPILOT_API_URL optional Copilot API base URL override
```
+**Ordered LLM fallback chain** (opt-in, TOML only — see
+`docs/llm-provider-fallback.md`, #648): the primary provider above always
+runs first; `[[llm_fallbacks]]` entries run only after a transient failure
+(429, 5xx, timeout, connection error), in declaration order, with the
+original request/schema/operation id preserved on every attempt. A
+deterministic failure (4xx other than 429, an unsupported schema, a
+malformed response) still stops on the first candidate — no chain-wide
+retry loop.
+
+```toml
+llm_provider = "opencode"
+llm_model = "mimo-v2.5-free"
+
+[[llm_fallbacks]]
+provider = "openai-compat" # same wire names as llm_provider
+model = "poolside/laguna-s-2.1-free"
+base_url = "http://127.0.0.1:49375/v1" # required for openai-compat, as above
+api_key_env = "AI_MEMORY_LOCAL_ROUTER_TOKEN" # env var *name*; the key itself
+ # never lives in config.toml
+
+[[llm_fallbacks]]
+provider = "gemini"
+model = "gemini-3.5-flash"
+api_key_env = "GEMINI_API_KEY"
+```
+
+`api_key_env` is required for any provider that needs an API key
+(`anthropic`, `openai`, `gemini`, `opencode`; optional for `openai-compat`,
+which may run keyless) — it is never inherited from the primary's own
+fixed env var, so a fallback cannot look configured while actually
+resolving no credential. It is optional only for a provider with a native
+credential source (`openai-oauth`, `copilot`, `anthropic-oauth`), which
+shares the primary's process-wide token material. `Config::load` validates
+every profile and resolves its credential once, at startup: a
+missing/empty provider or model, an unknown provider, or a missing
+credential fails startup rather than leaving a latent fallback that only
+fails once the primary is already down. Each candidate carries its own
+30s in-memory circuit (`ai_memory_llm::fallback::CIRCUIT_COOLDOWN`): a
+transient failure opens it, a success closes it, and a restart clears all
+circuit state — there is no durable circuit or forced chain-wide deadline.
+
+`GET /admin/status` (`ai-memory status`) reports an `llm_candidates` list
+alongside the existing `llm`/`embedding` roles: each candidate's
+provider/model label, whether it answered the most recently completed
+call, its last success/error timestamp, a redacted error class + HTTP
+status (never a response body or credential), and its circuit-open-until
+timestamp. It is empty for a plain single-provider setup; the top-level
+`llm` role fields are unchanged.
+
Every chat request carries `User-Agent: ai-memory/<version>`
(`ai_memory_llm::DEFAULT_USER_AGENT`, layered in `build_provider`). `reqwest`
sends no user agent unless configured, so provider requests used to arrive