Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(llm): ordered provider fallback chains for transient failures (#648)
AkitaOnRails14c7235e22b2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2a8800ce..00a0f277 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Added+- Ordered LLM provider fallback chains (#648). `[[llm_fallbacks]]` in+ `config.toml` configures one or more additional providers, each with its+ own `provider`, `model`, optional `base_url`, and optional `api_key_env`+ (the environment-variable *name* holding that profile's key — never the+ key itself in config). The primary provider (`llm_provider`) always runs+ first; fallbacks run only after a transient failure+ (`LlmError::is_transient()`: 429, 5xx, timeout, connection error) advances+ past the current candidate, in declaration order, with the original+ request, JSON schema, and logical operation id preserved on every+ attempt. A deterministic failure (400/401/403/404/422, an unsupported+ schema, or a malformed response) still stops immediately — the same+ policy a single provider already had. Each candidate carries a 30s+ in-memory circuit: a transient failure opens it, a success closes it,+ and a restart clears all circuit state. `Config::load` validates every+ profile and resolves its credential once, at startup — a missing/empty+ provider or model, an unknown provider, or a `RequiredApiKey` provider+ with no resolved `api_key_env` fails startup rather than leaving a+ latent fallback that only fails once the primary is already down; a+ provider with a native credential source (OpenAI OAuth, Copilot) needs+ no `api_key_env`. `GET /admin/status` (and `ai-memory status`) now+ reports each candidate's provider/model label, last-selected/last+ success/error state, and circuit-open-until timestamp alongside the+ existing top-level LLM provider/model fields, which are unchanged for+ every existing single-provider deployment.- `bootstrap --resume` recovers an interrupted run from durable per-chunkprogress instead of re-paying for every LLM call (#621). Each chunk's pagesare recorded (keyed by a fingerprint of the pruned sources + chunk budget, sodiff --git a/crates/ai-memory-cli/src/commands/serve.rs b/crates/ai-memory-cli/src/commands/serve.rsindex 91e2122d..f2b01fed 100644--- a/crates/ai-memory-cli/src/commands/serve.rs+++ b/crates/ai-memory-cli/src/commands/serve.rs@@ -16,7 +16,7 @@ use ai_memory_hooks::{DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, HookState, ProjectCacheStore, WorkstreamState, hook_router,workstream_router,};-use ai_memory_llm::{Embedder, LlmProvider, ProviderHealth, build_embedder, build_provider};+use ai_memory_llm::{Embedder, LlmProvider, ProviderHealth, build_embedder};use ai_memory_mcp::human_auth::{Cidr, HumanAuthRuntime, LoginLimiter};use ai_memory_mcp::{AdminState, AiMemoryServer, ScopeInvalidation, admin_router_with_sweep_tuning,@@ -1985,7 +1985,14 @@ fn configure_consolidator(let provider_name = cfg.provider.name().to_string();let model = cfg.model.clone();let retry_hint = llm_retry_hint(&provider_name, &model, cfg.base_url.as_deref());- let llm = build_provider(cfg).context("building LLM provider from config")?;+ // Routes through the same construction `llm_provider_config` just+ // confirmed is `Some`: wraps `[primary, fallbacks...]` in a+ // `FallbackLlmProvider` when `llm_fallbacks` is non-empty, else returns+ // the plain provider unchanged (existing single-provider behavior).+ let llm = config+ .llm_provider_chain()+ .context("building LLM provider chain from config")?+ .context("LLM provider configured but chain construction returned none")?;let llm = provider_health.wrap_llm_provider(llm, provider_name, model, Some(retry_hint));info!(provider = llm.name(),diff --git a/crates/ai-memory-cli/src/commands/status.rs b/crates/ai-memory-cli/src/commands/status.rsindex 607ff3bf..7a499e2d 100644--- a/crates/ai-memory-cli/src/commands/status.rs+++ b/crates/ai-memory-cli/src/commands/status.rs@@ -379,10 +379,36 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {{println!(" retry: {hint}");}+ // Only present when `llm_fallbacks` is configured — an older server+ // or a plain single-provider setup reports an empty list.+ if !report.providers.llm_candidates.is_empty() {+ println!(" fallback chain:");+ for candidate in &report.providers.llm_candidates {+ println!(" {}", fallback_candidate_line(candidate));+ }+ }}Ok(())}+/// One line per `llm_candidates` entry: label, whether it answered the last+/// completed call, and its most notable state (an open circuit takes+/// priority over a stale last-error status).+fn fallback_candidate_line(candidate: &ai_memory_llm::CandidateHealth) -> String {+ let mut line = format!("{}/{}", candidate.provider, candidate.model);+ if candidate.last_selected {+ line.push_str(" (last selected)");+ }+ if let Some(until) = &candidate.circuit_open_until {+ line.push_str(&format!(" — circuit open until {until}"));+ } else if let Some(status) = candidate.last_error_status {+ line.push_str(&format!(" — last error: status {status}"));+ } else if let Some(class) = &candidate.last_error_class {+ line.push_str(&format!(" — last error: {class}"));+ }+ line+}+/// Render a spool age (ms) as a compact human duration, or `-` when the spool/// holds no events. Allowed to saturate: an operator reading a stuck-spool/// diagnosis wants the magnitude, not sub-second precision.@@ -576,6 +602,42 @@ mod tests {assert!(provider_health_line(&role).contains("bad token"));}+ #[test]+ fn fallback_candidate_line_marks_the_last_selected_candidate() {+ let candidate = ai_memory_llm::CandidateHealth {+ provider: "gemini".to_string(),+ model: "gemini-3.5-flash".to_string(),+ last_selected: true,+ last_success_at: Some("2026-05-28T12:00:00Z".parse::<Timestamp>().unwrap()),+ last_error_at: None,+ last_error_status: None,+ last_error_class: None,+ circuit_open_until: None,+ };+ let line = fallback_candidate_line(&candidate);+ assert!(line.contains("gemini/gemini-3.5-flash"));+ assert!(line.contains("last selected"));+ }++ #[test]+ fn fallback_candidate_line_surfaces_an_open_circuit_over_a_stale_error() {+ let when = "2026-05-28T12:00:30Z".parse::<Timestamp>().unwrap();+ let candidate = ai_memory_llm::CandidateHealth {+ provider: "openai-compat".to_string(),+ model: "local-router".to_string(),+ last_selected: false,+ last_success_at: None,+ last_error_at: Some(when),+ last_error_status: Some(503),+ last_error_class: Some("provider".to_string()),+ circuit_open_until: Some(when),+ };+ let line = fallback_candidate_line(&candidate);+ assert!(line.contains("openai-compat/local-router"));+ assert!(line.contains("circuit open until"));+ assert!(!line.contains("last error"));+ }+#[test]fn spool_age_line_renders_durations_and_empty() {assert_eq!(spool_age_line(None), "-");diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex aac387e3..4b330060 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -7,10 +7,12 @@//! `getMergedEnv()`, masking the bug for weeks).use std::path::{Path, PathBuf};+use std::sync::Arc;use ai_memory_llm::{- AuthRequirement, EmbedderChoice, EmbedderConfig, ExtraHeaders, LlmError, LlmResult,- OPENCODE_DEFAULT_MODEL, ProviderAuth, ProviderChoice, ProviderConfig, ReasoningEffort,+ AuthRequirement, Candidate, EmbedderChoice, EmbedderConfig, ExtraHeaders, FallbackLlmProvider,+ LlmError, LlmProvider, LlmResult, OPENCODE_DEFAULT_MODEL, ProviderAuth, ProviderChoice,+ ProviderConfig, ReasoningEffort, build_provider,};use anyhow::{Context, Result};use figment::{@@ -113,6 +115,31 @@ impl DecaySettings {}}+/// One `[[llm_fallbacks]]` entry: an ordered LLM provider tried only after+/// the primary (`llm_provider`) fails a transient call+/// (`LlmError::is_transient()`). See `docs/llm-provider-fallback.md`.+///+/// `Config::load` validates every profile and resolves its credential once,+/// at startup — a missing/empty provider or model, an unknown provider, or+/// a missing credential fails startup rather than leaving a latent fallback+/// that only fails under an outage.+#[derive(Debug, Clone, Default, Serialize, Deserialize)]+#[serde(default)]+pub struct FallbackProfile {+ /// Provider selection (same wire names as `llm_provider`).+ pub provider: String,+ /// Model id for this candidate.+ pub model: String,+ /// Optional base URL override (required for `openai-compat`, same as+ /// the top-level `llm_base_url`).+ pub base_url: Option<String>,+ /// Environment-variable name holding this profile's API key. Optional+ /// only for a provider with a native credential source (OpenAI OAuth,+ /// Copilot); every other provider must set it — it is never inherited+ /// from the primary provider's own env var.+ pub api_key_env: Option<String>,+}+/// Top-level runtime configuration.////// `deny_unknown_fields` is intentionally NOT set: figment's@@ -194,6 +221,32 @@ pub struct Config {/// replaces, so a duplicate would break the request instead of/// overriding it. Values are never logged.pub llm_headers: Vec<String>,+ /// Ordered LLM fallback chain, tried after the primary provider only on+ /// a transient failure (`LlmError::is_transient()`); empty by default+ /// (no behavior change). See [`FallbackProfile`] and+ /// `docs/llm-provider-fallback.md`. Configure via TOML:+ /// ```toml+ /// [[llm_fallbacks]]+ /// provider = "openai-compat"+ /// model = "poolside/laguna-s-2.1-free"+ /// base_url = "http://127.0.0.1:49375/v1"+ /// api_key_env = "AI_MEMORY_LOCAL_ROUTER_TOKEN"+ /// ```+ /// No environment-variable shorthand: figment cannot round-trip+ /// `Vec<Struct>` from `AI_MEMORY_*` env vars (see+ /// `AI_MEMORY_ADMISSION_WEBHOOKS_JSON`'s comment below), and encoding+ /// per-profile credential names into one env var would be ambiguous.+ #[serde(default)]+ pub llm_fallbacks: Vec<FallbackProfile>,+ /// Validated [`ProviderConfig`] for each `llm_fallbacks` entry+ /// (credentials included), resolved once by `Config::load` in+ /// declaration order. Not a config-file key — see+ /// [`Self::llm_provider_chain`]. `pub` (like [`Self::runtime_env`])+ /// only so struct-update syntax (`..Config::default()`) keeps working+ /// from every call site; construct it through `Config::load`, not by+ /// hand.+ #[serde(skip)]+ pub llm_fallback_configs: Vec<ProviderConfig>,/// Opt-in: run LLM consolidation on SessionEnd (in addition to the/// always-written heuristic session page), when an LLM provider is/// configured. Off by default. Provider work is durably queued after the@@ -651,6 +704,8 @@ impl Default for Config {llm_timeout_secs: ai_memory_llm::DEFAULT_REQUEST_TIMEOUT_SECS,llm_reasoning_effort: None,llm_headers: Vec::new(),+ llm_fallbacks: Vec::new(),+ llm_fallback_configs: Vec::new(),consolidate_on_session_end: false,capture_assistant: false,strip_root_combinators: false,@@ -1058,6 +1113,34 @@ impl Config {.context("parsing AI_MEMORY_LLM_HEADERS / llm_headers")?;validate_auth_secrets(&config.auth)?;+ // Validated and resolved eagerly, unlike the primary provider (which+ // stays lazily checked at `serve` startup): a fallback can otherwise+ // sit unused for months and only fail once the primary is already+ // down. The environment is read here, once, per invariant #1 (no+ // `std::env::var` outside `load`).+ let mut fallback_configs = Vec::with_capacity(config.llm_fallbacks.len());+ for (i, profile) in config.llm_fallbacks.iter().enumerate() {+ let resolved_key = match non_empty(profile.api_key_env.as_deref()) {+ Some(name) => Some(SecretString::from(env_string(name).ok_or_else(|| {+ anyhow::anyhow!(+ "llm_fallbacks[{i}].api_key_env={name} is set but the environment \+ variable is missing or empty"+ )+ })?)),+ None => None,+ };+ let provider_cfg = config+ .fallback_provider_config(i, profile, resolved_key)+ .with_context(|| format!("validating llm_fallbacks[{i}]"))?;+ // Constructed and discarded so a malformed profile (missing+ // base_url, bad credential shape) fails startup instead of+ // silently sitting unused until the primary has an outage.+ build_provider(provider_cfg.clone())+ .with_context(|| format!("building llm_fallbacks[{i}]"))?;+ fallback_configs.push(provider_cfg);+ }+ config.llm_fallback_configs = fallback_configs;+Ok(config)}@@ -1087,22 +1170,12 @@ impl Config {let Some(provider_raw) = non_empty(self.llm_provider.as_deref()) else {return Ok(None);};- let provider = match provider_raw {- "anthropic" => ProviderChoice::Anthropic,- "openai" => ProviderChoice::OpenAi,- "gemini" | "google" => ProviderChoice::Gemini,- "openai-compat" | "openai_compat" => ProviderChoice::OpenAiCompat,- "openai-oauth" | "openai_oauth" => ProviderChoice::OpenAiOAuth,- "copilot" | "github-copilot" | "github_copilot" => ProviderChoice::Copilot,- "anthropic-oauth" | "anthropic_oauth" => ProviderChoice::AnthropicOAuth,- "opencode" | "opencode-zen" | "opencode_zen" => ProviderChoice::OpenCode,- other => {- return Err(LlmError::NotConfigured(format!(- "AI_MEMORY_LLM_PROVIDER={other} is not one of \- anthropic|openai|gemini|openai-compat|openai-oauth|copilot|anthropic-oauth|opencode"- )));- }- };+ let provider = provider_choice_from_str(provider_raw).ok_or_else(|| {+ LlmError::NotConfigured(format!(+ "AI_MEMORY_LLM_PROVIDER={provider_raw} is not one of \+ anthropic|openai|gemini|openai-compat|openai-oauth|copilot|anthropic-oauth|opencode"+ ))+ })?;let model = match non_empty(self.llm_model.as_deref()) {Some(s) => s.to_string(),None => match provider {@@ -1140,6 +1213,133 @@ impl Config {}))}+ /// Resolve one `[[llm_fallbacks]]` profile into a validated+ /// [`ProviderConfig`].+ ///+ /// `resolved_key` is the value already read from the profile's+ /// `api_key_env` (or `None` when it sets none) — the environment is+ /// read once, in [`Self::load`], and passed down as data so this stays+ /// directly testable without mutating process env (`std::env::set_var`+ /// is `unsafe` under edition 2024, forbidden workspace-wide).+ ///+ /// # Errors+ /// Returns [`LlmError::NotConfigured`] for an empty provider/model, an+ /// unknown provider, or a `RequiredApiKey` provider with no resolved+ /// credential.+ fn fallback_provider_config(+ &self,+ index: usize,+ profile: &FallbackProfile,+ resolved_key: Option<SecretString>,+ ) -> LlmResult<ProviderConfig> {+ let provider_raw = non_empty(Some(profile.provider.as_str())).ok_or_else(|| {+ LlmError::NotConfigured(format!("llm_fallbacks[{index}].provider must not be empty"))+ })?;+ let provider = provider_choice_from_str(provider_raw).ok_or_else(|| {+ LlmError::NotConfigured(format!(+ "llm_fallbacks[{index}].provider={provider_raw} is not one of \+ anthropic|openai|gemini|openai-compat|openai-oauth|copilot|anthropic-oauth|opencode"+ ))+ })?;+ let model = non_empty(Some(profile.model.as_str()))+ .ok_or_else(|| {+ LlmError::NotConfigured(format!("llm_fallbacks[{index}].model must not be empty"))+ })?+ .to_string();+ if resolved_key.is_none()+ && matches!(+ provider.auth_requirement(),+ AuthRequirement::RequiredApiKey { .. }+ )+ {+ return Err(LlmError::NotConfigured(format!(+ "llm_fallbacks[{index}] provider={provider_raw} requires api_key_env \+ (no native credential source for this provider)"+ )));+ }+ Ok(ProviderConfig {+ provider,+ model,+ auth: self.fallback_provider_auth(provider, resolved_key),+ base_url: non_empty(profile.base_url.as_deref()).map(str::to_string),+ compat_strict: self.llm_compat_strict,+ request_timeout_secs: self.llm_timeout_secs,+ reasoning_effort: self.llm_reasoning_effort,+ extra_headers: self.llm_extra_headers()?,+ })+ }++ /// Auth resolution for a fallback profile. Deliberately distinct from+ /// [`Self::provider_auth`]: that method falls back to the *primary*+ /// provider's own fixed env var+ /// (e.g. `ANTHROPIC_API_KEY`) when no override is given, which would let+ /// a fallback profile silently reuse the primary's credential even+ /// though it omitted `api_key_env` — defeating the "fails startup"+ /// validation in [`Self::fallback_provider_config`]. Native credential+ /// sources (OpenAI OAuth, Copilot, Anthropic OAuth) are process-wide by+ /// nature and are still shared with the primary provider.+ fn fallback_provider_auth(+ &self,+ provider: ProviderChoice,+ resolved_key: Option<SecretString>,+ ) -> ProviderAuth {+ match provider.auth_requirement() {+ AuthRequirement::RequiredApiKey { env_var } => {+ ProviderAuth::required_api_key_from_env(env_var, resolved_key)+ }+ AuthRequirement::OptionalApiKey { env_var } => {+ ProviderAuth::optional_api_key_from_env(env_var, resolved_key)+ }+ AuthRequirement::OpenAiOAuthToken => {+ ProviderAuth::openai_oauth_token_file(self.openai_oauth_token_path())+ }+ AuthRequirement::CopilotToken => ProviderAuth::copilot(+ self.copilot_token_path(),+ self.runtime_env.copilot_github_token.clone(),+ self.runtime_env.github_copilot_api_token.clone(),+ self.runtime_env.copilot_api_url.clone(),+ ),+ AuthRequirement::AnthropicOAuthToken => {+ ProviderAuth::anthropic_oauth_token(self.runtime_env.anthropic_oauth_token.clone())+ }+ }+ }++ /// Build the configured LLM provider, including any ordered+ /// `llm_fallbacks` chain.+ ///+ /// `None` when no LLM is configured; the plain provider when no+ /// fallback is configured (existing single-provider callers are+ /// unaffected); otherwise a [`FallbackLlmProvider`] wrapping the+ /// primary and its fallbacks in declaration order. See+ /// `docs/llm-provider-fallback.md`.+ ///+ /// # Errors+ /// Propagates any error from constructing the primary or a fallback+ /// provider (`build_provider` is the sole construction path for both).+ pub fn llm_provider_chain(&self) -> LlmResult<Option<Arc<dyn LlmProvider>>> {+ let Some(primary_cfg) = self.llm_provider_config()? else {+ return Ok(None);+ };+ if self.llm_fallback_configs.is_empty() {+ return Ok(Some(build_provider(primary_cfg)?));+ }+ let mut candidates = Vec::with_capacity(1 + self.llm_fallback_configs.len());+ candidates.push(Candidate::new(+ primary_cfg.provider.name(),+ primary_cfg.model.clone(),+ build_provider(primary_cfg)?,+ ));+ for cfg in &self.llm_fallback_configs {+ candidates.push(Candidate::new(+ cfg.provider.name(),+ cfg.model.clone(),+ build_provider(cfg.clone())?,+ ));+ }+ Ok(Some(Arc::new(FallbackLlmProvider::new(candidates))))+ }+/// OpenAI-compatible embedding key. `EMBEDDING_API_KEY` is checked first/// so an operator can point the embedder at one provider while the LLM/// uses another; without it, direct OpenAI keeps requiring@@ -1387,6 +1587,21 @@ impl Config {}}+/// Wire-name parsing shared by `llm_provider` and `llm_fallbacks[].provider`.+fn provider_choice_from_str(raw: &str) -> Option<ProviderChoice> {+ Some(match raw {+ "anthropic" => ProviderChoice::Anthropic,+ "openai" => ProviderChoice::OpenAi,+ "gemini" | "google" => ProviderChoice::Gemini,+ "openai-compat" | "openai_compat" => ProviderChoice::OpenAiCompat,+ "openai-oauth" | "openai_oauth" => ProviderChoice::OpenAiOAuth,+ "copilot" | "github-copilot" | "github_copilot" => ProviderChoice::Copilot,+ "anthropic-oauth" | "anthropic_oauth" => ProviderChoice::AnthropicOAuth,+ "opencode" | "opencode-zen" | "opencode_zen" => ProviderChoice::OpenCode,+ _ => return None,+ })+}+fn env_string(name: &str) -> Option<String> {std::env::var(name).ok().and_then(|s| {let trimmed = s.trim();@@ -2680,6 +2895,213 @@ mod tests {}}+ fn load_with_toml(toml: &str) -> anyhow::Result<Config> {+ let tmp = TempDir::new().unwrap();+ let config_path = tmp.path().join("config.toml");+ std::fs::write(&config_path, toml).unwrap();+ Config::load(Some(&config_path), Some(tmp.path().to_path_buf()))+ }++ #[test]+ fn llm_fallbacks_default_to_empty_and_no_behavior_change() {+ let cfg = Config::default();+ assert!(cfg.llm_fallbacks.is_empty());+ assert!(cfg.llm_fallback_configs.is_empty());+ }++ #[test]+ fn load_rejects_an_empty_fallback_provider() {+ let error = load_with_toml("[[llm_fallbacks]]\nprovider = \"\"\nmodel = \"m\"\n")+ .expect_err("an empty provider must fail closed");+ assert!(+ format!("{error:#}").contains("llm_fallbacks[0].provider must not be empty"),+ "unexpected error: {error:#}"+ );+ }++ #[test]+ fn load_rejects_an_unknown_fallback_provider() {+ let error = load_with_toml(+ "[[llm_fallbacks]]\nprovider = \"not-a-real-provider\"\nmodel = \"m\"\n",+ )+ .expect_err("an unknown provider must fail closed");+ assert!(+ format!("{error:#}").contains("is not one of"),+ "unexpected error: {error:#}"+ );+ }++ #[test]+ fn load_rejects_an_empty_fallback_model() {+ let error = load_with_toml("[[llm_fallbacks]]\nprovider = \"gemini\"\nmodel = \"\"\n")+ .expect_err("an empty model must fail closed");+ assert!(+ format!("{error:#}").contains("llm_fallbacks[0].model must not be empty"),+ "unexpected error: {error:#}"+ );+ }++ #[test]+ fn load_rejects_a_required_api_key_provider_with_no_api_key_env() {+ let error = load_with_toml("[[llm_fallbacks]]\nprovider = \"gemini\"\nmodel = \"m\"\n")+ .expect_err("a RequiredApiKey provider needs api_key_env");+ assert!(+ format!("{error:#}").contains("requires api_key_env"),+ "unexpected error: {error:#}"+ );+ }++ /// The fallback never silently inherits the *primary*'s fixed env var+ /// (`GEMINI_API_KEY`) — it must fail closed instead of looking healthy+ /// until the primary has an outage.+ #[test]+ fn load_rejects_a_fallback_that_would_otherwise_inherit_the_primary_credential() {+ let error = load_with_toml(+ "llm_provider = \"gemini\"\n[[llm_fallbacks]]\nprovider = \"gemini\"\nmodel = \"m\"\n",+ )+ .expect_err("omitting api_key_env must not fall back to GEMINI_API_KEY");+ assert!(+ format!("{error:#}").contains("requires api_key_env"),+ "unexpected error: {error:#}"+ );+ }++ #[test]+ fn load_rejects_a_missing_or_empty_api_key_env_value() {+ let error = load_with_toml(+ "[[llm_fallbacks]]\nprovider = \"gemini\"\nmodel = \"m\"\n\+ api_key_env = \"AI_MEMORY_TEST_FALLBACK_UNSET_KEY_648\"\n",+ )+ .expect_err("an unresolved api_key_env must fail closed");+ assert!(+ format!("{error:#}").contains(+ "llm_fallbacks[0].api_key_env=AI_MEMORY_TEST_FALLBACK_UNSET_KEY_648 is set but \+ the environment variable is missing or empty"+ ),+ "unexpected error: {error:#}"+ );+ }++ #[test]+ fn load_rejects_an_openai_compat_fallback_with_no_base_url() {+ let error =+ load_with_toml("[[llm_fallbacks]]\nprovider = \"openai-compat\"\nmodel = \"m\"\n")+ .expect_err("openai-compat needs a base_url");+ assert!(+ format!("{error:#}").contains("LLM_BASE_URL"),+ "unexpected error: {error:#}"+ );+ }++ #[test]+ fn load_accepts_a_well_formed_fallback_profile_with_no_primary() {+ // `llm_fallbacks` is meaningful even without a primary configured+ // (it is only ever consulted through `llm_provider_chain`, which+ // returns `None` when there is no primary) — the profile itself+ // must still validate independently.+ let cfg = load_with_toml(+ "[[llm_fallbacks]]\nprovider = \"openai-compat\"\nmodel = \"m\"\n\+ base_url = \"http://127.0.0.1:9\"\n",+ )+ .unwrap();+ let fallbacks = &cfg.llm_fallback_configs;+ assert_eq!(fallbacks.len(), 1);+ assert_eq!(fallbacks[0].provider, ProviderChoice::OpenAiCompat);+ assert_eq!(fallbacks[0].model, "m");+ assert_eq!(fallbacks[0].base_url.as_deref(), Some("http://127.0.0.1:9"));+ assert!(cfg.llm_provider_chain().unwrap().is_none());+ }++ /// `fallback_provider_config` takes the resolved credential as a plain+ /// parameter (see its doc comment): the environment is read once, in+ /// `Config::load`, and passed down as data so the happy path stays+ /// directly testable without mutating process env.+ #[test]+ fn fallback_provider_config_resolves_a_directly_supplied_credential() {+ let cfg = Config::default();+ let profile = FallbackProfile {+ provider: "gemini".into(),+ model: "gemini-3.5-flash".into(),+ base_url: None,+ api_key_env: Some("GEMINI_FALLBACK_KEY".into()),+ };+ let provider_cfg = cfg+ .fallback_provider_config(0, &profile, Some(SecretString::from("sk-fallback-test")))+ .unwrap();+ assert_eq!(provider_cfg.provider, ProviderChoice::Gemini);+ assert_eq!(provider_cfg.model, "gemini-3.5-flash");+ assert_eq!(+ provider_cfg.auth.require_api_key().unwrap().expose_secret(),+ "sk-fallback-test"+ );+ }++ /// `openai-oauth`/`copilot`/`anthropic-oauth` are native credential+ /// sources: no `api_key_env` is required, and the fallback shares the+ /// same process-wide token material as the primary.+ #[test]+ fn fallback_provider_config_allows_no_api_key_env_for_a_native_credential_provider() {+ let cfg = Config::default();+ let profile = FallbackProfile {+ provider: "anthropic-oauth".into(),+ model: "claude-sonnet-4-6".into(),+ base_url: None,+ api_key_env: None,+ };+ let provider_cfg = cfg.fallback_provider_config(0, &profile, None).unwrap();+ assert_eq!(provider_cfg.provider, ProviderChoice::AnthropicOAuth);+ }++ #[test]+ fn llm_provider_chain_is_the_plain_provider_when_no_fallback_is_configured() {+ let cfg = Config {+ llm_provider: Some("openai".into()),+ runtime_env: RuntimeEnv {+ openai_api_key: Some(SecretString::from("sk-test-key")),+ ..RuntimeEnv::default()+ },+ ..Config::default()+ };+ let chain = cfg.llm_provider_chain().unwrap().unwrap();+ assert_eq!(chain.name(), "openai");+ // A plain provider reports no candidates — only a fallback chain does.+ assert!(chain.candidate_health().is_empty());+ }++ /// End-to-end happy path through `Config::load` -> `llm_provider_chain`:+ /// a keyless `openai-compat` primary plus one keyless `openai-compat`+ /// fallback (distinct base URLs, as `docs/llm-provider-fallback.md`'s+ /// example does with distinct providers) produces a `FallbackLlmProvider`+ /// exposing both candidates, in declaration order.+ #[test]+ fn llm_provider_chain_wraps_the_primary_and_every_fallback_in_order() {+ let cfg = load_with_toml(+ "llm_provider = \"openai-compat\"\n\+ llm_model = \"primary-model\"\n\+ llm_base_url = \"http://127.0.0.1:9\"\n\+ [[llm_fallbacks]]\n\+ provider = \"openai-compat\"\n\+ model = \"fallback-model\"\n\+ base_url = \"http://127.0.0.1:9\"\n",+ )+ .unwrap();++ let chain = cfg.llm_provider_chain().unwrap().unwrap();+ assert_eq!(chain.name(), "openai-compat");+ assert_eq!(chain.model(), "primary-model");+ let candidates = chain.candidate_health();+ assert_eq!(candidates.len(), 2);+ assert_eq!(candidates[0].provider, "openai-compat");+ assert_eq!(candidates[0].model, "primary-model");+ assert_eq!(candidates[1].provider, "openai-compat");+ assert_eq!(candidates[1].model, "fallback-model");+ }++ #[test]+ fn llm_provider_chain_none_when_no_llm_is_configured() {+ assert!(Config::default().llm_provider_chain().unwrap().is_none());+ }+#[test]fn anthropic_oauth_provider_underscore_alias_also_resolves() {let cfg = Config {diff --git a/crates/ai-memory-llm/src/error.rs b/crates/ai-memory-llm/src/error.rsindex 9f9bf918..e78732da 100644--- a/crates/ai-memory-llm/src/error.rs+++ b/crates/ai-memory-llm/src/error.rs@@ -42,6 +42,20 @@ pub enum LlmError {/// JSON schema for structured output could not be derived.#[error("schema: {0}")]Schema(String),++ /// Every eligible candidate in an ordered LLM fallback chain failed, or+ /// every candidate's circuit was open. Carries only provider/model+ /// labels and error classes/status — never response bodies or secrets.+ /// See `docs/llm-provider-fallback.md`.+ #[error("llm fallback chain exhausted after {attempted} candidate(s): {summary}")]+ AllCandidatesFailed {+ /// Number of candidates actually attempted (skipped open circuits+ /// are excluded from this count).+ attempted: usize,+ /// Bounded `provider/model: class[ status]` summary, one entry per+ /// attempt, joined with `"; "`.+ summary: String,+ },}impl LlmError {@@ -63,6 +77,32 @@ impl LlmError {_ => false,}}++ /// HTTP status captured from this error, when the failure carries one.+ #[must_use]+ pub fn http_status(&self) -> Option<u16> {+ match self {+ Self::Http(e) => e.status().map(|status| status.as_u16()),+ Self::Provider { status, .. } => Some(*status),+ _ => None,+ }+ }++ /// Redacted error category used for health reporting and the fallback+ /// chain's aggregate error — never a response body or secret.+ #[must_use]+ pub const fn class(&self) -> &'static str {+ match self {+ Self::Http(_) => "http",+ Self::Provider { .. } => "provider",+ Self::Serde(_) => "serde",+ Self::UnexpectedShape(_) => "unexpected-shape",+ Self::NotConfigured(_) => "not-configured",+ Self::Auth(_) => "auth",+ Self::Schema(_) => "schema",+ Self::AllCandidatesFailed { .. } => "all-candidates-failed",+ }+ }}impl From<serde_json::Error> for LlmError {@@ -115,4 +155,61 @@ mod tests {assert!(!LlmError::UnexpectedShape("no tool block".into()).is_transient());assert!(!LlmError::NotConfigured("no key".into()).is_transient());}++ #[test]+ fn all_candidates_failed_is_not_transient() {+ // The aggregate error is terminal: nothing is left to advance to.+ assert!(+ !LlmError::AllCandidatesFailed {+ attempted: 2,+ summary: String::new(),+ }+ .is_transient()+ );+ }++ #[test]+ fn http_status_reads_provider_status_only() {+ assert_eq!(+ LlmError::Provider {+ status: 503,+ body: "boom".into()+ }+ .http_status(),+ Some(503)+ );+ assert_eq!(LlmError::Schema("bad".into()).http_status(), None);+ }++ #[test]+ fn class_is_a_stable_redacted_label_never_the_message() {+ let cases: &[(LlmError, &str)] = &[+ (+ LlmError::Provider {+ status: 500,+ body: "secret upstream body".into(),+ },+ "provider",+ ),+ (LlmError::Serde("nope".into()), "serde"),+ (+ LlmError::UnexpectedShape("no tool block".into()),+ "unexpected-shape",+ ),+ (LlmError::NotConfigured("no key".into()), "not-configured"),+ (LlmError::Auth("expired".into()), "auth"),+ (LlmError::Schema("bad".into()), "schema"),+ (+ LlmError::AllCandidatesFailed {+ attempted: 1,+ summary: String::new(),+ },+ "all-candidates-failed",+ ),+ ];+ for (err, expected) in cases {+ assert_eq!(err.class(), *expected, "unexpected class for {err}");+ assert!(!err.class().contains("secret upstream body"));+ }+ }}diff --git a/crates/ai-memory-llm/src/fallback.rs b/crates/ai-memory-llm/src/fallback.rsnew file mode 100644index 00000000..4186d6d1--- /dev/null+++ b/crates/ai-memory-llm/src/fallback.rs@@ -0,0 +1,753 @@+//! Ordered LLM provider fallback chain.+//!+//! See `docs/llm-provider-fallback.md` (issue #648) for the full design.+//! `FallbackLlmProvider` wraps an ordered list of already-constructed+//! providers behind the same [`LlmProvider`] trait: on a transient failure+//! (`LlmError::is_transient()`) it advances to the next eligible candidate,+//! preserving the original request, schema, and operation id; a+//! deterministic failure (bad request, auth, unsupported schema, malformed+//! response) returns immediately without trying the rest of the chain.+//! `Config::llm_provider_chain` in `ai-memory-cli` is the sole place that+//! builds one; `build_provider` remains the sole construction path for each+//! candidate.++use std::sync::{Arc, Mutex};+use std::time::{Duration, Instant};++use async_trait::async_trait;+use jiff::Timestamp;++use crate::error::{LlmError, LlmResult};+use crate::health::CandidateHealth;+use crate::provider::LlmProvider;+use crate::types::{ChatRequest, ChatResponse, LlmOperationId};++/// How long a candidate's circuit stays open after a transient failure+/// before it is eligible again. Restarting the process clears all circuit+/// state; there is no durable or cross-process cooldown.+pub const CIRCUIT_COOLDOWN: Duration = Duration::from_secs(30);++/// One provider in an ordered fallback chain.+///+/// Carries only the diagnostic labels used by health reporting and circuit+/// tracking — never the raw configuration or credentials that built+/// `inner`. `build_provider` remains the sole construction path; this type+/// only wraps its result.+pub struct Candidate {+ provider: &'static str,+ model: String,+ inner: Arc<dyn LlmProvider>,+}++impl Candidate {+ /// Wrap an already-constructed provider for use in a fallback chain.+ #[must_use]+ pub fn new(+ provider: &'static str,+ model: impl Into<String>,+ inner: Arc<dyn LlmProvider>,+ ) -> Self {+ Self {+ provider,+ model: model.into(),+ inner,+ }+ }+}++/// Per-candidate mutable state: circuit + last-call bookkeeping, shared+/// between the call path and health reporting.+#[derive(Default)]+struct CandidateState {+ last_success_at: Option<Timestamp>,+ last_error_at: Option<Timestamp>,+ last_error_status: Option<u16>,+ last_error_class: Option<&'static str>,+ circuit_open_until: Option<Instant>,+}++struct CandidateEntry {+ candidate: Candidate,+ state: Mutex<CandidateState>,+}++/// Ordered LLM provider fallback chain.+///+/// Tries each eligible candidate in declaration order, advancing only on a+/// transient failure; a deterministic failure returns immediately. See the+/// module docs and `docs/llm-provider-fallback.md`.+pub struct FallbackLlmProvider {+ entries: Vec<CandidateEntry>,+ /// Index of the candidate that answered the most recently completed+ /// call, for health reporting.+ last_selected: Mutex<Option<usize>>,+}++impl FallbackLlmProvider {+ /// Build a chain from an ordered list of candidates (primary first,+ /// then fallbacks in declaration order).+ #[must_use]+ pub fn new(candidates: Vec<Candidate>) -> Self {+ Self {+ entries: candidates+ .into_iter()+ .map(|candidate| CandidateEntry {+ candidate,+ state: Mutex::new(CandidateState::default()),+ })+ .collect(),+ last_selected: Mutex::new(None),+ }+ }++ fn circuit_open(&self, index: usize) -> bool {+ let state = self.entries[index]+ .state+ .lock()+ .unwrap_or_else(|e| e.into_inner());+ matches!(state.circuit_open_until, Some(until) if Instant::now() < until)+ }++ fn record_success(&self, index: usize) {+ {+ let mut state = self.entries[index]+ .state+ .lock()+ .unwrap_or_else(|e| e.into_inner());+ state.last_success_at = Some(Timestamp::now());+ state.circuit_open_until = None;+ }+ *self.last_selected.lock().unwrap_or_else(|e| e.into_inner()) = Some(index);+ }++ fn record_error(&self, index: usize, err: &LlmError) {+ let mut state = self.entries[index]+ .state+ .lock()+ .unwrap_or_else(|e| e.into_inner());+ state.last_error_at = Some(Timestamp::now());+ state.last_error_status = err.http_status();+ state.last_error_class = Some(err.class());+ if err.is_transient() {+ state.circuit_open_until = Some(Instant::now() + CIRCUIT_COOLDOWN);+ }+ }++ fn describe_failure(&self, index: usize, err: &LlmError) -> String {+ let candidate = &self.entries[index].candidate;+ match err.http_status() {+ Some(status) => format!(+ "{}/{}: {} {status}",+ candidate.provider,+ candidate.model,+ err.class()+ ),+ None => format!(+ "{}/{}: {}",+ candidate.provider,+ candidate.model,+ err.class()+ ),+ }+ }++ fn exhausted(&self, attempted: usize, summary: Vec<String>) -> LlmError {+ LlmError::AllCandidatesFailed {+ attempted,+ summary: summary.join("; "),+ }+ }+}++#[async_trait]+impl LlmProvider for FallbackLlmProvider {+ fn name(&self) -> &'static str {+ self.entries+ .first()+ .map_or("fallback-chain-empty", |entry| entry.candidate.provider)+ }++ fn model(&self) -> &str {+ self.entries+ .first()+ .map_or("", |entry| entry.candidate.model.as_str())+ }++ async fn complete(&self, request: ChatRequest) -> LlmResult<ChatResponse> {+ let mut attempted = 0usize;+ let mut summary = Vec::new();+ for (i, entry) in self.entries.iter().enumerate() {+ if self.circuit_open(i) {+ continue;+ }+ attempted += 1;+ match entry.candidate.inner.complete(request.clone()).await {+ Ok(response) => {+ self.record_success(i);+ return Ok(response);+ }+ Err(err) => {+ let transient = err.is_transient();+ summary.push(self.describe_failure(i, &err));+ self.record_error(i, &err);+ if !transient {+ return Err(err);+ }+ }+ }+ }+ Err(self.exhausted(attempted, summary))+ }++ async fn complete_with_operation_id(+ &self,+ request: ChatRequest,+ operation_id: LlmOperationId,+ ) -> LlmResult<ChatResponse> {+ let mut attempted = 0usize;+ let mut summary = Vec::new();+ for (i, entry) in self.entries.iter().enumerate() {+ if self.circuit_open(i) {+ continue;+ }+ attempted += 1;+ match entry+ .candidate+ .inner+ .complete_with_operation_id(request.clone(), operation_id)+ .await+ {+ Ok(response) => {+ self.record_success(i);+ return Ok(response);+ }+ Err(err) => {+ let transient = err.is_transient();+ summary.push(self.describe_failure(i, &err));+ self.record_error(i, &err);+ if !transient {+ return Err(err);+ }+ }+ }+ }+ Err(self.exhausted(attempted, summary))+ }++ async fn complete_structured_raw(+ &self,+ request: ChatRequest,+ schema: serde_json::Value,+ ) -> LlmResult<serde_json::Value> {+ let mut attempted = 0usize;+ let mut summary = Vec::new();+ for (i, entry) in self.entries.iter().enumerate() {+ if self.circuit_open(i) {+ continue;+ }+ attempted += 1;+ match entry+ .candidate+ .inner+ .complete_structured_raw(request.clone(), schema.clone())+ .await+ {+ Ok(value) => {+ self.record_success(i);+ return Ok(value);+ }+ Err(err) => {+ let transient = err.is_transient();+ summary.push(self.describe_failure(i, &err));+ self.record_error(i, &err);+ if !transient {+ return Err(err);+ }+ }+ }+ }+ Err(self.exhausted(attempted, summary))+ }++ async fn complete_structured_raw_with_operation_id(+ &self,+ request: ChatRequest,+ schema: serde_json::Value,+ operation_id: LlmOperationId,+ ) -> LlmResult<serde_json::Value> {+ let mut attempted = 0usize;+ let mut summary = Vec::new();+ for (i, entry) in self.entries.iter().enumerate() {+ if self.circuit_open(i) {+ continue;+ }+ attempted += 1;+ match entry+ .candidate+ .inner+ .complete_structured_raw_with_operation_id(+ request.clone(),+ schema.clone(),+ operation_id,+ )+ .await+ {+ Ok(value) => {+ self.record_success(i);+ return Ok(value);+ }+ Err(err) => {+ let transient = err.is_transient();+ summary.push(self.describe_failure(i, &err));+ self.record_error(i, &err);+ if !transient {+ return Err(err);+ }+ }+ }+ }+ Err(self.exhausted(attempted, summary))+ }++ fn candidate_health(&self) -> Vec<CandidateHealth> {+ let selected = *self.last_selected.lock().unwrap_or_else(|e| e.into_inner());+ self.entries+ .iter()+ .enumerate()+ .map(|(i, entry)| {+ let state = entry.state.lock().unwrap_or_else(|e| e.into_inner());+ let now = Instant::now();+ CandidateHealth {+ provider: entry.candidate.provider.to_string(),+ model: entry.candidate.model.clone(),+ last_selected: selected == Some(i),+ last_success_at: state.last_success_at,+ last_error_at: state.last_error_at,+ last_error_status: state.last_error_status,+ last_error_class: state.last_error_class.map(str::to_string),+ // Only report while the cooldown genuinely has not+ // elapsed yet — a stale past instant (no later call+ // resets it) would otherwise read as "still open".+ circuit_open_until: state.circuit_open_until.and_then(|until| {+ (until > now)+ .then(|| Timestamp::now() + until.saturating_duration_since(now))+ }),+ }+ })+ .collect()+ }+}++#[cfg(test)]+mod tests {+ use std::sync::atomic::{AtomicUsize, Ordering};++ use super::*;+ use crate::types::{ChatMessage, Usage};++ /// A scripted provider double: returns each configured outcome in+ /// order (repeating the last one once exhausted), and records every+ /// request/schema/operation id it was called with.+ struct ScriptedLlm {+ name: &'static str,+ model: String,+ outcomes: Mutex<Vec<LlmResult<ChatResponse>>>,+ calls: AtomicUsize,+ seen_requests: Mutex<Vec<ChatRequest>>,+ seen_schemas: Mutex<Vec<serde_json::Value>>,+ seen_operation_ids: Mutex<Vec<Option<LlmOperationId>>>,+ }++ impl ScriptedLlm {+ fn new(name: &'static str, model: &str, outcomes: Vec<LlmResult<ChatResponse>>) -> Self {+ Self {+ name,+ model: model.to_string(),+ outcomes: Mutex::new(outcomes),+ calls: AtomicUsize::new(0),+ seen_requests: Mutex::new(Vec::new()),+ seen_schemas: Mutex::new(Vec::new()),+ seen_operation_ids: Mutex::new(Vec::new()),+ }+ }++ fn ok(name: &'static str, model: &str, text: &str) -> Self {+ Self::new(+ name,+ model,+ vec![Ok(ChatResponse {+ text: text.to_string(),+ usage: Some(Usage::default()),+ model: model.to_string(),+ })],+ )+ }++ fn err(name: &'static str, model: &str, err: LlmError) -> Self {+ Self::new(name, model, vec![Err(err)])+ }++ fn calls(&self) -> usize {+ self.calls.load(Ordering::SeqCst)+ }++ fn next_outcome(&self) -> LlmResult<ChatResponse> {+ self.calls.fetch_add(1, Ordering::SeqCst);+ let mut outcomes = self.outcomes.lock().unwrap();+ if outcomes.len() > 1 {+ outcomes.remove(0)+ } else {+ clone_outcome(&outcomes[0])+ }+ }++ fn record(&self, request: &ChatRequest, operation_id: Option<LlmOperationId>) {+ self.seen_requests.lock().unwrap().push(request.clone());+ self.seen_operation_ids.lock().unwrap().push(operation_id);+ }+ }++ fn clone_outcome(outcome: &LlmResult<ChatResponse>) -> LlmResult<ChatResponse> {+ match outcome {+ Ok(response) => Ok(response.clone()),+ Err(err) => Err(clone_err(err)),+ }+ }++ fn clone_err(err: &LlmError) -> LlmError {+ match err {+ LlmError::Provider { status, body } => LlmError::Provider {+ status: *status,+ body: body.clone(),+ },+ LlmError::Serde(msg) => LlmError::Serde(msg.clone()),+ LlmError::UnexpectedShape(msg) => LlmError::UnexpectedShape(msg.clone()),+ LlmError::NotConfigured(msg) => LlmError::NotConfigured(msg.clone()),+ LlmError::Auth(msg) => LlmError::Auth(msg.clone()),+ LlmError::Schema(msg) => LlmError::Schema(msg.clone()),+ other => panic!("clone_err: unsupported variant in test double: {other}"),+ }+ }++ #[async_trait]+ impl LlmProvider for ScriptedLlm {+ fn name(&self) -> &'static str {+ self.name+ }++ fn model(&self) -> &str {+ &self.model+ }++ async fn complete(&self, request: ChatRequest) -> LlmResult<ChatResponse> {+ self.record(&request, None);+ self.next_outcome()+ }++ async fn complete_with_operation_id(+ &self,+ request: ChatRequest,+ operation_id: LlmOperationId,+ ) -> LlmResult<ChatResponse> {+ self.record(&request, Some(operation_id));+ self.next_outcome()+ }++ async fn complete_structured_raw(+ &self,+ request: ChatRequest,+ schema: serde_json::Value,+ ) -> LlmResult<serde_json::Value> {+ self.record(&request, None);+ self.seen_schemas.lock().unwrap().push(schema);+ self.next_outcome()+ .map(|response| serde_json::json!({ "text": response.text }))+ }++ async fn complete_structured_raw_with_operation_id(+ &self,+ request: ChatRequest,+ schema: serde_json::Value,+ operation_id: LlmOperationId,+ ) -> LlmResult<serde_json::Value> {+ self.record(&request, Some(operation_id));+ self.seen_schemas.lock().unwrap().push(schema);+ self.next_outcome()+ .map(|response| serde_json::json!({ "text": response.text }))+ }+ }++ fn transient(status: u16) -> LlmError {+ LlmError::Provider {+ status,+ body: "upstream body that must never leak".to_string(),+ }+ }++ #[tokio::test]+ async fn chain_tries_candidates_in_order_and_stops_at_first_success() {+ let first = Arc::new(ScriptedLlm::err("anthropic", "claude", transient(503)));+ let second = Arc::new(ScriptedLlm::ok("gemini", "gemini-flash", "second answer"));+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("anthropic", "claude", first.clone()),+ Candidate::new("gemini", "gemini-flash", second.clone()),+ ]);++ let response = chain+ .complete(ChatRequest::user_prompt("hi"))+ .await+ .unwrap();++ assert_eq!(response.text, "second answer");+ assert_eq!(first.calls(), 1);+ assert_eq!(second.calls(), 1);+ }++ #[tokio::test]+ async fn all_four_methods_preserve_request_schema_and_operation_id() {+ let first = Arc::new(ScriptedLlm::err("primary", "m1", transient(500)));+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "ok"));+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("primary", "m1", first.clone()),+ Candidate::new("secondary", "m2", second.clone()),+ ]);+ let request = ChatRequest::user_prompt("preserve me").with_max_tokens(42);+ let schema = serde_json::json!({ "type": "object" });+ let op_id = LlmOperationId::new();++ chain.complete(request.clone()).await.unwrap();+ chain+ .complete_with_operation_id(request.clone(), op_id)+ .await+ .unwrap();+ chain+ .complete_structured_raw(request.clone(), schema.clone())+ .await+ .unwrap();+ chain+ .complete_structured_raw_with_operation_id(request.clone(), schema.clone(), op_id)+ .await+ .unwrap();++ // Every call advanced past `first` (transient) onto `second`, which+ // must have seen the exact same request/schema/operation id on+ // each of the four entry points.+ assert_eq!(second.calls(), 4);+ let seen_requests = second.seen_requests.lock().unwrap();+ assert!(seen_requests.iter().all(+ |r| matches!(&r.messages[..], [ChatMessage { content, .. }] if content == "preserve me")+ && r.max_tokens == 42+ ));+ let seen_schemas = second.seen_schemas.lock().unwrap();+ assert_eq!(seen_schemas.len(), 2);+ assert!(seen_schemas.iter().all(|s| *s == schema));+ let seen_ops = second.seen_operation_ids.lock().unwrap();+ // complete/complete_structured_raw pass None; the *_with_operation_id+ // calls pass the same op_id every time.+ assert_eq!(*seen_ops, vec![None, Some(op_id), None, Some(op_id)]);+ }++ #[tokio::test]+ async fn transient_statuses_advance_to_the_next_candidate() {+ for status in [429, 500, 502, 503, 504] {+ let first = Arc::new(ScriptedLlm::err("primary", "m1", transient(status)));+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "ok"));+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("primary", "m1", first.clone()),+ Candidate::new("secondary", "m2", second.clone()),+ ]);++ let result = chain.complete(ChatRequest::user_prompt("hi")).await;+ assert!(result.is_ok(), "status {status} should have advanced");+ assert_eq!(second.calls(), 1, "status {status} should have advanced");+ }+ }++ #[tokio::test]+ async fn deterministic_errors_stop_on_the_first_candidate() {+ let deterministic = [+ LlmError::Provider {+ status: 400,+ body: String::new(),+ },+ LlmError::Provider {+ status: 401,+ body: String::new(),+ },+ LlmError::Provider {+ status: 403,+ body: String::new(),+ },+ LlmError::Provider {+ status: 404,+ body: String::new(),+ },+ LlmError::Provider {+ status: 422,+ body: String::new(),+ },+ LlmError::Schema("bad schema".into()),+ LlmError::Serde("deserialize failed".into()),+ LlmError::UnexpectedShape("no tool block".into()),+ ];+ for err in deterministic {+ let expected_class = err.class();+ let first = Arc::new(ScriptedLlm::err("primary", "m1", err));+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "unreachable"));+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("primary", "m1", first.clone()),+ Candidate::new("secondary", "m2", second.clone()),+ ]);++ let result = chain.complete(ChatRequest::user_prompt("hi")).await;+ let returned = result.expect_err("deterministic error must propagate, not succeed");+ assert_eq!(returned.class(), expected_class);+ assert_eq!(first.calls(), 1);+ assert_eq!(+ second.calls(),+ 0,+ "class {expected_class} must not try the next candidate"+ );+ }+ }++ #[tokio::test]+ async fn all_candidates_failing_returns_a_bounded_aggregate_error() {+ let first = Arc::new(ScriptedLlm::err(+ "primary",+ "m1",+ transient_with_body(500, "primary secret body"),+ ));+ let second = Arc::new(ScriptedLlm::err(+ "secondary",+ "m2",+ transient_with_body(503, "secondary secret body"),+ ));+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("primary", "m1", first.clone()),+ Candidate::new("secondary", "m2", second.clone()),+ ]);++ let err = chain+ .complete(ChatRequest::user_prompt("hi"))+ .await+ .expect_err("every candidate failed");+ match &err {+ LlmError::AllCandidatesFailed { attempted, summary } => {+ assert_eq!(*attempted, 2);+ assert!(summary.contains("primary/m1"));+ assert!(summary.contains("secondary/m2"));+ assert!(summary.contains("500"));+ assert!(summary.contains("503"));+ assert!(!summary.contains("secret body"));+ }+ other => panic!("expected AllCandidatesFailed, got {other}"),+ }+ }++ fn transient_with_body(status: u16, body: &str) -> LlmError {+ LlmError::Provider {+ status,+ body: body.to_string(),+ }+ }++ #[tokio::test]+ async fn an_open_circuit_skips_only_its_candidate() {+ let first = Arc::new(ScriptedLlm::err("primary", "m1", transient(500)));+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "ok"));+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("primary", "m1", first.clone()),+ Candidate::new("secondary", "m2", second.clone()),+ ]);++ // First call: primary fails transiently and opens its circuit;+ // secondary answers.+ chain+ .complete(ChatRequest::user_prompt("hi"))+ .await+ .unwrap();+ assert_eq!(first.calls(), 1);+ assert_eq!(second.calls(), 1);++ // Second call, still inside the cooldown: primary must be skipped+ // outright (not attempted again), secondary answers again.+ chain+ .complete(ChatRequest::user_prompt("hi"))+ .await+ .unwrap();+ assert_eq!(first.calls(), 1, "an open circuit must not be attempted");+ assert_eq!(second.calls(), 2);+ }++ #[test]+ fn a_success_closes_an_open_circuit_immediately() {+ let chain = FallbackLlmProvider::new(vec![Candidate::new(+ "primary",+ "m1",+ Arc::new(ScriptedLlm::ok("primary", "m1", "ok")),+ )]);+ // Simulate an open circuit far in the future — a real cooldown, not+ // yet elapsed.+ {+ let mut state = chain.entries[0].state.lock().unwrap();+ state.circuit_open_until = Some(Instant::now() + Duration::from_secs(600));+ }+ assert!(chain.circuit_open(0));++ chain.record_success(0);++ assert!(+ !chain.circuit_open(0),+ "a success must close the circuit immediately, not wait out the cooldown"+ );+ }++ #[test]+ fn an_elapsed_cooldown_closes_the_circuit_without_an_explicit_success() {+ let chain = FallbackLlmProvider::new(vec![Candidate::new(+ "primary",+ "m1",+ Arc::new(ScriptedLlm::ok("primary", "m1", "ok")),+ )]);+ {+ let mut state = chain.entries[0].state.lock().unwrap();+ state.circuit_open_until = Some(Instant::now() - Duration::from_secs(1));+ }+ assert!(!chain.circuit_open(0));+ }++ #[tokio::test]+ async fn candidate_health_reports_labels_last_selected_and_redacted_errors() {+ let first = Arc::new(ScriptedLlm::err(+ "primary",+ "m1",+ transient_with_body(500, "must not leak"),+ ));+ let second = Arc::new(ScriptedLlm::ok("secondary", "m2", "ok"));+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("primary", "m1", first),+ Candidate::new("secondary", "m2", second),+ ]);++ chain+ .complete(ChatRequest::user_prompt("hi"))+ .await+ .unwrap();++ let health = chain.candidate_health();+ assert_eq!(health.len(), 2);+ assert_eq!(health[0].provider, "primary");+ assert_eq!(health[0].model, "m1");+ assert!(!health[0].last_selected);+ assert_eq!(health[0].last_error_status, Some(500));+ assert_eq!(health[0].last_error_class.as_deref(), Some("provider"));+ assert!(health[0].circuit_open_until.is_some());+ assert!(health[1].last_selected);+ assert!(health[1].last_success_at.is_some());+ assert!(health[1].circuit_open_until.is_none());++ let rendered = format!("{health:?}");+ assert!(!rendered.contains("must not leak"));+ }+}diff --git a/crates/ai-memory-llm/src/health.rs b/crates/ai-memory-llm/src/health.rsindex 0860da57..6fd7ceb3 100644--- a/crates/ai-memory-llm/src/health.rs+++ b/crates/ai-memory-llm/src/health.rs@@ -21,14 +21,28 @@ const MAX_ERROR_MESSAGE_CHARS: usize = 1024;pub struct ProviderHealth {llm: ProviderRoleHealth,embedding: ProviderRoleHealth,+ /// The configured LLM provider, held only so `snapshot()` can read its+ /// [`LlmProvider::candidate_health`] — a plain provider always returns+ /// empty here, a [`crate::fallback::FallbackLlmProvider`] returns its+ /// ordered candidate list. Never dereferenced for anything but that+ /// read, so this adds no behavior beyond passive reporting.+ llm_provider: Arc<Mutex<Option<Arc<dyn LlmProvider>>>>,}impl ProviderHealth {/// Return a serializable snapshot of the latest recorded state.#[must_use]pub fn snapshot(&self) -> ProviderHealthSnapshot {+ let llm_candidates = self+ .llm_provider+ .lock()+ .unwrap_or_else(|e| e.into_inner())+ .as_ref()+ .map(|provider| provider.candidate_health())+ .unwrap_or_default();ProviderHealthSnapshot {llm: self.llm.snapshot(),+ llm_candidates,embedding: self.embedding.snapshot(),}}@@ -44,6 +58,7 @@ impl ProviderHealth {) -> Arc<dyn LlmProvider> {let health = self.llm.clone();health.configure(provider.into(), model.into(), None, retry_hint);+ *self.llm_provider.lock().unwrap_or_else(|e| e.into_inner()) = Some(inner.clone());Arc::new(HealthRecordingLlmProvider { inner, health })}@@ -95,10 +110,46 @@ impl ProviderHealthStatus {pub struct ProviderHealthSnapshot {/// LLM provider role.pub llm: ProviderRoleHealthSnapshot,+ /// Ordered LLM fallback-chain candidate state (`llm_fallbacks`), in+ /// declaration order (index 0 is the primary provider). Empty for a+ /// single configured provider or no provider at all.+ /// `#[serde(default)]` so a response from a server built before this+ /// field existed still deserializes for a newer CLI.+ #[serde(default)]+ pub llm_candidates: Vec<CandidateHealth>,/// Embedding provider role.pub embedding: ProviderRoleHealthSnapshot,}+/// One candidate's passive health state in an ordered LLM fallback chain.+/// See `docs/llm-provider-fallback.md`. Every entry here is by construction+/// a configured chain member; redaction is total — only labels, timestamps,+/// an HTTP status, and a short error class are recorded, never a response+/// body or credential.+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]+pub struct CandidateHealth {+ /// Candidate provider label.+ pub provider: String,+ /// Candidate model id.+ pub model: String,+ /// Whether this candidate answered the most recently completed chain+ /// call.+ pub last_selected: bool,+ /// Timestamp of the last call this candidate answered successfully.+ pub last_success_at: Option<Timestamp>,+ /// Timestamp of the last call this candidate failed.+ pub last_error_at: Option<Timestamp>,+ /// HTTP status captured from the last error, when available.+ pub last_error_status: Option<u16>,+ /// Redacted error class captured from the last error+ /// ([`LlmError::class`]), never a response body or secret.+ pub last_error_class: Option<String>,+ /// When this candidate's circuit reopens, if a transient failure has it+ /// currently open. `None` once the cooldown has elapsed, even if no+ /// later call has explicitly closed it.+ pub circuit_open_until: Option<Timestamp>,+}+/// Wire-format health snapshot for one provider role.#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]pub struct ProviderRoleHealthSnapshot {@@ -303,6 +354,10 @@ impl LlmProvider for HealthRecordingLlmProvider {self.health.record_result(&result);result}++ fn candidate_health(&self) -> Vec<CandidateHealth> {+ self.inner.candidate_health()+ }}struct HealthRecordingEmbedder {diff --git a/crates/ai-memory-llm/src/lib.rs b/crates/ai-memory-llm/src/lib.rsindex 5e4d4f32..1ba6307d 100644--- a/crates/ai-memory-llm/src/lib.rs+++ b/crates/ai-memory-llm/src/lib.rs@@ -60,6 +60,7 @@ pub mod copilot;pub mod embedding;pub mod error;pub mod factory;+pub mod fallback;pub mod gemini;pub mod google;pub mod health;@@ -94,10 +95,12 @@ pub use factory::{EmbedderChoice, EmbedderConfig, ProviderChoice, ProviderConfig, build_embedder, build_provider,default_embedding_dim, try_default_embedding_dim,};+pub use fallback::{CIRCUIT_COOLDOWN, Candidate, FallbackLlmProvider};pub use gemini::GeminiProvider;pub use google::{DEFAULT_MODEL as GOOGLE_DEFAULT_EMBED_MODEL, GoogleEmbedder};pub use health::{- ProviderHealth, ProviderHealthSnapshot, ProviderHealthStatus, ProviderRoleHealthSnapshot,+ CandidateHealth, ProviderHealth, ProviderHealthSnapshot, ProviderHealthStatus,+ ProviderRoleHealthSnapshot,};#[cfg(feature = "local-embeddings")]pub use local::{LOCAL_DIM, LOCAL_MODEL, LocalEmbedder, fetch_model, model_present};diff --git a/crates/ai-memory-llm/src/provider.rs b/crates/ai-memory-llm/src/provider.rsindex 63336b69..7ba2929b 100644--- a/crates/ai-memory-llm/src/provider.rs+++ b/crates/ai-memory-llm/src/provider.rs@@ -65,6 +65,19 @@ pub trait LlmProvider: Send + Sync {) -> LlmResult<serde_json::Value> {self.complete_structured_raw(request, schema).await}++ /// Per-candidate health entries for an ordered LLM fallback chain.+ ///+ /// Empty for a plain, non-chained provider (the default). A wrapping+ /// decorator (health recording) must forward this to its inner+ /// provider; [`FallbackLlmProvider`](crate::fallback::FallbackLlmProvider)+ /// is the only implementation that returns a non-empty list.+ /// [`ProviderHealth`](crate::health::ProviderHealth) reads this at+ /// snapshot time — it never probes providers or generates traffic on+ /// its own.+ fn candidate_health(&self) -> Vec<crate::health::CandidateHealth> {+ Vec::new()+ }}/// Typed wrapper around [`LlmProvider::complete_structured_raw`].diff --git a/crates/ai-memory-llm/tests/suite/fallback_provider.rs b/crates/ai-memory-llm/tests/suite/fallback_provider.rsnew file mode 100644index 00000000..b310abdc--- /dev/null+++ b/crates/ai-memory-llm/tests/suite/fallback_provider.rs@@ -0,0 +1,111 @@+//! Integration coverage for `FallbackLlmProvider` against real network+//! failures: a genuine connection-refused and a genuine client-side+//! timeout, each produced by a real `reqwest::Error` rather than a+//! hand-built `LlmError`. `fallback.rs`'s own unit tests cover the pure+//! chain-order/circuit/redaction logic with fake providers; this file+//! proves `LlmError::is_transient()` sees those two real failure classes+//! the same way when they arrive through an actual `OpenAiCompatProvider`+//! HTTP call, matching `docs/llm-provider-fallback.md`'s delivery-sequence+//! scenario (a candidate that first fails transiently, then a working one).++use std::time::Duration;++use ai_memory_llm::types::ChatRequest;+use ai_memory_llm::{Candidate, FallbackLlmProvider, LlmProvider, OpenAiCompatProvider};+use wiremock::matchers::{method, path};+use wiremock::{Mock, MockServer, ResponseTemplate};++fn compat_provider(+ base_url: impl Into<String>,+ model: &str,+ timeout_secs: u64,+) -> OpenAiCompatProvider {+ OpenAiCompatProvider::new(base_url.into(), None, model)+ .expect("provider builds")+ .with_timeout_secs(timeout_secs)+}++fn ok_response_body(content: &str) -> serde_json::Value {+ serde_json::json!({+ "id": "id",+ "object": "chat.completion",+ "created": 0,+ "model": "second-model",+ "choices": [{+ "index": 0,+ "message": { "role": "assistant", "content": content },+ "finish_reason": "stop",+ }],+ "usage": { "prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2 },+ })+}++/// A closed loopback port refuses the connection immediately, producing a+/// real `reqwest::Error` with `.is_connect() == true` — no wiremock or+/// external network access needed.+async fn closed_port_url() -> String {+ let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("bind an ephemeral port");+ let addr = listener.local_addr().expect("local addr");+ drop(listener); // nothing listens on `addr` from here on+ format!("http://{addr}/v1")+}++#[tokio::test]+async fn a_real_connection_error_advances_to_the_next_candidate() {+ let unreachable_base = closed_port_url().await;+ let working = MockServer::start().await;+ Mock::given(method("POST"))+ .and(path("/v1/chat/completions"))+ .respond_with(ResponseTemplate::new(200).set_body_json(ok_response_body("second answer")))+ .mount(&working)+ .await;++ let first = compat_provider(unreachable_base, "first-model", 5);+ let second = compat_provider(working.uri(), "second-model", 5);+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("openai-compat", "first-model", std::sync::Arc::new(first)),+ Candidate::new("openai-compat", "second-model", std::sync::Arc::new(second)),+ ]);++ let response = chain+ .complete(ChatRequest::user_prompt("hi"))+ .await+ .expect("a real connection error must advance to the next candidate");+ assert_eq!(response.text, "second answer");+}++#[tokio::test]+async fn a_real_client_timeout_advances_to_the_next_candidate() {+ let slow = MockServer::start().await;+ Mock::given(method("POST"))+ .and(path("/v1/chat/completions"))+ .respond_with(+ ResponseTemplate::new(200)+ .set_body_json(ok_response_body("too slow"))+ .set_delay(Duration::from_secs(3)),+ )+ .mount(&slow)+ .await;+ let working = MockServer::start().await;+ Mock::given(method("POST"))+ .and(path("/v1/chat/completions"))+ .respond_with(ResponseTemplate::new(200).set_body_json(ok_response_body("second answer")))+ .mount(&working)+ .await;++ // `with_timeout_secs` only accepts whole seconds, so the mock's+ // artificial delay is set well beyond the 1s client timeout — a real+ // `reqwest::Error::is_timeout() == true`, not a race.+ let first = compat_provider(slow.uri(), "first-model", 1);+ let second = compat_provider(working.uri(), "second-model", 5);+ let chain = FallbackLlmProvider::new(vec![+ Candidate::new("openai-compat", "first-model", std::sync::Arc::new(first)),+ Candidate::new("openai-compat", "second-model", std::sync::Arc::new(second)),+ ]);++ let response = chain+ .complete(ChatRequest::user_prompt("hi"))+ .await+ .expect("a real client timeout must advance to the next candidate");+ assert_eq!(response.text, "second answer");+}diff --git a/crates/ai-memory-llm/tests/suite/mod.rs b/crates/ai-memory-llm/tests/suite/mod.rsindex 0337bf86..5a50f954 100644--- a/crates/ai-memory-llm/tests/suite/mod.rs+++ b/crates/ai-memory-llm/tests/suite/mod.rs@@ -3,5 +3,6 @@//! no extra binary; a new file must be declared below.mod extra_headers_on_the_wire;+mod fallback_provider;mod openai_compat_embedder;mod openai_compat_strict;diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 2c581cbc..124fb98a 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -626,6 +626,55 @@ GITHUB_COPILOT_API_TOKEN optional pre-minted Copilot API tokenCOPILOT_API_URL optional Copilot API base URL override```+**Ordered LLM fallback chain** (opt-in, TOML only — see+`docs/llm-provider-fallback.md`, #648): the primary provider above always+runs first; `[[llm_fallbacks]]` entries run only after a transient failure+(429, 5xx, timeout, connection error), in declaration order, with the+original request/schema/operation id preserved on every attempt. A+deterministic failure (4xx other than 429, an unsupported schema, a+malformed response) still stops on the first candidate — no chain-wide+retry loop.++```toml+llm_provider = "opencode"+llm_model = "mimo-v2.5-free"++[[llm_fallbacks]]+provider = "openai-compat" # same wire names as llm_provider+model = "poolside/laguna-s-2.1-free"+base_url = "http://127.0.0.1:49375/v1" # required for openai-compat, as above+api_key_env = "AI_MEMORY_LOCAL_ROUTER_TOKEN" # env var *name*; the key itself+ # never lives in config.toml++[[llm_fallbacks]]+provider = "gemini"+model = "gemini-3.5-flash"+api_key_env = "GEMINI_API_KEY"+```++`api_key_env` is required for any provider that needs an API key+(`anthropic`, `openai`, `gemini`, `opencode`; optional for `openai-compat`,+which may run keyless) — it is never inherited from the primary's own+fixed env var, so a fallback cannot look configured while actually+resolving no credential. It is optional only for a provider with a native+credential source (`openai-oauth`, `copilot`, `anthropic-oauth`), which+shares the primary's process-wide token material. `Config::load` validates+every profile and resolves its credential once, at startup: a+missing/empty provider or model, an unknown provider, or a missing+credential fails startup rather than leaving a latent fallback that only+fails once the primary is already down. Each candidate carries its own+30s in-memory circuit (`ai_memory_llm::fallback::CIRCUIT_COOLDOWN`): a+transient failure opens it, a success closes it, and a restart clears all+circuit state — there is no durable circuit or forced chain-wide deadline.++`GET /admin/status` (`ai-memory status`) reports an `llm_candidates` list+alongside the existing `llm`/`embedding` roles: each candidate's+provider/model label, whether it answered the most recently completed+call, its last success/error timestamp, a redacted error class + HTTP+status (never a response body or credential), and its circuit-open-until+timestamp. It is empty for a plain single-provider setup; the top-level+`llm` role fields are unchanged.+Every chat request carries `User-Agent: ai-memory/<version>`(`ai_memory_llm::DEFAULT_USER_AGENT`, layered in `build_provider`). `reqwest`sends no user agent unless configured, so provider requests used to arrive