Skip to content

CubeLoop — a Pythonic, async-native agent harness core designed for high performance, readability, and production-grade persistence.

docs: document FallbackBoundModel in CHANGELOG, README, recipe, and providers guide

xf gong32063a53b2d5
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 707ec71..4b61510 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **`FallbackBoundModel`** — built-in failover chain at the `BoundModel` level.
+ Wrap an ordered `chain` of `BoundModel` instances; on `RateLimited`,
+ `ProviderUnavailable`, or `ContextLengthExceeded` (configurable via
+ `trigger_errors`), or on a first-event stream error, the next model in the
+ chain is tried transparently. Optional `on_failover` callback for
+ billing/metrics hooks. Exported from `cubepi` and `cubepi.providers`.
+
+- **`DEFAULT_TRIGGER_ERRORS`** — `frozenset({RateLimited, ProviderUnavailable,
+ ContextLengthExceeded})`. The default set of error types that trigger failover
+ in `FallbackBoundModel`.
+
- **`BoundModel.generate()` / `BoundModel.stream()`** — the handle returned by
`provider.model(...)` now drives a provider call directly. Useful for
utilities (summarizers, classifiers) where you already hold a `BoundModel`
diff --git a/README.md b/README.md
index 9ceaa19..7494306 100644
--- a/README.md
+++ b/README.md
@@ -100,6 +100,21 @@ faux = FauxProvider(provider_id="faux")
faux.set_responses(["Hello!", "How can I help?"])
```
+Use `FallbackBoundModel` to chain providers — on a rate limit, outage, or
+context-length error the next model in the chain is tried automatically:
+
+```python
+from cubepi import FallbackBoundModel
+
+model = FallbackBoundModel(
+ chain=(
+ anthropic.model("claude-opus-4-8"), # primary
+ openai.model("gpt-5"), # fallback
+ )
+)
+agent = Agent(model=model, system_prompt="...")
+```
+
### Tools
Decorate an async function with `@tool`: the input schema is derived from the
diff --git a/examples/multi_provider_failover.py b/examples/multi_provider_failover.py
index 7b65b59..e9ae265 100644
--- a/examples/multi_provider_failover.py
+++ b/examples/multi_provider_failover.py
@@ -1,150 +1,54 @@
"""Multi-provider failover — recipe example.
-Demonstrates a FailoverProvider wrapper that peeks at the first stream event
-from each inner provider and falls over to the next on error.
+Demonstrates FallbackBoundModel: the primary provider is given a bad key so its
+first stream event is an auth error, which triggers transparent failover to the
+real provider.
uv run python examples/multi_provider_failover.py
-The example constructs a primary provider with a bad key to force a failover to
-the real provider, proving the mechanism works end-to-end.
-
Set ANTHROPIC_API_KEY or OPENAI_API_KEY (and optionally MODEL) before running.
See _provider.py for details.
"""
+from __future__ import annotations
+
import asyncio
-import logging
import os
-import time
-from cubepi import Agent
-from cubepi.providers.openai import OpenAIProvider
+from cubepi import Agent, FallbackBoundModel
from cubepi.providers.anthropic import AnthropicProvider
-from cubepi.providers.base import (
- AssistantMessage,
- BaseProvider,
- BoundModel,
- Message,
- MessageStream,
- Model,
- StreamEvent,
- StreamOptions,
- ToolDefinition,
- Usage,
-)
+from cubepi.providers.base import BoundModel
+from cubepi.providers.openai import OpenAIProvider
from _provider import MODEL_ID, provider
-log = logging.getLogger(__name__)
-
-
-class FailoverProvider(BaseProvider):
- """Try providers in order; fall over on construction or first-event errors.
-
- Built-in providers swallow API/network errors and surface them as
- StreamEvent(type="error") on the returned stream. We peek at the first
- event from each inner stream and only commit once we see a non-error event.
-
- Limitation: errors that arrive *after* the first event are forwarded as-is.
- """
-
- def __init__(self, primary: BoundModel, *fallbacks: BoundModel) -> None:
- super().__init__(provider_id=primary.spec.provider_id)
- self._chain: list[BoundModel] = [primary, *fallbacks]
-
- async def stream(
- self,
- model: Model,
- messages: list[Message],
- *,
- system_prompt: str = "",
- tools: list[ToolDefinition] | None = None,
- options: StreamOptions | None = None,
- ) -> MessageStream:
- last_error: str | None = None
-
- for bound_model in self._chain:
- inner_provider = bound_model.provider
- mapped_model = bound_model.spec
- try:
- inner = await inner_provider.stream(
- mapped_model,
- messages,
- system_prompt=system_prompt,
- tools=tools,
- options=options,
- )
- except Exception as e:
- log.warning("provider %s failed at construction: %s", mapped_model.provider_id, e)
- last_error = repr(e)
- continue
-
- iterator = inner.__aiter__()
- try:
- first = await iterator.__anext__()
- except StopAsyncIteration:
- last_error = "stream ended before producing any events"
- continue
-
- if first.type == "error":
- log.warning(
- "provider %s errored on first event: %s",
- mapped_model.provider_id,
- first.error_message,
- )
- last_error = first.error_message or "stream error"
- continue
-
- outer = MessageStream()
-
- async def _forward(first_event=first, src=iterator, src_stream=inner):
- try:
- outer.push(first_event)
- async for ev in src:
- outer.push(ev)
- final = await src_stream.result()
- outer.set_result(final)
- except Exception as exc:
- fallback_msg = AssistantMessage(
- content=[],
- stop_reason="error",
- error_message=str(exc),
- usage=Usage(),
- timestamp=time.time(),
- )
- outer.push(StreamEvent(type="error", error_message=str(exc)))
- outer.set_result(fallback_msg)
-
- outer.attach_task(asyncio.create_task(_forward()))
- return outer
-
- raise RuntimeError(f"all providers exhausted; last error: {last_error!r}")
-
-
-def _bad_key_provider() -> tuple[BaseProvider, str]:
- """Build a provider identical to the real one but with an invalid key."""
+
+def _bad_key_primary() -> BoundModel:
+ """Build a bound model with a bad API key to force a first-event auth error."""
if os.environ.get("ANTHROPIC_API_KEY"):
base_url = os.environ.get("ANTHROPIC_BASE_URL") or None
- return AnthropicProvider(api_key="bad-key", base_url=base_url), MODEL_ID
+ return AnthropicProvider(api_key="bad-key", base_url=base_url).model(MODEL_ID)
else:
base_url = os.environ.get("OPENAI_BASE_URL") or None
- return OpenAIProvider(api_key="bad-key", base_url=base_url), MODEL_ID
+ return OpenAIProvider(api_key="bad-key", base_url=base_url).model(MODEL_ID)
async def main() -> None:
- logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(name)s: %(message)s")
-
- bad_provider, model_id = _bad_key_provider()
-
- # Primary: bad key → errors on first event → triggers failover.
- # Fallback: real provider from _provider.py.
- failover = FailoverProvider(
- bad_provider.model(model_id),
- provider.model(MODEL_ID),
+ # Primary has a bad key → first stream event will be type="error" → failover.
+ # Fallback uses the real provider from _provider.py.
+ model = FallbackBoundModel(
+ chain=(
+ _bad_key_primary(),
+ provider.model(MODEL_ID),
+ ),
+ on_failover=lambda failed, nxt, err: print(
+ f"[failover] {failed.spec.provider_id}/{failed.spec.id}"
+ f" → {nxt.spec.provider_id}/{nxt.spec.id if nxt else '—'}: {err}"
+ ),
)
agent = Agent(
- model=failover.model(MODEL_ID),
+ model=model,
system_prompt="You answer concisely.",
)
diff --git a/website/docs/guides/providers/overview.md b/website/docs/guides/providers/overview.md
index a259686..476b24e 100644
--- a/website/docs/guides/providers/overview.md
+++ b/website/docs/guides/providers/overview.md
@@ -322,9 +322,33 @@ provider, `provider.model("id", ...)` factory, typed `ProviderError`
failures, and a capability descriptor for backend wire differences. See
[Image Generation](./image-generation) for the full guide.
+## Failover chain
+
+`FallbackBoundModel` wraps an ordered chain of `BoundModel` instances. On a
+`RateLimited`, `ProviderUnavailable`, or `ContextLengthExceeded` error — or on
+a first-event stream error — it transparently tries the next model:
+
+```python
+from cubepi import FallbackBoundModel
+
+model = FallbackBoundModel(
+ chain=(
+ anthropic.model("claude-opus-4-8"), # primary
+ openai.model("gpt-5"), # fallback
+ )
+)
+agent = Agent(model=model, ...)
+```
+
+`trigger_errors` is configurable; an optional `on_failover` callback fires on
+each switchover for billing/metrics. See the
+[Multi-Provider Failover recipe](../../recipes/multi-provider-failover) for the
+full guide.
+
## See also
- [OpenAI Provider](./openai) — concrete OpenAI / OpenAI-compatible setup.
- [Anthropic Provider](./anthropic) — the `int_budget` reasoning shape in practice.
- [Writing a Custom Provider](./custom) — when the endpoint isn't even OpenAI/Anthropic-shaped.
+- [Multi-Provider Failover](../../recipes/multi-provider-failover) — `FallbackBoundModel` in action.
- [API Reference → `cubepi.providers`](../../api/cubepi-providers).
diff --git a/website/docs/recipes/multi-provider-failover.md b/website/docs/recipes/multi-provider-failover.md
index 2c4d869..d6dc905 100644
--- a/website/docs/recipes/multi-provider-failover.md
+++ b/website/docs/recipes/multi-provider-failover.md
@@ -1,267 +1,129 @@
---
title: Multi-Provider Failover
-description: "Configure multi-provider failover in CubePi for resilience — automatic fallback between providers."
+description: "Automatic failover between LLM providers using FallbackBoundModel."
---
# Recipe: Multi-Provider Failover
-When Anthropic is rate-limited or down, fail over to OpenAI without
-crashing the agent. We'll wrap both providers behind a single
-`Provider` adapter that does its own retry/failover logic.
+When the primary provider is rate-limited, unavailable, or has hit its context
+limit, fall over to the next one automatically — without crashing the agent.
+CubePi ships `FallbackBoundModel` for this out of the box.
-**Time to run:** 10 minutes.
-**Deps:** `cubepi`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`.
+**Time to read:** 5 minutes.
+**Deps:** `cubepi`, API keys for two providers.
-## The wrapper provider
+## The built-in: `FallbackBoundModel`
-```python title="failover.py"
-import asyncio
-import logging
-import time
-from typing import Sequence
-
-from cubepi.providers.base import (
- AssistantMessage,
- BaseProvider,
- BoundModel,
- Message,
- MessageStream,
- Model,
- StreamEvent,
- StreamOptions,
- ToolDefinition,
- Usage,
-)
+```python
+import os
+from cubepi import Agent, FallbackBoundModel
from cubepi.providers.anthropic import AnthropicProvider
from cubepi.providers.openai import OpenAIProvider
-log = logging.getLogger(__name__)
+anthropic = AnthropicProvider(api_key=os.environ["ANTHROPIC_API_KEY"])
+openai = OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"])
+model = FallbackBoundModel(
+ chain=(
+ anthropic.model("claude-opus-4-8"), # primary
+ openai.model("gpt-5"), # fallback
+ )
+)
-class FailoverProvider(BaseProvider):
- """Try providers in order; fall over on construction or first-event errors.
-
- Built-in providers swallow API/network errors and surface them as
- `StreamEvent(type="error")` on the returned stream — never as exceptions
- out of `provider.stream()`. So we peek at the first event from each
- inner stream and only commit to it once we see a non-error event.
-
- Limitation: errors that arrive *after* the first event (e.g. mid-stream
- rate limit, server disconnect) are forwarded to the agent as-is.
- Fully replaying a half-streamed turn against a fallback provider would
- require buffering the whole turn — out of scope here.
- """
-
- def __init__(self, primary: BoundModel, *fallbacks: BoundModel) -> None:
- super().__init__(provider_id=primary.spec.provider_id)
- self._chain: list[BoundModel] = [primary, *fallbacks]
-
- async def stream(
- self,
- model: Model,
- messages: list[Message],
- *,
- system_prompt: str = "",
- tools: list[ToolDefinition] | None = None,
- options: StreamOptions | None = None,
- ) -> MessageStream:
- last_error: str | None = None
-
- for bound_model in self._chain:
- provider = bound_model.provider
- mapped_model = bound_model.spec
- # Construction-time failures (rare — most stay inside the producer task).
- try:
- inner = await provider.stream(
- mapped_model,
- messages,
- system_prompt=system_prompt,
- tools=tools,
- options=options,
- )
- except Exception as e:
- log.warning("provider %s failed at construction: %s", mapped_model.provider_id, e)
- last_error = repr(e)
- continue
-
- # Peek at the first event to learn whether the stream is healthy.
- iterator = inner.__aiter__()
- try:
- first = await iterator.__anext__()
- except StopAsyncIteration:
- last_error = "stream ended before producing any events"
- continue
-
- if first.type == "error":
- log.warning("provider %s errored on first event: %s",
- mapped_model.provider_id, first.error_message)
- last_error = first.error_message or "stream error"
- continue
-
- # Healthy — commit to this provider. Forward `first` plus the rest
- # through a fresh outer MessageStream so the caller sees a complete
- # stream starting at the start event.
- outer = MessageStream()
-
- async def _forward(first_event=first, src=iterator, src_stream=inner):
- try:
- outer.push(first_event)
- async for ev in src:
- outer.push(ev)
- final = await src_stream.result()
- outer.set_result(final)
- except Exception as exc:
- fallback_msg = AssistantMessage(
- content=[],
- stop_reason="error",
- error_message=str(exc),
- usage=Usage(),
- timestamp=time.time(),
- )
- outer.push(StreamEvent(type="error", error_message=str(exc)))
- outer.set_result(fallback_msg)
-
- outer.attach_task(asyncio.create_task(_forward()))
- return outer
-
- raise RuntimeError(f"all providers exhausted; last error: {last_error!r}")
+agent = Agent(model=model, system_prompt="You answer concisely.")
+await agent.prompt("Capital of Mongolia?")
```
-## Use it
+`FallbackBoundModel` peeks at the first stream event from each provider. If it
+is a `type="error"` event, or if `stream()` raises a retriable error, the next
+model in the chain is tried. Once a non-error first event arrives the stream is
+forwarded as-is — mid-stream errors are not retried.
-```python title="main.py"
-import asyncio
-import os
+## Default trigger conditions
-from cubepi import Agent
-from cubepi.providers.anthropic import AnthropicProvider
-from cubepi.providers.openai import OpenAIProvider
-from failover import FailoverProvider
+By default failover is triggered on:
+| Error | Why |
+|---|---|
+| `RateLimited` | Quota hit; another provider can serve |
+| `ProviderUnavailable` | 5xx / timeout / connection failure |
+| `ContextLengthExceeded` | Fallback may have a larger context window |
-async def main():
- anthropic = AnthropicProvider(
- provider_id="anthropic",
- api_key=os.environ["ANTHROPIC_API_KEY"],
- )
- openai = OpenAIProvider(
- provider_id="openai",
- api_key=os.environ["OPENAI_API_KEY"],
- )
- failover = FailoverProvider(
- anthropic.model("claude-sonnet-4-6"),
- openai.model("gpt-5"),
- )
+Auth failures (`ProviderAuthFailed`) and bad requests (`ProviderBadRequest`)
+are **not** triggered by default — a bad key or a malformed request will fail
+the same way on every provider in the chain.
- # The model passed here is overridden inside FailoverProvider; pass any
- # placeholder. We use the primary's so usage tracking labels match the
- # happy path.
- agent = Agent(
- model=failover.model("claude-sonnet-4-6"),
- system_prompt="You answer concisely.",
- )
- agent.subscribe(lambda e, s=None: None)
- await agent.prompt("Capital of Mongolia?")
- last = agent.state.messages[-1]
- print(last.content[0].text)
+## Custom trigger conditions
+Pass `trigger_errors` to override:
-asyncio.run(main())
-```
+```python
+from cubepi import FallbackBoundModel
+from cubepi.errors import ProviderAuthFailed, ProviderUnavailable, RateLimited
-## What about smarter failover policies?
+model = FallbackBoundModel(
+ chain=(primary, fallback),
+ trigger_errors=frozenset({RateLimited, ProviderUnavailable, ProviderAuthFailed}),
+)
+```
-The example above falls back on **any** error event. That's fine for
-`RateLimitError`, `APIConnectionError`, or 5xx — but arguably wrong for
-`BadRequestError` (your code is wrong; the next provider will fail the
-same way).
+## Monitoring failovers
-The first-event `error_message` comes from `str(exc)` on the
-underlying SDK exception. Filter on substrings, or — better — wrap each
-provider's `_produce` to tag the error category:
+Pass `on_failover` to hook into billing or alerting:
```python
-NON_RETRYABLE_HINTS = ("bad request", "invalid_request_error", "401", "403")
-
-if first.type == "error":
- msg = (first.error_message or "").lower()
- if any(h in msg for h in NON_RETRYABLE_HINTS):
- raise RuntimeError(f"non-retryable error from {mapped_model.provider_id}: {msg}")
- last_error = first.error_message
- continue
-```
-
-A more robust approach is to fork the built-in providers and re-raise
-specific SDK exceptions from `_produce` so they reach `provider.stream()`
-as real Python exceptions — but that's a larger change against
-CubePi itself.
+import logging
-## Adding circuit breaking
+log = logging.getLogger(__name__)
-Don't keep retrying a provider that's clearly down. A simple counter:
+async def record_failover(failed, next_model, error):
+ log.warning(
+ "provider failover: %s/%s → %s/%s (%s)",
+ failed.spec.provider_id, failed.spec.id,
+ next_model.spec.provider_id if next_model else "none",
+ next_model.spec.id if next_model else "—",
+ error,
+ )
+ # e.g. await billing.record_fallback_failure(failed.spec, error)
-```python
-import time
-
-class CircuitBreaker:
- def __init__(self, failure_threshold: int = 3, recovery_seconds: float = 60) -> None:
- self._failures = 0
- self._opened_at: float | None = None
- self._threshold = failure_threshold
- self._recovery = recovery_seconds
-
- def can_attempt(self) -> bool:
- if self._opened_at and time.monotonic() - self._opened_at < self._recovery:
- return False
- if self._opened_at:
- self._opened_at = None # half-open
- return True
-
- def record_failure(self) -> None:
- self._failures += 1
- if self._failures >= self._threshold:
- self._opened_at = time.monotonic()
- self._failures = 0
-
- def record_success(self) -> None:
- self._failures = 0
+model = FallbackBoundModel(
+ chain=(primary, fallback),
+ on_failover=record_failover,
+)
```
-Hold one `CircuitBreaker` per provider in the `FailoverProvider`,
-skip if `can_attempt()` is False.
+`on_failover` receives `(failed: BoundModel, next_model: BoundModel | None,
+error: BaseException | str)`. Both sync and async callables are accepted.
+Exceptions raised inside the callback are logged and swallowed — a broken
+callback never aborts the failover.
-## Per-tool failover doesn't apply
+## `provider` and `spec` always reflect the primary
-This recipe handles **provider** failures. Tool failures are different
-— see [Middleware → Retries](../guides/middleware/examples#retries-with-backoff)
-for that pattern.
+`FallbackBoundModel.provider` and `.spec` proxy `chain[0]`. Tracing and
+billing code that reads `agent._model.provider` or `agent._model.spec` sees
+the primary — which is the intended provider. The `AssistantMessage` returned
+by the actual call carries `provider_id` and `model_id` of whichever model
+really responded.
## Common pitfalls
-- **Different tool schemas across providers** — Both built-in
- providers accept the same `ToolDefinition`, but extra-body
- customisations (e.g. OpenAI `parallel_tool_calls=False`) won't carry
- to Anthropic. Keep cross-provider behaviour in
- [`transform_context`](../guides/middleware/hooks#transform_context),
- not in `extra_body`.
-- **Different cost** — Failover from Anthropic to OpenAI changes
- per-token cost. Track which provider answered (via `on_response` or
- `AssistantMessage.provider_id`) and bill accordingly.
-- **Streaming consistency** — The wrapper forwards events through a
- fresh `MessageStream`, so consumers see the same `StreamEvent` shape
- regardless of which provider answered. The original `start` event
- comes from the inner provider unchanged.
-- **Mid-stream errors aren't recovered** — Once we've seen a healthy
- first event, the wrapper commits to that provider. If it errors
- halfway through a long response, the agent sees the error. Full
- mid-stream replay would require buffering — out of scope here.
+- **Different tool schemas across providers** — Both built-in providers accept
+ the same `ToolDefinition`, but vendor-specific extras (e.g. OpenAI
+ `parallel_tool_calls=False`) won't carry to Anthropic. Keep cross-provider
+ behaviour in `transform_context` middleware, not in `extra_body`.
+- **Different cost** — Failover changes per-token cost. Track which provider
+ answered via `AssistantMessage.provider_id` and bill accordingly; the
+ `on_failover` callback is the right place to record the switch.
+- **Mid-stream errors aren't retried** — Once a healthy first event arrives,
+ `FallbackBoundModel` commits to that provider. Errors during the rest of
+ the stream are forwarded to the agent as-is.
+- **`ContextLengthExceeded` is only useful if the fallback is larger** — If
+ both providers have the same context window the failover will fail the same
+ way. Consider pairing a standard-window primary with a large-context fallback.
## Run the example
-A self-contained, runnable version of this recipe is in the repository at
-[`examples/multi_provider_failover.py`](https://github.com/cubeplexai/cubepi/blob/main/examples/multi_provider_failover.py).
-It deliberately uses a bad key for the primary provider to trigger failover,
-then answers correctly via the fallback.
+A runnable version is in the repository:
```bash
git clone https://github.com/cubeplexai/cubepi && cd cubepi
@@ -271,9 +133,11 @@ export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY [+ OPENAI_BASE_URL]
uv run python examples/multi_provider_failover.py
```
+The example deliberately uses a bad key for the primary to trigger failover,
+then answers correctly via the real fallback.
+
## See also
-- [Providers / Anthropic](../guides/providers/anthropic) and
- [OpenAI](../guides/providers/openai) — provider-specific details.
-- [Writing a Custom Provider](../guides/providers/custom) — the same
- Protocol used by this wrapper.
+- [Providers Overview](../guides/providers/overview) — provider setup and `CapabilityDescriptor`.
+- [Providers / Anthropic](../guides/providers/anthropic) and [OpenAI](../guides/providers/openai) — provider-specific details.
+- [Writing a Custom Provider](../guides/providers/custom) — when neither built-in provider fits.