CubeLoop — a Pythonic, async-native agent harness core designed for high performance, readability, and production-grade persistence.
Python agent-frameworkagent-harnessagent-runtimeagentic-aiai-agentsai-frameworkanthropicharnessharness-engineeringharness-frameworklanggraphlanggraph-alternativellmmcpopenaipi-agentpi-agent-corepythonpython-agent
docs: document FallbackBoundModel in CHANGELOG, README, recipe, and providers guide
xf gong32063a53b2d5
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 707ec71..4b61510 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **`FallbackBoundModel`** — built-in failover chain at the `BoundModel` level.+ Wrap an ordered `chain` of `BoundModel` instances; on `RateLimited`,+ `ProviderUnavailable`, or `ContextLengthExceeded` (configurable via+ `trigger_errors`), or on a first-event stream error, the next model in the+ chain is tried transparently. Optional `on_failover` callback for+ billing/metrics hooks. Exported from `cubepi` and `cubepi.providers`.++- **`DEFAULT_TRIGGER_ERRORS`** — `frozenset({RateLimited, ProviderUnavailable,+ ContextLengthExceeded})`. The default set of error types that trigger failover+ in `FallbackBoundModel`.+- **`BoundModel.generate()` / `BoundModel.stream()`** — the handle returned by`provider.model(...)` now drives a provider call directly. Useful forutilities (summarizers, classifiers) where you already hold a `BoundModel`diff --git a/README.md b/README.mdindex 9ceaa19..7494306 100644--- a/README.md+++ b/README.md@@ -100,6 +100,21 @@ faux = FauxProvider(provider_id="faux")faux.set_responses(["Hello!", "How can I help?"])```+Use `FallbackBoundModel` to chain providers — on a rate limit, outage, or+context-length error the next model in the chain is tried automatically:++```python+from cubepi import FallbackBoundModel++model = FallbackBoundModel(+ chain=(+ anthropic.model("claude-opus-4-8"), # primary+ openai.model("gpt-5"), # fallback+ )+)+agent = Agent(model=model, system_prompt="...")+```+### ToolsDecorate an async function with `@tool`: the input schema is derived from thediff --git a/examples/multi_provider_failover.py b/examples/multi_provider_failover.pyindex 7b65b59..e9ae265 100644--- a/examples/multi_provider_failover.py+++ b/examples/multi_provider_failover.py@@ -1,150 +1,54 @@"""Multi-provider failover — recipe example.-Demonstrates a FailoverProvider wrapper that peeks at the first stream event-from each inner provider and falls over to the next on error.+Demonstrates FallbackBoundModel: the primary provider is given a bad key so its+first stream event is an auth error, which triggers transparent failover to the+real provider.uv run python examples/multi_provider_failover.py-The example constructs a primary provider with a bad key to force a failover to-the real provider, proving the mechanism works end-to-end.-Set ANTHROPIC_API_KEY or OPENAI_API_KEY (and optionally MODEL) before running.See _provider.py for details."""+from __future__ import annotations+import asyncio-import loggingimport os-import time-from cubepi import Agent-from cubepi.providers.openai import OpenAIProvider+from cubepi import Agent, FallbackBoundModelfrom cubepi.providers.anthropic import AnthropicProvider-from cubepi.providers.base import (- AssistantMessage,- BaseProvider,- BoundModel,- Message,- MessageStream,- Model,- StreamEvent,- StreamOptions,- ToolDefinition,- Usage,-)+from cubepi.providers.base import BoundModel+from cubepi.providers.openai import OpenAIProviderfrom _provider import MODEL_ID, provider-log = logging.getLogger(__name__)---class FailoverProvider(BaseProvider):- """Try providers in order; fall over on construction or first-event errors.-- Built-in providers swallow API/network errors and surface them as- StreamEvent(type="error") on the returned stream. We peek at the first- event from each inner stream and only commit once we see a non-error event.-- Limitation: errors that arrive *after* the first event are forwarded as-is.- """-- def __init__(self, primary: BoundModel, *fallbacks: BoundModel) -> None:- super().__init__(provider_id=primary.spec.provider_id)- self._chain: list[BoundModel] = [primary, *fallbacks]-- async def stream(- self,- model: Model,- messages: list[Message],- *,- system_prompt: str = "",- tools: list[ToolDefinition] | None = None,- options: StreamOptions | None = None,- ) -> MessageStream:- last_error: str | None = None-- for bound_model in self._chain:- inner_provider = bound_model.provider- mapped_model = bound_model.spec- try:- inner = await inner_provider.stream(- mapped_model,- messages,- system_prompt=system_prompt,- tools=tools,- options=options,- )- except Exception as e:- log.warning("provider %s failed at construction: %s", mapped_model.provider_id, e)- last_error = repr(e)- continue-- iterator = inner.__aiter__()- try:- first = await iterator.__anext__()- except StopAsyncIteration:- last_error = "stream ended before producing any events"- continue-- if first.type == "error":- log.warning(- "provider %s errored on first event: %s",- mapped_model.provider_id,- first.error_message,- )- last_error = first.error_message or "stream error"- continue-- outer = MessageStream()-- async def _forward(first_event=first, src=iterator, src_stream=inner):- try:- outer.push(first_event)- async for ev in src:- outer.push(ev)- final = await src_stream.result()- outer.set_result(final)- except Exception as exc:- fallback_msg = AssistantMessage(- content=[],- stop_reason="error",- error_message=str(exc),- usage=Usage(),- timestamp=time.time(),- )- outer.push(StreamEvent(type="error", error_message=str(exc)))- outer.set_result(fallback_msg)-- outer.attach_task(asyncio.create_task(_forward()))- return outer-- raise RuntimeError(f"all providers exhausted; last error: {last_error!r}")---def _bad_key_provider() -> tuple[BaseProvider, str]:- """Build a provider identical to the real one but with an invalid key."""++def _bad_key_primary() -> BoundModel:+ """Build a bound model with a bad API key to force a first-event auth error."""if os.environ.get("ANTHROPIC_API_KEY"):base_url = os.environ.get("ANTHROPIC_BASE_URL") or None- return AnthropicProvider(api_key="bad-key", base_url=base_url), MODEL_ID+ return AnthropicProvider(api_key="bad-key", base_url=base_url).model(MODEL_ID)else:base_url = os.environ.get("OPENAI_BASE_URL") or None- return OpenAIProvider(api_key="bad-key", base_url=base_url), MODEL_ID+ return OpenAIProvider(api_key="bad-key", base_url=base_url).model(MODEL_ID)async def main() -> None:- logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(name)s: %(message)s")-- bad_provider, model_id = _bad_key_provider()-- # Primary: bad key → errors on first event → triggers failover.- # Fallback: real provider from _provider.py.- failover = FailoverProvider(- bad_provider.model(model_id),- provider.model(MODEL_ID),+ # Primary has a bad key → first stream event will be type="error" → failover.+ # Fallback uses the real provider from _provider.py.+ model = FallbackBoundModel(+ chain=(+ _bad_key_primary(),+ provider.model(MODEL_ID),+ ),+ on_failover=lambda failed, nxt, err: print(+ f"[failover] {failed.spec.provider_id}/{failed.spec.id}"+ f" → {nxt.spec.provider_id}/{nxt.spec.id if nxt else '—'}: {err}"+ ),)agent = Agent(- model=failover.model(MODEL_ID),+ model=model,system_prompt="You answer concisely.",)diff --git a/website/docs/guides/providers/overview.md b/website/docs/guides/providers/overview.mdindex a259686..476b24e 100644--- a/website/docs/guides/providers/overview.md+++ b/website/docs/guides/providers/overview.md@@ -322,9 +322,33 @@ provider, `provider.model("id", ...)` factory, typed `ProviderError`failures, and a capability descriptor for backend wire differences. See[Image Generation](./image-generation) for the full guide.+## Failover chain++`FallbackBoundModel` wraps an ordered chain of `BoundModel` instances. On a+`RateLimited`, `ProviderUnavailable`, or `ContextLengthExceeded` error — or on+a first-event stream error — it transparently tries the next model:++```python+from cubepi import FallbackBoundModel++model = FallbackBoundModel(+ chain=(+ anthropic.model("claude-opus-4-8"), # primary+ openai.model("gpt-5"), # fallback+ )+)+agent = Agent(model=model, ...)+```++`trigger_errors` is configurable; an optional `on_failover` callback fires on+each switchover for billing/metrics. See the+[Multi-Provider Failover recipe](../../recipes/multi-provider-failover) for the+full guide.+## See also- [OpenAI Provider](./openai) — concrete OpenAI / OpenAI-compatible setup.- [Anthropic Provider](./anthropic) — the `int_budget` reasoning shape in practice.- [Writing a Custom Provider](./custom) — when the endpoint isn't even OpenAI/Anthropic-shaped.+- [Multi-Provider Failover](../../recipes/multi-provider-failover) — `FallbackBoundModel` in action.- [API Reference → `cubepi.providers`](../../api/cubepi-providers).diff --git a/website/docs/recipes/multi-provider-failover.md b/website/docs/recipes/multi-provider-failover.mdindex 2c4d869..d6dc905 100644--- a/website/docs/recipes/multi-provider-failover.md+++ b/website/docs/recipes/multi-provider-failover.md@@ -1,267 +1,129 @@---title: Multi-Provider Failover-description: "Configure multi-provider failover in CubePi for resilience — automatic fallback between providers."+description: "Automatic failover between LLM providers using FallbackBoundModel."---# Recipe: Multi-Provider Failover-When Anthropic is rate-limited or down, fail over to OpenAI without-crashing the agent. We'll wrap both providers behind a single-`Provider` adapter that does its own retry/failover logic.+When the primary provider is rate-limited, unavailable, or has hit its context+limit, fall over to the next one automatically — without crashing the agent.+CubePi ships `FallbackBoundModel` for this out of the box.-**Time to run:** 10 minutes.-**Deps:** `cubepi`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`.+**Time to read:** 5 minutes.+**Deps:** `cubepi`, API keys for two providers.-## The wrapper provider+## The built-in: `FallbackBoundModel`-```python title="failover.py"-import asyncio-import logging-import time-from typing import Sequence--from cubepi.providers.base import (- AssistantMessage,- BaseProvider,- BoundModel,- Message,- MessageStream,- Model,- StreamEvent,- StreamOptions,- ToolDefinition,- Usage,-)+```python+import os+from cubepi import Agent, FallbackBoundModelfrom cubepi.providers.anthropic import AnthropicProviderfrom cubepi.providers.openai import OpenAIProvider-log = logging.getLogger(__name__)+anthropic = AnthropicProvider(api_key=os.environ["ANTHROPIC_API_KEY"])+openai = OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"])+model = FallbackBoundModel(+ chain=(+ anthropic.model("claude-opus-4-8"), # primary+ openai.model("gpt-5"), # fallback+ )+)-class FailoverProvider(BaseProvider):- """Try providers in order; fall over on construction or first-event errors.-- Built-in providers swallow API/network errors and surface them as- `StreamEvent(type="error")` on the returned stream — never as exceptions- out of `provider.stream()`. So we peek at the first event from each- inner stream and only commit to it once we see a non-error event.-- Limitation: errors that arrive *after* the first event (e.g. mid-stream- rate limit, server disconnect) are forwarded to the agent as-is.- Fully replaying a half-streamed turn against a fallback provider would- require buffering the whole turn — out of scope here.- """-- def __init__(self, primary: BoundModel, *fallbacks: BoundModel) -> None:- super().__init__(provider_id=primary.spec.provider_id)- self._chain: list[BoundModel] = [primary, *fallbacks]-- async def stream(- self,- model: Model,- messages: list[Message],- *,- system_prompt: str = "",- tools: list[ToolDefinition] | None = None,- options: StreamOptions | None = None,- ) -> MessageStream:- last_error: str | None = None-- for bound_model in self._chain:- provider = bound_model.provider- mapped_model = bound_model.spec- # Construction-time failures (rare — most stay inside the producer task).- try:- inner = await provider.stream(- mapped_model,- messages,- system_prompt=system_prompt,- tools=tools,- options=options,- )- except Exception as e:- log.warning("provider %s failed at construction: %s", mapped_model.provider_id, e)- last_error = repr(e)- continue-- # Peek at the first event to learn whether the stream is healthy.- iterator = inner.__aiter__()- try:- first = await iterator.__anext__()- except StopAsyncIteration:- last_error = "stream ended before producing any events"- continue-- if first.type == "error":- log.warning("provider %s errored on first event: %s",- mapped_model.provider_id, first.error_message)- last_error = first.error_message or "stream error"- continue-- # Healthy — commit to this provider. Forward `first` plus the rest- # through a fresh outer MessageStream so the caller sees a complete- # stream starting at the start event.- outer = MessageStream()-- async def _forward(first_event=first, src=iterator, src_stream=inner):- try:- outer.push(first_event)- async for ev in src:- outer.push(ev)- final = await src_stream.result()- outer.set_result(final)- except Exception as exc:- fallback_msg = AssistantMessage(- content=[],- stop_reason="error",- error_message=str(exc),- usage=Usage(),- timestamp=time.time(),- )- outer.push(StreamEvent(type="error", error_message=str(exc)))- outer.set_result(fallback_msg)-- outer.attach_task(asyncio.create_task(_forward()))- return outer-- raise RuntimeError(f"all providers exhausted; last error: {last_error!r}")+agent = Agent(model=model, system_prompt="You answer concisely.")+await agent.prompt("Capital of Mongolia?")```-## Use it+`FallbackBoundModel` peeks at the first stream event from each provider. If it+is a `type="error"` event, or if `stream()` raises a retriable error, the next+model in the chain is tried. Once a non-error first event arrives the stream is+forwarded as-is — mid-stream errors are not retried.-```python title="main.py"-import asyncio-import os+## Default trigger conditions-from cubepi import Agent-from cubepi.providers.anthropic import AnthropicProvider-from cubepi.providers.openai import OpenAIProvider-from failover import FailoverProvider+By default failover is triggered on:+| Error | Why |+|---|---|+| `RateLimited` | Quota hit; another provider can serve |+| `ProviderUnavailable` | 5xx / timeout / connection failure |+| `ContextLengthExceeded` | Fallback may have a larger context window |-async def main():- anthropic = AnthropicProvider(- provider_id="anthropic",- api_key=os.environ["ANTHROPIC_API_KEY"],- )- openai = OpenAIProvider(- provider_id="openai",- api_key=os.environ["OPENAI_API_KEY"],- )- failover = FailoverProvider(- anthropic.model("claude-sonnet-4-6"),- openai.model("gpt-5"),- )+Auth failures (`ProviderAuthFailed`) and bad requests (`ProviderBadRequest`)+are **not** triggered by default — a bad key or a malformed request will fail+the same way on every provider in the chain.- # The model passed here is overridden inside FailoverProvider; pass any- # placeholder. We use the primary's so usage tracking labels match the- # happy path.- agent = Agent(- model=failover.model("claude-sonnet-4-6"),- system_prompt="You answer concisely.",- )- agent.subscribe(lambda e, s=None: None)- await agent.prompt("Capital of Mongolia?")- last = agent.state.messages[-1]- print(last.content[0].text)+## Custom trigger conditions+Pass `trigger_errors` to override:-asyncio.run(main())-```+```python+from cubepi import FallbackBoundModel+from cubepi.errors import ProviderAuthFailed, ProviderUnavailable, RateLimited-## What about smarter failover policies?+model = FallbackBoundModel(+ chain=(primary, fallback),+ trigger_errors=frozenset({RateLimited, ProviderUnavailable, ProviderAuthFailed}),+)+```-The example above falls back on **any** error event. That's fine for-`RateLimitError`, `APIConnectionError`, or 5xx — but arguably wrong for-`BadRequestError` (your code is wrong; the next provider will fail the-same way).+## Monitoring failovers-The first-event `error_message` comes from `str(exc)` on the-underlying SDK exception. Filter on substrings, or — better — wrap each-provider's `_produce` to tag the error category:+Pass `on_failover` to hook into billing or alerting:```python-NON_RETRYABLE_HINTS = ("bad request", "invalid_request_error", "401", "403")--if first.type == "error":- msg = (first.error_message or "").lower()- if any(h in msg for h in NON_RETRYABLE_HINTS):- raise RuntimeError(f"non-retryable error from {mapped_model.provider_id}: {msg}")- last_error = first.error_message- continue-```--A more robust approach is to fork the built-in providers and re-raise-specific SDK exceptions from `_produce` so they reach `provider.stream()`-as real Python exceptions — but that's a larger change against-CubePi itself.+import logging-## Adding circuit breaking+log = logging.getLogger(__name__)-Don't keep retrying a provider that's clearly down. A simple counter:+async def record_failover(failed, next_model, error):+ log.warning(+ "provider failover: %s/%s → %s/%s (%s)",+ failed.spec.provider_id, failed.spec.id,+ next_model.spec.provider_id if next_model else "none",+ next_model.spec.id if next_model else "—",+ error,+ )+ # e.g. await billing.record_fallback_failure(failed.spec, error)-```python-import time--class CircuitBreaker:- def __init__(self, failure_threshold: int = 3, recovery_seconds: float = 60) -> None:- self._failures = 0- self._opened_at: float | None = None- self._threshold = failure_threshold- self._recovery = recovery_seconds-- def can_attempt(self) -> bool:- if self._opened_at and time.monotonic() - self._opened_at < self._recovery:- return False- if self._opened_at:- self._opened_at = None # half-open- return True-- def record_failure(self) -> None:- self._failures += 1- if self._failures >= self._threshold:- self._opened_at = time.monotonic()- self._failures = 0-- def record_success(self) -> None:- self._failures = 0+model = FallbackBoundModel(+ chain=(primary, fallback),+ on_failover=record_failover,+)```-Hold one `CircuitBreaker` per provider in the `FailoverProvider`,-skip if `can_attempt()` is False.+`on_failover` receives `(failed: BoundModel, next_model: BoundModel | None,+error: BaseException | str)`. Both sync and async callables are accepted.+Exceptions raised inside the callback are logged and swallowed — a broken+callback never aborts the failover.-## Per-tool failover doesn't apply+## `provider` and `spec` always reflect the primary-This recipe handles **provider** failures. Tool failures are different-— see [Middleware → Retries](../guides/middleware/examples#retries-with-backoff)-for that pattern.+`FallbackBoundModel.provider` and `.spec` proxy `chain[0]`. Tracing and+billing code that reads `agent._model.provider` or `agent._model.spec` sees+the primary — which is the intended provider. The `AssistantMessage` returned+by the actual call carries `provider_id` and `model_id` of whichever model+really responded.## Common pitfalls-- **Different tool schemas across providers** — Both built-in- providers accept the same `ToolDefinition`, but extra-body- customisations (e.g. OpenAI `parallel_tool_calls=False`) won't carry- to Anthropic. Keep cross-provider behaviour in- [`transform_context`](../guides/middleware/hooks#transform_context),- not in `extra_body`.-- **Different cost** — Failover from Anthropic to OpenAI changes- per-token cost. Track which provider answered (via `on_response` or- `AssistantMessage.provider_id`) and bill accordingly.-- **Streaming consistency** — The wrapper forwards events through a- fresh `MessageStream`, so consumers see the same `StreamEvent` shape- regardless of which provider answered. The original `start` event- comes from the inner provider unchanged.-- **Mid-stream errors aren't recovered** — Once we've seen a healthy- first event, the wrapper commits to that provider. If it errors- halfway through a long response, the agent sees the error. Full- mid-stream replay would require buffering — out of scope here.+- **Different tool schemas across providers** — Both built-in providers accept+ the same `ToolDefinition`, but vendor-specific extras (e.g. OpenAI+ `parallel_tool_calls=False`) won't carry to Anthropic. Keep cross-provider+ behaviour in `transform_context` middleware, not in `extra_body`.+- **Different cost** — Failover changes per-token cost. Track which provider+ answered via `AssistantMessage.provider_id` and bill accordingly; the+ `on_failover` callback is the right place to record the switch.+- **Mid-stream errors aren't retried** — Once a healthy first event arrives,+ `FallbackBoundModel` commits to that provider. Errors during the rest of+ the stream are forwarded to the agent as-is.+- **`ContextLengthExceeded` is only useful if the fallback is larger** — If+ both providers have the same context window the failover will fail the same+ way. Consider pairing a standard-window primary with a large-context fallback.## Run the example-A self-contained, runnable version of this recipe is in the repository at-[`examples/multi_provider_failover.py`](https://github.com/cubeplexai/cubepi/blob/main/examples/multi_provider_failover.py).-It deliberately uses a bad key for the primary provider to trigger failover,-then answers correctly via the fallback.+A runnable version is in the repository:```bashgit clone https://github.com/cubeplexai/cubepi && cd cubepi@@ -271,9 +133,11 @@ export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY [+ OPENAI_BASE_URL]uv run python examples/multi_provider_failover.py```+The example deliberately uses a bad key for the primary to trigger failover,+then answers correctly via the real fallback.+## See also-- [Providers / Anthropic](../guides/providers/anthropic) and- [OpenAI](../guides/providers/openai) — provider-specific details.-- [Writing a Custom Provider](../guides/providers/custom) — the same- Protocol used by this wrapper.+- [Providers Overview](../guides/providers/overview) — provider setup and `CapabilityDescriptor`.+- [Providers / Anthropic](../guides/providers/anthropic) and [OpenAI](../guides/providers/openai) — provider-specific details.+- [Writing a Custom Provider](../guides/providers/custom) — when neither built-in provider fits.