DSPy: The framework for programming—not prompting—language models
Python
Declare HTTP providers to the native engine: one binding per LM, a fallback that keeps the connection (#10442)
Maxime Rivestfd52234363c0
diff --git a/docs/docs/community/normalized-lm-api-migration.md b/docs/docs/community/normalized-lm-api-migration.mdindex e76e9e9ec..8127affcd 100644--- a/docs/docs/community/normalized-lm-api-migration.md+++ b/docs/docs/community/normalized-lm-api-migration.md@@ -278,7 +278,10 @@ A custom engine owns its connection: `api_key`, `api_base`, `timeout`,on `copy()` and on every call, rather than dropped. `copy(engine=...)` replaces theengine pair, and each side must be its kind (a coroutine function on the async side).An engine is saved by `dump_state()`/`load_state()` of its own, and loading one is-gated by `allow_unsafe_lm_state=True` like a custom LM class. See the+gated by `allow_unsafe_lm_state=True` like a custom LM class. For an HTTP provider+lm15 can speak to, declare it with `dspy.lm15.register_provider(...)` instead of+writing an engine: each `dspy.LM` binds the registrations present at construction,+and a fallback to LiteLLM keeps the declared address and credential. See the[custom-engine tutorial](../tutorials/custom_lm_engines/index.md).Implementing custom LMs through `BaseLM.forward(prompt=None, messages=None, **kwargs)`diff --git a/docs/docs/tutorials/custom_lm_engines/index.md b/docs/docs/tutorials/custom_lm_engines/index.mdindex 975a5cf61..acbd9502f 100644--- a/docs/docs/tutorials/custom_lm_engines/index.md+++ b/docs/docs/tutorials/custom_lm_engines/index.md@@ -1,6 +1,9 @@# Custom LM Engines-You can supply your own execution engine to `dspy.LM`. An engine is for a backend that is not an HTTP provider DSPy's bundled lm15 can speak to: a CLI, an in-process model, an agent harness. It owns its whole connection, so the [rules below](#what-a-custom-engine-owns) apply.+You can supply your own execution engine to `dspy.LM`. Before you do, check which of two situations you are in:++- **An HTTP provider lm15 can already speak to** — an OpenAI-compatible service, a company gateway, a host DSPy's bundled lm15 does not list yet. Do not write an engine: [declare the provider](#declaring-a-provider-instead-of-writing-an-engine) and `dspy.LM("<provider>/<model>")` routes natively, with `api_key`, `api_base`, `timeout`, saving and loading all behaving as for a built-in provider.+- **Not an HTTP provider at all** — a CLI, an in-process model, an agent harness. Write an engine. It owns its whole connection, so the [rules below](#what-a-custom-engine-owns) apply.The minimum synchronous engine interface is:@@ -21,6 +24,48 @@ family and owns retries. Unexpected exceptions retain their original cause andare not guessed to be retryable from their message text. See[errors and retry ownership](../../community/normalized-lm-api-migration.md#errors-and-retry-ownership).+## Declaring a provider instead of writing an engine++lm15 routes model strings through a registry of providers it has verified against the wire. A provider it does not list can be declared for the process with the same three facts a registry entry is made of — an access policy (name, key variable, address), a wire dialect, and a compat policy describing the server's spellings:++```python+import dspy+from dspy.lm15 import AccessPolicy, EndpointSupport, ModelSupport, OpenAIChatCompat, ProviderDefinition++dspy.lm15.register_provider(+ ProviderDefinition.chat(+ AccessPolicy(+ provider="fireworks",+ supports=EndpointSupport(complete=True, stream=True, models=True),+ auth_modes=("bearer",),+ env_keys=("FIREWORKS_API_KEY",),+ base_url="https://api.fireworks.ai/inference/v1",+ ),+ compat=OpenAIChatCompat(max_tokens_field="max_tokens", thinking_format="reasoning_effort"),+ aliases=("fireworks-ai",), # accept LiteLLM's spelling, fireworks_ai/, as a model prefix+ ),+ metadata_namespaces=("fireworks_ai",), # this endpoint IS Fireworks: its snapshot entries may describe and price the models+)++lm = dspy.LM("fireworks/accounts/fireworks/models/deepseek-v4p1-flash") # or fireworks_ai/...+```++Do this at import time of your application, before the LMs that use it are constructed. **Each `dspy.LM` binds the registrations present when it is constructed and keeps them for its whole life** — selection, capabilities, pricing, and both its sync and async engines read that one binding. `register_provider(..., replace=True)` therefore changes only LMs constructed afterwards; it never moves an existing LM, and it can never leave one LM's sync calls on one definition and its async calls on another.++After that the provider behaves like a built-in one:++- `dspy.LM(..., api_key=..., api_base=..., timeout=...)` are honored; with no key given, `FIREWORKS_API_KEY` is read, and a missing key names that variable. A `FIREWORKS_API_BASE` variable in the environment does not redirect a declared provider (it does for registry providers, to preserve LiteLLM-era gateways): the declaration states its address itself.+- `engine="auto"` plans each request against the declared compat. A client setting only LiteLLM carries (`extra_headers`, `organization`, …) selects LiteLLM before any I/O — **at the declared address, with the declared credential**, through LiteLLM's generic OpenAI-compatible door (`openai/`, or `anthropic/` for that dialect). The credential travels only under the scheme the declaration selects for it — the `openai/` door sends a bearer header, the `anthropic/` door `x-api-key` — and a declaration whose scheme the door cannot send gets no fallback rather than its secret in an undeclared header. The LM's aliases are never handed to LiteLLM as a provider prefix, because LiteLLM may know that name as a different service. The call is still that provider's: errors name it and pricing comes from its `metadata_namespaces`, never from LiteLLM's reading of the door's model string. A refusal the declared compat produces (a reasoning dial on a wire with no reasoning field, say) is final: the declaration is the authority on what that server cannot do, and sending the request through LiteLLM anyway would drop the setting silently. `engine="lm15"` refuses both cases instead.+- `dump_state()`/`load_state()` and program `save()`/`load()` need nothing extra — the state is the model string. A registration is an environment fact, like the key variable: it must be in place when the program is loaded.++**Aliases are spellings, nothing more.** Which models a declared provider's endpoint supports, and what they cost, are separate statements:++- `metadata_namespaces=("fireworks_ai",)` says DSPy's model-metadata snapshot entries under that LiteLLM namespace describe (capabilities) and price this provider's models. Give it only when the endpoint really is that service. Without it nothing is inherited, and cost is reported as unknown.+- `supports=ModelSupport(function_calling=True, ...)` states model support for the whole provider; `models={"private-v1": ModelSupport(...)}` per model id. A snapshot entry, when one is found, is the more specific fact and wins over a statement. What neither states is treated as unsupported — the rule built-in providers already live with — and a statement never prices anything.+- The compat object still bounds everything: a wire with `thinking_format="none"` has no reasoning whatever the statement says.++What a declaration is not: a receipt. lm15's own registry entries are pinned from live captures; a declaration is your word, and the route says so (`Resolution.declared`). Registering the same registration twice is a no-op; a different one under the same id needs `replace=True`; a spelling lm15 or LiteLLM already uses is refused.+## What a custom engine ownsA custom engine is borrowed by `dspy.LM` and owns its connection. Three rules follow, and DSPy enforces each rather than guessing:diff --git a/dspy/clients/backend_selection.py b/dspy/clients/backend_selection.pyindex f7c0bbf21..8138a908d 100644--- a/dspy/clients/backend_selection.py+++ b/dspy/clients/backend_selection.py@@ -9,7 +9,7 @@ import osfrom dataclasses import dataclass, fieldfrom dspy.clients.engines.lm15_engine import LM15Engine, timeouts_for-from dspy.lm15 import RouterConfig, UnknownModelError, UnsupportedFeatureError+from dspy.lm15 import RouterConfig, UnknownModelError, UnsupportedFeatureError, _definitionsCLIENT_KEYS = {"api_key", "api_base", "base_url", "headers", "extra_headers", "timeout", "api_version","azure_ad_token_provider", "organization", "project", "extra_query", "custom_llm_provider"}@@ -35,13 +35,27 @@ def select_backend(lm, options=None):clients.update({key: val for key, val in (options or {}).items() if key in CLIENT_KEYS})native = spec != "litellm" and lm.model_type != "text"resolution = None- if native:+ providers = _definitions(getattr(lm, "_providers", ()))+ # A declared provider is resolved even under engine="litellm": its+ # LiteLLM route is built from the declaration (address, credential), not+ # from a model-string prefix LiteLLM may read as another service.+ if native or (providers and lm.model_type != "text"):+ try:+ resolution = LM15Engine(RouterConfig(env={}, providers=providers), model_type=lm.model_type).resolve(lm.model)+ except (UnknownModelError, UnsupportedFeatureError):+ if spec == "lm15":+ raise+ native = False+ if native and resolution is not None:try:- resolution = LM15Engine(RouterConfig(env={}), model_type=lm.model_type).resolve(lm.model)# Preserve legacy environment-configured gateways. A public native- # endpoint must not receive credentials meant for that gateway.+ # endpoint must not receive credentials meant for that gateway. A+ # declared provider states its address explicitly; an explicit+ # declaration beats an ambient variable.prefix = lm.model.split("/", 1)[0].upper()- if spec == "auto" and any(os.getenv(name) for name in (f"{prefix}_API_BASE", f"{prefix}_BASE_URL")):+ if spec == "auto" and not resolution.declared and any(+ os.getenv(name) for name in (f"{prefix}_API_BASE", f"{prefix}_BASE_URL")+ ):native = Falseif spec == "auto" and resolution.provider == "xai" and "api_key" not in clients and os.getenv("XAI_API_KEY"):clients["api_key"] = os.environ["XAI_API_KEY"]@@ -57,6 +71,8 @@ def select_backend(lm, options=None):if spec == "lm15":raisenative = False+ if spec == "litellm" and resolution is not None and not resolution.declared:+ resolution = None # as before: the compatibility engine reads the model string itselfif spec == "lm15" and not native:raise UnsupportedFeatureError("The requested client settings require the LiteLLM compatibility engine.")return BackendSelection(native, resolution, clients)diff --git a/dspy/clients/capabilities.py b/dspy/clients/capabilities.pyindex 4b214d078..c0324b298 100644--- a/dspy/clients/capabilities.py+++ b/dspy/clients/capabilities.py@@ -16,6 +16,7 @@ from dspy.clients.backend_selection import select_backendfrom dspy.clients.engines.litellm_errors import litellm_errorsfrom dspy.clients.errors import error_boundaryfrom dspy.clients.model_metadata import model_info+from dspy.lm15 import UnsupportedFeatureError@dataclass(frozen=True)@@ -28,9 +29,10 @@ class Capabilities:def resolve(lm):from dspy.clients.engines.lm15_engine import LM15Engine- from dspy.lm15 import RouterConfig+ from dspy.lm15 import RouterConfig, _definitions- return LM15Engine(RouterConfig(env={}), model_type=lm.model_type).resolve(lm.model)+ providers = _definitions(getattr(lm, "_providers", ()))+ return LM15Engine(RouterConfig(env={}, providers=providers), model_type=lm.model_type).resolve(lm.model)@dataclass(repr=False)@@ -128,9 +130,11 @@ def _capabilities(lm, options=None):frozenset(getattr(spec, "supported_params", ()) or ()),)selection = select_backend(lm, options)- if not selection.native:- return _litellm_capabilities(lm, selection)route = selection.resolution+ if not selection.native and not (route is not None and route.declared):+ return _litellm_capabilities(lm, selection)+ # A declared provider is described by its declaration on either route:+ # the LiteLLM fallback reaches the same server through the generic door.from dspy._vendor.lm15.compat import (AnthropicCompat,OpenAIChatCompat,@@ -140,21 +144,44 @@ def _capabilities(lm, options=None):resolve_openai_responses_compat,)from dspy._vendor.lm15.registry import lookup-- definition = lookup(route.provider)- info = model_info(route.provider, route.model)- tools = info.get("supports_function_calling") is True- reasoning = info.get("supports_reasoning") is True- schema = info.get("supports_response_schema") is True+ from dspy.lm15 import _binding_for++ binding = _binding_for(getattr(lm, "_providers", ()), route.provider) if route.declared else None+ if binding is not None:+ definition = binding.definition+ # Only the namespaces the registration named may describe this model.+ info = model_info(route.provider, route.model, namespaces=binding.metadata_namespaces)+ stated = binding.support_for(route.model)+ else:+ definition = lookup(route.provider)+ if definition is None:+ raise UnsupportedFeatureError(f"{route.provider!r} is not a provider DSPy knows the capabilities of")+ info = model_info(route.provider, route.model)+ stated = None++ def supported(key, statement):+ # A snapshot entry is the more specific fact; the registration's own+ # statement fills what the snapshot does not say; unknown is False.+ value = info.get(key)+ if value is None and statement is not None:+ value = statement+ return value is True++ tools = supported("supports_function_calling", stated.function_calling if stated else None)+ reasoning = supported("supports_reasoning", stated.reasoning if stated else None)+ schema = supported("supports_response_schema", stated.response_schema if stated else None)params = {"temperature", "max_tokens", "top_p", "stream"}# These describe lm15's canonical mappings. Engine-specific refusal still# validates the complete request, including combinations and model limits.if definition.dialect == "openai-chat":- preset = definition.compat or ("xai" if route.provider == "xai" else "openai")- compat = resolve_openai_chat_compat(OpenAIChatCompat.preset(preset).for_model(route.model))+ if isinstance(definition.compat, OpenAIChatCompat):+ partial = definition.compat # a declared provider carries its compat object+ else:+ partial = OpenAIChatCompat.preset(definition.compat or ("xai" if route.provider == "xai" else "openai"))+ compat = resolve_openai_chat_compat(partial.for_model(route.model))params.update({"stop", "max_completion_tokens", "logprobs", "top_logprobs", "seed","presence_penalty", "frequency_penalty", "logit_bias", "user"})- if info.get("supports_response_schema") is True or info.get("supports_function_calling") is True:+ if schema or tools:params.add("response_format")if compat.json_schema == "reject":schema = False@@ -163,7 +190,10 @@ def _capabilities(lm, options=None):if route.provider == "xai":params.difference_update({"logprobs", "top_logprobs"})elif definition.dialect == "anthropic":- compat = resolve_anthropic_compat(AnthropicCompat.preset(definition.compat or "anthropic"))+ if isinstance(definition.compat, AnthropicCompat):+ compat = resolve_anthropic_compat(definition.compat)+ else:+ compat = resolve_anthropic_compat(AnthropicCompat.preset(definition.compat or "anthropic"))params.update({"stop", "top_k"})if schema and compat.structured_output != "reject":params.add("response_format")@@ -174,7 +204,10 @@ def _capabilities(lm, options=None):elif definition.dialect == "gemini":params.update({"stop", "top_k", "logprobs", "top_logprobs", "response_format"})else:- compat = resolve_openai_responses_compat(OpenAIResponsesCompat.preset(definition.compat or "openai"))+ if isinstance(definition.compat, OpenAIResponsesCompat):+ compat = resolve_openai_responses_compat(definition.compat)+ else:+ compat = resolve_openai_responses_compat(OpenAIResponsesCompat.preset(definition.compat or "openai"))params.update({"response_format", "max_completion_tokens", "logprobs", "top_logprobs", "store"})if compat.reasoning_format == "none":reasoning = Falsediff --git a/dspy/clients/costs.py b/dspy/clients/costs.pyindex 834e4dfc7..56b0825be 100644--- a/dspy/clients/costs.py+++ b/dspy/clients/costs.py@@ -9,20 +9,23 @@ import refrom dspy.clients.model_metadata import model_info, rate, source_info-def estimate_cost(response, *, provider, requested_model, request=None):+def estimate_cost(response, *, provider, requested_model, request=None, namespaces=None):"""Return (estimated USD cost, provenance), or (None, reason metadata).Advisory metadata must never turn a completed generation into a failure.+ ``namespaces`` are a declared provider's metadata namespaces; the price+ comes from them alone, and the provenance names them."""try:- return _estimate_cost(response, provider=provider, requested_model=requested_model, request=request)+ return _estimate_cost(response, provider=provider, requested_model=requested_model, request=request,+ namespaces=namespaces)except Warning:raiseexcept Exception:return None, {"kind": "unknown", "reason": "pricing metadata could not be interpreted"}-def _estimate_cost(response, *, provider, requested_model, request=None):+def _estimate_cost(response, *, provider, requested_model, request=None, namespaces=None):from dspy.lm15 import BuiltinTooldef unknown(reason):@@ -36,10 +39,12 @@ def _estimate_cost(response, *, provider, requested_model, request=None):return unknown("subscription usage is not per-token API billing")if request is not None and any(isinstance(tool, BuiltinTool) for tool in request.tools):return unknown("hosted-tool fees are not included in token rates")- info = model_info(provider, response.model)+ if namespaces is not None and not namespaces:+ return unknown("declared provider names no metadata namespace for pricing")+ info = model_info(provider, response.model, namespaces=namespaces)priced_model = response.modelif not info:- info = model_info(provider, requested_model)+ info = model_info(provider, requested_model, namespaces=namespaces)priced_model = requested_modelu = response.usageif u.input_tokens is None or u.output_tokens is None:@@ -101,5 +106,5 @@ def _estimate_cost(response, *, provider, requested_model, request=None):return unknown("no matching pricing entry")return sum(count * amount for count, amount in dimensions if count and amount is not None), {"kind": "estimate", "currency": "USD", "provider": provider, "model": priced_model,- "metadata": source_info(),+ "metadata": {**source_info(), **({"namespaces": list(namespaces)} if namespaces else {})},}diff --git a/dspy/clients/engines/litellm_engine.py b/dspy/clients/engines/litellm_engine.pyindex 7f594aca3..8f4fa20f0 100644--- a/dspy/clients/engines/litellm_engine.py+++ b/dspy/clients/engines/litellm_engine.py@@ -97,7 +97,7 @@ class _ChatStream:class _LiteLLMConfig:- def __init__(self, *, model_type="chat", **client_options):+ def __init__(self, *, model_type="chat", wire_model=None, **client_options):if model_type not in {"chat", "responses", "text"}:raise UnsupportedFeatureError("Typed LiteLLM engines support chat and Responses APIs. Use ordinary LM calls for text completions."@@ -112,6 +112,10 @@ class _LiteLLMConfig:raise TypeError(f"Unknown engine client options: {sorted(unknown)}. Generation options belong in Request.config.")self.model_type = model_typeself.client_options = dict(client_options)+ # The model string LiteLLM receives when it differs from the LM's+ # (a declared provider reached through LiteLLM's generic door). The+ # LM's own string stays on the request, the history and the cache key.+ self.wire_model = wire_modelself._closed = Falsedef _arguments(self, request, *, streaming=False):@@ -139,7 +143,7 @@ class _LiteLLMConfig:# as a LiteLLM argument rather than dropped by the conversion.data["top_k"] = request.config.top_kdata.update(self.client_options)- data.update(model=request.model, num_retries=0, cache={"no-cache": True, "no-store": True})+ data.update(model=self.wire_model or request.model, num_retries=0, cache={"no-cache": True, "no-store": True})if self.model_type == "chat":data["n"] = 1if streaming:@@ -180,6 +184,8 @@ class LiteLLMEngine(_LiteLLMConfig):fn = {"chat": lm_module.litellm_completion, "text": lm_module.litellm_text_completion,"responses": lm_module.litellm_responses_completion}[self.model_type]+ if self.wire_model is not None:+ request = {**request, "model": self.wire_model, **self.client_options}with litellm_errors(model=lm.model):raw = fn(request=request, num_retries=0)completed_legacy(raw)@@ -224,6 +230,8 @@ class AsyncLiteLLMEngine(_LiteLLMConfig):fn = {"chat": lm_module.alitellm_completion, "text": lm_module.alitellm_text_completion,"responses": lm_module.alitellm_responses_completion}[self.model_type]+ if self.wire_model is not None:+ request = {**request, "model": self.wire_model, **self.client_options}with litellm_errors(model=lm.model):raw = await fn(request=request, num_retries=0)completed_legacy(raw)diff --git a/dspy/clients/engines/lm15_engine.py b/dspy/clients/engines/lm15_engine.pyindex 038945cbb..78da9568a 100644--- a/dspy/clients/engines/lm15_engine.py+++ b/dspy/clients/engines/lm15_engine.py@@ -5,6 +5,7 @@ import threadingfrom collections.abc import AsyncIterator, Iteratorfrom dataclasses import replace+from dspy._vendor.lm15.registry import canonical_providerfrom dspy._vendor.lm15.router import LITELLM_PROVIDER_PREFIXESfrom dspy._vendor.lm15.types import StreamEventfrom dspy.clients.engines.base import validate_request@@ -58,14 +59,24 @@ def timeouts_for(timeout) -> Timeouts | None:return Timeouts(**kwargs) if kwargs else None-def _model_string(model: str, model_type: str) -> str:- """Keep endpoint selection deliberate, including model ids containing ':' ."""+def _model_string(model: str, model_type: str, providers=()) -> str:+ """Keep endpoint selection deliberate, including model ids containing ':' .++ A ``provider/model`` prefix is a LiteLLM spelling lm15 has a door for, or+ a provider declared on this engine's config, by id or alias in either+ spelling.+ """head, separator, rest = model.partition("/")- if separator and head in LITELLM_PROVIDER_PREFIXES:- provider = LITELLM_PROVIDER_PREFIXES[head]- if model_type == "responses":- provider = {"openai-chat": "openai", "azure-chat": "azure"}.get(provider, provider)- return f"{provider}:{rest}"+ if separator and rest:+ if head in LITELLM_PROVIDER_PREFIXES:+ provider = LITELLM_PROVIDER_PREFIXES[head]+ if model_type == "responses":+ provider = {"openai-chat": "openai", "azure-chat": "azure"}.get(provider, provider)+ return f"{provider}:{rest}"+ spelling = canonical_provider(head)+ for definition in providers:+ if spelling in definition.spellings:+ return f"{definition.id}:{rest}"return model@@ -81,13 +92,14 @@ class _Routing:def resolve(self, model: str):"""Offline routing. Callers inspect canonical errors, not a wrapped cause."""- routed = _model_string(model, self.model_type)+ routed = _model_string(model, self.model_type, self.config.providers)if self.model_type == "chat":return self.router.resolve_openai_chat(routed)resolution = self.router.resolve(routed)from dspy._vendor.lm15.registry import lookup- definition = lookup(resolution.provider)+ declared = {d.id: d for d in self.config.providers}+ definition = declared.get(resolution.provider) or lookup(resolution.provider)if definition is None or definition.dialect != "openai-responses":raise UnsupportedFeatureError(f"{model!r} does not select a Responses API endpoint.")return resolutiondiff --git a/dspy/clients/execution.py b/dspy/clients/execution.pyindex 7962c1002..dccc74a6c 100644--- a/dspy/clients/execution.py+++ b/dspy/clients/execution.py@@ -2,6 +2,7 @@import asyncioimport copy+import osimport timefrom contextlib import nullcontextfrom dataclasses import dataclass, replace@@ -35,7 +36,7 @@ from dspy.lm15 import (UnsupportedFeatureError,request_from_openai_chat,)-from dspy.utils.exceptions import LMUnsupportedFeatureError, is_retryable_lm_error+from dspy.utils.exceptions import LMConfigurationError, LMUnsupportedFeatureError, is_retryable_lm_errorfrom dspy.utils.lazy_import import requireanyio = require("anyio")@@ -189,6 +190,70 @@ def _canonical(call, *, compat=None):return request+# LiteLLM's generic doors, by lm15 dialect: the route a declared provider+# takes when a call falls back, with the declared address and credential+# carried along. The provider's own aliases are never handed to LiteLLM as+# a prefix: LiteLLM may know that name as a different service. Each door+# sends the credential under one scheme (the OpenAI SDK's bearer header,+# the Anthropic SDK's x-api-key header) — the only scheme it can carry.+_LITELLM_GENERIC_DOOR = {+ "openai-chat": ("openai", "bearer"),+ "openai-responses": ("openai", "bearer"),+ "anthropic": ("anthropic", "x-api-key"),+}+++def _declared_fallback(lm, binding, resolution, clients):+ """(wire model, client options) for LiteLLM to reach a declared provider+ at its declared address with its own credential, sent under the scheme+ the declaration selects for it — never LiteLLM's idea of a similarly+ named service, never an ambient OPENAI_API_KEY, never a scheme the+ declaration did not name."""+ from dspy._vendor.lm15.access import select_scheme+ from dspy._vendor.lm15.credentials import AwsCredentials+ from dspy._vendor.lm15.providers.base import resolve_credential_value++ definition = binding.definition+ door, sends = _LITELLM_GENERIC_DOOR.get(definition.dialect, (None, None))+ if door is None or definition.hosted:+ raise LMUnsupportedFeatureError(+ f"{lm.model!r}: the declared provider {definition.id!r} has no LiteLLM route for these client "+ f"settings; use settings the native engine carries (api_key, api_base, timeout)",+ model=lm.model, provider=definition.id,+ )+ options = dict(clients)+ if not (options.get("api_base") or options.get("base_url")):+ options["api_base"] = definition.access.base_url+ key = options.get("api_key")+ if not key:+ key = next((os.environ[name] for name in definition.access.env_keys if os.environ.get(name)), None)+ key = key or definition.placeholder_key+ if not key:+ variables = " or ".join(definition.access.env_keys) or "api_key="+ raise LMConfigurationError(+ f"no credential for the declared provider {definition.id!r}: set {variables} or pass api_key=",+ model=lm.model, provider=definition.id,+ )+ # The scheme this credential travels under natively (lm15 AUTH-2). A+ # callable credential is invoked once per call, as the native path does.+ # The door renders exactly one scheme; a declaration that needs another+ # gets no fallback rather than its secret in a header it did not declare.+ credential = resolve_credential_value(key)+ scheme = select_scheme(definition.access, credential)+ if scheme != sends or isinstance(credential, AwsCredentials):+ raise LMUnsupportedFeatureError(+ f"{lm.model!r}: the declared provider {definition.id!r} authenticates with {scheme!r}, which the "+ f"LiteLLM {door}/ door cannot send (it sends {sends!r}); use settings the native engine carries "+ "(api_key, api_base, timeout)",+ model=lm.model, provider=definition.id, features=[scheme],+ )+ options["api_key"] = credential.value # LiteLLM takes the string+ static = dict(definition.access.headers)+ if static:+ options["extra_headers"] = {**static, **(options.get("extra_headers") or {})}+ return f"{door}/{resolution.model}", options++def _engine(lm, call, asynchronous):# Setup is outside the retry loop. Canonical routing/conversion refusals# still need the same public error projection as engine-call failures.@@ -225,19 +290,27 @@ def _select_engine(lm, call, asynchronous):return backend, _canonical(call), Noneselection = select_backend(lm, call.legacy)native, resolution, clients = selection.native, selection.resolution, selection.clients+ from dspy.lm15 import _binding_for, _definitions++ binding = _binding_for(lm._providers, resolution.provider) if resolution is not None and resolution.declared else Nonecanonical = call.requestif native:try:canonical = _canonical(call, compat=resolution.compat)except (LM15Error, TypeError, ValueError) as exc:- if spec == "lm15" or call.request is not None or call.legacy.get("prompt_cache") is not None:+ # A declared provider's compat is the caller's own statement of+ # what that server takes; a refusal it produces is final, not a+ # reason to send the request through LiteLLM anyway.+ if (spec == "lm15" or binding is not None or call.request is not None+ or call.legacy.get("prompt_cache") is not None):if isinstance(exc, UnsupportedFeatureError):raise # The outer boundary preserves its feature and other diagnostics.raise LMUnsupportedFeatureError(str(exc), model=lm.model) from exc# This is a representational refusal BEFORE execution. Preserve the# original body on the compatibility backend, never retry elsewhere.native = False- if not native:++ def compatibility_engine():if call.legacy.get("prompt_cache") is not None:raise LMUnsupportedFeatureError("This request requires LiteLLM, which has no prompt_cache bridge. "@@ -246,7 +319,18 @@ def _select_engine(lm, call, asynchronous):# None disables the bridge; it is never a provider keyword.call.legacy.pop("prompt_cache", None)cls = AsyncLiteLLMEngine if asynchronous else LiteLLMEngine+ if binding is not None:+ # A declared provider keeps its destination and credential across+ # the backend switch; only the LM's model string stays as it was+ # (history, cache key). It is still that provider: errors and+ # pricing name it and read its metadata namespaces.+ wire_model, options = _declared_fallback(lm, binding, resolution, clients)+ engine = cls(model_type=lm.model_type, wire_model=wire_model, **options)+ return engine, canonical if call.request else None, resolution.providerreturn cls(model_type=lm.model_type, **clients), canonical if call.request else None, None++ if not native:+ return compatibility_engine()# Long-lived sync pools and a separate async pool per event loop. Copies# share the store; a copy with changed client settings gets a distinct key.loop = asyncio.get_running_loop() if asynchronous else None@@ -260,7 +344,8 @@ def _select_engine(lm, call, asynchronous):provider = resolution.providerapi_keys = {provider: clients["api_key"]} if "api_key" in clients else Noneurl = clients.get("api_base") or clients.get("base_url")- config = RouterConfig(api_keys=api_keys, base_urls={provider: url} if url else None, timeouts=timeouts)+ config = RouterConfig(api_keys=api_keys, base_urls={provider: url} if url else None, timeouts=timeouts,+ providers=_definitions(lm._providers))cls = AsyncLM15Engine if asynchronous else LM15Enginebackend = cls(config, model_type=lm.model_type)lm._engine_store[key] = backend@@ -272,8 +357,12 @@ def _select_engine(lm, call, asynchronous):try:backend.plan(canonical)except UnsupportedFeatureError:- cls = AsyncLiteLLMEngine if asynchronous else LiteLLMEngine- return cls(model_type=lm.model_type, **clients), canonical if call.request else None, None+ if binding is not None:+ # The declaration is the authority on what that server cannot+ # do; sending the request through LiteLLM anyway would defeat+ # the refusal it encodes (a silent degrade). Refuse.+ raise+ return compatibility_engine()return backend, canonical, resolution.provider@@ -324,6 +413,23 @@ def _result(lm, response, call, provider, request=None, *, estimate=True):return result+def _price_legacy(lm, result, provider):+ """A declared provider reached through LiteLLM's generic door is still+ that provider: its usage is priced from the declaration's namespaces,+ replacing LiteLLM's reading of the door's model string (a colliding+ name would be priced as another vendor's; an unknown one not at all).+ Chat only; other model types keep LiteLLM's figure."""+ if provider is None or result.raw is None or lm.model_type != "chat":+ return+ try:+ from dspy._vendor.lm15.providers.openai_chat import response_from_openai_chat++ response = response_from_openai_chat(plain(result.raw), model=lm.model)+ except Exception:+ return # advisory: never fail a completed call over pricing+ _price_result(lm, result, response, provider, None)++def _price_result(lm, result, response, provider, request):"""Advisory pricing only: safe to finish in a worker after cancellation.@@ -333,12 +439,18 @@ def _price_result(lm, result, response, provider, request):try:wire_model = request.model if request is not None else lm.model+ namespaces = Noneif provider is not None:from dspy.clients.capabilities import resolve+ from dspy.lm15 import _binding_for- wire_model = resolve(lm).model+ route = resolve(lm)+ wire_model = route.model+ binding = _binding_for(getattr(lm, "_providers", ()), provider) if route.declared else None+ if binding is not None:+ namespaces = binding.metadata_namespacesresult.cost, result.cost_details = estimate_cost(- response, provider=provider, requested_model=wire_model, request=request,+ response, provider=provider, requested_model=wire_model, request=request, namespaces=namespaces,)except Warning:raise@@ -447,6 +559,7 @@ def _attempt(lm, call, backend, request, provider, state, results):result = backend.complete_legacy(lm, copy.deepcopy(call.legacy), prompt=call.prompt,messages=call.messages, call_kwargs=call.kwargs)_accept_legacy(state, result)+ _price_legacy(lm, result, provider)finally:_legacy_done(state, results, progress)elif stream is None:@@ -524,6 +637,7 @@ async def _aattempt(lm, call, backend, request, provider, state, results):result = await backend.complete_legacy(lm, copy.deepcopy(call.legacy), prompt=call.prompt,messages=call.messages, call_kwargs=call.kwargs)_accept_legacy(state, result)+ await asyncio.to_thread(_price_legacy, lm, result, provider) # metadata lookup off the loopfinally:_legacy_done(state, results, progress)elif stream is None:diff --git a/dspy/clients/lm.py b/dspy/clients/lm.pyindex 0ddf349f0..7b817bf8f 100644--- a/dspy/clients/lm.py+++ b/dspy/clients/lm.py@@ -109,7 +109,8 @@ def _refuse_client_settings(engine, kwargs, *, where: str = "dspy.LM") -> None:if present:raise ValueError(f"{where}: {present} configure a connection DSPy owns; a custom engine owns its own. "- "Configure them on the engine instead."+ "Configure them on the engine, or for an HTTP provider declare it with "+ "dspy.lm15.register_provider(...) and drop engine=")@@ -264,6 +265,12 @@ class LM(BaseLM):kwargs["prompt_cache"] = prompt_cacheself._engine_spec = engineself._async_engine_spec = async_engine+ # The declared providers this LM routes with, bound now and kept for+ # its life (copies share them): selection, capabilities, pricing and+ # both engines read one tuple, so a later registration cannot split them.+ from dspy.lm15 import registered_providers++ self._providers = registered_providers()self._engine_store = {}self._engine_lock = threading.RLock()super().__init__(@@ -450,6 +457,7 @@ class LM(BaseLM):self.__dict__.update(state)self._engine_spec = getattr(self, "_engine_spec", "auto")self._async_engine_spec = getattr(self, "_async_engine_spec", None)+ self._providers = getattr(self, "_providers", ())self._engine_lock = threading.RLock()self._engine_store = {}diff --git a/dspy/clients/model_metadata/__init__.py b/dspy/clients/model_metadata/__init__.pyindex 5767794a8..fcfbb1c44 100644--- a/dspy/clients/model_metadata/__init__.py+++ b/dspy/clients/model_metadata/__init__.py@@ -105,14 +105,19 @@ def _provider_names(provider):}.get(provider, (provider,))-def model_info(provider, model):+def model_info(provider, model, *, namespaces=None):"""Return a copy of the most specific matching entry, or an empty dict.Missing capability fields may inherit from a provider-matched bare entry;explicit false values do not. Prices never inherit from another entry.+ ``namespaces`` names the snapshot namespaces to read for a declared+ provider (its registration's ``metadata_namespaces``); an empty tuple+ reads nothing. Without it the provider's own namespace table applies."""data = _load()- names = _provider_names(provider)+ names = tuple(namespaces) if namespaces is not None else _provider_names(provider)+ if not names:+ return {}keys = [f"{name}/{model}" for name in names]bare = data.get(model)if isinstance(bare, dict) and bare.get("litellm_provider") in names:diff --git a/dspy/lm15.py b/dspy/lm15.pyindex df15e268f..3e39bda8d 100644--- a/dspy/lm15.py+++ b/dspy/lm15.py@@ -9,7 +9,201 @@ Examples:>>> request = Request(model="example", messages=(Message.user("Hello"),))>>> request.model'example'++This module also holds the providers declared to DSPy's native engine for+the process (`register_provider`). lm15 routes model strings through a+registry of receipted providers and lets a caller declare more with+`RouterConfig(providers=...)`. A registration is an environment fact, like+an API-key variable: a saved program records only the model string, and+loading it needs the same registration in place. Each `dspy.LM` binds the+registrations present when it is constructed and keeps them for its whole+life, so one LM never sees two definitions of a provider."""+import threading as _threading+from dataclasses import dataclass as _dataclass+from typing import Mapping as _Mapping+from dspy._vendor.lm15 import *-from dspy._vendor.lm15 import __all__ as __all__+from dspy._vendor.lm15 import __all__ as _vendored_all+from dspy._vendor.lm15.compat import AnthropicCompat, OpenAIChatCompat, OpenAIResponsesCompat+from dspy._vendor.lm15.registry import ProviderDefinition+from dspy._vendor.lm15.router import RouterConfig++__all__ = [+ *_vendored_all,+ "AnthropicCompat",+ "ModelSupport",+ "OpenAIChatCompat",+ "OpenAIResponsesCompat",+ "ProviderDefinition",+ "RegisteredProvider",+ "register_provider",+ "registered_providers",+ "unregister_provider",+]+++@_dataclass(frozen=True)+class ModelSupport:+ """What a model behind a declared provider supports, stated by the caller.++ A compat policy says what the wire accepts; whether a model honours it is+ model knowledge. None means "not stated": DSPy then reads the metadata+ snapshot through the provider's ``metadata_namespaces``, and treats what+ is still unknown as unsupported (the rule built-in providers live with).+ """++ function_calling: bool | None = None+ reasoning: bool | None = None+ response_schema: bool | None = None++ def __post_init__(self) -> None:+ for name in ("function_calling", "reasoning", "response_schema"):+ value = getattr(self, name)+ if value is not None and not isinstance(value, bool):+ raise TypeError(f"ModelSupport.{name} is True, False or None, got {type(value).__name__}")++ def over(self, other: "ModelSupport | None") -> "ModelSupport":+ """This support, with unstated fields taken from ``other``."""+ if other is None:+ return self+ return ModelSupport(+ function_calling=self.function_calling if self.function_calling is not None else other.function_calling,+ reasoning=self.reasoning if self.reasoning is not None else other.reasoning,+ response_schema=self.response_schema if self.response_schema is not None else other.response_schema,+ )+++@_dataclass(frozen=True)+class RegisteredProvider:+ """A declared provider as DSPy binds it: the lm15 definition (routing,+ credentials, wire) plus what DSPy alone needs — which metadata-snapshot+ namespaces may describe and price its models, and the caller's own+ statements of model support. Aliases on the definition are routing+ spellings only; they grant no metadata."""++ definition: ProviderDefinition+ metadata_namespaces: tuple[str, ...] = ()+ supports: ModelSupport | None = None+ models: tuple[tuple[str, ModelSupport], ...] = () # (model id, statement), immutable and picklable++ @property+ def id(self) -> str:+ return self.definition.id++ def support_for(self, model: str) -> ModelSupport:+ """The caller's statement for ``model``: its own entry over the+ provider-wide default; unstated fields stay None."""+ own = next((support for model_id, support in self.models if model_id == model), None)+ if own is None:+ return self.supports or ModelSupport()+ return own.over(self.supports)+++_registry_lock = _threading.Lock()+_registered: dict[str, RegisteredProvider] = {}+++def register_provider(+ definition: ProviderDefinition,+ *,+ supports: ModelSupport | None = None,+ models: _Mapping[str, ModelSupport] | None = None,+ metadata_namespaces: tuple[str, ...] = (),+ replace: bool = False,+) -> RegisteredProvider:+ """Declare a provider for the native ``dspy.LM``s constructed after this call.++ ``definition`` is an lm15 ``ProviderDefinition`` (``ProviderDefinition.chat(access,+ compat=...)`` for an OpenAI-compatible server). Afterwards+ ``dspy.LM("<id>/<model>")`` — and every alias — routes natively with the+ LM's ``api_key``, ``api_base`` and ``timeout`` honored, and the provider's+ own key variable read when none is given. A call the native route cannot+ carry falls back to LiteLLM at the declared address with the same+ credential, never at an address LiteLLM knows by a similar name.++ ``metadata_namespaces`` names the snapshot namespaces (LiteLLM's+ spellings, e.g. ``("fireworks_ai",)``) whose entries may describe and+ price this provider's models. Give it only when the endpoint really is+ that service; nothing is inherited from an alias. ``supports`` states+ model support for the whole provider and ``models`` per model id; a+ snapshot entry, when one is found, is the more specific fact and wins.++ Registering the same definition again with the same statements is a+ no-op. A different registration under the same id is refused unless+ ``replace=True``, and even then only LMs constructed afterwards bind it.+ A spelling another registered provider, lm15's registry or LiteLLM+ already uses is always refused.++ Returns:+ The registration as DSPy holds it.+ """+ if not isinstance(definition, ProviderDefinition):+ raise TypeError(+ "register_provider takes a dspy.lm15 ProviderDefinition; declare one with "+ f"ProviderDefinition.chat(access, compat=...), not {type(definition).__name__}"+ )+ if supports is not None and not isinstance(supports, ModelSupport):+ raise TypeError(f"supports= takes a dspy.lm15.ModelSupport, not {type(supports).__name__}")+ if not isinstance(metadata_namespaces, tuple) or not all(isinstance(n, str) and n for n in metadata_namespaces):+ raise TypeError("metadata_namespaces= is a tuple of non-empty snapshot namespace names")+ per_model: dict[str, ModelSupport] = {}+ for model_id, support in (models or {}).items():+ if not isinstance(model_id, str) or not model_id or not isinstance(support, ModelSupport):+ raise TypeError("models= maps model ids to dspy.lm15.ModelSupport")+ per_model[model_id] = support+ binding = RegisteredProvider(+ definition=definition,+ metadata_namespaces=metadata_namespaces,+ supports=supports,+ models=tuple(per_model.items()),+ )+ with _registry_lock:+ current = _registered.get(definition.id)+ if current is not None and current == binding:+ return current+ if current is not None and not replace:+ raise ValueError(+ f"provider {definition.id!r} is already registered with a different registration; "+ "pass replace=True to replace it (LMs constructed afterwards bind the replacement)"+ )+ for other in _registered.values():+ if other.id != definition.id and set(other.definition.spellings) & set(definition.spellings):+ clash = sorted(set(other.definition.spellings) & set(definition.spellings))+ raise ValueError(+ f"provider {definition.id!r} spells {clash} like the registered provider "+ f"{other.id!r}; unregister_provider({other.id!r}) first"+ )+ kept = {pid: b for pid, b in _registered.items() if pid != definition.id}+ # lm15 validates the set: type, hyphenation, and no spelling a+ # receipted entry or LiteLLM prefix already uses.+ RouterConfig(providers=(*(b.definition for b in kept.values()), definition))+ _registered.clear()+ _registered.update(kept)+ _registered[definition.id] = binding+ return binding+++def unregister_provider(provider: str) -> None:+ """Forget a registered provider. Unknown ids are ignored. LMs that+ already bound it keep it."""+ with _registry_lock:+ _registered.pop(provider, None)+++def registered_providers() -> tuple[RegisteredProvider, ...]:+ """The providers registered in this process, in registration order —+ the tuple a ``dspy.LM`` binds when constructed."""+ with _registry_lock:+ return tuple(_registered.values())+++def _definitions(bindings) -> tuple[ProviderDefinition, ...]:+ """The lm15 definitions of an LM's bound providers, for RouterConfig."""+ return tuple(b.definition for b in bindings)+++def _binding_for(bindings, provider: str) -> RegisteredProvider | None:+ """The LM's binding for a provider id, or None for a registry provider."""+ return next((b for b in bindings if b.id == provider), None)diff --git a/tests/clients/test_lm_async_execution.py b/tests/clients/test_lm_async_execution.pyindex 452bf72d1..49f10e787 100644--- a/tests/clients/test_lm_async_execution.py+++ b/tests/clients/test_lm_async_execution.py@@ -292,7 +292,7 @@ async def test_cost_lookup_runs_off_loop(endpoint, monkeypatch):threads = []loop_thread = threading.get_ident()- def info(*args):+ def info(*args, **kwargs):threads.append(threading.get_ident())return {"input_cost_per_token": 0.001, "output_cost_per_token": 0.002}diff --git a/tests/clients/test_registered_providers.py b/tests/clients/test_registered_providers.pynew file mode 100644index 000000000..44d5ff58e--- /dev/null+++ b/tests/clients/test_registered_providers.py@@ -0,0 +1,507 @@+"""dspy.lm15.register_provider: an HTTP provider lm15's registry does not+list becomes a native route for every dspy.LM constructed afterwards, with+the LM's client settings honored — the alternative to writing a custom+engine for it (cmpnd-ai/breaka-your-lm gauntlet/fireworks.py, 2026-09-16).++The rules pinned here came out of review of the first design (dspy#10441):+one binding per LM, a fallback that keeps the declared destination and+credential, and aliases that grant nothing but a spelling.+"""++import asyncio+import json+import threading+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++import pytest++import dspy+from dspy.clients.backend_selection import select_backend+from dspy.clients.execution import _select_engine, prepare+from dspy.lm15 import (+ AccessPolicy,+ EndpointSupport,+ ModelSupport,+ OpenAIChatCompat,+ ProviderDefinition,+ RegisteredProvider,+ register_provider,+ registered_providers,+ unregister_provider,+)++MODEL = "accounts/fireworks/models/deepseek-v4-flash" # in the bundled metadata snapshot (tools, reasoning, schema, prices)+++def server():+ seen = []++ class Handler(BaseHTTPRequestHandler):+ def do_POST(self):+ body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))+ seen.append({"auth": self.headers.get("Authorization"), "path": self.path, "body": body,+ "headers": dict(self.headers)})+ out = {"id": "x", "object": "chat.completion", "model": body["model"],+ "choices": [{"index": 0, "message": {"role": "assistant", "content": "Paris"}, "finish_reason": "stop"}],+ "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15}}+ data = json.dumps(out).encode()+ self.send_response(200)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(data)))+ self.end_headers()+ self.wfile.write(data)++ def log_message(self, *args):+ pass++ http = ThreadingHTTPServer(("127.0.0.1", 0), Handler)+ worker = threading.Thread(target=http.serve_forever, daemon=True)+ worker.start()+ try:+ yield f"http://127.0.0.1:{http.server_port}/v1", seen+ finally:+ http.shutdown()+ http.server_close()+ worker.join()+++def _definition(base_url, provider="fireworks", aliases=("fireworks-ai",), headers=(), auth_scheme=("bearer",), **compat):+ compat.setdefault("max_tokens_field", "max_tokens")+ return ProviderDefinition.chat(+ AccessPolicy(provider=provider, supports=EndpointSupport(complete=True, stream=True, models=True),+ auth_modes=("bearer",), auth_scheme=auth_scheme, env_keys=("FIREWORKS_API_KEY",),+ base_url=base_url, headers=tuple(headers)),+ compat=OpenAIChatCompat(**compat),+ aliases=aliases, note="Fireworks (test)",+ )++[email protected](autouse=True)+def clean_registry(monkeypatch):+ import dspy.clients.model_metadata as metadata++ for name in ("FIREWORKS_API_KEY", "FIREWORKS_API_BASE", "FIREWORKS_BASE_URL", "OPENAI_API_KEY"):+ monkeypatch.delenv(name, raising=False)+ # Capabilities and prices come from the bundled metadata snapshot, not+ # from whatever the live LiteLLM map says today (or whichever a sibling+ # test loaded first).+ monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")+ monkeypatch.setattr(metadata, "_data", None)+ monkeypatch.setattr(metadata, "_source", {})+ before = {b.id for b in registered_providers()}+ yield+ for binding in registered_providers():+ if binding.id not in before:+ unregister_provider(binding.id)+++def litellm_stub(monkeypatch):+ """LiteLLM as the fallback would call it: records the arguments, answers."""+ import litellm++ calls = []++ def completion(**kwargs):+ calls.append(kwargs)+ return litellm.ModelResponse(**{+ "id": "x", "object": "chat.completion", "model": kwargs["model"],+ "choices": [{"index": 0, "message": {"role": "assistant", "content": "[[ ## answer ## ]]\nParis\n\n[[ ## completed ## ]]"},+ "finish_reason": "stop"}],+ "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},+ })++ async def acompletion(**kwargs):+ return completion(**kwargs)++ monkeypatch.setattr(litellm, "completion", completion)+ monkeypatch.setattr(litellm, "acompletion", acompletion)+ return calls+++# ─── the registry ─────────────────────────────────────────────────────+++def test_register_is_idempotent_and_replace_is_explicit():+ definition = _definition("https://api.fireworks.test/v1")+ binding = register_provider(definition)+ assert isinstance(binding, RegisteredProvider) and binding.definition is definition+ assert register_provider(definition) is binding # same registration: no-op+ with pytest.raises(ValueError, match="already registered with a different registration"):+ register_provider(_definition("https://other.test/v1"))+ with pytest.raises(ValueError, match="already registered with a different registration"):+ register_provider(definition, metadata_namespaces=("fireworks_ai",)) # statements are part of it+ replaced = register_provider(_definition("https://other.test/v1"), replace=True)+ assert registered_providers() == (replaced,)+ unregister_provider("fireworks")+ assert registered_providers() == ()+ unregister_provider("fireworks") # unknown: ignored+++def test_arguments_are_type_checked():+ definition = _definition("https://x.test/v1")+ with pytest.raises(TypeError, match="ProviderDefinition"):+ register_provider("fireworks")+ with pytest.raises(TypeError, match="ModelSupport"):+ register_provider(definition, supports={"function_calling": True})+ with pytest.raises(TypeError, match="models= maps"):+ register_provider(definition, models={MODEL: True})+ with pytest.raises(TypeError, match="metadata_namespaces"):+ register_provider(definition, metadata_namespaces="fireworks_ai")+ with pytest.raises(TypeError, match="True, False or None"):+ ModelSupport(function_calling="yes")+++def test_spellings_lm15_or_litellm_already_use_are_refused():+ for taken in ("groq", "ollama-chat"):+ with pytest.raises(Exception, match="already names"):+ register_provider(_definition("https://x.test/v1", provider="new-door", aliases=(taken,)))+ register_provider(_definition("https://x.test/v1"))+ with pytest.raises(ValueError, match=r"spells \['fireworks-ai'\] like the registered provider 'fireworks'"):+ register_provider(_definition("https://x.test/v1", provider="other", aliases=("fireworks-ai",)))+++# ─── one binding per LM ───────────────────────────────────────────────+++def test_an_lm_binds_the_registrations_present_at_construction(server):+ base, _ = server+ before = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False)+ register_provider(_definition(base))+ after = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False, num_retries=0)+ assert not select_backend(before).native # bound nothing; stays as it was+ assert select_backend(after).native+ assert after("hi") == ["Paris"]+ assert after.copy(temperature=0.5)._providers is after._providers # copies share the binding+ after.close()+++def test_replacing_a_registration_never_splits_an_lm(server):+ base, _ = server+ register_provider(_definition(base, max_tokens_field="max_tokens"))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False, num_retries=0)+ call = prepare(lm, "hi", None, {"max_tokens": 7})+ sync_engine, _, _ = _select_engine(lm, call, False)+ register_provider(_definition("http://127.0.0.1:9/v1", max_tokens_field="max_completion_tokens"), replace=True)+ # Selection, the cached sync engine and a fresh async engine all read the+ # LM's own binding, not the registry as it is now.+ assert select_backend(lm).resolution.compat.max_tokens_field == "max_tokens"+ same, _, _ = _select_engine(lm, call, False)+ assert same is sync_engine and same.config.providers[0].access.base_url == base++ async def async_engine():+ engine, _, _ = _select_engine(lm, call, True)+ return engine.config.providers[0].access.base_url++ assert asyncio.run(async_engine()) == base+ assert lm("hi") == ["Paris"] # still the address it was built with+ fresh = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False, num_retries=0)+ assert select_backend(fresh).resolution.compat.max_tokens_field == "max_completion_tokens"+ lm.close()+++# ─── routing through dspy.LM ──────────────────────────────────────────++[email protected]("prefix", ["fireworks", "fireworks_ai", "fireworks-ai"])+def test_registered_provider_routes_natively_with_the_lm_key(server, prefix):+ base, seen = server+ register_provider(_definition(base))+ lm = dspy.LM(f"{prefix}/{MODEL}", api_key="lm-key", cache=False, num_retries=0)+ selection = select_backend(lm)+ assert selection.native and selection.resolution.provider == "fireworks" and selection.resolution.declared+ assert lm("capital of France?") == ["Paris"]+ assert seen[-1]["auth"] == "Bearer lm-key"+ assert seen[-1]["path"] == "/v1/chat/completions"+ assert seen[-1]["body"]["model"] == MODEL+ lm.close()+++def test_env_key_and_client_settings_are_honored(server, monkeypatch):+ base, seen = server+ register_provider(_definition(base))+ monkeypatch.setenv("FIREWORKS_API_KEY", "env-key")+ lm = dspy.LM(f"fireworks/{MODEL}", cache=False, num_retries=0, max_tokens=9, timeout=5)+ assert select_backend(lm).native+ assert lm("hi") == ["Paris"]+ assert seen[-1]["auth"] == "Bearer env-key"+ assert seen[-1]["body"]["max_tokens"] == 9 # the declared compat's spelling+ lm.close()+ moved = dspy.LM(f"fireworks/{MODEL}", api_base="http://127.0.0.1:9/v1", cache=False, num_retries=0)+ with pytest.raises(dspy.LMError):+ moved("hi") # api_base moves the call; a closed port fails, never a silent success elsewhere+ monkeypatch.delenv("FIREWORKS_API_KEY")+ with pytest.raises(dspy.LMError, match="FIREWORKS_API_KEY"):+ dspy.LM(f"fireworks/{MODEL}", cache=False, num_retries=0)("hi")+++def test_a_declaration_beats_the_ambient_gateway_variable(server, monkeypatch):+ # {PREFIX}_API_BASE redirects engine="auto" to LiteLLM for registry+ # providers (legacy gateways). A declaration states its address itself.+ base, seen = server+ register_provider(_definition(base))+ monkeypatch.setenv("FIREWORKS_API_BASE", "http://127.0.0.1:9/v1")+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False, num_retries=0)+ assert select_backend(lm).native+ assert lm("hi") == ["Paris"] and seen[-1]["path"] == "/v1/chat/completions"+ lm.close()+++async def test_async_path(server):+ base, seen = server+ register_provider(_definition(base))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="lm-key", cache=False, num_retries=0)+ assert await lm.acall("hi") == ["Paris"]+ assert seen[-1]["auth"] == "Bearer lm-key"+ await lm.aclose()+++def test_unregistered_prefix_is_not_native():+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False)+ assert not select_backend(lm).native+++# ─── the fallback keeps the declared connection ───────────────────────++[email protected]("prefix", ["fireworks", "fireworks_ai"])+def test_fallback_reaches_the_declared_address_with_the_declared_credential(litellm_stub, prefix, monkeypatch):+ # extra_headers is a client setting only LiteLLM carries. The route it+ # takes is LiteLLM's generic OpenAI-compatible door at the DECLARED+ # address, not LiteLLM's own idea of "fireworks_ai".+ register_provider(_definition("https://private-gateway.test/v1", headers=(("X-Gateway", "tenant-1"),)))+ lm = dspy.LM(f"{prefix}/{MODEL}", api_key="private-key", extra_headers={"X-Test": "1"}, cache=False, num_retries=0)+ selection = select_backend(lm)+ assert not selection.native and selection.resolution.declared+ assert lm("hi")[0].endswith("Paris\n\n[[ ## completed ## ]]")+ sent = litellm_stub[-1]+ assert sent["model"] == f"openai/{MODEL}"+ assert sent["api_base"] == "https://private-gateway.test/v1"+ assert sent["api_key"] == "private-key"+ assert sent["extra_headers"] == {"X-Gateway": "tenant-1", "X-Test": "1"}+ assert lm.history[-1]["model"] == f"{prefix}/{MODEL}" # the LM's string, not the wire one+++def test_fallback_reads_the_declared_key_variable_and_never_an_ambient_openai_key(litellm_stub, monkeypatch):+ register_provider(_definition("https://private-gateway.test/v1"))+ monkeypatch.setenv("OPENAI_API_KEY", "sk-openai-must-not-leak")+ lm = dspy.LM(f"fireworks/{MODEL}", extra_headers={"X-Test": "1"}, cache=False, num_retries=0)+ with pytest.raises(dspy.LMError, match="FIREWORKS_API_KEY") as info:+ lm("hi")+ assert not litellm_stub and "sk-openai" not in str(info.value)+ monkeypatch.setenv("FIREWORKS_API_KEY", "env-key")+ assert "Paris" in lm("hi")[0]+ assert litellm_stub[-1]["api_key"] == "env-key"+++def test_fallback_under_engine_litellm_takes_the_same_route(litellm_stub):+ register_provider(_definition("https://private-gateway.test/v1"))+ lm = dspy.LM(f"fireworks_ai/{MODEL}", engine="litellm", api_key="k", cache=False, num_retries=0)+ assert "Paris" in lm("hi")[0]+ assert litellm_stub[-1]["model"] == f"openai/{MODEL}"+ assert litellm_stub[-1]["api_base"] == "https://private-gateway.test/v1"+++async def test_async_fallback_keeps_the_connection_too(litellm_stub):+ register_provider(_definition("https://private-gateway.test/v1"))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", extra_headers={"X": "1"}, cache=False, num_retries=0)+ assert "Paris" in (await lm.acall("hi"))[0]+ assert litellm_stub[-1]["api_base"] == "https://private-gateway.test/v1"+++def test_capabilities_on_the_fallback_route_come_from_the_declaration(litellm_stub):+ # An adapter asks the LM what it supports before every call; on the+ # fallback route that must not be LiteLLM's reading of a prefix it+ # does not know. The whole program runs.+ register_provider(_definition("https://private-gateway.test/v1"),+ supports=ModelSupport(function_calling=True, response_schema=True))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", extra_headers={"X": "1"}, cache=False, num_retries=0)+ assert not select_backend(lm).native+ assert lm.supports_function_calling and lm.supports_response_schema and "tools" in lm.supported_params+ with dspy.context(lm=lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False)):+ prediction = dspy.Predict("question -> answer")(question="capital of France?")+ assert litellm_stub[-1]["api_base"] == "https://private-gateway.test/v1"+ assert prediction.answer == "Paris"+++def test_fallback_sends_the_credential_only_under_the_declared_scheme(litellm_stub):+ # LiteLLM's openai/ door renders the key as a bearer header; its+ # anthropic/ door as x-api-key. A declaration whose scheme the door+ # cannot send gets no fallback — never its secret in an undeclared+ # header (greptile on dspy#10442).+ from dspy.lm15 import AnthropicCompat++ register_provider(_definition("https://private-gateway.test/v1", provider="hdr", aliases=(), auth_scheme=("x-api-key",)))+ lm = dspy.LM(f"hdr/{MODEL}", api_key="secret", extra_headers={"X": "1"}, cache=False, num_retries=0)+ with pytest.raises(dspy.LMUnsupportedFeatureError, match=r"authenticates with 'x-api-key'.*openai/ door cannot send"):+ lm("hi")+ assert not litellm_stub+ register_provider(ProviderDefinition.anthropic(+ AccessPolicy(provider="claude-gw", auth_modes=("x-api-key",), auth_scheme=("x-api-key",),+ env_keys=("CLAUDE_GW_KEY",), base_url="https://claude-gateway.test"),+ compat=AnthropicCompat()))+ lm = dspy.LM("claude-gw/claude-x", api_key="secret", extra_headers={"X": "1"}, cache=False, num_retries=0)+ assert "Paris" in lm("hi")[0]+ assert litellm_stub[-1]["model"] == "anthropic/claude-x"+ assert litellm_stub[-1]["api_base"] == "https://claude-gateway.test"+ assert litellm_stub[-1]["api_key"] == "secret"+ register_provider(ProviderDefinition.anthropic(+ AccessPolicy(provider="claude-bearer", auth_modes=("bearer",), auth_scheme=("bearer",),+ env_keys=("CLAUDE_GW_KEY",), base_url="https://claude-gateway.test"),+ compat=AnthropicCompat()))+ lm = dspy.LM("claude-bearer/claude-x", api_key="secret", extra_headers={"X": "1"}, cache=False, num_retries=0)+ with pytest.raises(dspy.LMUnsupportedFeatureError, match=r"authenticates with 'bearer'.*anthropic/ door cannot send"):+ lm("hi")+++def test_fallback_resolves_a_callable_credential_per_call(litellm_stub):+ # lm15 credentials may be zero-argument callables (rotating tokens). The+ # fallback invokes them like the native path, on each call, and hands+ # LiteLLM the string (greptile on dspy#10442).+ register_provider(_definition("https://private-gateway.test/v1"))+ tokens = iter(["token-1", "token-2"])+ lm = dspy.LM(f"fireworks/{MODEL}", api_key=lambda: next(tokens), extra_headers={"X": "1"}, cache=False, num_retries=0)+ lm("hi")+ lm("hi again")+ assert [call["api_key"] for call in litellm_stub[-2:]] == ["token-1", "token-2"]+++async def test_fallback_is_still_the_declared_provider_for_pricing(litellm_stub):+ # The fallback reaches the same server; pricing comes from the+ # declaration's namespaces on every path, never from LiteLLM's reading+ # of the generic door's model string (greptile on dspy#10442).+ register_provider(_definition("https://private-gateway.test/v1"), metadata_namespaces=("fireworks_ai",))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", extra_headers={"X": "1"}, cache=False, num_retries=0)+ lm("hi") # legacy body path+ lm(dspy.lm15.Request(model=lm.model, messages=(dspy.lm15.Message.user("hi"),))) # canonical path+ await lm.acall("hi") # async legacy path+ costs = [entry["cost"] for entry in lm.history[-3:]]+ assert all(cost is not None and cost > 0 for cost in costs) and len(set(costs)) == 1+ for entry in lm.history[-3:]:+ assert entry["cost_details"]["provider"] == "fireworks"+ assert entry["cost_details"]["metadata"]["namespaces"] == ["fireworks_ai"]+ # A colliding name on a gateway with no namespace is unknown, not OpenAI's price.+ register_provider(_definition("https://private-gateway.test/v1", provider="gw", aliases=()))+ plain = dspy.LM("gw/gpt-4o", api_key="k", extra_headers={"X": "1"}, cache=False, num_retries=0)+ plain("hi")+ assert plain.history[-1]["cost"] is None+ assert "names no metadata namespace" in plain.history[-1]["cost_details"]["reason"]+++def test_a_declared_refusal_is_final_under_auto(litellm_stub, server):+ # thinking_format="none" says the wire has no reasoning field. A+ # reasoning dial on it is a refusal (lm15 MAP-7); sending it through+ # LiteLLM anyway would drop the dial silently, so engine="auto" refuses+ # too. (A setting lm15 adapts and records, e.g. json_schema="reject",+ # is not a refusal and never leaves the native route.)+ base, seen = server+ register_provider(_definition(base, thinking_format="none", json_schema="reject"))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False, num_retries=0)+ with pytest.raises(dspy.LMUnsupportedFeatureError):+ lm("hi", reasoning_effort="low")+ assert not litellm_stub+ assert lm("hi", response_format={"type": "json_schema", "json_schema": {"name": "x", "schema": {"type": "object"}}}) == ["Paris"]+ assert "response_format" not in seen[-1]["body"] # adapted and recorded, natively+ lm.close()+++def test_engine_lm15_refuses_client_settings_it_cannot_carry(server):+ base, _ = server+ register_provider(_definition(base))+ with pytest.raises(Exception, match="require the LiteLLM"):+ select_backend(dspy.LM(f"fireworks/{MODEL}", engine="lm15", api_key="k", extra_headers={"X": "1"}))+++# ─── aliases grant a spelling, nothing else ───────────────────────────+++def test_an_alias_grants_no_metadata(server):+ base, _ = server+ register_provider(_definition(base)) # alias fireworks-ai, no namespaces+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False, num_retries=0)+ assert not (lm.supports_function_calling or lm.supports_reasoning or lm.supports_response_schema)+ lm("hi")+ assert lm.history[-1]["cost"] is None+ assert "names no metadata namespace" in lm.history[-1]["cost_details"]["reason"]+ lm.close()+++def test_metadata_namespaces_are_an_explicit_opt_in(server):+ base, _ = server+ register_provider(_definition(base), metadata_namespaces=("fireworks_ai",))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False, num_retries=0)+ assert lm.supports_function_calling and lm.supports_reasoning and lm.supports_response_schema+ assert {"reasoning_effort", "max_tokens", "response_format", "tools"} <= lm.supported_params+ lm("hi")+ entry = lm.history[-1]+ assert entry["cost"] is not None and entry["cost_details"]["kind"] == "estimate"+ assert entry["cost_details"]["provider"] == "fireworks"+ assert entry["cost_details"]["metadata"]["namespaces"] == ["fireworks_ai"]+ lm.close()+++def test_stated_support_fills_what_no_snapshot_says(server):+ base, _ = server+ register_provider(+ _definition(base), supports=ModelSupport(function_calling=True),+ models={"private-v1": ModelSupport(response_schema=True, function_calling=False)},+ )+ default = dspy.LM("fireworks/private-v2", api_key="k")+ assert default.supports_function_calling and not default.supports_response_schema and not default.supports_reasoning+ assert "tools" in default.supported_params+ specific = dspy.LM("fireworks/private-v1", api_key="k")+ assert not specific.supports_function_calling and specific.supports_response_schema+ assert "response_format" in specific.supported_params and "tools" not in specific.supported_params+ # A statement never prices anything.+ specific = dspy.LM("fireworks/private-v1", api_key="k", cache=False, num_retries=0)+ specific("hi")+ assert specific.history[-1]["cost"] is None+ specific.close()+++def test_a_snapshot_entry_is_more_specific_than_a_statement(server):+ base, _ = server+ # The snapshot says this Fireworks model supports tools; the statement+ # says the provider does not. The per-model fact wins.+ register_provider(_definition(base), metadata_namespaces=("fireworks_ai",),+ supports=ModelSupport(function_calling=False))+ assert dspy.LM(f"fireworks/{MODEL}", api_key="k").supports_function_calling+ assert not dspy.LM("fireworks/private-v9", api_key="k").supports_function_calling+++def test_compat_object_shapes_capabilities(server):+ base, _ = server+ register_provider(_definition(base, json_schema="reject", thinking_format="none"),+ supports=ModelSupport(function_calling=True, reasoning=True, response_schema=True))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k")+ assert lm.supports_function_calling+ assert not lm.supports_response_schema and not lm.supports_reasoning # the wire cannot carry them+ assert "reasoning_effort" not in lm.supported_params+++# ─── state ────────────────────────────────────────────────────────────+++def test_state_is_the_model_string(server):+ base, _ = server+ register_provider(_definition(base))+ state = dspy.LM(f"fireworks/{MODEL}", api_key="k").dump_state()+ json.dumps(state)+ assert "engine" not in state and "api_key" not in state and "_providers" not in state+ loaded = dspy.LM.load_state(state)+ assert loaded.model == f"fireworks/{MODEL}" and select_backend(loaded).native # bound at load time+++def test_pickled_lm_keeps_its_binding(server):+ import pickle++ base, _ = server+ register_provider(_definition(base))+ lm = dspy.LM(f"fireworks/{MODEL}", api_key="k", cache=False)+ unregister_provider("fireworks")+ restored = pickle.loads(pickle.dumps(lm))+ assert select_backend(restored).native and restored._providers == lm._providersdiff --git a/tests/test_vendored_lm15.py b/tests/test_vendored_lm15.pyindex f226fb0f1..95eb7ad44 100644--- a/tests/test_vendored_lm15.py+++ b/tests/test_vendored_lm15.py@@ -17,9 +17,19 @@ def test_public_exports_are_the_vendored_objects():assert importlib.import_module("dspy.lm15") is dspy.lm15assert Request is vendored.Request- assert lm15.__all__ == vendored.__all__- for name in lm15.__all__:+ # Every vendored export is the vendored object; the facade adds only the+ # process-level provider registry and the compat/definition types it takes.+ assert lm15.__all__[: len(vendored.__all__)] == vendored.__all__+ for name in vendored.__all__:assert getattr(lm15, name) is getattr(vendored, name)+ assert set(lm15.__all__) - set(vendored.__all__) == {+ "AnthropicCompat", "ModelSupport", "OpenAIChatCompat", "OpenAIResponsesCompat", "ProviderDefinition",+ "RegisteredProvider", "register_provider", "registered_providers", "unregister_provider",+ }+ from dspy._vendor.lm15 import compat, registry++ assert lm15.OpenAIChatCompat is compat.OpenAIChatCompat+ assert lm15.ProviderDefinition is registry.ProviderDefinitionrequest = Request(model="example", messages=(lm15.Message.user("Hello"),))restored = pickle.loads(pickle.dumps(request))assert type(restored) is Request