Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Custom engines own their connection: refuse client settings at every door, save through engine state (#10441)

Maxime Rivest7b0d899f331c
diff --git a/docs/docs/community/normalized-lm-api-migration.md b/docs/docs/community/normalized-lm-api-migration.md
index 9689fa78a..e76e9e9ec 100644
--- a/docs/docs/community/normalized-lm-api-migration.md
+++ b/docs/docs/community/normalized-lm-api-migration.md
@@ -273,6 +273,14 @@ an async iterator for async calls. DSPy does not silently run a sync backend on
async path. Engines must not add another DSPy cache or retry loop. Custom engines
are caller-owned and are not closed by DSPy.
+A custom engine owns its connection: `api_key`, `api_base`, `timeout`,
+`extra_headers` and the other client settings are refused on `dspy.LM(engine=...)`,
+on `copy()` and on every call, rather than dropped. `copy(engine=...)` replaces the
+engine pair, and each side must be its kind (a coroutine function on the async side).
+An engine is saved by `dump_state()`/`load_state()` of its own, and loading one is
+gated by `allow_unsafe_lm_state=True` like a custom LM class. See the
+[custom-engine tutorial](../tutorials/custom_lm_engines/index.md).
+
Implementing custom LMs through `BaseLM.forward(prompt=None, messages=None, **kwargs)`
or `aforward` is **deprecated**. Implement an engine instead, as shown above and in
the [custom-engine tutorial](../tutorials/custom_lm_engines/index.md). The old subclass
diff --git a/docs/docs/tutorials/custom_lm_engines/index.md b/docs/docs/tutorials/custom_lm_engines/index.md
index 1c192c2f3..975a5cf61 100644
--- a/docs/docs/tutorials/custom_lm_engines/index.md
+++ b/docs/docs/tutorials/custom_lm_engines/index.md
@@ -1,6 +1,8 @@
# Custom LM Engines
-In the DSPy 3.4 development API, you can supply your own execution engine to `dspy.LM`. The minimum synchronous interface is:
+You can supply your own execution engine to `dspy.LM`. An engine is for a backend that is not an HTTP provider DSPy's bundled lm15 can speak to: a CLI, an in-process model, an agent harness. It owns its whole connection, so the [rules below](#what-a-custom-engine-owns) apply.
+
+The minimum synchronous engine interface is:
```python
class MyEngine:
@@ -19,6 +21,33 @@ family and owns retries. Unexpected exceptions retain their original cause and
are not guessed to be retryable from their message text. See
[errors and retry ownership](../../community/normalized-lm-api-migration.md#errors-and-retry-ownership).
+## What a custom engine owns
+
+A custom engine is borrowed by `dspy.LM` and owns its connection. Three rules follow, and DSPy enforces each rather than guessing:
+
+**Connection settings are refused.** `dspy.LM(engine=MyEngine(), api_key=...)` raises `ValueError`, and so do `api_base`, `base_url`, `timeout`, `headers`, `extra_headers` and the other client settings — on construction, on `copy()`, and on every call (`lm("hi", api_key=...)`), before any cache lookup. There is no channel from the LM to the engine for them, and dropping them silently would let a call run with the engine's key while the LM said another. Give them to the engine's constructor.
+
+**The engine pair is one unit.** `async_engine=` is only accepted with a custom `engine=`, and each side must be its kind: the sync engine's `complete` is a plain function returning a `Response`, the async engine's is a coroutine function (`async def complete`). A sync method on the async side, or the reverse, is refused at construction rather than on the first call. `lm.copy(engine=...)` replaces both: the copy has no async engine unless you pass a new `async_engine=` in the same call. A copy that does not mention `engine` keeps the pair.
+
+**Saving needs the engine's own state.** An engine is saved when it implements `dump_state() -> dict` (JSON-serializable, no secrets) and the classmethod `load_state(state) -> engine`:
+
+```python
+class MyEngine:
+ def __init__(self, model="gpt-6-astra"):
+ self.model = model
+
+ def complete(self, request): ...
+
+ def dump_state(self):
+ return {"model": self.model}
+
+ @classmethod
+ def load_state(cls, state):
+ return cls(**state)
+```
+
+`lm.dump_state()` then records `{"engine": {"class": "your.module:MyEngine", "state": {...}}}` (and `async_engine` likewise). Loading imports that class from the file, which is the same trust decision as a custom LM class, so it is gated the same way: `program.load(path, allow_unsafe_lm_state=True)` or `dspy.LM.load_state(state, allow_custom_lm_class=True)`. The class must be importable by that path in the process that loads the state. Define it at module level, not inside a function — `dump_state()` refuses a class it cannot import back — and for state that must outlive the session, in an importable module: a class defined in a script or notebook is recorded as `__main__:MyEngine`, which loads only in a process whose `__main__` defines it again. `dump_state()` also refuses engine state JSON cannot carry. An engine without the two methods cannot be saved either; `dump_state()` says so and the way out is to save the program without that LM and set it again after loading. Keep secrets out of engine state, as DSPy keeps `api_key` out of LM state.
+
This tutorial wraps the [Pi CLI](https://pi.dev) as a custom engine. Pi keeps its normal system prompt and tools, so a DSPy program can ask it to inspect a repository. The entire agent run, including tool calls, becomes one DSPy LM response.
## Prerequisites
@@ -78,7 +107,7 @@ program(
)
```
-Here `"pi"` is a DSPy model label for history; the engine's CLI arguments select the actual provider and model.
+Here `"pi"` is a DSPy model label for history; the engine's CLI arguments select the actual provider and model. `PiEngine` holds no state, so to make it saveable add `dump_state` returning `{}` and `load_state` returning `cls()`.
An answer for the DSPy repository can look like:
diff --git a/dspy/clients/execution.py b/dspy/clients/execution.py
index 64e39c8a1..7962c1002 100644
--- a/dspy/clients/execution.py
+++ b/dspy/clients/execution.py
@@ -136,6 +136,11 @@ def prepare(lm, prompt, messages, kwargs, *, asynchronous=False, direct=False):
legacy = {"model": lm.model}
use_cache = kwargs.get("cache", lm.cache) if managed and getattr(lm, "_cache_responses", True) else False
return PreparedCall(None, None, kwargs, legacy, request, use_cache, 1, managed)
+ if managed and not isinstance(getattr(lm, "_engine_spec", "auto"), str):
+ # A custom engine owns its connection; a call cannot hand it one either.
+ from dspy.clients.lm import _refuse_client_settings
+
+ _refuse_client_settings(lm._engine_spec, kwargs, where="LM call")
merged = {**lm.kwargs, **{key: val for key, val in kwargs.items() if key != "cache"}}
prompt_cache = merged.get("prompt_cache")
if prompt_cache is not None:
diff --git a/dspy/clients/lm.py b/dspy/clients/lm.py
index a4ccde3a8..0ddf349f0 100644
--- a/dspy/clients/lm.py
+++ b/dspy/clients/lm.py
@@ -44,6 +44,150 @@ def _is_openai_reasoning_model(model: str) -> bool:
) is not None
+_ENGINE_SELECTIONS = ("auto", "lm15", "litellm")
+# Serialized custom engines: {"class": "pkg.module:Qual.Name", "state": {...}}.
+_ENGINE_CLASS_KEY = "class"
+_ENGINE_STATE_KEY = "state"
+
+
+def _is_coroutine_function(fn) -> bool:
+ import inspect
+
+ if inspect.iscoroutinefunction(fn):
+ return True
+ # A callable object whose __call__ (own or inherited) is async: resolve
+ # it through the MRO as Python does. A plain function's type resolves
+ # __call__ to a slot wrapper, which is not a coroutine function.
+ call = next((vars(cls)["__call__"] for cls in type(fn).__mro__ if "__call__" in vars(cls)), None)
+ return inspect.iscoroutinefunction(call)
+
+
+def _check_engines(engine, async_engine) -> None:
+ """One rule for the constructor and copy(): a selection string with no
+ async counterpart, or a custom engine object with complete(Request) — a
+ plain function on the sync engine, a coroutine function on the async one
+ (dspy.clients.engines.base). Checked here, at construction, rather than
+ on the first call, where the wrong kind fails as an unexpected error."""
+ if isinstance(engine, str):
+ if engine not in _ENGINE_SELECTIONS:
+ raise ValueError("engine must be 'auto', 'lm15', 'litellm', or an engine object")
+ if async_engine is not None:
+ raise ValueError("async_engine is only used with a custom engine object")
+ return
+ complete = getattr(engine, "complete", None)
+ if not callable(complete):
+ raise TypeError("A custom engine must implement complete(Request) -> Response")
+ if _is_coroutine_function(complete):
+ raise TypeError(
+ "engine.complete is a coroutine function; the sync engine returns a Response directly. "
+ "Pass an async engine as async_engine="
+ )
+ if async_engine is None:
+ return
+ acomplete = getattr(async_engine, "complete", None)
+ if not callable(acomplete):
+ raise TypeError("A custom async engine must implement async complete(Request) -> Response")
+ if not _is_coroutine_function(acomplete):
+ raise TypeError(
+ "async_engine.complete is not a coroutine function; the async engine is awaited "
+ "(async def complete). Pass a sync engine as engine="
+ )
+
+
+def _refuse_client_settings(engine, kwargs, *, where: str = "dspy.LM") -> None:
+ """A custom engine owns its connection: api_key, api_base, timeout and the
+ other client settings have no way to reach it, so they are refused rather
+ than dropped (dspy#10409 follow-up: the Fireworks custom engine ran with
+ the engine's key while the LM said another). Applied at construction, on
+ copy(), and to every call's own keyword arguments — the same rule at
+ each door, before any cache lookup."""
+ if isinstance(engine, str):
+ return
+ from dspy.clients.backend_selection import CLIENT_KEYS
+
+ present = sorted(key for key in kwargs if key in CLIENT_KEYS and kwargs[key] is not None)
+ if present:
+ raise ValueError(
+ f"{where}: {present} configure a connection DSPy owns; a custom engine owns its own. "
+ "Configure them on the engine instead."
+ )
+
+
+def _engine_class_path(engine) -> str:
+ cls = type(engine)
+ return f"{cls.__module__}:{cls.__qualname__}"
+
+
+def _dump_engine(engine, name: str) -> dict[str, Any]:
+ dump = getattr(engine, "dump_state", None)
+ load = getattr(type(engine), "load_state", None)
+ if not callable(dump) or not callable(load):
+ raise TypeError(
+ f"{name}={type(engine).__name__} cannot be saved: a custom engine is saved when it implements "
+ "dump_state() -> dict (JSON-serializable, no secrets) and the classmethod load_state(state) "
+ "-> engine. Otherwise save the program without this LM and set it again after loading."
+ )
+ path = _engine_class_path(engine)
+ # The path must lead back to this class when loading (pickle's rule): a
+ # class defined inside a function or built dynamically would save fine
+ # and never load.
+ try:
+ found = _import_engine_class(path)
+ except (ImportError, TypeError, ValueError):
+ found = None
+ if found is not type(engine):
+ raise TypeError(
+ f"{name}={type(engine).__name__} cannot be saved: its class is not importable as `{path}`. "
+ "Define the engine class at module level so a loader can import it."
+ )
+ state = dump()
+ if not isinstance(state, dict):
+ raise TypeError(f"{name}.dump_state() must return a dict, got {type(state).__name__}")
+ import json
+
+ try:
+ json.dumps(state)
+ except (TypeError, ValueError) as exc:
+ raise TypeError(f"{name}.dump_state() must return JSON-serializable data: {exc}") from exc
+ return {_ENGINE_CLASS_KEY: path, _ENGINE_STATE_KEY: state}
+
+
+def _import_engine_class(class_path: str) -> type:
+ import importlib
+
+ module_name, _, qualname = class_path.partition(":")
+ if not module_name or not qualname:
+ raise ValueError(f"Serialized engine class path must be 'module:QualName', got {class_path!r}")
+ obj = importlib.import_module(module_name)
+ for part in qualname.split("."):
+ try:
+ obj = getattr(obj, part)
+ except AttributeError as exc:
+ raise ImportError(f"Serialized engine class `{class_path}` cannot be imported: {exc}") from exc
+ if not isinstance(obj, type):
+ raise TypeError(f"Serialized engine `{class_path}` is not a class")
+ return obj
+
+
+def _load_engine(record, name: str, *, allow_custom_lm_class: bool):
+ if not isinstance(record, dict) or _ENGINE_CLASS_KEY not in record:
+ raise ValueError(f"Serialized {name} must be a selection string or a {{'class', 'state'}} record")
+ class_path = record[_ENGINE_CLASS_KEY]
+ if not allow_custom_lm_class:
+ raise ValueError(
+ f"Refusing to import custom serialized engine class `{class_path}`. "
+ "Pass allow_unsafe_lm_state=True when loading trusted files to enable custom engines."
+ )
+ cls = _import_engine_class(class_path)
+ load = getattr(cls, "load_state", None)
+ if not callable(load):
+ raise TypeError(f"Serialized engine `{class_path}` has no load_state(state) classmethod")
+ engine = load(record.get(_ENGINE_STATE_KEY, {}))
+ if not callable(getattr(engine, "complete", None)):
+ raise TypeError(f"`{class_path}.load_state` must return an engine with complete(Request)")
+ return engine
+
+
class LM(BaseLM):
"""
A language model supporting chat or text completion requests for use with DSPy modules.
@@ -92,8 +236,11 @@ class LM(BaseLM):
backoff.
engine: 'auto' prefers lm15 for representable requests; 'litellm' preserves the compatibility backend;
'lm15' refuses unsupported mappings rather than selecting LiteLLM. A custom engine implements
- complete(Request) -> Response and optionally stream(Request). Engines are borrowed.
- async_engine: Async counterpart when supplying a custom engine object.
+ complete(Request) -> Response and optionally stream(Request). Engines are borrowed and own
+ their connection: api_key, api_base, timeout and the other client settings are refused with a
+ custom engine, at construction and on every call.
+ async_engine: Async counterpart when supplying a custom engine object. The pair is one unit:
+ copy(engine=...) replaces both unless async_engine= is given too.
prompt_cache: Optional lm15 CacheConfig for provider-side prompt caching on ordinary calls.
Separate from DSPy's response cache. Requires native lm15 or a canonical custom engine;
may incur cache-write/storage charges. A call-time value overrides this default, and None
@@ -107,13 +254,8 @@ class LM(BaseLM):
only affects generation when `temperature` is non-zero. This argument is
stripped before sending requests to the provider.
"""
- if isinstance(engine, str):
- if engine not in {"auto", "lm15", "litellm"}:
- raise ValueError("engine must be 'auto', 'lm15', 'litellm', or an engine object")
- if async_engine is not None:
- raise ValueError("async_engine is only used with a custom engine object")
- elif not callable(getattr(engine, "complete", None)):
- raise TypeError("A custom engine must implement complete(Request) -> Response")
+ _check_engines(engine, async_engine)
+ _refuse_client_settings(engine, kwargs)
if isinstance(num_retries, bool) or not isinstance(num_retries, int) or num_retries < 0:
raise ValueError("num_retries must be a nonnegative integer")
if prompt_cache is not None:
@@ -283,12 +425,16 @@ class LM(BaseLM):
def copy(self, **kwargs):
if kwargs.get("prompt_cache") is not None and not isinstance(kwargs["prompt_cache"], CacheConfig):
raise TypeError("prompt_cache must be a dspy.lm15.CacheConfig or None")
- spec = kwargs.pop("engine", self._engine_spec)
- async_spec = kwargs.pop("async_engine", self._async_engine_spec)
- if isinstance(spec, str) and spec not in {"auto", "lm15", "litellm"}:
- raise ValueError("Unknown engine selection")
- if not isinstance(spec, str) and not callable(getattr(spec, "complete", None)):
- raise TypeError("Custom engines must implement complete(Request)")
+ if "engine" in kwargs:
+ # The engine pair is one unit: a new engine drops the old async
+ # counterpart unless a new one comes with it.
+ spec = kwargs.pop("engine")
+ async_spec = kwargs.pop("async_engine", None)
+ else:
+ spec = self._engine_spec
+ async_spec = kwargs.pop("async_engine", self._async_engine_spec)
+ _check_engines(spec, async_spec)
+ _refuse_client_settings(spec, {**self.kwargs, **kwargs}, where="LM.copy")
copied = super().copy(**kwargs)
copied._engine_spec = spec
copied._async_engine_spec = async_spec
@@ -389,19 +535,31 @@ class LM(BaseLM):
def dump_state(self):
"""Return a sanitized reconstruction state for this LM.
+ A custom engine is recorded as its class path and its own
+ ``dump_state()``; the class must be importable by that path in the
+ process that loads the state. A class defined in ``__main__`` (a
+ script or notebook) loads only where ``__main__`` defines it again;
+ for durable state define the engine in an importable module.
+
Returns:
A dictionary that can be passed to `BaseLM.load_state` to
reconstruct this `LM`. The state excludes API keys.
"""
- if not isinstance(self._engine_spec, str):
- raise TypeError("Custom engine objects require custom dump_state/load_state methods; they cannot be stored in JSON LM state.")
state = super().dump_state()
if state.get("prompt_cache") is not None:
from dspy._vendor.lm15.serde import cache_config_to_dict
state["prompt_cache"] = cache_config_to_dict(state["prompt_cache"])
- if self._engine_spec != "auto":
- state["engine"] = self._engine_spec
+ if isinstance(self._engine_spec, str):
+ if self._engine_spec != "auto":
+ state["engine"] = self._engine_spec
+ else:
+ # A custom engine is saved as its class path and its own state;
+ # loading it imports a class from the file, so it is gated like a
+ # custom LM class (allow_unsafe_lm_state).
+ state["engine"] = _dump_engine(self._engine_spec, "engine")
+ if self._async_engine_spec is not None:
+ state["async_engine"] = _dump_engine(self._async_engine_spec, "async_engine")
state.update(
{
"finetuning_model": self.finetuning_model,
@@ -429,6 +587,15 @@ class LM(BaseLM):
state["max_tokens"] = state["max_completion_tokens"]
state.pop("max_completion_tokens")
+ if not isinstance(state.get("engine", "auto"), str):
+ state["engine"] = _load_engine(state["engine"], "engine", allow_custom_lm_class=allow_custom_lm_class)
+ if state.get("async_engine") is not None:
+ state["async_engine"] = _load_engine(
+ state["async_engine"], "async_engine", allow_custom_lm_class=allow_custom_lm_class
+ )
+ elif state.get("async_engine") is not None:
+ raise ValueError("Serialized async_engine without a custom engine")
+
return super().load_state(state, allow_custom_lm_class=allow_custom_lm_class)
def _check_truncation(self, results):
diff --git a/tests/clients/test_custom_engine_contract.py b/tests/clients/test_custom_engine_contract.py
new file mode 100644
index 000000000..1e18cdfec
--- /dev/null
+++ b/tests/clients/test_custom_engine_contract.py
@@ -0,0 +1,293 @@
+"""The custom-engine contract: a borrowed engine owns its connection, the
+engine pair is one unit, and a custom engine is saved through its own state.
+
+Pinned from cmpnd-ai/breaka-your-lm results/CUSTOM_ENGINE_FINDINGS.md
+(2026-09-16): api_key/api_base/timeout/extra_headers were dropped before a
+custom engine saw the request, the constructor and copy() applied different
+rules to async_engine, and dump_state refused custom engines with no
+documented way out.
+"""
+
+import json
+import os
+import re
+import subprocess
+import sys
+
+import pytest
+
+import dspy
+from dspy.lm15 import Message, Response, Usage
+
+
+class Echo:
+ """A stateful engine that knows how to save and restore itself."""
+
+ def __init__(self, tag="t"):
+ self.tag = tag
+
+ def complete(self, request):
+ return Response(id=None, model=request.model, message=Message.assistant(f"echo:{self.tag}"),
+ finish_reason="stop", usage=Usage())
+
+ def dump_state(self):
+ return {"tag": self.tag}
+
+ @classmethod
+ def load_state(cls, state):
+ return cls(**state)
+
+
+class AsyncEcho(Echo):
+ async def complete(self, request):
+ return Echo.complete(self, request)
+
+
+class BadState(Echo):
+ """An engine whose state is not a dict."""
+
+ def dump_state(self):
+ return ["tag"]
+
+
+class NonJsonState(Echo):
+ """An engine whose state is a dict that JSON cannot carry."""
+
+ def dump_state(self):
+ return {"tag": {1, 2}}
+
+
+class Stateless:
+ """An engine with no save/load protocol."""
+
+ def complete(self, request):
+ return Echo().complete(request)
+
+
+# ─── connection settings belong to the engine ─────────────────────────
+
+
[email protected]("setting", [
+ {"api_key": "k"}, {"api_base": "http://127.0.0.1:9/v1"}, {"base_url": "http://127.0.0.1:9/v1"},
+ {"timeout": 1}, {"extra_headers": {"X-Test": "1"}}, {"headers": {"X-Test": "1"}}, {"api_version": "v1"},
+ {"organization": "org"}, {"project": "p"}, {"extra_query": {"a": "b"}}, {"custom_llm_provider": "x"},
+])
+def test_client_settings_are_refused_with_a_custom_engine(setting):
+ with pytest.raises(ValueError, match=re.escape(f"dspy.LM: {sorted(setting)}") + ".*custom engine owns its own"):
+ dspy.LM("custom/echo", engine=Echo(), **setting)
+
+
[email protected]("setting", [{"api_key": "k"}, {"api_base": "http://127.0.0.1:9/v1"}, {"timeout": 0.001},
+ {"extra_headers": {"X-Test": "1"}}])
+async def test_client_settings_are_refused_on_every_call(setting):
+ # The constructor and copy() refused them; a call passed them in and the
+ # engine never saw them (greptile on dspy#10441 found the third door open).
+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=AsyncEcho(), cache=True)
+ refused = re.escape(f"LM call: {sorted(setting)}")
+ with pytest.raises(ValueError, match=refused):
+ lm("hi", **setting)
+ with pytest.raises(ValueError, match=refused):
+ await lm.acall("hi", **setting)
+ with pytest.raises(ValueError, match=refused):
+ lm.forward("hi", **setting)
+ with pytest.raises(ValueError, match=refused):
+ await lm.aforward("hi", **setting)
+ assert not lm.history # refused before any attempt or cache entry
+ assert lm("hi") == ["echo:t"]
+
+
+def test_client_settings_are_refused_on_copy_too():
+ lm = dspy.LM("custom/echo", engine=Echo(), cache=False)
+ with pytest.raises(ValueError, match=r"LM.copy: \['api_key'\]"):
+ lm.copy(api_key="k")
+ # And when a copy switches to a custom engine while the LM carries them.
+ native = dspy.LM("openai/gpt-4o", api_key="k", cache=False)
+ with pytest.raises(ValueError, match="api_key"):
+ native.copy(engine=Echo())
+ # Clearing them in the same copy is the way through.
+ switched = native.copy(engine=Echo(), api_key=None)
+ assert "api_key" not in switched.kwargs and isinstance(switched.engine, Echo)
+
+
+def test_none_is_not_a_setting():
+ lm = dspy.LM("custom/echo", engine=Echo(), api_key=None, timeout=None, cache=False)
+ assert lm("hi") == ["echo:t"]
+
+
+def test_built_in_engines_keep_taking_client_settings():
+ lm = dspy.LM("openai/gpt-4o", engine="lm15", api_key="k", api_base="http://127.0.0.1:9/v1", timeout=3)
+ assert lm.kwargs["api_key"] == "k"
+ assert lm.copy(api_key="k2").kwargs["api_key"] == "k2"
+
+
+# ─── the engine pair is one unit ──────────────────────────────────────
+
+
+def test_constructor_and_copy_apply_the_same_async_rule():
+ with pytest.raises(ValueError, match="async_engine is only used with a custom engine"):
+ dspy.LM("custom/echo", engine="auto", async_engine=AsyncEcho())
+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=AsyncEcho(), cache=False)
+ with pytest.raises(ValueError, match="async_engine is only used with a custom engine"):
+ lm.copy(engine="litellm", async_engine=AsyncEcho())
+
+
+def test_copy_with_a_new_engine_replaces_the_pair():
+ lm = dspy.LM("custom/echo", engine=Echo("a"), async_engine=AsyncEcho("a"), cache=False)
+ to_litellm = lm.copy(engine="litellm")
+ assert to_litellm.engine == "litellm" and to_litellm._async_engine_spec is None
+ other = lm.copy(engine=Echo("b"))
+ assert other.engine.tag == "b" and other._async_engine_spec is None
+ both = lm.copy(engine=Echo("c"), async_engine=AsyncEcho("c"))
+ assert both.engine.tag == "c" and both._async_engine_spec.tag == "c"
+ # A copy that does not mention the engine keeps the whole pair.
+ same = lm.copy(temperature=0.5)
+ assert same.engine is lm.engine and same._async_engine_spec is lm._async_engine_spec
+
+
+async def test_async_counterpart_must_be_an_engine():
+ with pytest.raises(TypeError, match="async complete"):
+ dspy.LM("custom/echo", engine=Echo(), async_engine=object())
+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=AsyncEcho("z"), cache=False)
+ assert await lm.acall("hi") == ["echo:z"]
+
+
+def test_each_side_of_the_pair_must_be_the_right_kind():
+ # A sync complete() as the async engine would be awaited as a Response
+ # on the first acall; an async complete() as the sync engine would hand
+ # execute() a coroutine. Both are refused at construction (greptile on
+ # dspy#10441).
+ with pytest.raises(TypeError, match=r"async_engine\.complete is not a coroutine function"):
+ dspy.LM("custom/echo", engine=Echo(), async_engine=Echo())
+ with pytest.raises(TypeError, match=r"engine\.complete is a coroutine function"):
+ dspy.LM("custom/echo", engine=AsyncEcho())
+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=AsyncEcho(), cache=False)
+ with pytest.raises(TypeError, match="not a coroutine function"):
+ lm.copy(async_engine=Echo())
+ with pytest.raises(TypeError, match="is a coroutine function"):
+ lm.copy(engine=AsyncEcho())
+
+
+class AsyncCallable:
+ async def __call__(self, request):
+ return Echo("inherited").complete(request)
+
+
+class InheritsAsyncCall(AsyncCallable):
+ pass
+
+
+class WithCallableComplete:
+ complete = InheritsAsyncCall()
+
+
+async def test_an_inherited_async_call_counts_as_a_coroutine_function():
+ # Python resolves __call__ through the MRO; so does the kind check.
+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=WithCallableComplete(), cache=False)
+ assert await lm.acall("hi") == ["echo:inherited"]
+
+
+# ─── saving and loading ───────────────────────────────────────────────
+
+
+def test_custom_engine_round_trips_through_json_state():
+ lm = dspy.LM("custom/echo", engine=Echo("a"), async_engine=AsyncEcho("b"), cache=False, num_retries=0)
+ state = lm.dump_state()
+ json.dumps(state) # JSON-serializable
+ assert state["engine"] == {"class": f"{__name__}:Echo", "state": {"tag": "a"}}
+ assert state["async_engine"] == {"class": f"{__name__}:AsyncEcho", "state": {"tag": "b"}}
+ loaded = dspy.LM.load_state(state, allow_custom_lm_class=True)
+ assert isinstance(loaded.engine, Echo) and loaded.engine.tag == "a"
+ assert isinstance(loaded._async_engine_spec, AsyncEcho) and loaded._async_engine_spec.tag == "b"
+ assert loaded.num_retries == 0 and loaded("hi") == ["echo:a"]
+
+
+def test_loading_a_custom_engine_is_gated_like_a_custom_lm_class():
+ state = dspy.LM("custom/echo", engine=Echo()).dump_state()
+ with pytest.raises(ValueError, match="Refusing to import custom serialized engine class"):
+ dspy.LM.load_state(state)
+ with pytest.raises(ValueError, match="Refusing to import custom serialized engine class"):
+ dspy.BaseLM.load_state(state)
+
+
+def test_program_save_and_load_with_a_custom_engine(tmp_path):
+ program = dspy.Predict("question -> answer")
+ program.lm = dspy.LM("custom/echo", engine=Echo("saved"), cache=False)
+ path = tmp_path / "program.json"
+ program.save(path)
+ fresh = dspy.Predict("question -> answer")
+ with pytest.raises(ValueError, match="allow_unsafe_lm_state"):
+ fresh.load(path)
+ fresh.load(path, allow_unsafe_lm_state=True)
+ assert isinstance(fresh.lm.engine, Echo) and fresh.lm.engine.tag == "saved"
+
+
+def test_engine_without_state_protocol_is_refused_with_the_way_out():
+ lm = dspy.LM("custom/echo", engine=Stateless())
+ with pytest.raises(TypeError, match=r"dump_state\(\) -> dict.*load_state\(state\)") as info:
+ lm.dump_state()
+ assert "save the program without this LM" in str(info.value)
+
+
+def test_function_local_engine_class_is_refused_at_save_time():
+ class Local(Echo):
+ pass
+
+ lm = dspy.LM("custom/echo", engine=Local())
+ with pytest.raises(TypeError, match=r"not importable as .*<locals>.*module level"):
+ lm.dump_state()
+
+
+def test_engine_state_must_be_a_dict():
+ with pytest.raises(TypeError, match="must return a dict"):
+ dspy.LM("custom/echo", engine=BadState()).dump_state()
+
+
+def test_engine_state_must_be_json_at_dump_time():
+ # Refused where the promise is made, not at file-write time.
+ with pytest.raises(TypeError, match=r"JSON-serializable.*set"):
+ dspy.LM("custom/echo", engine=NonJsonState()).dump_state()
+
+
+def test_saved_state_loads_in_a_fresh_process(tmp_path):
+ # The promise of a class path is that another process can follow it.
+ program = dspy.Predict("question -> answer")
+ program.lm = dspy.LM("custom/echo", engine=Echo("portable"), cache=False)
+ path = tmp_path / "program.json"
+ program.save(path)
+ script = (
+ "import sys, dspy\n"
+ "p = dspy.Predict('question -> answer')\n"
+ f"p.load({str(path)!r}, allow_unsafe_lm_state=True)\n"
+ "print(type(p.lm.engine).__name__, p.lm.engine.tag)\n"
+ )
+ result = subprocess.run([sys.executable, "-c", script], capture_output=True, text=True,
+ cwd=str(tmp_path), env={**os.environ, "PYTHONPATH": ":".join(sys.path)})
+ assert result.returncode == 0, result.stderr
+ assert result.stdout.strip() == "Echo portable"
+
+
+def test_serialized_engine_records_are_validated():
+ base = dspy.LM("custom/echo").dump_state()
+ with pytest.raises(ValueError, match="selection string or a"):
+ dspy.LM.load_state({**base, "engine": {"state": {}}}, allow_custom_lm_class=True)
+ with pytest.raises(ValueError, match="module:QualName"):
+ dspy.LM.load_state({**base, "engine": {"class": "nonsense", "state": {}}}, allow_custom_lm_class=True)
+ with pytest.raises(ValueError, match="without a custom engine"):
+ dspy.LM.load_state({**base, "async_engine": {"class": f"{__name__}:AsyncEcho", "state": {}}},
+ allow_custom_lm_class=True)
+ with pytest.raises(ImportError, match="cannot be imported"):
+ dspy.LM.load_state({**base, "engine": {"class": f"{__name__}:Missing", "state": {}}},
+ allow_custom_lm_class=True)
+ with pytest.raises(TypeError, match="no load_state"):
+ dspy.LM.load_state({**base, "engine": {"class": f"{__name__}:Stateless", "state": {}}},
+ allow_custom_lm_class=True)
+
+
+def test_built_in_engine_selection_state_is_unchanged():
+ assert "engine" not in dspy.LM("openai/gpt-4o").dump_state()
+ assert dspy.LM("openai/gpt-4o", engine="litellm").dump_state()["engine"] == "litellm"
+ loaded = dspy.LM.load_state(dspy.LM("openai/gpt-4o", engine="lm15").dump_state())
+ assert loaded.engine == "lm15"
diff --git a/tests/streaming/test_react_v2_native_tool_stream.py b/tests/streaming/test_react_v2_native_tool_stream.py
new file mode 100644
index 000000000..e6bde0357
--- /dev/null
+++ b/tests/streaming/test_react_v2_native_tool_stream.py
@@ -0,0 +1,116 @@
+"""ReActV2 under streamify with a backend that streams native tool calls.
+
+Pinned after cmpnd-ai/breaka-your-lm reported the streamed run failing on
+Fireworks (2026-09-16). These tests establish what DSPy's streaming path does
+with two kinds of model output: a native tool-call stream finishes like the
+plain run, and a call written as JSON text fails the same way with or without
+streamify. They do not establish what that live model produced; that needs
+the captured request and response.
+"""
+
+import json
+
+import pytest
+
+import dspy
+from dspy._vendor.lm15.types import (
+ StreamDeltaEvent,
+ StreamEndEvent,
+ StreamStartEvent,
+ TextDelta,
+ ToolCallDelta,
+ ToolCallPart,
+)
+from dspy.lm15 import Message, Response, Usage
+
+
+class ScriptedToolEngine:
+ """Turn 1 calls add(3, 4); turn 2 calls submit(answer=35)."""
+
+ def __init__(self, *, tool_call_as_text=False):
+ self.turns = 0
+ self.modes = []
+ self.tool_call_as_text = tool_call_as_text
+
+ def _turn(self):
+ self.turns += 1
+ return ("add", {"a": 3, "b": 4}) if self.turns == 1 else ("submit", {"answer": 35})
+
+ def complete(self, request):
+ self.modes.append("complete")
+ name, args = self._turn()
+ if self.tool_call_as_text:
+ return Response(id="r", model=request.model, message=Message.assistant(json.dumps({"name": name, "arguments": args})),
+ finish_reason="stop", usage=Usage())
+ return Response(id="r", model=request.model,
+ message=Message(role="assistant", parts=(ToolCallPart(id="c1", name=name, input=args),)),
+ finish_reason="tool_call", usage=Usage())
+
+ def stream(self, request):
+ self.modes.append("stream")
+ name, args = self._turn()
+ yield StreamStartEvent(id="r", model=request.model)
+ if self.tool_call_as_text:
+ yield StreamDeltaEvent(TextDelta(text=json.dumps({"name": name, "arguments": args})))
+ yield StreamEndEvent(finish_reason="stop", usage=Usage())
+ else:
+ yield StreamDeltaEvent(ToolCallDelta(input=json.dumps(args), id="c1", name=name))
+ yield StreamEndEvent(finish_reason="tool_call", usage=Usage())
+
+
+def add(a: int, b: int) -> int:
+ """Add two numbers."""
+ return a + b
+
+
+def multiply(a: int, b: int) -> int:
+ """Multiply two numbers."""
+ return a * b
+
+
+def _program():
+ return dspy.ReActV2("question -> answer: int", tools=[add, multiply], max_iters=5)
+
+
+def _drain(stream):
+ final = None
+ for item in stream:
+ if isinstance(item, dspy.Prediction):
+ final = item
+ return final
+
+
+def test_react_v2_streams_native_tool_calls_like_the_plain_run():
+ plain_engine = ScriptedToolEngine()
+ with dspy.context(lm=dspy.LM("scripted/tools", engine=plain_engine, cache=False, num_retries=0)):
+ plain = _program()(question="What is (3 + 4) * 5?")
+ assert plain.answer == 35 and plain.termination_reason == "submit" and plain_engine.modes == ["complete", "complete"]
+
+ streamed_engine = ScriptedToolEngine()
+ with dspy.context(lm=dspy.LM("scripted/tools", engine=streamed_engine, cache=False, num_retries=0)):
+ stream = dspy.streamify(_program(), async_streaming=False)
+ final = _drain(stream(question="What is (3 + 4) * 5?"))
+ assert final.answer == 35 and final.termination_reason == "submit"
+ assert streamed_engine.modes == ["stream", "stream"]
+
+
+def test_tool_call_written_as_text_fails_the_same_way_with_or_without_streamify():
+ # The model answers with the call as JSON text instead of a native tool
+ # call: no adapter can read it, so the loop ends in a parse error on
+ # both paths. Streamify only skips the JSON-adapter retry, because output
+ # was already streamed; it does not change what the model did.
+ question = "What is (3 + 4) * 5?"
+ with dspy.context(lm=dspy.LM("scripted/tools", engine=ScriptedToolEngine(tool_call_as_text=True),
+ cache=False, num_retries=0)):
+ with pytest.raises(dspy.utils.exceptions.AdapterParseError) as plain:
+ _program()(question=question)
+ with dspy.context(lm=dspy.LM("scripted/tools", engine=ScriptedToolEngine(tool_call_as_text=True),
+ cache=False, num_retries=0)):
+ stream = dspy.streamify(_program(), async_streaming=False)
+ with pytest.raises(BaseException) as streamed:
+ _drain(stream(question=question))
+ inner = streamed.value
+ while hasattr(inner, "exceptions"): # streamify wraps failures in an exception group (3.10 has no name for it)
+ inner = inner.exceptions[0]
+ assert type(inner) is type(plain.value)
+ assert '"name": "submit"' in str(inner) and '"name": "submit"' in str(plain.value)