DSPy: The framework for programming—not prompting—language models
Python
Custom engines own their connection: refuse client settings at every door, save through engine state (#10441)
Maxime Rivest7b0d899f331c
diff --git a/docs/docs/community/normalized-lm-api-migration.md b/docs/docs/community/normalized-lm-api-migration.mdindex 9689fa78a..e76e9e9ec 100644--- a/docs/docs/community/normalized-lm-api-migration.md+++ b/docs/docs/community/normalized-lm-api-migration.md@@ -273,6 +273,14 @@ an async iterator for async calls. DSPy does not silently run a sync backend onasync path. Engines must not add another DSPy cache or retry loop. Custom enginesare caller-owned and are not closed by DSPy.+A custom engine owns its connection: `api_key`, `api_base`, `timeout`,+`extra_headers` and the other client settings are refused on `dspy.LM(engine=...)`,+on `copy()` and on every call, rather than dropped. `copy(engine=...)` replaces the+engine pair, and each side must be its kind (a coroutine function on the async side).+An engine is saved by `dump_state()`/`load_state()` of its own, and loading one is+gated by `allow_unsafe_lm_state=True` like a custom LM class. See the+[custom-engine tutorial](../tutorials/custom_lm_engines/index.md).+Implementing custom LMs through `BaseLM.forward(prompt=None, messages=None, **kwargs)`or `aforward` is **deprecated**. Implement an engine instead, as shown above and inthe [custom-engine tutorial](../tutorials/custom_lm_engines/index.md). The old subclassdiff --git a/docs/docs/tutorials/custom_lm_engines/index.md b/docs/docs/tutorials/custom_lm_engines/index.mdindex 1c192c2f3..975a5cf61 100644--- a/docs/docs/tutorials/custom_lm_engines/index.md+++ b/docs/docs/tutorials/custom_lm_engines/index.md@@ -1,6 +1,8 @@# Custom LM Engines-In the DSPy 3.4 development API, you can supply your own execution engine to `dspy.LM`. The minimum synchronous interface is:+You can supply your own execution engine to `dspy.LM`. An engine is for a backend that is not an HTTP provider DSPy's bundled lm15 can speak to: a CLI, an in-process model, an agent harness. It owns its whole connection, so the [rules below](#what-a-custom-engine-owns) apply.++The minimum synchronous engine interface is:```pythonclass MyEngine:@@ -19,6 +21,33 @@ family and owns retries. Unexpected exceptions retain their original cause andare not guessed to be retryable from their message text. See[errors and retry ownership](../../community/normalized-lm-api-migration.md#errors-and-retry-ownership).+## What a custom engine owns++A custom engine is borrowed by `dspy.LM` and owns its connection. Three rules follow, and DSPy enforces each rather than guessing:++**Connection settings are refused.** `dspy.LM(engine=MyEngine(), api_key=...)` raises `ValueError`, and so do `api_base`, `base_url`, `timeout`, `headers`, `extra_headers` and the other client settings — on construction, on `copy()`, and on every call (`lm("hi", api_key=...)`), before any cache lookup. There is no channel from the LM to the engine for them, and dropping them silently would let a call run with the engine's key while the LM said another. Give them to the engine's constructor.++**The engine pair is one unit.** `async_engine=` is only accepted with a custom `engine=`, and each side must be its kind: the sync engine's `complete` is a plain function returning a `Response`, the async engine's is a coroutine function (`async def complete`). A sync method on the async side, or the reverse, is refused at construction rather than on the first call. `lm.copy(engine=...)` replaces both: the copy has no async engine unless you pass a new `async_engine=` in the same call. A copy that does not mention `engine` keeps the pair.++**Saving needs the engine's own state.** An engine is saved when it implements `dump_state() -> dict` (JSON-serializable, no secrets) and the classmethod `load_state(state) -> engine`:++```python+class MyEngine:+ def __init__(self, model="gpt-6-astra"):+ self.model = model++ def complete(self, request): ...++ def dump_state(self):+ return {"model": self.model}++ @classmethod+ def load_state(cls, state):+ return cls(**state)+```++`lm.dump_state()` then records `{"engine": {"class": "your.module:MyEngine", "state": {...}}}` (and `async_engine` likewise). Loading imports that class from the file, which is the same trust decision as a custom LM class, so it is gated the same way: `program.load(path, allow_unsafe_lm_state=True)` or `dspy.LM.load_state(state, allow_custom_lm_class=True)`. The class must be importable by that path in the process that loads the state. Define it at module level, not inside a function — `dump_state()` refuses a class it cannot import back — and for state that must outlive the session, in an importable module: a class defined in a script or notebook is recorded as `__main__:MyEngine`, which loads only in a process whose `__main__` defines it again. `dump_state()` also refuses engine state JSON cannot carry. An engine without the two methods cannot be saved either; `dump_state()` says so and the way out is to save the program without that LM and set it again after loading. Keep secrets out of engine state, as DSPy keeps `api_key` out of LM state.+This tutorial wraps the [Pi CLI](https://pi.dev) as a custom engine. Pi keeps its normal system prompt and tools, so a DSPy program can ask it to inspect a repository. The entire agent run, including tool calls, becomes one DSPy LM response.## Prerequisites@@ -78,7 +107,7 @@ program()```-Here `"pi"` is a DSPy model label for history; the engine's CLI arguments select the actual provider and model.+Here `"pi"` is a DSPy model label for history; the engine's CLI arguments select the actual provider and model. `PiEngine` holds no state, so to make it saveable add `dump_state` returning `{}` and `load_state` returning `cls()`.An answer for the DSPy repository can look like:diff --git a/dspy/clients/execution.py b/dspy/clients/execution.pyindex 64e39c8a1..7962c1002 100644--- a/dspy/clients/execution.py+++ b/dspy/clients/execution.py@@ -136,6 +136,11 @@ def prepare(lm, prompt, messages, kwargs, *, asynchronous=False, direct=False):legacy = {"model": lm.model}use_cache = kwargs.get("cache", lm.cache) if managed and getattr(lm, "_cache_responses", True) else Falsereturn PreparedCall(None, None, kwargs, legacy, request, use_cache, 1, managed)+ if managed and not isinstance(getattr(lm, "_engine_spec", "auto"), str):+ # A custom engine owns its connection; a call cannot hand it one either.+ from dspy.clients.lm import _refuse_client_settings++ _refuse_client_settings(lm._engine_spec, kwargs, where="LM call")merged = {**lm.kwargs, **{key: val for key, val in kwargs.items() if key != "cache"}}prompt_cache = merged.get("prompt_cache")if prompt_cache is not None:diff --git a/dspy/clients/lm.py b/dspy/clients/lm.pyindex a4ccde3a8..0ddf349f0 100644--- a/dspy/clients/lm.py+++ b/dspy/clients/lm.py@@ -44,6 +44,150 @@ def _is_openai_reasoning_model(model: str) -> bool:) is not None+_ENGINE_SELECTIONS = ("auto", "lm15", "litellm")+# Serialized custom engines: {"class": "pkg.module:Qual.Name", "state": {...}}.+_ENGINE_CLASS_KEY = "class"+_ENGINE_STATE_KEY = "state"+++def _is_coroutine_function(fn) -> bool:+ import inspect++ if inspect.iscoroutinefunction(fn):+ return True+ # A callable object whose __call__ (own or inherited) is async: resolve+ # it through the MRO as Python does. A plain function's type resolves+ # __call__ to a slot wrapper, which is not a coroutine function.+ call = next((vars(cls)["__call__"] for cls in type(fn).__mro__ if "__call__" in vars(cls)), None)+ return inspect.iscoroutinefunction(call)+++def _check_engines(engine, async_engine) -> None:+ """One rule for the constructor and copy(): a selection string with no+ async counterpart, or a custom engine object with complete(Request) — a+ plain function on the sync engine, a coroutine function on the async one+ (dspy.clients.engines.base). Checked here, at construction, rather than+ on the first call, where the wrong kind fails as an unexpected error."""+ if isinstance(engine, str):+ if engine not in _ENGINE_SELECTIONS:+ raise ValueError("engine must be 'auto', 'lm15', 'litellm', or an engine object")+ if async_engine is not None:+ raise ValueError("async_engine is only used with a custom engine object")+ return+ complete = getattr(engine, "complete", None)+ if not callable(complete):+ raise TypeError("A custom engine must implement complete(Request) -> Response")+ if _is_coroutine_function(complete):+ raise TypeError(+ "engine.complete is a coroutine function; the sync engine returns a Response directly. "+ "Pass an async engine as async_engine="+ )+ if async_engine is None:+ return+ acomplete = getattr(async_engine, "complete", None)+ if not callable(acomplete):+ raise TypeError("A custom async engine must implement async complete(Request) -> Response")+ if not _is_coroutine_function(acomplete):+ raise TypeError(+ "async_engine.complete is not a coroutine function; the async engine is awaited "+ "(async def complete). Pass a sync engine as engine="+ )+++def _refuse_client_settings(engine, kwargs, *, where: str = "dspy.LM") -> None:+ """A custom engine owns its connection: api_key, api_base, timeout and the+ other client settings have no way to reach it, so they are refused rather+ than dropped (dspy#10409 follow-up: the Fireworks custom engine ran with+ the engine's key while the LM said another). Applied at construction, on+ copy(), and to every call's own keyword arguments — the same rule at+ each door, before any cache lookup."""+ if isinstance(engine, str):+ return+ from dspy.clients.backend_selection import CLIENT_KEYS++ present = sorted(key for key in kwargs if key in CLIENT_KEYS and kwargs[key] is not None)+ if present:+ raise ValueError(+ f"{where}: {present} configure a connection DSPy owns; a custom engine owns its own. "+ "Configure them on the engine instead."+ )+++def _engine_class_path(engine) -> str:+ cls = type(engine)+ return f"{cls.__module__}:{cls.__qualname__}"+++def _dump_engine(engine, name: str) -> dict[str, Any]:+ dump = getattr(engine, "dump_state", None)+ load = getattr(type(engine), "load_state", None)+ if not callable(dump) or not callable(load):+ raise TypeError(+ f"{name}={type(engine).__name__} cannot be saved: a custom engine is saved when it implements "+ "dump_state() -> dict (JSON-serializable, no secrets) and the classmethod load_state(state) "+ "-> engine. Otherwise save the program without this LM and set it again after loading."+ )+ path = _engine_class_path(engine)+ # The path must lead back to this class when loading (pickle's rule): a+ # class defined inside a function or built dynamically would save fine+ # and never load.+ try:+ found = _import_engine_class(path)+ except (ImportError, TypeError, ValueError):+ found = None+ if found is not type(engine):+ raise TypeError(+ f"{name}={type(engine).__name__} cannot be saved: its class is not importable as `{path}`. "+ "Define the engine class at module level so a loader can import it."+ )+ state = dump()+ if not isinstance(state, dict):+ raise TypeError(f"{name}.dump_state() must return a dict, got {type(state).__name__}")+ import json++ try:+ json.dumps(state)+ except (TypeError, ValueError) as exc:+ raise TypeError(f"{name}.dump_state() must return JSON-serializable data: {exc}") from exc+ return {_ENGINE_CLASS_KEY: path, _ENGINE_STATE_KEY: state}+++def _import_engine_class(class_path: str) -> type:+ import importlib++ module_name, _, qualname = class_path.partition(":")+ if not module_name or not qualname:+ raise ValueError(f"Serialized engine class path must be 'module:QualName', got {class_path!r}")+ obj = importlib.import_module(module_name)+ for part in qualname.split("."):+ try:+ obj = getattr(obj, part)+ except AttributeError as exc:+ raise ImportError(f"Serialized engine class `{class_path}` cannot be imported: {exc}") from exc+ if not isinstance(obj, type):+ raise TypeError(f"Serialized engine `{class_path}` is not a class")+ return obj+++def _load_engine(record, name: str, *, allow_custom_lm_class: bool):+ if not isinstance(record, dict) or _ENGINE_CLASS_KEY not in record:+ raise ValueError(f"Serialized {name} must be a selection string or a {{'class', 'state'}} record")+ class_path = record[_ENGINE_CLASS_KEY]+ if not allow_custom_lm_class:+ raise ValueError(+ f"Refusing to import custom serialized engine class `{class_path}`. "+ "Pass allow_unsafe_lm_state=True when loading trusted files to enable custom engines."+ )+ cls = _import_engine_class(class_path)+ load = getattr(cls, "load_state", None)+ if not callable(load):+ raise TypeError(f"Serialized engine `{class_path}` has no load_state(state) classmethod")+ engine = load(record.get(_ENGINE_STATE_KEY, {}))+ if not callable(getattr(engine, "complete", None)):+ raise TypeError(f"`{class_path}.load_state` must return an engine with complete(Request)")+ return engine++class LM(BaseLM):"""A language model supporting chat or text completion requests for use with DSPy modules.@@ -92,8 +236,11 @@ class LM(BaseLM):backoff.engine: 'auto' prefers lm15 for representable requests; 'litellm' preserves the compatibility backend;'lm15' refuses unsupported mappings rather than selecting LiteLLM. A custom engine implements- complete(Request) -> Response and optionally stream(Request). Engines are borrowed.- async_engine: Async counterpart when supplying a custom engine object.+ complete(Request) -> Response and optionally stream(Request). Engines are borrowed and own+ their connection: api_key, api_base, timeout and the other client settings are refused with a+ custom engine, at construction and on every call.+ async_engine: Async counterpart when supplying a custom engine object. The pair is one unit:+ copy(engine=...) replaces both unless async_engine= is given too.prompt_cache: Optional lm15 CacheConfig for provider-side prompt caching on ordinary calls.Separate from DSPy's response cache. Requires native lm15 or a canonical custom engine;may incur cache-write/storage charges. A call-time value overrides this default, and None@@ -107,13 +254,8 @@ class LM(BaseLM):only affects generation when `temperature` is non-zero. This argument isstripped before sending requests to the provider."""- if isinstance(engine, str):- if engine not in {"auto", "lm15", "litellm"}:- raise ValueError("engine must be 'auto', 'lm15', 'litellm', or an engine object")- if async_engine is not None:- raise ValueError("async_engine is only used with a custom engine object")- elif not callable(getattr(engine, "complete", None)):- raise TypeError("A custom engine must implement complete(Request) -> Response")+ _check_engines(engine, async_engine)+ _refuse_client_settings(engine, kwargs)if isinstance(num_retries, bool) or not isinstance(num_retries, int) or num_retries < 0:raise ValueError("num_retries must be a nonnegative integer")if prompt_cache is not None:@@ -283,12 +425,16 @@ class LM(BaseLM):def copy(self, **kwargs):if kwargs.get("prompt_cache") is not None and not isinstance(kwargs["prompt_cache"], CacheConfig):raise TypeError("prompt_cache must be a dspy.lm15.CacheConfig or None")- spec = kwargs.pop("engine", self._engine_spec)- async_spec = kwargs.pop("async_engine", self._async_engine_spec)- if isinstance(spec, str) and spec not in {"auto", "lm15", "litellm"}:- raise ValueError("Unknown engine selection")- if not isinstance(spec, str) and not callable(getattr(spec, "complete", None)):- raise TypeError("Custom engines must implement complete(Request)")+ if "engine" in kwargs:+ # The engine pair is one unit: a new engine drops the old async+ # counterpart unless a new one comes with it.+ spec = kwargs.pop("engine")+ async_spec = kwargs.pop("async_engine", None)+ else:+ spec = self._engine_spec+ async_spec = kwargs.pop("async_engine", self._async_engine_spec)+ _check_engines(spec, async_spec)+ _refuse_client_settings(spec, {**self.kwargs, **kwargs}, where="LM.copy")copied = super().copy(**kwargs)copied._engine_spec = speccopied._async_engine_spec = async_spec@@ -389,19 +535,31 @@ class LM(BaseLM):def dump_state(self):"""Return a sanitized reconstruction state for this LM.+ A custom engine is recorded as its class path and its own+ ``dump_state()``; the class must be importable by that path in the+ process that loads the state. A class defined in ``__main__`` (a+ script or notebook) loads only where ``__main__`` defines it again;+ for durable state define the engine in an importable module.+Returns:A dictionary that can be passed to `BaseLM.load_state` toreconstruct this `LM`. The state excludes API keys."""- if not isinstance(self._engine_spec, str):- raise TypeError("Custom engine objects require custom dump_state/load_state methods; they cannot be stored in JSON LM state.")state = super().dump_state()if state.get("prompt_cache") is not None:from dspy._vendor.lm15.serde import cache_config_to_dictstate["prompt_cache"] = cache_config_to_dict(state["prompt_cache"])- if self._engine_spec != "auto":- state["engine"] = self._engine_spec+ if isinstance(self._engine_spec, str):+ if self._engine_spec != "auto":+ state["engine"] = self._engine_spec+ else:+ # A custom engine is saved as its class path and its own state;+ # loading it imports a class from the file, so it is gated like a+ # custom LM class (allow_unsafe_lm_state).+ state["engine"] = _dump_engine(self._engine_spec, "engine")+ if self._async_engine_spec is not None:+ state["async_engine"] = _dump_engine(self._async_engine_spec, "async_engine")state.update({"finetuning_model": self.finetuning_model,@@ -429,6 +587,15 @@ class LM(BaseLM):state["max_tokens"] = state["max_completion_tokens"]state.pop("max_completion_tokens")+ if not isinstance(state.get("engine", "auto"), str):+ state["engine"] = _load_engine(state["engine"], "engine", allow_custom_lm_class=allow_custom_lm_class)+ if state.get("async_engine") is not None:+ state["async_engine"] = _load_engine(+ state["async_engine"], "async_engine", allow_custom_lm_class=allow_custom_lm_class+ )+ elif state.get("async_engine") is not None:+ raise ValueError("Serialized async_engine without a custom engine")+return super().load_state(state, allow_custom_lm_class=allow_custom_lm_class)def _check_truncation(self, results):diff --git a/tests/clients/test_custom_engine_contract.py b/tests/clients/test_custom_engine_contract.pynew file mode 100644index 000000000..1e18cdfec--- /dev/null+++ b/tests/clients/test_custom_engine_contract.py@@ -0,0 +1,293 @@+"""The custom-engine contract: a borrowed engine owns its connection, the+engine pair is one unit, and a custom engine is saved through its own state.++Pinned from cmpnd-ai/breaka-your-lm results/CUSTOM_ENGINE_FINDINGS.md+(2026-09-16): api_key/api_base/timeout/extra_headers were dropped before a+custom engine saw the request, the constructor and copy() applied different+rules to async_engine, and dump_state refused custom engines with no+documented way out.+"""++import json+import os+import re+import subprocess+import sys++import pytest++import dspy+from dspy.lm15 import Message, Response, Usage+++class Echo:+ """A stateful engine that knows how to save and restore itself."""++ def __init__(self, tag="t"):+ self.tag = tag++ def complete(self, request):+ return Response(id=None, model=request.model, message=Message.assistant(f"echo:{self.tag}"),+ finish_reason="stop", usage=Usage())++ def dump_state(self):+ return {"tag": self.tag}++ @classmethod+ def load_state(cls, state):+ return cls(**state)+++class AsyncEcho(Echo):+ async def complete(self, request):+ return Echo.complete(self, request)+++class BadState(Echo):+ """An engine whose state is not a dict."""++ def dump_state(self):+ return ["tag"]+++class NonJsonState(Echo):+ """An engine whose state is a dict that JSON cannot carry."""++ def dump_state(self):+ return {"tag": {1, 2}}+++class Stateless:+ """An engine with no save/load protocol."""++ def complete(self, request):+ return Echo().complete(request)+++# ─── connection settings belong to the engine ─────────────────────────++[email protected]("setting", [+ {"api_key": "k"}, {"api_base": "http://127.0.0.1:9/v1"}, {"base_url": "http://127.0.0.1:9/v1"},+ {"timeout": 1}, {"extra_headers": {"X-Test": "1"}}, {"headers": {"X-Test": "1"}}, {"api_version": "v1"},+ {"organization": "org"}, {"project": "p"}, {"extra_query": {"a": "b"}}, {"custom_llm_provider": "x"},+])+def test_client_settings_are_refused_with_a_custom_engine(setting):+ with pytest.raises(ValueError, match=re.escape(f"dspy.LM: {sorted(setting)}") + ".*custom engine owns its own"):+ dspy.LM("custom/echo", engine=Echo(), **setting)++[email protected]("setting", [{"api_key": "k"}, {"api_base": "http://127.0.0.1:9/v1"}, {"timeout": 0.001},+ {"extra_headers": {"X-Test": "1"}}])+async def test_client_settings_are_refused_on_every_call(setting):+ # The constructor and copy() refused them; a call passed them in and the+ # engine never saw them (greptile on dspy#10441 found the third door open).+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=AsyncEcho(), cache=True)+ refused = re.escape(f"LM call: {sorted(setting)}")+ with pytest.raises(ValueError, match=refused):+ lm("hi", **setting)+ with pytest.raises(ValueError, match=refused):+ await lm.acall("hi", **setting)+ with pytest.raises(ValueError, match=refused):+ lm.forward("hi", **setting)+ with pytest.raises(ValueError, match=refused):+ await lm.aforward("hi", **setting)+ assert not lm.history # refused before any attempt or cache entry+ assert lm("hi") == ["echo:t"]+++def test_client_settings_are_refused_on_copy_too():+ lm = dspy.LM("custom/echo", engine=Echo(), cache=False)+ with pytest.raises(ValueError, match=r"LM.copy: \['api_key'\]"):+ lm.copy(api_key="k")+ # And when a copy switches to a custom engine while the LM carries them.+ native = dspy.LM("openai/gpt-4o", api_key="k", cache=False)+ with pytest.raises(ValueError, match="api_key"):+ native.copy(engine=Echo())+ # Clearing them in the same copy is the way through.+ switched = native.copy(engine=Echo(), api_key=None)+ assert "api_key" not in switched.kwargs and isinstance(switched.engine, Echo)+++def test_none_is_not_a_setting():+ lm = dspy.LM("custom/echo", engine=Echo(), api_key=None, timeout=None, cache=False)+ assert lm("hi") == ["echo:t"]+++def test_built_in_engines_keep_taking_client_settings():+ lm = dspy.LM("openai/gpt-4o", engine="lm15", api_key="k", api_base="http://127.0.0.1:9/v1", timeout=3)+ assert lm.kwargs["api_key"] == "k"+ assert lm.copy(api_key="k2").kwargs["api_key"] == "k2"+++# ─── the engine pair is one unit ──────────────────────────────────────+++def test_constructor_and_copy_apply_the_same_async_rule():+ with pytest.raises(ValueError, match="async_engine is only used with a custom engine"):+ dspy.LM("custom/echo", engine="auto", async_engine=AsyncEcho())+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=AsyncEcho(), cache=False)+ with pytest.raises(ValueError, match="async_engine is only used with a custom engine"):+ lm.copy(engine="litellm", async_engine=AsyncEcho())+++def test_copy_with_a_new_engine_replaces_the_pair():+ lm = dspy.LM("custom/echo", engine=Echo("a"), async_engine=AsyncEcho("a"), cache=False)+ to_litellm = lm.copy(engine="litellm")+ assert to_litellm.engine == "litellm" and to_litellm._async_engine_spec is None+ other = lm.copy(engine=Echo("b"))+ assert other.engine.tag == "b" and other._async_engine_spec is None+ both = lm.copy(engine=Echo("c"), async_engine=AsyncEcho("c"))+ assert both.engine.tag == "c" and both._async_engine_spec.tag == "c"+ # A copy that does not mention the engine keeps the whole pair.+ same = lm.copy(temperature=0.5)+ assert same.engine is lm.engine and same._async_engine_spec is lm._async_engine_spec+++async def test_async_counterpart_must_be_an_engine():+ with pytest.raises(TypeError, match="async complete"):+ dspy.LM("custom/echo", engine=Echo(), async_engine=object())+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=AsyncEcho("z"), cache=False)+ assert await lm.acall("hi") == ["echo:z"]+++def test_each_side_of_the_pair_must_be_the_right_kind():+ # A sync complete() as the async engine would be awaited as a Response+ # on the first acall; an async complete() as the sync engine would hand+ # execute() a coroutine. Both are refused at construction (greptile on+ # dspy#10441).+ with pytest.raises(TypeError, match=r"async_engine\.complete is not a coroutine function"):+ dspy.LM("custom/echo", engine=Echo(), async_engine=Echo())+ with pytest.raises(TypeError, match=r"engine\.complete is a coroutine function"):+ dspy.LM("custom/echo", engine=AsyncEcho())+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=AsyncEcho(), cache=False)+ with pytest.raises(TypeError, match="not a coroutine function"):+ lm.copy(async_engine=Echo())+ with pytest.raises(TypeError, match="is a coroutine function"):+ lm.copy(engine=AsyncEcho())+++class AsyncCallable:+ async def __call__(self, request):+ return Echo("inherited").complete(request)+++class InheritsAsyncCall(AsyncCallable):+ pass+++class WithCallableComplete:+ complete = InheritsAsyncCall()+++async def test_an_inherited_async_call_counts_as_a_coroutine_function():+ # Python resolves __call__ through the MRO; so does the kind check.+ lm = dspy.LM("custom/echo", engine=Echo(), async_engine=WithCallableComplete(), cache=False)+ assert await lm.acall("hi") == ["echo:inherited"]+++# ─── saving and loading ───────────────────────────────────────────────+++def test_custom_engine_round_trips_through_json_state():+ lm = dspy.LM("custom/echo", engine=Echo("a"), async_engine=AsyncEcho("b"), cache=False, num_retries=0)+ state = lm.dump_state()+ json.dumps(state) # JSON-serializable+ assert state["engine"] == {"class": f"{__name__}:Echo", "state": {"tag": "a"}}+ assert state["async_engine"] == {"class": f"{__name__}:AsyncEcho", "state": {"tag": "b"}}+ loaded = dspy.LM.load_state(state, allow_custom_lm_class=True)+ assert isinstance(loaded.engine, Echo) and loaded.engine.tag == "a"+ assert isinstance(loaded._async_engine_spec, AsyncEcho) and loaded._async_engine_spec.tag == "b"+ assert loaded.num_retries == 0 and loaded("hi") == ["echo:a"]+++def test_loading_a_custom_engine_is_gated_like_a_custom_lm_class():+ state = dspy.LM("custom/echo", engine=Echo()).dump_state()+ with pytest.raises(ValueError, match="Refusing to import custom serialized engine class"):+ dspy.LM.load_state(state)+ with pytest.raises(ValueError, match="Refusing to import custom serialized engine class"):+ dspy.BaseLM.load_state(state)+++def test_program_save_and_load_with_a_custom_engine(tmp_path):+ program = dspy.Predict("question -> answer")+ program.lm = dspy.LM("custom/echo", engine=Echo("saved"), cache=False)+ path = tmp_path / "program.json"+ program.save(path)+ fresh = dspy.Predict("question -> answer")+ with pytest.raises(ValueError, match="allow_unsafe_lm_state"):+ fresh.load(path)+ fresh.load(path, allow_unsafe_lm_state=True)+ assert isinstance(fresh.lm.engine, Echo) and fresh.lm.engine.tag == "saved"+++def test_engine_without_state_protocol_is_refused_with_the_way_out():+ lm = dspy.LM("custom/echo", engine=Stateless())+ with pytest.raises(TypeError, match=r"dump_state\(\) -> dict.*load_state\(state\)") as info:+ lm.dump_state()+ assert "save the program without this LM" in str(info.value)+++def test_function_local_engine_class_is_refused_at_save_time():+ class Local(Echo):+ pass++ lm = dspy.LM("custom/echo", engine=Local())+ with pytest.raises(TypeError, match=r"not importable as .*<locals>.*module level"):+ lm.dump_state()+++def test_engine_state_must_be_a_dict():+ with pytest.raises(TypeError, match="must return a dict"):+ dspy.LM("custom/echo", engine=BadState()).dump_state()+++def test_engine_state_must_be_json_at_dump_time():+ # Refused where the promise is made, not at file-write time.+ with pytest.raises(TypeError, match=r"JSON-serializable.*set"):+ dspy.LM("custom/echo", engine=NonJsonState()).dump_state()+++def test_saved_state_loads_in_a_fresh_process(tmp_path):+ # The promise of a class path is that another process can follow it.+ program = dspy.Predict("question -> answer")+ program.lm = dspy.LM("custom/echo", engine=Echo("portable"), cache=False)+ path = tmp_path / "program.json"+ program.save(path)+ script = (+ "import sys, dspy\n"+ "p = dspy.Predict('question -> answer')\n"+ f"p.load({str(path)!r}, allow_unsafe_lm_state=True)\n"+ "print(type(p.lm.engine).__name__, p.lm.engine.tag)\n"+ )+ result = subprocess.run([sys.executable, "-c", script], capture_output=True, text=True,+ cwd=str(tmp_path), env={**os.environ, "PYTHONPATH": ":".join(sys.path)})+ assert result.returncode == 0, result.stderr+ assert result.stdout.strip() == "Echo portable"+++def test_serialized_engine_records_are_validated():+ base = dspy.LM("custom/echo").dump_state()+ with pytest.raises(ValueError, match="selection string or a"):+ dspy.LM.load_state({**base, "engine": {"state": {}}}, allow_custom_lm_class=True)+ with pytest.raises(ValueError, match="module:QualName"):+ dspy.LM.load_state({**base, "engine": {"class": "nonsense", "state": {}}}, allow_custom_lm_class=True)+ with pytest.raises(ValueError, match="without a custom engine"):+ dspy.LM.load_state({**base, "async_engine": {"class": f"{__name__}:AsyncEcho", "state": {}}},+ allow_custom_lm_class=True)+ with pytest.raises(ImportError, match="cannot be imported"):+ dspy.LM.load_state({**base, "engine": {"class": f"{__name__}:Missing", "state": {}}},+ allow_custom_lm_class=True)+ with pytest.raises(TypeError, match="no load_state"):+ dspy.LM.load_state({**base, "engine": {"class": f"{__name__}:Stateless", "state": {}}},+ allow_custom_lm_class=True)+++def test_built_in_engine_selection_state_is_unchanged():+ assert "engine" not in dspy.LM("openai/gpt-4o").dump_state()+ assert dspy.LM("openai/gpt-4o", engine="litellm").dump_state()["engine"] == "litellm"+ loaded = dspy.LM.load_state(dspy.LM("openai/gpt-4o", engine="lm15").dump_state())+ assert loaded.engine == "lm15"diff --git a/tests/streaming/test_react_v2_native_tool_stream.py b/tests/streaming/test_react_v2_native_tool_stream.pynew file mode 100644index 000000000..e6bde0357--- /dev/null+++ b/tests/streaming/test_react_v2_native_tool_stream.py@@ -0,0 +1,116 @@+"""ReActV2 under streamify with a backend that streams native tool calls.++Pinned after cmpnd-ai/breaka-your-lm reported the streamed run failing on+Fireworks (2026-09-16). These tests establish what DSPy's streaming path does+with two kinds of model output: a native tool-call stream finishes like the+plain run, and a call written as JSON text fails the same way with or without+streamify. They do not establish what that live model produced; that needs+the captured request and response.+"""++import json++import pytest++import dspy+from dspy._vendor.lm15.types import (+ StreamDeltaEvent,+ StreamEndEvent,+ StreamStartEvent,+ TextDelta,+ ToolCallDelta,+ ToolCallPart,+)+from dspy.lm15 import Message, Response, Usage+++class ScriptedToolEngine:+ """Turn 1 calls add(3, 4); turn 2 calls submit(answer=35)."""++ def __init__(self, *, tool_call_as_text=False):+ self.turns = 0+ self.modes = []+ self.tool_call_as_text = tool_call_as_text++ def _turn(self):+ self.turns += 1+ return ("add", {"a": 3, "b": 4}) if self.turns == 1 else ("submit", {"answer": 35})++ def complete(self, request):+ self.modes.append("complete")+ name, args = self._turn()+ if self.tool_call_as_text:+ return Response(id="r", model=request.model, message=Message.assistant(json.dumps({"name": name, "arguments": args})),+ finish_reason="stop", usage=Usage())+ return Response(id="r", model=request.model,+ message=Message(role="assistant", parts=(ToolCallPart(id="c1", name=name, input=args),)),+ finish_reason="tool_call", usage=Usage())++ def stream(self, request):+ self.modes.append("stream")+ name, args = self._turn()+ yield StreamStartEvent(id="r", model=request.model)+ if self.tool_call_as_text:+ yield StreamDeltaEvent(TextDelta(text=json.dumps({"name": name, "arguments": args})))+ yield StreamEndEvent(finish_reason="stop", usage=Usage())+ else:+ yield StreamDeltaEvent(ToolCallDelta(input=json.dumps(args), id="c1", name=name))+ yield StreamEndEvent(finish_reason="tool_call", usage=Usage())+++def add(a: int, b: int) -> int:+ """Add two numbers."""+ return a + b+++def multiply(a: int, b: int) -> int:+ """Multiply two numbers."""+ return a * b+++def _program():+ return dspy.ReActV2("question -> answer: int", tools=[add, multiply], max_iters=5)+++def _drain(stream):+ final = None+ for item in stream:+ if isinstance(item, dspy.Prediction):+ final = item+ return final+++def test_react_v2_streams_native_tool_calls_like_the_plain_run():+ plain_engine = ScriptedToolEngine()+ with dspy.context(lm=dspy.LM("scripted/tools", engine=plain_engine, cache=False, num_retries=0)):+ plain = _program()(question="What is (3 + 4) * 5?")+ assert plain.answer == 35 and plain.termination_reason == "submit" and plain_engine.modes == ["complete", "complete"]++ streamed_engine = ScriptedToolEngine()+ with dspy.context(lm=dspy.LM("scripted/tools", engine=streamed_engine, cache=False, num_retries=0)):+ stream = dspy.streamify(_program(), async_streaming=False)+ final = _drain(stream(question="What is (3 + 4) * 5?"))+ assert final.answer == 35 and final.termination_reason == "submit"+ assert streamed_engine.modes == ["stream", "stream"]+++def test_tool_call_written_as_text_fails_the_same_way_with_or_without_streamify():+ # The model answers with the call as JSON text instead of a native tool+ # call: no adapter can read it, so the loop ends in a parse error on+ # both paths. Streamify only skips the JSON-adapter retry, because output+ # was already streamed; it does not change what the model did.+ question = "What is (3 + 4) * 5?"+ with dspy.context(lm=dspy.LM("scripted/tools", engine=ScriptedToolEngine(tool_call_as_text=True),+ cache=False, num_retries=0)):+ with pytest.raises(dspy.utils.exceptions.AdapterParseError) as plain:+ _program()(question=question)+ with dspy.context(lm=dspy.LM("scripted/tools", engine=ScriptedToolEngine(tool_call_as_text=True),+ cache=False, num_retries=0)):+ stream = dspy.streamify(_program(), async_streaming=False)+ with pytest.raises(BaseException) as streamed:+ _drain(stream(question=question))+ inner = streamed.value+ while hasattr(inner, "exceptions"): # streamify wraps failures in an exception group (3.10 has no name for it)+ inner = inner.exceptions[0]+ assert type(inner) is type(plain.value)+ assert '"name": "submit"' in str(inner) and '"name": "submit"' in str(plain.value)