Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Run the LM layer through lm15 engines and retire the 3.3 experimental types (#10366)

Maxime Rivest2cda3872d5d5
diff --git a/.github/.internal_dspyai/internals/release-checklist.md b/.github/.internal_dspyai/internals/release-checklist.md
index a08fc11d4..49e9c45aa 100644
--- a/.github/.internal_dspyai/internals/release-checklist.md
+++ b/.github/.internal_dspyai/internals/release-checklist.md
@@ -1,5 +1,11 @@
# Release Checklist
+* [ ] For DSPy 3.4, announce the [3.5 LM-interface cutoff](https://dspy.ai/community/normalized-lm-api-migration/#the-35-cutoff) in the release notes: legacy `forward()`/`aforward()` integrations, both legacy-engine wrappers, custom `complete_legacy()` shortcuts, and OpenAI-style `messages=` calls are deprecated. Include the request/response migration examples and clarify that `lm("hello")` stays supported.
+* [ ] Before releasing DSPy 3.5, verify that the deprecated LM interfaces and temporary adapter-message warning marker are removed, all adapters and engines use lm15 requests/responses, and examples/tests no longer rely on the old interfaces.
+
+* [ ] Before tagging, refresh the bundled model capability/pricing metadata from a selected **released** LiteLLM version. From the repository root, run `python scripts/update_model_metadata.py VERSION` (replace `VERSION` with the release you reviewed; the script's default is pinned, not "latest"). This downloads metadata and license files without importing LiteLLM or making model calls.
+ * Review and commit `dspy/clients/model_metadata/snapshot.json.gz`, `provenance.json`, and `LICENSE` before creating the release tag. Check the recorded version and source hashes; do not regenerate these files during the release build itself.
+ * Run `LITELLM_LOCAL_MODEL_COST_MAP=True python -m pytest tests/clients/test_native_metadata.py` against the updated snapshot. Review changed capabilities/prices rather than adjusting expectations just to make the tests pass.
* [ ] On `main` Create a git tag with pattern X.Y.Z where X, Y, and Z follow the [semver pattern](https://semver.org/). Then push the tag to the origin git repo (github).
* ```bash
git tag X.Y.Z
diff --git a/docs/docs/community/normalized-lm-api-migration.md b/docs/docs/community/normalized-lm-api-migration.md
index af6008580..4646dcba4 100644
--- a/docs/docs/community/normalized-lm-api-migration.md
+++ b/docs/docs/community/normalized-lm-api-migration.md
@@ -1,239 +1,309 @@
-# Typed LM API migration plan
+# DSPy 3.4 LM migration
-DSPy is moving toward a typed language-model boundary while keeping `dspy.BaseLM` as the public base class for language models.
+DSPy's LM layer now uses the lm15 objects bundled inside DSPy. Import them from
+`dspy.lm15`; no separate installation is needed. They are the original lm15
+classes, not DSPy wrappers or subclasses.
-**Most DSPy users do not need to change anything in DSPy 3.3**. Existing `lm(...)`, modules, and programs keep their current behavior by default. The typed LM API is opt-in in 3.3 with `dspy.context(experimental=True)`.
+## The 3.5 cutoff
-TLDR: `dspy.LM.forward` is currently untyped and mixes DSPy-specific behavior with OpenAI/LiteLLM-shaped inputs. We will migrate `BaseLM.forward` and `LM.forward` from:
+**3.4 is the transition release; 3.5 removes the old LM integration interfaces.**
-```python
-def forward(self, prompt=None, messages=None, **kwargs):
- ...
-```
-
-to:
-
-```python
-def forward(self, request: dspy.LMRequest) -> dspy.LMResponse:
- ...
-```
-
-!!! note "Status"
- This is a migration plan for the DSPy 3.3–3.6/4.0 series. Names and exact release timing may change before implementation lands, but the staged compatibility plan below should guide discussion.
-
-!!! info "Community feedback wanted"
- This plan mostly affects custom LMs and adapters. If you maintain one, please review the proposed one-line `forward_contract` migration and share feedback before the default LM path changes.
-
-## Who is affected?
-
-| Group | What to do now | Future requirement |
+| Interface | DSPy 3.4 | DSPy 3.5 |
| --- | --- | --- |
-| Most DSPy users | Nothing required. Optionally try the direct `lm(...)` API with `dspy.context(experimental=True)` and provide feedback. | DSPy programs will keep working before, during, and after this migration without user changes. |
-| Existing custom LM authors | Nothing required in 3.3. If you want to be explicit, add `forward_contract = "legacy"`. | Add an explicit `forward_contract`; eventually migrate to `forward_contract = "typed_lm"` before legacy support is removed. |
-| New custom LM authors | Use `forward_contract = "typed_lm"` and implement `forward(request: dspy.LMRequest) -> dspy.LMResponse`. | No later migration needed if you start with the typed contract. |
-| Custom adapter authors | Call `lm(...)`, not `lm.forward(...)`. | Build `LMRequest` objects and parse `LMResponse` directly. |
-
-
-## Background
-
-Today, `BaseLM` subclasses implement an untyped forward method, with a few optional parameters:
-
-```python
-def forward(self, prompt=None, messages=None, **kwargs):
- ...
-```
-
-That hook usually receives OpenAI/LiteLLM-shaped inputs and returns an OpenAI-like provider response. DSPy then post-processes that response into a `list[str | dict]` containing outputs.
-
-Because the current parameters are untyped, it is hard to know inside an LM exactly which inputs you will get and what types they will contain. The new contract is typed and provider-neutral. We have designed `LMRequest` and `LMResponse` to be flexible enough for LMs backed by many different provider APIs:
-
-```python
-def forward(self, request: dspy.LMRequest) -> dspy.LMResponse:
- ...
-```
-
-DSPy has settled on the internal LM type system around `LMRequest`, `LMResponse`, typed messages, parts, config, usage, and stream events. These types should be treated as the stable direction for LM implementations. Concrete LMs translate between these DSPy types and their provider API.
-
-## Why this matters
-
-The typed boundary gives DSPy one clear internal representation for LM calls:
-
-```text
-LMRequest -> BaseLM -> LMResponse
+| `lm("Hello")` | List-returning convenience | Kept as a convenience over the canonical engine path |
+| `lm(Request(...))` / `await lm.acall(Request(...))` | Returns an lm15 `Response` | The adapter and integration contract |
+| OpenAI-style `lm(messages=[{"role": ..., "content": ...}])` | Deprecated; use an explicit lm15 request | Removed |
+| Custom `BaseLM.forward()` / `aforward()` implementations | Deprecated; migrate to an engine | Removed as an integration interface |
+| `LegacyEngine` / `AsyncLegacyEngine` wrappers | Deprecated transition tools | Removed |
+| Custom-engine `complete_legacy()` shortcuts | Deprecated | Removed |
+
+In 3.5, DSPy's adapters will build lm15 requests and consume lm15 responses directly,
+not use the list-returning convenience path. Provider-wire dictionaries belong
+inside engines. LiteLLM itself is **not** deprecated: a LiteLLM engine must follow
+that same request/response contract.
+
+These changes do not happen silently in 3.4. Deprecated interfaces still execute;
+`DeprecationWarning` announces the cutoff and links to the replacement. Python may
+hide these warnings by default. To review them during development, use:
+
+```sh
+python -W default::DeprecationWarning your_program.py
```
-That gives DSPy and the community:
-
-- cleaner custom LM implementations,
-- less OpenAI/LiteLLM-shaped logic inside adapters,
-- first-class support for multimodal inputs, tool calls, reasoning, citations, usage, and provider metadata,
-- a more expressive direct `lm(...)` UX,
-- a clearer path for community packages to ship LMs that feel and are treated like first-class DSPy LMs.
-
-The migration is staged so existing code keeps working while new code can opt into the typed path.
+Normal DSPy program calls do not warn merely because built-in adapters still use
+an internal dictionary boundary in 3.4. That implementation is DSPy's migration
+responsibility. Using a legacy custom LM still warns, including through a program.
-## Guide for DSPy users
+## Ordinary programs
-Most users should not need to change anything in 3.3.
-
-Default behavior remains legacy:
+Keep using ordinary calls and DSPy modules:
```python
-outputs = lm("hello")
-# list[str | dict]
-```
+import dspy
-To try the typed LM API in 3.3, use the existing experimental switch:
-
-```python
-with dspy.context(experimental=True):
- response = lm("hello")
- print(response.text)
+lm = dspy.LM("openai/gpt-4o-mini")
+outputs = lm("Hello")
```
-Typed responses carry structured data:
+Ordinary calls return lists of strings or dictionaries. `experimental=True` no
+longer changes that return type; it still controls other experimental features.
+For multi-turn calls, use [explicit requests](#explicit-requests) rather than
+OpenAI-style message dictionaries.
+
+Engine selection:
+
+- `engine="auto"` (default): prefer native lm15 for supported routes and
+ representable inputs. Unsupported routes and ordinary provider-specific inputs
+ that cannot be represented faithfully select LiteLLM **before execution**.
+- `engine="lm15"`: require the native backend. Unsupported mappings raise.
+- `engine="litellm"`: explicitly use the compatibility backend.
+
+Authentication failures, timeouts and provider errors never cause a switch to
+another backend. Native capability errors also raise rather than dropping the
+requested feature. Text completions and client settings not implemented by the
+native integration remain on LiteLLM.
+
+### Caching, retries, usage and streaming
+
+DSPy owns response caching, retries, candidate fan-out, callbacks and history.
+Ordinary calls retain the old cache-key format and can read existing SDK response
+entries. Cache reads happen before engine construction. New native entries store
+plain serialized data and work with restricted cache deserialization. No old
+cache rewrite is required. Explicit typed calls have a separate cache namespace.
+
+Native `n` answers use `n` separate requests in sequence; LiteLLM compatibility
+calls retain the backend's native `n` behavior. Separate requests can take longer
+and bill input tokens more than once. A failed candidate does not restart earlier
+successful candidates, and incomplete sets are not cached. Bounded parallel
+execution is a follow-up tracked beside the candidate loops; it can reduce latency,
+but does not remove the input-token charges for separate requests.
+
+`num_retries` counts additional attempts, with exponential delays of 1, 2, 4…
+seconds, capped at 60 seconds unless the provider specifies `retry_after`.
+Backend retries are disabled on the managed LiteLLM path. A stream is not retried
+once a chunk has been sent to the caller. Caching a completed stream does not
+replay fake token timing on a later ordinary cache hit.
+
+`dspy.streamify` continues to use the existing listeners. Native events are
+assembled into an lm15 response and adapted to listener-facing chunks without
+mixing reasoning into answer text. LiteLLM's ordinary streaming chunks retain
+their original shape. Custom chunk consumers should not assume every native
+chunk is a LiteLLM class; its common fields remain available.
+
+`streamify` is the **program-level** API: it selects predictor fields, emits tool
+and module status messages, and delivers the final `Prediction`. lm15 events describe
+**one model call**, so they do not replace those responsibilities. The 3.5 migration
+will move the internal LM boundary to canonical requests/responses and events; it
+does not call for removing `streamify` or replacing its program-level role.
+
+Token usage is recorded once per public call, and cache hits add no billed usage.
+Native response counters stay provider-verbatim in `Response.usage`; DSPy's
+legacy prompt/completion counters are derived separately. A native cost estimate
+is **unknown**, not zero, when no reliable pricing is available. Existing cached
+SDK responses retain their historical cost metadata.
+
+## Migrating OpenAI-style messages
+
+The public `messages=` argument is deprecated, including provider SDK message
+objects as well as dictionaries. This does not deprecate `Request.messages`, which
+holds canonical lm15 `Message` objects. For example, this form still runs in 3.4:
```python
-response.text
-response.outputs
-response.usage
-response.cache_hit
-response.provider_data
+outputs = lm(messages=[
+ {"role": "system", "content": "Be concise."},
+ {"role": "user", "content": "What is DSPy?"},
+])
```
-The typed path also makes direct `lm(...)` calls more expressive. Strings, typed messages, media parts, previous responses, and explicit `LMRequest` objects all flow through one call API.
-
-!!! warning "Experimental 3.3 API"
- The typed LM symbols are importable without `experimental=True`. In DSPy 3.3, direct typed `lm(...)` calls are available behind `dspy.context(experimental=True)` while the API settles. Key helpers such as `dspy.LMRequest`, `dspy.LMResponse`, `dspy.System`, `dspy.User`, `dspy.Assistant`, `dspy.ToolCall`, and `dspy.ToolResult` are available at the top level. The complete typed LM vocabulary is available under `dspy.core.types`, e.g. `dspy.core.types.LMTextPart` and `dspy.core.types.LMImagePart`.
+Use the explicit request below instead. Put system instructions in `Request.system`,
+conversation turns in lm15 `Message` objects, and generation options in `Config`.
+The result is a `Response`, not a list: read `response.text`, `response.message`, or
+`response.tool_calls` as appropriate. Async calls use the same types.
-Multimodal request with instructions:
+## Explicit requests
```python
-from dspy.core.types import LMImagePart
-
-with dspy.context(experimental=True):
- response = lm(
- dspy.System("Be concise."),
- dspy.User("Describe this image.", dspy.Image("https://example.com/dog.png")), #Coming soon!
- temperature=0.2,
- )
-```
-
-Multi-turn conversation:
-
-```python
-with dspy.context(experimental=True):
- response = lm(
- dspy.User("What is DSPy?"),
- dspy.Assistant("DSPy is a framework for programming LM pipelines."),
- dspy.User("Say that in five words."),
- )
-```
-
-Tool-call transcript:
+from dspy.lm15 import Config, Message, Request
-```python
-with dspy.context(experimental=True):
- response = lm(
- dspy.User("What is the weather in Paris?"),
- dspy.Assistant(dspy.ToolCall(id="call_1", name="get_weather", args={"city": "Paris"})),
- dspy.ToolResult('{"temperature": "22 C"}', call_id="call_1", name="get_weather"),
- dspy.User("Summarize the result."),
- )
-```
-
-Passing a previous response back into the conversation:
-
-```python
-with dspy.context(experimental=True):
- first = lm("Explain DSPy in one sentence.")
- follow_up = lm(
- dspy.User("Explain DSPy in one sentence."),
- first,
- dspy.User("Now make it even shorter."),
- )
-```
-
-## Guide for custom LM authors
-
-Custom LM authors should declare which `forward()` contract their class implements.
-
-Legacy LMs should add:
-
-```python
-class MyLegacyLM(dspy.BaseLM):
- forward_contract = "legacy"
-
- def forward(self, prompt=None, messages=None, **kwargs):
- ...
-```
-
-Typed LMs should add:
-
-```python
-class MyTypedLM(dspy.BaseLM):
- forward_contract = "typed_lm"
-
- def forward(self, request: dspy.LMRequest) -> dspy.LMResponse:
- ...
-```
-
-In DSPy 3.3, classes without an explicit `forward_contract` are treated as legacy for compatibility. In later releases, missing declarations will warn and then may become errors or change defaults.
-
-A minimal typed LM looks like this:
-
-```python
-class EchoLM(dspy.BaseLM):
- forward_contract = "typed_lm"
-
- def forward(self, request: dspy.LMRequest) -> dspy.LMResponse:
- return dspy.LMResponse.from_text("hello", model=request.model)
-```
-
-## Guide for custom adapter authors
-
-Adapters should call the LM object, not `forward()` directly.
-
-Preferred typed boundary:
-
-```python
-request = dspy.LMRequest.from_call(
+request = Request(
model=lm.model,
- messages=messages,
- **lm_kwargs,
+ system="Be concise.",
+ messages=(Message.user("What is DSPy?"),),
+ config=Config(max_tokens=200),
)
response = lm(request)
+print(response.text)
+# Async equivalent: response = await lm.acall(request)
```
-Avoid this in adapters:
+The request's model must match the LM. Generation options belong in its config;
+LM generation defaults are not added. Client configuration still comes from the
+LM. `lm(request, cache=False, rollout_id=...)` controls DSPy's response cache.
+`Config.cache` instead controls provider-side prompt caching.
-```python
-lm.forward(...)
-```
+One typed request returns one response containing one assistant message. Use
+ordinary calls for `n` answers. To continue a conversation, add `response.message`
+to the next request's messages. LiteLLM typed streaming currently supports Chat
+Completions only; the native Responses engine supports Responses streaming.
-`BaseLM.__call__()` is the compatibility boundary. It owns input normalization, choosing the legacy or typed `forward()` path, adapting legacy outputs into `LMResponse`, and preserving public return behavior unless `experimental=True` is enabled.
+## Errors and retry ownership
-During the transition, adapters may still convert `LMResponse` back to legacy parser inputs. The long-term direction is for adapters to parse `LMResponse` directly.
+**Engines report failures; DSPy owns retry and fallback policy.** New engines raise
+specific errors from `dspy.lm15`, the same bundled vocabulary as their requests and
+responses. Use these imports, not a separately installed lm15 package whose Python
+classes have different identities.
-## Version sequence
-
-| Version | Custom `BaseLM.forward` contract | Public `lm(...)` behavior | LiteLLM role |
-| --- | --- | --- | --- |
-| 3.3 | Missing `forward_contract` is treated as legacy. | Typed returns available only through `experimental=True` or explicit `LMRequest` calls. | Current `dspy.LM` LiteLLM path remains the default. |
-| 3.4 | Missing `forward_contract` is treated as legacy and warns. | Still requires `experimental=True` or explicit `LMRequest` while migration continues. | Native typed LMs become preferred where available; LiteLLM is used as a compatibility fallback. |
-| 3.5 | Require explicit contract or flip default after final review. | Typed path becomes default with a legacy escape hatch. | Native typed LMs remain preferred; LiteLLM is used as a compatibility fallback but may require manual installation. |
-| 3.6 or 4.0 | Remove the legacy `forward(prompt, messages, **kwargs)` implementation contract after final review. | `forward(request: LMRequest) -> LMResponse` is the only supported `BaseLM` implementation contract. | TBD whether the LiteLLM fallback remains. |
+```python
+from dspy.lm15 import RateLimitError
-The important distinction is that removing the legacy `BaseLM.forward(prompt, messages, **kwargs)` contract does not require removing LiteLLM. LiteLLM can continue as a typed compatibility implementation that accepts `LMRequest` internally and returns `LMResponse`.
+# In an engine, after recognizing the backend's actual rate-limit response:
+raise RateLimitError("Provider rate limit", provider="my-backend", retry_after=2.0)
+```
-Before changing the default, DSPy will give custom LM authors enough time to add one of:
+Do not return an error dictionary or a fake successful `Response`. LiteLLM engines
+translate their SDK's exception classes and documented provider codes into lm15
+errors. Native engines propagate lm15 errors. Arbitrary custom exceptions are not
+classified by words such as "network" or "timeout".
+
+At its owned engine/capability boundaries, DSPy translates canonical errors into its
+existing public error family. Applications can keep catching `dspy.LMError`,
+`dspy.LMAuthError`, `dspy.ContextWindowExceededError`, and their siblings. These
+public error names are **not** part of the 3.5 interface removal. Already-public
+DSPy errors, including those from legacy plugins, are preserved by identity.
+
+- `LMLockTimeoutError` identifies local credential-lock contention, not a provider
+ timeout or bad credential. It is transient and eligible for managed retries.
+- `LMStreamAssemblyError` identifies an incomplete or invalid stream. It is not
+ automatically retried; `partial` may hold salvageable content, not a successful
+ result. Unknown usage stays unknown.
+- Unknown engine failures become `LMUnexpectedError`, with the original exception
+ as `__cause__`. The canonical cause retains its exact lm15 code and SDK cause;
+ the public error retains useful request IDs, provider codes, retry hints, routing
+ diagnostics, and partial responses where available.
+- Wrong Python API arguments still raise `TypeError`/`ValueError` before execution.
+ Missing dependencies retain `ImportError`. Cancellation, keyboard interrupts,
+ and warnings promoted to errors are not converted into retryable LM failures.
+
+Only an engine attempt is retryable. Pricing, cache writes, history, and callbacks
+must not cause a completed generation to run again. Reported usage for completed
+candidates is retained when later work fails. Invalid or non-finite retry hints use
+normal backoff; valid provider hints are honored, including HTTP-date headers.
+A network retry can still repeat a request the provider already processed or billed:
+this is **not an exactly-once guarantee**. Use `num_retries=0` when replay is unsafe.
+
+All engine streams pass through the same guard: one leading start, a final end,
+and no events after completion. Raised errors and canonical `StreamErrorEvent`s
+both fail the call. Incomplete results are never cached as success. Cleanup errors
+do not replace an active failure or cancellation; secondary diagnostics are kept in
+`cleanup_errors` where the exception supports them.
+
+Adapter fallback is separate from generation retry. `ChatAdapter` may make one
+additional JSON-format call for an `AdapterParseError`, but never after visible
+stream output. Unexpected parser bugs and engine/setup failures propagate. JSON
+schema fallback is decided **before** the model call, not after a failed execution.
+
+## Custom engines and legacy plugins
+
+New custom backends implement the small engine interface:
```python
-forward_contract = "legacy"
-```
+from dspy.lm15 import Message, Response, Usage
-or:
+class EchoEngine:
+ def complete(self, request):
+ return Response(
+ id=None, model=request.model, message=Message.assistant("hello"),
+ finish_reason="stop", usage=Usage(),
+ )
-```python
-forward_contract = "typed_lm"
+lm = dspy.LM("custom/echo", engine=EchoEngine())
```
+
+Implement `stream(request)` yielding canonical lm15 events to support streaming.
+Supply `async_engine=` with `async complete(request)` and `stream(request)` returning
+an async iterator for async calls. DSPy does not silently run a sync backend on the
+async path. Engines must not add another DSPy cache or retry loop. Custom engines
+are caller-owned and are not closed by DSPy.
+
+Implementing custom LMs through `BaseLM.forward(prompt=None, messages=None, **kwargs)`
+or `aforward` is **deprecated**. Implement an engine instead, as shown above and in
+the [custom-engine tutorial](../tutorials/custom_lm_engines/index.md). The old subclass
+interface remains supported throughout DSPy **3.4** and is scheduled for removal in
+**3.5**, along with both legacy-engine wrappers and `complete_legacy()` shortcuts.
+
+Calls through that old interface emit a `DeprecationWarning`, subject to Python's
+normal warning filters. Existing plugins are still automatically wrapped by legacy
+engines in 3.4, preserving ordinary inputs and outputs. DSPy does not add caching or
+retries around automatically wrapped plugins because they may already own those
+behaviors. A legacy plugin's private streaming implementation remains its responsibility.
+
+The explicit `LegacyEngine` and `AsyncLegacyEngine` wrappers from
+`dspy.clients.engines` are **3.4 transition tools only**, not a permanent escape
+hatch. Constructing either wrapper emits a deprecation warning; both are scheduled
+for removal in 3.5. Migrate the underlying implementation, rather than only wrapping
+it. During the transition, when a plugin already handles caching or retries, disable
+those on the outer `dspy.LM` with `cache=False, num_retries=0` to avoid a second layer.
+
+Custom engines must implement `complete(Request) -> Response`, plus their declared
+async/streaming counterparts. If DSPy selects a custom engine's `complete_legacy()`
+shortcut for an ordinary call, it warns. That shortcut will not exist in 3.5.
+
+An existing `forward_contract="legacy"` declaration is harmless. The experimental
+`forward_contract="typed_lm"` contract is removed and rejected explicitly.
+
+`DummyLM` now uses a canonical engine but retains its scripted answer modes,
+reasoning option, adapter formatting, and uncached consumption of repeated calls.
+
+### Copying, saving and cleanup
+
+`lm.copy()` isolates DSPy's history, callbacks and kwargs while sharing runtime
+resources. A model or client-setting change selects a distinct native pool.
+Native async pools are separate per event loop. Close owned sync pools with
+`lm.close()` and this loop's async pools with `await lm.aclose()` after calls finish.
+Copies share ownership: closing one releases the shared pools, which can be
+recreated on subsequent calls.
+
+JSON LM state preserves named engine selections and excludes API keys. Arbitrary
+custom engines require custom `dump_state`/`load_state` methods instead of guessed
+reconstruction. Whole-program pickle saving excludes native transport pools and
+locks; they are recreated lazily after loading. Existing trusted-loading safeguards
+remain in force.
+
+## Breaking replacement of the experimental 3.3 types
+
+The old `dspy.core.types` import raises a migration error. Old type names are no
+longer exported from `dspy` or `dspy.core`. This is not a drop-in rename:
+
+| Old experimental API | New API |
+| --- | --- |
+| `dspy.LMRequest` | `dspy.lm15.Request` |
+| `dspy.LMResponse` | `dspy.lm15.Response` |
+| `dspy.LMMessage`, `dspy.LMConfig` | `dspy.lm15.Message`, `dspy.lm15.Config` |
+| `dspy.System(text)` | `Request(system=text, ...)` |
+| `dspy.User(text)`, `dspy.Assistant(text)`, `dspy.Developer(text)` | `Message.user(text)`, `Message.assistant(text)`, `Message.developer(text)` |
+| `dspy.ToolCall(id=..., name=..., args=...)` | `ToolCallPart(id=..., name=..., input=...)` |
+| `dspy.ToolResult(content, call_id=...)` | `Message.tool(call_id, content)` |
+| `LMTextPart`, `LMImagePart`, etc. | `TextPart`, `ImagePart`, etc., from `dspy.lm15` |
+| `response.outputs[0].parts` | `response.message.parts` |
+
+lm15 uses frozen dataclasses, not Pydantic models. Required identities, roles,
+nonempty messages, and media-source validation differ. File names, document
+citation settings and arbitrary part metadata do not all have direct canonical
+equivalents; ordinary compatibility calls retain those provider-specific inputs.
+
+Old pickled objects containing the removed classes are not automatically migrated.
+Load and export them in their original environment first. This is separate from
+ordinary provider-response caches and ordinary saved-program configuration.
+
+Custom adapters should migrate to `lm(request)` / `await lm.acall(request)` and
+parse the returned lm15 `Response`. In 3.5 this is the internal contract throughout
+DSPy: adapters must not pass OpenAI-style dictionaries or use the list-returning
+prompt convenience. Existing 3.4 adapters can continue running during migration,
+but must not depend on the removed `dspy.clients.openai_format` module.
+
+The full built-in adapter migration is scheduled for 3.5; the 3.4 implementation
+still renders dictionary messages and parses list outputs. The temporary internal
+warning marker is not a public API for custom adapters to use.
+
+DSPy's signature types (`Image`, `Audio`, `File`, `Tool`, `ToolCalls`, `History`,
+`Reasoning`, etc.) and error classes are not removed.
diff --git a/docs/docs/tutorials/cache/index.md b/docs/docs/tutorials/cache/index.md
index a10c4b709..5fb12eebd 100644
--- a/docs/docs/tutorials/cache/index.md
+++ b/docs/docs/tutorials/cache/index.md
@@ -10,7 +10,7 @@ DSPy's caching system is architected in three distinct layers:
2. **On-disk cache**: Leveraging `diskcache.FanoutCache`, this layer offers persistent storage for cached items.
3. **Prompt cache (Server-side cache)**: This layer is managed by the LLM service provider (e.g., OpenAI, Anthropic).
-While DSPy does not directly control the server-side prompt cache, it offers users the flexibility to enable, disable, and customize the in-memory and on-disk caches to suit their specific requirements.
+DSPy controls its in-memory and on-disk answer caches. Provider-side prompt caching is separate: DSPy can pass caching instructions, but the provider determines eligibility, retention, and billing.
## Using DSPy Cache
@@ -52,7 +52,48 @@ Total usage: {}
In addition to DSPy's built-in caching mechanism, you can leverage provider-side prompt caching offered by LLM providers like Anthropic and OpenAI. This feature is particularly useful when working with modules like `dspy.ReAct()` that send similar prompts repeatedly, as it reduces both latency and costs by caching prompt prefixes on the provider's servers.
-You can enable prompt caching by passing the `cache_control_injection_points` parameter to `dspy.LM()`. This works with supported providers like Anthropic and OpenAI. For more details on this feature, see the [LiteLLM prompt caching documentation](https://docs.litellm.ai/docs/tutorials/prompt_caching#configuration).
+### Native lm15 engines (3.4 development API)
+
+Pass an actual `dspy.lm15.CacheConfig` as `prompt_cache`. This small bridge is intended for advanced users and adapter authors:
+
+```python
+import dspy
+
+lm = dspy.LM(
+ "anthropic/claude-haiku-4-5",
+ engine="lm15",
+ cache=False, # disable DSPy's answer cache, not the provider's prompt cache
+ prompt_cache=dspy.lm15.CacheConfig(prefix="stable"),
+)
+dspy.configure(lm=lm, track_usage=True)
+program = dspy.Predict("question -> answer")
+first = program(question="What is the capital of France?")
+second = program(question="What is the capital of Germany?")
+second.get_lm_usage()
+```
+
+`prefix="stable"` asks lm15 to mark the reusable system/tool prefix where supported. Providers with automatic caching may need no marker. This does not select demonstrations automatically, and this short example may be below the provider's cache minimum. Savings require an eligible, identical prefix across requests; each changing input still gets a fresh answer.
+
+For ordinary calls, a call-time value overrides the LM default:
+
+```python
+lm("A new question", prompt_cache=dspy.lm15.CacheConfig(prefix="history"))
+lm("Another question", prompt_cache=None) # remove the configured caching hint
+lm_without_hint = lm.copy(prompt_cache=None)
+```
+
+Adapter authors can put the same object in `lm_kwargs["prompt_cache"]`. DSPy attaches it to the canonical request **after** reading ordinary messages/options. `prefix_until_index`, when used, refers to lm15's canonical message list, not the original OpenAI-shaped rows. Use explicit lm15 requests when you need precise control of that boundary.
+
+- `cache=True/False` still controls DSPy's answer cache. `prompt_cache=None` removes this bridge's hint; it does **not** promise to disable the provider's automatic caching.
+- Explicit typed `Request` calls use only their own `Config.cache`, without inheriting `LM.prompt_cache` defaults or accepting a call-time `prompt_cache` override.
+- The option works with native lm15 and canonical custom engines. An ordinary request that needs LiteLLM raises rather than silently dropping or translating the policy. Do not combine it with provider-shaped options such as `prompt_cache_key`; use `CacheConfig.key` instead.
+- The policy survives `copy()` and JSON LM-state saving/loading. Different policies have distinct DSPy answer-cache keys; calls without a policy retain their existing keys.
+- Cache reads/writes appear in usage/history where the provider reports them. Cache writes and longer retention may cost extra, and cost estimates may be unknown when the pricing metadata is incomplete.
+- No stored cache is created, refreshed, or deleted automatically. `CacheConfig.resource`, if supplied, must identify a resource you already manage yourself.
+
+### LiteLLM compatibility engine
+
+With `engine="litellm"`, use LiteLLM's own options, such as `cache_control_injection_points`, rather than `prompt_cache`. See the [LiteLLM prompt caching documentation](https://docs.litellm.ai/docs/tutorials/prompt_caching#configuration) for provider-specific support.
```python
import dspy
@@ -61,6 +102,7 @@ import os
os.environ["ANTHROPIC_API_KEY"] = "{your_anthropic_key}"
lm = dspy.LM(
"anthropic/claude-sonnet-4-5-20250929",
+ engine="litellm",
cache_control_injection_points=[
{
"location": "message",
diff --git a/docs/docs/tutorials/core_development/index.md b/docs/docs/tutorials/core_development/index.md
index 6736ccd88..5d5376537 100644
--- a/docs/docs/tutorials/core_development/index.md
+++ b/docs/docs/tutorials/core_development/index.md
@@ -4,6 +4,9 @@ This section covers essential DSPy features and best practices for professional
## Integration and Tooling
+### [Custom LM Engines](../custom_lm_engines/index.md)
+Connect a custom backend through the lm15 request/response contract. The worked example runs the Pi CLI and its tools from a DSPy program.
+
### [Use MCP in DSPy](../mcp/index.md)
Learn to integrate Model Context Protocol (MCP) with DSPy applications. This tutorial shows how to leverage MCP for enhanced context management and more sophisticated AI interactions.
diff --git a/docs/docs/tutorials/custom_lm_engines/index.md b/docs/docs/tutorials/custom_lm_engines/index.md
new file mode 100644
index 000000000..1c192c2f3
--- /dev/null
+++ b/docs/docs/tutorials/custom_lm_engines/index.md
@@ -0,0 +1,104 @@
+# Custom LM Engines
+
+In the DSPy 3.4 development API, you can supply your own execution engine to `dspy.LM`. The minimum synchronous interface is:
+
+```python
+class MyEngine:
+ def complete(self, request: dspy.lm15.Request) -> dspy.lm15.Response:
+ ...
+```
+
+DSPy formats the program's inputs, converts them to an lm15 request, calls your engine, and parses the answer into a `Prediction`. You do not need to subclass `BaseLM` or return an OpenAI SDK object.
+
+!!! warning "DSPy 3.5 cutoff"
+ Custom `BaseLM.forward()`/`aforward()` integrations, `LegacyEngine`/`AsyncLegacyEngine`, and `complete_legacy()` shortcuts are deprecated in 3.4 and scheduled for removal in 3.5. Implement the request/response engine contract shown here; a legacy wrapper does not extend the migration deadline. OpenAI-style `lm(messages=[...])` calls are also being removed. `lm("hello")` remains a list-returning convenience; adapters use `lm(Request(...))` and consume `Response` directly. See the [migration guide](../../community/normalized-lm-api-migration.md).
+
+Expected backend failures should raise specific errors from `dspy.lm15`, such as
+`AuthError` or `RateLimitError`. DSPy translates them into its public `LMError`
+family and owns retries. Unexpected exceptions retain their original cause and
+are not guessed to be retryable from their message text. See
+[errors and retry ownership](../../community/normalized-lm-api-migration.md#errors-and-retry-ownership).
+
+This tutorial wraps the [Pi CLI](https://pi.dev) as a custom engine. Pi keeps its normal system prompt and tools, so a DSPy program can ask it to inspect a repository. The entire agent run, including tool calls, becomes one DSPy LM response.
+
+## Prerequisites
+
+- A DSPy build with the custom `engine=` interface.
+- Pi installed, available as `pi` on your PATH, and authenticated.
+- A model available to your Pi account. The example uses `openai-codex` and `gpt-6-astra`; change those two CLI arguments if needed.
+- A local Git repository to inspect.
+
+!!! warning "Pi has filesystem and shell access"
+ This example keeps Pi's normal tools, settings, and discovered resources. They can execute commands and modify files. Asking it not to modify anything is an instruction, not a sandbox. Run only against a trusted repository and use an OS-level sandbox if isolation matters.
+
+## Define the engine
+
+Pi's print mode runs its tool loop and writes the final assistant text to stdout. Wrap that text in an lm15 response; no event-stream parsing is needed for this demo.
+
+Append DSPy's system instructions with `--append-system-prompt`. This preserves Pi's normal system prompt while telling it how to format the answer for DSPy.
+
+```python
+import subprocess
+import dspy
+from dspy.lm15 import Message, Response, Usage
+
+
+class PiEngine:
+ def complete(self, request):
+ result = subprocess.run(
+ ["pi", "--print", "--no-session",
+ "--provider", "openai-codex", "--model", "gpt-6-astra",
+ "--append-system-prompt", request.system or ""],
+ input=request.messages[-1].text,
+ text=True, capture_output=True, check=True, timeout=120,
+ )
+ return Response(
+ id=None, model=request.model,
+ message=Message.assistant(result.stdout.strip()),
+ finish_reason="stop", usage=Usage(),
+ )
+```
+
+`Usage()` means **usage was not reported to DSPy**, not that the run consumed zero tokens. This minimal wrapper does not aggregate Pi's token usage or costs across its tool loop.
+
+## Run a DSPy program
+
+Run these cells in a notebook whose working directory is your repository root. The last expression displays the prediction without `print()`:
+
+```python
+dspy.configure(
+ lm=dspy.LM("pi", engine=PiEngine(), cache=False, num_retries=0),
+ adapter=dspy.ChatAdapter(use_json_adapter_fallback=False),
+)
+
+program = dspy.Predict("question -> answer")
+program(
+ question="Find the largest tracked file by byte size in this repository. "
+ "Use a tool to check. Report its path and size; do not modify anything."
+)
+```
+
+Here `"pi"` is a DSPy model label for history; the engine's CLI arguments select the actual provider and model.
+
+An answer for the DSPy repository can look like:
+
+```text
+Prediction(
+ answer='Largest tracked file: docs/docs/tutorials/observability/mlflow_trace_ui_navigation.gif\nSize: 8,704,325 bytes.'
+)
+```
+
+The answer depends on your repository. Pi still uses tools, but print mode does not expose tool events; this is not incremental streaming.
+
+## Demo boundaries
+
+- **Minimal input handling:** forwards only the last user text message and a plain-text system prompt. Earlier messages, generation options, and DSPy-declared tools are not forwarded or validated. Use this demo without demonstrations, conversation history, or media. Pi's own tools remain enabled.
+- **Synchronous only:** no async counterpart or streaming implementation.
+- **No saved Pi session:** `--no-session` avoids transcript persistence, but Pi may still update its own settings or credentials.
+- **No automatic DSPy replay:** caching, LM retries, and adapter fallback are disabled for the demonstrated program. Pi's own retry and compaction settings still apply.
+- **No detailed telemetry:** usage, costs, and tool events are not exposed by print mode. This demo labels a successful process exit as `stop`; it cannot distinguish a token-limit finish. Use `--mode json` for authoritative finish reasons and accounting across every assistant turn.
+- **Normal Pi output:** extensions can affect stdout. For a controlled production protocol, use JSON/RPC mode and explicitly select trusted extensions.
+- **Buffered subprocess output:** stdout and stderr are collected in memory. The timeout limits the direct Pi process, but this demo does not manage an entire descendant process tree.
+- **System instructions in arguments:** the appended prompt is visible to local process inspection and subject to OS argument-length limits. Avoid sensitive system instructions in this minimal example.
+
+These are deliberate shortcuts for demonstrating the custom-engine interface, not a production subprocess backend. For a more faithful integration, preserve complete messages and provider metadata, implement cancellation and bounded output handling, and explicitly map or reject each generation option.
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
index e05869553..fd59857b9 100644
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -73,6 +73,7 @@ nav:
- Tools, Development, and Deployment:
- Overview: tutorials/core_development/index.md
- Use MCP in DSPy: tutorials/mcp/index.md
+ - Custom LM Engines: tutorials/custom_lm_engines/index.md
- Output Refinement: tutorials/output_refinement/best-of-n-and-refine.md
- Saving and Loading: tutorials/saving/index.md
- Cache: tutorials/cache/index.md
diff --git a/dspy/__init__.py b/dspy/__init__.py
index 418f4b518..609b2ef03 100644
--- a/dspy/__init__.py
+++ b/dspy/__init__.py
@@ -9,18 +9,6 @@ from dspy.predict.flex import Flex
from dspy.evaluate import Evaluate # isort: skip
from dspy.clients import * # isort: skip
from dspy.adapters import Adapter, ChatAdapter, JSONAdapter, XMLAdapter, TwoStepAdapter, Image, Audio, File, History, Type, Tool, ToolCalls, ToolCallResults, Code, Reasoning # isort: skip
-from dspy.core import ( # isort: skip
- Assistant,
- Developer,
- LMConfig,
- LMMessage,
- LMRequest,
- LMResponse,
- System,
- ToolCall,
- ToolResult,
- User,
-)
from dspy.primitives.sandbox_serializable import SandboxSerializable # isort: skip
from dspy.utils.exceptions import (
AdapterParseError,
@@ -31,10 +19,12 @@ from dspy.utils.exceptions import (
LMConfigurationError,
LMError,
LMInvalidRequestError,
+ LMLockTimeoutError,
LMNotConfiguredError,
LMProviderError,
LMRateLimitError,
LMServerError,
+ LMStreamAssemblyError,
LMTimeoutError,
LMTransportError,
LMUnexpectedError,
diff --git a/dspy/_vendor/lm15-provenance.txt b/dspy/_vendor/lm15-provenance.txt
index 35bff7eed..eb4bfe548 100644
--- a/dspy/_vendor/lm15-provenance.txt
+++ b/dspy/_vendor/lm15-provenance.txt
@@ -1,5 +1,5 @@
source=https://github.com/cmpnd-ai/lm15-python.git
version=1.0.0a1
-commit=acb8090a100c41234300ee3adf5651c98b71fcb4
+commit=7896f82aa1b9d73401aa40f52ece350c3b2efdae
contract=e24b60e24b39a3e2b7bff8335bc84330b53bff39
-split=59e88ccd0f0395ed6da52d267c746b4315b0b407
+split=608cf26614f8581f7278efd242c5e8b2e23f6f2b
diff --git a/dspy/_vendor/lm15/errors.py b/dspy/_vendor/lm15/errors.py
index 1bafc5f56..b26c3b338 100644
--- a/dspy/_vendor/lm15/errors.py
+++ b/dspy/_vendor/lm15/errors.py
@@ -468,6 +468,7 @@ _CLASS_TO_CODE: dict[type[LM15Error], str] = {
AmbiguousModelError: "ambiguous_model",
TransportError: "transport",
LockTimeoutError: "lock_timeout",
+ StreamAssemblyError: "stream_assembly",
ProviderError: "provider",
}
diff --git a/dspy/_vendor/lm15/providers/async_base.py b/dspy/_vendor/lm15/providers/async_base.py
index 60faae70f..af6038761 100644
--- a/dspy/_vendor/lm15/providers/async_base.py
+++ b/dspy/_vendor/lm15/providers/async_base.py
@@ -46,7 +46,7 @@ from ..types import (
StreamEvent,
)
from .anthropic import AnthropicLM
-from .base import BaseProviderLM, Credential, HttpResponse, _attach_retry_after
+from .base import BaseProviderLM, Credential, HttpResponse, _attach_error_metadata
from .claude_code import DEFAULT_CLAUDE_CODE_VERSION, ClaudeCodeLM
from .gemini import GeminiLM
from .openai import OpenAILM
@@ -168,7 +168,7 @@ class AsyncBaseProviderLM:
error = self._inner.normalize_error(
resp.status, body.decode("utf-8", errors="replace")
)
- _attach_retry_after(error, resp.headers)
+ _attach_error_metadata(error, resp.headers)
raise error
async for raw in aparse_sse(_aiter_lines(resp)):
for event in self._inner.parse_stream_events(request, raw):
diff --git a/dspy/_vendor/lm15/providers/base.py b/dspy/_vendor/lm15/providers/base.py
index d3a61cc5c..469b1299a 100644
--- a/dspy/_vendor/lm15/providers/base.py
+++ b/dspy/_vendor/lm15/providers/base.py
@@ -1,6 +1,7 @@
from __future__ import annotations
import json
+import math
import os
from dataclasses import dataclass
from datetime import datetime
@@ -94,14 +95,14 @@ def resolve_credential(credential: Credential) -> str:
def _retry_after_seconds(value: str | None) -> float | None:
"""Parse an HTTP Retry-After header: delta-seconds or an HTTP-date."""
- if not value:
+ if value is None or value == "" or isinstance(value, bool):
return None
try:
seconds = float(value)
- except ValueError:
+ except (TypeError, ValueError, OverflowError):
pass
else:
- return seconds if seconds >= 0 else None
+ return seconds if math.isfinite(seconds) and seconds >= 0 else None
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
@@ -119,9 +120,11 @@ def _retry_after_seconds(value: str | None) -> float | None:
def _attach_retry_after(error: ProviderError, headers: "list[tuple[str, str]] | None") -> None:
"""Populate error.retry_after from a Retry-After header.
- A provider-body-derived value (set by the adapter's normalize_error)
- always wins; the header only fills the gap."""
- if getattr(error, "retry_after", None) is not None:
+ A valid provider-body-derived value wins; an invalid hint must never
+ become an infinite wait or replace the provider failure with a timer error."""
+ hint = _retry_after_seconds(getattr(error, "retry_after", None))
+ error.retry_after = hint
+ if hint is not None:
return
value = None
for key, val in headers or []:
@@ -133,6 +136,18 @@ def _attach_retry_after(error: ProviderError, headers: "list[tuple[str, str]] |
error.retry_after = seconds
+def _attach_error_metadata(error: ProviderError, headers: "list[tuple[str, str]] | None") -> None:
+ """Fill HTTP diagnostics absent from the body; never invent absent fields."""
+ _attach_retry_after(error, headers)
+ if error.request_id is not None:
+ return
+ values = {key.lower(): value for key, value in headers or []}
+ for name in ("x-request-id", "request-id", "x-amzn-requestid", "x-amz-request-id", "x-ms-request-id"):
+ if values.get(name):
+ error.request_id = values[name]
+ break
+
+
@dataclass(frozen=True, slots=True)
class HttpResponse:
"""Buffered provider-level HTTP response.
@@ -401,7 +416,7 @@ class BaseProviderLM:
error = self.normalize_error(
resp.status, body.decode("utf-8", errors="replace")
)
- _attach_retry_after(error, resp.headers)
+ _attach_error_metadata(error, resp.headers)
raise error
lines = resp.iter_lines() if hasattr(resp, "iter_lines") else _iter_lines(resp)
for raw in parse_sse(lines):
@@ -429,7 +444,7 @@ class BaseProviderLM:
def _http_error(self, response: HttpResponse) -> ProviderError:
"""Normalize an HTTP failure without losing its retry hint."""
error = self.normalize_error(response.status, response.text())
- _attach_retry_after(error, response.headers)
+ _attach_error_metadata(error, response.headers)
return error
def normalize_error(self, status: int, body: str) -> ProviderError:
diff --git a/dspy/_vendor/lm15/result.py b/dspy/_vendor/lm15/result.py
index beaffaa31..a9bb726e0 100644
--- a/dspy/_vendor/lm15/result.py
+++ b/dspy/_vendor/lm15/result.py
@@ -24,7 +24,7 @@ import json
from dataclasses import dataclass, field, replace
from typing import Any, AsyncIterator, Iterator
-from .errors import LM15Error, error_class_for_code
+from .errors import LM15Error, StreamAssemblyError, error_class_for_code
from .types import (
AudioDelta,
AudioPart,
@@ -289,12 +289,10 @@ class ResponseStream:
self._source_closed = False
self._event_iter = self._pump()
- def _close_source(self) -> None:
+ def _close_source(self, primary: BaseException | None = None) -> None:
if not self._source_closed:
self._source_closed = True
- close = getattr(self._source, "close", None)
- if close is not None:
- close()
+ _close_events(self._source, primary, self._response)
def close(self) -> None:
"""Stop reading and release the source, without draining it.
@@ -371,26 +369,30 @@ class ResponseStream:
return self._response
def _pump(self) -> Iterator[StreamEvent]:
+ primary = None
try:
for event in self._source:
- if event.type == "error":
- self._failure = _exception_from_error(event)
- raise self._failure
+ _check_terminal(event, self._response)
self._accumulator.push(event)
if event.type == "end":
self._response = self._accumulator.response()
- self._done = True
- yield event
- return
yield event
- self._response = self._accumulator.response()
+ if self._response is None:
+ raise _incomplete(self._accumulator)
except BaseException as exc:
- if self._failure is None and self._response is None:
- self._failure = exc
+ primary = _after_completion_error(exc, self._response)
+ if self._failure is None and not (isinstance(exc, GeneratorExit) and self._response is not None):
+ self._failure = primary
+ if primary is not exc:
+ raise primary from exc
raise
finally:
self._done = True
- self._close_source()
+ try:
+ self._close_source(primary)
+ except BaseException as exc:
+ self._failure = exc
+ raise
class AsyncResponseStream:
@@ -414,12 +416,10 @@ class AsyncResponseStream:
self._source_closed = False
self._event_gen = self._pump()
- async def _close_source(self) -> None:
+ async def _close_source(self, primary: BaseException | None = None) -> None:
if not self._source_closed:
self._source_closed = True
- close = getattr(self._source, "aclose", None)
- if close is not None:
- await close()
+ await _aclose_events(self._source, primary, self._response)
async def aclose(self) -> None:
"""Stop reading and release the source, without draining it."""
@@ -463,52 +463,132 @@ class AsyncResponseStream:
return self._response
async def _pump(self) -> AsyncIterator[StreamEvent]:
+ primary = None
try:
async for event in self._source:
- if event.type == "error":
- self._failure = _exception_from_error(event)
- raise self._failure
+ _check_terminal(event, self._response)
self._accumulator.push(event)
if event.type == "end":
self._response = self._accumulator.response()
- self._done = True
- yield event
- return
yield event
- self._response = self._accumulator.response()
+ if self._response is None:
+ raise _incomplete(self._accumulator)
except BaseException as exc:
- if self._failure is None and self._response is None:
- self._failure = exc
+ primary = _after_completion_error(exc, self._response)
+ if self._failure is None and not (isinstance(exc, GeneratorExit) and self._response is not None):
+ self._failure = primary
+ if primary is not exc:
+ raise primary from exc
raise
finally:
self._done = True
- await self._close_source()
+ try:
+ await self._close_source(primary)
+ except BaseException as exc:
+ self._failure = exc
+ raise
# ─── One-shot materialization ────────────────────────────────────────
def materialize_response(events: Iterator[StreamEvent], request: Request) -> Response:
- """Consume stream events and build a complete Response."""
+ """Consume a complete stream, requiring a final end event and closing its source."""
accumulator = StreamAccumulator(request)
- for event in events:
- if event.type == "error":
- raise _exception_from_error(event)
- accumulator.push(event)
- if event.type == "end":
- break
- return accumulator.response()
+ response = None
+ primary = None
+ try:
+ for event in events:
+ _check_terminal(event, response)
+ accumulator.push(event)
+ if event.type == "end":
+ response = accumulator.response()
+ if response is None:
+ raise _incomplete(accumulator)
+ return response
+ except BaseException as exc:
+ primary = _after_completion_error(exc, response)
+ if primary is not exc:
+ raise primary from exc
+ raise
+ finally:
+ _close_events(events, primary, response)
async def amaterialize_response(events: AsyncIterator[StreamEvent], request: Request) -> Response:
"""Async mirror of :func:`materialize_response`."""
accumulator = StreamAccumulator(request)
- async for event in events:
- if event.type == "error":
- raise _exception_from_error(event)
- accumulator.push(event)
- if event.type == "end":
- break
- return accumulator.response()
+ response = None
+ primary = None
+ try:
+ async for event in events:
+ _check_terminal(event, response)
+ accumulator.push(event)
+ if event.type == "end":
+ response = accumulator.response()
+ if response is None:
+ raise _incomplete(accumulator)
+ return response
+ except BaseException as exc:
+ primary = _after_completion_error(exc, response)
+ if primary is not exc:
+ raise primary from exc
+ raise
+ finally:
+ await _aclose_events(events, primary, response)
+
+
+def _check_terminal(event, response):
+ if response is not None:
+ raise StreamAssemblyError("Stream emitted an event after completion", partial=response)
+ if event.type == "error":
+ raise _exception_from_error(event)
+
+
+def _incomplete(accumulator):
+ try:
+ partial = accumulator.response()
+ except StreamAssemblyError as exc:
+ partial = exc.partial
+ except Exception:
+ partial = None
+ return StreamAssemblyError("Stream ended without a completion event", partial=partial)
+
+
+def _after_completion_error(exc, response):
+ if response is not None and isinstance(exc, Exception) and not isinstance(exc, (StreamAssemblyError, Warning)):
+ return StreamAssemblyError("Stream failed after response completion", partial=response)
+ return exc
+
+
+def _cleanup_failure(primary, cleanup, response):
+ if primary is None or isinstance(primary, GeneratorExit):
+ if response is not None:
+ raise StreamAssemblyError("Stream cleanup failed after response completion", partial=response) from cleanup
+ raise cleanup
+ try:
+ primary.cleanup_errors = (*getattr(primary, "cleanup_errors", ()), cleanup)
+ if hasattr(primary, "add_note"):
+ primary.add_note(f"Stream cleanup also failed ({type(cleanup).__name__}); see cleanup_errors.")
+ except Exception:
+ pass # Optional diagnostics must not replace the primary failure.
+
+
+def _close_events(source, primary=None, response=None):
+ try:
+ close = getattr(source, "close", None)
+ if close is not None:
+ close()
+ except Exception as cleanup:
+ _cleanup_failure(primary, cleanup, response)
+
+
+async def _aclose_events(source, primary=None, response=None):
+ try:
+ close = getattr(source, "aclose", None)
+ if close is not None:
+ await close()
+ except Exception as cleanup:
+ _cleanup_failure(primary, cleanup, response)
# ─── Conversion utilities ────────────────────────────────────────────
@@ -647,12 +727,14 @@ def coalesce_stream(
See docs/mapping-rules.md MAP-3 and MAP-4.
"""
+ primary = None
try:
yield from _coalesce_stream(events, model=model)
+ except BaseException as exc:
+ primary = exc
+ raise
finally:
- close = getattr(events, "close", None)
- if close is not None:
- close()
+ _close_events(events, primary)
def _coalesce_stream(events: Iterator[StreamEvent], *, model: str | None) -> Iterator[StreamEvent]:
@@ -702,13 +784,15 @@ async def acoalesce_stream(
StreamStartEvent for dialects without a start frame; duplicate starts
are dropped; error events never force a start.
"""
+ primary = None
try:
async for event in _acoalesce_stream(events, model=model):
yield event
+ except BaseException as exc:
+ primary = exc
+ raise
finally:
- close = getattr(events, "aclose", None)
- if close is not None:
- await close()
+ await _aclose_events(events, primary)
async def _acoalesce_stream(events: AsyncIterator[StreamEvent], *, model: str | None) -> AsyncIterator[StreamEvent]:
diff --git a/dspy/_vendor/lm15/transports/_async.py b/dspy/_vendor/lm15/transports/_async.py
index 747c16116..b299a901d 100644
--- a/dspy/_vendor/lm15/transports/_async.py
+++ b/dspy/_vendor/lm15/transports/_async.py
@@ -9,8 +9,8 @@ Mirror of _sync.py but async end-to-end. Key differences:
- Cancellation correctness: if the caller's task is cancelled mid-stream,
we must close the writer (not return it to the pool), then re-raise
CancelledError without awaiting anything that could itself be cancelled.
-- Timeouts use `asyncio.wait_for`. We avoid `asyncio.timeout` (3.11+) to
- stay Python 3.10-compatible.
+- Timeouts use a cancellation-safe wait helper, including on Python 3.10/3.11
+ where asyncio.wait_for can swallow simultaneous caller cancellation.
"""
from __future__ import annotations
@@ -36,6 +36,7 @@ from ._http11 import (
)
from ._proxy import ProxyRoute, connect_payload, proxy_route_for, route_origin
from ._ssl import make_ssl_context
+from ._timeouts import wait_for
from ._types import AsyncTransportResponse, TransportRequest
from ._url import ParsedURL, parse_url
@@ -115,7 +116,7 @@ class _AsyncConnectionPool:
await self._slot.acquire()
return
try:
- await asyncio.wait_for(self._slot.acquire(), timeout=timeout)
+ await wait_for(self._slot.acquire(), timeout=timeout, cancel_result=lambda _: self._slot.release())
except asyncio.TimeoutError:
raise TransportError("timed out waiting for connection pool slot")
@@ -294,7 +295,7 @@ class StdlibAsyncTransport:
return
while not decoder.complete:
try:
- data = await asyncio.wait_for(
+ data = await wait_for(
conn.reader.read(_READ_CHUNK),
timeout=read_timeout,
)
@@ -363,11 +364,12 @@ class StdlibAsyncTransport:
# gained start_tls only in 3.11).
tunnel = await self._connect_tunnel(parsed, proxy, timeout=connect_timeout)
try:
- reader, writer = await asyncio.wait_for(
+ reader, writer = await wait_for(
asyncio.open_connection(
sock=tunnel, ssl=ctx, server_hostname=parsed.host
),
timeout=connect_timeout,
+ cancel_result=lambda pair: pair[1].close(),
)
except asyncio.TimeoutError as exc:
tunnel.close()
@@ -383,7 +385,7 @@ class StdlibAsyncTransport:
connect_host = proxy.host if proxy is not None else parsed.host
connect_port = proxy.port if proxy is not None else parsed.port
try:
- reader, writer = await asyncio.wait_for(
+ reader, writer = await wait_for(
asyncio.open_connection(
host=connect_host,
port=connect_port,
@@ -391,6 +393,7 @@ class StdlibAsyncTransport:
server_hostname=parsed.host if ctx else None,
),
timeout=connect_timeout,
+ cancel_result=lambda pair: pair[1].close(),
)
except asyncio.TimeoutError as exc:
raise ConnectTimeout(
@@ -419,11 +422,12 @@ class StdlibAsyncTransport:
"""Open a raw socket to the proxy and CONNECT it to the TLS target."""
loop = asyncio.get_running_loop()
try:
- sock = await asyncio.wait_for(
+ sock = await wait_for(
asyncio.to_thread(
socket.create_connection, (proxy.host, proxy.port), timeout
),
timeout=timeout,
+ cancel_result=lambda sock: sock.close(),
)
except asyncio.TimeoutError as exc:
raise ConnectTimeout(
@@ -439,12 +443,12 @@ class StdlibAsyncTransport:
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
except OSError:
pass
- await asyncio.wait_for(
+ await wait_for(
loop.sock_sendall(sock, connect_payload(parsed, proxy)), timeout=timeout
)
parser = ResponseHeadParser()
while not parser.complete:
- data = await asyncio.wait_for(
+ data = await wait_for(
loop.sock_recv(sock, _READ_CHUNK), timeout=timeout
)
if not data:
@@ -504,7 +508,7 @@ class StdlibAsyncTransport:
conn.writer.write(head)
if body_length is not None and body:
conn.writer.write(body)
- await asyncio.wait_for(conn.writer.drain(), timeout=write_timeout)
+ await wait_for(conn.writer.drain(), timeout=write_timeout)
except asyncio.TimeoutError as exc:
raise WriteTimeout(f"write timed out: {exc}") from exc
except (BrokenPipeError, ConnectionResetError):
@@ -518,7 +522,7 @@ class StdlibAsyncTransport:
parser = ResponseHeadParser()
while not parser.complete:
try:
- data = await asyncio.wait_for(
+ data = await wait_for(
conn.reader.read(_READ_CHUNK), timeout=read_timeout
)
except asyncio.TimeoutError as exc:
diff --git a/dspy/_vendor/lm15/transports/_timeouts.py b/dspy/_vendor/lm15/transports/_timeouts.py
new file mode 100644
index 000000000..6e0b0645b
--- /dev/null
+++ b/dspy/_vendor/lm15/transports/_timeouts.py
@@ -0,0 +1,53 @@
+"""Cancellation-safe timeouts on every supported Python version.
+
+Python 3.10/3.11 asyncio.wait_for can swallow caller cancellation when its
+inner future completes concurrently (CPython #86296). Socket reads/writes
+must not continue after that cancellation. Use asyncio.wait to distinguish
+our timeout from caller cancellation, then explicitly cancel and drain the
+owned operation. No global event-loop patches or third-party dependency.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Awaitable, Callable, TypeVar
+
+_T = TypeVar("_T")
+
+
+async def wait_for(
+ awaitable: Awaitable[_T], timeout: float | None,
+ *, cancel_result: Callable[[_T], None] | None = None,
+) -> _T:
+ """Wait with a timeout, never replacing caller cancellation with success.
+
+ Like asyncio.wait_for, timeout cancellation waits for the operation's
+ cleanup. An operation that suppresses cancellation can return a value;
+ an operation that suppresses it indefinitely can exceed the timeout.
+ cancel_result releases a resource acquired concurrently with cancellation
+ (a socket, or a semaphore permit) before its caller took ownership.
+ """
+ if timeout is None:
+ return await awaitable
+ task = asyncio.ensure_future(awaitable)
+ try:
+ if timeout > 0:
+ done, _ = await asyncio.wait((task,), timeout=timeout)
+ if done:
+ return task.result()
+ elif task.done():
+ return task.result()
+ task.cancel()
+ # return_exceptions consumes only the child's cancellation. A caller
+ # cancellation still raises here and reaches the outer handler.
+ await asyncio.gather(task, return_exceptions=True)
+ try:
+ return task.result()
+ except asyncio.CancelledError as exc:
+ raise asyncio.TimeoutError() from exc
+ except BaseException:
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+ if cancel_result is not None and not task.cancelled() and task.exception() is None:
+ cancel_result(task.result())
+ raise
diff --git a/dspy/adapters/_legacy_type_markers.py b/dspy/adapters/_legacy_type_markers.py
index 3a3a788ac..906f65f84 100644
--- a/dspy/adapters/_legacy_type_markers.py
+++ b/dspy/adapters/_legacy_type_markers.py
@@ -9,7 +9,6 @@ from typing import Any
import json_repair
from dspy.adapters.types.base_type import CUSTOM_TYPE_END_IDENTIFIER, CUSTOM_TYPE_START_IDENTIFIER
-from dspy.core.types import LMAudioPart, LMBinaryPart, LMDocumentPart, LMImagePart, LMMessage, LMPart, LMTextPart
def _expand_legacy_custom_type_markers_in_chat_message(message: dict[str, Any]) -> dict[str, Any]:
@@ -22,21 +21,6 @@ def _expand_legacy_custom_type_markers_in_chat_message(message: dict[str, Any])
return {**message, "content": _split_legacy_custom_type_text_to_blocks(content)}
-def _expand_legacy_custom_type_markers_in_lm_message(message: LMMessage) -> LMMessage:
- """Expand legacy marker payloads in user `LMMessage`s into normalized parts."""
- if message.role != "user":
- return message
- expanded_parts: list[LMPart] = []
- changed = False
- for part in message.parts:
- if not isinstance(part, LMTextPart) or CUSTOM_TYPE_START_IDENTIFIER not in part.text:
- expanded_parts.append(part)
- continue
- changed = True
- expanded_parts.extend(_split_legacy_custom_type_text_to_parts(part.text))
- return message.model_copy(update={"parts": expanded_parts}) if changed else message
-
-
def _split_legacy_custom_type_text_to_blocks(text: str) -> list[dict[str, Any]]:
blocks: list[dict[str, Any]] = []
last_end = 0
@@ -51,20 +35,6 @@ def _split_legacy_custom_type_text_to_blocks(text: str) -> list[dict[str, Any]]:
return blocks
-def _split_legacy_custom_type_text_to_parts(text: str) -> list[LMPart]:
- parts: list[LMPart] = []
- last_end = 0
- for match in re.finditer(_marker_pattern(), text, re.DOTALL):
- start, end = match.span()
- if start > last_end:
- parts.append(LMTextPart(text=text[last_end:start]))
- parts.extend(_legacy_custom_type_payload_to_parts(match.group(1).strip()))
- last_end = end
- if last_end < len(text):
- parts.append(LMTextPart(text=text[last_end:]))
- return parts
-
-
def _legacy_custom_type_payload_to_blocks(payload: str) -> list[dict[str, Any]]:
parsed = _parse_legacy_payload(payload)
if isinstance(parsed, list):
@@ -72,13 +42,6 @@ def _legacy_custom_type_payload_to_blocks(payload: str) -> list[dict[str, Any]]:
return [{"type": "text", "text": payload}]
-def _legacy_custom_type_payload_to_parts(payload: str) -> list[LMPart]:
- parsed = _parse_legacy_payload(payload)
- if not isinstance(parsed, list):
- return [LMTextPart(text=payload)]
- return [_legacy_content_block_to_lm_part(block) for block in parsed]
-
-
def _parse_legacy_payload(payload: str) -> Any:
for parse_fn in (json.loads, _parse_doubly_quoted_json, json_repair.loads):
try:
@@ -94,46 +57,6 @@ def _parse_doubly_quoted_json(value: str) -> Any:
return json.loads(json.loads(f'"{value}"'))
-def _legacy_content_block_to_lm_part(block: Any) -> LMPart:
- if not isinstance(block, dict):
- return LMTextPart(text=str(block))
-
- block_type = block.get("type")
- if block_type == "text":
- return LMTextPart(text=block.get("text", ""))
- if block_type == "image_url":
- image_url = block.get("image_url", {})
- source = image_url.get("url") if isinstance(image_url, dict) else image_url
- if isinstance(source, str) and source.startswith("data:") and "," in source:
- media_type, data = _split_data_uri(source)
- return LMImagePart(data=data, media_type=media_type)
- return LMImagePart(url=source or "")
- if block_type == "input_audio":
- audio = block.get("input_audio", {})
- return LMAudioPart(data=audio.get("data", ""), media_type=f"audio/{audio.get('format', 'wav')}")
- if block_type == "file":
- file = block.get("file", {})
- if file.get("file_data") is not None:
- media_type, data = _split_data_uri(file["file_data"])
- return LMBinaryPart(data=data, media_type=media_type, filename=file.get("filename"), metadata={"legacy_content_block": block})
- return LMBinaryPart(file_id=file.get("file_id", ""), filename=file.get("filename"), metadata={"legacy_content_block": block})
- if block_type == "document":
- source = block.get("source", {})
- return LMDocumentPart(
- source=source if isinstance(source, dict) else {"type": "text", "data": str(source)},
- citations=block.get("citations") or {"enabled": True},
- title=block.get("title"),
- context=block.get("context"),
- )
-
- # Keep unknown provider-shaped blocks losslessly for the current BaseLM
- # compatibility path. `openai_format.part_to_openai_blocks()` rehydrates
- # this metadata before the legacy provider-shaped call. A later normalized
- # type pass should replace this with an explicit opaque part or remove it
- # when marker-based custom type rendering is retired.
- return LMTextPart(text="", metadata={"legacy_content_block": block})
-
-
def _marker_pattern() -> str:
return rf"{CUSTOM_TYPE_START_IDENTIFIER}(.*?){CUSTOM_TYPE_END_IDENTIFIER}"
diff --git a/dspy/adapters/base.py b/dspy/adapters/base.py
index 0e1329de3..0da1b46fd 100644
--- a/dspy/adapters/base.py
+++ b/dspy/adapters/base.py
@@ -6,19 +6,14 @@ import json_repair
from dspy.adapters._legacy_type_markers import (
_expand_legacy_custom_type_markers_in_chat_message,
- _expand_legacy_custom_type_markers_in_lm_message,
)
from dspy.adapters.types import History, Type
from dspy.adapters.types.reasoning import Reasoning
from dspy.adapters.types.tool import Tool, ToolCallResults, ToolCalls
from dspy.adapters.utils import apply_output_field_defaults, serialize_for_json
+from dspy.clients._deprecation import adapter_message_call
from dspy.clients.base_lm import BaseLM
-from dspy.clients.openai_format import (
- legacy_outputs_from_lm_response,
- lm_response_from_legacy_outputs,
- to_openai_chat_request,
-)
-from dspy.core.types import LMMessage, LMRequest, LMResponse
+from dspy.clients.capabilities import with_capability_planning
from dspy.experimental import Citations
from dspy.signatures.field import InputField
from dspy.signatures.signature import Signature
@@ -142,11 +137,6 @@ class Adapter:
lm: BaseLM,
lm_kwargs: dict[str, Any],
) -> list[dict[str, Any]]:
- # TODO(adapters-plan): This still parses legacy adapter output objects.
- # PR1 normalizes the LM boundary, then immediately converts back to this
- # shape to avoid changing parser semantics. A later PR should parse
- # `LMResponse` directly and merge text-parsed fields with explicit
- # native fields from `_AdapterPlan`.
values = []
tool_call_output_field_name = self._get_tool_call_output_field_name(original_signature)
@@ -206,112 +196,7 @@ class Adapter:
return values
- def _render_request(
- self,
- lm: BaseLM,
- lm_kwargs: dict[str, Any],
- messages: list[LMMessage | dict[str, Any]],
- ) -> LMRequest:
- """Build the normalized LM request for the current adapter call path.
-
- TODO(adapters-plan): This currently receives already-rendered messages.
- Once planning lands, this should render from `_AdapterPlan` and apply
- planned message/part insertions before creating `LMRequest`.
- """
- return LMRequest.from_call(
- model=lm.model,
- messages=self._coerce_lm_messages(messages),
- **lm_kwargs,
- )
-
- def _call_lm(self, lm: BaseLM, request: LMRequest) -> LMResponse:
- """Call current `BaseLM` through the normalized request/response boundary.
-
- TODO(language-models): When `BaseLM` is replaced by/updated to the
- normalized `BaseLM.forward(request: LMRequest) -> LMResponse` contract,
- remove this compatibility shim and let adapters call the normalized LM
- entry point directly. The OpenAI-shaped compatibility kwargs should live
- only inside concrete LM backends.
- """
- data = self._legacy_call_kwargs(request)
- outputs = lm(messages=data.pop("messages"), **data)
- return self._normalize_legacy_outputs(outputs, request)
-
- async def _acall_lm(self, lm: BaseLM, request: LMRequest) -> LMResponse:
- """Async variant of `_call_lm`.
-
- TODO(language-models): Same transitional boundary as `_call_lm()`; this
- should eventually call a normalized async LM method directly.
- """
- data = self._legacy_call_kwargs(request)
- outputs = await lm.acall(messages=data.pop("messages"), **data)
- return self._normalize_legacy_outputs(outputs, request)
-
- def _legacy_call_kwargs(self, request: LMRequest) -> dict[str, Any]:
- # TODO(language-models): Current `BaseLM` expects OpenAI/LiteLLM-shaped
- # chat kwargs. We intentionally use `dspy.clients.openai_format` here so
- # the conversion code lives in the future LM/client layer, not in
- # adapters. Remove this adapter helper once `BaseLM` accepts `LMRequest`.
- data = to_openai_chat_request(request)
- data.pop("model", None)
- # TODO(language-models): `cache` and `rollout_id` are DSPy BaseLM
- # execution controls, not provider request fields. The future
- # normalized LM base should own them before provider-format conversion.
- if request.config.cache is not None:
- if request.config.cache.enabled is not None:
- data["cache"] = request.config.cache.enabled
- if request.config.cache.rollout_id is not None:
- data["rollout_id"] = request.config.cache.rollout_id
- return data
-
- def _coerce_lm_messages(self, messages: list[LMMessage | dict[str, Any]]) -> list[LMMessage]:
- """Normalize subclass `format()` output before the LM boundary.
-
- TODO(adapters-normalized-rendering): Adapter `format()` methods still
- return OpenAI-chat-shaped dictionaries. This coercion is the bridge until
- adapters render `LMMessage` / `LMPart` directly.
- """
- return [
- _expand_legacy_custom_type_markers_in_lm_message(
- message if isinstance(message, LMMessage) else self._chat_dict_to_lm_message(message)
- )
- for message in messages
- ]
-
- def _chat_dict_to_lm_message(self, message: dict[str, Any]) -> LMMessage:
- try:
- return LMMessage(**message)
- except Exception:
- # TODO(legacy-custom-types): Unknown OpenAI content blocks are
- # temporarily preserved as `legacy_content_block` metadata on an
- # empty text part so `openai_format` can round-trip them back to
- # current BaseLM calls. Replace this with either explicit opaque
- # provider parts or remove it when marker-based custom type
- # serialization is retired.
- message = dict(message)
- content = message.get("content")
- if isinstance(content, list):
- sanitized = []
- supported = {"text", "image_url", "input_audio", "file", "document", "video"}
- for block in content:
- if isinstance(block, dict) and block.get("type") in supported:
- sanitized.append(block)
- elif isinstance(block, dict):
- sanitized.append({"type": "text", "text": "", "metadata": {"legacy_content_block": block}})
- else:
- sanitized.append({"type": "text", "text": json.dumps(block, ensure_ascii=False)})
- message["content"] = sanitized
- return LMMessage(**message)
-
- def _normalize_legacy_outputs(self, outputs: list[dict[str, Any] | str | None], request: LMRequest) -> LMResponse:
- """Convert current `BaseLM` outputs into a normalized `LMResponse`.
-
- TODO(language-models): Current `BaseLM` returns `list[str | dict | None]`.
- Future LMs should return `LMResponse` directly, making this method a
- compatibility-only path for old/custom LMs.
- """
- return lm_response_from_legacy_outputs(outputs, request)
-
+ @with_capability_planning
def __call__(
self,
lm: BaseLM,
@@ -338,15 +223,14 @@ class Adapter:
"""
processed_signature = self._call_preprocess(lm, lm_kwargs, signature, inputs)
messages = self.format(processed_signature, demos, inputs)
- request = self._render_request(lm, lm_kwargs, messages)
- response = self._call_lm(lm, request)
- # TODO(adapters-response): We normalize at the LM boundary, but still
- # convert back to legacy postprocess dictionaries here to keep this PR
- # behavior-preserving. Replace with direct `LMResponse` parsing once the
- # explicit adapter plan exists.
- outputs = legacy_outputs_from_lm_response(response)
+ if lm_kwargs.get("parallel_tool_calls") is not None:
+ lm_kwargs.setdefault("tool_choice", "auto")
+ # TODO(3.5): build Request and parse Response directly; remove this marker.
+ with adapter_message_call(lm, messages):
+ outputs = lm(messages=messages, **lm_kwargs)
return self._call_postprocess(processed_signature, signature, outputs, lm, lm_kwargs)
+ @with_capability_planning
async def acall(
self,
lm: BaseLM,
@@ -357,11 +241,11 @@ class Adapter:
) -> list[dict[str, Any]]:
processed_signature = self._call_preprocess(lm, lm_kwargs, signature, inputs)
messages = self.format(processed_signature, demos, inputs)
- request = self._render_request(lm, lm_kwargs, messages)
- response = await self._acall_lm(lm, request)
- # TODO(adapters-response): Keep in sync with `__call__()` until both use
- # direct `LMResponse` parsing.
- outputs = legacy_outputs_from_lm_response(response)
+ if lm_kwargs.get("parallel_tool_calls") is not None:
+ lm_kwargs.setdefault("tool_choice", "auto")
+ # TODO(3.5): share the canonical request/response boundary with __call__.
+ with adapter_message_call(lm, messages):
+ outputs = await lm.acall(messages=messages, **lm_kwargs)
return self._call_postprocess(processed_signature, signature, outputs, lm, lm_kwargs)
def format(
diff --git a/dspy/adapters/chat_adapter.py b/dspy/adapters/chat_adapter.py
index 06d2f90ed..a31663c87 100644
--- a/dspy/adapters/chat_adapter.py
+++ b/dspy/adapters/chat_adapter.py
@@ -15,9 +15,10 @@ from dspy.adapters.utils import (
translate_field_type,
)
from dspy.clients.base_lm import BaseLM
+from dspy.clients.call_context import adapter_fallback_scope
from dspy.signatures.signature import Signature
from dspy.utils.callback import BaseCallback
-from dspy.utils.exceptions import AdapterParseError, LMError
+from dspy.utils.exceptions import AdapterParseError
field_header_pattern = re.compile(r"\[\[ ## (\w+) ## \]\]")
@@ -52,9 +53,9 @@ class ChatAdapter(Adapter):
callbacks: List of callback functions to execute during adapter methods.
use_native_function_calling: Whether to enable native function calling capabilities.
native_response_types: List of output field types handled by native LM features.
- use_json_adapter_fallback: Whether to automatically fallback to JSONAdapter if the ChatAdapter fails.
- If True, when an error occurs (except ContextWindowExceededError), the adapter will retry using
- JSONAdapter. Defaults to True.
+ use_json_adapter_fallback: Whether to try JSONAdapter after an AdapterParseError.
+ Only invalid model output can trigger this extra call, and never after visible stream output.
+ Configuration errors, engine failures and programming bugs propagate. Defaults to True.
parallel_tool_calls: Whether to request provider-side parallel tool-call generation when native function
calling is active. If None, the adapter does not set the provider option.
"""
@@ -82,17 +83,15 @@ class ChatAdapter(Adapter):
demos: list[dict[str, Any]],
inputs: dict[str, Any],
) -> list[dict[str, Any]]:
- try:
- return super().__call__(lm, lm_kwargs, signature, demos, inputs)
- except Exception as e:
- # fallback to JSONAdapter
- from dspy.adapters.json_adapter import JSONAdapter
-
- if isinstance(e, LMError) or isinstance(self, JSONAdapter) or not self.use_json_adapter_fallback:
- # On LM errors, already using JSONAdapter, or use_json_adapter_fallback is False, we don't want to
- # retry with a different adapter. Raise the original error instead of the fallback error.
- raise
- return self._make_json_adapter_fallback()(lm, lm_kwargs, signature, demos, inputs)
+ with adapter_fallback_scope() as progress:
+ try:
+ return super().__call__(lm, lm_kwargs, signature, demos, inputs)
+ except AdapterParseError:
+ from dspy.adapters.json_adapter import JSONAdapter
+
+ if progress.emitted or isinstance(self, JSONAdapter) or not self.use_json_adapter_fallback:
+ raise
+ return self._make_json_adapter_fallback()(lm, lm_kwargs, signature, demos, inputs)
async def acall(
self,
@@ -102,17 +101,15 @@ class ChatAdapter(Adapter):
demos: list[dict[str, Any]],
inputs: dict[str, Any],
) -> list[dict[str, Any]]:
- try:
- return await super().acall(lm, lm_kwargs, signature, demos, inputs)
- except Exception as e:
- # fallback to JSONAdapter
- from dspy.adapters.json_adapter import JSONAdapter
-
- if isinstance(e, LMError) or isinstance(self, JSONAdapter) or not self.use_json_adapter_fallback:
- # On LM errors, already using JSONAdapter, or use_json_adapter_fallback is False, we don't want to
- # retry with a different adapter. Raise the original error instead of the fallback error.
- raise
- return await self._make_json_adapter_fallback().acall(lm, lm_kwargs, signature, demos, inputs)
+ with adapter_fallback_scope() as progress:
+ try:
+ return await super().acall(lm, lm_kwargs, signature, demos, inputs)
+ except AdapterParseError:
+ from dspy.adapters.json_adapter import JSONAdapter
+
+ if progress.emitted or isinstance(self, JSONAdapter) or not self.use_json_adapter_fallback:
+ raise
+ return await self._make_json_adapter_fallback().acall(lm, lm_kwargs, signature, demos, inputs)
def format_field_description(self, signature: type[Signature]) -> str:
return (
@@ -236,7 +233,7 @@ class ChatAdapter(Adapter):
if (k not in fields) and (k in signature.output_fields):
try:
fields[k] = parse_value(v, signature.output_fields[k].annotation)
- except Exception as e:
+ except ValueError as e:
raise AdapterParseError(
adapter_name="ChatAdapter",
signature=signature,
diff --git a/dspy/adapters/json_adapter.py b/dspy/adapters/json_adapter.py
index 286f3e4ac..d665be0e7 100644
--- a/dspy/adapters/json_adapter.py
+++ b/dspy/adapters/json_adapter.py
@@ -18,9 +18,10 @@ from dspy.adapters.utils import (
translate_field_type,
)
from dspy.clients.base_lm import BaseLM
+from dspy.clients.capabilities import with_capability_planning
from dspy.signatures.signature import Signature, SignatureMeta
from dspy.utils.callback import BaseCallback
-from dspy.utils.exceptions import AdapterParseError, LMError
+from dspy.utils.exceptions import AdapterParseError
logger = logging.getLogger(__name__)
@@ -52,19 +53,22 @@ class JSONAdapter(ChatAdapter):
parallel_tool_calls=parallel_tool_calls,
)
- def _json_adapter_call_common(self, lm, lm_kwargs, signature, demos, inputs, call_fn):
- """Common call logic to be used for both sync and async calls."""
+ def _prepare_response_format(self, lm, lm_kwargs, signature):
+ """Choose a format before execution; schema fallback never encloses an LM call."""
if "response_format" not in lm.supported_params:
- return call_fn(lm, lm_kwargs, signature, demos, inputs)
-
+ return
has_tool_calls = any(field.annotation == ToolCalls for field in signature.output_fields.values())
-
if _has_open_ended_mapping(signature) or (not self.use_native_function_calling and has_tool_calls) or not lm.supports_response_schema:
- # We found that structured output mode doesn't work well with dspy.ToolCalls as output field.
- # So we fall back to json mode if native function calling is disabled and ToolCalls is present.
lm_kwargs["response_format"] = {"type": "json_object"}
- return call_fn(lm, lm_kwargs, signature, demos, inputs)
+ return
+ try:
+ format_ = _get_structured_outputs_response_format(signature, self.use_native_function_calling)
+ except (ValueError, pydantic.PydanticInvalidForJsonSchema, pydantic.PydanticSchemaGenerationError):
+ logger.warning("Failed to build structured output schema; using JSON mode before execution.")
+ format_ = {"type": "json_object"}
+ lm_kwargs["response_format"] = format_
+ @with_capability_planning
def __call__(
self,
lm: BaseLM,
@@ -73,25 +77,10 @@ class JSONAdapter(ChatAdapter):
demos: list[dict[str, Any]],
inputs: dict[str, Any],
) -> list[dict[str, Any]]:
- result = self._json_adapter_call_common(lm, lm_kwargs, signature, demos, inputs, super().__call__)
- if result:
- return result
-
- try:
- structured_output_model = _get_structured_outputs_response_format(
- signature, self.use_native_function_calling
- )
- lm_kwargs["response_format"] = structured_output_model
- return super().__call__(lm, lm_kwargs, signature, demos, inputs)
- except LMError:
- # Provider/backend failures should propagate; the fallback below is only for local structured-output
- # setup/schema failures where retrying in JSON mode is appropriate.
- raise
- except Exception:
- logger.warning("Failed to use structured output format, falling back to JSON mode.")
- lm_kwargs["response_format"] = {"type": "json_object"}
- return super().__call__(lm, lm_kwargs, signature, demos, inputs)
+ self._prepare_response_format(lm, lm_kwargs, signature)
+ return super().__call__(lm, lm_kwargs, signature, demos, inputs)
+ @with_capability_planning
async def acall(
self,
lm: BaseLM,
@@ -100,24 +89,8 @@ class JSONAdapter(ChatAdapter):
demos: list[dict[str, Any]],
inputs: dict[str, Any],
) -> list[dict[str, Any]]:
- result = self._json_adapter_call_common(lm, lm_kwargs, signature, demos, inputs, super().acall)
- if result:
- return await result
-
- try:
- structured_output_model = _get_structured_outputs_response_format(
- signature, self.use_native_function_calling
- )
- lm_kwargs["response_format"] = structured_output_model
- return await super().acall(lm, lm_kwargs, signature, demos, inputs)
- except LMError:
- # Provider/backend failures should propagate; the fallback below is only for local structured-output
- # setup/schema failures where retrying in JSON mode is appropriate.
- raise
- except Exception:
- logger.warning("Failed to use structured output format, falling back to JSON mode.")
- lm_kwargs["response_format"] = {"type": "json_object"}
- return await super().acall(lm, lm_kwargs, signature, demos, inputs)
+ self._prepare_response_format(lm, lm_kwargs, signature)
+ return await super().acall(lm, lm_kwargs, signature, demos, inputs)
def format_field_structure(self, signature: type[Signature]) -> str:
parts = []
@@ -188,7 +161,13 @@ class JSONAdapter(ChatAdapter):
# Attempt to cast each value to type signature.output_fields[k].annotation.
for k, v in fields.items():
if k in signature.output_fields:
- fields[k] = parse_value(v, signature.output_fields[k].annotation)
+ try:
+ fields[k] = parse_value(v, signature.output_fields[k].annotation)
+ except ValueError as exc:
+ raise AdapterParseError(
+ adapter_name=type(self).__name__, signature=signature, lm_response=completion,
+ message=f"Failed to parse field {k}: {exc}", parsed_result=fields,
+ ) from exc
fields = apply_output_field_defaults(signature, fields)
if fields.keys() != signature.output_fields.keys():
diff --git a/dspy/adapters/two_step_adapter.py b/dspy/adapters/two_step_adapter.py
index 012fa5d9b..1ce1f4e88 100644
--- a/dspy/adapters/two_step_adapter.py
+++ b/dspy/adapters/two_step_adapter.py
@@ -6,10 +6,11 @@ from dspy.adapters.base import Adapter
from dspy.adapters.chat_adapter import ChatAdapter
from dspy.adapters.types import ToolCalls
from dspy.adapters.utils import get_field_description_string
+from dspy.clients._deprecation import adapter_message_call
from dspy.clients.base_lm import BaseLM
from dspy.signatures.field import InputField
from dspy.signatures.signature import Signature, make_signature
-from dspy.utils.exceptions import AdapterParseError, LMError
+from dspy.utils.exceptions import AdapterParseError
"""
NOTE/TODO/FIXME:
@@ -101,9 +102,7 @@ class TwoStepAdapter(Adapter):
)
return parsed_result[0]
- except LMError:
- raise
- except Exception as e:
+ except AdapterParseError as e:
raise AdapterParseError(
adapter_name="TwoStepAdapter",
signature=signature,
@@ -121,7 +120,9 @@ class TwoStepAdapter(Adapter):
) -> list[dict[str, Any]]:
inputs = self.format(signature, demos, inputs)
- outputs = await lm.acall(messages=inputs, **lm_kwargs)
+ # TODO(3.5): use Request/Response for both the main and extraction calls.
+ with adapter_message_call(lm, inputs):
+ outputs = await lm.acall(messages=inputs, **lm_kwargs)
# The signature is supposed to be "text -> {original output fields}"
extractor_signature = self._create_extractor_signature(signature)
@@ -149,9 +150,7 @@ class TwoStepAdapter(Adapter):
)
value = value[0]
- except LMError:
- raise
- except Exception as e:
+ except AdapterParseError as e:
raise AdapterParseError(
adapter_name="TwoStepAdapter",
signature=signature,
diff --git a/dspy/adapters/utils.py b/dspy/adapters/utils.py
index 37cdc654b..2df6915e3 100644
--- a/dspy/adapters/utils.py
+++ b/dspy/adapters/utils.py
@@ -232,7 +232,7 @@ def parse_value(value, annotation):
try:
# For dspy.Type, try parsing from the original value in case it has a custom parser
return TypeAdapter(annotation).validate_python(value)
- except Exception:
+ except pydantic.ValidationError:
raise e
raise
diff --git a/dspy/adapters/xml_adapter.py b/dspy/adapters/xml_adapter.py
index e3617e560..8d6c42e67 100644
--- a/dspy/adapters/xml_adapter.py
+++ b/dspy/adapters/xml_adapter.py
@@ -118,7 +118,7 @@ class XMLAdapter(ChatAdapter):
except pydantic.ValidationError:
fields[name] = adapter.validate_python(value, by_name=True)
break
- except Exception as e:
+ except ValueError as e:
error = e
else:
raise AdapterParseError(
diff --git a/dspy/clients/_deprecation.py b/dspy/clients/_deprecation.py
new file mode 100644
index 000000000..8bf3156b6
--- /dev/null
+++ b/dspy/clients/_deprecation.py
@@ -0,0 +1,87 @@
+"""Temporary warning policy for the DSPy 3.4 to 3.5 LM-interface migration.
+
+Warnings belong to public API use, not provider-wire conversion. Remove this
+module and the adapter-origin marker with the legacy interfaces in 3.5.
+"""
+
+import inspect
+import warnings
+from contextlib import contextmanager
+from contextvars import ContextVar
+
+_GUIDE = "https://dspy.ai/community/normalized-lm-api-migration/"
+_adapter_call = ContextVar("dspy_legacy_adapter_message_call", default=None)
+
+
+def _warn(message: str) -> None:
+ # Python 3.10/3.11 lack warnings.warn(skip_file_prefixes=...). Skip DSPy's
+ # own forwarding frames for attribution only, never to decide eligibility.
+ # Keep Python's filtering rather than storing warning state on LM objects.
+ frame = inspect.currentframe()
+ stacklevel = 1
+ try:
+ while frame is not None:
+ module = frame.f_globals.get("__name__", "")
+ if module != "dspy" and not module.startswith("dspy."):
+ break
+ stacklevel += 1
+ frame = frame.f_back
+ finally:
+ del frame
+ warnings.warn(message, DeprecationWarning, stacklevel=stacklevel)
+
+
+def warn_legacy_lm() -> None:
+ _warn(
+ "Implementing custom LMs through BaseLM.forward() or aforward() is deprecated. "
+ "Implement an engine with complete(Request) -> Response and pass it to dspy.LM(engine=...) instead. "
+ "The old subclass interface remains supported throughout DSPy 3.4 and is scheduled for removal in 3.5, "
+ "along with LegacyEngine and AsyncLegacyEngine. "
+ f"See {_GUIDE}#custom-engines-and-legacy-plugins."
+ )
+
+
+def warn_legacy_engine() -> None:
+ _warn(
+ "LegacyEngine and AsyncLegacyEngine are deprecated and scheduled for removal in DSPy 3.5. "
+ "They are transition wrappers for 3.4 only. Migrate the underlying implementation to "
+ "complete(Request) -> Response, with an async counterpart when needed. "
+ f"See {_GUIDE}#custom-engines-and-legacy-plugins."
+ )
+
+
+def warn_legacy_shortcut() -> None:
+ _warn(
+ "The custom-engine complete_legacy() shortcut is deprecated and scheduled for removal in DSPy 3.5. "
+ "Implement complete(Request) -> Response (async complete for async engines). "
+ f"See {_GUIDE}#custom-engines-and-legacy-plugins."
+ )
+
+
+def warn_openai_messages(lm, messages) -> None:
+ if messages is None:
+ return
+ origin = _adapter_call.get()
+ if origin is not None and origin[0] is lm and origin[1] is messages:
+ return
+ _warn(
+ "Passing OpenAI-style message dictionaries or other message objects to an LM through messages= is deprecated and "
+ "scheduled for removal in DSPy 3.5. Use lm(dspy.lm15.Request(...)) with lm15.Message objects; "
+ "it returns an lm15.Response. lm('hello') remains supported as a convenience returning a list. "
+ f"See {_GUIDE}#migrating-openai-style-messages."
+ )
+
+
+@contextmanager
+def adapter_message_call(lm, messages):
+ """Identify only the exact dictionary call owned by a 3.4 adapter.
+
+ Normal program users cannot change this internal boundary. Calls with a
+ different LM or message object still warn, even when nested or concurrent.
+ This is not a general suppression of warnings in the adapter's context.
+ """
+ token = _adapter_call.set((lm, messages))
+ try:
+ yield
+ finally:
+ _adapter_call.reset(token)
diff --git a/dspy/clients/_http.py b/dspy/clients/_http.py
new file mode 100644
index 000000000..e698c4cf3
--- /dev/null
+++ b/dspy/clients/_http.py
@@ -0,0 +1,30 @@
+"""Conservative parsing of advisory HTTP retry hints."""
+
+import math
+from datetime import datetime, timezone
+from email.utils import parsedate_to_datetime
+
+
+def finite_seconds(value):
+ if isinstance(value, bool) or not isinstance(value, (int, float, str)):
+ return None
+ try:
+ value = float(value)
+ except (ValueError, OverflowError):
+ return None
+ return value if math.isfinite(value) and value >= 0 else None
+
+
+def retry_after_seconds(value):
+ seconds = finite_seconds(value)
+ if seconds is not None:
+ return seconds
+ if not isinstance(value, str):
+ return None
+ try:
+ when = parsedate_to_datetime(value)
+ if when.tzinfo is None:
+ when = when.replace(tzinfo=timezone.utc)
+ return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
+ except (TypeError, ValueError, OverflowError):
+ return None
diff --git a/dspy/clients/backend_selection.py b/dspy/clients/backend_selection.py
new file mode 100644
index 000000000..b0beef6db
--- /dev/null
+++ b/dspy/clients/backend_selection.py
@@ -0,0 +1,54 @@
+"""Offline configuration-based backend selection, shared by planning and execution.
+
+This decides which backend owns a model/client configuration. Native request
+conversion still checks the actual message and generation-option representation
+before execution. No provider client is constructed and no credential is invoked.
+"""
+
+import os
+from dataclasses import dataclass, field
+
+from dspy.clients.engines.lm15_engine import LM15Engine
+from dspy.lm15 import RouterConfig, UnknownModelError, UnsupportedFeatureError
+
+CLIENT_KEYS = {"api_key", "api_base", "base_url", "headers", "extra_headers", "timeout", "api_version",
+ "azure_ad_token_provider", "organization", "project", "extra_query", "custom_llm_provider"}
+NATIVE_CLIENT_KEYS = {"api_key", "api_base", "base_url"}
+
+
+@dataclass(frozen=True)
+class BackendSelection:
+ native: bool
+ resolution: object = None
+ clients: dict = field(default_factory=dict, repr=False)
+
+
+def select_backend(lm, options=None):
+ """Select using LM defaults plus call overrides, without inference I/O."""
+ spec = lm._engine_spec
+ if not isinstance(spec, str):
+ raise TypeError("Custom engines supply their own capability contract")
+ clients = {key: val for key, val in lm.kwargs.items() if key in CLIENT_KEYS}
+ clients.update({key: val for key, val in (options or {}).items() if key in CLIENT_KEYS})
+ native = spec != "litellm" and lm.model_type != "text"
+ resolution = None
+ if native:
+ try:
+ resolution = LM15Engine(RouterConfig(env={}), model_type=lm.model_type).resolve(lm.model)
+ # Preserve legacy environment-configured gateways. A public native
+ # endpoint must not receive credentials meant for that gateway.
+ prefix = lm.model.split("/", 1)[0].upper()
+ if spec == "auto" and any(os.getenv(name) for name in (f"{prefix}_API_BASE", f"{prefix}_BASE_URL")):
+ native = False
+ if spec == "auto" and resolution.provider == "xai" and "api_key" not in clients and os.getenv("XAI_API_KEY"):
+ clients["api_key"] = os.environ["XAI_API_KEY"]
+ if (set(clients) - NATIVE_CLIENT_KEYS) or (resolution.provider.startswith("azure") and
+ any(key in clients for key in ("api_base", "base_url"))):
+ native = False
+ except (UnknownModelError, UnsupportedFeatureError):
+ if spec == "lm15":
+ raise
+ native = False
+ if spec == "lm15" and not native:
+ raise UnsupportedFeatureError("The requested client settings require the LiteLLM compatibility engine.")
+ return BackendSelection(native, resolution, clients)
diff --git a/dspy/clients/base_lm.py b/dspy/clients/base_lm.py
index 172f8e904..224b16531 100644
--- a/dspy/clients/base_lm.py
+++ b/dspy/clients/base_lm.py
@@ -3,18 +3,10 @@ import datetime
import importlib
import inspect
import uuid
-import warnings
-from typing import Any, Literal, TextIO
-
-from dspy.clients.openai_format import (
- completion_to_lm_response,
- cost_from_response,
- lm_response_from_legacy_outputs,
- responses_to_lm_response,
- to_openai_chat_request,
- usage_from_response,
-)
-from dspy.core.types import LMHistoryEntry, LMRequest, LMResponse
+from typing import Any, TextIO
+
+from dspy.clients._deprecation import warn_legacy_lm, warn_openai_messages
+from dspy.clients.legacy_outputs import responses_outputs
from dspy.dsp.utils import settings
from dspy.utils.callback import BaseCallback, with_callbacks
from dspy.utils.inspect_history import pretty_print_history
@@ -23,7 +15,6 @@ MAX_HISTORY_SIZE = 10_000
GLOBAL_HISTORY = []
LM_CLASS_STATE_KEY = "_dspy_lm_class"
_BUILTIN_LM_CLASS_PATH = "dspy.clients.lm.LM"
-ForwardContract = Literal["legacy", "typed_lm"]
def _import_lm_class(class_path: str) -> type:
@@ -57,115 +48,22 @@ def _import_lm_class(class_path: str) -> type:
class BaseLM:
"""Base class for DSPy language models.
- Most users should use `dspy.LM`, which is a `BaseLM` subclass.
-
- For advanced use cases, such as custom language model backends, users can
- subclass `BaseLM` and implement `forward()`.
-
- DSPy is migrating `forward()` from the legacy OpenAI/LiteLLM-shaped
- contract to a typed DSPy contract. During this migration, subclasses should
- declare which contract they implement with `forward_contract`:
-
- - `forward_contract = "typed_lm"`: implement
- `forward(request: dspy.LMRequest) -> dspy.LMResponse`. This is the
- preferred contract for new custom LMs.
- - `forward_contract = "legacy"`: implement
- `forward(prompt=None, messages=None, **kwargs)` and return an OpenAI-like
- provider response. This remains the default during the migration.
-
- `BaseLM.__call__()` is the compatibility boundary. In DSPy 3.3 and 3.4,
- ordinary calls preserve the legacy public return value, `list[str | dict]`:
-
- ```python
- outputs = lm("What is DSPy?")
- outputs = lm(messages=[{"role": "user", "content": "What is DSPy?"}])
- ```
-
- Calls can flow internally through the typed `LMRequest` / `LMResponse` path
- without changing the public return shape. The typed path is used when the
- caller passes an explicit `dspy.LMRequest`, when
- `dspy.context(experimental=True)` is active, or when the subclass declares
- `forward_contract = "typed_lm"`. It accepts richer direct-call inputs,
- including `dspy.System`, `dspy.User`, `dspy.Assistant`, `dspy.ToolResult`,
- content parts, and prior `dspy.LMResponse` objects. The public return value
- remains legacy outputs unless the caller explicitly opts into typed output
- with an `LMRequest` or `experimental=True`.
-
- Example typed direct call:
-
- ```python
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("You are concise."),
- dspy.User("What is DSPy?"),
- )
- print(response.text)
- ```
-
- `LMResponse` is designed to feel familiar to users of the legacy output
- list while carrying substantially more structure, including typed outputs,
- usage, cache status, provider metadata, tool calls, reasoning, citations,
- and multimodal content.
-
- LMs must be serializable as part of saved DSPy programs. The default
- `dump_state()` and `load_state()` implementations support subclasses whose
- persistent state is fully captured by `BaseLM.__init__()` arguments. If a
- subclass stores additional persistent state, override both methods.
-
- Examples:
- Preferred typed custom LM:
-
- ```python
- import dspy
-
-
- class EchoLM(dspy.BaseLM):
- forward_contract = "typed_lm"
-
- def forward(self, request: dspy.LMRequest) -> dspy.LMResponse:
- return dspy.LMResponse.from_text("hello", model=request.model)
-
-
- lm = EchoLM(model="test/echo")
-
- with dspy.context(experimental=True):
- response = lm(dspy.User("Say hello."))
- print(response.text)
- ```
-
- Legacy custom LM for an OpenAI-like provider:
-
- ```python
- from openai import OpenAI
-
- import dspy
-
-
- class MyLegacyLM(dspy.BaseLM):
- forward_contract = "legacy"
-
- def forward(self, prompt=None, messages=None, **kwargs):
- client = OpenAI()
- return client.chat.completions.create(
- model=self.model,
- messages=messages or [{"role": "user", "content": prompt}],
- **self.kwargs,
- **kwargs,
- )
-
-
- lm = MyLegacyLM(model="gpt-4o-mini")
- dspy.configure(lm=lm)
- print(dspy.Predict("q -> a")(q="Why did the chicken cross the kitchen?"))
- ```
- """
-
- forward_contract: ForwardContract = "legacy"
- """The `forward()` implementation contract used by this LM.
-
- `"legacy"` means `forward(prompt=None, messages=None, **kwargs)` returns an
- OpenAI-like provider response. `"typed_lm"` means
- `forward(request: dspy.LMRequest) -> dspy.LMResponse`.
+ Legacy subclasses implement forward(prompt=None, messages=None, **kwargs)
+ and optionally aforward with the same arguments. Ordinary calls return
+ lists of strings or dictionaries. The built-in LM also accepts explicit
+ dspy.lm15.Request calls. The experimental setting does not change outputs.
+
+ Implementing custom LMs through forward()/aforward() is deprecated. The old
+ subclass interface remains supported throughout DSPy 3.4 and is scheduled
+ for removal in 3.5. Implement an engine with complete(Request) -> Response
+ and pass it to dspy.LM(engine=...) instead. LegacyEngine and AsyncLegacyEngine
+ are transition wrappers for 3.4 only and are also scheduled for removal in 3.5.
+ OpenAI-style messages= dictionaries are deprecated too: use lm15.Request
+ and Message objects instead. lm("hello") remains a list-returning convenience.
+ See the [migration guide](https://dspy.ai/community/normalized-lm-api-migration/#custom-engines-and-legacy-plugins).
+
+ Persistent custom state belongs in dump_state/load_state. Runtime clients
+ are shared by copy(), while DSPy history, callbacks and kwargs are isolated.
"""
def __init__(
@@ -205,64 +103,6 @@ class BaseLM:
def _get_initial_kwargs(self, *, temperature, max_tokens, **kwargs) -> dict[str, Any]:
return dict(temperature=temperature, max_tokens=max_tokens, **kwargs)
- def _declares_forward_contract(self) -> bool:
- """Return whether this concrete LM class declares `forward_contract`."""
- return "forward_contract" in type(self).__dict__
-
- def _get_forward_contract(self) -> ForwardContract:
- """Return the declared `forward()` contract for this LM.
-
- DSPy deliberately does not inspect `forward()` signatures during the
- migration to typed LM calls. Subclasses opt into the typed contract by
- setting `forward_contract = "typed_lm"`; all existing custom LMs remain
- on the legacy contract unless they say otherwise.
- """
- contract = getattr(type(self), "forward_contract", "legacy")
- if contract not in {"legacy", "typed_lm"}:
- raise ValueError(
- f"{type(self).__name__}.forward_contract must be 'legacy' or 'typed_lm', "
- f"but got {contract!r}."
- )
- return contract
-
- def _validate_typed_lm_response(self, response: Any) -> LMResponse:
- """Validate the result of a typed `forward(request)` implementation."""
- if isinstance(response, LMResponse):
- return response
- raise TypeError(
- f"{type(self).__name__}.forward_contract='typed_lm' requires forward(request) "
- f"to return dspy.LMResponse, but got {type(response).__name__}."
- )
-
- def _validate_legacy_lm_response(
- self,
- response: Any,
- *,
- stacklevel: int = 2,
- ) -> LMResponse | None:
- """Validate a legacy `forward()` result that already looks typed.
-
- During the 3.3 migration, custom LMs that inherit the default legacy
- contract may accidentally return `LMResponse`; warn so authors can add
- `forward_contract = "typed_lm"`. If a class explicitly declares
- `forward_contract = "legacy"`, treat `LMResponse` as a contract
- mismatch and fail fast.
- """
- if not isinstance(response, LMResponse):
- return None
- if self._declares_forward_contract():
- raise TypeError(
- f"{type(self).__name__}.forward_contract='legacy' requires forward() to return an "
- "OpenAI-like provider response, but got dspy.LMResponse. Set forward_contract='typed_lm'."
- )
- warnings.warn(
- f"{type(self).__name__}.forward() returned dspy.LMResponse while using the default legacy "
- "forward_contract. Set forward_contract='typed_lm' before the typed LM API becomes the default.",
- DeprecationWarning,
- stacklevel=stacklevel,
- )
- return response
-
@property
def supports_function_calling(self) -> bool:
"""Whether the model supports function calling (tool use)."""
@@ -291,13 +131,16 @@ class BaseLM:
else:
outputs = self._process_completion(response, merged_kwargs)
+ return self._record_response(response, prompt, messages, outputs, kwargs)
+
+ def _record_response(self, response, prompt, messages, outputs, kwargs, request=None):
if not getattr(response, "cache_hit", False) and settings.usage_tracker:
settings.usage_tracker.add_usage(self.model, dict(getattr(response, "usage", {}) or {}))
if settings.disable_history:
return outputs
- # Logging, with removed api key & where `cost` is None on cache hit.
+ # Cache hits retain historical cost metadata but add no billed usage.
kwargs = {k: v for k, v in kwargs.items() if not k.startswith("api_")}
entry = {
"prompt": prompt,
@@ -314,353 +157,44 @@ class BaseLM:
"model_type": self.model_type,
}
+ if request is not None:
+ entry["request"] = request
self.update_history(entry)
return outputs
@with_callbacks
- def __call__(
- self,
- *items: Any,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- request: LMRequest | None = None,
- **kwargs,
- ) -> LMResponse | list[dict[str, Any] | str]:
- """Call the language model synchronously.
+ def __call__(self, prompt=None, *, messages=None, **kwargs):
+ """Return legacy outputs, or one lm15.Response for an explicit Request."""
+ from dspy.clients.execution import execute, finalize, prepare
- The default call path preserves DSPy's legacy behavior and returns `list[str | dict]`. Calls flow internally
- through `LMRequest` / `LMResponse` when either:
-
- - `request=` is provided or the first positional argument is an `LMRequest`, or
- - `dspy.context(experimental=True)` is active, or
- - the subclass declares `forward_contract = "typed_lm"`.
-
- In the typed request path, positional `items` are normalized with `LMRequest.from_call()`. This supports
- a single prompt string, direct message objects such as `dspy.System(...)`, `dspy.User(...)`, and
- `dspy.Assistant(...)`, `dspy.ToolResult(...)` tool messages, and prior `LMResponse` values as assistant turns.
-
- Args:
- *items: Optional direct-call inputs. In the legacy path this may contain at most one prompt string. In the
- typed path it may contain normalized messages, message sequences, prior `LMResponse` values, or content
- parts accepted by `LMRequest.from_call()`.
- prompt: Optional prompt string. Do not combine with positional prompt input.
- messages: Optional OpenAI-chat-shaped messages. Do not combine with `items` or `prompt` in the typed path.
- request: Optional explicit normalized request. Call kwargs override request config when provided.
- **kwargs: Per-call generation parameters.
-
- Returns:
- `LMResponse` for explicit `LMRequest` calls or `experimental=True`; otherwise DSPy's legacy list of output
- strings or dictionaries, even when a typed LM subclass uses the typed path internally.
- """
- return_typed_response, forward_contract, normalized_request = self._prepare_lm_call(
- items=items,
- prompt=prompt,
- messages=messages,
- request=request,
- kwargs=kwargs,
- )
- if normalized_request is None:
- return self._legacy_call_direct(*items, prompt=prompt, messages=messages, **kwargs)
-
- if forward_contract == "typed_lm":
- response = self.forward(normalized_request)
- response = self._finalize_lm_response(normalized_request, self._validate_typed_lm_response(response))
- else:
- response = self._legacy_forward_as_lm_response(normalized_request)
- if return_typed_response:
- return response
- return response.to_legacy_outputs()
+ call = prepare(self, prompt, messages, kwargs)
+ if not call.managed:
+ warn_legacy_lm()
+ warn_openai_messages(self, messages)
+ return finalize(self, call, execute(self, call))
@with_callbacks
- async def acall(
- self,
- *items: Any,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- request: LMRequest | None = None,
- **kwargs,
- ) -> LMResponse | list[dict[str, Any] | str]:
- """Asynchronously call the language model.
-
- This is the async equivalent of `__call__()`. It preserves legacy outputs by default and returns
- `dspy.LMResponse` for explicit `LMRequest` calls or experimental direct calls.
- """
- return_typed_response, forward_contract, normalized_request = self._prepare_lm_call(
- items=items,
- prompt=prompt,
- messages=messages,
- request=request,
- kwargs=kwargs,
- )
- if normalized_request is None:
- return await self._legacy_acall_direct(*items, prompt=prompt, messages=messages, **kwargs)
-
- if forward_contract == "typed_lm":
- response = await self.aforward(normalized_request)
- response = self._finalize_lm_response(normalized_request, self._validate_typed_lm_response(response))
- else:
- response = await self._legacy_aforward_as_lm_response(normalized_request)
- if return_typed_response:
- return response
- return response.to_legacy_outputs()
-
- def _prepare_lm_call(
- self,
- *,
- items: tuple[Any, ...],
- prompt: str | None,
- messages: list[dict[str, Any]] | None,
- request: LMRequest | None,
- kwargs: dict[str, Any],
- ) -> tuple[bool, ForwardContract, LMRequest | None]:
- explicit_request = request is not None or bool(items and isinstance(items[0], LMRequest))
- return_typed_response = explicit_request or bool(settings.get("experimental", False))
- forward_contract = self._get_forward_contract()
- if not return_typed_response and forward_contract != "typed_lm":
- return return_typed_response, forward_contract, None
-
- normalized_request = self._normalize_lm_call(
- *items,
- prompt=prompt,
- messages=messages,
- request=request,
- **kwargs,
- )
- return return_typed_response, forward_contract, normalized_request
-
- def _legacy_call_direct(
- self,
- *items: Any,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- **kwargs: Any,
- ) -> list[dict[str, Any] | str]:
- """Execute the pre-typed synchronous call path and return legacy outputs."""
- prompt = self._legacy_prompt_from_items(items, prompt=prompt)
- response = self.forward(prompt=prompt, messages=messages, **kwargs)
- if isinstance(response, LMResponse):
- raise TypeError(
- f"{type(self).__name__}.forward() returned dspy.LMResponse on the legacy direct path. "
- "Set forward_contract='typed_lm' or pass an LMRequest/use dspy.context(experimental=True)."
- )
- return self._process_lm_response(response, prompt, messages, **kwargs)
-
- async def _legacy_acall_direct(
- self,
- *items: Any,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- **kwargs: Any,
- ) -> list[dict[str, Any] | str]:
- """Execute the pre-typed asynchronous call path and return legacy outputs."""
- prompt = self._legacy_prompt_from_items(items, prompt=prompt)
- response = await self.aforward(prompt=prompt, messages=messages, **kwargs)
- if isinstance(response, LMResponse):
- raise TypeError(
- f"{type(self).__name__}.aforward() returned dspy.LMResponse on the legacy direct path. "
- "Set forward_contract='typed_lm' or pass an LMRequest/use dspy.context(experimental=True)."
- )
- return self._process_lm_response(response, prompt, messages, **kwargs)
-
- def _legacy_prompt_from_items(self, items: tuple[Any, ...], *, prompt: str | None) -> str | None:
- """Validate and extract the one positional prompt accepted by legacy calls."""
- if len(items) > 1:
- raise TypeError(
- "Legacy BaseLM calls accept at most one positional prompt. "
- "Use dspy.context(experimental=True) or pass an LMRequest for typed multi-item LM calls."
- )
- if items and prompt is not None:
- raise TypeError("Pass a prompt either positionally or by keyword, not both.")
- if items and isinstance(items[0], LMRequest):
- raise TypeError("LMRequest calls require the typed LM path; this should be unreachable.")
- return items[0] if items else prompt
-
- def _normalize_lm_call(
- self,
- *items: Any,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- request: LMRequest | None = None,
- **kwargs: Any,
- ) -> LMRequest:
- """Normalize direct call inputs or an explicit request into `LMRequest`."""
- if request is None and items and isinstance(items[0], LMRequest):
- request = items[0]
- items = items[1:]
-
- if request is not None:
- if items or prompt is not None or messages is not None:
- raise ValueError(
- "Pass either an LMRequest or direct-call inputs, not both. "
- "Use call kwargs to override request config."
- )
- return request.with_config_overrides(**kwargs) if kwargs else request
-
- merged_kwargs = {**self.kwargs, **kwargs}
- merged_kwargs.setdefault("cache", self.cache)
- return LMRequest.from_call(
- model=self.model,
- items=items,
- prompt=prompt,
- messages=messages,
- **merged_kwargs,
- )
-
- def _legacy_forward_as_lm_response(self, request: LMRequest) -> LMResponse:
- """Call a legacy `forward()` implementation and normalize its provider response."""
- data = self._legacy_forward_kwargs(request)
- messages = data.pop("messages", None)
- prompt = self._prompt_from_lm_request(request)
- if prompt is not None:
- messages = None
- response = self.forward(prompt=prompt, messages=messages, **data)
- typed_response = self._validate_legacy_lm_response(response, stacklevel=4)
- if typed_response is not None:
- return self._finalize_lm_response(request, typed_response)
- with settings.context(disable_history=True, usage_tracker=None):
- outputs = self._process_lm_response(response, prompt, messages, **data)
- lm_response = self._legacy_outputs_to_lm_response(outputs, request=request, provider_response=response)
- return self._finalize_lm_response(request, lm_response)
-
- async def _legacy_aforward_as_lm_response(self, request: LMRequest) -> LMResponse:
- """Async variant of `_legacy_forward_as_lm_response()`."""
- data = self._legacy_forward_kwargs(request)
- messages = data.pop("messages", None)
- prompt = self._prompt_from_lm_request(request)
- if prompt is not None:
- messages = None
- response = await self.aforward(prompt=prompt, messages=messages, **data)
- typed_response = self._validate_legacy_lm_response(response, stacklevel=4)
- if typed_response is not None:
- return self._finalize_lm_response(request, typed_response)
- with settings.context(disable_history=True, usage_tracker=None):
- outputs = self._process_lm_response(response, prompt, messages, **data)
- lm_response = self._legacy_outputs_to_lm_response(outputs, request=request, provider_response=response)
- return self._finalize_lm_response(request, lm_response)
-
- def _legacy_outputs_to_lm_response(
- self,
- outputs: list[dict[str, Any] | str | None],
- *,
- request: LMRequest,
- provider_response: Any,
- ) -> LMResponse:
- """Convert legacy post-processed outputs and provider metadata into `LMResponse`."""
- if self.model_type == "responses":
- response = responses_to_lm_response(provider_response, request)
- elif self.model_type in {"chat", "text"}:
- response = completion_to_lm_response(provider_response, request)
- else:
- response = lm_response_from_legacy_outputs(outputs, request)
-
- return response.model_copy(
- update={
- "model": getattr(provider_response, "model", None) or response.model,
- "usage": usage_from_response(provider_response),
- "cost": cost_from_response(provider_response),
- "cache_hit": bool(getattr(provider_response, "cache_hit", False)),
- "provider_response": provider_response,
- }
- )
-
- def _legacy_forward_kwargs(self, request: LMRequest) -> dict[str, Any]:
- """Convert a normalized request into kwargs for legacy `forward()` implementations."""
- data = to_openai_chat_request(request)
- data.pop("model", None)
- if request.config.cache is not None:
- if request.config.cache.enabled is not None:
- data["cache"] = request.config.cache.enabled
- if request.config.cache.rollout_id is not None:
- data["rollout_id"] = request.config.cache.rollout_id
- return data
-
- def _prompt_from_lm_request(self, request: LMRequest) -> str | None:
- """Return the legacy prompt when a normalized request is exactly one text user message."""
- if len(request.messages) != 1:
- return None
- message = request.messages[0]
- if message.role != "user" or len(message.parts) != 1:
- return None
- part = message.parts[0]
- return part.text if getattr(part, "type", None) == "text" else None
-
- def _finalize_lm_response(self, request: LMRequest, response: LMResponse) -> LMResponse:
- """Record usage and typed history for a normalized LM response."""
- if not getattr(response, "cache_hit", False) and settings.usage_tracker:
- usage = response.usage_as_dict()
- if usage:
- settings.usage_tracker.add_usage(self.model, usage)
-
- if not settings.disable_history:
- entry = LMHistoryEntry(
- request=request,
- response=response,
- timestamp=datetime.datetime.now().isoformat(),
- uuid=str(uuid.uuid4()),
- model_type=getattr(self, "model_type", None),
- )
- self.update_history(entry)
- return response
-
- def forward(
- self,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- **kwargs
- ):
- """Forward pass for the language model.
-
- Subclasses must implement this method according to `forward_contract`.
-
- For `forward_contract = "legacy"`, implement
- `forward(prompt=None, messages=None, **kwargs)` and return one of these OpenAI-like provider responses:
-
- - [OpenAI response format](https://platform.openai.com/docs/api-reference/responses/object)
- - [OpenAI chat completion format](https://platform.openai.com/docs/api-reference/chat/object)
- - [OpenAI text completion format](https://platform.openai.com/docs/api-reference/completions/object)
-
- For `forward_contract = "typed_lm"`, implement `forward(request: dspy.LMRequest) -> dspy.LMResponse`.
-
- Raises:
- dspy.LMError: Base class for LM configuration, transport, provider,
- and unsupported-feature failures. Notable subclasses include
- `dspy.ContextWindowExceededError` for context-window failures,
- which adapters use to avoid inappropriate fallback retries when
- the prompt is too long. Each subclass should catch its
- provider's native context-window error and re-raise it as
- `dspy.ContextWindowExceededError`.
- """
+ async def acall(self, prompt=None, *, messages=None, **kwargs):
+ """Async equivalent of __call__, with the same execution ownership."""
+ import asyncio
+
+ from dspy.clients.execution import aexecute, finalize, prepare
+
+ # Canonical media snapshots may read local files. Context variables
+ # propagate to the worker; callbacks/finalization stay on the caller.
+ call = await asyncio.to_thread(prepare, self, prompt, messages, kwargs, asynchronous=True)
+ if not call.managed:
+ warn_legacy_lm()
+ warn_openai_messages(self, messages)
+ return finalize(self, call, await aexecute(self, call))
+
+ def forward(self, prompt=None, messages=None, **kwargs):
+ """Return an OpenAI-shaped provider response for a legacy LM call."""
raise NotImplementedError("Subclasses must implement this method.")
- async def aforward(
- self,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- **kwargs
- ):
- """Async forward pass for the language model.
-
- Subclasses that support async calls must implement this method according to `forward_contract`.
-
- For `forward_contract = "legacy"`, implement
- `aforward(prompt=None, messages=None, **kwargs)` and return one of these OpenAI-like provider responses:
-
- - [OpenAI response format](https://platform.openai.com/docs/api-reference/responses/object)
- - [OpenAI chat completion format](https://platform.openai.com/docs/api-reference/chat/object)
- - [OpenAI text completion format](https://platform.openai.com/docs/api-reference/completions/object)
-
- For `forward_contract = "typed_lm"`, implement `aforward(request: dspy.LMRequest) -> dspy.LMResponse`.
-
- Raises:
- dspy.LMError: Base class for LM configuration, transport, provider,
- and unsupported-feature failures. Notable subclasses include
- `dspy.ContextWindowExceededError` for context-window failures,
- which adapters use to avoid inappropriate fallback retries when
- the prompt is too long. Each subclass should catch its
- provider's native context-window error and re-raise it as
- `dspy.ContextWindowExceededError`.
- """
+ async def aforward(self, prompt=None, messages=None, **kwargs):
+ """Asynchronously return an OpenAI-shaped provider response."""
raise NotImplementedError("Subclasses must implement this method.")
def dump_state(self) -> dict[str, Any]:
@@ -810,18 +344,20 @@ class BaseLM:
Returns:
List of processed outputs
"""
+ from dspy.clients.legacy_outputs import value
+
outputs = []
- for c in response.choices:
- output = {}
- output["text"] = c.message.content if hasattr(c, "message") else c["text"]
+ for c in value(response, "choices", []) or []:
+ message = value(c, "message")
+ output = {"text": value(message, "content") if message is not None else value(c, "text")}
- if hasattr(c, "message") and hasattr(c.message, "reasoning_content") and c.message.reasoning_content:
- output["reasoning_content"] = c.message.reasoning_content
+ if reasoning := value(message, "reasoning_content"):
+ output["reasoning_content"] = reasoning
if merged_kwargs.get("logprobs"):
- output["logprobs"] = c.logprobs if hasattr(c, "logprobs") else c["logprobs"]
- if hasattr(c, "message") and getattr(c.message, "tool_calls", None):
- output["tool_calls"] = c.message.tool_calls
+ output["logprobs"] = value(c, "logprobs")
+ if calls := value(message, "tool_calls"):
+ output["tool_calls"] = calls
# Extract citations from LiteLLM response if available
citations = self._extract_citations_from_response(c)
@@ -847,9 +383,12 @@ class BaseLM:
"""
try:
# Check for citations in LiteLLM provider_specific_fields
- citations_data = choice.message.provider_specific_fields.get("citations")
+ from dspy.clients.legacy_outputs import value
+
+ fields = value(value(choice, "message"), "provider_specific_fields", {}) or {}
+ citations_data = fields.get("citations")
if isinstance(citations_data, list):
- return [citation for citations in citations_data for citation in citations]
+ return [citation for group in citations_data for citation in (group if isinstance(group, list) else [group])]
except Exception:
return None
@@ -865,9 +404,7 @@ class BaseLM:
Each output is a dictionary in the same shape the chat path produces: `text` is always present, and
`reasoning_content`, `tool_calls`, and `citations` appear when present.
"""
- request = LMRequest(model=self.model, messages=[])
- lm_response = responses_to_lm_response(response, request)
- return [output.to_output_dict() for output in lm_response.outputs]
+ return responses_outputs(response)
def inspect_history(n: int = 1, file: "TextIO | None" = None) -> None:
diff --git a/dspy/clients/call_context.py b/dspy/clients/call_context.py
new file mode 100644
index 000000000..9046abf39
--- /dev/null
+++ b/dspy/clients/call_context.py
@@ -0,0 +1,44 @@
+"""Task-local progress for retry/fallback policy, never part of a model Request."""
+
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass
+
+from dspy.dsp.utils.settings import settings
+
+
+@dataclass
+class StreamProgress:
+ emitted: bool = False
+
+
+_adapter_calls = ContextVar("dspy_adapter_stream_progress", default=())
+
+
+@contextmanager
+def adapter_fallback_scope():
+ progress = StreamProgress()
+ token = _adapter_calls.set((*_adapter_calls.get(), progress))
+ try:
+ yield progress
+ finally:
+ _adapter_calls.reset(token)
+
+
+def stream_emitted():
+ # A nested extraction call's visible output also prevents its parent from
+ # replaying a program segment. Independent concurrent calls have own scopes.
+ for progress in _adapter_calls.get():
+ progress.emitted = True
+ progress = settings.get("_lm_stream_progress")
+ if progress is not None:
+ progress["emitted"] = True
+
+
+def completed_legacy(raw):
+ # Transitional LiteLLM bookkeeping: retain billed usage even if SDK stream
+ # cleanup or legacy output conversion fails after a full reply was obtained.
+ progress = settings.get("_lm_stream_progress")
+ if progress is not None:
+ progress["completed"] = True
+ progress["raw"] = raw
diff --git a/dspy/clients/call_result.py b/dspy/clients/call_result.py
new file mode 100644
index 000000000..3d96bdc42
--- /dev/null
+++ b/dspy/clients/call_result.py
@@ -0,0 +1,168 @@
+"""DSPy execution metadata around unmodified lm15 responses."""
+
+from dataclasses import dataclass, field
+from typing import Any
+
+from dspy._vendor.lm15.serde import response_from_dict, response_to_dict
+from dspy.clients.legacy_outputs import plain, value
+from dspy.clients.lm15_boundary import response_value
+from dspy.lm15 import Request, Response, TextPart, ThinkingPart
+
+CACHE_FORMAT = "dspy-lm15-result-v1"
+
+
+class AttributeDict(dict):
+ """Legacy output fields remain accessible by key and by attribute."""
+
+ def __getattr__(self, name):
+ try:
+ return self[name]
+ except KeyError:
+ raise AttributeError(name) from None
+
+ def model_dump(self, **kwargs):
+ return plain(self)
+
+
+def attributes(obj):
+ if isinstance(obj, dict):
+ return AttributeDict({key: attributes(item) for key, item in obj.items()})
+ if isinstance(obj, list):
+ return [attributes(item) for item in obj]
+ return obj
+
+
+def legacy_output(response: Response, *, logprobs=False):
+ import json
+
+ texts = [part.text for part in response.message.parts if isinstance(part, TextPart)]
+ thinking = [part.text for part in response.message.parts if isinstance(part, ThinkingPart)]
+ output = {"text": "".join(texts) if texts else None}
+ if any(thinking):
+ output["reasoning_content"] = "".join(thinking)
+ if response.tool_calls:
+ output["tool_calls"] = [attributes({
+ "id": call.id, "type": "function",
+ "function": {"name": call.name, "arguments": json.dumps(call.input)},
+ }) for call in response.tool_calls]
+ if response.citations:
+ output["citations"] = [
+ {key: val for key, val in {"text": part.text, "title": part.title, "url": part.url}.items() if val is not None}
+ for part in response.citations
+ ]
+ if logprobs:
+ output["logprobs"] = attributes({"content": [
+ {"token": token.token, "logprob": token.logprob,
+ "bytes": list(token.bytes) if token.bytes is not None else None,
+ "top_logprobs": [{"token": top.token, "logprob": top.logprob,
+ "bytes": list(top.bytes) if top.bytes is not None else None} for top in token.top]}
+ for token in response.logprobs or ()
+ ]})
+ return output
+
+
+def usage_dict(response: Response, provider=None):
+ u = response.usage
+ inputs, outputs = u.input_tokens, u.output_tokens
+ # LiteLLM's public prompt/completion counters include cached Anthropic
+ # input and Gemini thinking output. lm15's counters are provider-verbatim.
+ if provider and "anthropic" in provider and inputs is not None:
+ inputs += (u.cache_read_tokens or 0) + (u.cache_write_tokens or 0)
+ if provider in {"gemini", "vertex", "vertex-express", "xai"} and outputs is not None:
+ outputs += u.reasoning_tokens or 0
+ data = {key: val for key, val in {
+ "prompt_tokens": inputs, "completion_tokens": outputs,
+ "total_tokens": (inputs + outputs if provider and "anthropic" in provider and inputs is not None and outputs is not None
+ else u.total_tokens),
+ }.items() if val is not None}
+ prompt_details = {key: val for key, val in {
+ "cached_tokens": u.cache_read_tokens, "cache_creation_tokens": u.cache_write_tokens,
+ "audio_tokens": u.input_audio_tokens,
+ }.items() if val is not None}
+ completion_details = {key: val for key, val in {
+ "reasoning_tokens": u.reasoning_tokens, "audio_tokens": u.output_audio_tokens,
+ }.items() if val is not None}
+ if data or prompt_details or completion_details:
+ data["prompt_tokens_details"] = prompt_details or None
+ data["completion_tokens_details"] = completion_details or None
+ return data
+
+
+@dataclass
+class CallResult:
+ responses: tuple[Response, ...] = ()
+ outputs: list = field(default_factory=list)
+ usage: dict = field(default_factory=dict)
+ cost: float | None = None
+ raw: Any = None
+ response_model: str | None = None
+ cache_hit: bool = False
+ cost_details: dict = field(default_factory=dict)
+
+ @classmethod
+ def native(cls, response, *, model_type="chat", logprobs=False, provider=None):
+ output = legacy_output(response, logprobs=logprobs)
+ outputs = [output] if model_type == "responses" or len(output) > 1 else [output["text"]]
+ return cls((response,), outputs, usage_dict(response, provider), raw=response, response_model=response.model)
+
+ @classmethod
+ def legacy(cls, lm, raw, request=None, *, kwargs=None):
+ kwargs = kwargs or {}
+ outputs = (lm._process_response(raw) if lm.model_type == "responses"
+ else lm._process_completion(raw, {**lm.kwargs, **kwargs}))
+ return cls(outputs=outputs, usage=plain(dict(value(raw, "usage", {}) or {})),
+ cost=getattr(raw, "_hidden_params", {}).get("response_cost"), raw=raw,
+ response_model=value(raw, "model", lm.model))
+
+ def typed(self, request: Request, model_type):
+ if self.responses:
+ if len(self.responses) != 1:
+ raise ValueError("An explicit Request returns exactly one Response")
+ return self.responses[0]
+ return response_value(self.raw, model_type, request)
+
+ def dump(self):
+ return {"_dspy_format": CACHE_FORMAT,
+ "responses": [response_to_dict(r, include_provider_data=True) for r in self.responses],
+ "outputs": plain(self.outputs), "usage": plain(self.usage), "cost": self.cost,
+ "cost_details": plain(self.cost_details),
+ "raw": None if self.responses else plain(self.raw), "response_model": self.response_model}
+
+ @classmethod
+ def load(cls, record):
+ responses = tuple(response_from_dict(r) for r in record["responses"])
+ return cls(responses, attributes(record["outputs"]), {}, record["cost"],
+ responses[0] if len(responses) == 1 else attributes(record.get("raw")),
+ record["response_model"], True, record.get("cost_details", {}))
+
+ def provider_response(self):
+ if self.raw is not None:
+ if not isinstance(self.raw, Response):
+ return self.raw
+ choices = []
+ for index, output in enumerate(self.outputs):
+ output = output if isinstance(output, dict) else {"text": output}
+ choices.append({"index": index, "finish_reason": "stop", "message": {
+ "content": output.get("text"), "tool_calls": output.get("tool_calls"),
+ "reasoning_content": output.get("reasoning_content"),
+ }, "logprobs": output.get("logprobs")})
+ return attributes({"model": self.response_model, "choices": choices, "usage": self.usage,
+ "cache_hit": self.cache_hit})
+
+
+def combine(results, *, model_type):
+ from dspy.utils.usage_tracker import UsageTracker
+
+ tracker = UsageTracker()
+ for result in results:
+ tracker.add_usage("combined", result.usage)
+ outputs = [out for result in results for out in result.outputs]
+ if any(isinstance(out, dict) for out in outputs):
+ outputs = [out if isinstance(out, dict) else {"text": out} for out in outputs]
+ costs = [r.cost for r in results]
+ return CallResult(tuple(r for item in results for r in item.responses), outputs,
+ tracker.get_total_tokens().get("combined", {}),
+ sum(costs) if costs and all(c is not None for c in costs) else None,
+ raw=results[0].raw if len(results) == 1 else None,
+ response_model=results[0].response_model if results else None,
+ cost_details={"candidates": [r.cost_details for r in results]} if any(r.cost_details for r in results) else {})
diff --git a/dspy/clients/capabilities.py b/dspy/clients/capabilities.py
new file mode 100644
index 000000000..4b214d078
--- /dev/null
+++ b/dspy/clients/capabilities.py
@@ -0,0 +1,191 @@
+"""Adapter-facing capability hints for native and custom execution engines.
+
+The catalog describes models; lm15's policies describe the selected API.
+A serializer's ability to send a field alone is not evidence a model honours it.
+Public booleans preserve BaseLM's existing unknown-as-false convention.
+"""
+
+import asyncio
+import inspect
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass, field
+from functools import wraps
+
+from dspy.clients.backend_selection import select_backend
+from dspy.clients.engines.litellm_errors import litellm_errors
+from dspy.clients.errors import error_boundary
+from dspy.clients.model_metadata import model_info
+
+
+@dataclass(frozen=True)
+class Capabilities:
+ function_calling: bool = False
+ reasoning: bool = False
+ response_schema: bool = False
+ params: frozenset[str] = field(default_factory=frozenset)
+
+
+def resolve(lm):
+ from dspy.clients.engines.lm15_engine import LM15Engine
+ from dspy.lm15 import RouterConfig
+
+ return LM15Engine(RouterConfig(env={}), model_type=lm.model_type).resolve(lm.model)
+
+
+@dataclass(repr=False)
+class _PlanningScope:
+ lm: object
+ options: dict
+ value: Capabilities | None = None
+
+
+_planning = ContextVar("dspy_lm_capability_planning", default=None)
+
+
+@contextmanager
+def planning_scope(lm, options):
+ current = _planning.get()
+ if current is not None and current.lm is lm and current.options is options:
+ yield
+ return
+ token = _planning.set(_PlanningScope(lm, options))
+ try:
+ yield
+ finally:
+ _planning.reset(token)
+
+
+def with_capability_planning(fn):
+ """Scope adapter capability reads to this call's client settings.
+
+ Do not copy or mutate the LM: callbacks and custom hooks retain its identity.
+ Nested JSON/base-adapter calls reuse the same task-local planning scope.
+ """
+ if inspect.iscoroutinefunction(fn):
+ @wraps(fn)
+ async def async_wrapper(self, lm, lm_kwargs, *args, **kwargs):
+ with planning_scope(lm, lm_kwargs):
+ await prepare_async(lm)
+ return await fn(self, lm, lm_kwargs, *args, **kwargs)
+ return async_wrapper
+
+ @wraps(fn)
+ def wrapper(self, lm, lm_kwargs, *args, **kwargs):
+ with planning_scope(lm, lm_kwargs):
+ return fn(self, lm, lm_kwargs, *args, **kwargs)
+ return wrapper
+
+
+async def prepare_async(lm):
+ """Resolve built-in capability hints off-loop, including LiteLLM lookup.
+
+ Native calls still never import LiteLLM. Custom/legacy LM capability hooks
+ keep their own contract and are not moved to worker threads.
+ """
+ scope = _planning.get()
+ if scope is not None and scope.lm is lm and scope.value is not None:
+ return
+ if isinstance(getattr(lm, "_engine_spec", None), str):
+ await asyncio.to_thread(capabilities, lm)
+
+
+def capabilities(lm):
+ with error_boundary(lm.model, unexpected=True):
+ scope = _planning.get()
+ if scope is not None and scope.lm is lm:
+ if scope.value is None:
+ scope.value = _capabilities(lm, scope.options)
+ return scope.value
+ return _capabilities(lm)
+
+
+def _litellm_capabilities(lm, selection):
+ from dspy.clients.lm import _get_litellm
+
+ litellm = _get_litellm()
+ provider = selection.clients.get("custom_llm_provider")
+ kwargs = {"custom_llm_provider": provider} if provider is not None else {}
+ with litellm_errors(model=lm.model, provider=provider or lm._provider_name):
+ params = litellm.get_supported_openai_params(model=lm.model, custom_llm_provider=provider or lm._provider_name)
+ return Capabilities(
+ bool(litellm.supports_function_calling(model=lm.model, **kwargs)),
+ bool(litellm.supports_reasoning(lm.model, **kwargs)),
+ bool(litellm.supports_response_schema(model=lm.model, custom_llm_provider=provider or lm._provider_name)),
+ frozenset(params or ()),
+ )
+
+
+def _capabilities(lm, options=None):
+ spec = lm.engine
+ if not isinstance(spec, str):
+ # A custom engine can declare the same hints as BaseLM. Missing
+ # declarations remain false; do not consult an unrelated model catalog.
+ return Capabilities(
+ bool(getattr(spec, "supports_function_calling", False)),
+ bool(getattr(spec, "supports_reasoning", False)),
+ bool(getattr(spec, "supports_response_schema", False)),
+ frozenset(getattr(spec, "supported_params", ()) or ()),
+ )
+ selection = select_backend(lm, options)
+ if not selection.native:
+ return _litellm_capabilities(lm, selection)
+ route = selection.resolution
+ from dspy._vendor.lm15.compat import (
+ AnthropicCompat,
+ OpenAIChatCompat,
+ OpenAIResponsesCompat,
+ resolve_anthropic_compat,
+ resolve_openai_chat_compat,
+ resolve_openai_responses_compat,
+ )
+ from dspy._vendor.lm15.registry import lookup
+
+ definition = lookup(route.provider)
+ info = model_info(route.provider, route.model)
+ tools = info.get("supports_function_calling") is True
+ reasoning = info.get("supports_reasoning") is True
+ schema = info.get("supports_response_schema") is True
+ params = {"temperature", "max_tokens", "top_p", "stream"}
+ # These describe lm15's canonical mappings. Engine-specific refusal still
+ # validates the complete request, including combinations and model limits.
+ if definition.dialect == "openai-chat":
+ preset = definition.compat or ("xai" if route.provider == "xai" else "openai")
+ compat = resolve_openai_chat_compat(OpenAIChatCompat.preset(preset).for_model(route.model))
+ params.update({"stop", "max_completion_tokens", "logprobs", "top_logprobs", "seed",
+ "presence_penalty", "frequency_penalty", "logit_bias", "user"})
+ if info.get("supports_response_schema") is True or info.get("supports_function_calling") is True:
+ params.add("response_format")
+ if compat.json_schema == "reject":
+ schema = False
+ if compat.thinking_format == "none":
+ reasoning = False
+ if route.provider == "xai":
+ params.difference_update({"logprobs", "top_logprobs"})
+ elif definition.dialect == "anthropic":
+ compat = resolve_anthropic_compat(AnthropicCompat.preset(definition.compat or "anthropic"))
+ params.update({"stop", "top_k"})
+ if schema and compat.structured_output != "reject":
+ params.add("response_format")
+ else:
+ schema = False
+ if compat.sampling_params == "reject":
+ params.difference_update({"temperature", "top_p", "top_k"})
+ elif definition.dialect == "gemini":
+ params.update({"stop", "top_k", "logprobs", "top_logprobs", "response_format"})
+ else:
+ compat = resolve_openai_responses_compat(OpenAIResponsesCompat.preset(definition.compat or "openai"))
+ params.update({"response_format", "max_completion_tokens", "logprobs", "top_logprobs", "store"})
+ if compat.reasoning_format == "none":
+ reasoning = False
+ if route.provider == "openai-codex":
+ params.difference_update({"max_tokens", "max_completion_tokens", "store"})
+ if tools:
+ params.update({"tools", "tool_choice", "parallel_tool_calls"})
+ if definition.dialect == "gemini" or (
+ definition.dialect == "anthropic" and compat.parallel_tool_calls == "reject"
+ ):
+ params.discard("parallel_tool_calls")
+ if reasoning:
+ params.add("reasoning_effort")
+ return Capabilities(tools, reasoning, schema, frozenset(params))
diff --git a/dspy/clients/costs.py b/dspy/clients/costs.py
new file mode 100644
index 000000000..834e4dfc7
--- /dev/null
+++ b/dspy/clients/costs.py
@@ -0,0 +1,105 @@
+"""Token-cost estimation from provider-verbatim lm15 usage and upstream prices.
+
+Unknown pricing is not zero. Hosted tools and unmodelled billing dimensions
+return no estimate instead of claiming a complete bill from token prices alone.
+"""
+
+import re
+
+from dspy.clients.model_metadata import model_info, rate, source_info
+
+
+def estimate_cost(response, *, provider, requested_model, request=None):
+ """Return (estimated USD cost, provenance), or (None, reason metadata).
+
+ Advisory metadata must never turn a completed generation into a failure.
+ """
+ try:
+ return _estimate_cost(response, provider=provider, requested_model=requested_model, request=request)
+ except Warning:
+ raise
+ except Exception:
+ return None, {"kind": "unknown", "reason": "pricing metadata could not be interpreted"}
+
+
+def _estimate_cost(response, *, provider, requested_model, request=None):
+ from dspy.lm15 import BuiltinTool
+
+ def unknown(reason):
+ return None, {"kind": "unknown", "reason": reason}
+
+ if provider is None:
+ return unknown("custom engine has no pricing provider")
+ if any(part.type in {"image", "audio", "video", "document", "binary"} for part in response.message.parts):
+ return unknown("generated-media billing is not covered by text token rates")
+ if provider in {"claude-code", "openai-codex"}:
+ return unknown("subscription usage is not per-token API billing")
+ if request is not None and any(isinstance(tool, BuiltinTool) for tool in request.tools):
+ return unknown("hosted-tool fees are not included in token rates")
+ info = model_info(provider, response.model)
+ priced_model = response.model
+ if not info:
+ info = model_info(provider, requested_model)
+ priced_model = requested_model
+ u = response.usage
+ if u.input_tokens is None or u.output_tokens is None:
+ return unknown("input or output usage was not reported")
+ anthropic = provider in {"anthropic", "aws-anthropic", "bedrock-anthropic", "azure-anthropic",
+ "vertex-anthropic", "deepseek-anthropic", "meta-anthropic", "moonshotai-anthropic"}
+ separate_reasoning = provider in {"gemini", "vertex", "vertex-express", "xai"}
+ cached, written = u.cache_read_tokens or 0, u.cache_write_tokens or 0
+ inputs = u.input_tokens if anthropic else u.input_tokens - cached - written
+ reasoning = u.reasoning_tokens or 0
+ outputs = u.output_tokens if separate_reasoning else u.output_tokens - reasoning
+ input_audio, output_audio = u.input_audio_tokens or 0, u.output_audio_tokens or 0
+ inputs -= input_audio
+ outputs -= output_audio
+ if min(inputs, outputs) < 0:
+ return unknown("usage dimensions cannot be separated safely")
+ total_input = u.input_tokens + cached + written if anthropic else u.input_tokens
+ tier = request.config.service_tier if request is not None else None
+ # Prefer the provider's actual served tier when it reports one.
+ raw = response.provider_data or {}
+ tier = raw.get("service_tier") or (raw.get("usageMetadata") or {}).get("serviceTier") or tier
+ tier = {"default": None, "standard": None, "auto": None, "ON_DEMAND": None}.get(tier, tier)
+ if tier not in (None, "priority", "flex"):
+ return unknown("unrecognised service-tier pricing")
+ suffix = f"_{tier}" if tier else ""
+ thresholds = sorted({int(m.group(1)) * 1000 for key in info
+ if (m := re.search(r"_above_(\d+)k_tokens", key))})
+ threshold = max((n for n in thresholds if total_input > n), default=None)
+ long_context = f"_above_{threshold // 1000}k_tokens" if threshold else ""
+
+ def price(key, *, fallback=None, long_write=False):
+ duration = "_above_1hr" if long_write else ""
+ candidate = key + duration + long_context + suffix
+ found = rate(info, candidate)
+ if found is not None:
+ return found
+ # Reasoning/audio/cache use their own rate where supplied; otherwise
+ # use the matching tier's ordinary token rate, never a cheaper tier.
+ if fallback and not duration:
+ found = rate(info, fallback + long_context + suffix)
+ if found is not None:
+ return found
+ return None
+
+ long_write = bool(request is not None and request.config.cache is not None
+ and request.config.cache.retention == "long" and anthropic)
+ dimensions = [
+ (inputs, price("input_cost_per_token")),
+ (outputs, price("output_cost_per_token")),
+ (reasoning, price("output_cost_per_reasoning_token", fallback="output_cost_per_token")),
+ (cached, price("cache_read_input_token_cost", fallback="input_cost_per_token")),
+ (written, price("cache_creation_input_token_cost", fallback="input_cost_per_token", long_write=long_write)),
+ (input_audio, price("input_cost_per_audio_token", fallback="input_cost_per_token")),
+ (output_audio, price("output_cost_per_audio_token", fallback="output_cost_per_token")),
+ ]
+ if any(count and amount is None for count, amount in dimensions):
+ return unknown("a billed token dimension has no known rate")
+ if not info:
+ return unknown("no matching pricing entry")
+ return sum(count * amount for count, amount in dimensions if count and amount is not None), {
+ "kind": "estimate", "currency": "USD", "provider": provider, "model": priced_model,
+ "metadata": source_info(),
+ }
diff --git a/dspy/clients/engines/__init__.py b/dspy/clients/engines/__init__.py
new file mode 100644
index 000000000..96c6aee2d
--- /dev/null
+++ b/dspy/clients/engines/__init__.py
@@ -0,0 +1,22 @@
+"""Execution backends for DSPy's LM layer.
+
+The outer LM layer selects these engines and owns caching, retries, fan-out
+and bookkeeping. No engine silently delegates a failed request to another
+backend.
+"""
+
+from dspy.clients.engines.base import AsyncEngine, Engine
+from dspy.clients.engines.legacy_engine import AsyncLegacyEngine, LegacyEngine
+from dspy.clients.engines.litellm_engine import AsyncLiteLLMEngine, LiteLLMEngine
+from dspy.clients.engines.lm15_engine import AsyncLM15Engine, LM15Engine
+
+__all__ = [
+ "Engine",
+ "AsyncEngine",
+ "LM15Engine",
+ "AsyncLM15Engine",
+ "LiteLLMEngine",
+ "AsyncLiteLLMEngine",
+ "LegacyEngine",
+ "AsyncLegacyEngine",
+]
diff --git a/dspy/clients/engines/base.py b/dspy/clients/engines/base.py
new file mode 100644
index 000000000..37c569d45
--- /dev/null
+++ b/dspy/clients/engines/base.py
@@ -0,0 +1,51 @@
+"""One-attempt execution interfaces for DSPy's LM layer.
+
+Engines consume one canonical request and produce one response. They do not
+own DSPy's cache, retries, candidate fan-out, history or callbacks. A stream
+emits one leading start event and one final end event only on successful completion.
+Expected failures raise dspy.lm15 errors. Canonical StreamErrorEvents are also
+accepted as terminal failures by DSPy's shared stream guard, never as Responses.
+SDK-specific errors are translated by the owning engine; engines do not import
+DSPy's public error hierarchy or implement a second retry loop.
+
+DSPy 3.5 uses this contract throughout adapters and engines. Legacy forward()
+plugins, LegacyEngine/AsyncLegacyEngine, and complete_legacy() shortcuts are
+3.4 transition interfaces, deprecated for removal in 3.5.
+"""
+
+from collections.abc import AsyncIterator, Iterator
+from typing import Protocol
+
+from dspy._vendor.lm15.types import StreamEvent
+from dspy.lm15 import Request, Response
+
+
+class Engine(Protocol):
+ def complete(self, request: Request) -> Response: ...
+
+ def stream(self, request: Request) -> Iterator[StreamEvent]: ...
+
+ def close(self) -> None: ...
+
+
+class AsyncEngine(Protocol):
+ async def complete(self, request: Request) -> Response: ...
+
+ def stream(self, request: Request) -> AsyncIterator[StreamEvent]: ...
+
+ async def aclose(self) -> None: ...
+
+
+def validate_request(request: Request) -> None:
+ if not isinstance(request, Request):
+ raise TypeError("An engine requires a dspy.lm15.Request")
+ # Execution controls may not bypass the single-response contract through
+ # provider passthrough fields. They are owned by the outer DSPy LM layer.
+ forbidden = {
+ "n", "num_generations", "stream", "stream_options", "num_retries", "max_retries",
+ "retry_strategy", "cache", "caching", "rollout_id", "model", "messages", "input",
+ "api_key", "api_base", "base_url", "headers", "extra_headers", "timeout",
+ }
+ extra = forbidden.intersection(request.config.extensions or {})
+ if extra:
+ raise ValueError(f"Engine execution controls do not belong in Config.extensions: {sorted(extra)}")
diff --git a/dspy/clients/engines/dummy_engine.py b/dspy/clients/engines/dummy_engine.py
new file mode 100644
index 000000000..df82c1ebe
--- /dev/null
+++ b/dspy/clients/engines/dummy_engine.py
@@ -0,0 +1,76 @@
+"""Canonical engine used by DummyLM; no provider SDK objects are needed."""
+
+from dspy.clients.engines.base import validate_request
+from dspy.lm15 import Message, Response, TextPart, ThinkingPart, Usage, response_to_events
+
+
+class DummyEngine:
+ def __init__(self, owner):
+ self.owner = owner
+
+ def complete(self, request):
+ validate_request(request)
+ messages = []
+ if request.system is not None:
+ system = request.system if isinstance(request.system, str) else "".join(p.text for p in request.system)
+ messages.append({"role": "system", "content": system})
+ messages.extend({"role": m.role, "content": m.text or ""} for m in request.messages)
+ return self._complete_messages(messages)
+
+ def complete_legacy(self, lm, request, *, prompt=None, messages=None, **context):
+ """Scripted answers do not require provider-compatible input blocks.
+
+ Keep the original messages (including filenames and opaque blocks).
+ The dummy generates canonical responses, but does not send inputs to a
+ provider or pretend those inputs round-trip through a narrower schema.
+ """
+ from dspy.clients.call_result import CallResult, combine
+
+ if messages is None and isinstance(prompt, (list, tuple)) and all(
+ isinstance(message, dict) and "role" in message for message in prompt
+ ):
+ messages = prompt
+ else:
+ messages = request["messages"]
+ count = request.get("n") or 1
+ results = [CallResult.native(self._complete_messages(messages)) for _ in range(count)]
+ return results[0] if count == 1 else combine(results, model_type="chat")
+
+ def _complete_messages(self, messages):
+ owner = self.owner
+ if owner.follow_examples:
+ output = owner._use_example(messages)
+ elif isinstance(owner.answers, dict):
+ output = next((owner._format_answer_fields(v) for k, v in owner.answers.items()
+ if k in messages[-1]["content"]), "No more responses")
+ else:
+ output = owner._format_answer_fields(next(owner.answers, {"answer": "No more responses"}))
+ parts = [TextPart(output or "")]
+ if owner.reasoning:
+ parts.insert(0, ThinkingPart("Some reasoning"))
+ return Response(id=None, model="dummy", message=Message.assistant(parts), finish_reason="stop",
+ usage=Usage(input_tokens=0, output_tokens=0, total_tokens=0))
+
+ def stream(self, request):
+ return response_to_events(self.complete(request))
+
+ def close(self):
+ pass
+
+
+class AsyncDummyEngine:
+ def __init__(self, sync):
+ self.sync = sync
+
+ async def complete(self, request):
+ return self.sync.complete(request)
+
+ async def complete_legacy(self, lm, request, **context):
+ return self.sync.complete_legacy(lm, request, **context)
+
+ async def stream(self, request):
+ for event in self.sync.stream(request):
+ yield event
+
+ async def aclose(self):
+ pass
diff --git a/dspy/clients/engines/legacy_engine.py b/dspy/clients/engines/legacy_engine.py
new file mode 100644
index 000000000..071184756
--- /dev/null
+++ b/dspy/clients/engines/legacy_engine.py
@@ -0,0 +1,123 @@
+"""Adapt a legacy custom LM without invoking BaseLM's public-call bookkeeping."""
+
+from dspy.clients._deprecation import warn_legacy_engine
+from dspy.clients.engines.base import validate_request
+from dspy.clients.errors import wrap_error
+from dspy.clients.lm15_boundary import request_kwargs, response_value
+from dspy.lm15 import Request, Response
+from dspy.utils.exceptions import LMError, LMUnsupportedFeatureError
+
+
+class _LegacyConfig:
+ """Borrow a plugin; its runtime state and lifecycle remain caller-owned.
+
+ No retries or caching are added here. Third-party forward implementations
+ may perform their own; this wrapper cannot disable undocumented behavior.
+ """
+
+ def __init__(self, lm, *, _implicit=False):
+ from dspy.clients.base_lm import BaseLM
+ from dspy.clients.lm import LM
+
+ if not isinstance(lm, BaseLM):
+ raise TypeError("LegacyEngine requires a BaseLM instance")
+ if type(lm) is LM and not {"forward", "aforward"}.intersection(vars(lm)):
+ raise TypeError("Use LiteLLMEngine for the built-in LM, not a nested LegacyEngine")
+ if getattr(type(lm), "forward_contract", "legacy") != "legacy":
+ raise TypeError("The removed DSPy 3.3 typed_lm contract is not a legacy plugin")
+ if not _implicit:
+ # Automatic wrapping already warns at the public LM call boundary.
+ warn_legacy_engine()
+ self.lm = lm
+ self._closed = False
+
+ def _arguments(self, request):
+ validate_request(request)
+ if self._closed:
+ raise RuntimeError("Engine is closed")
+ if request.model != self.lm.model:
+ raise ValueError("Request.model must match the wrapped legacy LM")
+ data = request_kwargs(request, "chat")
+ messages = data.pop("messages")
+ # Explicitly neutralize absent generation defaults; do not mutate the
+ # plugin or make a runtime copy that would lose its call-state updates.
+ client_keys = {
+ "api_key", "api_base", "base_url", "api_version", "headers", "extra_headers",
+ "timeout", "azure_ad_token_provider", "organization", "project",
+ }
+ kwargs = {key: None for key in self.lm.kwargs if key not in client_keys}
+ kwargs.update(data)
+ kwargs["n"] = 1
+ return {"prompt": None, "messages": messages, **kwargs}
+
+ def _response(self, raw, request):
+ if isinstance(raw, Response):
+ return raw
+ return response_value(raw, "chat", request)
+
+ def __repr__(self):
+ return f"{type(self).__name__}(plugin={type(self.lm).__name__})"
+
+
+class LegacyEngine(_LegacyConfig):
+ """Deprecated 3.4 transition wrapper, scheduled for removal in DSPy 3.5.
+
+ Migrate the underlying implementation to complete(Request) -> Response.
+ Wrapping it explicitly does not extend the legacy interface's lifetime.
+ """
+
+ def complete_legacy(self, lm, request, *, prompt=None, messages=None, call_kwargs=None):
+ from dspy.clients.call_result import CallResult
+
+ raw = self.lm.forward(prompt=prompt, messages=messages, **(call_kwargs or {}))
+ return CallResult.legacy(self.lm, raw, kwargs=call_kwargs)
+
+ def complete(self, request: Request):
+ kwargs = self._arguments(request)
+ try:
+ return self._response(self.lm.forward(**kwargs), request)
+ except LMError:
+ raise
+ except Exception as exc:
+ raise wrap_error(exc, model=request.model) from exc
+
+ def stream(self, request: Request):
+ validate_request(request)
+ raise LMUnsupportedFeatureError(
+ "Legacy forward() has no standard streaming contract. "
+ "Use the plugin's ordinary streaming path or provide a native streaming engine.",
+ model=request.model, features=["stream"],
+ )
+
+ def close(self):
+ # The plugin may be used elsewhere; never close a borrowed object.
+ self._closed = True
+
+
+class AsyncLegacyEngine(_LegacyConfig):
+ """Deprecated async transition wrapper, scheduled for removal in DSPy 3.5."""
+
+ async def complete_legacy(self, lm, request, *, prompt=None, messages=None, call_kwargs=None):
+ from dspy.clients.call_result import CallResult
+
+ raw = await self.lm.aforward(prompt=prompt, messages=messages, **(call_kwargs or {}))
+ return CallResult.legacy(self.lm, raw, kwargs=call_kwargs)
+
+ async def complete(self, request: Request):
+ kwargs = self._arguments(request)
+ try:
+ return self._response(await self.lm.aforward(**kwargs), request)
+ except LMError:
+ raise
+ except Exception as exc:
+ raise wrap_error(exc, model=request.model) from exc
+
+ def stream(self, request: Request):
+ validate_request(request)
+ raise LMUnsupportedFeatureError(
+ "Legacy aforward() has no standard streaming contract.",
+ model=request.model, features=["stream"],
+ )
+
+ async def aclose(self):
+ self._closed = True
diff --git a/dspy/clients/engines/lifecycle.py b/dspy/clients/engines/lifecycle.py
new file mode 100644
index 000000000..45c543692
--- /dev/null
+++ b/dspy/clients/engines/lifecycle.py
@@ -0,0 +1,61 @@
+"""Release stream sources without replacing an active failure or cancellation."""
+
+import inspect
+from contextlib import asynccontextmanager, contextmanager
+
+import anyio
+
+
+def _secondary(primary, cleanup):
+ # Keep diagnostics on the original error across Python 3.10-3.14. Do not
+ # interpolate exception messages: transport errors may contain credentials.
+ try:
+ primary.cleanup_errors = (*getattr(primary, "cleanup_errors", ()), cleanup)
+ if hasattr(primary, "add_note"):
+ primary.add_note(f"Stream cleanup also failed ({type(cleanup).__name__}); see cleanup_errors.")
+ except Exception:
+ # An exception type may prohibit extra attributes. Preserving the
+ # original failure takes precedence over attaching optional diagnostics.
+ pass
+
+
+@contextmanager
+def closing_stream(source):
+ primary = None
+ try:
+ yield source
+ except BaseException as exc:
+ primary = exc
+ raise
+ finally:
+ try:
+ close = getattr(source, "close", None)
+ if close is not None:
+ close()
+ except Exception as cleanup:
+ if primary is None or isinstance(primary, GeneratorExit):
+ # Let the outer closer attach this to its actual failure.
+ raise
+ _secondary(primary, cleanup)
+
+
+@asynccontextmanager
+async def aclosing_stream(source):
+ primary = None
+ try:
+ yield source
+ except BaseException as exc:
+ primary = exc
+ raise
+ finally:
+ try:
+ close = getattr(source, "aclose", None) or getattr(source, "close", None)
+ if close is not None:
+ with anyio.CancelScope(shield=True):
+ result = close()
+ if inspect.isawaitable(result):
+ await result
+ except Exception as cleanup:
+ if primary is None or isinstance(primary, GeneratorExit):
+ raise
+ _secondary(primary, cleanup)
diff --git a/dspy/clients/engines/litellm_engine.py b/dspy/clients/engines/litellm_engine.py
new file mode 100644
index 000000000..b233ea256
--- /dev/null
+++ b/dspy/clients/engines/litellm_engine.py
@@ -0,0 +1,227 @@
+"""LiteLLM behind the same single-response interface as native lm15 engines."""
+
+import json
+from dataclasses import replace
+
+from dspy._vendor.lm15.sse import SSEEvent
+from dspy.clients._litellm import get_litellm
+from dspy.clients.call_context import completed_legacy
+from dspy.clients.engines.base import validate_request
+from dspy.clients.engines.lifecycle import aclosing_stream, closing_stream
+from dspy.clients.engines.litellm_errors import litellm_errors
+from dspy.clients.legacy_outputs import plain
+from dspy.clients.lm15_boundary import request_kwargs, response_value
+from dspy.lm15 import (
+ ConfigurationError,
+ OpenAIChatLM,
+ ProviderError,
+ Request,
+ StreamAssemblyError,
+ StreamEndEvent,
+ StreamStartEvent,
+ UnsupportedFeatureError,
+)
+
+
+class _ChatStream:
+ """Map chunks with the bundled dialect while keeping part indices distinct."""
+
+ def __init__(self, request):
+ self.request = request
+ self.dialect = OpenAIChatLM(api_key="conversion-only", transport=None)
+ self.indices = {}
+ self.started = False
+ self.terminal = False
+ self.chunks = []
+
+ def feed(self, chunk):
+ body = plain(chunk)
+ choices = body.get("choices") or []
+ if len(choices) > 1 or any(choice.get("index", 0) != 0 for choice in choices):
+ raise ProviderError("A single-response engine received multiple streaming candidates.")
+ for choice in choices:
+ delta = choice.get("delta") or {}
+ supported = {"role", "content", "reasoning_content", "reasoning", "tool_calls", "provider_specific_fields"}
+ unknown = [key for key, val in delta.items() if key not in supported and val not in (None, "", [], {})]
+ extras = delta.get("provider_specific_fields") or {}
+ unknown.extend(f"provider_specific_fields.{key}" for key, val in extras.items()
+ if key != "citation" and val not in (None, "", [], {}))
+ if unknown:
+ raise UnsupportedFeatureError(
+ f"LiteLLM streaming fields have no implemented lm15 event mapping: {sorted(unknown)}",
+ )
+ self.chunks.append(chunk)
+ if not self.started:
+ self.started = True
+ yield StreamStartEvent(id=body.get("id") or None, model=body.get("model") or self.request.model)
+ if choices and choices[0].get("finish_reason") is not None:
+ self.terminal = True
+ for event in self.dialect.parse_stream_events(
+ self.request, SSEEvent(event=None, data=json.dumps(body)),
+ ):
+ if event.type == "error":
+ from dspy._vendor.lm15.errors import error_class_for_code
+
+ raise error_class_for_code(event.error.code)(event.error.message, provider_code=event.error.provider_code)
+ if event.type != "delta":
+ continue
+ delta = event.delta
+ key = (delta.type, delta.part_index)
+ index = self.indices.setdefault(key, len(self.indices))
+ yield replace(event, delta=replace(delta, part_index=index))
+ # LiteLLM exposes Anthropic citations as a provider-specific delta.
+ for choice in choices:
+ delta = choice.get("delta") or {}
+ citation = (delta.get("provider_specific_fields") or {}).get("citation")
+ if isinstance(citation, dict):
+ from dspy.lm15 import CitationDelta, StreamDeltaEvent
+
+ text = citation.get("cited_text") or citation.get("text") or citation.get("supported_text")
+ title = citation.get("document_title") or citation.get("title")
+ url = citation.get("url")
+ if not any((text, title, url)):
+ raise UnsupportedFeatureError("This citation chunk has no lm15 text, title or URL representation.")
+ index = self.indices.setdefault(("citation", len(self.chunks)), len(self.indices))
+ yield StreamDeltaEvent(CitationDelta(text=text, title=title, url=url, part_index=index))
+
+ def finish(self, litellm):
+ if not self.terminal:
+ raise StreamAssemblyError("LiteLLM stream ended without a completion reason; no successful end event emitted.")
+ raw = litellm.stream_chunk_builder(self.chunks)
+ response = response_value(raw, "chat", self.request)
+ return StreamEndEvent(
+ finish_reason=response.finish_reason, usage=response.usage, provider_data=response.provider_data,
+ )
+
+
+class _LiteLLMConfig:
+ def __init__(self, *, model_type="chat", **client_options):
+ if model_type not in {"chat", "responses", "text"}:
+ raise UnsupportedFeatureError(
+ "Typed LiteLLM engines support chat and Responses APIs. Use ordinary LM calls for text completions."
+ )
+ allowed = {
+ "api_key", "api_base", "base_url", "api_version", "organization", "project",
+ "headers", "extra_headers", "extra_query", "timeout", "azure_ad_token_provider",
+ "custom_llm_provider",
+ }
+ unknown = set(client_options) - allowed
+ if unknown:
+ raise TypeError(f"Unknown engine client options: {sorted(unknown)}. Generation options belong in Request.config.")
+ self.model_type = model_type
+ self.client_options = dict(client_options)
+ self._closed = False
+
+ def _arguments(self, request, *, streaming=False):
+ validate_request(request)
+ if self._closed:
+ raise ConfigurationError("Engine is closed")
+ if self.model_type == "text":
+ raise UnsupportedFeatureError("Text-completion models accept ordinary prompt/messages calls, not typed requests.")
+ if streaming and self.model_type != "chat":
+ raise UnsupportedFeatureError(
+ "LiteLLMEngine streaming currently requires model_type='chat'. "
+ "Use the native Responses engine for Responses streaming.",
+ )
+ data = request_kwargs(request, self.model_type)
+ data.update(self.client_options)
+ data.update(model=request.model, num_retries=0, cache={"no-cache": True, "no-store": True})
+ if self.model_type == "chat":
+ data["n"] = 1
+ if streaming:
+ data["stream"] = True
+ data["stream_options"] = {"include_usage": True}
+ return data
+
+ def __repr__(self):
+ return f"{type(self).__name__}(model_type={self.model_type!r})"
+
+
+class LiteLLMEngine(_LiteLLMConfig):
+ """One synchronous attempt. LiteLLM's process-global clients are borrowed."""
+
+ def complete_legacy(self, lm, request, **context):
+ """Carry ordinary provider-specific inputs without a lossy typed conversion."""
+ from dspy.clients import lm as lm_module
+ from dspy.clients.call_result import CallResult
+
+ fn = {"chat": lm_module.litellm_completion, "text": lm_module.litellm_text_completion,
+ "responses": lm_module.litellm_responses_completion}[self.model_type]
+ with litellm_errors(model=lm.model):
+ raw = fn(request=request, num_retries=0)
+ completed_legacy(raw)
+ lm._check_truncation(raw)
+ return CallResult.legacy(lm, raw, kwargs=request)
+
+ def complete(self, request: Request):
+ data = self._arguments(request)
+ litellm = get_litellm(feature="LiteLLM engine")
+ fn = litellm.responses if self.model_type == "responses" else litellm.completion
+ with litellm_errors(model=request.model):
+ raw = fn(**data)
+ return response_value(raw, self.model_type, request)
+
+ def stream(self, request: Request):
+ data = self._arguments(request, streaming=True)
+ litellm = get_litellm(feature="LiteLLM engine streaming")
+ with litellm_errors(model=request.model):
+ source = litellm.completion(**data)
+ codec = _ChatStream(request)
+ with closing_stream(source):
+ iterator = iter(source)
+ while True:
+ with litellm_errors(model=request.model):
+ try:
+ chunk = next(iterator)
+ except StopIteration:
+ break
+ yield from codec.feed(chunk)
+ yield codec.finish(litellm)
+
+ def close(self):
+ self._closed = True
+
+
+class AsyncLiteLLMEngine(_LiteLLMConfig):
+ """Async equivalent; cancellation propagates without retrying."""
+
+ async def complete_legacy(self, lm, request, **context):
+ from dspy.clients import lm as lm_module
+ from dspy.clients.call_result import CallResult
+
+ fn = {"chat": lm_module.alitellm_completion, "text": lm_module.alitellm_text_completion,
+ "responses": lm_module.alitellm_responses_completion}[self.model_type]
+ with litellm_errors(model=lm.model):
+ raw = await fn(request=request, num_retries=0)
+ completed_legacy(raw)
+ lm._check_truncation(raw)
+ return CallResult.legacy(lm, raw, kwargs=request)
+
+ async def complete(self, request: Request):
+ data = self._arguments(request)
+ litellm = get_litellm(feature="LiteLLM engine")
+ fn = litellm.aresponses if self.model_type == "responses" else litellm.acompletion
+ with litellm_errors(model=request.model):
+ raw = await fn(**data)
+ return response_value(raw, self.model_type, request)
+
+ async def stream(self, request: Request):
+ data = self._arguments(request, streaming=True)
+ litellm = get_litellm(feature="LiteLLM engine streaming")
+ with litellm_errors(model=request.model):
+ source = await litellm.acompletion(**data)
+ codec = _ChatStream(request)
+ async with aclosing_stream(source):
+ iterator = source.__aiter__()
+ while True:
+ with litellm_errors(model=request.model):
+ try:
+ chunk = await iterator.__anext__()
+ except StopAsyncIteration:
+ break
+ for event in codec.feed(chunk):
+ yield event
+ yield codec.finish(litellm)
+
+ async def aclose(self):
+ self._closed = True
diff --git a/dspy/clients/engines/litellm_errors.py b/dspy/clients/engines/litellm_errors.py
new file mode 100644
index 000000000..b2c97eba2
--- /dev/null
+++ b/dspy/clients/engines/litellm_errors.py
@@ -0,0 +1,125 @@
+"""LiteLLM/SDK exception interpretation, confined to the compatibility engine."""
+
+import sys
+from contextlib import contextmanager
+
+from dspy import lm15
+from dspy.clients._http import finite_seconds, retry_after_seconds
+
+
+def _header(headers, name):
+ # SDKs use both case-insensitive header objects and plain mappings.
+ items = headers.items() if hasattr(headers, "items") else headers or ()
+ return next((value for key, value in items if str(key).lower() == name), None)
+
+
+def _structured_class(provider, code):
+ # Refine generic SDK errors with documented provider codes, never prose.
+ # In particular an OpenAI quota failure also travels as HTTP 429.
+ if provider in {"openai", "openai-chat", "azure", "azure-chat"}:
+ return {
+ "insufficient_quota": lm15.BillingError,
+ "context_length_exceeded": lm15.ContextLengthError,
+ "model_not_found": lm15.UnsupportedModelError,
+ "DeploymentNotFound": lm15.UnsupportedModelError,
+ "invalid_api_key": lm15.AuthError,
+ }.get(code)
+ if provider == "anthropic":
+ return {
+ "billing_error": lm15.BillingError,
+ "authentication_error": lm15.AuthError,
+ "permission_error": lm15.AuthError,
+ }.get(code)
+ return None
+
+
+def to_lm15_error(exc: Exception, *, model=None, provider=None) -> Exception:
+ """Specific SDK classes and provider codes win over status; never use prose.
+
+ Unknown exceptions without an HTTP status stay unknown. The DSPy engine
+ boundary handles those as unexpected failures, preserving the original cause.
+ This function is only for errors raised by the compatibility backend, not
+ arbitrary custom engines. It never imports LiteLLM to classify an error.
+ """
+ if isinstance(exc, (lm15.LM15Error, Warning, ImportError)):
+ return exc
+ module = sys.modules.get("litellm")
+ mappings = (
+ ("ContextWindowExceededError", lm15.ContextLengthError),
+ ("BudgetExceededError", lm15.BillingError),
+ ("RouterRateLimitError", lm15.RateLimitError),
+ ("UnsupportedParamsError", lm15.UnsupportedFeatureError),
+ ("AuthenticationError", lm15.AuthError),
+ ("PermissionDeniedError", lm15.AuthError),
+ ("Timeout", lm15.TimeoutError),
+ ("APIConnectionError", lm15.TransportError),
+ ("APIResponseValidationError", lm15.ProviderError),
+ ("RateLimitError", lm15.RateLimitError),
+ ("NotFoundError", lm15.UnsupportedModelError),
+ ("UnprocessableEntityError", lm15.InvalidRequestError),
+ ("ContentPolicyViolationError", lm15.InvalidRequestError),
+ ("BadRequestError", lm15.InvalidRequestError),
+ ("InvalidRequestError", lm15.InvalidRequestError),
+ ("InternalServerError", lm15.ServerError),
+ ("ServiceUnavailableError", lm15.ServerError),
+ ("BadGatewayError", lm15.ServerError),
+ )
+ target = None
+ for name, canonical in mappings:
+ cls = getattr(module, name, None)
+ if isinstance(cls, type) and isinstance(exc, cls):
+ target = canonical
+ break
+ response = getattr(exc, "response", None)
+ status = getattr(exc, "status_code", None)
+ if status is None:
+ status = getattr(response, "status_code", None)
+ if isinstance(status, str) and status.isdecimal():
+ status = int(status)
+ if isinstance(status, bool) or not isinstance(status, int) or not 100 <= status <= 599:
+ status = None
+ if target is None and status is None:
+ return exc
+ headers = getattr(response, "headers", None) or getattr(exc, "headers", None) or {}
+ body = getattr(exc, "body", None)
+ detail = body.get("error", body) if isinstance(body, dict) else {}
+ code = detail.get("code") if isinstance(detail, dict) else None
+ if code is None and isinstance(detail, dict):
+ code = detail.get("type")
+ code = str(code) if code is not None else None
+ provider = getattr(exc, "llm_provider", None) or provider
+ if target in (None, lm15.ProviderError, lm15.InvalidRequestError, lm15.RateLimitError, lm15.ServerError):
+ target = _structured_class(provider, code) or target
+ message = str(getattr(exc, "message", None) or exc)
+ metadata = {
+ "provider": provider,
+ "provider_code": code,
+ "request_id": getattr(exc, "request_id", None) or next((
+ value for name in ("x-request-id", "request-id", "x-amzn-requestid", "x-amz-request-id", "x-ms-request-id")
+ if (value := _header(headers, name)) is not None
+ ), None),
+ "retry_after": finite_seconds(getattr(exc, "retry_after", None)),
+ }
+ if metadata["retry_after"] is None:
+ metadata["retry_after"] = retry_after_seconds(_header(headers, "retry-after"))
+ if target is None:
+ from dspy._vendor.lm15.errors import map_http_error
+
+ wrapped = map_http_error(status, message, **metadata)
+ else:
+ wrapped = target(message, status=status, **metadata)
+ # lm15's generic errors have no model constructor argument. Keep the SDK's
+ # model label for the existing DSPy public metadata projection.
+ wrapped.model = getattr(exc, "model", None) or model
+ return wrapped
+
+
+@contextmanager
+def litellm_errors(*, model=None, provider=None):
+ try:
+ yield
+ except Exception as exc:
+ mapped = to_lm15_error(exc, model=model, provider=provider)
+ if mapped is exc:
+ raise
+ raise mapped from exc
diff --git a/dspy/clients/engines/lm15_engine.py b/dspy/clients/engines/lm15_engine.py
new file mode 100644
index 000000000..fd15705a0
--- /dev/null
+++ b/dspy/clients/engines/lm15_engine.py
@@ -0,0 +1,139 @@
+"""Native lm15 routing, with canonical errors and no retry policy."""
+
+import threading
+from collections.abc import AsyncIterator, Iterator
+from contextlib import AsyncExitStack, ExitStack
+from dataclasses import replace
+
+from dspy._vendor.lm15.router import LITELLM_PROVIDER_PREFIXES
+from dspy._vendor.lm15.types import StreamEvent
+from dspy.clients.engines.base import validate_request
+from dspy.clients.engines.lifecycle import aclosing_stream, closing_stream
+from dspy.clients.engines.stream_guard import achecked_stream, checked_stream
+from dspy.lm15 import (
+ AsyncLMRouter,
+ ConfigurationError,
+ LMRouter,
+ Request,
+ Response,
+ RouterConfig,
+ UnsupportedFeatureError,
+)
+
+
+def _model_string(model: str, model_type: str) -> str:
+ """Keep endpoint selection deliberate, including model ids containing ':' ."""
+ head, separator, rest = model.partition("/")
+ if separator and head in LITELLM_PROVIDER_PREFIXES:
+ provider = LITELLM_PROVIDER_PREFIXES[head]
+ if model_type == "responses":
+ provider = {"openai-chat": "openai", "azure-chat": "azure"}.get(provider, provider)
+ return f"{provider}:{rest}"
+ return model
+
+
+class _Routing:
+ def _init_routing(self, config, model_type):
+ if model_type not in {"chat", "responses"}:
+ raise UnsupportedFeatureError("lm15 engines support chat and Responses APIs, not text completions.")
+ self.model_type = model_type
+ self.config = config if config is not None else RouterConfig()
+ self._closed = False
+ self._lock = threading.RLock()
+ self._providers = {}
+
+ def resolve(self, model: str):
+ """Offline routing. Callers inspect canonical errors, not a wrapped cause."""
+ routed = _model_string(model, self.model_type)
+ if self.model_type == "chat":
+ return self.router.resolve_openai_chat(routed)
+ resolution = self.router.resolve(routed)
+ from dspy._vendor.lm15.registry import lookup
+
+ definition = lookup(resolution.provider)
+ if definition is None or definition.dialect != "openai-responses":
+ raise UnsupportedFeatureError(f"{model!r} does not select a Responses API endpoint.")
+ return resolution
+
+ def _target(self, request, *, streaming=False):
+ validate_request(request)
+ resolution = self.resolve(request.model)
+ with self._lock:
+ if self._closed:
+ raise ConfigurationError("Engine is closed")
+ # Serialize lazy construction so parallel first calls do not leak
+ # duplicate owned transports. Borrowed transports stay caller-owned.
+ provider = self._providers.get(resolution.provider)
+ if provider is None:
+ provider = self.router.lm(f"{resolution.provider}:{resolution.model}")
+ self._providers[resolution.provider] = provider
+ surface = "stream" if streaming else "complete"
+ if not getattr(provider.supports, surface):
+ raise UnsupportedFeatureError(
+ f"{resolution.provider} does not support {surface}.", provider=resolution.provider,
+ )
+ return provider, replace(request, model=resolution.model)
+
+
+class LM15Engine(_Routing):
+ """One synchronous native attempt. Close after its active calls finish."""
+
+ def __init__(self, config: RouterConfig | None = None, *, model_type="chat"):
+ self._init_routing(config, model_type)
+ self.router = LMRouter(self.config)
+
+ def complete(self, request: Request) -> Response:
+ provider, routed = self._target(request)
+ return provider.complete(routed)
+
+ def stream(self, request: Request) -> Iterator[StreamEvent]:
+ provider, routed = self._target(request, streaming=True)
+ events = checked_stream(provider.stream(routed), provider=provider.provider)
+ with closing_stream(events):
+ yield from events
+
+ def close(self):
+ with self._lock:
+ self._closed = True
+ providers = list(self._providers.values())
+ self._providers.clear()
+ self.router._lms.clear()
+ if self.config.transport is None:
+ with ExitStack() as cleanup:
+ for provider in providers:
+ cleanup.callback(provider.close)
+
+
+class AsyncLM15Engine(_Routing):
+ """One async attempt; owned pools are confined to their event loop."""
+
+ def __init__(self, config: RouterConfig | None = None, *, model_type="chat"):
+ self._init_routing(config, model_type)
+ self.router = AsyncLMRouter(self.config)
+
+ async def complete(self, request: Request) -> Response:
+ import asyncio
+
+ # Construction may load or refresh credentials synchronously.
+ provider, routed = await asyncio.to_thread(self._target, request)
+ return await provider.complete(routed)
+
+ async def stream(self, request: Request) -> AsyncIterator[StreamEvent]:
+ import asyncio
+
+ provider, routed = await asyncio.to_thread(self._target, request, streaming=True)
+ events = achecked_stream(provider.stream(routed), provider=provider.provider)
+ async with aclosing_stream(events):
+ async for event in events:
+ yield event
+
+ async def aclose(self):
+ with self._lock:
+ self._closed = True
+ providers = list(self._providers.values())
+ self._providers.clear()
+ self.router._lms.clear()
+ if self.config.transport is None:
+ async with AsyncExitStack() as cleanup:
+ for provider in providers:
+ cleanup.push_async_callback(provider.aclose)
diff --git a/dspy/clients/engines/stream_guard.py b/dspy/clients/engines/stream_guard.py
new file mode 100644
index 000000000..2b0f905d2
--- /dev/null
+++ b/dspy/clients/engines/stream_guard.py
@@ -0,0 +1,57 @@
+"""One stream protocol guard for native, compatibility and custom engines."""
+
+from dspy._vendor.lm15.errors import error_class_for_code
+from dspy._vendor.lm15.types import STREAM_EVENT_CLASSES
+from dspy.clients.engines.lifecycle import aclosing_stream, closing_stream
+from dspy.lm15 import StreamAssemblyError
+
+
+def error_from_event(event, *, provider=None):
+ return error_class_for_code(event.error.code)(
+ event.error.message, provider=provider, provider_code=event.error.provider_code,
+ )
+
+
+class _Sequence:
+ def __init__(self, provider):
+ self.provider = provider
+ self.started = False
+ self.ended = False
+
+ def accept(self, event):
+ if not isinstance(event, STREAM_EVENT_CLASSES):
+ raise StreamAssemblyError(f"Engine.stream yielded {type(event).__name__}, not an lm15 stream event")
+ if self.ended:
+ raise StreamAssemblyError("Engine stream emitted an event after its final end event")
+ if event.type == "error":
+ raise error_from_event(event, provider=self.provider)
+ if event.type == "start":
+ if self.started:
+ raise StreamAssemblyError("Engine stream emitted more than one start event")
+ self.started = True
+ elif not self.started:
+ raise StreamAssemblyError("Engine stream emitted data before its start event")
+ elif event.type == "end":
+ self.ended = True
+
+ def finish(self):
+ if not self.ended:
+ raise StreamAssemblyError("Engine stream ended without a completion event")
+
+
+def checked_stream(source, *, provider=None):
+ sequence = _Sequence(provider)
+ with closing_stream(source):
+ for event in source:
+ sequence.accept(event)
+ yield event
+ sequence.finish()
+
+
+async def achecked_stream(source, *, provider=None):
+ sequence = _Sequence(provider)
+ async with aclosing_stream(source):
+ async for event in source:
+ sequence.accept(event)
+ yield event
+ sequence.finish()
diff --git a/dspy/clients/engines/streaming.py b/dspy/clients/engines/streaming.py
new file mode 100644
index 000000000..740cf437a
--- /dev/null
+++ b/dspy/clients/engines/streaming.py
@@ -0,0 +1,62 @@
+"""Bridge canonical events to DSPy's existing streaming listeners."""
+
+import json
+
+from dspy.clients.call_result import AttributeDict, attributes
+
+
+class EngineChunk(AttributeDict):
+ """Listener-facing chunk. Canonical assembly uses the original event."""
+
+ def json(self):
+ return json.dumps(self)
+
+
+class ListenerBridge:
+ def __init__(self, model, predict_id=None):
+ self.model = model
+ self.predict_id = predict_id
+ self.id = None
+ self.tools = {}
+
+ def chunk(self, event):
+ if event.type == "start":
+ self.id = event.id
+ self.model = event.model or self.model
+ return None
+ delta = {}
+ finish = None
+ if event.type == "end":
+ finish = "tool_calls" if event.finish_reason == "tool_call" else event.finish_reason
+ elif event.type == "delta":
+ part = event.delta
+ if part.type == "text":
+ delta["content"] = part.text
+ elif part.type == "thinking":
+ delta["reasoning_content"] = part.text
+ elif part.type == "tool_call":
+ index = self.tools.setdefault(part.part_index, len(self.tools))
+ function = {"arguments": part.input}
+ if part.name is not None:
+ function["name"] = part.name
+ call = {"index": index, "type": "function", "function": function}
+ if part.id is not None:
+ call["id"] = part.id
+ delta["tool_calls"] = [call]
+ elif part.type == "citation":
+ delta["provider_specific_fields"] = {"citation": {
+ key: val for key, val in {"text": part.text, "title": part.title, "url": part.url}.items()
+ if val is not None
+ }}
+ elif part.type == "continuation":
+ # Opaque replay state is retained by canonical assembly, not
+ # exposed as visible answer text or mistaken for a tool call.
+ return None
+ else:
+ delta[part.type] = {"data": part.data, "url": part.url, "file_id": part.file_id,
+ "media_type": part.media_type}
+ else:
+ return None
+ delta.setdefault("content", None)
+ return EngineChunk(id=self.id, model=self.model, predict_id=self.predict_id,
+ choices=[attributes({"index": 0, "delta": delta, "finish_reason": finish})])
diff --git a/dspy/clients/errors.py b/dspy/clients/errors.py
new file mode 100644
index 000000000..9ccaa6af1
--- /dev/null
+++ b/dspy/clients/errors.py
@@ -0,0 +1,99 @@
+"""DSPy's application-facing projection of engine failures.
+
+Engines speak dspy.lm15 errors. Only DSPy-owned boundaries translate them into
+public DSPy errors; SDK-specific interpretation belongs to the owning engine.
+"""
+
+from contextlib import contextmanager
+
+from dspy import lm15
+from dspy.utils.exceptions import (
+ ContextWindowExceededError,
+ LMAuthError,
+ LMBillingError,
+ LMConfigurationError,
+ LMError,
+ LMInvalidRequestError,
+ LMLockTimeoutError,
+ LMNotConfiguredError,
+ LMProviderError,
+ LMRateLimitError,
+ LMServerError,
+ LMStreamAssemblyError,
+ LMTimeoutError,
+ LMTransportError,
+ LMUnexpectedError,
+ LMUnsupportedFeatureError,
+ LMUnsupportedModelError,
+)
+
+# Specific classes precede their parents. Unknown future subclasses inherit the
+# nearest known meaning; an unknown root error fails closed without retrying.
+ERROR_MAPPING = (
+ (lm15.ContextLengthError, ContextWindowExceededError),
+ (lm15.AuthError, LMAuthError),
+ (lm15.BillingError, LMBillingError),
+ (lm15.RateLimitError, LMRateLimitError),
+ (lm15.UnsupportedModelError, LMUnsupportedModelError),
+ (lm15.InvalidRequestError, LMInvalidRequestError),
+ (lm15.TimeoutError, LMTimeoutError),
+ (lm15.LockTimeoutError, LMLockTimeoutError),
+ (lm15.StreamAssemblyError, LMStreamAssemblyError),
+ (lm15.ServerError, LMServerError),
+ (lm15.CapabilityError, LMUnsupportedFeatureError),
+ (lm15.NotConfiguredError, LMNotConfiguredError),
+ (lm15.ConfigurationError, LMConfigurationError),
+ (lm15.ToolDerivationError, LMConfigurationError),
+ (lm15.TransportError, LMTransportError),
+ (lm15.ProviderError, LMProviderError),
+)
+
+
+def wrap_error(exc: Exception, *, model: str, provider: str | None = None) -> Exception:
+ """Project a canonical failure without guessing from arbitrary attributes/text.
+
+ Preserve public DSPy errors by identity (including legacy 3.4 plugins).
+ Warnings promoted to errors and missing dependencies remain Python errors.
+ The caller raises the returned exception from the original, retaining its
+ exact canonical code, diagnostics, and any original SDK cause.
+ """
+ if isinstance(exc, (LMError, Warning, ImportError)):
+ return exc
+ if not isinstance(exc, lm15.LM15Error):
+ return LMUnexpectedError(str(exc), model=model, provider=provider)
+ target = next((target for source, target in ERROR_MAPPING if isinstance(exc, source)), LMUnexpectedError)
+ wrapped = target(
+ message=exc.message,
+ model=getattr(exc, "model", None) or model,
+ provider=exc.provider or provider,
+ provider_code=exc.provider_code,
+ status=exc.status,
+ request_id=exc.request_id,
+ retry_after=exc.retry_after,
+ )
+ for name in (
+ "partial", "part_index", "env_keys", "credential_hint", "path", "lock_path",
+ "providers", "candidates", "rules_tried", "catalog_searched", "cleanup_errors",
+ ):
+ if hasattr(exc, name):
+ setattr(wrapped, name, getattr(exc, name))
+ return wrapped
+
+
+@contextmanager
+def error_boundary(model: str, *, provider: str | None = None, unexpected: bool = False):
+ """Translate once at a DSPy boundary, not around arbitrary application code.
+
+ Setup preserves local TypeError/ValueError. The actual engine invocation
+ opts into unexpected failures so an engine bug cannot become a bad model
+ prediction or a parsing fallback. Cancellation is never caught here.
+ """
+ try:
+ yield
+ except Exception as exc:
+ if not unexpected and not isinstance(exc, lm15.LM15Error):
+ raise
+ error = wrap_error(exc, model=model, provider=provider)
+ if error is exc:
+ raise
+ raise error from exc
diff --git a/dspy/clients/execution.py b/dspy/clients/execution.py
new file mode 100644
index 000000000..b1dc8c342
--- /dev/null
+++ b/dspy/clients/execution.py
@@ -0,0 +1,563 @@
+"""One execution path for native, compatibility and custom DSPy LMs."""
+
+import asyncio
+import copy
+import time
+from contextlib import nullcontext
+from dataclasses import dataclass, replace
+from typing import Any
+
+import anyio
+import pydantic
+
+from dspy._vendor.lm15.result import StreamAccumulator
+from dspy._vendor.lm15.serde import request_to_dict
+from dspy.clients._deprecation import warn_legacy_shortcut
+from dspy.clients._http import finite_seconds
+from dspy.clients.backend_selection import CLIENT_KEYS, select_backend
+from dspy.clients.call_context import stream_emitted
+from dspy.clients.call_result import CACHE_FORMAT, CallResult, combine, usage_dict
+from dspy.clients.engines.legacy_engine import AsyncLegacyEngine, LegacyEngine
+from dspy.clients.engines.lifecycle import aclosing_stream, closing_stream
+from dspy.clients.engines.litellm_engine import AsyncLiteLLMEngine, LiteLLMEngine
+from dspy.clients.engines.lm15_engine import AsyncLM15Engine, LM15Engine
+from dspy.clients.engines.stream_guard import achecked_stream, checked_stream
+from dspy.clients.engines.streaming import ListenerBridge
+from dspy.clients.errors import error_boundary
+from dspy.clients.legacy_outputs import plain, value
+from dspy.dsp.utils.settings import settings
+from dspy.lm15 import (
+ CacheConfig,
+ LM15Error,
+ Request,
+ Response,
+ RouterConfig,
+ StreamAssemblyError,
+ request_from_openai_chat,
+)
+from dspy.utils.exceptions import LMUnsupportedFeatureError, is_retryable_lm_error
+
+IGNORED_CACHE_KEYS = ["api_key", "api_base", "base_url"]
+
+
+@dataclass
+class PreparedCall:
+ prompt: Any
+ messages: Any
+ kwargs: dict
+ legacy: dict
+ request: Request | None
+ cache: bool
+ n: int
+ managed: bool
+
+ def key(self, lm, asynchronous):
+ if self.request is not None:
+ return {"_fn_identifier": "dspy.clients.lm15.complete.async" if asynchronous else "dspy.clients.lm15.complete",
+ "request": request_to_dict(self.request), "rollout_id": self.kwargs.get("rollout_id")}
+ suffix = {"chat": "completion", "text": "text_completion", "responses": "responses_completion"}[lm.model_type]
+ name = ("alitellm_" if asynchronous else "litellm_") + suffix
+ key = {**self.legacy, "_fn_identifier": f"dspy.clients.lm.{name}"}
+ prompt_cache = key.pop("prompt_cache", None)
+ if prompt_cache is not None:
+ from dspy._vendor.lm15.serde import cache_config_to_dict
+
+ key["prompt_cache"] = cache_config_to_dict(prompt_cache)
+ return key
+
+
+def prepare(lm, prompt, messages, kwargs, *, asynchronous=False, direct=False):
+ kwargs = dict(kwargs)
+ request = kwargs.pop("request", None)
+ if isinstance(prompt, Request):
+ if request is not None:
+ raise TypeError("Pass a Request once, positionally or by keyword")
+ request, prompt = prompt, None
+ managed = hasattr(lm, "_engine_spec")
+ if managed and not direct:
+ from dspy.clients.lm import LM
+ from dspy.utils.dummies import DummyLM
+
+ base = DummyLM if isinstance(lm, DummyLM) else LM if isinstance(lm, LM) else None
+ if base is not None:
+ method = "aforward" if asynchronous else "forward"
+ managed = method not in vars(lm) and getattr(type(lm), method) is getattr(base, method)
+ if base is DummyLM and asynchronous:
+ # DummyLM's inherited aforward delegates to forward, including
+ # a subclass's override. Do not bypass that override either.
+ managed = managed and "forward" not in vars(lm) and type(lm).forward is DummyLM.forward
+ if getattr(type(lm), "forward_contract", "legacy") != "legacy":
+ raise TypeError("The DSPy 3.3 typed_lm contract was removed; implement an lm15 engine instead.")
+ if request is not None:
+ if not isinstance(request, Request):
+ raise TypeError("request must be a dspy.lm15.Request")
+ if prompt is not None or messages is not None:
+ raise TypeError("Do not combine a Request with prompt/messages")
+ if request.model != lm.model:
+ raise ValueError("Request.model must match LM.model")
+ from dspy.clients.engines.base import validate_request
+
+ validate_request(request)
+ extra = set(kwargs) - {"cache", "rollout_id"}
+ if extra:
+ raise TypeError(f"Generation options belong in Request.config: {sorted(extra)}")
+ from dspy.clients.lm15_boundary import snapshot_request
+
+ request = snapshot_request(request)
+ legacy = {"model": lm.model}
+ use_cache = kwargs.get("cache", lm.cache) if managed and getattr(lm, "_cache_responses", True) else False
+ return PreparedCall(None, None, kwargs, legacy, request, use_cache, 1, managed)
+ merged = {**lm.kwargs, **{key: val for key, val in kwargs.items() if key != "cache"}}
+ prompt_cache = merged.get("prompt_cache")
+ if prompt_cache is not None:
+ if not isinstance(prompt_cache, CacheConfig):
+ raise TypeError("prompt_cache must be a dspy.lm15.CacheConfig or None")
+ if not managed or getattr(lm, "_engine_spec", None) == "litellm" or lm.model_type == "text":
+ raise LMUnsupportedFeatureError("prompt_cache requires native lm15 or a canonical custom engine.")
+ if merged.get("rollout_id") is None:
+ merged.pop("rollout_id", None)
+ if hasattr(lm, "_warn_zero_temp_rollout"):
+ lm._warn_zero_temp_rollout(merged.get("temperature"), merged.get("rollout_id"))
+ rendered = messages or [{"role": "user", "content": prompt}]
+ if getattr(lm, "use_developer_role", False) and lm.model_type == "responses":
+ rendered = [{**m, "role": "developer"} if m.get("role") == "system" else m for m in rendered]
+ n = merged.get("n", 1)
+ if n is None:
+ n = 1
+ if isinstance(n, bool) or not isinstance(n, int) or n < 1:
+ raise ValueError("n must be a positive integer")
+ use_cache = kwargs.get("cache", lm.cache) if managed and getattr(lm, "_cache_responses", True) else False
+ return PreparedCall(prompt, messages, kwargs, {"model": lm.model, "messages": rendered, **merged},
+ None, use_cache, n, managed)
+
+
+def _canonical(call, *, compat=None):
+ if call.request is not None:
+ return call.request
+ body = {key: val for key, val in call.legacy.items()
+ if key not in CLIENT_KEYS | {"n", "rollout_id", "num_generations", "prompt_cache"} and val is not None}
+ format_ = body.get("response_format")
+ if isinstance(format_, type) and issubclass(format_, pydantic.BaseModel):
+ from dspy.clients.legacy_requests import _close_object_schemas
+
+ # Match the legacy Responses path's preparation of generated schemas.
+ # Raw caller-supplied schemas remain unchanged.
+ body["response_format"] = {"type": "json_schema", "json_schema": {
+ "name": format_.__name__, "schema": _close_object_schemas(format_.model_json_schema()), "strict": True,
+ }}
+ request = request_from_openai_chat(body, compat=compat)
+ prompt_cache = call.legacy.get("prompt_cache")
+ if prompt_cache is not None:
+ if request.config.cache is not None:
+ raise LMUnsupportedFeatureError("Do not combine prompt_cache with provider-shaped prompt-cache options.")
+ request = replace(request, config=replace(request.config, cache=prompt_cache))
+ return request
+
+
+def _engine(lm, call, asynchronous):
+ # Setup is outside the retry loop. Canonical routing/conversion refusals
+ # still need the same public error projection as engine-call failures.
+ with error_boundary(lm.model):
+ return _select_engine(lm, call, asynchronous)
+
+
+def _select_engine(lm, call, asynchronous):
+ if not call.managed:
+ backend = AsyncLegacyEngine(lm, _implicit=True) if asynchronous else LegacyEngine(lm, _implicit=True)
+ return backend, _canonical(call) if call.request else None, None
+ spec = lm._engine_spec
+ if not isinstance(spec, str):
+ backend = lm._async_engine_spec if asynchronous else spec
+ if backend is None:
+ raise LMUnsupportedFeatureError("This custom engine has no async counterpart; pass async_engine=.")
+ # TODO(3.5): remove the legacy shortcut after moving DummyLM and all
+ # adapter execution to canonical requests/responses. Do not warn users
+ # about DSPy's own temporary implementation, or repeat wrapper warnings.
+ legacy_complete = getattr(backend, "complete_legacy", None)
+ if call.request is None and call.legacy.get("prompt_cache") is None and callable(legacy_complete):
+ from dspy.clients.engines.dummy_engine import AsyncDummyEngine, DummyEngine
+
+ builtin_methods = (
+ DummyEngine.complete_legacy, AsyncDummyEngine.complete_legacy,
+ LiteLLMEngine.complete_legacy, AsyncLiteLLMEngine.complete_legacy,
+ )
+ implementation = getattr(legacy_complete, "__func__", legacy_complete)
+ if not isinstance(backend, (LegacyEngine, AsyncLegacyEngine)) and not any(
+ implementation is method for method in builtin_methods
+ ):
+ warn_legacy_shortcut()
+ return backend, None, None
+ return backend, _canonical(call), None
+ selection = select_backend(lm, call.legacy)
+ native, resolution, clients = selection.native, selection.resolution, selection.clients
+ canonical = call.request
+ if native:
+ try:
+ canonical = _canonical(call, compat=resolution.compat)
+ except (LM15Error, TypeError, ValueError) as exc:
+ if spec == "lm15" or call.request is not None or call.legacy.get("prompt_cache") is not None:
+ raise LMUnsupportedFeatureError(str(exc), model=lm.model) from exc
+ # This is a representational refusal BEFORE execution. Preserve the
+ # original body on the compatibility backend, never retry elsewhere.
+ native = False
+ if not native:
+ if call.legacy.get("prompt_cache") is not None:
+ raise LMUnsupportedFeatureError(
+ "This request requires LiteLLM, which has no prompt_cache bridge. "
+ "Use native-compatible inputs/settings or omit prompt_cache."
+ )
+ # None disables the bridge; it is never a provider keyword.
+ call.legacy.pop("prompt_cache", None)
+ cls = AsyncLiteLLMEngine if asynchronous else LiteLLMEngine
+ return cls(model_type=lm.model_type, **clients), canonical if call.request else None, None
+ # Long-lived sync pools and a separate async pool per event loop. Copies
+ # share the store; a copy with changed client settings gets a distinct key.
+ loop = asyncio.get_running_loop() if asynchronous else None
+ key = (loop, lm.model, lm.model_type, tuple(sorted((k, v if isinstance(v, (str, int, float, type(None))) else id(v))
+ for k, v in clients.items())))
+ with lm._engine_lock:
+ backend = lm._engine_store.get(key)
+ if backend is None:
+ provider = resolution.provider
+ api_keys = {provider: clients["api_key"]} if "api_key" in clients else None
+ url = clients.get("api_base") or clients.get("base_url")
+ config = RouterConfig(api_keys=api_keys, base_urls={provider: url} if url else None)
+ cls = AsyncLM15Engine if asynchronous else LM15Engine
+ backend = cls(config, model_type=lm.model_type)
+ lm._engine_store[key] = backend
+ return backend, canonical, resolution.provider
+
+
+def _cached(lm, call, asynchronous):
+ if not call.cache:
+ return None
+ import dspy
+
+ record = dspy.cache.get(call.key(lm, asynchronous), IGNORED_CACHE_KEYS)
+ if record is None:
+ return None
+ if isinstance(record, dict) and record.get("_dspy_format") == CACHE_FORMAT:
+ return CallResult.load(record)
+ result = CallResult.legacy(lm, record, kwargs=call.legacy)
+ result.cache_hit = True
+ result.usage = {}
+ return result
+
+
+def _store(lm, call, result, asynchronous):
+ if call.cache:
+ import dspy
+
+ # Preserve the original SDK cache format on the compatibility path.
+ # Native responses use only plain data, compatible with restricted mode.
+ record = result.dump() if result.responses or result.raw is None else result.raw
+ dspy.cache.put(call.key(lm, asynchronous), record, IGNORED_CACHE_KEYS)
+
+
+def _delay(exc, attempt):
+ hint = finite_seconds(getattr(exc, "retry_after", None))
+ return hint if hint is not None else min(2 ** min(attempt, 6), 60)
+
+
+def _result(lm, response, call, provider, request=None, *, estimate=True):
+ if not isinstance(response, Response):
+ actual = f"{type(response).__module__}.{type(response).__qualname__}"
+ raise TypeError(f"Engine.complete must return dspy.lm15.Response, got {actual}")
+ result = CallResult.native(response, model_type=lm.model_type,
+ logprobs=(call.request.config.logprobs is not None if call.request else bool(call.legacy.get("logprobs"))),
+ provider=provider)
+ if estimate:
+ _price_result(lm, result, response, provider, request)
+ if response.finish_reason == "length":
+ import logging
+
+ logging.getLogger("dspy.clients.lm").warning("LM response was truncated; increase max_tokens or inspect history.")
+ return result
+
+
+def _price_result(lm, result, response, provider, request):
+ """Advisory pricing only: safe to finish in a worker after cancellation.
+
+ Never update usage, history, or the response cache from this worker.
+ """
+ from dspy.clients.costs import estimate_cost
+
+ try:
+ wire_model = request.model if request is not None else lm.model
+ if provider is not None:
+ from dspy.clients.capabilities import resolve
+
+ wire_model = resolve(lm).model
+ result.cost, result.cost_details = estimate_cost(
+ response, provider=provider, requested_model=wire_model, request=request,
+ )
+ except Warning:
+ raise
+ except Exception:
+ result.cost = None
+ result.cost_details = {"kind": "unknown", "reason": "pricing metadata could not be interpreted"}
+
+
+@dataclass
+class _Attempt:
+ emitted: bool = False
+ completed: bool = False
+ recorded: bool = False
+ response: Response | None = None
+ result: CallResult | None = None
+
+
+def _retain_response_usage(state, results, response, provider):
+ if isinstance(response, Response) and not state.recorded:
+ # Record billing before output conversion, final delivery or cleanup.
+ # If conversion fails this placeholder is accounted, never returned.
+ results.append(CallResult(usage=usage_dict(response, provider)))
+ state.recorded = True
+
+
+def _accept_response(lm, call, state, results, response, provider, request):
+ state.completed = True
+ state.response = response
+ _retain_response_usage(state, results, response, provider)
+ state.result = _result(lm, response, call, provider, request, estimate=False)
+ results[-1] = state.result
+
+
+def _accept_end(lm, call, state, results, accumulator, provider, request):
+ state.completed = True
+ try:
+ response = accumulator.response()
+ except StreamAssemblyError as exc:
+ _retain_response_usage(state, results, exc.partial, provider)
+ raise
+ _accept_response(lm, call, state, results, response, provider, request)
+
+
+def _partial(accumulator):
+ try:
+ present = any((accumulator.started_model, accumulator.started_id, accumulator.text_parts,
+ accumulator.thinking_parts, accumulator.tool_call_meta, accumulator.image_parts,
+ accumulator.audio_chunks, accumulator.citation_parts, accumulator.message_continuation))
+ return accumulator.response() if present else None
+ except StreamAssemblyError as exc:
+ return exc.partial
+ except Exception:
+ # Salvaging diagnostics must never replace the primary error.
+ return None
+
+
+def _legacy_done(state, results, progress):
+ state.emitted = progress.get("emitted", False)
+ state.completed = state.completed or progress.get("completed", False)
+ if state.result is not None:
+ results.append(state.result)
+ state.recorded = True
+ elif state.completed and "raw" in progress:
+ # A completed compatibility response can outlive a failed SDK close or
+ # output conversion. Extract only reported usage, not invented outputs.
+ try:
+ usage = plain(dict(value(progress["raw"], "usage", {}) or {}))
+ except Exception:
+ usage = {}
+ results.append(CallResult(usage=usage))
+ state.recorded = True
+
+
+def _accept_legacy(state, result):
+ state.completed = True
+ if not isinstance(result, CallResult):
+ raise TypeError("Engine.complete_legacy must return a CallResult")
+ state.result = result
+
+
+def _account_failed_call(lm, results, primary):
+ if settings.usage_tracker:
+ for result in results:
+ try:
+ settings.usage_tracker.add_usage(lm.model, result.usage)
+ except Exception as secondary:
+ # A custom tracker must not replace an engine error or cancel.
+ try:
+ primary.usage_errors = (*getattr(primary, "usage_errors", ()), secondary)
+ if hasattr(primary, "add_note"):
+ primary.add_note(f"Usage accounting also failed ({type(secondary).__name__}); see usage_errors.")
+ except Exception:
+ pass
+
+
+def _retryable(state, exc, attempt, retries):
+ return not state.completed and not state.emitted and attempt < retries and is_retryable_lm_error(exc)
+
+
+def _attempt(lm, call, backend, request, provider, state, results):
+ stream = settings.send_stream
+ if request is None:
+ progress = {"emitted": False}
+ try:
+ with settings.context(_lm_stream_progress=progress):
+ result = backend.complete_legacy(lm, copy.deepcopy(call.legacy), prompt=call.prompt,
+ messages=call.messages, call_kwargs=call.kwargs)
+ _accept_legacy(state, result)
+ finally:
+ _legacy_done(state, results, progress)
+ elif stream is None:
+ try:
+ response = backend.complete(request)
+ except StreamAssemblyError as exc:
+ _retain_response_usage(state, results, exc.partial, provider)
+ raise
+ _accept_response(lm, call, state, results, response, provider, request)
+ else:
+ accumulator = StreamAccumulator(request)
+ bridge = ListenerBridge(lm.model, id(settings.caller_predict) if settings.caller_predict else None)
+ events = checked_stream(backend.stream(request), provider=provider)
+ try:
+ with closing_stream(events):
+ for event in events:
+ accumulator.push(event)
+ if event.type == "end":
+ _accept_end(lm, call, state, results, accumulator, provider, request)
+ if chunk := bridge.chunk(event):
+ state.emitted = True
+ stream_emitted()
+ anyio.from_thread.run(stream.send, chunk)
+ except StreamAssemblyError as exc:
+ if exc.partial is None:
+ exc.partial = _partial(accumulator)
+ raise
+
+
+def execute(lm, call):
+ if result := _cached(lm, call, False):
+ return result
+ backend, request, provider = _engine(lm, call, False)
+ results = []
+ # Ordinary compatibility calls keep backend-native n; canonical engines
+ # produce one candidate per attempt.
+ # TODO(candidate-parallelism): bound concurrent canonical requests while
+ # respecting engine concurrency guarantees, output order, per-candidate
+ # retries/cancellation accounting, and whole-call caching. Define candidate
+ # identity for listeners before multiplexing streams. Concurrency reduces
+ # latency, not the input-token charges for separate requests.
+ count = call.n if request is not None else 1
+ retries = lm.num_retries if call.managed else 0
+ try:
+ for _ in range(count):
+ for attempt in range(retries + 1):
+ state = _Attempt()
+ try:
+ boundary = error_boundary(lm.model, provider=provider, unexpected=True) if call.managed else nullcontext()
+ with boundary:
+ _attempt(lm, call, backend, request, provider, state, results)
+ break
+ except Exception as exc:
+ if not _retryable(state, exc, attempt, retries):
+ raise
+ time.sleep(_delay(exc, attempt))
+ # Completion is irreversible. Advisory pricing and storage never
+ # run inside the retry region, even if custom code raises here.
+ if request is not None:
+ _price_result(lm, state.result, state.response, provider, request)
+ result = results[0] if len(results) == 1 else combine(results, model_type=lm.model_type)
+ _store(lm, call, result, False)
+ return result
+ except BaseException as exc:
+ _account_failed_call(lm, results, exc)
+ raise
+
+
+async def _aattempt(lm, call, backend, request, provider, state, results):
+ stream = settings.send_stream
+ if request is None:
+ progress = {"emitted": False}
+ try:
+ with settings.context(_lm_stream_progress=progress):
+ result = await backend.complete_legacy(lm, copy.deepcopy(call.legacy), prompt=call.prompt,
+ messages=call.messages, call_kwargs=call.kwargs)
+ _accept_legacy(state, result)
+ finally:
+ _legacy_done(state, results, progress)
+ elif stream is None:
+ try:
+ response = await backend.complete(request)
+ except StreamAssemblyError as exc:
+ _retain_response_usage(state, results, exc.partial, provider)
+ raise
+ _accept_response(lm, call, state, results, response, provider, request)
+ else:
+ accumulator = StreamAccumulator(request)
+ bridge = ListenerBridge(lm.model, id(settings.caller_predict) if settings.caller_predict else None)
+ events = achecked_stream(backend.stream(request), provider=provider)
+ try:
+ async with aclosing_stream(events):
+ async for event in events:
+ accumulator.push(event)
+ if event.type == "end":
+ _accept_end(lm, call, state, results, accumulator, provider, request)
+ if chunk := bridge.chunk(event):
+ state.emitted = True
+ stream_emitted()
+ await stream.send(chunk)
+ except StreamAssemblyError as exc:
+ if exc.partial is None:
+ exc.partial = _partial(accumulator)
+ raise
+
+
+async def aexecute(lm, call):
+ if call.cache:
+ if result := await asyncio.to_thread(_cached, lm, call, True):
+ return result
+ backend, request, provider = _engine(lm, call, True)
+ results = []
+ # TODO(candidate-parallelism): apply the same guarantees as execute() above;
+ # cancellation must account for every completed candidate exactly once.
+ count = call.n if request is not None else 1
+ retries = lm.num_retries if call.managed else 0
+ try:
+ for _ in range(count):
+ for attempt in range(retries + 1):
+ state = _Attempt()
+ try:
+ boundary = error_boundary(lm.model, provider=provider, unexpected=True) if call.managed else nullcontext()
+ with boundary:
+ await _aattempt(lm, call, backend, request, provider, state, results)
+ break
+ except Exception as exc:
+ if not _retryable(state, exc, attempt, retries):
+ raise
+ await asyncio.sleep(_delay(exc, attempt))
+ if request is not None:
+ await asyncio.to_thread(_price_result, lm, state.result, state.response, provider, request)
+ result = results[0] if len(results) == 1 else combine(results, model_type=lm.model_type)
+ if call.cache:
+ # Cancellation may leave this worker writing a complete result,
+ # never a partial one. Usage still belongs to this public call.
+ await asyncio.to_thread(_store, lm, call, result, True)
+ return result
+ except BaseException as exc:
+ _account_failed_call(lm, results, exc)
+ raise
+
+
+def finalize(lm, call, result):
+ if not result.cache_hit and settings.usage_tracker:
+ settings.usage_tracker.add_usage(lm.model, result.usage)
+ if not settings.disable_history:
+ import datetime
+ import uuid
+
+ from dspy.clients.lm15_boundary import history_messages
+
+ entry = {"prompt": call.prompt, "messages": history_messages(call.request) if call.request else call.messages,
+ "kwargs": {key: val for key, val in call.kwargs.items() if not key.startswith("api_")},
+ "response": result.raw if result.raw is not None else result.responses,
+ "outputs": result.outputs, "usage": result.usage, "cost": result.cost,
+ "timestamp": datetime.datetime.now().isoformat(), "uuid": str(uuid.uuid4()),
+ "model": lm.model, "response_model": result.response_model, "model_type": lm.model_type}
+ if result.cost_details:
+ entry["cost_details"] = result.cost_details
+ if call.request:
+ entry["request"] = call.request
+ lm.update_history(entry)
+ return result.typed(call.request, lm.model_type) if call.request else result.outputs
diff --git a/dspy/clients/legacy_outputs.py b/dspy/clients/legacy_outputs.py
new file mode 100644
index 000000000..bd28980c9
--- /dev/null
+++ b/dspy/clients/legacy_outputs.py
@@ -0,0 +1,68 @@
+"""Preserve DSPy's public output shapes without the retired experimental types."""
+
+import json
+from typing import Any
+
+
+def value(obj: Any, key: str, default=None):
+ return obj.get(key, default) if isinstance(obj, dict) else getattr(obj, key, default)
+
+
+def plain(obj: Any):
+ """Read SDK fields without invoking deferred Pydantic serializers."""
+ from pydantic import BaseModel
+
+ if isinstance(obj, BaseModel):
+ obj = dict(obj)
+ if isinstance(obj, dict):
+ return {key: plain(item) for key, item in obj.items()}
+ if isinstance(obj, (list, tuple)):
+ return [plain(item) for item in obj]
+ return obj
+
+
+def responses_outputs(response: Any) -> list[dict[str, Any]]:
+ """Render a Responses API answer using DSPy's existing list-of-dicts API."""
+ texts, reasoning, calls, citations = [], [], [], []
+ for item in value(response, "output", []) or []:
+ kind = value(item, "type")
+ if kind == "message":
+ for part in value(item, "content", []) or []:
+ if value(part, "type") in {"text", "output_text", "input_text"}:
+ text = value(part, "text")
+ if text is not None:
+ texts.append(text)
+ for annotation in value(part, "annotations", []) or []:
+ raw = plain(annotation)
+ fields = {"cited_text", "text", "supported_text", "document_title", "title", "url"}
+ citation = {
+ "type": "citation",
+ "metadata": {key: val for key, val in raw.items() if key not in fields and val is not None},
+ "text": raw.get("cited_text") or raw.get("text") or raw.get("supported_text"),
+ "title": raw.get("document_title") or raw.get("title"),
+ "url": raw.get("url"),
+ }
+ citations.append({key: val for key, val in citation.items() if val is not None})
+ elif kind == "function_call":
+ arguments = value(item, "arguments", {})
+ if isinstance(arguments, str):
+ try:
+ arguments = json.loads(arguments)
+ except ValueError:
+ arguments = {}
+ call = {"type": "function", "function": {"name": value(item, "name", ""), "arguments": json.dumps(arguments)}}
+ if value(item, "call_id") is not None:
+ call["id"] = value(item, "call_id")
+ calls.append(call)
+ elif kind == "reasoning":
+ for part in value(item, "content") or value(item, "summary") or []:
+ if text := value(part, "text"):
+ reasoning.append(text)
+ output = {"text": "".join(texts) if texts else None}
+ if reasoning:
+ output["reasoning_content"] = "".join(reasoning)
+ if calls:
+ output["tool_calls"] = calls
+ if citations:
+ output["citations"] = citations
+ return [output]
diff --git a/dspy/clients/legacy_requests.py b/dspy/clients/legacy_requests.py
new file mode 100644
index 000000000..a7aaa5cc8
--- /dev/null
+++ b/dspy/clients/legacy_requests.py
@@ -0,0 +1,122 @@
+"""Legacy chat-to-Responses translation, independent of the typed LM API.
+
+This boundary preserves provider-native fields accepted by ordinary DSPy calls.
+It deliberately does not force those fields through lm15's narrower vocabulary.
+"""
+
+import copy
+from typing import Any
+
+import pydantic
+
+
+def _close_object_schemas(schema: Any) -> Any:
+ if not isinstance(schema, dict):
+ return schema
+ closed = dict(schema)
+ for key in ("items", "additionalProperties", "not", "if", "then", "else", "contains", "propertyNames"):
+ if isinstance(closed.get(key), dict):
+ closed[key] = _close_object_schemas(closed[key])
+ for key in ("items", "prefixItems", "anyOf", "oneOf", "allOf"):
+ if isinstance(closed.get(key), list):
+ closed[key] = [_close_object_schemas(item) for item in closed[key]]
+ for key in ("properties", "patternProperties", "$defs", "definitions"):
+ if isinstance(closed.get(key), dict):
+ closed[key] = {name: _close_object_schemas(sub) for name, sub in closed[key].items()}
+ if closed.get("type") == "object" and "additionalProperties" not in closed:
+ closed["additionalProperties"] = False
+ return closed
+
+
+def _content(content, role):
+ text_type = "output_text" if role == "assistant" else "input_text"
+ if isinstance(content, str):
+ return [{"type": text_type, "text": content}]
+ blocks = []
+ for block in content or []:
+ kind = block.get("type")
+ if kind in {"text", "input_text", "output_text"}:
+ blocks.append({"type": text_type, "text": block.get("text", "")})
+ elif kind == "image_url":
+ image = block["image_url"]
+ image = image if isinstance(image, dict) else {"url": image}
+ converted = {"type": "input_image", "image_url": image.get("url", "")}
+ if image.get("detail") is not None:
+ converted["detail"] = image["detail"]
+ blocks.append(converted)
+ elif kind == "file":
+ file = block.get("file", {})
+ blocks.append({"type": "input_file", **{key: file.get(key) for key in ("file_data", "filename", "file_id")}})
+ else:
+ blocks.append(copy.deepcopy(block))
+ return blocks
+
+
+def chat_to_responses(request: dict[str, Any]) -> dict[str, Any]:
+ data = copy.deepcopy(request)
+ messages = data.pop("messages", [])
+ inputs = []
+ for message in messages:
+ role = message.get("role")
+ content = message.get("content")
+ if role == "tool":
+ if isinstance(content, list):
+ content = "".join(
+ block.get("text", "") if isinstance(block, dict) and block.get("type") in {"text", "input_text"}
+ else str(block) for block in content
+ )
+ item = {"type": "function_call_output", "output": content or ""}
+ if message.get("tool_call_id") is not None:
+ item["call_id"] = message["tool_call_id"]
+ inputs.append(item)
+ continue
+ calls = message.get("tool_calls") or []
+ blocks = _content(content, role)
+ if blocks or role != "assistant" or not calls:
+ item = {"role": role, "content": blocks}
+ if message.get("name") is not None:
+ item["name"] = message["name"]
+ inputs.append(item)
+ if role == "assistant":
+ for call in calls:
+ function = call.get("function") or call
+ item = {"type": "function_call", "name": function.get("name", ""),
+ "arguments": function.get("arguments", "{}")}
+ call_id = call.get("call_id") or call.get("id")
+ if call_id is not None:
+ item["call_id"] = call_id
+ inputs.append(item)
+ data["input"] = inputs
+ if "max_completion_tokens" in data and "max_tokens" not in data:
+ data["max_tokens"] = data.pop("max_completion_tokens")
+ if data.get("max_tokens") is not None:
+ data["max_output_tokens"] = data.pop("max_tokens")
+ else:
+ data.pop("max_tokens", None)
+ if "reasoning_effort" in data:
+ effort = data.pop("reasoning_effort")
+ if data.get("reasoning") is None:
+ data["reasoning"] = {"effort": effort, "summary": "auto"}
+ if data.get("tools"):
+ data["tools"] = [
+ {"type": "function", **{key: val for key, val in tool.items() if key not in {"type", "function"}}, **tool["function"]}
+ if "function" in tool else tool for tool in data["tools"]
+ ]
+ for tool in data["tools"]:
+ if tool.get("type") == "function" and tool.get("description") is None:
+ tool.pop("description", None)
+ choice = data.get("tool_choice")
+ if isinstance(choice, dict) and "function" in choice:
+ data["tool_choice"] = {"type": "function", "name": choice["function"]["name"]}
+ format_ = data.pop("response_format", None)
+ if format_ is not None:
+ if isinstance(format_, type) and issubclass(format_, pydantic.BaseModel):
+ format_ = {"name": format_.__name__, "type": "json_schema", "schema": _close_object_schemas(format_.model_json_schema())}
+ data["text"] = {**data.get("text", {}), "format": format_}
+ # The former config serializer omitted absent common generation parameters.
+ for key in ("temperature", "top_p", "n", "logprobs", "reasoning", "tool_choice"):
+ if data.get(key) is None:
+ data.pop(key, None)
+ if not data.get("stop"):
+ data.pop("stop", None)
+ return data
diff --git a/dspy/clients/lm.py b/dspy/clients/lm.py
index b6fa8ffb4..0d0487b08 100644
--- a/dspy/clients/lm.py
+++ b/dspy/clients/lm.py
@@ -10,31 +10,17 @@ import anyio.from_thread
from anyio.streams.memory import MemoryObjectSendStream
import dspy
-from dspy.clients._litellm import get_litellm, is_litellm_context_window_error
+from dspy.clients._litellm import get_litellm
from dspy.clients.cache import request_cache
+from dspy.clients.call_context import completed_legacy, stream_emitted
+from dspy.clients.engines.lifecycle import aclosing_stream
+from dspy.clients.legacy_requests import chat_to_responses
from dspy.clients.openai import OpenAIProvider
-from dspy.clients.openai_format import to_openai_responses_request
from dspy.clients.provider import Provider, ReinforceJob, TrainingJob
from dspy.clients.utils_finetune import TrainDataFormat
-from dspy.core.types import LMRequest
+from dspy.lm15 import CacheConfig
from dspy.utils.callback import BaseCallback
-from dspy.utils.exceptions import (
- ContextWindowExceededError,
- LMAuthError,
- LMBillingError,
- LMConfigurationError,
- LMError,
- LMInvalidRequestError,
- LMNotConfiguredError,
- LMProviderError,
- LMRateLimitError,
- LMServerError,
- LMTimeoutError,
- LMTransportError,
- LMUnexpectedError,
- LMUnsupportedFeatureError,
- LMUnsupportedModelError,
-)
+from dspy.utils.exceptions import LMConfigurationError, LMError, LMUnsupportedFeatureError
from .base_lm import BaseLM
@@ -56,9 +42,13 @@ def _is_openai_reasoning_model(model: str) -> bool:
class LM(BaseLM):
"""
A language model supporting chat or text completion requests for use with DSPy modules.
- """
- forward_contract = "legacy"
+ Use lm("hello") for a list-returning convenience call, or pass an explicit
+ dspy.lm15.Request to receive a dspy.lm15.Response. OpenAI-style messages=
+ dictionaries are deprecated and scheduled for removal in DSPy 3.5. Adapters
+ and custom engines must migrate to the canonical request/response contract.
+ See https://dspy.ai/community/normalized-lm-api-migration/.
+ """
def __init__(
self,
@@ -74,6 +64,9 @@ class LM(BaseLM):
launch_kwargs: dict[str, Any] | None = None,
train_kwargs: dict[str, Any] | None = None,
use_developer_role: bool = False,
+ engine: Any = "auto",
+ async_engine: Any = None,
+ prompt_cache: CacheConfig | None = None,
**kwargs,
):
"""Create a new language model instance for use with DSPy modules and programs.
@@ -92,7 +85,15 @@ class LM(BaseLM):
num_retries: The number of times to retry a request if it fails transiently due to
network error, rate limiting, etc. Requests are retried with exponential
backoff.
- provider: The provider to use. If not specified, the provider will be inferred from the model.
+ engine: 'auto' prefers lm15 for representable requests; 'litellm' preserves the compatibility backend;
+ 'lm15' refuses unsupported mappings rather than selecting LiteLLM. A custom engine implements
+ complete(Request) -> Response and optionally stream(Request). Engines are borrowed.
+ async_engine: Async counterpart when supplying a custom engine object.
+ prompt_cache: Optional lm15 CacheConfig for provider-side prompt caching on ordinary calls.
+ Separate from DSPy's response cache. Requires native lm15 or a canonical custom engine;
+ may incur cache-write/storage charges. A call-time value overrides this default, and None
+ removes it. Explicit Request calls use only Request.config.cache. No cache resource is created.
+ provider: The training/launch provider. This does not select the inference engine.
finetuning_model: The model to finetune. In some providers, the models available for finetuning is different
from the models available for inference.
rollout_id: Optional integer used to differentiate cache entries for otherwise
@@ -101,6 +102,23 @@ class LM(BaseLM):
only affects generation when `temperature` is non-zero. This argument is
stripped before sending requests to the provider.
"""
+ if isinstance(engine, str):
+ if engine not in {"auto", "lm15", "litellm"}:
+ raise ValueError("engine must be 'auto', 'lm15', 'litellm', or an engine object")
+ if async_engine is not None:
+ raise ValueError("async_engine is only used with a custom engine object")
+ elif not callable(getattr(engine, "complete", None)):
+ raise TypeError("A custom engine must implement complete(Request) -> Response")
+ if isinstance(num_retries, bool) or not isinstance(num_retries, int) or num_retries < 0:
+ raise ValueError("num_retries must be a nonnegative integer")
+ if prompt_cache is not None:
+ if not isinstance(prompt_cache, CacheConfig):
+ raise TypeError("prompt_cache must be a dspy.lm15.CacheConfig or None")
+ kwargs["prompt_cache"] = prompt_cache
+ self._engine_spec = engine
+ self._async_engine_spec = async_engine
+ self._engine_store = {}
+ self._engine_lock = threading.RLock()
super().__init__(
model=model,
model_type=model_type,
@@ -147,20 +165,27 @@ class LM(BaseLM):
@property
def supports_function_calling(self) -> bool:
- return _get_litellm().supports_function_calling(model=self.model)
+ from dspy.clients.capabilities import capabilities
+
+ return capabilities(self).function_calling
@property
def supports_reasoning(self) -> bool:
- return _get_litellm().supports_reasoning(self.model)
+ from dspy.clients.capabilities import capabilities
+
+ return capabilities(self).reasoning
@property
def supports_response_schema(self) -> bool:
- return _get_litellm().supports_response_schema(model=self.model, custom_llm_provider=self._provider_name)
+ from dspy.clients.capabilities import capabilities
+
+ return capabilities(self).response_schema
@property
def supported_params(self) -> set[str]:
- params = _get_litellm().get_supported_openai_params(model=self.model, custom_llm_provider=self._provider_name)
- return set(params) if params else set()
+ from dspy.clients.capabilities import capabilities
+
+ return set(capabilities(self).params)
def _warn_zero_temp_rollout(self, temperature: float | None, rollout_id):
if not self._warned_zero_temp_rollout and rollout_id is not None and temperature == 0:
@@ -183,144 +208,99 @@ class LM(BaseLM):
return completion_fn, litellm_cache_args
def _wrap_litellm_exception(self, exc: Exception) -> LMError:
- """Convert exceptions raised at the LiteLLM boundary into DSPy LM exceptions."""
- if isinstance(exc, LMError):
- return exc
-
- status = _exception_status(exc)
- provider = getattr(exc, "llm_provider", None) or self._provider_name
- model = getattr(exc, "model", None) or self.model
- message = _exception_message(exc)
- metadata = {
- "model": model,
- "provider": provider,
- "provider_code": _exception_provider_code(exc),
- "status": status,
- "request_id": _exception_request_id(exc),
- "retry_after": _exception_retry_after(exc),
- }
-
- if is_litellm_context_window_error(exc):
- return ContextWindowExceededError(message=message or "Context window exceeded", **metadata)
-
- exc_cls = _lm_error_class_from_litellm_exception(exc) or _lm_error_class_from_status(status)
- return exc_cls(message, **metadata)
-
- def forward(
- self,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- **kwargs
- ):
- """Call the configured LM synchronously.
+ from dspy.clients.engines.litellm_errors import to_lm15_error
+ from dspy.clients.errors import wrap_error
- LiteLLM/provider exceptions are wrapped in DSPy's structured LM error
- hierarchy before they are re-raised.
+ canonical = to_lm15_error(exc, model=self.model, provider=self._provider_name)
+ return wrap_error(canonical, model=self.model, provider=self._provider_name)
- Args:
- prompt: Optional prompt text. Ignored when `messages` is provided.
- messages: Optional chat messages to send to the LM.
- **kwargs: Per-call LM parameters that override defaults from `LM(...)`.
-
- Raises:
- dspy.LMError: Base class for wrapped LM configuration, transport,
- provider, and unsupported-feature failures. Notable subclasses
- include `dspy.ContextWindowExceededError` for context-window
- failures, which adapters use to avoid inappropriate fallback
- retries when the prompt is too long.
- """
- # Build the request.
- kwargs = dict(kwargs)
- cache = kwargs.pop("cache", self.cache)
-
- messages = messages or [{"role": "user", "content": prompt}]
- if self.use_developer_role and self.model_type == "responses":
- messages = [{**m, "role": "developer"} if m.get("role") == "system" else m for m in messages]
- kwargs = {**self.kwargs, **kwargs}
- self._warn_zero_temp_rollout(kwargs.get("temperature"), kwargs.get("rollout_id"))
- if kwargs.get("rollout_id") is None:
- kwargs.pop("rollout_id", None)
-
- if self.model_type == "chat":
- completion = litellm_completion
- elif self.model_type == "text":
- completion = litellm_text_completion
- elif self.model_type == "responses":
- completion = litellm_responses_completion
- completion, litellm_cache_args = self._get_cached_completion_fn(completion, cache)
+ def forward(self, prompt=None, messages=None, **kwargs):
+ """Compatibility forward entry point; public calls also record history."""
+ from dspy.clients.execution import execute, prepare
- try:
- results = completion(
- request=dict(model=self.model, messages=messages, **kwargs),
- num_retries=self.num_retries,
- cache=litellm_cache_args,
- )
- except Exception as e:
- if isinstance(e, LMError):
- raise
- raise self._wrap_litellm_exception(e) from e
+ return execute(self, prepare(self, prompt, messages, kwargs, direct=True)).provider_response()
- self._check_truncation(results)
+ async def aforward(self, prompt=None, messages=None, **kwargs):
+ import asyncio
- return results
+ from dspy.clients.execution import aexecute, prepare
- async def aforward(
- self,
- prompt: str | None = None,
- messages: list[dict[str, Any]] | None = None,
- **kwargs,
- ):
- """Call the configured LM asynchronously.
+ call = await asyncio.to_thread(prepare, self, prompt, messages, kwargs, asynchronous=True, direct=True)
+ return (await aexecute(self, call)).provider_response()
+
+ @property
+ def engine(self):
+ """Configured engine selection ('auto', 'lm15', 'litellm', or an engine object)."""
+ return self._engine_spec
- LiteLLM/provider exceptions are wrapped in DSPy's structured LM error
- hierarchy before they are re-raised.
+ def close(self):
+ """Close owned synchronous engine pools after active calls have finished.
- Args:
- prompt: Optional prompt text. Ignored when `messages` is provided.
- messages: Optional chat messages to send to the LM.
- **kwargs: Per-call LM parameters that override defaults from `LM(...)`.
-
- Raises:
- dspy.LMError: Base class for wrapped LM configuration, transport,
- provider, and unsupported-feature failures. Notable subclasses
- include `dspy.ContextWindowExceededError` for context-window
- failures, which adapters use to avoid inappropriate fallback
- retries when the prompt is too long.
+ Custom engines are borrowed. Async pools must be closed with aclose().
+ Copies share owned pools; closing one releases those shared resources.
"""
- # Build the request.
- kwargs = dict(kwargs)
- cache = kwargs.pop("cache", self.cache)
-
- messages = messages or [{"role": "user", "content": prompt}]
- if self.use_developer_role and self.model_type == "responses":
- messages = [{**m, "role": "developer"} if m.get("role") == "system" else m for m in messages]
- kwargs = {**self.kwargs, **kwargs}
- self._warn_zero_temp_rollout(kwargs.get("temperature"), kwargs.get("rollout_id"))
- if kwargs.get("rollout_id") is None:
- kwargs.pop("rollout_id", None)
-
- if self.model_type == "chat":
- completion = alitellm_completion
- elif self.model_type == "text":
- completion = alitellm_text_completion
- elif self.model_type == "responses":
- completion = alitellm_responses_completion
- completion, litellm_cache_args = self._get_cached_completion_fn(completion, cache)
-
+ with self._engine_lock:
+ keys = [key for key in self._engine_store if key[0] is None]
+ engines = [self._engine_store.pop(key) for key in keys]
+ from contextlib import ExitStack
+
+ with ExitStack() as cleanup:
+ for engine in engines:
+ cleanup.callback(engine.close)
+
+ async def aclose(self):
+ """Close this event loop's owned pools and the synchronous pools."""
+ import asyncio
+ from contextlib import AsyncExitStack
+
+ loop = asyncio.get_running_loop()
+ with self._engine_lock:
+ keys = [key for key in self._engine_store if key[0] is loop]
+ engines = [self._engine_store.pop(key) for key in keys]
try:
- results = await completion(
- request=dict(model=self.model, messages=messages, **kwargs),
- num_retries=self.num_retries,
- cache=litellm_cache_args,
- )
- except Exception as e:
- if isinstance(e, LMError):
- raise
- raise self._wrap_litellm_exception(e) from e
-
- self._check_truncation(results)
+ async with AsyncExitStack() as cleanup:
+ for engine in engines:
+ cleanup.push_async_callback(engine.aclose)
+ finally:
+ self.close()
+
+ def __copy__(self):
+ import types
+
+ copied = object.__new__(type(self))
+ copied.__dict__.update(self.__dict__)
+ for cls in type(self).__mro__:
+ for name, descriptor in vars(cls).items():
+ if isinstance(descriptor, types.MemberDescriptorType) and hasattr(self, name):
+ setattr(copied, name, getattr(self, name))
+ return copied
+
+ def copy(self, **kwargs):
+ if kwargs.get("prompt_cache") is not None and not isinstance(kwargs["prompt_cache"], CacheConfig):
+ raise TypeError("prompt_cache must be a dspy.lm15.CacheConfig or None")
+ spec = kwargs.pop("engine", self._engine_spec)
+ async_spec = kwargs.pop("async_engine", self._async_engine_spec)
+ if isinstance(spec, str) and spec not in {"auto", "lm15", "litellm"}:
+ raise ValueError("Unknown engine selection")
+ if not isinstance(spec, str) and not callable(getattr(spec, "complete", None)):
+ raise TypeError("Custom engines must implement complete(Request)")
+ copied = super().copy(**kwargs)
+ copied._engine_spec = spec
+ copied._async_engine_spec = async_spec
+ return copied
+
+ def __getstate__(self):
+ state = dict(self.__dict__)
+ state.pop("_engine_lock", None)
+ state.pop("_engine_store", None)
+ return state
- return results
+ def __setstate__(self, state):
+ self.__dict__.update(state)
+ self._engine_spec = getattr(self, "_engine_spec", "auto")
+ self._async_engine_spec = getattr(self, "_async_engine_spec", None)
+ self._engine_lock = threading.RLock()
+ self._engine_store = {}
def launch(self, launch_kwargs: dict[str, Any] | None = None):
self.provider.launch(self, launch_kwargs)
@@ -408,7 +388,15 @@ class LM(BaseLM):
A dictionary that can be passed to `BaseLM.load_state` to
reconstruct this `LM`. The state excludes API keys.
"""
+ if not isinstance(self._engine_spec, str):
+ raise TypeError("Custom engine objects require custom dump_state/load_state methods; they cannot be stored in JSON LM state.")
state = super().dump_state()
+ if state.get("prompt_cache") is not None:
+ from dspy._vendor.lm15.serde import cache_config_to_dict
+
+ state["prompt_cache"] = cache_config_to_dict(state["prompt_cache"])
+ if self._engine_spec != "auto":
+ state["engine"] = self._engine_spec
state.update(
{
"finetuning_model": self.finetuning_model,
@@ -425,6 +413,10 @@ class LM(BaseLM):
@classmethod
def load_state(cls, state: dict[str, Any], *, allow_custom_lm_class: bool = False):
state = dict(state)
+ if isinstance(state.get("prompt_cache"), dict):
+ from dspy._vendor.lm15.serde import cache_config_from_dict
+
+ state["prompt_cache"] = cache_config_from_dict(state["prompt_cache"])
model = state.get("model")
if isinstance(model, str) and _is_openai_reasoning_model(model) and "max_completion_tokens" in state:
@@ -435,9 +427,11 @@ class LM(BaseLM):
return super().load_state(state, allow_custom_lm_class=allow_custom_lm_class)
def _check_truncation(self, results):
- if self.model_type != "responses" and any(c.finish_reason == "length" for c in results["choices"]):
+ from dspy.clients.legacy_outputs import value
+
+ if self.model_type != "responses" and any(value(c, "finish_reason") == "length" for c in value(results, "choices", []) or []):
logger.warning(
- f"LM response was truncated due to exceeding max_tokens={self.kwargs['max_tokens']}. "
+ f"LM response was truncated due to exceeding max_tokens={self.kwargs.get('max_tokens', self.kwargs.get('max_completion_tokens'))}. "
"You can inspect the latest LM interactions with `dspy.inspect_history()`. "
"To avoid truncation, consider passing a larger max_tokens when setting up dspy.LM. "
f"You may also consider increasing the temperature (currently {self.kwargs['temperature']}) "
@@ -466,19 +460,23 @@ def _get_stream_completion_fn(
async def stream_completion(request: dict[str, Any], cache_kwargs: dict[str, Any]):
response = await _get_litellm().acompletion(
+ num_retries=0,
cache=cache_kwargs,
stream=True,
headers=headers,
**request,
)
chunks = []
- async for chunk in response:
- if caller_predict_id:
- # Add the predict id to the chunk so that the stream listener can identify which predict produces it.
- chunk.predict_id = caller_predict_id
- chunks.append(chunk)
- await stream.send(chunk)
- return _get_litellm().stream_chunk_builder(chunks)
+ async with aclosing_stream(response):
+ async for chunk in response:
+ if caller_predict_id:
+ chunk.predict_id = caller_predict_id
+ chunks.append(chunk)
+ stream_emitted()
+ await stream.send(chunk)
+ raw = _get_litellm().stream_chunk_builder(chunks)
+ completed_legacy(raw)
+ return raw
def sync_stream_completion():
return anyio.from_thread.run(functools.partial(stream_completion, request, cache_kwargs))
@@ -619,92 +617,12 @@ async def alitellm_responses_completion(request: dict[str, Any], num_retries: in
def _convert_chat_request_to_responses_request(request: dict[str, Any]):
- """Convert legacy chat-shaped LM kwargs into Responses API kwargs.
-
- This is the legacy door into the normalized Responses mapper. Inputs the
- old pass-through converter forwarded verbatim — Responses-native shapes and
- provider-SDK dumps — are tolerated here and only here; the typed path stays
- strict.
- """
- request = dict(request)
- model = request.pop("model")
- messages = [_sanitize_legacy_message(message) for message in request.pop("messages", [])]
- tools = list(request.pop("tools", None) or [])
-
- # Reasoning models use `max_completion_tokens` in the chat path. The
- # normalized Responses mapper expects the shared `max_tokens` name and emits
- # `max_output_tokens`.
- if "max_completion_tokens" in request and "max_tokens" not in request:
- request["max_tokens"] = request.pop("max_completion_tokens")
-
- # Preserve the legacy `reasoning_effort=...` Responses behavior from this LM
- # compatibility shim: requesting reasoning effort also asks OpenAI for an
- # automatic reasoning summary, which DSPy can expose as `reasoning_content`.
- # An explicit per-call `reasoning` dict wins verbatim; the effort shorthand
- # is discarded rather than merged.
- if "reasoning_effort" in request:
- effort = request.pop("reasoning_effort")
- if request.get("reasoning") is None:
- request["reasoning"] = {"effort": effort, "summary": "auto"}
-
- # Hosted Responses tools (web_search, file_search, code_interpreter, mcp, ...)
- # have no normalized LMToolSpec representation; send them through unchanged.
- function_tools = [tool for tool in tools if not _is_hosted_responses_tool(tool)]
-
- # tool_choice dicts that aren't the chat-nested function form are already
- # Responses-native (flat function, hosted, allowed_tools, ...); send them
- # through unchanged.
- responses_native_tool_choice = None
- if isinstance(request.get("tool_choice"), dict) and "function" not in request["tool_choice"]:
- responses_native_tool_choice = request.pop("tool_choice")
-
- lm_request = LMRequest.from_call(model=model, messages=messages, tools=function_tools or None, **request)
- # The old converter never validated reasoning/temperature combinations
- # client-side; keep the provider as the authority on this door.
- data = to_openai_responses_request(lm_request, enforce_reasoning_temperature=False)
- if tools:
- normalized_function_tools = iter(data.pop("tools", []))
- data["tools"] = [
- tool if _is_hosted_responses_tool(tool) else next(normalized_function_tools) for tool in tools
- ]
- if responses_native_tool_choice is not None:
- data["tool_choice"] = responses_native_tool_choice
- return data
-
-
-def _is_hosted_responses_tool(tool: Any) -> bool:
- return isinstance(tool, dict) and "function" not in tool and tool.get("type") not in (None, "function")
-
-
-def _sanitize_legacy_message(message: dict[str, Any]) -> dict[str, Any]:
- """Normalize one legacy message the old converter would have passed through.
-
- Strips provider-SDK output fields that are not message inputs and rewrites
- Responses-native content blocks into their chat forms so the mapper can
- re-emit them with role-correct direction.
- """
- if not isinstance(message, dict):
- return message
- message_keys = {"role", "content", "name", "metadata", "tool_calls", "tool_call_id"}
- cleaned = {key: value for key, value in message.items() if key in message_keys}
- content = cleaned.get("content")
- if isinstance(content, list):
- cleaned["content"] = [_normalize_legacy_content_block(block) for block in content]
- return cleaned
-
-
-def _normalize_legacy_content_block(block: Any) -> Any:
- if not isinstance(block, dict):
- return block
- block_type = block.get("type")
- if block_type in ("input_text", "output_text"):
- return {"type": "text", "text": block.get("text", "")}
- if block_type == "input_image" and isinstance(block.get("image_url"), str):
- return {"type": "image_url", "image_url": {"url": block["image_url"]}}
- if block_type == "input_file":
- file = {key: block[key] for key in ("file_data", "file_id", "filename") if block.get(key) is not None}
- return {"type": "file", "file": file}
- return block
+ """Translate ordinary calls without the retired experimental types."""
+ if "input" in request:
+ data = dict(request)
+ data.pop("messages", None)
+ return data
+ return chat_to_responses(request)
def _add_dspy_identifier_to_headers(headers: dict[str, Any] | None = None):
@@ -714,129 +632,3 @@ def _add_dspy_identifier_to_headers(headers: dict[str, Any] | None = None):
**headers,
}
-#--------
-# Errors
-#--------
-
-def _safe_litellm_exception_class(name: str) -> type[Exception] | None:
- cls = getattr(_get_litellm(), name, None)
- return cls if isinstance(cls, type) and issubclass(cls, Exception) else None
-
-
-def _lm_error_class_from_litellm_exception(exc: Exception) -> type[LMError] | None:
- message = _exception_message(exc).lower()
- class_name = type(exc).__name__.lower()
- if _exception_status(exc) is None and any(
- phrase in message for phrase in ("api key", "apikey", "credentials", "environment variable")
- ):
- return LMNotConfiguredError
- if "timeout" in class_name or "timed out" in message or "timeout" in message:
- return LMTimeoutError
- if "connection" in class_name or "network" in message or "connection" in message:
- return LMTransportError
-
- mappings = [
- ("AuthenticationError", LMAuthError),
- ("RateLimitError", LMRateLimitError),
- ("NotFoundError", LMUnsupportedModelError),
- ("UnsupportedParamsError", LMUnsupportedFeatureError),
- ("UnprocessableEntityError", LMInvalidRequestError),
- ("ContentPolicyViolationError", LMInvalidRequestError),
- ("BadRequestError", LMInvalidRequestError),
- ("InvalidRequestError", LMInvalidRequestError),
- ("InternalServerError", LMServerError),
- ("ServiceUnavailableError", LMServerError),
- ("APIConnectionError", LMTransportError),
- ("APIResponseValidationError", LMProviderError),
- ("BudgetExceededError", LMBillingError),
- ("RouterRateLimitError", LMRateLimitError),
- ]
- for litellm_name, dspy_cls in mappings:
- litellm_cls = _safe_litellm_exception_class(litellm_name)
- if litellm_cls is not None and isinstance(exc, litellm_cls):
- return dspy_cls
- return None
-
-
-def _lm_error_class_from_status(status: int | None) -> type[LMError]:
- if status in (401, 403):
- return LMAuthError
- if status == 402:
- return LMBillingError
- if status == 404:
- return LMUnsupportedModelError
- if status == 408:
- return LMTimeoutError
- if status == 429:
- return LMRateLimitError
- if status is not None and 400 <= status < 500:
- return LMInvalidRequestError
- if status is not None and status >= 500:
- return LMServerError
- return LMUnexpectedError if status is None else LMProviderError
-
-
-# Best-effort LiteLLM/provider exception metadata extraction.
-#
-# LiteLLM exception metadata is not exposed as a single stable typed shape across providers, exception classes, and
-# LiteLLM versions. Keep the defensive getattr-based extraction localized here so the rest of DSPy sees structured
-# DSPyError metadata.
-def _exception_status(exc: Exception) -> int | None:
- status = getattr(exc, "status_code", None)
- if status is None:
- response = getattr(exc, "response", None)
- status = getattr(response, "status_code", None)
- try:
- return int(status) if status is not None else None
- except (TypeError, ValueError):
- return None
-
-
-def _exception_message(exc: Exception) -> str:
- message = getattr(exc, "message", None)
- if message is None:
- message = str(exc)
- return str(message)
-
-
-def _exception_headers(exc: Exception):
- response = getattr(exc, "response", None)
- return getattr(response, "headers", None) or getattr(exc, "headers", None) or {}
-
-
-def _exception_header(exc: Exception, name: str) -> str | None:
- headers = _exception_headers(exc)
- if not headers:
- return None
- try:
- return headers.get(name) or headers.get(name.lower())
- except AttributeError:
- return None
-
-
-def _exception_request_id(exc: Exception) -> str | None:
- return (
- _exception_header(exc, "x-request-id")
- or _exception_header(exc, "request-id")
- or _exception_header(exc, "x-amzn-requestid")
- or _exception_header(exc, "x-ms-request-id")
- )
-
-
-def _exception_retry_after(exc: Exception) -> float | None:
- retry_after = _exception_header(exc, "retry-after")
- try:
- return float(retry_after) if retry_after is not None else None
- except (TypeError, ValueError):
- return None
-
-
-def _exception_provider_code(exc: Exception) -> str | None:
- body = getattr(exc, "body", None)
- if isinstance(body, dict):
- error = body.get("error")
- if isinstance(error, dict) and error.get("code") is not None:
- return str(error["code"])
- if body.get("code") is not None:
- return str(body["code"])
- return None
diff --git a/dspy/clients/lm15_boundary.py b/dspy/clients/lm15_boundary.py
new file mode 100644
index 000000000..9ad72e6b4
--- /dev/null
+++ b/dspy/clients/lm15_boundary.py
@@ -0,0 +1,93 @@
+"""Typed direct calls using the exact lm15 objects bundled with DSPy.
+
+Execution controls (response caching, retries and history) stay in DSPy. The
+provider-neutral Request contains only what is sent to the model.
+"""
+
+import json
+
+from dspy._vendor.lm15.providers.base import HttpResponse
+from dspy._vendor.lm15.serde import part_to_dict
+from dspy.clients.legacy_outputs import plain
+from dspy.lm15 import OpenAIChatLM, OpenAILM, Request, response_from_openai_chat
+
+
+class _NoTransport:
+ def stream(self, request):
+ raise RuntimeError("The typed conversion boundary must not perform network I/O")
+
+
+def request_kwargs(request: Request, model_type: str) -> dict:
+ """Serialize with lm15's dialect, never a second copy of its type system."""
+ if "n" in (request.config.extensions or {}):
+ raise ValueError("A typed Request produces one Response; do not put n in Config.extensions.")
+ if model_type not in {"chat", "responses", "text"}:
+ raise ValueError(f"Unsupported model_type: {model_type!r}")
+ # Private dialect mapping is confined here because lm15 does not expose a
+ # public outbound-body converter. No credentials are resolved or sent.
+ dialect = OpenAIChatLM(api_key="conversion-only", transport=_NoTransport())
+ if model_type == "responses":
+ dialect = OpenAILM(api_key="conversion-only", transport=_NoTransport())
+ data = dialect._payload(request, stream=False)
+ data.pop("model", None)
+ data.pop("stream", None)
+ return data
+
+
+def snapshot_request(request: Request) -> Request:
+ """Read local media once so the cache key describes the bytes actually sent."""
+ import base64
+ from dataclasses import replace
+
+ def snapshot(part):
+ path = getattr(part, "path", None)
+ if path is not None:
+ return replace(part, path=None, data=base64.b64encode(path.read_bytes()).decode("ascii"))
+ if part.type == "tool_result":
+ content = tuple(snapshot(item) for item in part.content)
+ if any(left is not right for left, right in zip(content, part.content, strict=True)):
+ return replace(part, content=content)
+ return part
+
+ messages = []
+ for message in request.messages:
+ parts = tuple(snapshot(part) for part in message.parts)
+ messages.append(replace(message, parts=parts) if any(left is not right for left, right in zip(parts, message.parts, strict=True))
+ else message)
+ system = request.system
+ if isinstance(system, tuple):
+ frozen = tuple(snapshot(part) for part in system)
+ if any(left is not right for left, right in zip(frozen, system, strict=True)):
+ system = frozen
+ if system is request.system and all(left is right for left, right in zip(messages, request.messages, strict=True)):
+ return request
+ return replace(request, messages=tuple(messages), system=system)
+
+
+def history_messages(request: Request) -> list[dict]:
+ """A display snapshot; never read local files while recording history."""
+ messages = []
+ if request.system is not None:
+ system = request.system
+ messages.append({"role": "system", "content": system if isinstance(system, str)
+ else [part_to_dict(part) for part in system]})
+ for message in request.messages:
+ messages.append({"role": message.role, "content": message.text if message.text is not None
+ else [part_to_dict(part) for part in message.parts]})
+ return messages
+
+
+def response_value(response, model_type: str, request: Request):
+ body = plain(response)
+ if model_type == "responses":
+ dialect = OpenAILM(api_key="conversion-only", transport=_NoTransport())
+ return dialect.parse_response(request, HttpResponse(
+ status=200, reason="OK", headers=[], body=json.dumps(body).encode(),
+ ))
+ if model_type == "text":
+ body = dict(body)
+ body["choices"] = [
+ {**choice, "message": {"role": "assistant", "content": choice.get("text")}}
+ for choice in body.get("choices", [])
+ ]
+ return response_from_openai_chat(body, model=request.model)
diff --git a/dspy/clients/model_metadata/LICENSE b/dspy/clients/model_metadata/LICENSE
new file mode 100644
index 000000000..3bfef5bae
--- /dev/null
+++ b/dspy/clients/model_metadata/LICENSE
@@ -0,0 +1,26 @@
+Portions of this software are licensed as follows:
+
+* All content that resides under the "enterprise/" directory of this repository, if that directory exists, is licensed under the license defined in "enterprise/LICENSE".
+* Content outside of the above mentioned directories or restrictions above is available under the MIT license as defined below.
+---
+MIT License
+
+Copyright (c) 2023 Berri AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/dspy/clients/model_metadata/__init__.py b/dspy/clients/model_metadata/__init__.py
new file mode 100644
index 000000000..5767794a8
--- /dev/null
+++ b/dspy/clients/model_metadata/__init__.py
@@ -0,0 +1,152 @@
+"""Model metadata without an inference SDK dependency.
+
+Match the existing remote-first loading policy: once on first use, fetch the
+upstream map with a five-second timeout, otherwise use the packaged backup.
+LITELLM_LOCAL_MODEL_COST_MAP and LITELLM_MODEL_COST_MAP_URL retain their meaning.
+No persistent remote cache is introduced. Metadata is advisory, not routing.
+"""
+
+import copy
+import gzip
+import json
+import logging
+import math
+import os
+import re
+import threading
+import urllib.request
+from importlib.resources import files
+
+logger = logging.getLogger(__name__)
+_URL = "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"
+_RESERVED = {"sample_spec", "fallback_generalizations"}
+_lock = threading.RLock()
+_data = None
+_source = {}
+
+
+def _snapshot():
+ return json.loads(gzip.decompress(files(__package__).joinpath("snapshot.json.gz").read_bytes()))
+
+
+def _count(data):
+ return sum(key not in _RESERVED and isinstance(value, dict) for key, value in data.items())
+
+
+async def apreload():
+ """Load once without blocking an event loop (including lock contention)."""
+ if _data is None:
+ import asyncio
+
+ await asyncio.to_thread(_load)
+
+
+def _load():
+ global _data, _source
+ # The map is published only after it is fully built. Normal reads need
+ # neither a network call nor a blocking lock after initialization.
+ if _data is not None:
+ return _data
+ with _lock:
+ if _data is not None:
+ return _data
+ local = _snapshot()
+ forced = os.getenv("LITELLM_LOCAL_MODEL_COST_MAP", "").lower() == "true"
+ url = os.getenv("LITELLM_MODEL_COST_MAP_URL", _URL)
+ data = local
+ reason = None
+ remote = False
+ if not forced:
+ try:
+ request = urllib.request.Request(url, headers={"User-Agent": "DSPy/model-metadata"})
+ with urllib.request.urlopen(request, timeout=5) as response:
+ payload = response.read(32 * 1024 * 1024 + 1)
+ if len(payload) > 32 * 1024 * 1024:
+ raise ValueError("metadata exceeds 32 MB")
+ candidate = json.loads(payload)
+ if not isinstance(candidate, dict) or _count(candidate) < max(100, _count(local) * 0.5):
+ raise ValueError("metadata is empty or has shrunk unexpectedly")
+ if any(not isinstance(v, dict) for k, v in candidate.items() if k not in _RESERVED):
+ raise ValueError("model entries must be objects")
+ data, remote = candidate, True
+ except Exception as exc:
+ # The configured URL may contain credentials; do not log it or
+ # exceptions from the HTTP client verbatim.
+ reason = type(exc).__name__
+ logger.warning("Model metadata refresh failed (%s); using bundled snapshot.", reason)
+ _source = {"source": "remote" if remote else "local", "is_env_forced": forced,
+ "fallback_reason": reason,
+ "snapshot_version": json.loads(files(__package__).joinpath("provenance.json").read_text())["version"]}
+ _data = data
+ return data
+
+
+def source_info():
+ """Describe the loaded map without exposing configured URL credentials."""
+ _load()
+ return dict(_source)
+
+
+def _provider_names(provider):
+ # Metadata namespaces, NOT inference routes. Cloud deployment prices must
+ # not fall back to a public provider's prices.
+ return {
+ "openai-chat": ("openai",), "claude-code": ("anthropic",),
+ "openai-codex": ("openai",), "vertex": ("vertex_ai", "gemini"),
+ "vertex-express": ("vertex_ai", "gemini"), "vertex-anthropic": ("vertex_ai",),
+ "azure-chat": ("azure",), "azure-anthropic": ("azure_ai",),
+ "bedrock-chat": ("bedrock",), "bedrock-mantle-chat": ("bedrock_mantle",),
+ "bedrock-anthropic": ("bedrock",), "aws-anthropic": ("aws-anthropic",),
+ "deepseek-anthropic": ("deepseek",), "meta-chat": ("meta",),
+ "meta-anthropic": ("meta",), "moonshotai": ("moonshot", "moonshotai"),
+ "moonshotai-responses": ("moonshot", "moonshotai"),
+ "moonshotai-anthropic": ("moonshot", "moonshotai"),
+ "vllm": ("hosted_vllm", "vllm"),
+ }.get(provider, (provider,))
+
+
+def model_info(provider, model):
+ """Return a copy of the most specific matching entry, or an empty dict.
+
+ Missing capability fields may inherit from a provider-matched bare entry;
+ explicit false values do not. Prices never inherit from another entry.
+ """
+ data = _load()
+ names = _provider_names(provider)
+ keys = [f"{name}/{model}" for name in names]
+ bare = data.get(model)
+ if isinstance(bare, dict) and bare.get("litellm_provider") in names:
+ keys.append(model)
+ found = next((data[key] for key in keys if key in data and key not in _RESERVED), None)
+ if found is None:
+ for key, entry in data.items():
+ if key in _RESERVED or entry.get("litellm_provider") not in names:
+ continue
+ aliases = entry.get("aliases")
+ if isinstance(aliases, list) and any(alias in keys or alias == model for alias in aliases):
+ found = entry
+ break
+ result = copy.deepcopy(found or {})
+ if isinstance(bare, dict) and bare.get("litellm_provider") in names:
+ for key, value in bare.items():
+ if key.startswith("supports_") and result.get(key) is None:
+ result[key] = value
+ # Upstream family declarations supply capabilities, never guessed pricing.
+ rules = data.get("fallback_generalizations", {}).get("rules", [])
+ for rule in rules if isinstance(rules, list) else []:
+ if not isinstance(rule, dict) or not isinstance(rule.get("pattern"), str):
+ continue
+ try:
+ matches = re.search(rule["pattern"], model) is not None
+ except re.error:
+ continue
+ if matches:
+ for key, value in (rule.get("model_info") or {}).items():
+ if key.startswith("supports_") and result.get(key) is None:
+ result[key] = value
+ return result
+
+
+def rate(info, key):
+ value = info.get(key)
+ return float(value) if isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) and value >= 0 else None
diff --git a/dspy/clients/model_metadata/provenance.json b/dspy/clients/model_metadata/provenance.json
new file mode 100644
index 000000000..e2604ee65
--- /dev/null
+++ b/dspy/clients/model_metadata/provenance.json
@@ -0,0 +1,6 @@
+{
+ "source": "https://files.pythonhosted.org/packages/9b/c8/01cd494661e5909f5e308ea1dc8b5e7e1e6ac027b784c9d66be2d3514d3a/litellm-1.96.0-cp310-cp310-macosx_10_12_x86_64.whl",
+ "version": "1.96.0",
+ "wheel_sha256": "01a6a9c8a5e9214e02b76c869031d4d31eca576559b288defc1f2eda93acd58d",
+ "snapshot_sha256": "54f7c04dda28383a81bd44c433a9bd6f3ec2a06fa6fbef35db45b2e9b04d39d4"
+}
diff --git a/dspy/clients/model_metadata/snapshot.json.gz b/dspy/clients/model_metadata/snapshot.json.gz
new file mode 100644
index 000000000..8e3a59083
Binary files /dev/null and b/dspy/clients/model_metadata/snapshot.json.gz differ
diff --git a/dspy/clients/openai_format.py b/dspy/clients/openai_format.py
deleted file mode 100644
index 2a18b136e..000000000
--- a/dspy/clients/openai_format.py
+++ /dev/null
@@ -1,1032 +0,0 @@
-"""Translate between DSPy's LM types and OpenAI-shaped JSON.
-
-This module does not call any provider. It only maps data shapes:
-
-```text
-LMRequest -> OpenAI Chat / Responses / text-completion kwargs
-provider response -> LMResponse
-```
-
-The concrete OpenAI backends import these functions, then add transport,
-streaming, and caching around them. Read the file in
-this order when learning it:
-
-1. `to_openai_chat_request()` maps chat-completion requests.
-2. `to_openai_responses_request()` maps Responses API requests.
-3. `to_openai_text_request()` maps legacy text-completion requests.
-4. `completion_to_lm_response()` and `responses_to_lm_response()` map outputs.
-5. The final utility section handles media sources, data URIs, and object/dict
- access.
-"""
-
-from __future__ import annotations
-
-import base64
-import json
-import mimetypes
-import os
-from typing import Any
-
-import pydantic
-
-from dspy.core.types import (
- LMAudioPart,
- LMBinaryPart,
- LMCitationPart,
- LMConfig,
- LMDocumentPart,
- LMImagePart,
- LMMessage,
- LMOutput,
- LMRefusalPart,
- LMRequest,
- LMResponse,
- LMTextPart,
- LMThinkingPart,
- LMToolCallPart,
- LMToolChoice,
- LMToolResultPart,
- LMToolSpec,
- LMUsage,
- LMVideoPart,
-)
-
-__all__ = [
- "to_openai_chat_request",
- "to_openai_responses_request",
- "to_openai_text_request",
- "tool_to_openai_responses",
- "completion_to_lm_response",
- "responses_to_lm_response",
- "provider_tool_call_to_part",
- "responses_function_call_to_part",
- "usage_from_response",
-]
-
-
-# ---------------------------------------------------------------------------
-# DSPy request -> OpenAI Chat Completions
-#
-# Chat Completions expects:
-# {"model": ..., "messages": [...], "tools": [...], ...config}
-# Each DSPy message stays one OpenAI message, except assistant tool calls are
-# split into OpenAI's top-level `tool_calls` field on the assistant message.
-# ---------------------------------------------------------------------------
-
-
-def to_openai_chat_request(request: LMRequest) -> dict[str, Any]:
- """Convert a normalized DSPy request into Chat Completions kwargs."""
- data = {"model": request.model, "messages": [message_to_openai_chat(message) for message in request.messages]}
- data.update(common_config_kwargs(request.config, model=request.model, endpoint="chat"))
- if request.config.tool_choice is not None:
- data.update(tool_choice_to_openai(request.config.tool_choice))
- if request.tools:
- data["tools"] = [tool_to_openai(tool) for tool in request.tools]
- return data
-
-
-def message_to_openai_chat(message: LMMessage) -> dict[str, Any]:
- """Convert one DSPy message into one Chat Completions message."""
- output: dict[str, Any] = {"role": message.role}
- if message.name is not None:
- output["name"] = message.name
-
- if message.role == "assistant":
- tool_calls = [part for part in message.parts if isinstance(part, LMToolCallPart)]
- content_parts = [part for part in message.parts if not isinstance(part, LMToolCallPart)]
- output["content"] = None if tool_calls and not content_parts else parts_to_openai_content(content_parts)
- if tool_calls:
- output["tool_calls"] = [assistant_tool_call_to_openai(part) for part in tool_calls]
- return output
-
- if message.role == "tool" and len(message.parts) == 1 and isinstance(message.parts[0], LMToolResultPart):
- result = message.parts[0]
- output.update(tool_result_to_openai(result))
- if result.call_id is not None:
- output["tool_call_id"] = result.call_id
- if result.name is not None:
- output["name"] = result.name
- return output
-
- output["content"] = parts_to_openai_content(message.parts)
- return output
-
-
-# ---------------------------------------------------------------------------
-# DSPy request -> OpenAI Responses
-#
-# Responses expects a flat `input` list, not a chat `messages` list. A single
-# DSPy assistant message can therefore become two kinds of input items: one
-# message item for content and one function_call item for each tool call.
-# ---------------------------------------------------------------------------
-
-
-def to_openai_responses_request(request: LMRequest, *, enforce_reasoning_temperature: bool = True) -> dict[str, Any]:
- """Convert a normalized DSPy request into Responses API kwargs."""
- config = request.config
- data: dict[str, Any] = {
- "model": request.model,
- "input": [item for message in request.messages for item in message_to_responses_input_items(message)],
- }
- data.update(
- responses_config_kwargs(config, model=request.model, enforce_reasoning_temperature=enforce_reasoning_temperature)
- )
- if config.tool_choice is not None:
- data.update(tool_choice_to_openai_responses(config.tool_choice))
- if request.tools:
- data["tools"] = [tool_to_openai_responses(tool) for tool in request.tools]
- return data
-
-
-def message_to_responses_input_items(message: LMMessage) -> list[dict[str, Any]]:
- """Convert one DSPy message into one or more Responses input items."""
- if message.role == "tool" and len(message.parts) == 1 and isinstance(message.parts[0], LMToolResultPart):
- result = message.parts[0]
- item = {"type": "function_call_output", "output": responses_tool_output_text(tool_result_to_openai(result)["content"])}
- if result.call_id is not None:
- item["call_id"] = result.call_id
- return [item]
-
- tool_calls = [part for part in message.parts if isinstance(part, LMToolCallPart)]
- content_parts = [part for part in message.parts if not isinstance(part, LMToolCallPart)]
- content = parts_to_responses_content(content_parts, role=message.role)
- items: list[dict[str, Any]] = []
-
- if content or message.role != "assistant" or not tool_calls:
- item: dict[str, Any] = {"role": message.role, "content": content}
- if message.name is not None:
- item["name"] = message.name
- items.append(item)
-
- if message.role == "assistant":
- items.extend(tool_call_to_responses_input(tool_call) for tool_call in tool_calls)
- return items
-
-
-def parts_to_responses_content(parts: list[Any], role: str = "user") -> list[dict[str, Any]]:
- blocks = parts_to_openai_content(parts)
- if isinstance(blocks, str):
- return [{"type": _responses_text_type(role), "text": blocks}]
- return [content_block_to_responses(block, role) for block in blocks]
-
-
-def _responses_text_type(role: str) -> str:
- # The Responses API types text by direction, not just by role slot: assistant
- # input items replay *model output*, so their text must be "output_text";
- # "input_text" on an assistant item is rejected with a 400.
- return "output_text" if role == "assistant" else "input_text"
-
-
-def tool_call_to_responses_input(tool_call_part: LMToolCallPart) -> dict[str, Any]:
- tool_call = assistant_tool_call_to_openai(tool_call_part)
- function = tool_call.get("function", {})
- item = {"type": "function_call", "name": function.get("name", ""), "arguments": function.get("arguments", "{}")}
- call_id = tool_call.get("id") or tool_call.get("call_id")
- if call_id is not None:
- item["call_id"] = call_id
- return item
-
-
-def content_block_to_responses(block: dict[str, Any], role: str = "user") -> dict[str, Any]:
- block_type = block.get("type")
- if block_type == "text":
- return {"type": _responses_text_type(role), "text": block.get("text", "")}
- if block_type == "image_url":
- image_url = block.get("image_url", {})
- out = {"type": "input_image", "image_url": image_url.get("url", "")}
- if image_url.get("detail") is not None:
- out["detail"] = image_url["detail"]
- return out
- if block_type == "input_audio":
- return block
- if block_type == "file":
- file = block.get("file", {})
- return {
- "type": "input_file",
- "file_data": file.get("file_data"),
- "filename": file.get("filename"),
- "file_id": file.get("file_id"),
- }
- return block
-
-
-# ---------------------------------------------------------------------------
-# DSPy request -> OpenAI text completions
-#
-# Text completions have no native message roles. We concatenate text-only
-# messages with blank lines and append DSPy's historical response marker.
-# ---------------------------------------------------------------------------
-
-
-def to_openai_text_request(request: LMRequest) -> dict[str, Any]:
- """Convert a normalized DSPy request into text-completion kwargs."""
- data = {"model": request.model, "prompt": messages_to_text_prompt(request.messages)}
- data.update(text_config_kwargs(request.config))
- return data
-
-
-def messages_to_text_prompt(messages: list[LMMessage]) -> str:
- """Flatten text-only messages into the prompt used by text completions."""
- chunks = []
- for message in messages:
- texts = []
- for part in message.parts:
- if not isinstance(part, LMTextPart):
- raise ValueError(f"OpenAI text completions only support text parts, but received {type(part).__name__}.")
- texts.append(part.text)
- chunks.append("".join(texts))
- return "\n\n".join(chunks + ["BEGIN RESPONSE:"])
-
-
-# ---------------------------------------------------------------------------
-# Shared request mappers
-#
-# The three OpenAI endpoint families use slightly different outer envelopes, but most
-# leaf values are the same: text parts, media parts, tools, tool choices, and
-# generation config. The helpers below keep those conversions in one place.
-# ---------------------------------------------------------------------------
-
-
-def parts_to_openai_content(parts: list[Any]) -> str | list[dict[str, Any]]:
- """Convert DSPy parts into OpenAI `content`.
-
- OpenAI accepts a bare string for the common single-text case. Mixed content
- becomes a list of content blocks.
- """
- if len(parts) == 1 and isinstance(parts[0], LMTextPart) and "legacy_content_block" not in parts[0].metadata:
- return parts[0].text
- blocks: list[dict[str, Any]] = []
- for part in parts:
- blocks.extend(part_to_openai_blocks(part))
- return blocks
-
-
-def part_to_openai_blocks(part: Any) -> list[dict[str, Any]]:
- """Convert one DSPy part into one or more OpenAI content blocks."""
- legacy_block = getattr(part, "metadata", {}).get("legacy_content_block")
- if legacy_block is not None:
- return [dict(legacy_block)]
- if isinstance(part, LMTextPart):
- return [{"type": "text", "text": part.text}]
- if isinstance(part, LMImagePart):
- return [image_to_openai(part)]
- if isinstance(part, LMDocumentPart):
- return document_to_openai_blocks(part)
- if isinstance(part, LMAudioPart):
- return [audio_to_openai(part)]
- if isinstance(part, LMVideoPart):
- return [video_to_openai(part)]
- if isinstance(part, LMBinaryPart):
- return [binary_to_openai(part)]
- if isinstance(part, LMThinkingPart):
- return [{"type": "text", "text": part.text}]
- if isinstance(part, LMCitationPart):
- citation = " ".join(value for value in (part.title, part.text, part.url) if value)
- return [{"type": "text", "text": citation}]
- if isinstance(part, LMToolResultPart):
- return part_to_openai_blocks(LMTextPart(text="".join(part_text(value) for value in part.content)))
- return [{"type": "text", "text": str(part)}]
-
-
-def image_to_openai(image: LMImagePart) -> dict[str, Any]:
- image_url: dict[str, Any] = {"url": media_source(image)}
- if image.detail is not None:
- image_url["detail"] = image.detail
- return {"type": "image_url", "image_url": image_url}
-
-
-def audio_to_openai(audio: LMAudioPart) -> dict[str, Any]:
- if audio.data is not None:
- data = audio.data
- media_type = audio.media_type
- elif audio.path is not None:
- data = read_path_base64(audio.path)
- media_type = media_type_for_path(audio.path, fallback=audio.media_type)
- else:
- raise ValueError("OpenAI-format audio input requires base64 `data` or local `path`.")
- return {"type": "input_audio", "input_audio": {"data": data, "format": media_format(media_type)}}
-
-
-def document_to_openai_blocks(document: LMDocumentPart) -> list[dict[str, Any]]:
- block: dict[str, Any] = {"type": "document"}
- if document.source is not None:
- block["source"] = document.source
- else:
- block["source"] = media_source(document)
- block["media_type"] = document.media_type
- if document.citations:
- block["citations"] = document.citations
- if document.title is not None:
- block["title"] = document.title
- if document.context is not None:
- block["context"] = document.context
- return [block]
-
-
-def video_to_openai(video: LMVideoPart) -> dict[str, Any]:
- filename = os.path.basename(video.path) if video.path is not None else None
- return binary_to_openai(
- LMBinaryPart(
- data=video.data,
- url=video.url,
- file_id=video.file_id,
- path=video.path,
- media_type=video.media_type,
- filename=filename,
- )
- )
-
-
-def binary_to_openai(binary: LMBinaryPart) -> dict[str, Any]:
- file_data: dict[str, Any] = {}
- if binary.data is not None:
- file_data["file_data"] = data_uri(binary.media_type, binary.data)
- elif binary.path is not None:
- file_data["file_data"] = data_uri_from_path(binary.path, fallback_media_type=binary.media_type)
- file_data["filename"] = binary.filename or os.path.basename(binary.path)
- elif binary.url is not None:
- file_data["file_data"] = binary.url
- if binary.file_id is not None:
- file_data["file_id"] = binary.file_id
- if binary.filename is not None:
- file_data["filename"] = binary.filename
- return {"type": "file", "file": file_data}
-
-
-_TOOL_SPEC_WIRE_KEYS = {"type", "name", "description", "parameters", "strict"}
-
-
-def _tool_provider_extras(tool: LMToolSpec) -> dict[str, Any]:
- return {key: value for key, value in tool.provider_data.items() if key not in _TOOL_SPEC_WIRE_KEYS}
-
-
-def tool_to_openai(tool: LMToolSpec) -> dict[str, Any]:
- # provider_data holds function-scoped extras; each dialect emits them where
- # it puts function fields — nested under "function" here, flattened to the
- # top level in the Responses shape.
- data = {"type": "function", "function": {"name": tool.name, "parameters": tool.parameters}}
- if tool.description is not None:
- data["function"]["description"] = tool.description
- if tool.strict is not None:
- data["function"]["strict"] = tool.strict
- data["function"].update(_tool_provider_extras(tool))
- return data
-
-
-def tool_to_openai_responses(tool: LMToolSpec) -> dict[str, Any]:
- """Convert a normalized tool spec into Responses API function-tool shape."""
- data = {"type": "function", "name": tool.name, "parameters": tool.parameters}
- if tool.description is not None:
- data["description"] = tool.description
- if tool.strict is not None:
- data["strict"] = tool.strict
- data.update(_tool_provider_extras(tool))
- return data
-
-
-def tool_choice_to_openai(choice: LMToolChoice) -> dict[str, Any]:
- if choice.allowed:
- if len(choice.allowed) != 1 or choice.mode not in {"required", "auto"}:
- raise ValueError(
- "OpenAI-format tool_choice only supports constraining to a single allowed tool "
- "with mode 'required' or 'auto'."
- )
- data: dict[str, Any] = {"tool_choice": {"type": "function", "function": {"name": choice.allowed[0]}}}
- else:
- data = {"tool_choice": choice.mode}
- if choice.parallel is not None:
- data["parallel_tool_calls"] = choice.parallel
- return data
-
-
-def tool_choice_to_openai_responses(choice: LMToolChoice) -> dict[str, Any]:
- if choice.allowed:
- if len(choice.allowed) != 1 or choice.mode not in {"required", "auto"}:
- raise ValueError(
- "OpenAI Responses tool_choice only supports constraining to a single allowed tool "
- "with mode 'required' or 'auto'."
- )
- data: dict[str, Any] = {"tool_choice": {"type": "function", "name": choice.allowed[0]}}
- else:
- data = {"tool_choice": choice.mode}
- if choice.parallel is not None:
- data["parallel_tool_calls"] = choice.parallel
- return data
-
-
-# provider_data on parsed tool calls echoes raw provider fields (the fc_* item
-# id, status) and parser diagnostics; none of these may override the canonical
-# wire fields or leak into requests.
-_TOOL_CALL_INTERNAL_KEYS = {"id", "call_id", "status", "raw_arguments", "arguments_parse_error"}
-
-
-def assistant_tool_call_to_openai(call: LMToolCallPart) -> dict[str, Any]:
- data = {key: value for key, value in call.provider_data.items() if key not in _TOOL_CALL_INTERNAL_KEYS}
- data["type"] = "function"
- data["function"] = {"name": call.name, "arguments": json.dumps(call.args)}
- if call.id is not None:
- data["id"] = call.id
- return data
-
-
-def tool_result_to_openai(result: LMToolResultPart) -> dict[str, Any]:
- return {"content": parts_to_openai_content(result.content)}
-
-
-def common_config_kwargs(config: LMConfig, *, model: str | None = None, endpoint: str = "chat") -> dict[str, Any]:
- """Convert shared DSPy config fields into Chat Completions kwargs."""
- data = dict(config.extensions)
- _validate_openai_reasoning_temperature(config, model=model, endpoint=endpoint)
- for key in ("temperature", "top_p"):
- value = getattr(config, key)
- if value is not None:
- data[key] = value
- if config.max_tokens is not None:
- token_key = "max_completion_tokens" if _uses_max_completion_tokens(model) else "max_tokens"
- data[token_key] = config.max_tokens
- if config.stop:
- data["stop"] = config.stop
- if config.logprobs is not None:
- data["logprobs"] = config.logprobs
- if config.n is not None:
- data["n"] = config.n
- if config.response_format is not None:
- data["response_format"] = config.response_format
- if config.reasoning is not None:
- data.update(reasoning_to_chat_kwargs(config.reasoning))
- if config.prompt_cache is not None:
- data.update(prompt_cache_to_kwargs(config.prompt_cache))
- return data
-
-
-def responses_config_kwargs(
- config: LMConfig, *, model: str | None = None, enforce_reasoning_temperature: bool = True
-) -> dict[str, Any]:
- """Convert shared DSPy config fields into Responses API kwargs."""
- data = dict(config.extensions) if config.extensions else {}
- if enforce_reasoning_temperature:
- _validate_openai_reasoning_temperature(config, model=model, endpoint="responses")
- for key in ("temperature", "top_p"):
- value = getattr(config, key)
- if value is not None:
- data[key] = value
- if config.max_tokens is not None:
- data["max_output_tokens"] = config.max_tokens
- if config.n is not None:
- data["n"] = config.n
- if config.logprobs is not None:
- data["logprobs"] = config.logprobs
- if config.stop:
- data["stop"] = config.stop
- if config.reasoning is not None:
- data.update(reasoning_to_responses_kwargs(config.reasoning))
- if config.prompt_cache is not None:
- data.update(prompt_cache_to_kwargs(config.prompt_cache))
- if config.response_format is not None:
- text = data.pop("text", {})
- data["text"] = {**text, "format": response_format_to_responses(config.response_format)}
- return data
-
-
-def text_config_kwargs(config: LMConfig) -> dict[str, Any]:
- """Convert shared DSPy config fields into text-completion kwargs."""
- data = dict(config.extensions)
- for key in ("temperature", "max_tokens", "top_p"):
- value = getattr(config, key)
- if value is not None:
- data[key] = value
- if config.stop:
- data["stop"] = config.stop
- if config.logprobs is not None:
- data["logprobs"] = config.logprobs
- if config.n is not None:
- data["n"] = config.n
- return data
-
-
-def reasoning_to_chat_kwargs(reasoning: Any) -> dict[str, Any]:
- data = {}
- if reasoning.effort is not None:
- data["reasoning_effort"] = reasoning.effort
- return data
-
-
-def reasoning_to_responses_kwargs(reasoning: Any) -> dict[str, Any]:
- data = {}
- if reasoning.effort is not None:
- data["effort"] = reasoning.effort
- if reasoning.summary is not None:
- data["summary"] = reasoning.summary
- return {"reasoning": data} if data else {}
-
-
-def _validate_openai_reasoning_temperature(config: LMConfig, *, model: str | None, endpoint: str) -> None:
- if not _is_openai_reasoning_model(model):
- return
- effort = getattr(config.reasoning, "effort", None) if config.reasoning is not None else None
- if effort in {None, "none"}:
- return
- if config.temperature in {None, 1}:
- return
-
- from dspy.utils.exceptions import LMUnsupportedFeatureError
-
- raise LMUnsupportedFeatureError(
- "OpenAI reasoning models only support the default temperature when reasoning effort is active. "
- "Use temperature=None or temperature=1, or set reasoning_effort='none'.",
- model=model,
- provider="openai",
- features=["temperature", "reasoning"],
- issues=[
- f"{endpoint} request used reasoning effort {effort!r} with temperature={config.temperature!r}.",
- ],
- )
-
-
-def _uses_max_completion_tokens(model: str | None) -> bool:
- return _is_openai_reasoning_model(model)
-
-
-def _is_openai_reasoning_model(model: str | None) -> bool:
- if not isinstance(model, str):
- return False
- model_name = model.removeprefix("openai/").lower()
- if "chat" in model_name:
- return False
- return model_name.startswith(("o1", "o3", "o4", "gpt-5"))
-
-
-def prompt_cache_to_kwargs(cache: Any) -> dict[str, Any]:
- data = {}
- if cache.key is not None:
- data["prompt_cache_key"] = cache.key
- if cache.enabled is False:
- data["prompt_cache"] = False
- return data
-
-
-def response_format_to_responses(value: Any) -> Any:
- if isinstance(value, type) and issubclass(value, pydantic.BaseModel):
- return {
- "name": value.__name__,
- "type": "json_schema",
- "schema": _close_object_schemas(value.model_json_schema()),
- }
- return value
-
-
-def _close_object_schemas(schema: Any) -> Any:
- """Recursively supply ``additionalProperties: false`` on object schemas.
-
- The Responses API rejects ``json_schema`` formats whose object schemas leave
- ``additionalProperties`` unspecified; pydantic's ``model_json_schema()``
- omits it. Schemas that already set it (e.g. ``dict[str, T]`` fields) are
- left untouched. Recursion follows JSON Schema keyword positions only, so
- schema-shaped values inside ``default``/``examples``/``const`` are never
- rewritten.
- """
- if not isinstance(schema, dict):
- return schema
- closed = dict(schema)
- for key in ("items", "additionalProperties", "not", "if", "then", "else", "contains", "propertyNames"):
- if isinstance(closed.get(key), dict):
- closed[key] = _close_object_schemas(closed[key])
- for key in ("items", "prefixItems", "anyOf", "oneOf", "allOf"):
- if isinstance(closed.get(key), list):
- closed[key] = [_close_object_schemas(item) for item in closed[key]]
- for key in ("properties", "patternProperties", "$defs", "definitions"):
- if isinstance(closed.get(key), dict):
- closed[key] = {name: _close_object_schemas(sub) for name, sub in closed[key].items()}
- if closed.get("type") == "object" and "additionalProperties" not in closed:
- closed["additionalProperties"] = False
- return closed
-
-
-def responses_tool_output_text(content: Any) -> str:
- if isinstance(content, str):
- return content
- if isinstance(content, list):
- return "".join(
- block.get("text", "") if isinstance(block, dict) and block.get("type") in {"text", "input_text"} else str(block)
- for block in content
- )
- return str(content)
-
-
-# ---------------------------------------------------------------------------
-# OpenAI response -> DSPy response
-#
-# Chat and text completions return `choices`; Responses returns one `output`
-# list. Both are normalized to an `LMResponse` with one or more `LMOutput`s and
-# typed DSPy parts.
-# ---------------------------------------------------------------------------
-
-
-def completion_to_lm_response(response: Any, request: LMRequest) -> LMResponse:
- """Convert an OpenAI Chat or text completion response into `LMResponse`."""
- choices = get_value(response, "choices", []) or []
- return LMResponse(
- model=get_value(response, "model") or request.model,
- outputs=[choice_to_lm_output(choice) for choice in choices],
- usage=usage_from_response(response),
- cache_hit=bool(get_value(response, "cache_hit", False)),
- response_id=get_value(response, "id"),
- provider_response=response,
- )
-
-
-def choice_to_lm_output(choice: Any) -> LMOutput:
- """Convert one completion choice into one DSPy output candidate."""
- message = get_value(choice, "message")
- parts = []
- if message is not None:
- reasoning = get_value(message, "reasoning_content")
- if reasoning:
- parts.append(LMThinkingPart(text=str(reasoning)))
- content = get_value(message, "content")
- if content:
- parts.extend(message_content_to_parts(content))
- for tool_call in get_value(message, "tool_calls") or []:
- parts.append(provider_tool_call_to_part(tool_call))
- parts.extend(extract_citations_from_choice(choice))
- else:
- text = get_value(choice, "text")
- if text:
- parts.extend(message_content_to_parts(text))
- finish_reason = get_value(choice, "finish_reason")
- return LMOutput(
- parts=parts,
- finish_reason=finish_reason,
- truncated=finish_reason == "length",
- logprobs=get_value(choice, "logprobs"),
- provider_output=choice,
- )
-
-
-def responses_to_lm_response(response: Any, request: LMRequest) -> LMResponse:
- """Convert an OpenAI Responses object into `LMResponse`.
-
- The Responses API represents one assistant answer as a sequence of output
- items: messages, function calls, reasoning, binary artifacts, images, and refusals.
- DSPy stores those as typed parts on one `LMOutput`.
- """
- parts = []
- for output_item in get_value(response, "output", []) or []:
- output_type = get_value(output_item, "type")
- if output_type == "message":
- for content_item in get_value(output_item, "content", []) or []:
- parts.extend(response_content_item_to_parts(content_item))
- parts.extend(responses_annotations_to_citations(content_item))
- elif output_type == "function_call":
- parts.append(responses_function_call_to_part(output_item))
- elif output_type in {"image", "output_image", "image_generation_call"}:
- parts.append(output_image_to_part(output_item))
- elif output_type in {"audio", "output_audio"}:
- parts.append(output_audio_to_part(output_item))
- elif output_type in {"file", "output_file"}:
- parts.append(output_file_to_part(output_item))
- elif output_type == "refusal":
- parts.append(refusal_to_part(output_item))
- elif output_type == "reasoning":
- for item in get_value(output_item, "content") or get_value(output_item, "summary") or []:
- text = get_value(item, "text")
- if text:
- parts.append(LMThinkingPart(text=text))
- return LMResponse(
- model=get_value(response, "model") or request.model,
- outputs=[LMOutput(parts=parts, provider_output=response)],
- usage=usage_from_response(response),
- cache_hit=bool(get_value(response, "cache_hit", False)),
- response_id=get_value(response, "id"),
- provider_response=response,
- )
-
-
-def message_content_to_parts(content: Any) -> list[Any]:
- if isinstance(content, str):
- return [LMTextPart(text=content)]
- if not isinstance(content, list):
- return [LMTextPart(text=str(content))]
- parts = []
- for item in content:
- parts.extend(response_content_item_to_parts(item))
- return parts
-
-
-def response_content_item_to_parts(item: Any) -> list[Any]:
- item_type = get_value(item, "type")
- text = get_value(item, "text")
- if item_type in {"text", "output_text", "input_text"} or (text is not None and item_type is None):
- return [LMTextPart(text=text)]
- if item_type in {"refusal", "output_refusal"}:
- return [refusal_to_part(item)]
- if item_type in {"image", "output_image", "image_url"}:
- return [output_image_to_part(item)]
- if item_type in {"audio", "output_audio", "input_audio"}:
- return [output_audio_to_part(item)]
- if item_type in {"file", "output_file", "input_file"}:
- return [output_file_to_part(item)]
- if item_type in {"tool_call", "function_call"}:
- return [provider_tool_call_to_part(item)]
- return []
-
-
-def provider_tool_call_to_part(tool_call: Any) -> LMToolCallPart:
- """Convert an OpenAI-shaped tool call into a DSPy tool-call part."""
- function = get_value(tool_call, "function", {})
- name = get_value(function, "name") or get_value(tool_call, "name")
- arguments = get_value(function, "arguments", get_value(tool_call, "arguments", "{}"))
- provider_data = model_dump(tool_call)
- try:
- args = json.loads(arguments) if isinstance(arguments, str) else dict(arguments)
- except Exception as error:
- args = {}
- provider_data["raw_arguments"] = arguments
- provider_data["arguments_parse_error"] = str(error)
- call_id = get_value(tool_call, "call_id") or get_value(tool_call, "id")
- return LMToolCallPart(id=call_id, name=name or "", args=args, provider_data=provider_data)
-
-
-def responses_function_call_to_part(output_item: Any) -> LMToolCallPart:
- """Convert one Responses function_call item into a DSPy tool-call part."""
- args = get_value(output_item, "arguments", {})
- provider_data = model_dump(output_item)
- if isinstance(args, str):
- try:
- args = json.loads(args)
- except Exception as error:
- provider_data["raw_arguments"] = args
- provider_data["arguments_parse_error"] = str(error)
- args = {}
- return LMToolCallPart(id=get_value(output_item, "call_id"), name=get_value(output_item, "name", ""), args=args, provider_data=provider_data)
-
-
-def citation_to_part(citation: Any) -> LMCitationPart:
- if hasattr(citation, "model_dump"):
- citation = model_dump(citation)
- if not isinstance(citation, dict):
- citation = {"text": str(citation)}
- citation_fields = {"cited_text", "text", "supported_text", "document_title", "title", "url"}
- return LMCitationPart(
- text=citation.get("cited_text") or citation.get("text") or citation.get("supported_text"),
- title=citation.get("document_title") or citation.get("title"),
- url=citation.get("url"),
- metadata={key: value for key, value in citation.items() if key not in citation_fields},
- )
-
-
-def extract_citations_from_choice(choice: Any) -> list[LMCitationPart]:
- try:
- message = get_value(choice, "message")
- provider_specific_fields = get_value(message, "provider_specific_fields", {}) or {}
- citations_data = provider_specific_fields.get("citations")
- if isinstance(citations_data, list):
- citations = []
- for item in citations_data:
- citations.extend(item if isinstance(item, list) else [item])
- return [citation_to_part(citation) for citation in citations]
- except Exception:
- return []
- return []
-
-
-def responses_annotations_to_citations(content_item: Any) -> list[LMCitationPart]:
- return [citation_to_part(annotation) for annotation in get_value(content_item, "annotations", []) or []]
-
-
-def output_image_to_part(value: Any) -> LMImagePart:
- data = model_dump(value)
- image_url = data.get("image_url")
- if isinstance(image_url, dict):
- image_url = image_url.get("url")
- source = image_url or data.get("url")
- b64_data = data.get("b64_json") or data.get("data")
- file_id = data.get("file_id")
- media_type = data.get("media_type") or data.get("mime_type") or "image/png"
- detail = data.get("detail")
- if b64_data is not None:
- if isinstance(b64_data, str) and b64_data.startswith("data:"):
- media_type, b64_data = split_data_uri(b64_data)
- return LMImagePart(data=b64_data, media_type=media_type, detail=detail)
- if source is not None:
- return LMImagePart(url=source, media_type=media_type, detail=detail)
- if file_id is not None:
- return LMImagePart(file_id=file_id, media_type=media_type, detail=detail)
- raise ValueError("Provider image output did not include data, url, or file_id.")
-
-
-def output_audio_to_part(value: Any) -> LMAudioPart:
- data = model_dump(value)
- audio = data.get("audio") if isinstance(data.get("audio"), dict) else data
- source = audio.get("url")
- b64_data = audio.get("data") or audio.get("b64_json")
- file_id = audio.get("file_id")
- media_type = audio.get("media_type") or audio.get("mime_type") or "audio/wav"
- if b64_data is not None:
- if isinstance(b64_data, str) and b64_data.startswith("data:"):
- media_type, b64_data = split_data_uri(b64_data)
- return LMAudioPart(data=b64_data, media_type=media_type)
- if source is not None:
- return LMAudioPart(url=source, media_type=media_type)
- if file_id is not None:
- return LMAudioPart(file_id=file_id, media_type=media_type)
- raise ValueError("Provider audio output did not include data, url, or file_id.")
-
-
-def output_file_to_part(value: Any) -> LMBinaryPart:
- data = model_dump(value)
- file = data.get("file") if isinstance(data.get("file"), dict) else data
- source = file.get("url")
- b64_data = file.get("file_data") or file.get("data")
- file_id = file.get("file_id") or file.get("id")
- filename = file.get("filename")
- media_type = file.get("media_type") or file.get("mime_type") or "application/octet-stream"
- if b64_data is not None:
- if isinstance(b64_data, str) and b64_data.startswith("data:"):
- media_type, b64_data = split_data_uri(b64_data)
- return LMBinaryPart(data=b64_data, media_type=media_type, filename=filename)
- if source is not None:
- return LMBinaryPart(url=source, media_type=media_type, filename=filename)
- if file_id is not None:
- return LMBinaryPart(file_id=file_id, media_type=media_type, filename=filename)
- raise ValueError("Provider file output did not include data, url, or file_id.")
-
-
-def refusal_to_part(value: Any) -> LMRefusalPart:
- text = get_value(value, "refusal") or get_value(value, "text") or get_value(value, "content") or str(value)
- return LMRefusalPart(text=str(text))
-
-
-def cost_from_response(response: Any) -> float | None:
- hidden = getattr(response, "_hidden_params", None) or {}
- return hidden.get("response_cost") if isinstance(hidden, dict) else None
-
-
-def usage_from_response(response: Any) -> LMUsage | None:
- """Convert provider usage objects or dictionaries into `LMUsage`."""
- usage = get_value(response, "usage")
- if usage is None:
- return None
- if hasattr(usage, "model_dump"):
- usage = model_dump(usage)
- elif not isinstance(usage, dict):
- data = {}
- for key in dir(usage):
- if key.startswith("_"):
- continue
- try:
- value = getattr(usage, key)
- except Exception:
- continue
- if value is not None and not callable(value):
- data[key] = value
- usage = data
- return LMUsage(**dict(usage))
-
-
-# ---------------------------------------------------------------------------
-# Small utilities
-# ---------------------------------------------------------------------------
-
-
-def media_source(part: LMImagePart | LMAudioPart | LMDocumentPart | LMBinaryPart) -> str:
- if part.data is not None:
- return data_uri(part.media_type, part.data)
- if part.url is not None:
- return part.url
- if part.file_id is not None:
- return part.file_id
- if part.path is not None:
- return data_uri_from_path(part.path, fallback_media_type=part.media_type)
- raise ValueError(f"{type(part).__name__} has no media source.")
-
-
-def read_path_base64(path: str) -> str:
- with open(path, "rb") as file:
- return base64.b64encode(file.read()).decode("ascii")
-
-
-def media_type_for_path(path: str, *, fallback: str) -> str:
- return mimetypes.guess_type(path)[0] or fallback
-
-
-def data_uri_from_path(path: str, *, fallback_media_type: str) -> str:
- return data_uri(media_type_for_path(path, fallback=fallback_media_type), read_path_base64(path))
-
-
-def data_uri(media_type: str, data: str) -> str:
- if data.startswith("data:"):
- return data
- return f"data:{media_type};base64,{data}"
-
-
-def split_data_uri(value: str) -> tuple[str, str]:
- if not value.startswith("data:") or "," not in value:
- return "application/octet-stream", value
- header, data = value.split(",", 1)
- media_type = header.removeprefix("data:").split(";", 1)[0]
- return media_type, data
-
-
-def media_format(media_type: str) -> str:
- format_ = media_type.split("/", 1)[1] if "/" in media_type else media_type
- return {"x-wav": "wav", "mpeg": "mp3"}.get(format_, format_)
-
-
-def part_text(value: Any) -> str:
- return value.text if isinstance(value, LMTextPart) else str(value)
-
-
-def get_value(value: Any, key: str, default: Any = None) -> Any:
- return value.get(key, default) if isinstance(value, dict) else getattr(value, key, default)
-
-
-def _rebuild_pydantic_serializers(value: Any) -> None:
- stack = [value]
- seen = set()
- for item in stack:
- if id(item) in seen:
- continue
- seen.add(id(item))
- if isinstance(item, pydantic.BaseModel):
- type(item).model_rebuild(force=True, raise_errors=False)
- stack.extend([*item.__dict__.values(), *((item.__pydantic_extra__ or {}).values())])
- elif isinstance(item, (dict, list, tuple)):
- stack.extend(item.values() if isinstance(item, dict) else item)
-
-
-def model_dump(value: Any) -> dict[str, Any]:
- if hasattr(value, "model_dump"):
- try:
- return value.model_dump(exclude_none=True)
- except TypeError as error:
- if "MockValSer" not in str(error):
- raise
- _rebuild_pydantic_serializers(value)
- return value.model_dump(exclude_none=True)
- if isinstance(value, dict):
- return dict(value)
- data = {}
- for key in ("id", "call_id", "type", "name", "arguments", "status", "text", "refusal", "url", "data", "file_id", "filename", "media_type", "mime_type"):
- item = getattr(value, key, None)
- if item is not None:
- data[key] = item
- return data
-
-
-# ---------------------------------------------------------------------------
-# Current BaseLM adapter compatibility helpers
-#
-# These helpers keep today's adapter->BaseLM boundary normalized internally while
-# BaseLM still returns legacy `list[str | dict | None]` outputs. Future native
-# LanguageModel implementations should use completion_to_lm_response() /
-# responses_to_lm_response() directly.
-# ---------------------------------------------------------------------------
-
-
-def lm_response_from_legacy_outputs(outputs: list[dict[str, Any] | str | None], request: LMRequest) -> LMResponse:
- """Normalize current legacy `BaseLM` outputs into an `LMResponse`."""
- if not outputs:
- return LMResponse(model=request.model, outputs=[LMOutput(parts=[], metadata={"empty_legacy_outputs": True})])
- return LMResponse(model=request.model, outputs=[lm_output_from_legacy_output(output) for output in outputs])
-
-
-def legacy_outputs_from_lm_response(response: LMResponse) -> list[dict[str, Any] | str | None]:
- """Return legacy adapter postprocess values from a normalized response.
-
- This is a temporary compatibility helper for current adapter postprocessing.
- Prefer the exact original legacy output when it is available so compatibility
- hooks that distinguish `str` from provider dictionaries keep working.
- """
- outputs = []
- for output in response.outputs:
- if output.metadata.get("empty_legacy_outputs"):
- continue
- if output.provider_output is not None:
- outputs.append(output.provider_output)
- elif output.text is not None and not output.reasoning_content and not output.tool_calls and not output.citations and output.logprobs is None:
- outputs.append(output.text)
- else:
- outputs.append(output.to_output_dict())
- return outputs
-
-
-def lm_output_from_legacy_output(output: dict[str, Any] | str | None) -> LMOutput:
- """Normalize one current legacy `BaseLM` output item into an `LMOutput`."""
- if isinstance(output, str):
- return LMOutput(parts=[LMTextPart(text=output)], provider_output=output)
- if output is None:
- return LMOutput(parts=[])
-
- parts = []
- text = output.get("text")
- if text:
- parts.append(LMTextPart(text=text))
- reasoning = output.get("reasoning_content")
- if reasoning:
- parts.append(LMThinkingPart(text=str(reasoning)))
- for tool_call in output.get("tool_calls") or []:
- parts.append(provider_tool_call_to_part(tool_call))
- for citation in output.get("citations") or []:
- parts.append(citation_to_part(citation))
- return LMOutput(parts=parts, logprobs=output.get("logprobs"), provider_output=output)
diff --git a/dspy/core/__init__.py b/dspy/core/__init__.py
index 2f72f0024..d9c93a758 100644
--- a/dspy/core/__init__.py
+++ b/dspy/core/__init__.py
@@ -1,4 +1,4 @@
-"""Core typed request/response objects used by DSPy internals."""
+"""The experimental LM vocabulary moved to :mod:`dspy.lm15`.
-from dspy.core.types import *
-from dspy.core.types import __all__ as __all__
+No old type aliases are provided: the new objects have different semantics.
+"""
diff --git a/dspy/core/types.py b/dspy/core/types.py
index 62f708ef2..7db0a2a3e 100644
--- a/dspy/core/types.py
+++ b/dspy/core/types.py
@@ -1,2063 +1,10 @@
-"""Normalized request, response, and stream types for DSPy language models."""
-
-from __future__ import annotations
-
-import json
-import mimetypes
-from collections.abc import AsyncIterator, Callable, Iterator, Mapping
-from dataclasses import dataclass
-from dataclasses import field as dataclass_field
-from pprint import pformat
-from typing import Annotated, Any, Literal
-from urllib.parse import urlparse
-
-import pydantic
-from pydantic import BaseModel, ConfigDict, Field, model_validator
-
-__all__ = [
- "Assistant",
- "Developer",
- "LMAudioPart",
- "LMBinaryPart",
- "LMCacheConfig",
- "LMCitationPart",
- "LMConfig",
- "LMDocumentPart",
- "LMImagePart",
- "LMMessage",
- "LMOutput",
- "LMPart",
- "LMPromptCacheConfig",
- "LMReasoningConfig",
- "LMRefusalPart",
- "LMRequest",
- "LMResponse",
- "LMTextPart",
- "LMThinkingPart",
- "LMToolCallPart",
- "LMToolChoice",
- "LMToolResultPart",
- "LMToolSpec",
- "LMUsage",
- "LMVideoPart",
- "System",
- "ToolCall",
- "ToolResult",
- "User",
-]
-
-
-class LMBasePart(BaseModel):
- """A single content item in an LM message or output."""
-
- type: str
- metadata: dict[str, Any] = Field(default_factory=dict)
-
- model_config = ConfigDict(extra="forbid")
-
-
-class LMSourcePart(LMBasePart):
- """Content from exactly one provider-addressable source."""
-
- media_type: str
- data: str | None = None
- url: str | None = None
- file_id: str | None = None
- path: str | None = None
-
- @model_validator(mode="after")
- def validate_one_source(self) -> LMSourcePart:
- _validate_one_source(self)
- return self
-
-
-class LMTextPart(LMBasePart):
- """Text content."""
-
- type: Literal["text"] = "text"
- text: str
-
-
-class LMImagePart(LMSourcePart):
- """Image content from data, a URL, a file ID, or a local path."""
-
- type: Literal["image"] = "image"
- media_type: str = "image/png"
- detail: Literal["low", "high", "auto"] | None = None
-
-
-class LMAudioPart(LMSourcePart):
- """Audio content from data, a URL, a file ID, or a local path."""
-
- type: Literal["audio"] = "audio"
- media_type: str = "audio/wav"
-
-
-class LMVideoPart(LMSourcePart):
- """Video content from data, a URL, a file ID, or a local path."""
-
- type: Literal["video"] = "video"
- media_type: str = "video/mp4"
-
-
-class LMDocumentPart(LMBasePart):
- """Semantic source/document content, optionally citation-enabled.
-
- Documents are source material: text, PDFs, reports, contracts, or other
- provider-addressable evidence that benefits from title/context/citation
- semantics. Use `LMBinaryPart` for opaque attachments or arbitrary bytes.
- """
-
- type: Literal["document"] = "document"
- media_type: str = "application/pdf"
- data: str | None = None
- url: str | None = None
- file_id: str | None = None
- path: str | None = None
- source: dict[str, Any] | None = None
- citations: dict[str, Any] = Field(default_factory=dict)
- title: str | None = None
- context: str | None = None
-
- @model_validator(mode="after")
- def validate_source(self) -> LMDocumentPart:
- has_media_source = any(value is not None for value in (self.data, self.url, self.file_id, self.path))
- if self.source is not None and has_media_source:
- raise ValueError("LMDocumentPart accepts either source or one of data, url, file_id, or path, not both.")
- if self.source is None:
- _validate_one_source(self)
- elif not self.source:
- raise ValueError("LMDocumentPart.source must be non-empty when provided.")
- return self
-
-
-class LMBinaryPart(LMSourcePart):
- """Opaque binary content from data, a URL, a file ID, or a local path."""
-
- type: Literal["binary"] = "binary"
- media_type: str = "application/octet-stream"
- filename: str | None = None
-
-
-class LMToolCallPart(LMBasePart):
- """A model request to call a tool.
-
- Use `dspy.ToolCall(...)` as a shorter public alias when constructing
- assistant messages by hand.
-
- Args:
- id: Provider call ID, when the backend uses one.
- name: Name of the tool to call.
- args: JSON-like arguments for the tool.
- provider_data: Raw provider fields to keep with the tool call.
-
- Examples:
- ```python
- import dspy
-
- assistant = dspy.Assistant(
- dspy.ToolCall(
- id="call_1",
- name="search",
- args={"query": "DSPy"},
- )
- )
- ```
- """
-
- type: Literal["tool_call"] = "tool_call"
- id: str | None = None
- name: str
- args: dict[str, Any] = Field(default_factory=dict)
- provider_data: dict[str, Any] = Field(default_factory=dict)
-
-
-class LMToolResultPart(LMBasePart):
- """A tool execution result sent back to a model."""
-
- type: Literal["tool_result"] = "tool_result"
- call_id: str | None = None
- name: str | None = None
- content: list[LMPart] = Field(default_factory=list)
- is_error: bool = False
- provider_data: dict[str, Any] = Field(default_factory=dict)
-
- @model_validator(mode="before")
- @classmethod
- def normalize_content(cls, data: Any) -> Any:
- if isinstance(data, dict) and "content" in data:
- data = dict(data)
- content = data["content"]
- if content is None:
- data["content"] = []
- elif isinstance(content, list):
- data["content"] = [_coerce_part(item) for item in content]
- else:
- data["content"] = [_coerce_part(content)]
- return data
-
-
-class LMThinkingPart(LMBasePart):
- """Reasoning or thinking content returned by a model."""
-
- type: Literal["thinking"] = "thinking"
- text: str
- redacted: bool = False
-
-
-class LMCitationPart(LMBasePart):
- """A source citation returned by a model."""
-
- type: Literal["citation"] = "citation"
- text: str | None = None
- title: str | None = None
- url: str | None = None
- metadata: dict[str, Any] = Field(default_factory=dict)
-
- @model_validator(mode="after")
- def validate_has_content(self) -> LMCitationPart:
- if self.text is None and self.title is None and self.url is None:
- raise ValueError("LMCitationPart requires at least one of text, title, or url.")
- return self
-
-
-class LMRefusalPart(LMBasePart):
- """A model refusal."""
-
- type: Literal["refusal"] = "refusal"
- text: str
-
-
-LMPart = Annotated[
- LMTextPart
- | LMImagePart
- | LMAudioPart
- | LMVideoPart
- | LMDocumentPart
- | LMBinaryPart
- | LMToolCallPart
- | LMToolResultPart
- | LMThinkingPart
- | LMCitationPart
- | LMRefusalPart,
- Field(discriminator="type"),
-]
-
-
-class LMMessage(BaseModel):
- """A role-attributed sequence of LM parts."""
-
- role: str
- parts: list[LMPart]
- name: str | None = None
- metadata: dict[str, Any] = Field(default_factory=dict)
-
- model_config = ConfigDict(extra="forbid")
-
- @model_validator(mode="before")
- @classmethod
- def normalize_parts(cls, data: Any) -> Any:
- if isinstance(data, cls):
- return data
- if isinstance(data, dict):
- data = dict(data)
- if data.get("role") == "tool" and "parts" not in data:
- content = data.pop("content", None)
- call_id = data.pop("tool_call_id", None)
- name = data.pop("name", None)
- data["parts"] = [
- LMToolResultPart(
- call_id=call_id,
- name=name,
- content=_parts_from_openai_content(content),
- )
- ]
- elif "parts" not in data:
- parts = _parts_from_openai_content(data.pop("content", None)) if "content" in data else []
- if "tool_calls" in data:
- parts.extend(_tool_calls_from_openai(data.pop("tool_calls") or []))
- data["parts"] = parts
- else:
- data["parts"] = [_coerce_part(part) for part in data["parts"]]
- if "tool_calls" in data:
- data["parts"].extend(_tool_calls_from_openai(data.pop("tool_calls") or []))
- return data
-
- @property
- def text(self) -> str | None:
- texts = [part.text for part in self.parts if isinstance(part, LMTextPart)]
- return "".join(texts) if texts else None
-
-
-class LMToolSpec(BaseModel):
- """Provider-independent schema for a tool available to an LM."""
-
- type: Literal["function"] = "function"
- name: str
- description: str | None = None
- parameters: dict[str, Any] = Field(default_factory=dict)
- strict: bool | None = None
- metadata: dict[str, Any] = Field(default_factory=dict)
- provider_data: dict[str, Any] = Field(default_factory=dict)
-
- model_config = ConfigDict(extra="forbid")
-
-
-class LMReasoningConfig(BaseModel):
- """Reasoning controls for models with native reasoning support."""
-
- effort: str | None = None
- max_tokens: int | None = None
- summary: str | None = None
-
- model_config = ConfigDict(extra="forbid")
-
- @classmethod
- def from_value(cls, value: Any = None, **overrides: Any) -> LMReasoningConfig:
- data = _config_data(value, str_field="effort")
- if "reasoning_effort" in data and "effort" not in data:
- data["effort"] = data.pop("reasoning_effort")
- data.update({key: value for key, value in overrides.items() if value is not _MISSING})
- return cls(**data)
-
-
-class LMToolChoice(BaseModel):
- """Tool-choice controls for native tool-capable models."""
-
- mode: Literal["auto", "required", "none"] = "auto"
- # Tool names that are allowed for a model request.
- allowed: list[str] | None = None
- parallel: bool | None = None
-
- model_config = ConfigDict(extra="forbid")
-
- @classmethod
- def from_value(cls, value: Any = None, **overrides: Any) -> LMToolChoice:
- if isinstance(value, Mapping) and value.get("type") == "function" and isinstance(value.get("function"), Mapping):
- # Accept OpenAI/LiteLLM-shaped tool choices for the current BaseLM
- # compatibility path. Internally, DSPy represents this as requiring
- # one allowed tool.
- data = {"mode": "required", "allowed": [value["function"].get("name", "")]}
- else:
- data = _config_data(value, str_field="mode")
- if "parallel_tool_calls" in data and "parallel" not in data:
- data["parallel"] = data.pop("parallel_tool_calls")
- data.update({key: value for key, value in overrides.items() if value is not _MISSING})
- return cls(**data)
-
-
-class LMCacheConfig(BaseModel):
- """DSPy memoization cache controls for a normalized LM request.
-
- This cache skips the provider call entirely when DSPy finds an exact
- request match. Use `LMPromptCacheConfig` for provider-side prompt/token
- caching that still sends the request to the provider.
- """
-
- enabled: bool | None = None
- rollout_id: int | str | None = None
-
- model_config = ConfigDict(extra="forbid")
-
- @classmethod
- def from_value(cls, value: Any = None, **overrides: Any) -> LMCacheConfig:
- data = _config_data(value, bool_field="enabled")
- data.update({key: value for key, value in overrides.items() if value is not _MISSING})
- return cls(**data)
-
-
-class LMPromptCacheConfig(BaseModel):
- """Provider-side prompt/token cache controls.
-
- Prompt caching is not DSPy memoization. The provider call still happens,
- but the backend may reuse cached prompt prefixes or KV state for lower
- latency or lower input-token cost.
- """
-
- enabled: bool | None = None
- key: str | None = None
-
- model_config = ConfigDict(extra="forbid")
-
- @classmethod
- def from_value(cls, value: Any = None, **overrides: Any) -> LMPromptCacheConfig:
- data = _config_data(value, bool_field="enabled")
- if "prompt_cache_key" in data and "key" not in data:
- data["key"] = data.pop("prompt_cache_key")
- data.update({key: value for key, value in overrides.items() if value is not _MISSING})
- return cls(**data)
-
-
-_MISSING = object()
-
-
-def _config_data(value: Any, *, str_field: str | None = None, bool_field: str | None = None) -> dict[str, Any]:
- if value is None:
- return {}
- if isinstance(value, BaseModel):
- return value.model_dump(exclude_none=True)
- if isinstance(value, Mapping):
- return dict(value)
- if str_field is not None and isinstance(value, str):
- return {str_field: value}
- if bool_field is not None and isinstance(value, bool):
- return {bool_field: value}
- raise TypeError(f"Cannot convert {type(value)!r} to a config object.")
-
-
-# Keep existing `dspy.LM(...)` / `lm(...)` keyword aliases in the call compatibility path.
-_KNOWN_CONFIG_KEYS = {
- "temperature",
- "max_tokens",
- "top_p",
- "stop",
- "n",
- "logprobs",
- "response_format",
- "reasoning",
- "reasoning_effort",
- "tool_choice",
- "parallel_tool_calls",
- "cache",
- "rollout_id",
- "prompt_cache",
- "prompt_cache_key",
- "extensions",
-}
-
-
-def _split_config_kwargs(kwargs: Mapping[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]:
- data: dict[str, Any] = {}
- extensions = dict(kwargs.get("extensions", {}) or {})
-
- for key, value in kwargs.items():
- if key == "extensions":
- continue
- if key in _KNOWN_CONFIG_KEYS:
- data[key] = value
- else:
- extensions[key] = value
- return data, extensions
-
-
-def _normalize_reasoning_config(data: dict[str, Any]) -> None:
- if "reasoning_effort" in data:
- effort = data.pop("reasoning_effort")
- if data.get("reasoning") is not None or effort is not None:
- data["reasoning"] = LMReasoningConfig.from_value(data.get("reasoning"), effort=effort)
- elif "reasoning" in data and data["reasoning"] is not None:
- data["reasoning"] = LMReasoningConfig.from_value(data["reasoning"])
-
-
-def _normalize_tool_choice_config(data: dict[str, Any]) -> None:
- parallel = data.pop("parallel_tool_calls", _MISSING)
- if data.get("tool_choice") is not None or parallel not in (_MISSING, None):
- data["tool_choice"] = LMToolChoice.from_value(data.get("tool_choice"), parallel=parallel)
-
-
-def _normalize_cache_config(data: dict[str, Any]) -> None:
- rollout_id = data.pop("rollout_id", _MISSING)
- if data.get("cache") is not None or rollout_id not in (_MISSING, None):
- data["cache"] = LMCacheConfig.from_value(data.get("cache"), rollout_id=rollout_id)
-
-
-def _normalize_prompt_cache_config(data: dict[str, Any]) -> None:
- prompt_cache_key = data.pop("prompt_cache_key", _MISSING)
- if data.get("prompt_cache") is not None or prompt_cache_key not in (_MISSING, None):
- data["prompt_cache"] = LMPromptCacheConfig.from_value(data.get("prompt_cache"), key=prompt_cache_key)
-
-
-class LMConfig(BaseModel):
- """Common generation controls for an LM request."""
-
- temperature: float | None = None
- max_tokens: int | None = None
- top_p: float | None = None
- stop: list[str] | None = None
- n: int | None = None
- logprobs: bool | int | None = None
- response_format: Any | None = None
- reasoning: LMReasoningConfig | None = None
- tool_choice: LMToolChoice | None = None
- cache: LMCacheConfig | None = None
- prompt_cache: LMPromptCacheConfig | None = None
- extensions: dict[str, Any] = Field(default_factory=dict)
-
- model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid")
-
- @classmethod
- def from_kwargs(cls, **kwargs: Any) -> LMConfig:
- data, extensions = _split_config_kwargs(kwargs)
- _normalize_reasoning_config(data)
- _normalize_tool_choice_config(data)
- _normalize_cache_config(data)
- _normalize_prompt_cache_config(data)
- data["extensions"] = extensions
- return cls(**data)
-
-
-def _merge_lm_config(left: LMConfig | None, right: LMConfig | None) -> LMConfig | None:
- if left is None:
- return right
- if right is None:
- return left
-
- data = left.model_dump()
- right_data = right.model_dump(exclude_none=True)
- for key in ("reasoning", "tool_choice", "cache", "prompt_cache"):
- if key in right_data and isinstance(data.get(key), dict) and isinstance(right_data[key], dict):
- right_data[key] = {**data[key], **right_data[key]}
- extensions = {**left.extensions, **right.extensions}
- data.update(right_data)
- data["extensions"] = extensions
- return LMConfig(**data)
-
-
-def _merge_config_overrides(config: LMConfig, kwargs: dict[str, Any]) -> LMConfig:
- data = config.model_dump()
- extensions = dict(config.extensions)
-
- direct_keys = {
- "temperature",
- "max_tokens",
- "top_p",
- "stop",
- "n",
- "logprobs",
- "response_format",
- }
- for key in direct_keys & kwargs.keys():
- data[key] = kwargs[key]
-
- if "reasoning" in kwargs or "reasoning_effort" in kwargs:
- data["reasoning"] = LMReasoningConfig.from_value(
- kwargs.get("reasoning", data.get("reasoning")),
- effort=kwargs.get("reasoning_effort", _MISSING),
- )
-
- if "tool_choice" in kwargs or "parallel_tool_calls" in kwargs:
- data["tool_choice"] = LMToolChoice.from_value(
- kwargs.get("tool_choice", data.get("tool_choice")),
- parallel=kwargs.get("parallel_tool_calls", _MISSING),
- )
-
- if "cache" in kwargs or "rollout_id" in kwargs:
- data["cache"] = LMCacheConfig.from_value(
- kwargs.get("cache", data.get("cache")),
- rollout_id=kwargs.get("rollout_id", _MISSING),
- )
-
- if "prompt_cache" in kwargs or "prompt_cache_key" in kwargs:
- data["prompt_cache"] = LMPromptCacheConfig.from_value(
- kwargs.get("prompt_cache", data.get("prompt_cache")),
- key=kwargs.get("prompt_cache_key", _MISSING),
- )
-
- if "extensions" in kwargs:
- extra = kwargs["extensions"]
- if extra is None:
- extensions = {}
- elif isinstance(extra, Mapping):
- extensions.update(extra)
- else:
- raise TypeError("`extensions` override must be a mapping or None.")
-
- handled = _KNOWN_CONFIG_KEYS
- for key, value in kwargs.items():
- if key not in handled:
- extensions[key] = value
- data["extensions"] = extensions
- return LMConfig(**data)
-
-
-@dataclass
-class LMRequestPatch:
- """A partial normalized LM request contributed while rendering a DSPy call.
-
- `LMRequest` is the complete object a `LanguageModel` receives. A patch is
- the smaller, composable unit that DSPy type strategies can contribute while
- an adapter is still building that request: extra messages, extra parts,
- native tools, native config, or signature fields that should be hidden from
- the outer adapter's ordinary text/JSON/XML rendering.
- """
-
- messages: list[LMMessage] = dataclass_field(default_factory=list)
- system_parts: list[LMPart] = dataclass_field(default_factory=list)
- user_parts: list[LMPart] = dataclass_field(default_factory=list)
- assistant_parts: list[LMPart] = dataclass_field(default_factory=list)
- tools: list[LMToolSpec] = dataclass_field(default_factory=list)
- config: LMConfig | None = None
- delete_input_fields: tuple[str, ...] = ()
- delete_output_fields: tuple[str, ...] = ()
- metadata: dict[str, Any] = dataclass_field(default_factory=dict)
-
- def merge(self, other: LMRequestPatch) -> LMRequestPatch:
- """Return a new patch containing this patch followed by `other`."""
- return LMRequestPatch(
- messages=[*self.messages, *other.messages],
- system_parts=[*self.system_parts, *other.system_parts],
- user_parts=[*self.user_parts, *other.user_parts],
- assistant_parts=[*self.assistant_parts, *other.assistant_parts],
- tools=[*self.tools, *other.tools],
- config=_merge_lm_config(self.config, other.config),
- delete_input_fields=(*self.delete_input_fields, *other.delete_input_fields),
- delete_output_fields=(*self.delete_output_fields, *other.delete_output_fields),
- metadata={**self.metadata, **other.metadata},
- )
-
- def as_lm_kwargs(self) -> dict[str, Any]:
- """Return the legacy kwargs implied by this patch.
-
- This keeps the first implementation usable with today's adapter call
- path, which still passes `lm_kwargs` rather than an `LMRequestPatch` all
- the way down. Message and part patches are intentionally not flattened
- here; they require the next adapter-call refactor.
- """
- kwargs = self.config.model_dump(exclude_none=True) if self.config is not None else {}
- extensions = kwargs.pop("extensions", {}) or {}
- kwargs = {**extensions, **kwargs}
- if self.tools:
- kwargs["tools"] = list(self.tools)
- return kwargs
-
-
-class LMRequest(BaseModel):
- """A normalized request passed to a `LanguageModel`."""
-
- model: str
- messages: list[LMMessage]
- tools: list[LMToolSpec] = Field(default_factory=list)
- config: LMConfig = Field(default_factory=LMConfig)
- metadata: dict[str, Any] = Field(default_factory=dict)
-
- model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid")
-
- @classmethod
- def from_call(
- cls,
- *,
- model: str,
- items: tuple[Any, ...] = (),
- prompt: str | None = None,
- messages: list[dict[str, Any] | LMMessage] | None = None,
- tools: list[Any] | None = None,
- **kwargs: Any,
- ) -> LMRequest:
- if messages is not None and (items or prompt is not None):
- raise ValueError("Pass messages or direct-call inputs, not both.")
-
- collected_tools: list[Any] = list(tools or [])
- if messages is not None:
- normalized_messages = [_coerce_message(message) for message in messages]
- else:
- normalized_messages, positional_tools = _messages_from_items(items, prompt=prompt)
- collected_tools.extend(positional_tools)
-
- config = LMConfig.from_kwargs(**kwargs)
- return cls(
- model=model,
- messages=normalized_messages,
- tools=[_coerce_tool_spec(tool) for tool in collected_tools],
- config=config,
- )
-
- @classmethod
- def from_prompt_or_messages(
- cls,
- *,
- model: str,
- prompt: str | None = None,
- messages: list[dict[str, Any] | LMMessage] | None = None,
- **kwargs: Any,
- ) -> LMRequest:
- return cls.from_call(model=model, prompt=prompt, messages=messages, **kwargs)
-
- def with_config_overrides(self, **kwargs: Any) -> LMRequest:
- """Return a copy with explicit request config overrides applied.
-
- Only fields implied by the supplied keyword arguments are changed. This
- preserves existing grouped config such as `cache`, `prompt_cache`,
- `tool_choice`, `reasoning`, `stop`, and provider-specific
- `extensions` when an unrelated setting is overridden.
- """
- if not kwargs:
- return self
- merged = _merge_config_overrides(self.config, kwargs)
- return self.model_copy(update={"config": merged}, deep=True)
-
-
-class LMUsage(BaseModel):
- """Token and timing usage for one LM request.
-
- Both DSPy token names and provider token names are populated because both
- are existing user-visible history/usage interfaces.
- """
-
- input_tokens: int | None = None
- output_tokens: int | None = None
- total_tokens: int | None = None
- prompt_tokens: int | None = None
- completion_tokens: int | None = None
- reasoning_tokens: int | None = None
- cache_read_tokens: int | None = None
- cache_write_tokens: int | None = None
- input_audio_tokens: int | None = None
- output_audio_tokens: int | None = None
- details: dict[str, Any] = Field(default_factory=dict)
-
- model_config = ConfigDict(extra="allow")
-
- @model_validator(mode="after")
- def fill_aliases(self) -> LMUsage:
- if self.input_tokens is None and self.prompt_tokens is not None:
- self.input_tokens = self.prompt_tokens
- if self.output_tokens is None and self.completion_tokens is not None:
- self.output_tokens = self.completion_tokens
- if self.prompt_tokens is None and self.input_tokens is not None:
- self.prompt_tokens = self.input_tokens
- if self.completion_tokens is None and self.output_tokens is not None:
- self.completion_tokens = self.output_tokens
- if self.total_tokens is None and self.input_tokens is not None and self.output_tokens is not None:
- self.total_tokens = self.input_tokens + self.output_tokens
- return self
-
-
-class LMOutput(BaseModel):
- """One generated candidate in an LM response."""
-
- parts: list[LMPart] = Field(default_factory=list)
- finish_reason: str | None = None
- truncated: bool = False
- logprobs: Any | None = None
- provider_output: Any | None = Field(default=None, exclude=True)
- provider_data: dict[str, Any] = Field(default_factory=dict)
- metadata: dict[str, Any] = Field(default_factory=dict)
-
- model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid")
-
- @model_validator(mode="before")
- @classmethod
- def normalize_parts(cls, data: Any) -> Any:
- if isinstance(data, dict) and "parts" in data:
- data = dict(data)
- data["parts"] = [_coerce_part(part) for part in data["parts"]]
- return data
-
- @property
- def text(self) -> str | None:
- texts = [part.text for part in self.parts if isinstance(part, LMTextPart)]
- return "".join(texts) if texts else None
-
- @property
- def reasoning_content(self) -> str | None:
- texts = [part.text for part in self.parts if isinstance(part, LMThinkingPart)]
- return "".join(texts) if texts else None
-
- @property
- def tool_calls(self) -> list[LMToolCallPart]:
- return [part for part in self.parts if isinstance(part, LMToolCallPart)]
-
- @property
- def citations(self) -> list[LMCitationPart]:
- return [part for part in self.parts if isinstance(part, LMCitationPart)]
-
- @property
- def images(self) -> list[LMImagePart]:
- return [part for part in self.parts if isinstance(part, LMImagePart)]
-
- @property
- def audio(self) -> list[LMAudioPart]:
- return [part for part in self.parts if isinstance(part, LMAudioPart)]
-
- @property
- def videos(self) -> list[LMVideoPart]:
- return [part for part in self.parts if isinstance(part, LMVideoPart)]
-
- @property
- def documents(self) -> list[LMDocumentPart]:
- return [part for part in self.parts if isinstance(part, LMDocumentPart)]
-
- @property
- def binaries(self) -> list[LMBinaryPart]:
- return [part for part in self.parts if isinstance(part, LMBinaryPart)]
-
- @property
- def refusal(self) -> str | None:
- refusals = [part.text for part in self.parts if isinstance(part, LMRefusalPart)]
- return "".join(refusals) if refusals else None
-
- def to_value(self) -> Any:
- values = [_part_to_value(part) for part in self.parts]
- values = [value for value in values if value is not None]
- if len(values) == 1 and isinstance(values[0], str) and self.logprobs is None:
- return values[0]
- return values
-
- def to_output_dict(self) -> dict[str, Any]:
- data: dict[str, Any] = {"text": self.text}
- if self.reasoning_content is not None:
- data["reasoning_content"] = self.reasoning_content
- if self.tool_calls:
- data["tool_calls"] = [_tool_call_to_provider_dict(call) for call in self.tool_calls]
- if self.citations:
- data["citations"] = [citation.model_dump(exclude_none=True) for citation in self.citations]
- if self.logprobs is not None:
- data["logprobs"] = self.logprobs
- return data
-
-
-class LMResponse(BaseModel):
- """The normalized result of one LM request."""
-
- model: str | None = None
- outputs: list[LMOutput] = Field(min_length=1)
- usage: LMUsage | dict[str, Any] | None = None
- cost: float | None = None
- cache_hit: bool = False
- response_id: str | None = None
- provider_response: Any | None = Field(default=None, exclude=True)
- provider_data: dict[str, Any] = Field(default_factory=dict)
- metadata: dict[str, Any] = Field(default_factory=dict)
-
- model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid")
-
- @model_validator(mode="before")
- @classmethod
- def normalize_usage(cls, data: Any) -> Any:
- if isinstance(data, dict) and isinstance(data.get("usage"), dict):
- data = dict(data)
- data["usage"] = LMUsage(**data["usage"])
- return data
-
- @classmethod
- def from_text(
- cls,
- text: str,
- *,
- model: str | None = None,
- usage: LMUsage | dict[str, Any] | None = None,
- cost: float | None = None,
- cache_hit: bool = False,
- **kwargs: Any,
- ) -> LMResponse:
- return cls(
- model=model,
- outputs=[LMOutput(parts=[LMTextPart(text=text)])],
- usage=usage,
- cost=cost,
- cache_hit=cache_hit,
- **kwargs,
- )
-
- def __iter__(self):
- return iter(self.to_values())
-
- def __getitem__(self, index: int) -> Any:
- return self.to_values()[index]
-
- def __len__(self) -> int:
- return len(self.outputs)
-
- @property
- def output(self) -> LMOutput:
- return self.outputs[0]
-
- @property
- def parts(self) -> list[LMPart]:
- return self.output.parts
-
- @property
- def text(self) -> str | None:
- return self.output.text
-
- @property
- def reasoning_content(self) -> str | None:
- return self.output.reasoning_content
-
- @property
- def tool_calls(self) -> list[LMToolCallPart]:
- return self.output.tool_calls
-
- @property
- def citations(self) -> list[LMCitationPart]:
- return self.output.citations
-
- @property
- def images(self) -> list[LMImagePart]:
- return self.output.images
-
- @property
- def audio(self) -> list[LMAudioPart]:
- return self.output.audio
-
- @property
- def videos(self) -> list[LMVideoPart]:
- return self.output.videos
-
- @property
- def documents(self) -> list[LMDocumentPart]:
- return self.output.documents
-
- @property
- def binaries(self) -> list[LMBinaryPart]:
- return self.output.binaries
-
- def to_values(self) -> list[Any]:
- return [output.to_value() for output in self.outputs]
-
- def to_outputs(self) -> list[Any]:
- outputs: list[Any] = []
- for output in self.outputs:
- if _requires_output_dict(output):
- outputs.append(output.to_output_dict())
- else:
- outputs.append(output.to_value())
- return outputs
-
- def to_legacy_outputs(self) -> list[Any]:
- return self.to_outputs()
-
- def usage_as_dict(self) -> dict[str, Any]:
- if self.usage is None:
- return {}
- if isinstance(self.usage, LMUsage):
- return self.usage.model_dump(exclude_none=True)
- return dict(self.usage)
-
-
-class LMHistoryEntry(BaseModel, Mapping[str, Any]):
- """A typed history record that can be read like a dictionary.
-
- Store the canonical request and response, then derive legacy convenience
- fields such as `outputs`, `usage`, `messages`, and `kwargs` on demand.
- Because this class implements `Mapping`, existing history code can keep
- using `entry["messages"]`, `entry.get("prompt")`, `entry.items()`, and
- `dict(entry)`.
- """
-
- request: LMRequest
- response: LMResponse
- timestamp: str
- uuid: str
- model_type: str | None = None
-
- model_config = ConfigDict(arbitrary_types_allowed=True, extra="allow")
-
- @model_validator(mode="before")
- @classmethod
- def drop_derived_fields(cls, data: Any) -> Any:
- if isinstance(data, dict):
- data = dict(data)
- for key in _HISTORY_DERIVED_KEYS:
- data.pop(key, None)
- return data
-
- @property
- def outputs(self) -> list[Any]:
- return self.response.to_outputs()
-
- @property
- def usage(self) -> dict[str, Any]:
- return self.response.usage_as_dict()
-
- @property
- def cost(self) -> float | None:
- return self.response.cost
-
- @property
- def model(self) -> str:
- return self.request.model
-
- @property
- def prompt(self) -> str | None:
- return _history_request_prompt(self.request)
-
- @property
- def messages(self) -> list[dict[str, Any]] | None:
- return _history_request_messages_as_openai(self.request)
-
- @property
- def kwargs(self) -> dict[str, Any]:
- return _history_request_kwargs(self.request)
-
- @property
- def response_model(self) -> str | None:
- return self.response.model
-
- def __getitem__(self, key: str) -> Any:
- return self._mapping()[key]
-
- def __iter__(self) -> Iterator[str]:
- return iter(self._mapping())
-
- def __len__(self) -> int:
- return len(self._mapping())
-
- def __repr__(self) -> str:
- formatted = pformat(self._essential_mapping(), width=100, sort_dicts=False)
- return f"LMHistoryEntry(\n{formatted}\n)"
-
- def __str__(self) -> str:
- return repr(self)
-
- def to_dict(self, *, mode: str = "python", exclude_none: bool = False, **kwargs: Any) -> dict[str, Any]:
- """Return this history entry as a plain dictionary."""
- if kwargs:
- return self.model_dump(mode=mode, exclude_none=exclude_none, **kwargs)
- data = self._mapping()
- if mode != "python":
- data = json.loads(json.dumps(data, default=_json_default))
- if exclude_none:
- data = {key: value for key, value in data.items() if value is not None}
- return data
-
- def _essential_mapping(self) -> dict[str, Any]:
- data = self.model_dump(mode="python", exclude_none=True)
- data.update(self.model_extra or {})
- return data
-
- def _mapping(self) -> dict[str, Any]:
- data = self._essential_mapping()
- data.update({key: getattr(self, key) for key in _HISTORY_DERIVED_KEYS})
- return {key: value for key, value in data.items() if value is not None}
-
-
-_HISTORY_DERIVED_KEYS = (
- "outputs",
- "usage",
- "cost",
- "model",
- "prompt",
- "messages",
- "kwargs",
- "response_model",
+"""Migration notice for the retired DSPy 3.3 experimental LM types."""
+
+raise ImportError(
+ "DSPy's 3.3 experimental LM types have been removed. "
+ "Import Request, Response, Message, Config and content parts from dspy.lm15 instead. "
+ "These are different objects, not drop-in aliases: system instructions use Request.system, "
+ "tool arguments use ToolCallPart.input, and each Response contains one Message. "
+ "Ordinary lm(prompt=..., messages=...) calls remain supported. "
+ "See https://dspy.ai/community/normalized-lm-api-migration/."
)
-
-
-def _json_default(value: Any) -> Any:
- if hasattr(value, "model_dump"):
- return value.model_dump(mode="json", exclude_none=True)
- return str(value)
-
-
-class LMDelta(BaseModel):
- """Base class for streamed content deltas."""
-
- type: str
-
-
-class LMTextDelta(LMDelta):
- type: Literal["text_delta"] = "text_delta"
- text: str
-
-
-class LMThinkingDelta(LMDelta):
- type: Literal["thinking_delta"] = "thinking_delta"
- text: str
-
-
-class LMToolCallDelta(LMDelta):
- type: Literal["tool_call_delta"] = "tool_call_delta"
- id: str | None = None
- name: str | None = None
- args_delta: str | None = None
-
-
-class LMCitationDelta(LMDelta):
- type: Literal["citation_delta"] = "citation_delta"
- citation: LMCitationPart
-
-
-class LMImageDelta(LMDelta):
- type: Literal["image_delta"] = "image_delta"
- image: LMImagePart
-
-
-class LMAudioDelta(LMDelta):
- type: Literal["audio_delta"] = "audio_delta"
- audio: LMAudioPart
-
-
-LMAnyDelta = Annotated[
- LMTextDelta | LMThinkingDelta | LMToolCallDelta | LMCitationDelta | LMImageDelta | LMAudioDelta,
- Field(discriminator="type"),
-]
-
-
-class LMStreamEvent(BaseModel):
- """Base class for normalized LM stream events."""
-
- type: str
-
-
-class LMStreamStartEvent(LMStreamEvent):
- type: Literal["start"] = "start"
- model: str | None = None
-
-
-class LMStreamDeltaEvent(LMStreamEvent):
- type: Literal["delta"] = "delta"
- output_index: int = Field(default=0, ge=0)
- part_index: int = Field(ge=0)
- delta: LMAnyDelta
-
-
-class LMStreamOutputEndEvent(LMStreamEvent):
- type: Literal["output_end"] = "output_end"
- output_index: int = Field(default=0, ge=0)
- finish_reason: str | None = None
- truncated: bool = False
-
-
-class LMStreamEndEvent(LMStreamEvent):
- type: Literal["end"] = "end"
- usage: LMUsage | dict[str, Any] | None = None
- cost: float | None = None
- response: LMResponse | None = None
-
-
-class LMStreamErrorEvent(LMStreamEvent):
- type: Literal["error"] = "error"
- error: Exception
-
- model_config = ConfigDict(arbitrary_types_allowed=True)
-
-
-class LMOutputBuilder:
- """Assemble streamed LM events into a final `LMResponse`."""
-
- def __init__(self):
- self.model: str | None = None
- self._parts: dict[int, list[LMPart | None]] = {}
- self._finish_reasons: dict[int, str | None] = {}
- self._truncated: dict[int, bool] = {}
-
- def apply(self, event: LMStreamEvent) -> LMResponse | None:
- if isinstance(event, LMStreamStartEvent):
- self.model = event.model
- return None
- if isinstance(event, LMStreamDeltaEvent):
- self._apply_delta(event)
- return None
- if isinstance(event, LMStreamOutputEndEvent):
- self._finish_reasons[event.output_index] = event.finish_reason
- self._truncated[event.output_index] = event.truncated
- return None
- if isinstance(event, LMStreamEndEvent):
- if event.response is not None:
- return event.response
- return self.to_response(usage=event.usage, cost=event.cost)
- if isinstance(event, LMStreamErrorEvent):
- raise event.error
- return None
-
- def to_response(self, *, usage: LMUsage | dict[str, Any] | None = None, cost: float | None = None) -> LMResponse:
- output_indices = set(self._parts) | set(self._finish_reasons) | set(self._truncated)
- if not output_indices:
- output_indices = {0}
- max_index = max(output_indices)
- expected_indices = set(range(max_index + 1))
- if output_indices != expected_indices:
- missing = sorted(expected_indices - output_indices)
- raise ValueError(f"Stream output indices must be contiguous from 0; missing indices: {missing}.")
-
- outputs = []
- for output_index in range(max_index + 1):
- part_buffer = self._parts.get(output_index, [])
- missing_part_indices = [index for index, part in enumerate(part_buffer) if part is None]
- if missing_part_indices:
- raise ValueError(
- f"Stream part indices for output {output_index} must be contiguous; "
- f"missing indices: {missing_part_indices}."
- )
- parts = [_finalize_stream_part(part) for part in part_buffer]
- outputs.append(
- LMOutput(
- parts=parts,
- finish_reason=self._finish_reasons.get(output_index),
- truncated=self._truncated.get(output_index, False),
- )
- )
- return LMResponse(model=self.model, outputs=outputs, usage=usage, cost=cost)
-
- def _apply_delta(self, event: LMStreamDeltaEvent) -> None:
- parts = self._parts.setdefault(event.output_index, [])
- while len(parts) <= event.part_index:
- parts.append(None)
-
- current = parts[event.part_index]
- delta = event.delta
- if isinstance(delta, LMThinkingDelta):
- if current is not None and not isinstance(current, LMThinkingPart):
- raise ValueError("Cannot apply thinking delta to a non-thinking stream part.")
- text = (current.text if isinstance(current, LMThinkingPart) else "") + delta.text
- parts[event.part_index] = LMThinkingPart(text=text)
- elif isinstance(delta, LMTextDelta):
- if current is not None and not isinstance(current, LMTextPart):
- raise ValueError("Cannot apply text delta to a non-text stream part.")
- text = (current.text if isinstance(current, LMTextPart) else "") + delta.text
- parts[event.part_index] = LMTextPart(text=text)
- elif isinstance(delta, LMToolCallDelta):
- if current is not None and not isinstance(current, LMToolCallPart):
- raise ValueError("Cannot apply tool-call delta to a non-tool-call stream part.")
- buffer = ""
- if isinstance(current, LMToolCallPart):
- buffer = current.provider_data.get("args_buffer", "")
- buffer += delta.args_delta or ""
- args = _parse_json_object(buffer)
- parts[event.part_index] = LMToolCallPart(
- id=delta.id if delta.id is not None else getattr(current, "id", None),
- name=delta.name if delta.name is not None else getattr(current, "name", ""),
- args=args,
- provider_data={"args_buffer": buffer},
- )
- elif isinstance(delta, LMCitationDelta):
- if current is not None and not isinstance(current, LMCitationPart):
- raise ValueError("Cannot apply citation delta to a different stream part type.")
- parts[event.part_index] = delta.citation
- elif isinstance(delta, LMImageDelta):
- if current is not None and not isinstance(current, LMImagePart):
- raise ValueError("Cannot apply image delta to a different stream part type.")
- parts[event.part_index] = delta.image
- elif isinstance(delta, LMAudioDelta):
- if current is not None and not isinstance(current, LMAudioPart):
- raise ValueError("Cannot apply audio delta to a different stream part type.")
- parts[event.part_index] = delta.audio
-
-
-class LMStream:
- """Synchronous LM stream with a final `LMResponse` result."""
-
- def __init__(
- self,
- *,
- request: LMRequest,
- events: Iterator[LMStreamEvent],
- finalize: Callable[[LMRequest, LMResponse], LMResponse],
- ):
- self.request = request
- self._events = events
- self._finalize = finalize
- self._builder = LMOutputBuilder()
- self._result: LMResponse | None = None
-
- def __iter__(self) -> Iterator[LMStreamEvent]:
- for event in self._events:
- response = self._builder.apply(event)
- if response is not None:
- self._result = self._finalize(self.request, response)
- yield event
-
- def result(self) -> LMResponse:
- if self._result is None:
- raise RuntimeError("Stream has not completed yet.")
- return self._result
-
-
-class AsyncLMStream:
- """Asynchronous LM stream with a final `LMResponse` result."""
-
- def __init__(
- self,
- *,
- request: LMRequest,
- events: AsyncIterator[LMStreamEvent],
- finalize: Callable[[LMRequest, LMResponse], LMResponse],
- ):
- self.request = request
- self._events = events
- self._finalize = finalize
- self._builder = LMOutputBuilder()
- self._result: LMResponse | None = None
-
- async def __aiter__(self) -> AsyncIterator[LMStreamEvent]:
- async for event in self._events:
- response = self._builder.apply(event)
- if response is not None:
- self._result = self._finalize(self.request, response)
- yield event
-
- def result(self) -> LMResponse:
- if self._result is None:
- raise RuntimeError("Stream has not completed yet.")
- return self._result
-
-
-def System(*parts: Any, name: str | None = None, metadata: dict[str, Any] | None = None) -> LMMessage: # noqa: N802
- """Create a system message for a direct LM call.
-
- A system message gives model-level instructions, such as tone, scope, or
- formatting rules. Pass text or normalized `LMPart` objects; DSPy stores
- them as one `LMMessage` with role `"system"`.
-
- Args:
- *parts: Text or normalized LM parts to include in the message.
- name: Optional sender name for providers that support named messages.
- metadata: Extra information to keep with the message.
-
- Returns:
- An `LMMessage` that can be passed to `dspy.LM` or `dspy.LMRequest`.
-
- Examples:
- System instruction with a user turn:
-
- ```python
- import dspy
-
- request = dspy.LMRequest.from_call(
- model="openai/gpt-4o-mini",
- items=(
- dspy.System("You are concise."),
- dspy.User("What is DSPy?"),
- ),
- )
- ```
-
- See Also:
- [`dspy.User`][dspy.User]
- [`dspy.Assistant`][dspy.Assistant]
- [`dspy.LMRequest`][dspy.LMRequest]
- """
- return LMMessage(role="system", parts=[_coerce_part(part) for part in parts], name=name, metadata=metadata or {})
-
-
-def Developer(*parts: Any, name: str | None = None, metadata: dict[str, Any] | None = None) -> LMMessage: # noqa: N802
- """Create a developer message for a direct LM call.
-
- A developer message carries instructions that sit between system guidance
- and user content. Use it when a provider supports a `"developer"` role and
- you want to keep implementation guidance separate from the user's request.
-
- Args:
- *parts: Text or normalized LM parts to include in the message.
- name: Optional sender name for providers that support named messages.
- metadata: Extra information to keep with the message.
-
- Returns:
- An `LMMessage` with role `"developer"`.
-
- Examples:
- Add house-style instructions:
-
- ```python
- import dspy
-
- request = dspy.LMRequest.from_call(
- model="openai/gpt-4o-mini",
- items=(
- dspy.System("You are a technical editor."),
- dspy.Developer("Prefer short examples."),
- dspy.User("Explain callbacks."),
- ),
- )
- ```
-
- See Also:
- [`dspy.System`][dspy.System]
- [`dspy.User`][dspy.User]
- """
- return LMMessage(role="developer", parts=[_coerce_part(part) for part in parts], name=name, metadata=metadata or {})
-
-
-def User(*parts: Any, name: str | None = None, metadata: dict[str, Any] | None = None) -> LMMessage: # noqa: N802
- """Create a user message for a direct LM call.
-
- A user message contains the request or data you want the model to answer.
- Pass plain text for simple prompts, or mix text with normalized image, audio,
- document, binary, and other `LMPart` objects for multimodal calls.
-
- Args:
- *parts: Text or normalized LM parts to include in the message.
- name: Optional sender name for providers that support named messages.
- metadata: Extra information to keep with the message.
-
- Returns:
- An `LMMessage` with role `"user"`.
-
- Examples:
- Multi-turn LM call:
-
- ```python
- import dspy
-
- lm = dspy.LM("openai/gpt-4o-mini")
- with dspy.context(experimental=True):
- response = lm(
- dspy.User("What is DSPy?"),
- dspy.Assistant("DSPy is a framework for programming LM pipelines."),
- dspy.User("Say that in five words."),
- )
- ```
-
- Multi-turn call with media:
-
- ```python
- import dspy
- from dspy.core.types import LMImagePart
-
- lm = dspy.LM("openai/gpt-4o-mini")
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Answer in one sentence."),
- dspy.User(
- "Describe this image.",
- LMImagePart(url="https://example.com/dog.png"),
- ),
- )
- ```
-
- For a single user turn, pass the parts directly to `lm(...)` instead:
-
- ```python
- with dspy.context(experimental=True):
- response = lm("Describe this image.", LMImagePart(url="https://example.com/dog.png"))
- ```
-
- Explicit `LMRequest` for custom LM authors and advanced users:
-
- ```python
- import dspy
- from dspy.core.types import LMImagePart
-
- lm = dspy.LM("openai/gpt-4o-mini")
- request = dspy.LMRequest(
- model="openai/gpt-4o-mini",
- messages=[
- dspy.System("You are concise."),
- dspy.User(
- "Describe this image.",
- LMImagePart(url="https://example.com/dog.png"),
- ),
- ],
- config=dspy.LMConfig(temperature=0.2, max_tokens=200),
- )
-
- response = lm(request)
- ```
-
- See Also:
- [`dspy.System`][dspy.System]
- [`dspy.Assistant`][dspy.Assistant]
- [`dspy.ToolResult`][dspy.ToolResult]
- """
- return LMMessage(role="user", parts=[_coerce_part(part) for part in parts], name=name, metadata=metadata or {})
-
-
-def Assistant(*parts: Any, name: str | None = None, metadata: dict[str, Any] | None = None) -> LMMessage: # noqa: N802
- """Create an assistant message for a direct LM call.
-
- An assistant message represents a previous model response. Use it when you
- build a multi-turn request by hand, or when you send tool calls back to the
- model before adding tool results.
-
- Args:
- *parts: Text, tool calls, citations, media parts, or other normalized LM
- parts from an assistant turn.
- name: Optional sender name for providers that support named messages.
- metadata: Extra information to keep with the message.
-
- Returns:
- An `LMMessage` with role `"assistant"`.
-
- Examples:
- Continue a conversation:
-
- ```python
- import dspy
-
- request = dspy.LMRequest.from_call(
- model="openai/gpt-4o-mini",
- items=(
- dspy.User("What is DSPy?"),
- dspy.Assistant("DSPy is a framework for programming LM pipelines."),
- dspy.User("Say that in five words."),
- ),
- )
- ```
-
- See Also:
- [`dspy.User`][dspy.User]
- [`dspy.ToolCall`][dspy.ToolCall]
- [`dspy.ToolResult`][dspy.ToolResult]
- """
- return LMMessage(role="assistant", parts=[_coerce_part(part) for part in parts], name=name, metadata=metadata or {})
-
-
-ToolCall = LMToolCallPart
-"""Create a tool-call part for an assistant message.
-
-`ToolCall` is an alias for `LMToolCallPart`. Use it inside `dspy.Assistant(...)`
-when you want to include a model-requested tool call in a normalized
-conversation.
-"""
-
-
-def ToolResult( # noqa: N802
- *parts: Any,
- call_id: str | None = None,
- name: str | None = None,
- content: Any | None = None,
- is_error: bool = False,
-) -> LMMessage:
- """Create a tool-result message for a direct LM call.
-
- A tool-result message sends the output of a tool back to the model. Pass the
- returned text or media as `*parts` or with `content=...`, and include the
- `call_id` from the matching assistant tool call when the provider uses call
- IDs.
-
- Args:
- *parts: Text, DSPy media objects, or normalized LM parts returned by the
- tool. If you pass one `LMToolResultPart`, DSPy uses it directly.
- call_id: Identifier of the assistant tool call this result answers.
- name: Tool name associated with the result.
- content: Optional tool output passed by keyword. Use this when adapting
- OpenAI-style code that stores the result under `content`.
- is_error: Whether this result represents a failed tool execution.
-
- Returns:
- An `LMMessage` with role `"tool"` and one `LMToolResultPart`.
-
- Examples:
- Send a weather result back to the model:
-
- ```python
- import dspy
-
- messages = [
- dspy.User("What is the weather in Paris?"),
- dspy.Assistant(
- dspy.ToolCall(
- id="call_1",
- name="get_weather",
- args={"location": "Paris"},
- )
- ),
- dspy.ToolResult(
- '{"temperature": "22", "unit": "celsius"}',
- call_id="call_1",
- name="get_weather",
- ),
- dspy.User("Summarize the result."),
- ]
- ```
-
- See Also:
- [`dspy.ToolCall`][dspy.ToolCall]
- [`dspy.Assistant`][dspy.Assistant]
- [`dspy.Tool`][dspy.Tool]
- """
- if content is not None:
- if parts:
- raise TypeError("Pass tool output either as positional parts or as `content=...`, not both.")
- parts = tuple(content if isinstance(content, list) else [content])
-
- if len(parts) == 1 and isinstance(parts[0], LMToolResultPart):
- result = parts[0]
- else:
- result = LMToolResultPart(
- call_id=call_id,
- name=name,
- content=[_coerce_part(part) for part in parts],
- is_error=is_error,
- )
- return LMMessage(role="tool", parts=[result])
-
-
-def _history_request_prompt(request: LMRequest) -> str | None:
- if len(request.messages) != 1:
- return None
- message = request.messages[0]
- if message.role != "user" or len(message.parts) != 1:
- return None
- part = message.parts[0]
- return part.text if isinstance(part, LMTextPart) else None
-
-
-def _history_request_messages_as_openai(request: LMRequest) -> list[dict[str, Any]]:
- messages = []
- for message in request.messages:
- if message.role == "assistant":
- tool_calls = [part for part in message.parts if isinstance(part, LMToolCallPart)]
- content_parts = [part for part in message.parts if not isinstance(part, LMToolCallPart)]
- item: dict[str, Any] = {
- "role": "assistant",
- "content": _history_message_parts_as_openai_content(content_parts) if content_parts else None,
- }
- if tool_calls:
- item["tool_calls"] = [_history_tool_call_as_openai(call) for call in tool_calls]
- elif message.role == "tool" and len(message.parts) == 1 and isinstance(message.parts[0], LMToolResultPart):
- result = message.parts[0]
- item = {"role": "tool", "content": _history_tool_result_content(result)}
- if result.call_id is not None:
- item["tool_call_id"] = result.call_id
- if result.name is not None:
- item["name"] = result.name
- else:
- item = {
- "role": message.role,
- "content": _history_message_parts_as_openai_content(message.parts),
- }
- if message.name is not None and "name" not in item:
- item["name"] = message.name
- messages.append(item)
- return messages
-
-
-def _history_tool_call_as_openai(call: LMToolCallPart) -> dict[str, Any]:
- data: dict[str, Any] = {
- "type": "function",
- "function": {
- "name": call.name,
- "arguments": json.dumps(call.args),
- },
- }
- if call.id is not None:
- data["id"] = call.id
- return data
-
-
-def _history_tool_result_content(result: LMToolResultPart) -> str:
- chunks = []
- for part in result.content:
- if isinstance(part, LMTextPart):
- chunks.append(part.text)
- else:
- chunks.append(json.dumps(part.model_dump(mode="json", exclude_none=True), ensure_ascii=False))
- return "".join(chunks)
-
-
-def _history_message_parts_as_openai_content(parts: list[LMPart]) -> str | list[dict[str, Any]]:
- if len(parts) == 1 and isinstance(parts[0], LMTextPart):
- return parts[0].text
- return [_history_part_as_openai_content(part) for part in parts]
-
-
-def _history_part_as_openai_content(part: LMPart) -> dict[str, Any]:
- if isinstance(part, LMTextPart):
- return {"type": "text", "text": part.text}
- if isinstance(part, LMImagePart):
- return {"type": "image_url", "image_url": {"url": _history_part_source(part)}}
- if isinstance(part, LMAudioPart):
- input_audio = {"format": _history_media_format(part.media_type)}
- if part.data is not None:
- if part.data.startswith("data:"):
- media_type, data = _split_data_uri(part.data)
- input_audio["format"] = _history_media_format(media_type)
- input_audio["data"] = data
- else:
- input_audio["data"] = part.data
- elif part.url is not None:
- input_audio["url"] = part.url
- elif part.file_id is not None:
- input_audio["file_id"] = part.file_id
- elif part.path is not None:
- input_audio["path"] = part.path
- return {
- "type": "input_audio",
- "input_audio": input_audio,
- }
- if isinstance(part, LMVideoPart):
- video = {"media_type": part.media_type}
- if part.data is not None:
- video["data"] = _history_part_source(part)
- elif part.url is not None:
- video["url"] = part.url
- elif part.file_id is not None:
- video["file_id"] = part.file_id
- elif part.path is not None:
- video["path"] = part.path
- return {"type": "video", "video": video}
- if isinstance(part, LMDocumentPart):
- data = {"type": "document"}
- if part.source is not None:
- data["source"] = part.source
- else:
- data["source"] = _history_part_source(part)
- data["media_type"] = part.media_type
- if part.citations:
- data["citations"] = part.citations
- if part.title is not None:
- data["title"] = part.title
- if part.context is not None:
- data["context"] = part.context
- return data
- if isinstance(part, LMBinaryPart):
- return {
- "type": "binary",
- "binary": {
- key: value
- for key, value in {
- "data": _history_part_source(part),
- "file_id": part.file_id,
- "filename": part.filename,
- "media_type": part.media_type,
- }.items()
- if value is not None
- },
- }
- return part.model_dump(exclude_none=True)
-
-
-def _history_part_source(part: LMImagePart | LMAudioPart | LMVideoPart | LMDocumentPart | LMBinaryPart) -> str | None:
- if part.data is not None:
- return part.data if part.data.startswith("data:") else f"data:{part.media_type};base64,{part.data}"
- return part.url or part.file_id or part.path
-
-
-def _history_media_format(media_type: str) -> str:
- return media_type.split("/", 1)[1] if "/" in media_type else media_type
-
-
-def _history_request_kwargs(request: LMRequest) -> dict[str, Any]:
- data = request.config.model_dump(exclude_none=True)
- extensions = data.pop("extensions", {}) or {}
- return {**extensions, **data}
-
-
-def _validate_one_source(part: Any) -> None:
- sources = {name: getattr(part, name) for name in ("data", "url", "file_id", "path") if getattr(part, name) is not None}
- class_name = type(part).__name__
- if len(sources) != 1:
- raise ValueError(f"{class_name} requires exactly one of data, url, file_id, or path.")
- name, value = next(iter(sources.items()))
- if isinstance(value, str) and not value:
- raise ValueError(f"{class_name}.{name} must be non-empty.")
-
-
-def _coerce_message(value: dict[str, Any] | LMMessage) -> LMMessage:
- if isinstance(value, LMMessage):
- return value
- return LMMessage(**value)
-
-
-def _messages_from_items(items: tuple[Any, ...], *, prompt: str | None = None) -> tuple[list[LMMessage], list[Any]]:
- # TODO: Normalize DSPy-specific LM(...) objects in the LM call layer before building LMRequest.
- if prompt is not None:
- items = (prompt, *items)
- if not items:
- items = ("",)
-
- if len(items) == 1 and _is_message_sequence(items[0]):
- items = tuple(items[0])
-
- if all(isinstance(item, LMMessage) or isinstance(item, LMResponse) for item in items):
- messages: list[LMMessage] = []
- for item in items:
- if isinstance(item, LMMessage):
- messages.append(item)
- else:
- messages.extend(_messages_from_response(item))
- return messages, []
-
- parts = [_coerce_part(item) for item in items]
- return [LMMessage(role="user", parts=parts)], []
-
-
-def _messages_from_response(response: LMResponse) -> list[LMMessage]:
- return [LMMessage(role="assistant", parts=output.parts) for output in response.outputs]
-
-
-def _is_message_sequence(value: Any) -> bool:
- return isinstance(value, (list, tuple)) and all(
- isinstance(item, LMMessage) or isinstance(item, LMResponse) for item in value
- )
-
-
-def _coerce_part(value: Any) -> LMPart:
- if isinstance(
- value,
- (
- LMTextPart,
- LMImagePart,
- LMAudioPart,
- LMVideoPart,
- LMDocumentPart,
- LMBinaryPart,
- LMToolCallPart,
- LMToolResultPart,
- LMThinkingPart,
- LMCitationPart,
- LMRefusalPart,
- ),
- ):
- return value
- if isinstance(value, str):
- return LMTextPart(text=value)
- if isinstance(value, dict) and "type" in value:
- return pydantic.TypeAdapter(LMPart).validate_python(value)
- raise TypeError(f"Cannot convert {type(value)!r} to an LMPart.")
-
-
-def _parts_from_openai_content(content: Any) -> list[LMPart]:
- if content is None:
- return []
- if isinstance(content, str):
- return [LMTextPart(text=content)]
- if not isinstance(content, list):
- return [_coerce_part(content)]
-
- parts = []
- for item in content:
- item_type = item.get("type") if isinstance(item, dict) else None
- if item_type == "text":
- parts.append(LMTextPart(text=item.get("text", ""), metadata=item.get("metadata", {}) or {}))
- elif item_type == "image_url":
- image = item.get("image_url", {})
- if not isinstance(image, dict):
- raise TypeError("Image content block must be a mapping.")
- url = image.get("url")
- if url is None:
- raise ValueError("Image content block requires url.")
- parts.append(_image_source_to_part(url))
- elif item_type == "input_audio":
- audio = item.get("input_audio", {})
- parts.append(_audio_dict_to_part(audio))
- elif item_type == "file":
- parts.append(_binary_dict_to_part(item.get("file", {})))
- elif item_type == "document":
- parts.append(_document_dict_to_part(item))
- elif item_type == "video":
- video = item.get("video", {})
- parts.append(_media_dict_to_video_part(video))
- else:
- parts.append(_coerce_part(item))
- return parts
-
-
-def _tool_calls_from_openai(tool_calls: list[Any]) -> list[LMToolCallPart]:
- return [_tool_call_from_openai(tool_call) for tool_call in tool_calls]
-
-
-def _tool_call_from_openai(tool_call: Any) -> LMToolCallPart:
- if not isinstance(tool_call, Mapping):
- part = _coerce_part(tool_call)
- if isinstance(part, LMToolCallPart):
- return part
- raise TypeError(f"Cannot convert {type(tool_call)!r} to an LMToolCallPart.")
-
- function = tool_call.get("function", {})
- if not isinstance(function, Mapping):
- function = {}
-
- args = function.get("arguments")
- if args is None:
- args = tool_call.get("arguments", {})
- if isinstance(args, str):
- args = _parse_json_object(args)
- elif isinstance(args, Mapping):
- args = dict(args)
- else:
- args = {}
-
- return LMToolCallPart(
- id=tool_call.get("call_id") or tool_call.get("id"),
- name=function.get("name") or tool_call.get("name") or "",
- args=args,
- )
-
-
-def _image_source_to_part(source: str) -> LMImagePart:
- if not isinstance(source, str):
- raise TypeError("Image URL must be a string.")
- if source.startswith("data:"):
- media_type, data = _split_data_uri(source)
- return LMImagePart(data=data, media_type=media_type)
- media_type = mimetypes.guess_type(urlparse(source).path)[0] or "image/png"
- return LMImagePart(url=source, media_type=media_type)
-
-
-def _audio_dict_to_part(audio: dict[str, Any]) -> LMAudioPart:
- if not isinstance(audio, dict):
- raise TypeError("Audio content block must be a mapping.")
-
- audio_format = audio.get("format") or "wav"
- if not isinstance(audio_format, str):
- raise TypeError("Audio format must be a string.")
- media_type = audio_format if "/" in audio_format else f"audio/{audio_format}"
-
- if audio.get("data") is not None:
- data = audio["data"]
- if isinstance(data, str) and data.startswith("data:"):
- media_type, data = _split_data_uri(data)
- return LMAudioPart(data=data, media_type=media_type)
- if audio.get("url") is not None:
- return LMAudioPart(url=audio["url"], media_type=media_type)
- if audio.get("file_id") is not None:
- return LMAudioPart(file_id=audio["file_id"], media_type=media_type)
- if audio.get("path") is not None:
- return LMAudioPart(path=audio["path"], media_type=media_type)
- raise ValueError("Audio content block requires data, url, file_id, or path.")
-
-
-def _binary_dict_to_part(file: dict[str, Any]) -> LMBinaryPart:
- # Stash the original block whenever the typed model can't re-emit it
- # losslessly: both file_data and file_id present (a part holds one source),
- # or file_data that isn't a data: URI (re-wrapping would invent a
- # media type the caller never declared).
- file_data = file.get("file_data")
- needs_legacy_block = file_data is not None and (
- file.get("file_id") is not None or not str(file_data).startswith("data:")
- )
- legacy_block = {"type": "file", "file": dict(file)} if needs_legacy_block else None
- if file_data is not None:
- media_type, data = _split_data_uri(file_data)
- metadata = {"legacy_content_block": legacy_block} if legacy_block is not None else {}
- return LMBinaryPart(data=data, media_type=media_type, filename=file.get("filename"), metadata=metadata)
- if file.get("data") is not None:
- media_type, data = _split_data_uri(file["data"])
- return LMBinaryPart(data=data, media_type=media_type, filename=file.get("filename"))
- if file.get("file_id") is not None:
- return LMBinaryPart(file_id=file["file_id"], filename=file.get("filename"))
- raise ValueError("Binary content block requires data, file_data, or file_id.")
-
-
-def _document_dict_to_part(item: dict[str, Any]) -> LMDocumentPart:
- common = {"title": item.get("title"), "context": item.get("context")}
- media_type = item.get("media_type") or "application/pdf"
- for source_key in ("data", "url", "file_id", "path"):
- if item.get(source_key) is not None:
- return LMDocumentPart(**{source_key: item[source_key]}, media_type=media_type, **common)
-
- source = item.get("source")
- if isinstance(source, dict):
- return LMDocumentPart(
- source=source,
- citations=item.get("citations") or {},
- **common,
- )
- if isinstance(source, str):
- kwargs = _media_source_kwargs(source, default_media_type=media_type)
- return LMDocumentPart(**kwargs, **common)
- raise ValueError("Document content block requires source.")
-
-
-def _media_dict_to_video_part(video: dict[str, Any]) -> LMVideoPart:
- if video.get("data") is not None:
- data = video["data"]
- if isinstance(data, str) and data.startswith("data:"):
- media_type, data = _split_data_uri(data)
- else:
- media_type = video.get("media_type") or "video/mp4"
- return LMVideoPart(data=data, media_type=media_type)
- if video.get("url") is not None:
- return LMVideoPart(url=video["url"], media_type=video.get("media_type") or "video/mp4")
- if video.get("file_id") is not None:
- return LMVideoPart(file_id=video["file_id"], media_type=video.get("media_type") or "video/mp4")
- if video.get("path") is not None:
- return LMVideoPart(path=video["path"], media_type=video.get("media_type") or "video/mp4")
- raise ValueError("Video content block requires data, url, file_id, or path.")
-
-
-def _split_data_uri(value: str) -> tuple[str, str]:
- if not value.startswith("data:") or "," not in value:
- return "application/octet-stream", value
- header, data = value.split(",", 1)
- media_type = header.removeprefix("data:").split(";", 1)[0]
- return media_type, data
-
-
-def _media_source_kwargs(source: str, *, default_media_type: str) -> dict[str, str]:
- if source.startswith("data:"):
- media_type, data = _split_data_uri(source)
- return {"data": data, "media_type": media_type}
-
- parsed = urlparse(source)
- if parsed.scheme in {"http", "https"}:
- media_type = mimetypes.guess_type(parsed.path)[0] or default_media_type
- return {"url": source, "media_type": media_type}
-
- return {"file_id": source, "media_type": default_media_type}
-
-
-def _coerce_tool_spec(tool: Any) -> LMToolSpec:
- if isinstance(tool, LMToolSpec):
- return tool
- if hasattr(tool, "to_lm_tool_spec"):
- return tool.to_lm_tool_spec()
- if isinstance(tool, dict):
- if "function" in tool:
- function = tool["function"]
- # provider_data carries function-scoped extras (see tool_to_openai*);
- # top-level extras on the nested form are folded into the same scope.
- provider_data = {key: value for key, value in tool.items() if key not in {"type", "function"}}
- provider_data.update(
- {
- key: value
- for key, value in function.items()
- if key not in {"name", "description", "parameters", "strict"}
- }
- )
- return LMToolSpec(
- name=function.get("name"),
- description=function.get("description"),
- parameters=function.get("parameters", {}),
- strict=function.get("strict"),
- provider_data=provider_data,
- )
- known_fields = set(LMToolSpec.model_fields)
- extras = {key: value for key, value in tool.items() if key not in known_fields}
- if extras:
- tool = {key: value for key, value in tool.items() if key in known_fields}
- tool["provider_data"] = {**tool.get("provider_data", {}), **extras}
- return LMToolSpec(**tool)
- raise TypeError(f"Cannot convert {type(tool)!r} to LMToolSpec.")
-
-
-def _requires_output_dict(output: LMOutput) -> bool:
- return bool(
- output.logprobs is not None or output.reasoning_content is not None or output.tool_calls or output.citations
- )
-
-
-def _part_to_value(part: LMPart) -> Any:
- if isinstance(part, LMTextPart):
- return part.text
- if isinstance(part, LMThinkingPart):
- return part
- if isinstance(part, LMToolCallPart):
- return part
- if isinstance(part, LMRefusalPart):
- return part.text
- return part
-
-
-def _finalize_stream_part(part: LMPart) -> LMPart:
- if isinstance(part, LMToolCallPart) and "args_buffer" in part.provider_data:
- return part.model_copy(update={"args": _parse_json_object_strict(part.provider_data["args_buffer"])})
- return part
-
-
-def _tool_call_to_provider_dict(call: LMToolCallPart) -> dict[str, Any]:
- data = {
- "type": "function",
- "function": {
- "name": call.name,
- "arguments": json.dumps(call.args),
- },
- }
- if call.id is not None:
- data["id"] = call.id
- return data
-
-
-def _parse_json_object(value: str) -> dict[str, Any]:
- if not value:
- return {}
- try:
- parsed = json.loads(value)
- except json.JSONDecodeError:
- return {}
- return parsed if isinstance(parsed, dict) else {}
-
-
-def _parse_json_object_strict(value: str) -> dict[str, Any]:
- if not value:
- return {}
- try:
- parsed = json.loads(value)
- except json.JSONDecodeError as exc:
- raise ValueError("Streamed tool-call arguments must be a JSON object.") from exc
- if not isinstance(parsed, dict):
- raise ValueError("Streamed tool-call arguments must be a JSON object.")
- return parsed
diff --git a/dspy/lm15.py b/dspy/lm15.py
index 64e92473f..df15e268f 100644
--- a/dspy/lm15.py
+++ b/dspy/lm15.py
@@ -1,8 +1,8 @@
"""Use the lm15 types and providers bundled with DSPy.
Import public names here rather than from DSPy's private vendor directory.
-These are the original lm15 objects, not copies or subclasses. Bundling them
-alone does not change how `dspy.LM` makes requests.
+These are the original lm15 objects, not copies or subclasses. `dspy.LM`
+accepts explicit Request objects and uses them in its native engine path.
Examples:
>>> from dspy.lm15 import Message, Request
diff --git a/dspy/predict/rlm.py b/dspy/predict/rlm.py
index 3141d443a..559dcd2f2 100644
--- a/dspy/predict/rlm.py
+++ b/dspy/predict/rlm.py
@@ -280,14 +280,14 @@ class RLM(Module):
"No LM configured. Use dspy.configure(lm=...) or pass sub_lm to RLM."
)
response = target_lm(prompt)
- if isinstance(response, dspy.LMResponse):
+ if isinstance(response, dspy.lm15.Response):
text = response.text
elif isinstance(response, list) and response:
first_output = response[0]
text = first_output.get("text") if isinstance(first_output, dict) else first_output
else:
raise TypeError(
- "Sub-LM must return dspy.LMResponse or a non-empty list of text outputs, "
+ "Sub-LM must return dspy.lm15.Response or a non-empty list of text outputs, "
f"got {type(response).__name__}."
)
diff --git a/dspy/streaming/streamify.py b/dspy/streaming/streamify.py
index 70bdbab16..5fd3ea436 100644
--- a/dspy/streaming/streamify.py
+++ b/dspy/streaming/streamify.py
@@ -20,6 +20,10 @@ logger = logging.getLogger(__name__)
def _is_litellm_model_response_stream(value: Any) -> bool:
+ from dspy.clients.engines.streaming import EngineChunk
+
+ if isinstance(value, EngineChunk):
+ return True
cls = type(value)
return cls.__name__ == "ModelResponseStream" and cls.__module__.startswith("litellm")
diff --git a/dspy/utils/dummies.py b/dspy/utils/dummies.py
index a5ac8fc1e..3ecc3f837 100644
--- a/dspy/utils/dummies.py
+++ b/dspy/utils/dummies.py
@@ -69,8 +69,6 @@ class DummyLM(BaseLM):
"""
- forward_contract = "legacy"
-
def __init__(
self,
answers: list[dict[str, Any]] | dict[str, dict[str, Any]],
@@ -91,6 +89,13 @@ class DummyLM(BaseLM):
adapter = ChatAdapter()
self.adapter = adapter
+ from dspy.clients.engines.dummy_engine import AsyncDummyEngine, DummyEngine
+
+ self._engine_spec = DummyEngine(self)
+ self._async_engine_spec = AsyncDummyEngine(self._engine_spec)
+ # DummyLM has always consumed scripted answers even for repeated calls.
+ self._cache_responses = False
+
def _use_example(self, messages):
# find all field names
fields = defaultdict(int)
@@ -125,35 +130,22 @@ class DummyLM(BaseLM):
return adapter.format_field_with_value(fields_with_values)
def forward(self, prompt=None, messages=None, **kwargs):
- messages = messages or [{"role": "user", "content": prompt}]
- kwargs = {**self.kwargs, **kwargs}
-
- choices = []
- for _ in range(kwargs.get("n", 1)):
- if self.follow_examples:
- current_output = self._use_example(messages)
- elif isinstance(self.answers, dict):
- current_output = next(
- (self._format_answer_fields(v) for k, v in self.answers.items() if k in messages[-1]["content"]),
- "No more responses",
- )
- else:
- current_output = self._format_answer_fields(next(self.answers, {"answer": "No more responses"}))
-
- message = dotdict(content=current_output, tool_calls=None)
- if self.reasoning:
- message.reasoning_content = "Some reasoning"
- choices.append(dotdict(message=message, finish_reason="stop"))
-
- return dotdict(
- choices=choices,
- usage=dotdict(prompt_tokens=0, completion_tokens=0, total_tokens=0),
- model="dummy",
- )
+ from dspy.clients.execution import execute, prepare
+
+ return execute(self, prepare(self, prompt, messages, kwargs, direct=True)).provider_response()
async def aforward(self, prompt=None, messages=None, **kwargs):
+ # Preserve the historical subclass extension point on async calls.
return self.forward(prompt=prompt, messages=messages, **kwargs)
+ def copy(self, **kwargs):
+ from dspy.clients.engines.dummy_engine import AsyncDummyEngine, DummyEngine
+
+ copied = super().copy(**kwargs)
+ copied._engine_spec = DummyEngine(copied)
+ copied._async_engine_spec = AsyncDummyEngine(copied._engine_spec)
+ return copied
+
def get_convo(self, index):
"""Get the prompt + answer from the ith message."""
return self.history[index]["messages"], self.history[index]["outputs"]
diff --git a/dspy/utils/exceptions.py b/dspy/utils/exceptions.py
index 4e786b710..1d0248450 100644
--- a/dspy/utils/exceptions.py
+++ b/dspy/utils/exceptions.py
@@ -50,10 +50,11 @@ class DSPyError(Exception):
class LMError(DSPyError):
"""Base class for language model errors.
- Catch this class to handle any failure raised while configuring or calling
- an LM. Concrete subclasses identify whether the failure came from local
- configuration, transport, provider authentication, rate limits, invalid
- requests, unsupported features, or provider server errors.
+ Catch this class for failures at DSPy's engine and capability boundaries.
+ Concrete subclasses identify local configuration, transport, authentication,
+ rate limits, invalid requests, unsupported features, and provider failures.
+ Invalid Python API arguments still raise TypeError/ValueError; missing
+ dependencies raise ImportError. Cancellation and warning policies propagate.
"""
default_code = "lm_error"
@@ -69,6 +70,17 @@ class LMTransportError(LMError):
default_code = "transport"
+class LMLockTimeoutError(LMError):
+ """Local credential-lock contention, not a provider or authentication failure."""
+
+ default_code = "lock_timeout"
+
+ def __init__(self, message: str = "", *, path: str = "", lock_path: str = "", **kwargs: Any):
+ self.path = path
+ self.lock_path = lock_path
+ super().__init__(message, **kwargs)
+
+
class LMConfigurationError(LMError):
"""The LM or provider client is not configured correctly."""
@@ -122,16 +134,31 @@ class LMProviderError(LMError):
class LMUnexpectedError(LMError):
"""An unexpected failure occurred at the LM provider boundary.
- DSPy raises this when an exception is raised while calling the LM backend,
- but the exception does not match a known provider error class and does not
- include an HTTP status code. This keeps adapter fallback behavior from
- treating unknown LM-boundary failures as parse failures while avoiding
- over-classifying them as provider response errors.
+ DSPy raises this for an unclassified engine failure and preserves its
+ original exception as the cause. Only the owning engine interprets SDK
+ errors: arbitrary message text or status-like attributes do not make a
+ custom engine failure retryable. This is not a provider-response error or
+ a model-output parsing error.
"""
default_code = "unexpected"
+class LMStreamAssemblyError(LMUnexpectedError):
+ """An incomplete or invalid stream cannot be accepted as a successful response.
+
+ ``partial`` carries salvageable content when available. It is not a success
+ and must not be cached. Unknown usage remains unknown.
+ """
+
+ default_code = "stream_assembly"
+
+ def __init__(self, message: str = "", *, partial=None, part_index: int | None = None, **kwargs: Any):
+ self.partial = partial
+ self.part_index = part_index
+ super().__init__(message, **kwargs)
+
+
class LMAuthError(LMProviderError):
"""The provider rejected the request because authentication failed."""
@@ -163,10 +190,10 @@ class LMInvalidRequestError(LMProviderError):
class ContextWindowExceededError(LMInvalidRequestError):
"""Raised when the prompt exceeds the model's context window.
- Any LM subclass should raise this error, or a subclass of it, when the
- request fails because the input is too long for the model. Adapters and
- some modules rely on catching this specific type to decide whether a
- fallback retry is appropriate.
+ Custom engines raise dspy.lm15.ContextLengthError; DSPy projects it to
+ this public type. Legacy 3.4 LM subclasses may still raise this error
+ directly. Modules such as ReAct catch it to shorten an overlong history;
+ it does not trigger a generation retry or adapter-format fallback.
Args:
model: The model identifier that rejected the request.
@@ -205,16 +232,17 @@ class LMServerError(LMProviderError):
default_code = "server"
-_RETRYABLE_LM_ERRORS = (LMRateLimitError, LMTimeoutError, LMServerError, LMTransportError)
+_RETRYABLE_LM_ERRORS = (LMRateLimitError, LMTimeoutError, LMServerError, LMTransportError, LMLockTimeoutError)
def is_retryable_lm_error(error: Exception) -> bool:
"""Return whether an LM error is generally safe to retry.
- DSPy's built-in `LM` delegates provider retries to LiteLLM, but callers and
- higher-level orchestration code can use this helper to classify wrapped LM
- failures after provider retries are exhausted. Retryability is advisory:
- callers should still respect provider policy and `retry_after` when present.
+ DSPy owns retries; managed engines perform one attempt. This classification
+ describes transient failures, not proof that the provider did no work.
+ Never replay after visible stream output or a completed generation, and
+ respect provider policy and valid `retry_after` hints. Network retries can
+ still repeat a request already processed or billed by the provider.
Args:
error: The exception to classify.
diff --git a/dspy/utils/parallelizer.py b/dspy/utils/parallelizer.py
index 2f71c0f12..7afcebd5e 100644
--- a/dspy/utils/parallelizer.py
+++ b/dspy/utils/parallelizer.py
@@ -144,7 +144,13 @@ class ParallelExecutor:
def handler(sig, frame):
self.cancel_jobs.set()
logger.warning("SIGINT received. Cancelling.")
- orig_handler(sig, frame)
+ if callable(orig_handler):
+ orig_handler(sig, frame)
+ else:
+ # SIG_DFL and SIG_IGN are enum sentinels, not functions.
+ # While evaluation owns SIGINT it means cancellation;
+ # restore the original disposition on leaving this scope.
+ signal.default_int_handler(sig, frame)
signal.signal(signal.SIGINT, handler)
try:
diff --git a/pyproject.toml b/pyproject.toml
index 7f36c054c..04f1eb7de 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -78,6 +78,7 @@ exclude = ["tests", "tests.*"]
[tool.setuptools.package-data]
dspy = ["primitives/*.js"]
+"dspy.clients.model_metadata" = ["snapshot.json.gz", "LICENSE", "provenance.json"]
"dspy._vendor" = ["lm15-LICENSE", "lm15-provenance.txt", "UPDATING.md"]
"dspy._vendor.lm15" = ["py.typed"]
diff --git a/scripts/update_model_metadata.py b/scripts/update_model_metadata.py
new file mode 100644
index 000000000..1fa22e161
--- /dev/null
+++ b/scripts/update_model_metadata.py
@@ -0,0 +1,46 @@
+"""Refresh DSPy's bundled metadata from a released LiteLLM wheel, without importing it.
+
+Run: python scripts/update_model_metadata.py [version]
+The runtime loader fetches the current upstream map; this snapshot is its fallback.
+Refresh and review it before tagging a release; see
+.github/.internal_dspyai/internals/release-checklist.md.
+"""
+
+import gzip
+import hashlib
+import io
+import json
+import sys
+import urllib.request
+import zipfile
+from pathlib import Path
+
+
+def main(version="1.96.0"):
+ with urllib.request.urlopen(f"https://pypi.org/pypi/litellm/{version}/json", timeout=30) as response:
+ release = json.load(response)
+ artifact = next(item for item in release["urls"] if item["filename"].endswith(".whl"))
+ with urllib.request.urlopen(artifact["url"], timeout=60) as response:
+ wheel = response.read()
+ if hashlib.sha256(wheel).hexdigest() != artifact["digests"]["sha256"]:
+ raise ValueError("Wheel checksum mismatch")
+ with zipfile.ZipFile(io.BytesIO(wheel)) as archive:
+ data = archive.read("litellm/model_prices_and_context_window_backup.json")
+ license_path = next(name for name in archive.namelist() if name.endswith("/LICENSE") and ".dist-info/" in name)
+ license_text = archive.read(license_path)
+ if not isinstance(json.loads(data), dict):
+ raise ValueError("Metadata must be a JSON object")
+ target = Path(__file__).resolve().parents[1] / "dspy/clients/model_metadata"
+ target.mkdir(parents=True, exist_ok=True)
+ (target / "snapshot.json.gz").write_bytes(gzip.compress(data, mtime=0))
+ (target / "LICENSE").write_bytes(license_text)
+ (target / "provenance.json").write_text(json.dumps({
+ "source": artifact["url"], "version": version,
+ "wheel_sha256": artifact["digests"]["sha256"],
+ "snapshot_sha256": hashlib.sha256(data).hexdigest(),
+ }, indent=2) + "\n")
+ print(f"Updated {target} from LiteLLM {version}")
+
+
+if __name__ == "__main__":
+ main(*sys.argv[1:])
diff --git a/tests/adapters/conftest.py b/tests/adapters/conftest.py
index e1d089dce..971114286 100644
--- a/tests/adapters/conftest.py
+++ b/tests/adapters/conftest.py
@@ -1,3 +1,12 @@
+"""Shared adapter test helpers.
+
+TODO(3.5): migrate general adapter LM doubles (including CapturingLM below)
+and tests mocking litellm.completion/acompletion to canonical Request/Response
+engines, with native-transport coverage where wire conversion matters. Preserve
+prompt-format and parsing assertions. Keep dedicated LiteLLM compatibility tests
+and historical cache fixtures; those are not candidates for replacement.
+"""
+
import dspy
diff --git a/tests/adapters/test_baml_adapter.py b/tests/adapters/test_baml_adapter.py
index 725a6c284..4af802f6e 100644
--- a/tests/adapters/test_baml_adapter.py
+++ b/tests/adapters/test_baml_adapter.py
@@ -457,7 +457,7 @@ def test_baml_adapter_with_code():
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
CodeGenSignature,
[],
@@ -553,7 +553,7 @@ async def test_baml_adapter_async_functionality():
adapter = BAMLAdapter()
result = await adapter.acall(
- dspy.LM(model="openai/gpt-4o", cache=False), {}, TestSignature, [], {"question": "Extract patient info"}
+ dspy.LM(engine="litellm", model="openai/gpt-4o", cache=False), {}, TestSignature, [], {"question": "Extract patient info"}
)
assert result[0]["patient"].name == "John Doe"
diff --git a/tests/adapters/test_chat_adapter.py b/tests/adapters/test_chat_adapter.py
index 67ebdfa42..0cbdef49a 100644
--- a/tests/adapters/test_chat_adapter.py
+++ b/tests/adapters/test_chat_adapter.py
@@ -72,7 +72,7 @@ def test_chat_adapter_quotes_literals_as_expected(
program = dspy.Predict(TestSignature)
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o"), adapter=dspy.ChatAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o"), adapter=dspy.ChatAdapter())
with mock.patch("litellm.completion") as mock_completion:
program(input_text=input_value)
@@ -2004,7 +2004,7 @@ def test_chat_adapter_with_pydantic_models():
question: str = dspy.InputField()
output: Answer = dspy.OutputField()
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o"), adapter=dspy.ChatAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o"), adapter=dspy.ChatAdapter())
program = dspy.Predict(TestSignature)
with mock.patch("litellm.completion") as mock_completion:
@@ -2040,7 +2040,7 @@ def test_chat_adapter_signature_information():
input2: int = dspy.InputField(desc="Integer Input")
output: str = dspy.OutputField(desc="String Output")
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o"), adapter=dspy.ChatAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o"), adapter=dspy.ChatAdapter())
program = dspy.Predict(TestSignature)
with mock.patch("litellm.completion") as mock_completion:
@@ -2272,7 +2272,7 @@ def test_chat_adapter_with_code():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
CodeGeneration,
[],
@@ -2347,7 +2347,7 @@ def test_chat_adapter_fallback_to_json_adapter_on_exception():
model="openai/gpt-4o-mini",
)
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
with mock.patch("dspy.adapters.json_adapter.JSONAdapter.__call__") as mock_json_adapter_call:
adapter(lm, {}, signature, [], {"question": "What is the capital of France?"})
@@ -2372,7 +2372,7 @@ def test_chat_adapter_fallback_preserves_native_function_calling_flag():
choices=[Choices(message=Message(content="nonsense"))],
model="openai/gpt-4o-mini",
)
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
with mock.patch("dspy.adapters.json_adapter.JSONAdapter.__call__", new=fake_json_adapter_call):
result = adapter(lm, {}, signature, [], {"question": "What is the capital of France?"})
@@ -2391,7 +2391,7 @@ def test_chat_adapter_respects_use_json_adapter_fallback_flag():
model="openai/gpt-4o-mini",
)
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
with mock.patch("dspy.adapters.json_adapter.JSONAdapter.__call__") as mock_json_adapter_call:
with pytest.raises(dspy.utils.exceptions.AdapterParseError):
@@ -2411,7 +2411,7 @@ async def test_chat_adapter_fallback_to_json_adapter_on_exception_async():
model="openai/gpt-4o-mini",
)
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
with mock.patch("dspy.adapters.json_adapter.JSONAdapter.acall") as mock_json_adapter_acall:
await adapter.acall(lm, {}, signature, [], {"question": "What is the capital of France?"})
@@ -2437,7 +2437,7 @@ async def test_chat_adapter_async_fallback_preserves_native_function_calling_fla
choices=[Choices(message=Message(content="nonsense"))],
model="openai/gpt-4o-mini",
)
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
with mock.patch("dspy.adapters.json_adapter.JSONAdapter.acall", new=fake_json_adapter_acall):
result = await adapter.acall(lm, {}, signature, [], {"question": "What is the capital of France?"})
@@ -2483,7 +2483,7 @@ def test_chat_adapter_toolcalls_native_function_calling():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
MySignature,
[],
@@ -2509,7 +2509,7 @@ def test_chat_adapter_toolcalls_native_function_calling():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
MySignature,
[],
@@ -2545,7 +2545,7 @@ def test_chat_adapter_toolcalls_vague_match():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
MySignature,
[],
@@ -2568,7 +2568,7 @@ def test_chat_adapter_toolcalls_vague_match():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
MySignature,
[],
@@ -2600,7 +2600,7 @@ def test_chat_adapter_native_reasoning():
model="anthropic/claude-3-7-sonnet-20250219",
)
modified_signature = adapter._call_preprocess(
- dspy.LM(model="anthropic/claude-3-7-sonnet-20250219", reasoning_effort="low", cache=False),
+ dspy.LM(engine="litellm", model="anthropic/claude-3-7-sonnet-20250219", reasoning_effort="low", cache=False),
{},
MySignature,
{"question": "What is the capital of France?"},
@@ -2608,7 +2608,7 @@ def test_chat_adapter_native_reasoning():
assert "reasoning" not in modified_signature.output_fields
result = adapter(
- dspy.LM(model="anthropic/claude-3-7-sonnet-20250219", reasoning_effort="low", cache=False),
+ dspy.LM(engine="litellm", model="anthropic/claude-3-7-sonnet-20250219", reasoning_effort="low", cache=False),
{},
MySignature,
[],
@@ -2641,7 +2641,7 @@ def test_chat_adapter_parses_float_with_underscores():
model="openai/gpt-4o-mini",
)
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
result = adapter(lm, {}, MySignature, [], {"question": "What is the score?"})
# The underscore-separated float should be parsed as a normal float
@@ -2685,7 +2685,7 @@ def test_null_content_raises_adapter_parse_error():
the adapter should raise AdapterParseError instead of silently returning None fields."""
from dspy.utils.exceptions import AdapterParseError
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
response = ModelResponse(
choices=[Choices(message=Message(content=None))],
model="openai/gpt-4o-mini",
@@ -2702,7 +2702,7 @@ def test_empty_string_content_raises_adapter_parse_error():
"""Same as above but with empty string content."""
from dspy.utils.exceptions import AdapterParseError
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
response = ModelResponse(
choices=[Choices(message=Message(content=""))],
model="openai/gpt-4o-mini",
@@ -2781,7 +2781,7 @@ def test_responses_model_native_tool_calling_round_trips_tool_only_output():
)
adapter = dspy.ChatAdapter(use_native_function_calling=True)
- lm = dspy.LM("openai/dspy-test-model", model_type="responses", cache=False)
+ lm = dspy.LM("openai/dspy-test-model", engine="litellm", model_type="responses", cache=False)
with mock.patch("litellm.supports_function_calling", return_value=True), mock.patch(
"litellm.responses", return_value=api_response
diff --git a/tests/adapters/test_json_adapter.py b/tests/adapters/test_json_adapter.py
index 63e07e171..c384ebb67 100644
--- a/tests/adapters/test_json_adapter.py
+++ b/tests/adapters/test_json_adapter.py
@@ -737,7 +737,7 @@ def test_json_adapter_passes_structured_output_when_supported_by_model():
program = dspy.Predict(TestSignature)
# Configure DSPy to use an OpenAI LM that supports structured outputs
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o"), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o"), adapter=dspy.JSONAdapter())
with mock.patch("litellm.completion") as mock_completion:
program(input1="Test input")
@@ -768,7 +768,7 @@ def test_json_adapter_not_using_structured_outputs_when_not_supported_by_model()
program = dspy.Predict(TestSignature)
# Configure DSPy to use a model from a fake provider that doesn't support structured outputs
- dspy.configure(lm=dspy.LM(model="fakeprovider/fakemodel", cache=False), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="fakeprovider/fakemodel", cache=False), adapter=dspy.JSONAdapter())
with mock.patch("litellm.completion") as mock_completion:
mock_completion.return_value = ModelResponse(
choices=[Choices(message=Message(content=("{'output1': 'Test output', 'output2': True}")))],
@@ -795,7 +795,7 @@ def test_json_adapter_with_structured_outputs_does_not_mutate_original_signature
output3: OutputField3 = dspy.OutputField(desc="Nested output field")
output4_unannotated = dspy.OutputField(desc="Unannotated output field")
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o"), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o"), adapter=dspy.JSONAdapter())
program = dspy.Predict(TestSignature)
with mock.patch("litellm.completion"):
program(input1="Test input")
@@ -841,7 +841,7 @@ def test_json_adapter_on_pydantic_model():
program = dspy.Predict(TestSignature)
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o", cache=False), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o", cache=False), adapter=dspy.JSONAdapter())
with mock.patch("litellm.completion") as mock_completion:
mock_completion.return_value = ModelResponse(
@@ -919,7 +919,7 @@ def test_json_adapter_parse_raise_error_on_mismatch_fields():
],
model="openai/gpt-4o",
)
- lm = dspy.LM(model="openai/gpt-4o-mini")
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
with pytest.raises(dspy.utils.exceptions.AdapterParseError) as e:
adapter(lm, {}, signature, [], {"question": "What is the capital of France?"})
@@ -1122,7 +1122,7 @@ def test_json_adapter_with_tool():
assert "{'country': {'type': 'string'}, 'year': {'type': 'integer'}}" in messages[1]["content"]
with mock.patch("litellm.completion") as mock_completion:
- lm = dspy.LM(model="openai/gpt-4o-mini")
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
adapter(lm, {}, MySignature, [], {"question": "What is the weather in Tokyo?", "tools": tools})
mock_completion.assert_called_once()
@@ -1200,7 +1200,7 @@ def test_json_adapter_with_code():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
CodeGeneration,
[],
@@ -1264,7 +1264,7 @@ async def test_json_adapter_on_pydantic_model_async():
model="openai/gpt-4o",
)
- with dspy.context(lm=dspy.LM(model="openai/gpt-4o", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM(engine="litellm", model="openai/gpt-4o", cache=False), adapter=dspy.JSONAdapter()):
result = await program.acall(
user={"id": 5, "name": "name_test", "email": "email_test"}, question="What is the capital of France?"
)
@@ -1325,7 +1325,7 @@ def test_json_adapter_does_not_fallback_to_json_mode_on_structured_output_lm_err
question: str = dspy.InputField()
answer: str = dspy.OutputField(desc="String output field")
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter())
program = dspy.Predict(TestSignature)
with mock.patch("litellm.completion") as mock_completion:
@@ -1344,7 +1344,7 @@ def test_json_adapter_json_mode_no_structured_outputs():
question: str = dspy.InputField()
answer: str = dspy.OutputField(desc="String output field")
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o", cache=False), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o", cache=False), adapter=dspy.JSONAdapter())
program = dspy.Predict(TestSignature)
with (
@@ -1388,7 +1388,7 @@ async def test_json_adapter_json_mode_no_structured_outputs_async():
mock_get_supported_openai_params.return_value = ["response_format"]
mock_supports_response_schema.return_value = False
- with dspy.context(lm=dspy.LM(model="openai/gpt-4o", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM(engine="litellm", model="openai/gpt-4o", cache=False), adapter=dspy.JSONAdapter()):
result = await program.acall(question="Dummy question!")
assert mock_acompletion.call_count == 1
@@ -1409,7 +1409,7 @@ async def test_json_adapter_does_not_fallback_to_json_mode_on_structured_output_
with mock.patch("litellm.acompletion") as mock_acompletion:
mock_acompletion.side_effect = RuntimeError("Structured output failed!")
- with dspy.context(lm=dspy.LM(model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
with pytest.raises(dspy.LMUnexpectedError, match="Structured output failed"):
await program.acall(question="Dummy question!")
@@ -1425,7 +1425,7 @@ def test_error_message_on_json_adapter_failure():
program = dspy.Predict(TestSignature)
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter())
with mock.patch("litellm.completion") as mock_completion:
mock_completion.side_effect = RuntimeError("RuntimeError!")
@@ -1451,7 +1451,7 @@ async def test_error_message_on_json_adapter_failure_async():
program = dspy.Predict(TestSignature)
with mock.patch("litellm.acompletion") as mock_acompletion:
- with dspy.context(lm=dspy.LM(model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
mock_acompletion.side_effect = RuntimeError("RuntimeError!")
with pytest.raises(dspy.LMUnexpectedError) as error:
await program.acall(question="Dummy question!")
@@ -1502,7 +1502,7 @@ def test_json_adapter_toolcalls_native_function_calling():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
MySignature,
[],
@@ -1528,7 +1528,7 @@ def test_json_adapter_toolcalls_native_function_calling():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
MySignature,
[],
@@ -1559,7 +1559,7 @@ def test_json_adapter_toolcalls_no_native_function_calling():
model="openai/gpt-4o-mini",
)
adapter = dspy.JSONAdapter(use_native_function_calling=False)
- lm = dspy.LM(model="openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False)
adapter(lm, {}, MySignature, [], {"question": "What is the weather in Tokyo?", "tools": tools})
# _get_structured_outputs_response_format is not called because without using native function calling,
@@ -1591,7 +1591,7 @@ def test_json_adapter_native_reasoning():
model="anthropic/claude-3-7-sonnet-20250219",
)
modified_signature = adapter._call_preprocess(
- dspy.LM(model="anthropic/claude-3-7-sonnet-20250219", reasoning_effort="low", cache=False),
+ dspy.LM(engine="litellm", model="anthropic/claude-3-7-sonnet-20250219", reasoning_effort="low", cache=False),
{},
MySignature,
{"question": "What is the capital of France?"},
@@ -1599,7 +1599,7 @@ def test_json_adapter_native_reasoning():
assert "reasoning" not in modified_signature.output_fields
result = adapter(
- dspy.LM(model="anthropic/claude-3-7-sonnet-20250219", reasoning_effort="low", cache=False),
+ dspy.LM(engine="litellm", model="anthropic/claude-3-7-sonnet-20250219", reasoning_effort="low", cache=False),
{},
MySignature,
[],
@@ -1648,7 +1648,7 @@ def test_json_adapter_with_responses_api():
user=None,
)
- lm = dspy.LM(model="openai/gpt-4o", model_type="responses", cache=False)
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o", model_type="responses", cache=False)
dspy.configure(lm=lm, adapter=dspy.JSONAdapter())
program = dspy.Predict(TestSignature)
diff --git a/tests/adapters/test_tool.py b/tests/adapters/test_tool.py
index bfce1df79..e21e3e929 100644
--- a/tests/adapters/test_tool.py
+++ b/tests/adapters/test_tool.py
@@ -6,8 +6,8 @@ from pydantic import BaseModel, TypeAdapter
import dspy
from dspy.adapters.types.tool import Tool, ToolCallResults, ToolCalls, convert_input_schema_to_tool_args
-from dspy.clients.openai_format import to_openai_chat_request
-from dspy.core.types import LMMessage, LMRequest, LMToolResultPart
+from dspy.clients.lm15_boundary import request_kwargs
+from dspy.lm15 import Message, Request, ToolResultPart, tool_result
# Test fixtures
@@ -566,17 +566,17 @@ def test_tool_call_results_can_round_trip_as_native_tool_result_message():
results = ToolCallResults.from_tool_calls_and_values([tool_call], ['{"items": ["cat"]}'])
result = results.tool_call_results[0]
- message = LMMessage(role="tool", tool_call_id=result.call_id, name=result.name, content=result.value)
+ message = Message.tool(tool_result(result.call_id, result.value, name=result.name))
assert len(message.parts) == 1
- assert isinstance(message.parts[0], LMToolResultPart)
- assert message.parts[0].call_id == "call_1"
+ assert isinstance(message.parts[0], ToolResultPart)
+ assert message.parts[0].id == "call_1"
assert message.parts[0].name == "search"
assert message.parts[0].content[0].text == '{"items": ["cat"]}'
- request = LMRequest(model="test-model", messages=[message])
- assert to_openai_chat_request(request)["messages"] == [
- {"role": "tool", "content": '{"items": ["cat"]}', "tool_call_id": "call_1", "name": "search"}
+ request = Request(model="test-model", messages=[message])
+ assert request_kwargs(request, "chat")["messages"] == [
+ {"role": "tool", "content": '{"items": ["cat"]}', "tool_call_id": "call_1"}
]
diff --git a/tests/adapters/test_xml_adapter.py b/tests/adapters/test_xml_adapter.py
index 77de0fc44..36340e3f2 100644
--- a/tests/adapters/test_xml_adapter.py
+++ b/tests/adapters/test_xml_adapter.py
@@ -432,7 +432,7 @@ def test_xml_adapter_with_code():
model="openai/gpt-4o-mini",
)
result = adapter(
- dspy.LM(model="openai/gpt-4o-mini", cache=False),
+ dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False),
{},
CodeGeneration,
[],
diff --git a/tests/clients/fixtures/lm_3_3_0/README.md b/tests/clients/fixtures/lm_3_3_0/README.md
new file mode 100644
index 000000000..dde382367
--- /dev/null
+++ b/tests/clients/fixtures/lm_3_3_0/README.md
@@ -0,0 +1,71 @@
+# DSPy 3.3.0 LM compatibility fixtures
+
+These fixtures were generated by **unchanged DSPy 3.3.0**, commit
+`e4e97aae29b8ad8aa2fb7e99ffae6fd52970fad8`. They are not generated by the
+migration implementation. `manifest.json` records the Python/dependency
+versions, calls, exact cache keys, synthetic provider responses, outputs,
+and stable history/usage fields.
+
+Each ZIP is a real diskcache directory written and closed by that DSPy
+version. It contains SQLite databases with pickled LiteLLM responses, not
+responses reserialized by the current implementation. Tests extract a copy
+into a temporary directory and read it through DSPy's normal LM call path.
+No credentials, personal prompts, or live model responses are included.
+Only load these trusted repository-owned archives: pickle is executable data.
+
+Cases cover text, two rich answers (reasoning, tool calls, citations, logprobs),
+and reasoning-model token-parameter normalization, in sync and async forms.
+Sync and async keys differ in 3.3. On a cache hit the usage tracker stays empty,
+while history retains the response's original cost metadata; the fixtures
+capture that observed behavior, not a new billing policy.
+
+## Why these fixtures remain
+
+These protect **historical cache compatibility**, not the removed experimental
+3.3 request/response types. `test_lm_migration_compatibility.py` checks the recorded
+outputs, cache keys, usage, and history. `test_old_disk_cache_hits_before_native_routing`
+in `test_lm_engine_execution.py` also reads all six archives with the default engine
+selection and forbids engine construction on a cache hit.
+
+Keep the archives, manifest, and generator while maintaining that compatibility
+promise. Replacing their contents with lm15 responses would stop testing whether
+actual old cache files can be read. The generator is the reproducible record of
+how they were made, not a script to run during the 3.5 interface migration. Removing
+legacy LM interfaces does not by itself justify dropping old-cache coverage.
+
+## Regeneration (deliberate, never automatic during tests)
+
+1. Create a clean checkout at the exact commit above.
+2. Install that checkout with the dependency versions in `manifest.json`.
+3. From the old checkout, run the generator from this directory:
+
+ ```sh
+ python /absolute/path/to/generate.py /tmp/lm-3.3-fixtures
+ ```
+
+4. Review JSON and archive differences before replacing fixtures. Do not
+ refresh expected values merely to make migration tests pass.
+
+`generate.py` verifies the checkout commit and imports DSPy from that checkout.
+Only LiteLLM's network completion function is replaced with synthetic data;
+DSPy's normalization, cache key generation, cache serialization, output
+processing, usage accounting and history recording execute unchanged.
+
+## Scope
+
+These are migration tripwires, not a snapshot of every private method. The
+primary checks exercise public calls. Adapt the synthetic backend seam as engine
+integration evolves, not the recorded expectations. The key check intentionally
+inspects the cache: exact old-key compatibility is itself the requirement.
+
+The six archives occupy about 140 KiB. We keep full closed cache directories
+rather than invent a portable pickle wrapper, to test the actual old disk
+format. This pins one recorded dependency environment, not every historical
+LiteLLM version. Time stamps and random IDs are checked for presence but not
+compared with recorded values.
+
+Existing `test_lm.py`, `test_disk_serialization.py` and `tests/streaming/` retain
+coverage for errors, copy/save/load, serialization safety, and streaming.
+Text-completion and Responses-API historical disk fixtures, multimodal input
+variants, and arbitrary provider-specific parameters are not covered by this
+small first set. They need targeted additions if integration touches those paths.
diff --git a/tests/clients/fixtures/lm_3_3_0/generate.py b/tests/clients/fixtures/lm_3_3_0/generate.py
new file mode 100644
index 000000000..65bfed154
--- /dev/null
+++ b/tests/clients/fixtures/lm_3_3_0/generate.py
@@ -0,0 +1,110 @@
+"""Run with Python from a clean DSPy 3.3.0 checkout, not the migration branch.
+
+Usage: python /path/to/generate.py OUTPUT_DIRECTORY
+The subprocess working directory must be the 3.3.0 checkout. No network calls
+are made; only LiteLLM's response constructors and DSPy's actual cache are used.
+"""
+
+import asyncio
+import copy
+import importlib
+import importlib.metadata
+import json
+import platform
+import subprocess
+import sys
+import tempfile
+import zipfile
+from pathlib import Path
+from unittest.mock import patch
+
+SOURCE_COMMIT = "e4e97aae29b8ad8aa2fb7e99ffae6fd52970fad8"
+HISTORY_KEYS = ("prompt", "messages", "kwargs", "outputs", "usage", "cost", "model", "response_model", "model_type")
+
+
+def plain(value):
+ return json.loads(json.dumps(value, default=lambda obj: obj.model_dump(mode="json")))
+
+
+def main():
+ assert subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip() == SOURCE_COMMIT
+ sys.path.insert(0, str(Path.cwd()))
+ import litellm
+
+ import dspy
+ from dspy.clients.cache import Cache
+ from dspy.utils.usage_tracker import track_usage
+
+ assert Path(dspy.__file__).resolve().is_relative_to(Path.cwd())
+ output_dir = Path(sys.argv[1]).resolve()
+ output_dir.mkdir(parents=True, exist_ok=True)
+ cases = [
+ {"name": "text", "init": {"model": "openai/gpt-4o-mini", "temperature": 0.7, "max_tokens": 32},
+ "call": {"prompt": "Say hello."}, "texts": ["Hello."]},
+ {"name": "rich_n", "init": {"model": "openai/gpt-4o-mini", "temperature": 0.7, "max_tokens": 32},
+ "call": {"messages": [{"role": "user", "content": "Check Paris."}], "n": 2, "logprobs": True,
+ "rollout_id": 7}, "texts": ["First answer.", "Second answer."]},
+ {"name": "reasoning_key", "init": {"model": "openai/o3-mini", "temperature": 1.0, "max_tokens": 16000},
+ "call": {"prompt": "What is two plus two?"}, "texts": ["Four."]},
+ ]
+ cases = [{**copy.deepcopy(case), "mode": mode} for case in cases for mode in ("sync", "async")]
+ for case in cases:
+ rich = case["name"] == "rich_n"
+ case["name"] += "_" + case["mode"]
+ choices = []
+ for index, text in enumerate(case.pop("texts")):
+ message = {"role": "assistant", "content": text}
+ choice = {"index": index, "finish_reason": "stop", "message": message}
+ if rich:
+ message.update({
+ "reasoning_content": "Check the evidence.",
+ "tool_calls": [{"id": f"call_{index}", "type": "function",
+ "function": {"name": "weather", "arguments": '{"city":"Paris"}'}}],
+ "provider_specific_fields": {"citations": [[{"type": "char_location", "cited_text": "Paris",
+ "document_index": 0, "start_char_index": 0,
+ "end_char_index": 5}]]},
+ })
+ choice["logprobs"] = {"content": [{"token": "First", "logprob": -0.1, "bytes": [70],
+ "top_logprobs": []}]}
+ choices.append(choice)
+ case["provider_response"] = {"id": f"fixture-{case['name']}", "created": 1,
+ "model": case["init"]["model"], "choices": choices,
+ "usage": {"prompt_tokens": 12, "completion_tokens": 8, "total_tokens": 20}}
+ with tempfile.TemporaryDirectory() as directory:
+ cache = Cache(True, True, directory)
+ dspy.cache = cache
+ response = litellm.ModelResponse(**case["provider_response"])
+ response._hidden_params["response_cost"] = 0.01
+ lm = dspy.LM(**case["init"])
+ backend = importlib.import_module("dspy.clients.lm")._get_litellm()
+ method = "completion" if case["mode"] == "sync" else "acompletion"
+
+ def invoke():
+ return lm(**case["call"]) if case["mode"] == "sync" else asyncio.run(lm.acall(**case["call"]))
+
+ with patch.object(backend, method, return_value=copy.deepcopy(response)) as completion:
+ with track_usage() as usage:
+ case["outputs"] = plain(invoke())
+ case["cold_usage"] = usage.get_total_tokens()
+ case["cold_history"] = plain({key: lm.history[-1][key] for key in HISTORY_KEYS})
+ with track_usage() as usage:
+ assert plain(invoke()) == case["outputs"]
+ case["warm_usage"] = usage.get_total_tokens()
+ case["warm_history"] = plain({key: lm.history[-1][key] for key in HISTORY_KEYS})
+ assert completion.call_count == 1
+ assert len(cache.memory_cache) == 1
+ case["cache_key"] = next(iter(cache.memory_cache))
+ cache.disk_cache.close()
+ with zipfile.ZipFile(output_dir / f"{case['name']}.zip", "w", zipfile.ZIP_DEFLATED) as archive:
+ for path in sorted(Path(directory).rglob("*")):
+ if path.is_file():
+ archive.write(path, path.relative_to(directory))
+ manifest = {"source_commit": SOURCE_COMMIT, "python": platform.python_version(),
+ "dependencies": {name: importlib.metadata.version(name)
+ for name in ("dspy", "litellm", "diskcache", "cloudpickle", "pydantic", "orjson")},
+ "history_keys": HISTORY_KEYS, "cases": cases}
+ (output_dir / "manifest.json").write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/clients/fixtures/lm_3_3_0/manifest.json b/tests/clients/fixtures/lm_3_3_0/manifest.json
new file mode 100644
index 000000000..8b4acd323
--- /dev/null
+++ b/tests/clients/fixtures/lm_3_3_0/manifest.json
@@ -0,0 +1,1119 @@
+{
+ "cases": [
+ {
+ "cache_key": "9d5fcfd793295ebdb683e3a30b0e251e9c0c565caa672c0079d3a4701e95b002",
+ "call": {
+ "prompt": "Say hello."
+ },
+ "cold_history": {
+ "cost": 0.01,
+ "kwargs": {},
+ "messages": null,
+ "model": "openai/gpt-4o-mini",
+ "model_type": "chat",
+ "outputs": [
+ "Hello."
+ ],
+ "prompt": "Say hello.",
+ "response_model": "openai/gpt-4o-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "cold_usage": {
+ "openai/gpt-4o-mini": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "init": {
+ "max_tokens": 32,
+ "model": "openai/gpt-4o-mini",
+ "temperature": 0.7
+ },
+ "mode": "sync",
+ "name": "text_sync",
+ "outputs": [
+ "Hello."
+ ],
+ "provider_response": {
+ "choices": [
+ {
+ "finish_reason": "stop",
+ "index": 0,
+ "message": {
+ "content": "Hello.",
+ "role": "assistant"
+ }
+ }
+ ],
+ "created": 1,
+ "id": "fixture-text_sync",
+ "model": "openai/gpt-4o-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "prompt_tokens": 12,
+ "total_tokens": 20
+ }
+ },
+ "warm_history": {
+ "cost": 0.01,
+ "kwargs": {},
+ "messages": null,
+ "model": "openai/gpt-4o-mini",
+ "model_type": "chat",
+ "outputs": [
+ "Hello."
+ ],
+ "prompt": "Say hello.",
+ "response_model": "openai/gpt-4o-mini",
+ "usage": {}
+ },
+ "warm_usage": {}
+ },
+ {
+ "cache_key": "778c7834c0c2946eee028269cfe6d95bb2b7fced39079cda9b4049dbad0f0786",
+ "call": {
+ "prompt": "Say hello."
+ },
+ "cold_history": {
+ "cost": 0.01,
+ "kwargs": {},
+ "messages": null,
+ "model": "openai/gpt-4o-mini",
+ "model_type": "chat",
+ "outputs": [
+ "Hello."
+ ],
+ "prompt": "Say hello.",
+ "response_model": "openai/gpt-4o-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "cold_usage": {
+ "openai/gpt-4o-mini": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "init": {
+ "max_tokens": 32,
+ "model": "openai/gpt-4o-mini",
+ "temperature": 0.7
+ },
+ "mode": "async",
+ "name": "text_async",
+ "outputs": [
+ "Hello."
+ ],
+ "provider_response": {
+ "choices": [
+ {
+ "finish_reason": "stop",
+ "index": 0,
+ "message": {
+ "content": "Hello.",
+ "role": "assistant"
+ }
+ }
+ ],
+ "created": 1,
+ "id": "fixture-text_async",
+ "model": "openai/gpt-4o-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "prompt_tokens": 12,
+ "total_tokens": 20
+ }
+ },
+ "warm_history": {
+ "cost": 0.01,
+ "kwargs": {},
+ "messages": null,
+ "model": "openai/gpt-4o-mini",
+ "model_type": "chat",
+ "outputs": [
+ "Hello."
+ ],
+ "prompt": "Say hello.",
+ "response_model": "openai/gpt-4o-mini",
+ "usage": {}
+ },
+ "warm_usage": {}
+ },
+ {
+ "cache_key": "48d470a107728ef8cea49490502659e6b89418695562f945725e52184cb8ece1",
+ "call": {
+ "logprobs": true,
+ "messages": [
+ {
+ "content": "Check Paris.",
+ "role": "user"
+ }
+ ],
+ "n": 2,
+ "rollout_id": 7
+ },
+ "cold_history": {
+ "cost": 0.01,
+ "kwargs": {
+ "logprobs": true,
+ "n": 2,
+ "rollout_id": 7
+ },
+ "messages": [
+ {
+ "content": "Check Paris.",
+ "role": "user"
+ }
+ ],
+ "model": "openai/gpt-4o-mini",
+ "model_type": "chat",
+ "outputs": [
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "First answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_0",
+ "type": "function"
+ }
+ ]
+ },
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "Second answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_1",
+ "type": "function"
+ }
+ ]
+ }
+ ],
+ "prompt": null,
+ "response_model": "openai/gpt-4o-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "cold_usage": {
+ "openai/gpt-4o-mini": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "init": {
+ "max_tokens": 32,
+ "model": "openai/gpt-4o-mini",
+ "temperature": 0.7
+ },
+ "mode": "sync",
+ "name": "rich_n_sync",
+ "outputs": [
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "First answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_0",
+ "type": "function"
+ }
+ ]
+ },
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "Second answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_1",
+ "type": "function"
+ }
+ ]
+ }
+ ],
+ "provider_response": {
+ "choices": [
+ {
+ "finish_reason": "stop",
+ "index": 0,
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "message": {
+ "content": "First answer.",
+ "provider_specific_fields": {
+ "citations": [
+ [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ]
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "role": "assistant",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_0",
+ "type": "function"
+ }
+ ]
+ }
+ },
+ {
+ "finish_reason": "stop",
+ "index": 1,
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "message": {
+ "content": "Second answer.",
+ "provider_specific_fields": {
+ "citations": [
+ [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ]
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "role": "assistant",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_1",
+ "type": "function"
+ }
+ ]
+ }
+ }
+ ],
+ "created": 1,
+ "id": "fixture-rich_n_sync",
+ "model": "openai/gpt-4o-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "prompt_tokens": 12,
+ "total_tokens": 20
+ }
+ },
+ "warm_history": {
+ "cost": 0.01,
+ "kwargs": {
+ "logprobs": true,
+ "n": 2,
+ "rollout_id": 7
+ },
+ "messages": [
+ {
+ "content": "Check Paris.",
+ "role": "user"
+ }
+ ],
+ "model": "openai/gpt-4o-mini",
+ "model_type": "chat",
+ "outputs": [
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "First answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_0",
+ "type": "function"
+ }
+ ]
+ },
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "Second answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_1",
+ "type": "function"
+ }
+ ]
+ }
+ ],
+ "prompt": null,
+ "response_model": "openai/gpt-4o-mini",
+ "usage": {}
+ },
+ "warm_usage": {}
+ },
+ {
+ "cache_key": "f28cf7bece73d7c0c0d5f04e421d257c6f9751f1fa3d6d58ebd73e3c611d38f4",
+ "call": {
+ "logprobs": true,
+ "messages": [
+ {
+ "content": "Check Paris.",
+ "role": "user"
+ }
+ ],
+ "n": 2,
+ "rollout_id": 7
+ },
+ "cold_history": {
+ "cost": 0.01,
+ "kwargs": {
+ "logprobs": true,
+ "n": 2,
+ "rollout_id": 7
+ },
+ "messages": [
+ {
+ "content": "Check Paris.",
+ "role": "user"
+ }
+ ],
+ "model": "openai/gpt-4o-mini",
+ "model_type": "chat",
+ "outputs": [
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "First answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_0",
+ "type": "function"
+ }
+ ]
+ },
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "Second answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_1",
+ "type": "function"
+ }
+ ]
+ }
+ ],
+ "prompt": null,
+ "response_model": "openai/gpt-4o-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "cold_usage": {
+ "openai/gpt-4o-mini": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "init": {
+ "max_tokens": 32,
+ "model": "openai/gpt-4o-mini",
+ "temperature": 0.7
+ },
+ "mode": "async",
+ "name": "rich_n_async",
+ "outputs": [
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "First answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_0",
+ "type": "function"
+ }
+ ]
+ },
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "Second answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_1",
+ "type": "function"
+ }
+ ]
+ }
+ ],
+ "provider_response": {
+ "choices": [
+ {
+ "finish_reason": "stop",
+ "index": 0,
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "message": {
+ "content": "First answer.",
+ "provider_specific_fields": {
+ "citations": [
+ [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ]
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "role": "assistant",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_0",
+ "type": "function"
+ }
+ ]
+ }
+ },
+ {
+ "finish_reason": "stop",
+ "index": 1,
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "message": {
+ "content": "Second answer.",
+ "provider_specific_fields": {
+ "citations": [
+ [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ]
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "role": "assistant",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_1",
+ "type": "function"
+ }
+ ]
+ }
+ }
+ ],
+ "created": 1,
+ "id": "fixture-rich_n_async",
+ "model": "openai/gpt-4o-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "prompt_tokens": 12,
+ "total_tokens": 20
+ }
+ },
+ "warm_history": {
+ "cost": 0.01,
+ "kwargs": {
+ "logprobs": true,
+ "n": 2,
+ "rollout_id": 7
+ },
+ "messages": [
+ {
+ "content": "Check Paris.",
+ "role": "user"
+ }
+ ],
+ "model": "openai/gpt-4o-mini",
+ "model_type": "chat",
+ "outputs": [
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "First answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_0",
+ "type": "function"
+ }
+ ]
+ },
+ {
+ "citations": [
+ {
+ "cited_text": "Paris",
+ "document_index": 0,
+ "end_char_index": 5,
+ "start_char_index": 0,
+ "type": "char_location"
+ }
+ ],
+ "logprobs": {
+ "content": [
+ {
+ "bytes": [
+ 70
+ ],
+ "logprob": -0.1,
+ "token": "First",
+ "top_logprobs": []
+ }
+ ]
+ },
+ "reasoning_content": "Check the evidence.",
+ "text": "Second answer.",
+ "tool_calls": [
+ {
+ "function": {
+ "arguments": "{\"city\":\"Paris\"}",
+ "name": "weather"
+ },
+ "id": "call_1",
+ "type": "function"
+ }
+ ]
+ }
+ ],
+ "prompt": null,
+ "response_model": "openai/gpt-4o-mini",
+ "usage": {}
+ },
+ "warm_usage": {}
+ },
+ {
+ "cache_key": "ccdb678ca227fe3ebda50fe7265b8e5d1001d8ace76422dd186d6b6a6fb289a1",
+ "call": {
+ "prompt": "What is two plus two?"
+ },
+ "cold_history": {
+ "cost": 0.01,
+ "kwargs": {},
+ "messages": null,
+ "model": "openai/o3-mini",
+ "model_type": "chat",
+ "outputs": [
+ "Four."
+ ],
+ "prompt": "What is two plus two?",
+ "response_model": "openai/o3-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "cold_usage": {
+ "openai/o3-mini": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "init": {
+ "max_tokens": 16000,
+ "model": "openai/o3-mini",
+ "temperature": 1.0
+ },
+ "mode": "sync",
+ "name": "reasoning_key_sync",
+ "outputs": [
+ "Four."
+ ],
+ "provider_response": {
+ "choices": [
+ {
+ "finish_reason": "stop",
+ "index": 0,
+ "message": {
+ "content": "Four.",
+ "role": "assistant"
+ }
+ }
+ ],
+ "created": 1,
+ "id": "fixture-reasoning_key_sync",
+ "model": "openai/o3-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "prompt_tokens": 12,
+ "total_tokens": 20
+ }
+ },
+ "warm_history": {
+ "cost": 0.01,
+ "kwargs": {},
+ "messages": null,
+ "model": "openai/o3-mini",
+ "model_type": "chat",
+ "outputs": [
+ "Four."
+ ],
+ "prompt": "What is two plus two?",
+ "response_model": "openai/o3-mini",
+ "usage": {}
+ },
+ "warm_usage": {}
+ },
+ {
+ "cache_key": "93185b1a3b1bd2abcd106efe23749f7c38e8e5dacf0bd4d285c4c75fa631e51e",
+ "call": {
+ "prompt": "What is two plus two?"
+ },
+ "cold_history": {
+ "cost": 0.01,
+ "kwargs": {},
+ "messages": null,
+ "model": "openai/o3-mini",
+ "model_type": "chat",
+ "outputs": [
+ "Four."
+ ],
+ "prompt": "What is two plus two?",
+ "response_model": "openai/o3-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "cold_usage": {
+ "openai/o3-mini": {
+ "completion_tokens": 8,
+ "completion_tokens_details": null,
+ "prompt_tokens": 12,
+ "prompt_tokens_details": null,
+ "total_tokens": 20
+ }
+ },
+ "init": {
+ "max_tokens": 16000,
+ "model": "openai/o3-mini",
+ "temperature": 1.0
+ },
+ "mode": "async",
+ "name": "reasoning_key_async",
+ "outputs": [
+ "Four."
+ ],
+ "provider_response": {
+ "choices": [
+ {
+ "finish_reason": "stop",
+ "index": 0,
+ "message": {
+ "content": "Four.",
+ "role": "assistant"
+ }
+ }
+ ],
+ "created": 1,
+ "id": "fixture-reasoning_key_async",
+ "model": "openai/o3-mini",
+ "usage": {
+ "completion_tokens": 8,
+ "prompt_tokens": 12,
+ "total_tokens": 20
+ }
+ },
+ "warm_history": {
+ "cost": 0.01,
+ "kwargs": {},
+ "messages": null,
+ "model": "openai/o3-mini",
+ "model_type": "chat",
+ "outputs": [
+ "Four."
+ ],
+ "prompt": "What is two plus two?",
+ "response_model": "openai/o3-mini",
+ "usage": {}
+ },
+ "warm_usage": {}
+ }
+ ],
+ "dependencies": {
+ "cloudpickle": "3.1.2",
+ "diskcache": "5.6.3",
+ "dspy": "3.3.1",
+ "litellm": "1.96.0",
+ "orjson": "3.12.0",
+ "pydantic": "2.13.4"
+ },
+ "history_keys": [
+ "prompt",
+ "messages",
+ "kwargs",
+ "outputs",
+ "usage",
+ "cost",
+ "model",
+ "response_model",
+ "model_type"
+ ],
+ "python": "3.12.12",
+ "source_commit": "e4e97aae29b8ad8aa2fb7e99ffae6fd52970fad8"
+}
diff --git a/tests/clients/fixtures/lm_3_3_0/reasoning_key_async.zip b/tests/clients/fixtures/lm_3_3_0/reasoning_key_async.zip
new file mode 100644
index 000000000..b2652991b
Binary files /dev/null and b/tests/clients/fixtures/lm_3_3_0/reasoning_key_async.zip differ
diff --git a/tests/clients/fixtures/lm_3_3_0/reasoning_key_sync.zip b/tests/clients/fixtures/lm_3_3_0/reasoning_key_sync.zip
new file mode 100644
index 000000000..d105b08e1
Binary files /dev/null and b/tests/clients/fixtures/lm_3_3_0/reasoning_key_sync.zip differ
diff --git a/tests/clients/fixtures/lm_3_3_0/rich_n_async.zip b/tests/clients/fixtures/lm_3_3_0/rich_n_async.zip
new file mode 100644
index 000000000..ffdc598d0
Binary files /dev/null and b/tests/clients/fixtures/lm_3_3_0/rich_n_async.zip differ
diff --git a/tests/clients/fixtures/lm_3_3_0/rich_n_sync.zip b/tests/clients/fixtures/lm_3_3_0/rich_n_sync.zip
new file mode 100644
index 000000000..d3e03f20d
Binary files /dev/null and b/tests/clients/fixtures/lm_3_3_0/rich_n_sync.zip differ
diff --git a/tests/clients/fixtures/lm_3_3_0/text_async.zip b/tests/clients/fixtures/lm_3_3_0/text_async.zip
new file mode 100644
index 000000000..adcee14df
Binary files /dev/null and b/tests/clients/fixtures/lm_3_3_0/text_async.zip differ
diff --git a/tests/clients/fixtures/lm_3_3_0/text_sync.zip b/tests/clients/fixtures/lm_3_3_0/text_sync.zip
new file mode 100644
index 000000000..aa7f6aebe
Binary files /dev/null and b/tests/clients/fixtures/lm_3_3_0/text_sync.zip differ
diff --git a/tests/clients/test_backend_capabilities.py b/tests/clients/test_backend_capabilities.py
new file mode 100644
index 000000000..ac9a6a8a1
--- /dev/null
+++ b/tests/clients/test_backend_capabilities.py
@@ -0,0 +1,185 @@
+"""Capabilities must follow the configuration-based execution backend."""
+
+import asyncio
+import threading
+from types import SimpleNamespace
+
+import pytest
+
+import dspy
+from dspy.clients import capabilities as hints
+from dspy.clients.call_result import CallResult
+from dspy.clients.engines import AsyncLiteLLMEngine, AsyncLM15Engine, LiteLLMEngine, LM15Engine
+from dspy.clients.execution import _engine, prepare
+from dspy.lm15 import Message, Response, Usage
+
+
+def conflicting_metadata(monkeypatch):
+ calls = []
+
+ def capability(name, result):
+ def lookup(*args, **kwargs):
+ calls.append((name, args, kwargs, threading.get_ident()))
+ return result
+ return lookup
+
+ litellm = SimpleNamespace(
+ supports_function_calling=capability("tools", False),
+ supports_reasoning=capability("reasoning", False),
+ supports_response_schema=capability("schema", False),
+ get_supported_openai_params=capability("params", []),
+ )
+ monkeypatch.setattr("dspy.clients.lm._get_litellm", lambda: litellm)
+ monkeypatch.setattr(hints, "model_info", lambda *args: {
+ "supports_function_calling": True, "supports_reasoning": True, "supports_response_schema": True,
+ })
+ monkeypatch.setattr("dspy.clients.costs.estimate_cost", lambda *args, **kwargs: (None, {}))
+ for name in ("OPENAI_API_BASE", "OPENAI_BASE_URL", "AZURE_API_BASE", "AZURE_BASE_URL"):
+ monkeypatch.delenv(name, raising=False)
+ return calls
+
+
[email protected]("model,options", [
+ ("openai/gpt-4o", {"custom_llm_provider": "alternate"}),
+ ("openai/gpt-4o", {"headers": {"x-test": "yes"}}),
+ ("openai/gpt-4o", {"timeout": 10}),
+ ("azure/gpt-4o", {"api_base": "https://example.invalid"}),
+ ("unmapped/model", {}),
+ ("openai/gpt-4o", {"model_type": "text"}),
+])
+def test_auto_compatibility_settings_use_litellm_capabilities(model, options, conflicting_metadata):
+ lm = dspy.LM(model, cache=False, **options)
+ backend, _, _ = _engine(lm, prepare(lm, "hello", None, {}), False)
+ try:
+ assert isinstance(backend, LiteLLMEngine)
+ assert not lm.supports_function_calling
+ assert not lm.supports_reasoning
+ assert not lm.supports_response_schema
+ assert lm.supported_params == set()
+ assert conflicting_metadata
+ if options.get("custom_llm_provider"):
+ assert all(c[2]["custom_llm_provider"] == "alternate" for c in conflicting_metadata)
+ finally:
+ lm.close()
+
+
[email protected]("variable", ["OPENAI_API_BASE", "OPENAI_BASE_URL"])
+def test_environment_gateway_uses_same_backend_for_capabilities(variable, conflicting_metadata, monkeypatch):
+ monkeypatch.setenv(variable, "https://gateway.invalid")
+ lm = dspy.LM("openai/gpt-4o", cache=False)
+ assert isinstance(_engine(lm, prepare(lm, "hello", None, {}), False)[0], LiteLLMEngine)
+ assert not lm.supports_function_calling
+ assert not lm.supports_response_schema
+ assert conflicting_metadata
+
+
+def test_native_compatible_settings_do_not_consult_litellm(conflicting_metadata):
+ lm = dspy.LM("openai/gpt-4o", api_key="fake", api_base="https://example.invalid", cache=False)
+ try:
+ assert isinstance(_engine(lm, prepare(lm, "hello", None, {}), False)[0], LM15Engine)
+ assert lm.supports_function_calling
+ assert lm.supports_reasoning
+ assert lm.supports_response_schema
+ assert "response_format" in lm.supported_params
+ assert not conflicting_metadata
+ finally:
+ lm.close()
+
+
+def test_copy_recomputes_effective_backend(conflicting_metadata):
+ lm = dspy.LM("openai/gpt-4o")
+ assert lm.supports_response_schema
+ assert not lm.copy(custom_llm_provider="alternate").supports_response_schema
+ assert lm.supports_response_schema
+
+
+def test_forced_native_configuration_error_matches_execution(conflicting_metadata):
+ lm = dspy.LM("openai/gpt-4o", engine="lm15", headers={"x-test": "yes"})
+ with pytest.raises(dspy.LMUnsupportedFeatureError):
+ _engine(lm, prepare(lm, "hello", None, {}), False)
+ with pytest.raises(dspy.LMUnsupportedFeatureError):
+ _ = lm.supports_response_schema
+ assert not conflicting_metadata
+
+
[email protected]("asynchronous", [False, True])
+async def test_json_adapter_honours_call_time_backend(asynchronous, conflicting_metadata, monkeypatch):
+ requests = []
+
+ def complete(self, lm, request, **context):
+ requests.append(request)
+ assert request["custom_llm_provider"] == "alternate"
+ assert "response_format" not in request
+ return CallResult(outputs=['{"answer":"compatible"}'], response_model=lm.model)
+
+ async def acomplete(self, lm, request, **context):
+ await asyncio.sleep(0)
+ return complete(self, lm, request, **context)
+
+ monkeypatch.setattr(LiteLLMEngine, "complete_legacy", complete)
+ monkeypatch.setattr(AsyncLiteLLMEngine, "complete_legacy", acomplete)
+ lm = dspy.LM("openai/gpt-4o", cache=False)
+ signature = dspy.Signature("question -> answer")
+ kwargs = {"custom_llm_provider": "alternate"}
+ args = (lm, kwargs, signature, [], {"question": "hello"})
+ adapter = dspy.JSONAdapter()
+ thread = threading.get_ident()
+ result = await adapter.acall(*args) if asynchronous else adapter(*args)
+ assert result == [{"answer": "compatible"}]
+ assert len(requests) == 1
+ if asynchronous:
+ assert all(c[3] != thread for c in conflicting_metadata)
+ assert "custom_llm_provider" not in lm.kwargs
+ assert lm.supports_response_schema # planning override did not leak
+
+
+def test_chat_adapter_does_not_enable_unsupported_tools(conflicting_metadata, monkeypatch):
+ def lookup(query: str) -> str:
+ return query
+
+ signature = dspy.Signature({
+ "tools": (list[dspy.Tool], dspy.InputField()),
+ "calls": (dspy.ToolCalls, dspy.OutputField()),
+ })
+
+ def complete(self, lm, request, **context):
+ assert "tools" not in request
+ assert "tool_choice" not in request
+ return CallResult(outputs=['[[ ## calls ## ]]\n{"tool_calls": []}'], response_model=lm.model)
+
+ monkeypatch.setattr(LiteLLMEngine, "complete_legacy", complete)
+ lm = dspy.LM("openai/gpt-4o", cache=False)
+ result = dspy.ChatAdapter(use_native_function_calling=True, use_json_adapter_fallback=False)(
+ lm, {"custom_llm_provider": "alternate"}, signature, [], {"tools": [dspy.Tool(lookup)]},
+ )
+ assert result[0]["calls"].tool_calls == []
+
+
+async def test_concurrent_planning_scopes_are_isolated(conflicting_metadata, monkeypatch):
+ async def native(self, request):
+ await asyncio.sleep(0)
+ assert request.config.response_format is not None
+ return Response(None, request.model, Message.assistant('{"answer":"native"}'), "stop", Usage())
+
+ async def compatibility(self, lm, request, **context):
+ await asyncio.sleep(0)
+ assert "response_format" not in request
+ return CallResult(outputs=['{"answer":"compatible"}'], response_model=lm.model)
+
+ monkeypatch.setattr(AsyncLM15Engine, "complete", native)
+ monkeypatch.setattr(AsyncLiteLLMEngine, "complete_legacy", compatibility)
+ lm = dspy.LM("openai/gpt-4o", cache=False)
+ adapter = dspy.JSONAdapter()
+ signature = dspy.Signature("question -> answer")
+ try:
+ result = await asyncio.gather(
+ adapter.acall(lm, {}, signature, [], {"question": "a"}),
+ adapter.acall(lm, {"custom_llm_provider": "alternate"}, signature, [], {"question": "b"}),
+ )
+ assert result == [[{"answer": "native"}], [{"answer": "compatible"}]]
+ assert lm.supports_response_schema
+ finally:
+ await lm.aclose()
diff --git a/tests/clients/test_disk_serialization.py b/tests/clients/test_disk_serialization.py
index c801a6847..8b8ed06bb 100644
--- a/tests/clients/test_disk_serialization.py
+++ b/tests/clients/test_disk_serialization.py
@@ -19,8 +19,8 @@ from dspy.clients.disk_serialization import (
_restricted_load,
restricted_disk,
)
-from dspy.clients.openai_format import completion_to_lm_response, responses_to_lm_response
-from dspy.core.types import LMRequest
+from dspy.clients.lm15_boundary import response_value
+from dspy.lm15 import Message, Request
class AllowedPydanticModel(pydantic.BaseModel):
@@ -99,11 +99,11 @@ def _assert_cached_tool_call_normalizes(directory):
assert type(type(tool_call).__pydantic_serializer__).__name__ == "MockValSer"
assert type(type(tool_call.function).__pydantic_serializer__).__name__ == "MockValSer"
- lm_response = completion_to_lm_response(response, LMRequest(model="dummy", messages=[]))
- part = lm_response.outputs[0].tool_calls[0]
+ lm_response = response_value(response, "chat", Request(model="dummy", messages=(Message.user("-"),)))
+ part = lm_response.tool_calls[0]
assert part.name == "get_weather"
- assert part.args == {"city": "SF"}
- assert part.provider_data["function"]["name"] == "get_weather"
+ assert part.input == {"city": "SF"}
+ assert lm_response.provider_data["choices"][0]["message"]["tool_calls"][0]["function"]["name"] == "get_weather"
assert lm_response.usage.total_tokens == 3
@@ -148,11 +148,11 @@ def _assert_cached_responses_function_call_normalizes(directory):
function_call = response.output[0]
assert type(type(function_call).__pydantic_serializer__).__name__ == "MockValSer"
- lm_response = responses_to_lm_response(response, LMRequest(model="dummy", messages=[]))
- part = lm_response.outputs[0].tool_calls[0]
+ lm_response = response_value(response, "responses", Request(model="dummy", messages=(Message.user("-"),)))
+ part = lm_response.tool_calls[0]
assert part.name == "get_weather"
- assert part.args == {"city": "SF"}
- assert part.provider_data["call_id"] == "call_1"
+ assert part.input == {"city": "SF"}
+ assert lm_response.provider_data["output"][0]["call_id"] == "call_1"
assert lm_response.usage.total_tokens == 3
diff --git a/tests/clients/test_engine_errors.py b/tests/clients/test_engine_errors.py
new file mode 100644
index 000000000..b63a96d61
--- /dev/null
+++ b/tests/clients/test_engine_errors.py
@@ -0,0 +1,340 @@
+"""The public engine-error contract, including failures before inference starts."""
+
+import asyncio
+import inspect
+from datetime import datetime, timezone
+from types import SimpleNamespace
+
+import anyio
+import litellm
+import pytest
+
+import dspy
+from dspy import lm15
+from dspy._vendor.lm15._authlock import CredentialLockTimeout
+from dspy._vendor.lm15.authkit import DeviceCodeExpiredError
+from dspy._vendor.lm15.router import MissingCredentialError
+from dspy.clients.engines import AsyncLiteLLMEngine, LiteLLMEngine
+from dspy.clients.engines.litellm_errors import to_lm15_error
+from dspy.clients.errors import wrap_error
+from dspy.utils.callback import BaseCallback
+
+EXPECTED = {
+ lm15.LM15Error: dspy.LMUnexpectedError,
+ lm15.TransportError: dspy.LMTransportError,
+ lm15.LockTimeoutError: dspy.LMLockTimeoutError,
+ lm15.StreamAssemblyError: dspy.LMStreamAssemblyError,
+ lm15.ConfigurationError: dspy.LMConfigurationError,
+ lm15.NotConfiguredError: dspy.LMNotConfiguredError,
+ lm15.UnknownModelError: dspy.LMConfigurationError,
+ lm15.AmbiguousModelError: dspy.LMConfigurationError,
+ lm15.CapabilityError: dspy.LMUnsupportedFeatureError,
+ lm15.UnsupportedFeatureError: dspy.LMUnsupportedFeatureError,
+ lm15.ProviderError: dspy.LMProviderError,
+ lm15.AuthError: dspy.LMAuthError,
+ lm15.BillingError: dspy.LMBillingError,
+ lm15.RateLimitError: dspy.LMRateLimitError,
+ lm15.InvalidRequestError: dspy.LMInvalidRequestError,
+ lm15.ContextLengthError: dspy.ContextWindowExceededError,
+ lm15.UnsupportedModelError: dspy.LMUnsupportedModelError,
+ lm15.TimeoutError: dspy.LMTimeoutError,
+ lm15.ServerError: dspy.LMServerError,
+ lm15.ToolDerivationError: dspy.LMConfigurationError,
+ CredentialLockTimeout: dspy.LMLockTimeoutError,
+ DeviceCodeExpiredError: dspy.LMAuthError,
+ MissingCredentialError: dspy.LMNotConfiguredError,
+}
+
+
+class Engine:
+ def __init__(self, error=None):
+ self.error = error
+ self.calls = 0
+
+ def complete(self, request):
+ self.calls += 1
+ if self.error is not None:
+ raise self.error
+ return lm15.Response(None, request.model, lm15.Message.assistant("ok"), "stop", lm15.Usage(input_tokens=2, output_tokens=1))
+
+ def stream(self, request):
+ yield from lm15.response_to_events(self.complete(request))
+
+
+class AsyncEngine:
+ def __init__(self, sync):
+ self.sync = sync
+
+ async def complete(self, request):
+ return self.sync.complete(request)
+
+ async def stream(self, request):
+ for event in self.sync.stream(request):
+ yield event
+
+
+class Sink:
+ async def send(self, chunk):
+ pass
+
+
+async def invoke(lm, asynchronous, *args, **kwargs):
+ if asynchronous:
+ return await lm.acall(*args, **kwargs)
+ return await anyio.to_thread.run_sync(lambda: lm(*args, **kwargs))
+
+
+def test_all_exported_canonical_error_classes_have_an_explicit_policy():
+ exported = {value for name in lm15.__all__ if inspect.isclass(value := getattr(lm15, name))
+ and issubclass(value, lm15.LM15Error)}
+ assert exported <= set(EXPECTED)
+
+
[email protected]("asynchronous", [False, True])
[email protected]("streaming", [False, True])
[email protected]("source,target", EXPECTED.items(), ids=[cls.__name__ for cls in EXPECTED])
+async def test_public_projection_preserves_metadata_and_cause(source, target, asynchronous, streaming):
+ original = source("original failure", provider="fake", provider_code="code", status=429,
+ request_id="request-1", retry_after=7.0)
+ engine = Engine(original)
+ errors = []
+
+ class Callback(BaseCallback):
+ def on_lm_end(self, call_id, outputs, exception):
+ errors.append(exception)
+
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False,
+ num_retries=0, callbacks=[Callback()])
+ request = lm15.Request(model=lm.model, messages=(lm15.Message.user("hello"),))
+ with dspy.context(send_stream=Sink() if streaming else None):
+ with pytest.raises(target) as caught:
+ await invoke(lm, asynchronous, request)
+ error = caught.value
+ assert type(error) is target
+ assert error.__cause__ is original
+ assert error.message == original.message
+ assert error.model == lm.model
+ assert (error.provider, error.provider_code, error.status, error.request_id, error.retry_after) == (
+ "fake", "code", 429, "request-1", 7.0,
+ )
+ assert errors == [error]
+ assert engine.calls == 1
+ assert not lm.history
+
+
+def test_routing_lock_and_partial_diagnostics_survive():
+ partial = Engine().complete(lm15.Request(model="custom", messages=(lm15.Message.user("hi"),)))
+ for exc, fields in [
+ (lm15.AmbiguousModelError("choose", model="ambiguous", providers=("a", "b")), {"providers": ("a", "b"), "model": "ambiguous"}),
+ (lm15.UnknownModelError("unknown", rules_tried=("rule",), catalog_searched=True), {"rules_tried": ("rule",), "catalog_searched": True}),
+ (lm15.LockTimeoutError("locked", path="keys", lock_path="keys.lock"), {"path": "keys", "lock_path": "keys.lock"}),
+ (lm15.StreamAssemblyError("broken", partial=partial, part_index=3), {"partial": partial, "part_index": 3}),
+ (lm15.AuthError("denied", env_keys=("KEY",), credential_hint="log in"), {"env_keys": ("KEY",), "credential_hint": "log in"}),
+ ]:
+ projected = wrap_error(exc, model="custom")
+ for name, value in fields.items():
+ assert getattr(projected, name) == value
+ assert not isinstance(wrap_error(lm15.LockTimeoutError(), model="custom"), dspy.LMProviderError)
+
+
[email protected]("asynchronous", [False, True])
[email protected]("text", ["network invariant failed", "timeout must be positive", "credentials parser bug"])
+async def test_unexpected_engine_errors_are_not_guessed_or_retried(asynchronous, text, monkeypatch):
+ original = RuntimeError(text)
+ original.status_code = 503 # An arbitrary plugin attribute is not SDK evidence.
+ engine = Engine(original)
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False, num_retries=3)
+ monkeypatch.setattr("dspy.clients.lm._get_litellm", lambda: pytest.fail("Native errors must not load LiteLLM"))
+ with pytest.raises(dspy.LMUnexpectedError) as caught:
+ await invoke(lm, asynchronous, "hello")
+ assert caught.value.__cause__ is original
+ assert engine.calls == 1
+
+
[email protected]("error", [dspy.LMAuthError("existing"), DeprecationWarning("stop"), ImportError("missing SDK")])
+def test_already_public_and_python_control_errors_keep_identity(error):
+ assert wrap_error(error, model="custom") is error
+
+
[email protected]("asynchronous", [False, True])
+async def test_cancellation_is_not_translated(asynchronous):
+ original = asyncio.CancelledError("cancelled")
+ engine = Engine(original)
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False, num_retries=3)
+ with pytest.raises(asyncio.CancelledError):
+ await invoke(lm, asynchronous, "hello")
+ assert engine.calls == 1
+
+
[email protected]("error,target", [
+ (litellm.BadRequestError("timeout must be positive", model="wire", llm_provider="openai"), lm15.InvalidRequestError),
+ (litellm.AuthenticationError("network access denied", model="wire", llm_provider="openai"), lm15.AuthError),
+ (litellm.UnsupportedParamsError("timeout unsupported", model="wire", llm_provider="openai"), lm15.UnsupportedFeatureError),
+ (litellm.BudgetExceededError(current_cost=2, max_budget=1), lm15.BillingError),
+ (litellm.APIConnectionError("failed", model="wire", llm_provider="openai"), lm15.TransportError),
+ (litellm.Timeout("failed", model="wire", llm_provider="openai"), lm15.TimeoutError),
+])
+def test_sdk_classes_override_messages_and_status(error, target):
+ mapped = to_lm15_error(error)
+ assert isinstance(mapped, target)
+ if target in (lm15.InvalidRequestError, lm15.AuthError, lm15.UnsupportedFeatureError, lm15.BillingError):
+ assert not dspy.is_retryable_lm_error(wrap_error(mapped, model="custom"))
+
+
[email protected]("asynchronous", [False, True])
+async def test_litellm_engine_raises_canonical_errors_and_public_lm_projects_once(asynchronous, monkeypatch):
+ sdk = litellm.BadRequestError("timeout must be positive", model="wire", llm_provider="openai")
+
+ def fail(**kwargs):
+ raise sdk
+
+ async def afail(**kwargs):
+ raise sdk
+
+ monkeypatch.setattr(litellm, "completion", fail)
+ monkeypatch.setattr(litellm, "acompletion", afail)
+ request = lm15.Request(model="openai/test", messages=(lm15.Message.user("hi"),))
+ with pytest.raises(lm15.InvalidRequestError) as canonical:
+ if asynchronous:
+ await AsyncLiteLLMEngine().complete(request)
+ else:
+ LiteLLMEngine().complete(request)
+ assert canonical.value.__cause__ is sdk
+ lm = dspy.LM(request.model, engine="litellm", cache=False, num_retries=0)
+ with pytest.raises(dspy.LMInvalidRequestError) as public:
+ await invoke(lm, asynchronous, request)
+ assert isinstance(public.value.__cause__, lm15.InvalidRequestError)
+ assert public.value.__cause__.__cause__ is sdk
+ assert public.value.model == "wire"
+
+
+def test_billing_code_on_generic_429_is_not_retryable():
+ sdk = litellm.RateLimitError("quota", model="wire", llm_provider="openai")
+ sdk.body = {"error": {"code": "insufficient_quota"}}
+ canonical = to_lm15_error(sdk)
+ assert isinstance(canonical, lm15.BillingError)
+ assert canonical.status == 429
+ assert not dspy.is_retryable_lm_error(wrap_error(canonical, model="custom"))
+ sdk.llm_provider = "unrelated-provider"
+ assert isinstance(to_lm15_error(sdk), lm15.RateLimitError) # no cross-provider code guesses
+
+
[email protected]("asynchronous", [False, True])
[email protected]("error_class", [lm15.RateLimitError, lm15.TimeoutError, lm15.TransportError, lm15.ServerError, lm15.LockTimeoutError])
+async def test_only_transient_failures_retry_within_budget(asynchronous, error_class):
+ engine = Engine(error_class("transient", retry_after=0))
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False, num_retries=1)
+ with pytest.raises(dspy.LMError):
+ await invoke(lm, asynchronous, "hello")
+ assert engine.calls == 2
+
+
[email protected]("asynchronous", [False, True])
+async def test_invalid_request_controls_fail_before_engine_execution(asynchronous):
+ engine = Engine()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False)
+ request = lm15.Request(model=lm.model, messages=(lm15.Message.user("hi"),), config=lm15.Config(extensions={"n": 2}))
+ with pytest.raises(ValueError, match="execution controls"):
+ await invoke(lm, asynchronous, request)
+ assert engine.calls == 0
+
+
+def test_headers_are_case_insensitive_and_body_metadata_wins(monkeypatch):
+ import dspy.clients._http as http
+
+ class Clock(datetime):
+ @classmethod
+ def now(cls, tz=None):
+ return cls(2025, 1, 1, tzinfo=timezone.utc)
+
+ monkeypatch.setattr(http, "datetime", Clock)
+ error = litellm.RateLimitError("wait", model="wire", llm_provider="openai")
+ error.response = SimpleNamespace(status_code=429, headers={
+ "ReTrY-AfTeR": "Wed, 01 Jan 2025 00:00:09 GMT", "X-Request-ID": "header-id",
+ })
+ assert to_lm15_error(error).retry_after == 9.0
+ assert to_lm15_error(error).request_id == "header-id"
+ error.retry_after = 0.0
+ error.request_id = "body-id"
+ assert to_lm15_error(error).retry_after == 0.0
+ assert to_lm15_error(error).request_id == "body-id"
+
+
[email protected]("hint", [float("nan"), float("inf"), -1, "bad", True, object()])
+def test_invalid_retry_hint_uses_backoff_without_masking_error(hint):
+ from dspy.clients.execution import _delay
+
+ assert _delay(SimpleNamespace(retry_after=hint), 0) == 1
+ assert _delay(SimpleNamespace(retry_after=hint), 1000) == 60
+ assert _delay(SimpleNamespace(retry_after=120), 0) == 120
+
+
[email protected]("asynchronous", [False, True])
+async def test_custom_request_refusal_is_projected_before_adapter_fallback(asynchronous, monkeypatch):
+ engine = Engine()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False)
+ adapter = dspy.ChatAdapter()
+ monkeypatch.setattr(adapter, "_make_json_adapter_fallback", lambda: pytest.fail("Setup failures are not parse failures"))
+ args = (lm, {"prediction": {"type": "content", "content": "x"}}, dspy.Signature("question -> answer"), [], {"question": "hi"})
+ with pytest.raises(dspy.LMUnsupportedFeatureError) as caught:
+ if asynchronous:
+ await adapter.acall(*args)
+ else:
+ adapter(*args)
+ assert isinstance(caught.value.__cause__, lm15.UnsupportedFeatureError)
+ assert engine.calls == 0
+
+
[email protected]("code", ["auth", "billing", "rate_limit", "invalid_request", "context_length", "timeout", "server", "unsupported_model", "unsupported_feature", "not_configured", "unknown_model", "ambiguous_model", "transport", "lock_timeout", "stream_assembly", "provider"])
+def test_every_canonical_error_event_uses_the_same_projection(code):
+ from dspy._vendor.lm15.errors import error_class_for_code
+ from dspy.clients.engines.stream_guard import error_from_event
+
+ event = lm15.StreamErrorEvent(lm15.ErrorDetail(code=code, message="failure"))
+ canonical = error_from_event(event, provider="fake")
+ assert canonical.code == code
+ assert type(wrap_error(canonical, model="custom")) is EXPECTED[error_class_for_code(code)]
+
+
[email protected]("asynchronous", [False, True])
[email protected]("streaming", [False, True])
+async def test_native_http_request_id_reaches_public_error(asynchronous, streaming, monkeypatch):
+ from tests.clients.test_lm_engine_execution import DualResponse, native_transport
+
+ transport = native_transport(monkeypatch, [DualResponse(
+ status=429, body=b'{"error":{"type":"rate_limit_error","message":"wait"}}',
+ headers=[("X-Request-ID", "request-1"), ("rEtRy-AfTeR", "3")],
+ )])
+ lm = dspy.LM("openai/gpt-4o-mini", engine="lm15", cache=False, num_retries=0)
+ try:
+ with dspy.context(send_stream=Sink() if streaming else None), pytest.raises(dspy.LMRateLimitError) as caught:
+ await invoke(lm, asynchronous, "hello")
+ assert caught.value.request_id == "request-1"
+ assert caught.value.retry_after == 3.0
+ assert caught.value.status == 429
+ assert len(transport.requests) == 1
+ finally:
+ await lm.aclose()
+
+
+def test_capability_routing_failure_is_projected_without_backend_fallback(monkeypatch):
+ from dspy.clients.engines import LM15Engine
+
+ error = lm15.AuthError("denied")
+
+ def fail(*args):
+ raise error
+
+ monkeypatch.setattr(LM15Engine, "resolve", fail)
+ monkeypatch.setattr("dspy.clients.lm._get_litellm", lambda: pytest.fail("No backend switch on auth errors"))
+ with pytest.raises(dspy.LMAuthError) as caught:
+ _ = dspy.LM("openai/test").supports_function_calling
+ assert caught.value.__cause__ is error
diff --git a/tests/clients/test_engine_failure_lifecycle.py b/tests/clients/test_engine_failure_lifecycle.py
new file mode 100644
index 000000000..a119b6ecc
--- /dev/null
+++ b/tests/clients/test_engine_failure_lifecycle.py
@@ -0,0 +1,524 @@
+"""Failed streams, cancellation, and post-completion work cannot replay a generation."""
+
+import asyncio
+
+import anyio
+import pytest
+
+import dspy
+from dspy import lm15
+from dspy.clients.engines.lifecycle import aclosing_stream, closing_stream
+from dspy.utils.callback import BaseCallback
+from dspy.utils.usage_tracker import track_usage
+from tests.clients.test_engine_errors import AsyncEngine, Engine, Sink, invoke
+
+
+def events(case):
+ start = lm15.StreamStartEvent() # a missing model/id is legitimate
+ text = lm15.StreamDeltaEvent(lm15.TextDelta("partial"))
+ end = lm15.StreamEndEvent(finish_reason="stop", usage=lm15.Usage(input_tokens=2, output_tokens=1))
+ error = lm15.StreamErrorEvent(lm15.ErrorDetail(code="auth", message="denied"))
+ return {
+ "empty": [], "missing-end": [start, text], "missing-start": [text, end],
+ "duplicate-start": [start, start], "duplicate-end": [start, text, end, end],
+ "after-end": [start, text, end, text], "unknown-event": [start, object()],
+ "error-first": [error], "error-midstream": [start, text, error],
+ }[case]
+
+
+class StreamEngine(Engine):
+ def __init__(self, case):
+ super().__init__()
+ self.case = case
+ self.closed = 0
+
+ def stream(self, request):
+ self.calls += 1
+ try:
+ yield from events(self.case)
+ finally:
+ self.closed += 1
+
+
[email protected]("asynchronous", [False, True])
[email protected]("case", ["empty", "missing-end", "missing-start", "duplicate-start", "duplicate-end", "after-end", "unknown-event", "error-first", "error-midstream"])
+async def test_all_streams_are_validated_and_failures_are_not_cached(case, asynchronous):
+ engine = StreamEngine(case)
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=True, num_retries=2)
+ expected = dspy.LMAuthError if case.startswith("error-") else dspy.LMStreamAssemblyError
+ with dspy.context(send_stream=Sink()), track_usage() as tracker:
+ for _ in range(2):
+ with pytest.raises(expected) as caught:
+ await invoke(lm, asynchronous, "hello")
+ if case == "missing-end":
+ assert caught.value.partial.text == "partial"
+ assert engine.calls == 2 # no retries and no partial-success cache hits
+ assert engine.closed == 2
+ assert not lm.history
+ assert not dspy.cache.memory_cache
+ usage = tracker.get_total_tokens()
+ if case in ("duplicate-end", "after-end"):
+ assert usage[lm.model]["total_tokens"] == 6
+ else:
+ assert usage == {}
+
+
[email protected]("asynchronous", [False, True])
+async def test_transient_stream_open_failure_retries_but_failure_after_output_does_not(asynchronous):
+ class Transient(Engine):
+ def stream(self, request):
+ self.calls += 1
+ if self.calls == 1:
+ raise lm15.RateLimitError("wait", retry_after=0)
+ yield lm15.StreamStartEvent()
+ yield lm15.StreamDeltaEvent(lm15.TextDelta("ok"))
+ yield lm15.StreamEndEvent(finish_reason="stop")
+
+ engine = Transient()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False, num_retries=1)
+ with dspy.context(send_stream=Sink()):
+ assert await invoke(lm, asynchronous, "hello") == ["ok"]
+ assert engine.calls == 2
+
+ class Partial(Engine):
+ def stream(self, request):
+ self.calls += 1
+ yield lm15.StreamStartEvent()
+ yield lm15.StreamDeltaEvent(lm15.TextDelta("partial"))
+ raise lm15.RateLimitError("wait", retry_after=0)
+
+ engine = Partial()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False, num_retries=1)
+ with dspy.context(send_stream=Sink()), pytest.raises(dspy.LMRateLimitError):
+ await invoke(lm, asynchronous, "hello")
+ assert engine.calls == 1
+
+
+class BrokenClose:
+ def __init__(self, primary):
+ self.primary = primary
+ self.cleanup = RuntimeError("cleanup failure")
+ self.closed = 0
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ raise self.primary
+
+ def __aiter__(self):
+ return self
+
+ async def __anext__(self):
+ raise self.primary
+
+ def close(self):
+ self.closed += 1
+ raise self.cleanup
+
+ async def aclose(self):
+ self.close()
+
+
[email protected]("asynchronous", [False, True])
[email protected]("cancelled", [False, True])
+async def test_primary_error_and_cancellation_survive_cleanup(asynchronous, cancelled):
+ primary = asyncio.CancelledError("cancelled") if cancelled else lm15.AuthError("denied")
+ source = BrokenClose(primary)
+ if asynchronous:
+ with pytest.raises(type(primary)) as caught:
+ async with aclosing_stream(source):
+ await source.__anext__()
+ else:
+ with pytest.raises(type(primary)) as caught:
+ with closing_stream(source):
+ next(source)
+ assert caught.value is primary
+ assert primary.cleanup_errors == (source.cleanup,)
+ assert source.closed == 1
+
+
[email protected]("asynchronous", [False, True])
+async def test_cleanup_diagnostics_survive_public_projection(asynchronous):
+ source = BrokenClose(lm15.AuthError("denied"))
+
+ class Backend(Engine):
+ def stream(self, request):
+ return source
+
+ class AsyncBackend(AsyncEngine):
+ def stream(self, request):
+ return source
+
+ lm = dspy.LM("custom", engine=Backend(), async_engine=AsyncBackend(None), cache=False, num_retries=0)
+ with dspy.context(send_stream=Sink()), pytest.raises(dspy.LMAuthError) as caught:
+ await invoke(lm, asynchronous, "hello")
+ assert caught.value.__cause__ is source.primary
+ assert caught.value.cleanup_errors == (source.cleanup,)
+
+
[email protected]("asynchronous", [False, True])
[email protected]("phase", ["pricing", "cache", "output-conversion"])
+async def test_post_completion_failure_does_not_retry_and_keeps_usage(phase, asynchronous, monkeypatch):
+ import dspy.clients.execution as execution
+
+ engine = Engine()
+ errors = []
+
+ class Callback(BaseCallback):
+ def on_lm_end(self, call_id, outputs, exception):
+ errors.append(exception)
+
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=True,
+ num_retries=3, callbacks=[Callback()])
+ failure = dspy.LMRateLimitError("injected post-completion failure", retry_after=0)
+
+ def fail(*args, **kwargs):
+ raise failure
+
+ monkeypatch.setattr(execution, {"pricing": "_price_result", "cache": "_store", "output-conversion": "_result"}[phase], fail)
+ with track_usage() as tracker, pytest.raises(dspy.LMRateLimitError) as caught:
+ await invoke(lm, asynchronous, "hello")
+ assert caught.value is failure
+ assert engine.calls == 1
+ assert tracker.get_total_tokens()[lm.model]["total_tokens"] == 3
+ assert errors == [failure]
+ assert not lm.history
+ assert not dspy.cache.memory_cache
+
+
[email protected]("asynchronous", [False, True])
+async def test_completed_stream_cleanup_failure_does_not_retry_and_keeps_usage(asynchronous):
+ failure = lm15.TransportError("close failed")
+
+ class Source:
+ def __init__(self):
+ self.iterator = iter([lm15.StreamStartEvent(), lm15.StreamDeltaEvent(lm15.TextDelta("ok")),
+ lm15.StreamEndEvent(finish_reason="stop", usage=lm15.Usage(input_tokens=2, output_tokens=1))])
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ return next(self.iterator)
+
+ def __aiter__(self):
+ return self
+
+ async def __anext__(self):
+ try:
+ return next(self.iterator)
+ except StopIteration:
+ raise StopAsyncIteration from None
+
+ def close(self):
+ raise failure
+
+ async def aclose(self):
+ self.close()
+
+ class Backend(Engine):
+ def stream(self, request):
+ self.calls += 1
+ return Source()
+
+ class AsyncBackend(AsyncEngine):
+ def stream(self, request):
+ return self.sync.stream(request)
+
+ engine = Backend()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncBackend(engine), cache=True, num_retries=2)
+ with dspy.context(send_stream=Sink()), track_usage() as tracker, pytest.raises(dspy.LMTransportError):
+ await invoke(lm, asynchronous, "hello")
+ assert engine.calls == 1
+ assert tracker.get_total_tokens()[lm.model]["total_tokens"] == 3
+ assert not dspy.cache.memory_cache
+
+
[email protected]("asynchronous", [False, True])
+async def test_unnamed_tool_call_preserves_partial_usage(asynchronous):
+ class Backend(Engine):
+ def stream(self, request):
+ yield lm15.StreamStartEvent()
+ yield lm15.StreamDeltaEvent(lm15.ToolCallDelta(input="{}", id="call-1"))
+ yield lm15.StreamEndEvent(finish_reason="tool_call", usage=lm15.Usage(input_tokens=2, output_tokens=1))
+
+ engine = Backend()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=True, num_retries=2)
+ with dspy.context(send_stream=Sink()), track_usage() as tracker, pytest.raises(dspy.LMStreamAssemblyError) as caught:
+ await invoke(lm, asynchronous, "hello")
+ assert caught.value.partial is not None
+ assert tracker.get_total_tokens()[lm.model]["total_tokens"] == 3
+ assert not dspy.cache.memory_cache
+
+
[email protected]("asynchronous", [False, True])
+async def test_parser_bug_is_not_hidden_by_adapter_fallback(asynchronous):
+ class Broken(dspy.ChatAdapter):
+ def parse(self, signature, completion):
+ raise RuntimeError("parser implementation bug")
+
+ engine = Engine()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False)
+ args = (lm, {}, dspy.Signature("question -> answer"), [], {"question": "hi"})
+ with pytest.raises(RuntimeError, match="parser implementation bug"):
+ if asynchronous:
+ await Broken().acall(*args)
+ else:
+ Broken()(*args)
+ assert engine.calls == 1
+
+
[email protected]("asynchronous", [False, True])
+async def test_parse_failure_after_visible_stream_never_falls_back(asynchronous):
+ # Engine returns plain 'ok', which is not a valid ChatAdapter answer.
+ engine = Engine()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False)
+ args = (lm, {}, dspy.Signature("question -> answer"), [], {"question": "hi"})
+ adapter = dspy.ChatAdapter()
+ with dspy.context(send_stream=Sink()), pytest.raises(dspy.AdapterParseError):
+ if asynchronous:
+ await adapter.acall(*args)
+ else:
+ await anyio.to_thread.run_sync(lambda: adapter(*args))
+ assert engine.calls == 1
+
+
[email protected]("asynchronous", [False, True])
+async def test_json_schema_fallback_is_only_for_setup(asynchronous, monkeypatch):
+ import dspy.adapters.json_adapter as json_adapter
+
+ class Backend(Engine):
+ supports_response_schema = True
+ supported_params = frozenset({"response_format"})
+
+ def complete(self, request):
+ response = super().complete(request)
+ self.format = request.config.response_format
+ return lm15.Response(None, request.model, lm15.Message.assistant('{"answer":"ok"}'), "stop", response.usage)
+
+ engine = Backend()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False)
+ adapter = dspy.JSONAdapter()
+ args = (lm, {}, dspy.Signature("question -> answer"), [], {"question": "hi"})
+
+ def bad_schema(*args):
+ raise ValueError("schema cannot be represented")
+
+ monkeypatch.setattr(json_adapter, "_get_structured_outputs_response_format", bad_schema)
+ result = await adapter.acall(*args) if asynchronous else adapter(*args)
+ assert result == [{"answer": "ok"}]
+ assert engine.calls == 1
+ assert engine.format == {"type": "json_object"}
+
+ def bug(*args):
+ raise RuntimeError("schema builder bug")
+
+ monkeypatch.setattr(json_adapter, "_get_structured_outputs_response_format", bug)
+ with pytest.raises(RuntimeError, match="schema builder bug"):
+ if asynchronous:
+ await adapter.acall(*args)
+ else:
+ adapter(*args)
+ assert engine.calls == 1
+
+
[email protected]("asynchronous", [False, True])
+async def test_materializing_backend_keeps_completed_usage_on_cleanup_failure(asynchronous):
+ from dspy._vendor.lm15.result import amaterialize_response, materialize_response
+
+ class Source(BrokenClose):
+ def __init__(self):
+ super().__init__(None)
+ response = lm15.Response(None, "custom", lm15.Message.assistant("ok"), "stop", lm15.Usage(input_tokens=2, output_tokens=1))
+ self.iterator = iter(lm15.response_to_events(response))
+
+ def __next__(self):
+ return next(self.iterator)
+
+ async def __anext__(self):
+ try:
+ return next(self.iterator)
+ except StopIteration:
+ raise StopAsyncIteration from None
+
+ class Backend(Engine):
+ def source(self):
+ self.calls += 1
+ return Source()
+
+ def complete(self, request):
+ return materialize_response(self.source(), request)
+
+ class AsyncBackend(AsyncEngine):
+ async def complete(self, request):
+ return await amaterialize_response(self.sync.source(), request)
+
+ engine = Backend()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncBackend(engine), cache=True, num_retries=2)
+ with track_usage() as tracker, pytest.raises(dspy.LMStreamAssemblyError) as caught:
+ await invoke(lm, asynchronous, "hello")
+ assert caught.value.partial.text == "ok"
+ assert tracker.get_total_tokens()[lm.model]["total_tokens"] == 3
+ assert engine.calls == 1
+ assert not dspy.cache.memory_cache
+
+
[email protected]("asynchronous", [False, True])
+async def test_compatibility_stream_keeps_usage_if_sdk_close_fails(asynchronous, monkeypatch):
+ import litellm
+
+ class Source:
+ def __init__(self):
+ self.sent = False
+
+ def __aiter__(self):
+ return self
+
+ async def __anext__(self):
+ if self.sent:
+ raise StopAsyncIteration
+ self.sent = True
+ return {"choices": [{"delta": {"content": "ok"}, "finish_reason": "stop"}]}
+
+ async def aclose(self):
+ raise lm15.TransportError("cleanup failed")
+
+ calls = []
+
+ async def complete(**kwargs):
+ calls.append(kwargs)
+ return Source()
+
+ raw = litellm.ModelResponse(model="fake", choices=[{"message": {"role": "assistant", "content": "ok"}}],
+ usage={"prompt_tokens": 2, "completion_tokens": 1, "total_tokens": 3})
+ monkeypatch.setattr(litellm, "acompletion", complete)
+ monkeypatch.setattr(litellm, "stream_chunk_builder", lambda chunks: raw)
+ lm = dspy.LM("openai/fake", engine="litellm", cache=True, num_retries=2)
+ with dspy.context(send_stream=Sink()), track_usage() as tracker, pytest.raises(dspy.LMTransportError):
+ await invoke(lm, asynchronous, "hello")
+ assert len(calls) == 1
+ assert tracker.get_total_tokens()[lm.model]["total_tokens"] == 3
+ assert not dspy.cache.memory_cache
+
+
[email protected]("adapter,module,text", [
+ (dspy.ChatAdapter(), "dspy.adapters.chat_adapter", '[[ ## answer ## ]]\n{"value": 1}'),
+ (dspy.JSONAdapter(), "dspy.adapters.json_adapter", '{"answer":{"value":1}}'),
+ (dspy.XMLAdapter(), "dspy.adapters.xml_adapter", '<answer>{"value":1}</answer>'),
+])
+def test_unexpected_field_parser_errors_are_not_labelled_as_model_output_errors(adapter, module, text, monkeypatch):
+ def fail(*args):
+ raise RuntimeError("validator implementation bug")
+
+ monkeypatch.setattr(module + ".parse_value", fail)
+ with pytest.raises(RuntimeError, match="validator implementation bug"):
+ adapter.parse(dspy.Signature("question -> answer"), text)
+
+
[email protected]("asynchronous", [False, True])
+async def test_usage_reporting_failure_does_not_replace_primary_error(asynchronous):
+ reporting_error = RuntimeError("tracker failed")
+
+ class Tracker:
+ def add_usage(self, model, usage):
+ raise reporting_error
+
+ class Backend(Engine):
+ def complete(self, request):
+ if self.calls:
+ self.error = lm15.AuthError("denied")
+ return super().complete(request)
+
+ engine = Backend()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False, num_retries=0)
+ with dspy.context(usage_tracker=Tracker()), pytest.raises(dspy.LMAuthError) as caught:
+ await invoke(lm, asynchronous, "hello", n=2)
+ assert caught.value.usage_errors == (reporting_error,)
+ assert isinstance(caught.value.__cause__, lm15.AuthError)
+ assert engine.calls == 2
+
+
[email protected]("asynchronous", [False, True])
+async def test_direct_native_engine_stream_is_also_guarded(asynchronous, monkeypatch):
+ from types import SimpleNamespace
+
+ from dspy.clients.engines import AsyncLM15Engine, LM15Engine
+
+ class Provider:
+ provider = "fake"
+
+ def stream(self, request):
+ return iter([lm15.StreamStartEvent(), lm15.StreamDeltaEvent(lm15.TextDelta("partial"))])
+
+ request = lm15.Request(model="custom", messages=(lm15.Message.user("hi"),))
+ if asynchronous:
+ async def source(request):
+ for event in Provider().stream(request):
+ yield event
+
+ engine = AsyncLM15Engine()
+ provider = SimpleNamespace(provider="fake", stream=source)
+ monkeypatch.setattr(engine, "_target", lambda *args, **kwargs: (provider, request))
+ with pytest.raises(lm15.StreamAssemblyError):
+ _ = [event async for event in engine.stream(request)]
+ else:
+ engine = LM15Engine()
+ monkeypatch.setattr(engine, "_target", lambda *args, **kwargs: (Provider(), request))
+ with pytest.raises(lm15.StreamAssemblyError):
+ list(engine.stream(request))
+
+
+async def test_stream_fallback_progress_is_isolated_across_concurrent_calls():
+ emitted, release = asyncio.Event(), asyncio.Event()
+
+ class Backend(AsyncEngine):
+ calls = 0
+
+ async def complete(self, request):
+ self.calls += 1
+ return lm15.Response(None, request.model, lm15.Message.assistant('{"answer":"ok"}'), "stop", lm15.Usage())
+
+ async def stream(self, request):
+ yield lm15.StreamStartEvent()
+ yield lm15.StreamDeltaEvent(lm15.TextDelta('{"answer":"ok"}'))
+ await release.wait()
+ yield lm15.StreamEndEvent(finish_reason="stop")
+
+ class SignallingSink(Sink):
+ async def send(self, chunk):
+ emitted.set()
+
+ backend = Backend(None)
+ lm = dspy.LM("custom", engine=Engine(), async_engine=backend, cache=False)
+ adapter = dspy.ChatAdapter()
+ args = (lm, {}, dspy.Signature("question -> answer"), [], {"question": "hi"})
+
+ async def streaming():
+ with dspy.context(send_stream=SignallingSink()):
+ with pytest.raises(dspy.AdapterParseError):
+ await adapter.acall(*args)
+
+ task = asyncio.create_task(streaming())
+ try:
+ await asyncio.wait_for(emitted.wait(), 5)
+ assert await adapter.acall(*args) == [{"answer": "ok"}]
+ assert backend.calls == 2 # non-streaming Chat parse failure still gets one JSON fallback
+ finally:
+ release.set()
+ await task
diff --git a/tests/clients/test_lazy_litellm_import.py b/tests/clients/test_lazy_litellm_import.py
index 8d3b7b59e..9b6047fac 100644
--- a/tests/clients/test_lazy_litellm_import.py
+++ b/tests/clients/test_lazy_litellm_import.py
@@ -40,7 +40,7 @@ def test_lm_litellm_use_raises_helpful_error_without_litellm(monkeypatch):
_hide_litellm(monkeypatch)
with pytest.raises(ImportError) as exc_info:
- _ = dspy.LM("openai/gpt-4o-mini").supports_function_calling
+ _ = dspy.LM("openai/gpt-4o-mini", engine="litellm").supports_function_calling
msg = str(exc_info.value)
assert "[litellm]" in msg
diff --git a/tests/clients/test_legacy_lm_deprecation.py b/tests/clients/test_legacy_lm_deprecation.py
new file mode 100644
index 000000000..fd170fe04
--- /dev/null
+++ b/tests/clients/test_legacy_lm_deprecation.py
@@ -0,0 +1,253 @@
+"""The 3.5 cutoff applies to legacy integration and explicit wrappers alike."""
+
+import inspect
+import warnings
+from pathlib import Path
+
+import pytest
+
+import dspy
+from dspy.clients.engines import AsyncLegacyEngine, LegacyEngine
+from dspy.lm15 import Message, Request, Response, Usage
+from dspy.utils.callback import BaseCallback
+from dspy.utils.dummies import DummyLM
+
+
+class LegacyLM(dspy.BaseLM):
+ forward_contract = "legacy"
+
+ def forward(self, prompt=None, messages=None, **kwargs):
+ self.calls = getattr(self, "calls", 0) + 1
+ return {
+ "model": self.model,
+ "choices": [{"message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}],
+ "usage": {},
+ }
+
+ async def aforward(self, **kwargs):
+ return self.forward(**kwargs)
+
+
+def migration_warnings(recorded):
+ return [item for item in recorded if "https://dspy.ai/community/normalized-lm-api-migration/" in str(item.message)]
+
+
[email protected]("asynchronous", [False, True])
[email protected]("typed", [False, True])
[email protected]("callbacks", [False, True])
+async def test_legacy_warning_identifies_the_call_site(asynchronous, typed, callbacks):
+ lm = LegacyLM("custom", callbacks=[BaseCallback()] if callbacks else None)
+ argument = Request(model=lm.model, messages=(Message.user("hello"),)) if typed else "hello"
+
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ if asynchronous:
+ line = inspect.currentframe().f_lineno + 1
+ result = await lm.acall(argument)
+ else:
+ line = inspect.currentframe().f_lineno + 1
+ result = lm(argument)
+
+ found = migration_warnings(recorded)
+ assert len(found) == 1
+ assert found[0].category is DeprecationWarning
+ assert Path(found[0].filename).resolve() == Path(__file__).resolve()
+ assert found[0].lineno == line
+ message = str(found[0].message)
+ assert "complete(Request) -> Response" in message
+ assert "supported throughout DSPy 3.4" in message
+ assert "scheduled for removal in 3.5" in message
+ assert "along with LegacyEngine and AsyncLegacyEngine" in message
+ assert "https://dspy.ai/community/normalized-lm-api-migration/#custom-engines-and-legacy-plugins" in message
+ if typed:
+ assert isinstance(result, Response)
+ assert result.text == "ok"
+ else:
+ assert result == ["ok"]
+ assert lm.calls == 1
+ assert len(lm.history) == 1
+
+
+def test_python_warning_filters_control_repetition_and_copies():
+ lm = LegacyLM("custom")
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ lm("hello")
+ lm("hello")
+ copied = lm.copy()
+ copied("hello")
+ assert len(migration_warnings(recorded)) == 3
+ assert not hasattr(lm, "_warned_legacy_engine")
+ assert not hasattr(copied, "_warned_legacy_engine")
+
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("default", DeprecationWarning)
+ for instance in (lm, lm, copied):
+ instance("hello")
+ assert len(migration_warnings(recorded)) == 1
+
+
[email protected]("asynchronous", [False, True])
[email protected]("mode", ["direct", "ordinary", "typed"])
+async def test_explicit_legacy_engines_warn_at_construction_not_each_execution(asynchronous, mode):
+ plugin = LegacyLM("custom")
+ request = Request(model=plugin.model, messages=(Message.user("hello"),))
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ sync_line = inspect.currentframe().f_lineno + 1
+ sync_engine = LegacyEngine(plugin)
+ async_line = inspect.currentframe().f_lineno + 1
+ async_engine = AsyncLegacyEngine(plugin)
+ if mode != "direct":
+ lm = dspy.LM(plugin.model, engine=sync_engine, async_engine=async_engine, cache=False, num_retries=0)
+ argument = request if mode == "typed" else "hello"
+ result = await lm.acall(argument) if asynchronous else lm(argument)
+ if mode == "typed":
+ assert isinstance(result, Response)
+ assert result.text == "ok"
+ else:
+ assert result == ["ok"]
+ else:
+ result = await async_engine.complete(request) if asynchronous else sync_engine.complete(request)
+ assert result.text == "ok"
+ found = migration_warnings(recorded)
+ assert len(found) == 2
+ assert [item.lineno for item in found] == [sync_line, async_line]
+ assert all(Path(item.filename).resolve() == Path(__file__).resolve() for item in found)
+ assert all("LegacyEngine and AsyncLegacyEngine are deprecated" in str(item.message) for item in found)
+ assert all("scheduled for removal in DSPy 3.5" in str(item.message) for item in found)
+ assert plugin.calls == 1
+
+
[email protected]("asynchronous", [False, True])
[email protected]("base", [dspy.LM, DummyLM])
+async def test_builtin_forward_overrides_warn(asynchronous, base):
+ class OverriddenLM(base):
+ forward = LegacyLM.forward
+ aforward = LegacyLM.aforward
+
+ lm = OverriddenLM([]) if base is DummyLM else OverriddenLM("custom", cache=False)
+ with pytest.warns(DeprecationWarning, match="scheduled for removal in 3.5"):
+ result = await lm.acall("hello") if asynchronous else lm("hello")
+ assert result == ["ok"]
+ assert lm.calls == 1
+
+
[email protected]("asynchronous", [False, True])
+async def test_engine_integrations_do_not_warn(asynchronous):
+ class Engine:
+ def complete(self, request):
+ return Response(None, request.model, Message.assistant("ok"), "stop", Usage())
+
+ class AsyncEngine:
+ async def complete(self, request):
+ return Engine().complete(request)
+
+ models = [DummyLM([{"answer": "ok"}]), dspy.LM("custom", engine=Engine(), async_engine=AsyncEngine(), cache=False)]
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ for lm in models:
+ if asynchronous:
+ await lm.acall("hello")
+ else:
+ lm("hello")
+ assert not migration_warnings(recorded)
+
+
[email protected]("asynchronous", [False, True])
+async def test_removed_typed_contract_does_not_receive_legacy_warning(asynchronous):
+ class RemovedLM(LegacyLM):
+ forward_contract = "typed_lm"
+
+ lm = RemovedLM("custom")
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with pytest.raises(TypeError, match="typed_lm contract was removed"):
+ if asynchronous:
+ await lm.acall("hello")
+ else:
+ lm("hello")
+ assert not migration_warnings(recorded)
+ assert not hasattr(lm, "calls")
+
+
[email protected]("engine_class", [LegacyEngine, AsyncLegacyEngine])
+def test_wrapper_warning_as_error_stops_at_construction(engine_class):
+ plugin = LegacyLM("custom")
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", DeprecationWarning)
+ with pytest.raises(DeprecationWarning, match="LegacyEngine and AsyncLegacyEngine are deprecated"):
+ engine_class(plugin)
+ assert not hasattr(plugin, "calls")
+
+
[email protected]("asynchronous", [False, True])
+async def test_program_warns_about_legacy_lm_not_internal_messages(asynchronous):
+ class ProgramLM(LegacyLM):
+ def forward(self, **kwargs):
+ raw = super().forward(**kwargs)
+ raw["choices"][0]["message"]["content"] = "[[ ## answer ## ]]\nok"
+ return raw
+
+ lm = ProgramLM("custom")
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with dspy.context(lm=lm):
+ program = dspy.Predict("question -> answer")
+ result = await program.acall(question="hello") if asynchronous else program(question="hello")
+ assert result.answer == "ok"
+ found = migration_warnings(recorded)
+ assert len(found) == 1
+ assert "Implementing custom LMs" in str(found[0].message)
+ assert lm.calls == 1
+
+
[email protected]("asynchronous", [False, True])
[email protected]("adapter_class", [dspy.ChatAdapter, dspy.JSONAdapter])
+async def test_warning_as_error_does_not_trigger_adapter_fallback(asynchronous, adapter_class):
+ starts = []
+
+ class Callback(BaseCallback):
+ def on_lm_start(self, call_id, instance, inputs):
+ starts.append(call_id)
+
+ class SchemaLM(LegacyLM):
+ supported_params = frozenset({"response_format"})
+ supports_response_schema = True
+
+ lm = SchemaLM("custom", callbacks=[Callback()])
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", DeprecationWarning)
+ with dspy.context(lm=lm, adapter=adapter_class()):
+ program = dspy.Predict("question -> answer")
+ with pytest.raises(DeprecationWarning, match="Implementing custom LMs"):
+ if asynchronous:
+ await program.acall(question="hello")
+ else:
+ program(question="hello")
+ assert len(starts) == 1
+ assert not hasattr(lm, "calls")
+
+
[email protected]("asynchronous", [False, True])
+async def test_two_step_extraction_preserves_warning_as_error(asynchronous):
+ extraction = LegacyLM("custom")
+ main = DummyLM([{"answer": "ok"}])
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", DeprecationWarning)
+ with dspy.context(lm=main, adapter=dspy.TwoStepAdapter(extraction)):
+ program = dspy.Predict("question -> answer")
+ with pytest.raises(DeprecationWarning, match="Implementing custom LMs"):
+ if asynchronous:
+ await program.acall(question="hello")
+ else:
+ program(question="hello")
+ assert not hasattr(extraction, "calls")
diff --git a/tests/clients/test_lm.py b/tests/clients/test_lm.py
index 658834809..f863b52e5 100644
--- a/tests/clients/test_lm.py
+++ b/tests/clients/test_lm.py
@@ -51,7 +51,7 @@ def test_chat_lms_can_be_queried(litellm_test_server):
expected_response = ["Hi!"]
openai_lm = dspy.LM(
- model="openai/dspy-test-model",
+ engine="litellm", model="openai/dspy-test-model",
api_base=api_base,
api_key="fakekey",
model_type="chat",
@@ -59,7 +59,7 @@ def test_chat_lms_can_be_queried(litellm_test_server):
assert openai_lm("openai query") == expected_response
azure_openai_lm = dspy.LM(
- model="azure/dspy-test-model",
+ engine="litellm", model="azure/dspy-test-model",
api_base=api_base,
api_key="fakekey",
model_type="chat",
@@ -79,7 +79,7 @@ def test_dspy_cache(litellm_test_server, tmp_path):
cache = dspy.cache
lm = dspy.LM(
- model="openai/dspy-test-model",
+ engine="litellm", model="openai/dspy-test-model",
api_base=api_base,
api_key="fakekey",
model_type="text",
@@ -121,7 +121,7 @@ def test_disabled_cache_skips_cache_key(monkeypatch):
monkeypatch.setattr(litellm, "completion", fake_completion)
- lm = dspy.LM("dummy", model_type="chat")
+ lm = dspy.LM("dummy", engine="litellm", model_type="chat")
lm(messages=[{"role": "user", "content": "Hello"}])
cache_key_spy.assert_not_called()
@@ -151,7 +151,7 @@ def test_rollout_id_bypasses_cache(monkeypatch, tmp_path):
disk_cache_dir=tmp_path / ".disk_cache",
)
- lm = dspy.LM(model="openai/dspy-test-model", model_type="chat")
+ lm = dspy.LM(engine="litellm", model="openai/dspy-test-model", model_type="chat")
with track_usage() as usage_tracker:
lm(messages=[{"role": "user", "content": "Query"}], rollout_id=1)
@@ -188,7 +188,7 @@ def test_zero_temperature_rollout_warns_once(monkeypatch):
monkeypatch.setattr(litellm, "completion", fake_completion)
- lm = dspy.LM(model="openai/dspy-test-model", model_type="chat", temperature=0)
+ lm = dspy.LM(engine="litellm", model="openai/dspy-test-model", model_type="chat", temperature=0)
with pytest.warns(UserWarning, match="rollout_id has no effect"):
lm("Query", rollout_id=1)
with warnings.catch_warnings(record=True) as record:
@@ -209,7 +209,7 @@ def test_rollout_id_with_default_temperature_does_not_warn(monkeypatch):
with warnings.catch_warnings(record=True) as record:
warnings.simplefilter("always")
- lm = dspy.LM(model="openai/gpt-5-nano", model_type="chat", rollout_id=1)
+ lm = dspy.LM(engine="litellm", model="openai/gpt-5-nano", model_type="chat", rollout_id=1)
lm("Query")
assert len(record) == 0
@@ -219,7 +219,7 @@ def test_text_lms_can_be_queried(litellm_test_server):
expected_response = ["Hi!"]
openai_lm = dspy.LM(
- model="openai/dspy-test-model",
+ engine="litellm", model="openai/dspy-test-model",
api_base=api_base,
api_key="fakekey",
model_type="text",
@@ -227,7 +227,7 @@ def test_text_lms_can_be_queried(litellm_test_server):
assert openai_lm("openai query") == expected_response
azure_openai_lm = dspy.LM(
- model="azure/dspy-test-model",
+ engine="litellm", model="azure/dspy-test-model",
api_base=api_base,
api_key="fakekey",
model_type="text",
@@ -244,7 +244,7 @@ def test_lm_calls_support_callables(litellm_test_server):
return None
lm_with_callable = dspy.LM(
- model="openai/dspy-test-model",
+ engine="litellm", model="openai/dspy-test-model",
api_base=api_base,
api_key="fakekey",
azure_ad_token_provider=azure_ad_token_provider,
@@ -268,7 +268,7 @@ def test_lm_calls_support_pydantic_models(litellm_test_server):
response: str
lm = dspy.LM(
- model="openai/dspy-test-model",
+ engine="litellm", model="openai/dspy-test-model",
api_base=api_base,
api_key="fakekey",
response_format=ResponseFormat,
@@ -315,7 +315,7 @@ def test_lm_wraps_unknown_boundary_error_as_unexpected_error():
def test_lm_preserves_existing_lm_error_without_self_cause():
error = dspy.LMRateLimitError("rate limited", model="openai/gpt-4o-mini")
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
with mock.patch("dspy.clients.lm.litellm_completion", side_effect=error):
with pytest.raises(dspy.LMRateLimitError) as exc_info:
@@ -328,7 +328,7 @@ def test_lm_preserves_existing_lm_error_without_self_cause():
@pytest.mark.asyncio
async def test_lm_preserves_existing_lm_error_without_self_cause_async():
error = dspy.LMRateLimitError("rate limited", model="openai/gpt-4o-mini")
- lm = dspy.LM("openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
with mock.patch("dspy.clients.lm.alitellm_completion", side_effect=error):
with pytest.raises(dspy.LMRateLimitError) as exc_info:
@@ -339,11 +339,12 @@ async def test_lm_preserves_existing_lm_error_without_self_cause_async():
def test_retry_number_set_correctly():
- lm = dspy.LM("openai/gpt-4o-mini", num_retries=3)
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", num_retries=3)
with mock.patch("litellm.completion") as mock_completion:
lm("query")
- assert mock_completion.call_args.kwargs["num_retries"] == 3
+ # DSPy owns retries; every individual backend attempt disables them.
+ assert mock_completion.call_args.kwargs["num_retries"] == 0
def test_retry_made_on_system_errors():
@@ -363,7 +364,7 @@ def test_retry_made_on_system_errors():
kwargs["sleep"] = lambda _: None
return original_retrying(*args, **kwargs)
- lm = dspy.LM(model="openai/gpt-4o-mini", max_tokens=250, num_retries=3)
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini", max_tokens=250, num_retries=3)
with (
mock.patch("tenacity.Retrying", side_effect=immediate_retrying),
mock.patch.object(litellm.OpenAIChatCompletion, "completion", side_effect=mock_create),
@@ -464,86 +465,18 @@ def test_base_lm_init_uses_lm_defaults_and_isolates_callback_list():
assert lm.callbacks is not callbacks
-def test_base_lm_forward_contract_defaults_to_legacy():
- class CustomLM(dspy.BaseLM):
- pass
-
- lm = CustomLM("custom-model")
- assert lm._get_forward_contract() == "legacy"
- assert not lm._declares_forward_contract()
-def test_base_lm_forward_contract_accepts_explicit_values():
- class LegacyLM(dspy.BaseLM):
- forward_contract = "legacy"
- class TypedLM(dspy.BaseLM):
- forward_contract = "typed_lm"
- assert LegacyLM("custom-model")._get_forward_contract() == "legacy"
- assert LegacyLM("custom-model")._declares_forward_contract()
- assert TypedLM("custom-model")._get_forward_contract() == "typed_lm"
- assert TypedLM("custom-model")._declares_forward_contract()
-def test_base_lm_forward_contract_rejects_unknown_values():
- class CustomLM(dspy.BaseLM):
- forward_contract = "normalized"
- with pytest.raises(ValueError, match="forward_contract must be 'legacy' or 'typed_lm'"):
- CustomLM("custom-model")._get_forward_contract()
-def test_base_lm_validates_typed_lm_response():
- lm = dspy.BaseLM("custom-model")
- response = dspy.LMResponse.from_text("ok", model="custom-model")
- assert lm._validate_typed_lm_response(response) is response
- with pytest.raises(TypeError, match=r"requires forward\(request\).*dspy.LMResponse"):
- lm._validate_typed_lm_response(["ok"])
-
-
-def test_base_lm_warns_when_inherited_legacy_forward_returns_lm_response():
- class CustomLM(dspy.BaseLM):
- pass
-
- lm = CustomLM("custom-model")
- response = dspy.LMResponse.from_text("ok", model="custom-model")
-
- with pytest.warns(DeprecationWarning, match="default legacy forward_contract"):
- assert lm._validate_legacy_lm_response(response) is response
-
- assert lm._validate_legacy_lm_response(["ok"]) is None
-
-
-def test_base_lm_errors_when_explicit_legacy_forward_returns_lm_response():
- class CustomLM(dspy.BaseLM):
- forward_contract = "legacy"
-
- lm = CustomLM("custom-model")
- response = dspy.LMResponse.from_text("ok", model="custom-model")
-
- with pytest.raises(TypeError, match=r"forward_contract='legacy'.*got dspy.LMResponse"):
- lm._validate_legacy_lm_response(response)
-
-
-def test_base_lm_inherited_legacy_forward_returning_lm_response_errors_on_direct_call():
- class CustomLM(dspy.BaseLM):
- def forward(self, prompt=None, messages=None, **kwargs):
- return dspy.LMResponse.from_text(
- "ok",
- model="custom-model",
- usage={"prompt_tokens": 1, "completion_tokens": 2, "total_tokens": 3},
- )
-
- lm = CustomLM("custom-model")
-
- with pytest.raises(TypeError, match="legacy direct path"):
- lm("Query")
-
- assert len(lm.history) == 0
# BaseLM direct-call compatibility tests.
@@ -566,91 +499,12 @@ def test_base_lm_default_call_keeps_legacy_outputs():
assert CustomLM("custom-model")("Query") == ["Hi!"]
-def test_base_lm_experimental_call_returns_lm_response_through_legacy_bridge():
- class CustomLM(dspy.BaseLM):
- def __init__(self, *args, **kwargs):
- super().__init__(*args, **kwargs)
- self.seen = None
-
- def forward(self, prompt=None, messages=None, **kwargs):
- self.seen = {"prompt": prompt, "messages": messages, "kwargs": kwargs}
- return ModelResponse(
- choices=[Choices(message=Message(role="assistant", content="Hi!"), finish_reason="stop")],
- usage={"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
- model="custom-model",
- )
-
- lm = CustomLM("custom-model", temperature=0.2)
- with dspy.context(experimental=True):
- response = lm("Query", rollout_id=7)
-
- assert isinstance(response, dspy.LMResponse)
- assert response.text == "Hi!"
- assert response.output.finish_reason == "stop"
- assert lm.seen["prompt"] == "Query"
- assert lm.seen["messages"] is None
- assert lm.seen["kwargs"]["temperature"] == 0.2
- assert lm.seen["kwargs"]["cache"] is True
- assert lm.seen["kwargs"]["rollout_id"] == 7
-
-
-def test_base_lm_explicit_lm_request_returns_lm_response_without_experimental():
- class CustomLM(dspy.BaseLM):
- def forward(self, prompt=None, messages=None, **kwargs):
- return ModelResponse(
- choices=[Choices(message=Message(role="assistant", content="Hi!"))],
- usage={},
- model="custom-model",
- )
-
- request = dspy.LMRequest.from_call(model="custom-model", prompt="Query")
- response = CustomLM("custom-model")(request)
- assert isinstance(response, dspy.LMResponse)
- assert response.text == "Hi!"
-def test_base_lm_legacy_bridge_records_typed_history_and_usage_once():
- class CustomLM(dspy.BaseLM):
- def forward(self, prompt=None, messages=None, **kwargs):
- return ModelResponse(
- choices=[Choices(message=Message(role="assistant", content="Hi!"))],
- usage={"prompt_tokens": 1, "completion_tokens": 2, "total_tokens": 3},
- model="custom-model",
- )
- lm = CustomLM("custom-model")
- request = dspy.LMRequest.from_call(model="custom-model", prompt="Query")
-
- with track_usage() as usage_tracker:
- response = lm(request)
-
- assert isinstance(response, dspy.LMResponse)
- assert response.text == "Hi!"
- assert len(lm.history) == 1
- assert lm.history[0].request == request
- assert lm.history[0].response == response
- total_usage = usage_tracker.get_total_tokens()["custom-model"]
- assert total_usage["prompt_tokens"] == 1
- assert total_usage["completion_tokens"] == 2
- assert total_usage["total_tokens"] == 3
-
-
-def test_base_lm_typed_forward_contract_uses_lm_request():
- class CustomLM(dspy.BaseLM):
- forward_contract = "typed_lm"
-
- def forward(self, request):
- assert isinstance(request, dspy.LMRequest)
- return dspy.LMResponse.from_text(f"model={request.model}; text={request.messages[0].text}")
- lm = CustomLM("custom-model")
- assert lm("Query") == ["model=custom-model; text=Query"]
- with dspy.context(experimental=True):
- response = lm("Query")
- assert isinstance(response, dspy.LMResponse)
- assert response.text == "model=custom-model; text=Query"
def test_base_lm_typed_forward_contract_rejects_non_lm_response_at_call_time():
@@ -660,18 +514,10 @@ def test_base_lm_typed_forward_contract_rejects_non_lm_response_at_call_time():
def forward(self, request):
return ["not typed"]
- with pytest.raises(TypeError, match="forward_contract='typed_lm'"):
+ with pytest.raises(TypeError, match="typed_lm contract was removed"):
CustomLM("custom-model")("Query")
-def test_base_lm_request_call_rejects_mixed_inputs():
- class CustomLM(dspy.BaseLM):
- def forward(self, prompt=None, messages=None, **kwargs):
- raise AssertionError("forward should not be called")
-
- request = dspy.LMRequest.from_call(model="custom-model", prompt="Query")
- with pytest.raises(ValueError, match="Pass either an LMRequest or direct-call inputs"):
- CustomLM("custom-model")(request, "extra")
def _model_response(text: str) -> ModelResponse:
@@ -682,165 +528,16 @@ def _model_response(text: str) -> ModelResponse:
)
-class _TypedContractLM(dspy.BaseLM):
- """Test double that records normalized requests received through the typed LM contract."""
-
- forward_contract = "typed_lm"
-
- def __init__(self, *args, outputs: list[str], **kwargs):
- super().__init__(*args, **kwargs)
- self.outputs = outputs
- self.requests = []
-
- def forward(self, request):
- assert isinstance(request, dspy.LMRequest)
- self.requests.append(request)
- return dspy.LMResponse.from_text(self.outputs[len(self.requests) - 1], model=request.model)
-
-
-def _direct_lm_case(lm_kind: str, outputs: list[str]):
- """Return a direct-call test double and helpers for inspecting normalized messages."""
- if lm_kind == "current_lm":
- patcher = mock.patch(
- "dspy.clients.lm.litellm_completion",
- side_effect=[_model_response(output) for output in outputs],
- )
- completion = patcher.start()
- lm = dspy.LM("custom-model", cache=False)
-
- def get_messages(index: int) -> list[dict[str, object]]:
- return completion.call_args_list[index].kwargs["request"]["messages"]
-
- def get_request(index: int):
- return None
-
- return lm, get_messages, get_request, patcher
-
- if lm_kind == "typed_lm":
- lm = _TypedContractLM("custom-model", outputs=outputs)
-
- def get_messages(index: int) -> list[dict[str, object]]:
- from dspy.clients.openai_format import to_openai_chat_request
-
- return to_openai_chat_request(lm.requests[index])["messages"]
-
- def get_request(index: int):
- return lm.requests[index]
- return lm, get_messages, get_request, None
-
- raise ValueError(f"Unknown lm_kind: {lm_kind}")
-
-
[email protected]("lm_kind", ["current_lm", "typed_lm"])
-def test_base_lm_experimental_direct_messages_support_system_user_and_assistant_turns(lm_kind):
- lm, get_messages, get_request, patcher = _direct_lm_case(lm_kind, ["Five-word answer."])
- try:
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Be concise."),
- dspy.User("What is DSPy?"),
- dspy.Assistant("DSPy is a framework for programming LM pipelines."),
- dspy.User("Say that in five words."),
- temperature=0.2,
- )
- finally:
- if patcher is not None:
- patcher.stop()
-
- assert isinstance(response, dspy.LMResponse)
- assert response.text == "Five-word answer."
- assert get_messages(0) == [
- {"role": "system", "content": "Be concise."},
- {"role": "user", "content": "What is DSPy?"},
- {"role": "assistant", "content": "DSPy is a framework for programming LM pipelines."},
- {"role": "user", "content": "Say that in five words."},
- ]
- if lm_kind == "typed_lm":
- assert get_request(0).config.temperature == 0.2
[email protected]("lm_kind", ["current_lm", "typed_lm"])
-def test_base_lm_experimental_direct_messages_support_tool_call_transcripts(lm_kind):
- lm, get_messages, get_request, patcher = _direct_lm_case(lm_kind, ["It is 22 C in Paris."])
- try:
- with dspy.context(experimental=True):
- response = lm(
- dspy.User("What is the weather in Paris?"),
- dspy.Assistant(dspy.ToolCall(id="call_1", name="get_weather", args={"city": "Paris"})),
- dspy.ToolResult('{"temperature": "22 C"}', call_id="call_1", name="get_weather"),
- dspy.User("Summarize the result."),
- )
- finally:
- if patcher is not None:
- patcher.stop()
- assert response.text == "It is 22 C in Paris."
- assert get_messages(0) == [
- {"role": "user", "content": "What is the weather in Paris?"},
- {
- "role": "assistant",
- "content": None,
- "tool_calls": [
- {
- "type": "function",
- "function": {"name": "get_weather", "arguments": json.dumps({"city": "Paris"})},
- "id": "call_1",
- }
- ],
- },
- {"role": "tool", "content": '{"temperature": "22 C"}', "tool_call_id": "call_1", "name": "get_weather"},
- {"role": "user", "content": "Summarize the result."},
- ]
- if lm_kind == "typed_lm":
- assert isinstance(get_request(0), dspy.LMRequest)
[email protected]("lm_kind", ["current_lm", "typed_lm"])
-def test_base_lm_experimental_direct_messages_can_reuse_lm_response_as_assistant_turn(lm_kind):
- lm, get_messages, get_request, patcher = _direct_lm_case(
- lm_kind,
- ["DSPy programs LM pipelines.", "DSPy programs pipelines."],
- )
- try:
- with dspy.context(experimental=True):
- first = lm("Explain DSPy in one sentence.")
- follow_up = lm(
- dspy.User("Explain DSPy in one sentence."),
- first,
- dspy.User("Now make it even shorter."),
- )
- finally:
- if patcher is not None:
- patcher.stop()
-
- assert first.text == "DSPy programs LM pipelines."
- assert follow_up.text == "DSPy programs pipelines."
- assert get_messages(0) == [{"role": "user", "content": "Explain DSPy in one sentence."}]
- assert get_messages(1) == [
- {"role": "user", "content": "Explain DSPy in one sentence."},
- {"role": "assistant", "content": "DSPy programs LM pipelines."},
- {"role": "user", "content": "Now make it even shorter."},
- ]
- if lm_kind == "typed_lm":
- assert isinstance(get_request(1), dspy.LMRequest)
-async def test_base_lm_async_explicit_lm_request_returns_lm_response():
- class CustomLM(dspy.BaseLM):
- async def aforward(self, prompt=None, messages=None, **kwargs):
- return ModelResponse(
- choices=[Choices(message=Message(role="assistant", content="Hi async!"))],
- usage={},
- model="custom-model",
- )
- request = dspy.LMRequest.from_call(model="custom-model", prompt="Query")
- response = await CustomLM("custom-model").acall(request)
- assert isinstance(response, dspy.LMResponse)
- assert response.text == "Hi async!"
def test_base_lm_tracks_usage_for_custom_subclasses():
@@ -1012,7 +709,7 @@ def test_lm_load_state_forwards_allow_custom_lm_class(monkeypatch):
monkeypatch.setattr(dspy.BaseLM, "load_state", classmethod(spy_load_state))
- dspy.LM.load_state(dspy.LM("openai/gpt-4o-mini").dump_state(), allow_custom_lm_class=True)
+ dspy.LM.load_state(dspy.LM("openai/gpt-4o-mini", engine="litellm").dump_state(), allow_custom_lm_class=True)
assert calls == [True]
@@ -1027,25 +724,22 @@ def test_exponential_backoff_retry():
mock_response.status_code = 429
raise RateLimitError(response=mock_response, message="message", body="error")
- original_retrying = tenacity.Retrying
-
- def immediate_retrying(*args, **kwargs):
- kwargs["sleep"] = retry_delays.append
- return original_retrying(*args, **kwargs)
-
- lm = dspy.LM(model="openai/gpt-3.5-turbo", max_tokens=250, num_retries=3)
+ lm = dspy.LM(engine="litellm", model="openai/gpt-3.5-turbo", max_tokens=250, num_retries=3)
with (
- mock.patch("tenacity.Retrying", side_effect=immediate_retrying),
- mock.patch.object(litellm.OpenAIChatCompletion, "completion", side_effect=mock_create),
+ # Replace this module's reference, not the process-wide time.sleep:
+ # background SDK threads may also sleep while the request runs.
+ mock.patch("dspy.clients.execution.time", mock.Mock(sleep=retry_delays.append)),
+ mock.patch.object(litellm.OpenAIChatCompletion, "completion", side_effect=mock_create) as completion,
):
with pytest.raises(dspy.LMRateLimitError):
lm("question")
- assert retry_delays == [1.0, 2.0]
+ assert retry_delays == [1, 2, 4]
+ assert completion.call_count == 4
def test_logprobs_included_when_requested():
- lm = dspy.LM(model="dspy-test-model", logprobs=True, cache=False)
+ lm = dspy.LM(engine="litellm", model="dspy-test-model", logprobs=True, cache=False)
with mock.patch("litellm.completion") as mock_completion:
mock_completion.return_value = ModelResponse(
choices=[
@@ -1091,7 +785,7 @@ async def test_async_lm_call():
with patch("litellm.acompletion") as mock_acompletion:
mock_acompletion.return_value = mock_response
- lm = dspy.LM(model="openai/gpt-4o-mini", cache=False)
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False)
result = await lm.acall("question")
assert result == ["answer"]
@@ -1109,7 +803,7 @@ async def test_async_lm_call_with_cache(tmp_path):
)
cache = dspy.cache
- lm = dspy.LM(model="openai/gpt-4o-mini")
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
with mock.patch("dspy.clients.lm.alitellm_completion") as mock_alitellm_completion:
mock_alitellm_completion.return_value = ModelResponse(
@@ -1137,7 +831,7 @@ async def test_async_lm_call_with_cache(tmp_path):
def test_lm_history_size_limit():
- lm = dspy.LM(model="openai/gpt-4o-mini")
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
with dspy.context(max_history_size=5):
with mock.patch("litellm.completion") as mock_completion:
mock_completion.return_value = ModelResponse(
@@ -1152,7 +846,7 @@ def test_lm_history_size_limit():
def test_disable_history():
- lm = dspy.LM(model="openai/gpt-4o-mini")
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
with dspy.context(disable_history=True):
with mock.patch("litellm.completion") as mock_completion:
mock_completion.return_value = ModelResponse(
@@ -1198,7 +892,7 @@ def test_responses_api():
with mock.patch("litellm.responses", autospec=True, return_value=api_response) as dspy_responses:
lm = dspy.LM(
- model="openai/gpt-5-mini",
+ engine="litellm", model="openai/gpt-5-mini",
model_type="responses",
cache=False,
temperature=1.0,
@@ -1221,7 +915,7 @@ def test_lm_replaces_system_with_developer_role():
with mock.patch("dspy.clients.lm.litellm_responses_completion", return_value={"choices": []}) as mock_completion:
lm = dspy.LM(
"openai/gpt-4o-mini",
- cache=False,
+ engine="litellm", cache=False,
model_type="responses",
use_developer_role=True,
)
@@ -1270,7 +964,7 @@ def test_responses_api_tool_calls(litellm_test_server, provider_fields):
with mock.patch("litellm.responses", autospec=True, return_value=api_response) as dspy_responses:
lm = dspy.LM(
- model="openai/dspy-test-model",
+ engine="litellm", model="openai/dspy-test-model",
api_base=api_base,
api_key="fakekey",
model_type="responses",
@@ -1304,7 +998,7 @@ def test_responses_api_cache_hit_preserves_outputs_and_skips_usage(tmp_path):
dspy.configure_cache(enable_disk_cache=True, enable_memory_cache=True, disk_cache_dir=tmp_path / ".dspy_cache")
try:
with mock.patch("litellm.responses", autospec=True, return_value=api_response) as responses:
- lm = dspy.LM("openai/dspy-test-model", model_type="responses")
+ lm = dspy.LM("openai/dspy-test-model", engine="litellm", model_type="responses")
with track_usage() as first_usage:
first = lm("cache me")
with track_usage() as second_usage:
@@ -1343,7 +1037,7 @@ def test_responses_api_joins_multiple_text_outputs():
)
with mock.patch("litellm.responses", autospec=True, return_value=api_response):
- lm = dspy.LM("openai/dspy-test-model", model_type="responses", cache=False)
+ lm = dspy.LM("openai/dspy-test-model", engine="litellm", model_type="responses", cache=False)
outputs = lm("multi part query")
assert outputs == [{"text": "part one. part two."}]
@@ -1353,7 +1047,7 @@ def test_reasoning_effort_responses_api():
"""Test that reasoning_effort gets normalized to reasoning format for Responses API."""
with mock.patch("litellm.responses", return_value=make_response([])) as mock_responses:
lm = dspy.LM(
- model="openai/gpt-5", model_type="responses", reasoning_effort="low", max_tokens=16000, temperature=1.0
+ engine="litellm", model="openai/gpt-5", model_type="responses", reasoning_effort="low", max_tokens=16000, temperature=1.0
)
lm("openai query")
call_kwargs = mock_responses.call_args.kwargs
@@ -1382,7 +1076,7 @@ def test_call_reasoning_model_with_chat_api():
with mock.patch("litellm.supports_reasoning", return_value=True):
# Create reasoning model with chat API
lm = dspy.LM(
- model="anthropic/claude-3-7-sonnet-20250219",
+ engine="litellm", model="anthropic/claude-3-7-sonnet-20250219",
model_type="chat",
temperature=1.0,
max_tokens=16000,
@@ -1650,7 +1344,7 @@ def test_responses_api_with_image_input():
with mock.patch("litellm.responses", autospec=True, return_value=api_response) as dspy_responses:
lm = dspy.LM(
- model="openai/gpt-5-mini",
+ engine="litellm", model="openai/gpt-5-mini",
model_type="responses",
cache=False,
temperature=1.0,
@@ -1715,7 +1409,7 @@ def test_responses_api_with_pydantic_model_input():
)
lm = dspy.LM(
- model="openai/gpt-5-mini",
+ engine="litellm", model="openai/gpt-5-mini",
model_type="responses",
cache=False,
temperature=1.0,
@@ -1788,7 +1482,7 @@ def test_responses_api_with_none_usage():
with mock.patch("litellm.responses", autospec=True, return_value=api_response):
lm = dspy.LM(
- model="openai/gpt-5-mini",
+ engine="litellm", model="openai/gpt-5-mini",
model_type="responses",
cache=False,
temperature=1.0,
@@ -1845,7 +1539,7 @@ async def test_responses_api_with_none_usage_async():
with mock.patch("litellm.aresponses", autospec=True, return_value=api_response):
lm = dspy.LM(
- model="openai/gpt-5-mini",
+ engine="litellm", model="openai/gpt-5-mini",
model_type="responses",
cache=False,
temperature=1.0,
@@ -1936,92 +1630,8 @@ class ContractSchema(pydantic.BaseModel):
answer: str
-def test_openai_format_responses_request_maps_tools_choices_and_config():
- from dspy.clients.openai_format import to_openai_responses_request
- from dspy.core.types import LMRequest
-
- tool = _chat_shaped_weather_tool()
- tool["function"]["strict"] = True
- request = LMRequest.from_call(
- model="openai/gpt-5-mini",
- messages=[{"role": "user", "content": "What is the weather in Paris?"}],
- tools=[tool],
- tool_choice={"type": "function", "function": {"name": "get_weather"}},
- parallel_tool_calls=False,
- reasoning={"effort": "low", "summary": "auto"},
- response_format=ContractSchema,
- max_tokens=123,
- )
-
- data = to_openai_responses_request(request)
-
- assert data["input"] == [
- {
- "role": "user",
- "content": [{"type": "input_text", "text": "What is the weather in Paris?"}],
- }
- ]
- assert data["tools"] == [
- {
- "type": "function",
- "name": "get_weather",
- "description": "Get weather.",
- "parameters": {
- "type": "object",
- "properties": {"city": {"type": "string"}},
- "required": ["city"],
- },
- "strict": True,
- }
- ]
- assert data["tool_choice"] == {"type": "function", "name": "get_weather"}
- assert data["parallel_tool_calls"] is False
- assert data["reasoning"] == {"effort": "low", "summary": "auto"}
- assert data["max_output_tokens"] == 123
- # Closed schema, not a raw model_json_schema(): the Responses API rejects
- # schemas without explicit additionalProperties: false.
- assert data["text"]["format"] == {
- "type": "json_schema",
- "name": "ContractSchema",
- "schema": {**ContractSchema.model_json_schema(), "additionalProperties": False},
- }
-
-
-def test_responses_request_converts_assistant_tool_calls_and_tool_results():
- from dspy.clients.openai_format import to_openai_responses_request
- from dspy.core.types import LMRequest
- request = LMRequest.from_call(
- model="openai/gpt-5-mini",
- messages=[
- {"role": "user", "content": "What is the weather?"},
- {
- "role": "assistant",
- "content": None,
- "tool_calls": [
- {
- "id": "call_1",
- "type": "function",
- "function": {"name": "get_weather", "arguments": json.dumps({"city": "Paris"})},
- }
- ],
- },
- {"role": "tool", "tool_call_id": "call_1", "name": "get_weather", "content": "sunny"},
- ],
- )
- data = to_openai_responses_request(request)
-
- assert data["input"] == [
- {"role": "user", "content": [{"type": "input_text", "text": "What is the weather?"}]},
- {
- "type": "function_call",
- "name": "get_weather",
- "arguments": json.dumps({"city": "Paris"}),
- "call_id": "call_1",
- },
- {"type": "function_call_output", "output": "sunny", "call_id": "call_1"},
- ]
def test_lm_responses_passes_hosted_tools_through_unchanged():
@@ -2035,7 +1645,7 @@ def test_lm_responses_passes_hosted_tools_through_unchanged():
}
with mock.patch("litellm.responses", autospec=True, return_value=response) as responses:
- lm = dspy.LM("openai/dspy-test-model", model_type="responses", cache=False)
+ lm = dspy.LM("openai/dspy-test-model", engine="litellm", model_type="responses", cache=False)
lm("What is in the news?", tools=[hosted_tool, _chat_shaped_weather_tool(), flat_tool])
sent_tools = responses.call_args.kwargs["tools"]
@@ -2052,7 +1662,7 @@ def test_lm_responses_passes_native_tool_choice_shapes_through():
{"type": "allowed_tools", "mode": "auto", "tools": [{"type": "function", "name": "get_weather"}]},
):
with mock.patch("litellm.responses", autospec=True, return_value=response) as responses:
- lm = dspy.LM("openai/dspy-test-model", model_type="responses", cache=False)
+ lm = dspy.LM("openai/dspy-test-model", engine="litellm", model_type="responses", cache=False)
lm("What is the weather?", tools=[_chat_shaped_weather_tool()], tool_choice=native_choice)
assert responses.call_args.kwargs["tool_choice"] == native_choice
@@ -2062,7 +1672,7 @@ def test_lm_responses_tolerates_native_content_and_sdk_message_dumps():
response = _responses_text_response()
with mock.patch("litellm.responses", autospec=True, return_value=response) as responses:
- lm = dspy.LM("openai/dspy-test-model", model_type="responses", cache=False)
+ lm = dspy.LM("openai/dspy-test-model", engine="litellm", model_type="responses", cache=False)
lm(
messages=[
{"role": "user", "content": [{"type": "input_text", "text": "Say hi."}]},
@@ -2090,7 +1700,7 @@ def test_lm_responses_explicit_reasoning_wins_over_constructor_effort():
response = _responses_text_response()
with mock.patch("litellm.responses", autospec=True, return_value=response) as responses:
- lm = dspy.LM("openai/dspy-test-model", model_type="responses", cache=False, reasoning_effort="low")
+ lm = dspy.LM("openai/dspy-test-model", engine="litellm", model_type="responses", cache=False, reasoning_effort="low")
lm("Say hi.", reasoning={"effort": "high"})
sent = responses.call_args.kwargs
@@ -2103,7 +1713,7 @@ def test_lm_responses_forwards_raw_base64_file_data_verbatim():
raw_base64 = "JVBERi0xLjQK"
with mock.patch("litellm.responses", autospec=True, return_value=response) as responses:
- lm = dspy.LM("openai/dspy-test-model", model_type="responses", cache=False)
+ lm = dspy.LM("openai/dspy-test-model", engine="litellm", model_type="responses", cache=False)
lm(
messages=[
{
@@ -2125,50 +1735,9 @@ def test_lm_responses_does_not_validate_reasoning_temperature_client_side():
response = _responses_text_response()
with mock.patch("litellm.responses", autospec=True, return_value=response) as responses:
- lm = dspy.LM("openai/gpt-5-nano", model_type="responses", cache=False, max_tokens=16000)
+ lm = dspy.LM("openai/gpt-5-nano", engine="litellm", model_type="responses", cache=False, max_tokens=16000)
lm("Say hi.", temperature=0.7, reasoning_effort="low")
sent = responses.call_args.kwargs
assert sent["temperature"] == 0.7
assert sent["reasoning"] == {"effort": "low", "summary": "auto"}
-def test_responses_to_lm_response_normalizes_mixed_text_reasoning_and_tool_calls():
- from dspy.clients.openai_format import responses_to_lm_response
- from dspy.core.types import LMRequest, LMThinkingPart
-
- response = make_response(
- [
- ResponseOutputMessage(
- id="msg_1",
- type="message",
- role="assistant",
- status="completed",
- content=[{"type": "output_text", "text": "I should use weather.", "annotations": []}],
- ),
- {
- "type": "function_call",
- "name": "get_weather",
- "arguments": '{"city": "Paris",}',
- "call_id": "call_1",
- "id": "fc_1",
- "status": "completed",
- },
- ResponseReasoningItem(
- id="reasoning_1",
- type="reasoning",
- summary=[Summary(type="summary_text", text="Need live weather.")],
- ),
- ]
- )
-
- lm_response = responses_to_lm_response(response, LMRequest(model="openai/dspy-test-model", messages=[]))
- output = lm_response.outputs[0]
-
- assert output.text == "I should use weather."
- assert output.reasoning_content == "Need live weather."
- assert isinstance(output.parts[2], LMThinkingPart)
- assert output.tool_calls[0].id == "call_1"
- assert output.tool_calls[0].name == "get_weather"
- assert output.tool_calls[0].args == {}
- assert output.tool_calls[0].provider_data["raw_arguments"] == '{"city": "Paris",}'
- assert "arguments_parse_error" in output.tool_calls[0].provider_data
- assert lm_response.usage.total_tokens == 2
diff --git a/tests/clients/test_lm_async_execution.py b/tests/clients/test_lm_async_execution.py
new file mode 100644
index 000000000..452bf72d1
--- /dev/null
+++ b/tests/clients/test_lm_async_execution.py
@@ -0,0 +1,465 @@
+"""Async execution boundaries exercised against a real, local HTTP transport."""
+
+import asyncio
+import json
+import threading
+import time
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from unittest.mock import patch
+
+import anyio
+import pytest
+
+import dspy
+from dspy.clients.engines import AsyncLM15Engine
+from dspy.clients.model_metadata import _snapshot
+from dspy.lm15 import Message, Response, Usage
+from dspy.utils.callback import BaseCallback
+from dspy.utils.usage_tracker import track_usage
+
+
+def endpoint(monkeypatch):
+ monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
+ state = {"active": 0, "peak": 0, "calls": 0, "status": 200, "delay": 0.01, "retry_after": "9"}
+ lock = threading.Lock()
+ entered = threading.Event()
+
+ class Handler(BaseHTTPRequestHandler):
+ protocol_version = "HTTP/1.1"
+
+ def log_message(self, *args):
+ pass
+
+ def do_POST(self):
+ data = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
+ with lock:
+ state["active"] += 1
+ state["calls"] += 1
+ state["peak"] = max(state["peak"], state["active"])
+ status = state["status"]
+ if state.get("recover"):
+ state["status"] = 200
+ entered.set()
+ if state.get("release") is not None:
+ state["release"].wait()
+ time.sleep(state["delay"])
+ answer = "A" if "alpha" in json.dumps(data) else "B"
+ body = {"model": "fake", "choices": [{"index": 0, "message": {"role": "assistant",
+ "content": f"[[ ## answer ## ]]\n{answer}\n\n[[ ## completed ## ]]"}, "finish_reason": "stop"}],
+ "usage": {"prompt_tokens": 7, "completion_tokens": 3, "total_tokens": 10}}
+ if status != 200:
+ body = {"error": {"message": "try later", "type": "rate_limit_error", "code": "rate_limit_exceeded"}}
+ encoded = json.dumps(body).encode()
+ streaming = data.get("stream") and status == 200
+ if streaming:
+ frames = [{"choices": [{"index": 0, "delta": {"content": t}, "finish_reason": None}]}
+ for t in ("[[", " ## answer ## ]]\n", answer, "\n\n[[ ## completed ## ]]")]
+ frames.append({"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]})
+ frames.append({"choices": [], "usage": body["usage"]})
+ encoded = "".join("data: " + json.dumps(f) + "\n\n" for f in frames).encode() + b"data: [DONE]\n\n"
+ try:
+ self.send_response(status)
+ self.send_header("Content-Type", "text/event-stream" if streaming else "application/json")
+ if state["retry_after"] is not None:
+ self.send_header("rEtRy-AfTeR", state["retry_after"])
+ self.send_header("Content-Length", str(len(encoded)))
+ self.end_headers()
+ self.wfile.write(encoded)
+ except (BrokenPipeError, ConnectionResetError):
+ pass
+ finally:
+ with lock:
+ state["active"] -= 1
+
+ http = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+ worker = threading.Thread(target=http.serve_forever, daemon=True)
+ worker.start()
+ try:
+ yield f"http://127.0.0.1:{http.server_port}/v1", state, entered
+ finally:
+ if state.get("release") is not None:
+ state["release"].set()
+ http.shutdown()
+ http.server_close()
+ worker.join()
+
+
+def native_lm(endpoint, **kwargs):
+ return dspy.LM("openai/fake", engine="lm15", api_base=endpoint[0], api_key="fake",
+ cache=False, num_retries=0, **kwargs)
+
+
+async def test_concurrent_programs_and_usage_are_isolated(endpoint):
+ lm = native_lm(endpoint)
+ endpoint[1]["delay"] = 0.1
+ program = dspy.Predict("question -> answer")
+
+ async def call(question):
+ with dspy.context(lm=lm, track_usage=True, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False)):
+ return await program.acall(question=question)
+
+ try:
+ results = await asyncio.wait_for(asyncio.gather(call("alpha"), call("beta"), call("alpha")), 10)
+ assert [r.answer for r in results] == ["A", "B", "A"]
+ assert endpoint[1]["peak"] > 1
+ assert all(r.get_lm_usage()[lm.model]["total_tokens"] == 10 for r in results)
+ assert len(lm.history) == 3
+ finally:
+ await lm.aclose()
+
+
+async def test_native_async_program_streaming(endpoint):
+ lm = native_lm(endpoint)
+ program = dspy.Predict("question -> answer")
+ stream = dspy.streamify(program, is_async_program=True,
+ stream_listeners=[dspy.streaming.StreamListener("answer")])
+ try:
+ with dspy.context(lm=lm, track_usage=True):
+ async def collect():
+ return [item async for item in stream(question="alpha")]
+ items = await asyncio.wait_for(collect(), 10)
+ assert items[-1].answer == "A"
+ assert "".join(i.chunk for i in items if isinstance(i, dspy.streaming.StreamResponse)) == "A"
+ assert items[-1].get_lm_usage()[lm.model]["total_tokens"] == 10
+ finally:
+ await lm.aclose()
+
+
+async def test_http_cancellation_releases_pool_without_history(endpoint):
+ lm = native_lm(endpoint)
+ # Hold the response until after cancellation: a fixed delay races with
+ # thread scheduling under the parallel Python-version matrix.
+ endpoint[1]["release"] = threading.Event()
+ task = asyncio.create_task(lm.acall("alpha"))
+ try:
+ assert await asyncio.to_thread(endpoint[2].wait, 5)
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ backend = next(iter(lm._engine_store.values()))
+ provider = next(iter(backend._providers.values()))
+ assert provider.transport.pool_stats()["in_use"] == 0
+ assert not lm.history
+ assert endpoint[1]["calls"] == 1
+ finally:
+ if not task.done():
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+ await lm.aclose()
+
+
+def test_same_lm_on_successive_loops(endpoint):
+ lm = native_lm(endpoint)
+
+ async def once():
+ try:
+ return await lm.acall("alpha")
+ finally:
+ await lm.aclose()
+
+ assert asyncio.run(once()) == asyncio.run(once())
+ assert endpoint[1]["calls"] == 2
+ assert lm._engine_store == {}
+
+
[email protected]("asynchronous", [False, True])
[email protected]("streaming", [False, True])
[email protected]("header,expected", [("9", 9), ("invalid", None), (None, None)])
+async def test_http_retry_header_reaches_dspy_error(endpoint, asynchronous, streaming, header, expected):
+ lm = native_lm(endpoint)
+ endpoint[1].update(status=429, retry_after=header)
+
+ class Sink:
+ async def send(self, chunk):
+ raise AssertionError("Rejected stream must not emit chunks")
+
+ try:
+ with dspy.context(send_stream=Sink() if streaming else None):
+ with pytest.raises(dspy.LMRateLimitError) as caught:
+ if asynchronous:
+ await lm.acall("alpha")
+ else:
+ await anyio.to_thread.run_sync(lambda: lm("alpha"))
+ assert caught.value.retry_after == expected
+ assert caught.value.status == 429
+ assert endpoint[1]["calls"] == 1
+ finally:
+ await lm.aclose()
+
+
+async def test_retry_loop_uses_preserved_header(endpoint, monkeypatch):
+ import dspy.clients.execution as execution
+
+ lm = native_lm(endpoint)
+ lm.num_retries = 1
+ endpoint[1].update(status=429, recover=True)
+ delays = []
+ original = execution._delay
+
+ def delay(exc, attempt):
+ delays.append(original(exc, attempt))
+ return 0 # verify the selected delay without waiting nine seconds
+
+ monkeypatch.setattr(execution, "_delay", delay)
+ try:
+ await lm.acall("alpha")
+ assert delays == [9]
+ assert endpoint[1]["calls"] == 2
+ assert len(lm.history) == 1
+ finally:
+ await lm.aclose()
+
+
+async def test_async_model_listing_preserves_retry_header():
+ from dspy._vendor.lm15.providers.async_base import AsyncOpenAIChatLM
+ from dspy._vendor.lm15.providers.base import HttpResponse
+ from dspy.lm15 import RateLimitError
+
+ async def send(self, request):
+ return HttpResponse(429, "Too Many Requests", [("Retry-After", "9")],
+ b'{"error":{"type":"rate_limit_error","message":"wait"}}')
+
+ provider = AsyncOpenAIChatLM(api_key="fake")
+ try:
+ with patch.object(AsyncOpenAIChatLM, "_send", send):
+ with pytest.raises(RateLimitError) as caught:
+ await provider.list_models()
+ assert caught.value.retry_after == 9
+ finally:
+ await provider.aclose()
+
+
[email protected]("adapter", [dspy.ChatAdapter, dspy.JSONAdapter])
+async def test_adapter_metadata_load_runs_off_loop_once(adapter, monkeypatch):
+ import io
+
+ import dspy.clients.model_metadata as metadata
+
+ monkeypatch.delenv("LITELLM_LOCAL_MODEL_COST_MAP", raising=False)
+ monkeypatch.setattr(metadata, "_data", None)
+ monkeypatch.setattr(metadata, "_source", {})
+ entered, release = threading.Event(), threading.Event()
+ threads = []
+ payload = json.dumps(_snapshot()).encode()
+ loop_thread = threading.get_ident()
+
+ def download(*args, **kwargs):
+ threads.append(threading.get_ident())
+ entered.set()
+ release.wait(5)
+ return io.BytesIO(payload)
+
+ monkeypatch.setattr(metadata.urllib.request, "urlopen", download)
+
+ async def complete(self, request):
+ text = '{"answer":"A"}' if adapter is dspy.JSONAdapter else "[[ ## answer ## ]]\nA"
+ return Response(None, request.model, Message.assistant(text), "stop", Usage(input_tokens=1, output_tokens=1))
+
+ monkeypatch.setattr(AsyncLM15Engine, "complete", complete)
+ lm = dspy.LM("openai/gpt-4o", engine="lm15", cache=False)
+ program = dspy.Predict("question -> answer")
+
+ async def call():
+ with dspy.context(lm=lm, adapter=adapter()):
+ return await program.acall(question="alpha")
+
+ tasks = [asyncio.create_task(call()), asyncio.create_task(call())]
+ try:
+ assert await asyncio.to_thread(entered.wait, 5)
+ assert len(threads) == 1 and threads[0] != loop_thread
+ release.set()
+ assert [r.answer for r in await asyncio.gather(*tasks)] == ["A", "A"]
+ assert len(threads) == 1
+ finally:
+ release.set()
+ await asyncio.gather(*tasks, return_exceptions=True)
+ await lm.aclose()
+
+
+async def test_cost_lookup_runs_off_loop(endpoint, monkeypatch):
+ import dspy.clients.costs as costs
+
+ lm = native_lm(endpoint)
+ threads = []
+ loop_thread = threading.get_ident()
+
+ def info(*args):
+ threads.append(threading.get_ident())
+ return {"input_cost_per_token": 0.001, "output_cost_per_token": 0.002}
+
+ monkeypatch.setattr(costs, "model_info", info)
+ try:
+ await lm.acall("alpha")
+ assert threads and all(t != loop_thread for t in threads)
+ assert lm.history[-1]["cost"] == pytest.approx(0.013)
+ finally:
+ await lm.aclose()
+
+
+class Candidates:
+ def __init__(self, phase):
+ self.calls = 0
+ self.phase = phase
+ self.waiting = asyncio.Event()
+
+ async def complete(self, request):
+ self.calls += 1
+ if self.calls == 2:
+ self.waiting.set()
+ if self.phase == "backoff":
+ raise dspy.LMRateLimitError("wait", retry_after=60)
+ await asyncio.Event().wait()
+ return Response(None, request.model, Message.assistant("ok"), "stop", Usage(input_tokens=7, output_tokens=3))
+
+ async def stream(self, request):
+ from dspy.lm15 import response_to_events
+
+ for event in response_to_events(await self.complete(request)):
+ yield event
+
+
[email protected]("streaming", [False, True])
[email protected]("phase", ["provider", "backoff", "pricing", "cache"])
+async def test_cancellation_accounts_completed_candidates_once(phase, streaming, monkeypatch):
+ import dspy.clients.execution as execution
+
+ engine = Candidates(phase)
+ lm = dspy.LM("custom", engine=object_engine(), async_engine=engine, cache=phase == "cache", num_retries=3)
+ worker_entered, release, worker_finished = threading.Event(), threading.Event(), threading.Event()
+ worker_threads = []
+ loop_thread = threading.get_ident()
+
+ def block(*args):
+ worker_threads.append(threading.get_ident())
+ worker_entered.set()
+ try:
+ if not release.wait(10):
+ raise TimeoutError("Test did not release the pricing/cache worker")
+ finally:
+ worker_finished.set()
+
+ def instrument(name, observer):
+ original = getattr(execution, name)
+
+ def scoped(owner, *args, **kwargs):
+ # Assert this LM's behavior, not activity from unrelated callers.
+ if owner is lm:
+ return observer(owner, *args, **kwargs)
+ return original(owner, *args, **kwargs)
+
+ monkeypatch.setattr(execution, name, scoped)
+
+ if phase == "pricing":
+ instrument("_price_result", block)
+ if phase == "cache":
+ instrument("_store", block)
+ calls_stored = []
+ if phase != "cache":
+ instrument("_store", lambda *args: calls_stored.append(args))
+ errors = []
+
+ class Trace(BaseCallback):
+ def on_lm_end(self, call_id, outputs, exception):
+ errors.append(exception)
+
+ class Sink:
+ async def send(self, chunk):
+ pass
+
+ lm.callbacks = [Trace()]
+ with dspy.context(send_stream=Sink() if streaming else None), track_usage() as tracker:
+ task = asyncio.create_task(lm.acall("hello", n=1 if phase == "cache" else 2))
+ try:
+ if phase in ("provider", "backoff"):
+ await asyncio.wait_for(engine.waiting.wait(), 5)
+ await asyncio.sleep(0) # let the caller enter its retry backoff
+ else:
+ assert await asyncio.to_thread(worker_entered.wait, 5)
+ assert worker_threads[0] != loop_thread
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert tracker.get_total_tokens()[lm.model]["total_tokens"] == 10
+ assert not lm.history
+ assert len(errors) == 1 and isinstance(errors[0], asyncio.CancelledError)
+ assert not calls_stored
+ assert engine.calls == (2 if phase in ("provider", "backoff") else 1)
+ finally:
+ release.set()
+ if not task.done():
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+ if worker_entered.is_set():
+ assert await asyncio.to_thread(worker_finished.wait, 10)
+
+
+async def test_cancellation_during_final_stream_delivery_preserves_usage():
+ engine = Candidates("provider")
+ lm = dspy.LM("custom", engine=object_engine(), async_engine=engine, cache=False)
+ waiting = asyncio.Event()
+
+ class Sink:
+ async def send(self, chunk):
+ if chunk.choices[0].finish_reason is not None:
+ waiting.set()
+ await asyncio.Event().wait()
+
+ with dspy.context(send_stream=Sink()), track_usage() as tracker:
+ task = asyncio.create_task(lm.acall("hello"))
+ try:
+ await asyncio.wait_for(waiting.wait(), 5)
+ task.cancel()
+ with pytest.raises(asyncio.CancelledError):
+ await task
+ assert tracker.get_total_tokens()[lm.model]["total_tokens"] == 10
+ assert not lm.history
+ assert engine.calls == 1
+ finally:
+ task.cancel()
+ await asyncio.gather(task, return_exceptions=True)
+
+
+def object_engine():
+ class Engine:
+ def complete(self, request):
+ raise AssertionError("Async call must not use sync engine")
+ return Engine()
+
+
+async def test_async_cache_and_preparation_use_workers(endpoint, monkeypatch):
+ import dspy.clients.execution as execution
+
+ lm = native_lm(endpoint)
+ lm.cache = True
+ threads = {name: [] for name in ("prepare", "_cached", "_store")}
+ loop_thread = threading.get_ident()
+ for name in threads:
+ original = getattr(execution, name)
+
+ def record(*args, _name=name, _original=original, **kwargs):
+ threads[_name].append(threading.get_ident())
+ return _original(*args, **kwargs)
+
+ monkeypatch.setattr(execution, name, record)
+ try:
+ with track_usage() as tracker:
+ first = await lm.acall("alpha")
+ assert await lm.acall("alpha") == first
+ assert tracker.get_total_tokens()[lm.model]["total_tokens"] == 10
+ assert endpoint[1]["calls"] == 1
+ assert all(values and all(t != loop_thread for t in values) for values in threads.values())
+ assert lm.history[-1]["usage"] == {}
+ finally:
+ await lm.aclose()
diff --git a/tests/clients/test_lm_direct_live.py b/tests/clients/test_lm_direct_live.py
index 01fe2e67f..4c4576dcc 100644
--- a/tests/clients/test_lm_direct_live.py
+++ b/tests/clients/test_lm_direct_live.py
@@ -1,12 +1,4 @@
-"""Live coverage for the experimental direct LM call interface.
-
-These tests exercise provider behavior that cannot be verified with the mocked unit tests in `test_lm.py`: typed
-message turns, tool-call transcripts, and reusing an `LMResponse` as an assistant turn across chat and Responses API
-providers.
-
-They are intentionally flat rather than parametrized so each test can be run individually from an editor or notebook-like
-workflow. Each test skips unless the required provider credential is available.
-"""
+"""Explicitly selected live coverage for canonical requests and ordinary calls."""
import json
import os
@@ -15,302 +7,62 @@ import pydantic
import pytest
import dspy
+from dspy.lm15 import Config, Message, Request, Response, ToolCallPart
-def _require_env(*keys: str) -> None:
+def _require_env(*keys):
missing = [key for key in keys if not os.getenv(key)]
if missing:
pytest.skip(f"Missing live LM credentials: {', '.join(missing)}")
-def _text(response: dspy.LMResponse) -> str:
- assert isinstance(response, dspy.LMResponse)
+def _text(response):
+ assert isinstance(response, Response)
assert response.text is not None
return response.text.strip()
@pytest.mark.llm_call
-def test_live_openai_chat_direct_system_user_assistant_multiturn():
- _require_env("OPENAI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_OPENAI_CHAT", "openai/gpt-5.5"),
- model_type="chat",
- cache=False,
- max_completion_tokens=64,
- )
-
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Follow the user's requested exact final token. No punctuation."),
- dspy.User("Reply with exactly: alpha"),
- dspy.Assistant("alpha"),
- dspy.User("Now reply with exactly: beta"),
- )
-
- assert "beta" in _text(response).lower()
- assert response.output.finish_reason is not None
-
-
-def test_live_openai_chat_direct_tool_call_transcript():
- _require_env("OPENAI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_OPENAI_CHAT", "openai/gpt-4o-mini"),
- model_type="chat",
- cache=False,
- max_tokens=64,
- )
-
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Use the supplied tool result. Keep the answer short."),
- dspy.User("What is the weather in Paris?"),
- dspy.Assistant(dspy.ToolCall(id="call_1", name="get_weather", args={"city": "Paris"})),
- dspy.ToolResult('{"temperature": "22 C"}', call_id="call_1", name="get_weather"),
- dspy.User("Answer with the temperature string from the tool result."),
- )
-
- text = _text(response).lower()
- assert "22" in text
- assert "c" in text
-
-
-def test_live_openai_chat_direct_reuse_lm_response_as_assistant_turn():
- _require_env("OPENAI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_OPENAI_CHAT", "openai/gpt-4o-mini"),
- model_type="chat",
- cache=False,
- max_tokens=64,
- )
-
- with dspy.context(experimental=True):
- first = lm(dspy.User("Reply with exactly: DSPy"))
- follow_up = lm(
- dspy.User("Reply with exactly: DSPy"),
- first,
- dspy.User("Repeat the previous assistant answer exactly."),
- )
-
- assert "dspy" in _text(first).lower()
- assert "dspy" in _text(follow_up).lower()
-
-
-def test_live_openai_responses_direct_system_user_assistant_multiturn():
- _require_env("OPENAI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_OPENAI_RESPONSES", "openai/gpt-4.1-mini"),
- model_type="responses",
- cache=False,
- temperature=1.0,
- max_tokens=16000,
- )
-
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Follow the user's requested exact final token. No punctuation."),
- dspy.User("Reply with exactly: alpha"),
- dspy.Assistant("alpha"),
- dspy.User("Now reply with exactly: beta"),
- )
-
- assert "beta" in _text(response).lower()
-
-
-def test_live_openai_responses_direct_tool_call_transcript():
- _require_env("OPENAI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_OPENAI_RESPONSES", "openai/gpt-4.1-mini"),
- model_type="responses",
- cache=False,
- temperature=1.0,
- max_tokens=16000,
- )
-
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Use the supplied tool result. Keep the answer short."),
- dspy.User("What is the weather in Paris?"),
- dspy.Assistant(dspy.ToolCall(id="call_1", name="get_weather", args={"city": "Paris"})),
- dspy.ToolResult('{"temperature": "22 C"}', call_id="call_1", name="get_weather"),
- dspy.User("Answer with the temperature string from the tool result."),
- )
-
- text = _text(response).lower()
- assert "22" in text
- assert "c" in text
-
-
-def test_live_openai_responses_direct_reuse_lm_response_as_assistant_turn():
- _require_env("OPENAI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_OPENAI_RESPONSES", "openai/gpt-4.1-mini"),
- model_type="responses",
- cache=False,
- temperature=1.0,
- max_tokens=16000,
- )
-
- with dspy.context(experimental=True):
- first = lm(dspy.User("Reply with exactly: DSPy"))
- follow_up = lm(
- dspy.User("Reply with exactly: DSPy"),
- first,
- dspy.User("Repeat the previous assistant answer exactly."),
- )
-
- assert "dspy" in _text(first).lower()
- assert "dspy" in _text(follow_up).lower()
-
-
-def test_live_anthropic_chat_direct_system_user_assistant_multiturn():
- _require_env("ANTHROPIC_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_ANTHROPIC", "anthropic/claude-3-5-haiku-latest"),
- model_type="chat",
- cache=False,
- max_tokens=64,
- )
-
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Follow the user's requested exact final token. No punctuation."),
- dspy.User("Reply with exactly: alpha"),
- dspy.Assistant("alpha"),
- dspy.User("Now reply with exactly: beta"),
- )
-
- assert "beta" in _text(response).lower()
-
-
-def test_live_anthropic_chat_direct_tool_call_transcript():
- _require_env("ANTHROPIC_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_ANTHROPIC", "anthropic/claude-3-5-haiku-latest"),
- model_type="chat",
- cache=False,
- max_tokens=64,
- )
-
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Use the supplied tool result. Keep the answer short."),
- dspy.User("What is the weather in Paris?"),
- dspy.Assistant(dspy.ToolCall(id="call_1", name="get_weather", args={"city": "Paris"})),
- dspy.ToolResult('{"temperature": "22 C"}', call_id="call_1", name="get_weather"),
- dspy.User("Answer with the temperature string from the tool result."),
- )
-
- text = _text(response).lower()
- assert "22" in text
- assert "c" in text
-
-
-def test_live_anthropic_chat_direct_reuse_lm_response_as_assistant_turn():
- _require_env("ANTHROPIC_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_ANTHROPIC", "anthropic/claude-3-5-haiku-latest"),
- model_type="chat",
- cache=False,
- max_tokens=64,
- )
-
- with dspy.context(experimental=True):
- first = lm(dspy.User("Reply with exactly: DSPy"))
- follow_up = lm(
- dspy.User("Reply with exactly: DSPy"),
- first,
- dspy.User("Repeat the previous assistant answer exactly."),
- )
-
- assert "dspy" in _text(first).lower()
- assert "dspy" in _text(follow_up).lower()
-
-
-def test_live_gemini_chat_direct_system_user_assistant_multiturn():
- _require_env("GEMINI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_GEMINI", "gemini/gemini-2.0-flash"),
- model_type="chat",
- cache=False,
- max_tokens=64,
- )
-
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Follow the user's requested exact final token. No punctuation."),
- dspy.User("Reply with exactly: alpha"),
- dspy.Assistant("alpha"),
- dspy.User("Now reply with exactly: beta"),
- )
-
- assert "beta" in _text(response).lower()
-
-
-def test_live_gemini_chat_direct_tool_call_transcript():
- _require_env("GEMINI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_GEMINI", "gemini/gemini-2.0-flash"),
- model_type="chat",
- cache=False,
- max_tokens=64,
- )
-
- with dspy.context(experimental=True):
- response = lm(
- dspy.System("Use the supplied tool result. Keep the answer short."),
- dspy.User("What is the weather in Paris?"),
- dspy.Assistant(dspy.ToolCall(id="call_1", name="get_weather", args={"city": "Paris"})),
- dspy.ToolResult('{"temperature": "22 C"}', call_id="call_1", name="get_weather"),
- dspy.User("Answer with the temperature string from the tool result."),
- )
-
- text = _text(response).lower()
- assert "22" in text
- assert "c" in text
-
-
-def test_live_gemini_chat_direct_reuse_lm_response_as_assistant_turn():
- _require_env("GEMINI_API_KEY")
-
- lm = dspy.LM(
- os.getenv("LM_FOR_TEST_DIRECT_GEMINI", "gemini/gemini-2.0-flash"),
- model_type="chat",
- cache=False,
- max_tokens=64,
- )
-
- with dspy.context(experimental=True):
- first = lm(dspy.User("Reply with exactly: DSPy"))
- follow_up = lm(
- dspy.User("Reply with exactly: DSPy"),
- first,
- dspy.User("Repeat the previous assistant answer exactly."),
- )
-
- assert "dspy" in _text(first).lower()
- assert "dspy" in _text(follow_up).lower()
[email protected]("model,model_type,key", [
+ ("openai/gpt-4.1-mini", "chat", "OPENAI_API_KEY"),
+ ("openai/gpt-4.1-mini", "responses", "OPENAI_API_KEY"),
+ ("anthropic/claude-haiku-4-5", "chat", "ANTHROPIC_API_KEY"),
+ ("gemini/gemini-2.5-flash", "chat", "GEMINI_API_KEY"),
+])
+def test_live_canonical_multiturn_and_response_replay(model, model_type, key):
+ _require_env(key)
+ lm = dspy.LM(model, model_type=model_type, engine="lm15", cache=False, num_retries=0)
+ request = Request(model=model, system="Answer briefly.", messages=(Message.user("Say DSPy."),),
+ config=Config(max_tokens=256))
+ try:
+ first = lm(request)
+ followup = Request(model=model, system=request.system,
+ messages=(*request.messages, first.message, Message.user("Repeat your answer.")),
+ config=request.config)
+ assert "dspy" in _text(lm(followup)).lower()
+ finally:
+ lm.close()
+
+
[email protected]("model,model_type,key", [
+ ("openai/gpt-4.1-mini", "chat", "OPENAI_API_KEY"),
+ ("openai/gpt-4.1-mini", "responses", "OPENAI_API_KEY"),
+ ("anthropic/claude-haiku-4-5", "chat", "ANTHROPIC_API_KEY"),
+ ("gemini/gemini-2.5-flash", "chat", "GEMINI_API_KEY"),
+])
+def test_live_canonical_tool_result(model, model_type, key):
+ _require_env(key)
+ lm = dspy.LM(model, model_type=model_type, engine="lm15", cache=False, num_retries=0)
+ request = Request(model=model, messages=(
+ Message.user("What is the weather in Paris?"),
+ Message.assistant(ToolCallPart(id="call_1", name="weather", input={"city": "Paris"})),
+ Message.tool("call_1", "22 C"), Message.user("Repeat the temperature."),
+ ), config=Config(max_tokens=256))
+ try:
+ assert "22" in _text(lm(request))
+ finally:
+ lm.close()
# ---------------------------------------------------------------------------
diff --git a/tests/clients/test_lm_engine_compatibility_fixes.py b/tests/clients/test_lm_engine_compatibility_fixes.py
new file mode 100644
index 000000000..2a039354a
--- /dev/null
+++ b/tests/clients/test_lm_engine_compatibility_fixes.py
@@ -0,0 +1,106 @@
+"""Regression coverage for the first coordinated engine matrix failures."""
+
+import pytest
+
+import dspy
+from dspy.clients.legacy_requests import chat_to_responses
+from dspy.utils.dummies import DummyLM
+
+
[email protected]("asynchronous", [False, True])
+async def test_dummy_forward_override_is_not_bypassed(asynchronous):
+ error = dspy.LMRateLimitError("original error")
+
+ class DownLM(DummyLM):
+ def forward(self, *args, **kwargs):
+ raise error
+
+ lm = DownLM([])
+ with pytest.raises(dspy.LMRateLimitError) as caught:
+ if asynchronous:
+ await lm.acall("hello")
+ else:
+ lm("hello")
+ assert caught.value is error
+ assert lm.history == []
+
+
+async def test_dummy_override_can_delegate_to_super_without_recursion():
+ class RecordingLM(DummyLM):
+ calls = 0
+
+ def forward(self, *args, **kwargs):
+ self.calls += 1
+ return super().forward(*args, **kwargs)
+
+ lm = RecordingLM([{"answer": "first"}, {"answer": "second"}])
+ assert lm("hello") == ["[[ ## answer ## ]]\nfirst"]
+ assert await lm.acall("hello") == ["[[ ## answer ## ]]\nsecond"]
+ assert lm.calls == 2
+ assert len(lm.history) == 2
+
+
+async def test_dummy_instance_forward_override_is_respected():
+ lm = DummyLM([])
+ error = dspy.LMTransportError("offline")
+
+ def fail(**kwargs):
+ raise error
+
+ lm.forward = fail
+ with pytest.raises(dspy.LMTransportError) as caught:
+ await lm.acall("hello")
+ assert caught.value is error
+
+
+async def test_dummy_accepts_opaque_blocks_and_positional_message_lists():
+ lm = DummyLM([{"answer": "first"}, {"answer": "second"}], reasoning=True)
+ messages = [{"role": "user", "content": [
+ {"type": "file", "file": {"file_id": "file-1", "filename": "report.txt"}},
+ {"type": "custom_event", "payload": {"value": 1}},
+ ]}]
+ first = lm(messages)
+ second = await lm.acall(messages=messages)
+ assert first[0]["text"] == "[[ ## answer ## ]]\nfirst"
+ assert second[0]["text"] == "[[ ## answer ## ]]\nsecond"
+ assert first[0]["reasoning_content"] == "Some reasoning"
+ assert messages[0]["content"][0]["file"]["filename"] == "report.txt"
+ assert lm.history[-1]["messages"] == messages
+
+
+def test_dictionary_responses_preserve_rich_outputs():
+ class DictionaryLM(dspy.BaseLM):
+ def forward(self, **kwargs):
+ return {"model": "custom", "usage": {"total_tokens": 2}, "choices": [{
+ "message": {"content": "answer", "reasoning_content": "reasoning",
+ "tool_calls": [{"id": "call-1", "type": "function", "function": {
+ "name": "lookup", "arguments": "{}"}}],
+ "provider_specific_fields": {"citations": [[{"cited_text": "source"}]]}},
+ "logprobs": {"content": []}, "finish_reason": "stop",
+ }]}
+
+ lm = DictionaryLM("custom")
+ result = lm("hello", logprobs=True)[0]
+ assert result["text"] == "answer"
+ assert result["reasoning_content"] == "reasoning"
+ assert result["tool_calls"][0]["function"]["name"] == "lookup"
+ assert result["citations"] == [{"cited_text": "source"}]
+ assert result["logprobs"] == {"content": []}
+ assert lm.history[0]["usage"] == {"total_tokens": 2}
+
+
+def test_absent_tool_description_is_omitted_without_dropping_other_fields():
+ original = {"model": "openai/example", "messages": [{"role": "user", "content": "hello"}],
+ "tools": [{"type": "function", "function": {
+ "name": "lookup", "description": None, "strict": False, "vendor_field": None,
+ "parameters": {"type": "object"},
+ }}]}
+ tool = chat_to_responses(original)["tools"][0]
+ assert "description" not in tool
+ assert tool["strict"] is False
+ assert "vendor_field" in tool and tool["vendor_field"] is None
+ assert "description" in original["tools"][0]["function"]
diff --git a/tests/clients/test_lm_engine_execution.py b/tests/clients/test_lm_engine_execution.py
new file mode 100644
index 000000000..327c7501a
--- /dev/null
+++ b/tests/clients/test_lm_engine_execution.py
@@ -0,0 +1,283 @@
+"""Offline checks for the coordinated LM engine path."""
+
+import json
+import zipfile
+from pathlib import Path
+
+import pytest
+
+import dspy
+from dspy._vendor.lm15.testing import FakeResponse, FakeTransport
+from dspy.clients.engines import LiteLLMEngine
+from dspy.lm15 import Config, Message, Request, Response, RouterConfig
+from dspy.utils.callback import BaseCallback
+from dspy.utils.usage_tracker import track_usage
+
+
+class DualResponse(FakeResponse):
+ async def __aenter__(self):
+ source = self
+
+ class AsyncResponse:
+ def __getattr__(self, name):
+ return getattr(source, name)
+
+ async def read(self):
+ return source.read()
+
+ async def __aiter__(self):
+ for chunk in source:
+ yield chunk
+
+ return AsyncResponse()
+
+ async def __aexit__(self, *exc):
+ pass
+
+
+def wire(text="hello", status=200):
+ body = {"id": "response-1", "model": "gpt-4o-mini", "choices": [
+ {"index": 0, "message": {"role": "assistant", "content": text}, "finish_reason": "stop"}
+ ], "usage": {"prompt_tokens": 2, "completion_tokens": 1, "total_tokens": 3}}
+ if status != 200:
+ body = {"error": {"message": "try later", "type": "authentication_error" if status == 401 else "rate_limit_error"}}
+ return DualResponse(status=status, body=json.dumps(body).encode())
+
+
+def native_transport(monkeypatch, responses):
+ import dspy.clients.execution as execution
+
+ transport = FakeTransport(responses)
+ monkeypatch.setattr(execution, "RouterConfig", lambda **kwargs: RouterConfig(
+ **{**kwargs, "api_keys": {"openai-chat": "fake"}, "transport": transport},
+ ))
+ return transport
+
+
+def test_default_native_cache_and_typed_request(monkeypatch):
+ transport = native_transport(monkeypatch, [wire(), wire("typed")])
+ lm = dspy.LM("openai/gpt-4o-mini")
+ with track_usage() as usage:
+ assert lm("hello") == ["hello"]
+ assert lm("hello") == ["hello"]
+ assert len(transport.requests) == 1
+ assert usage.get_total_tokens()[lm.model]["total_tokens"] == 3
+ assert lm.history[-1]["usage"] == {}
+ request = Request(model=lm.model, messages=(Message.user("explicit"),), config=Config(max_tokens=12))
+ response = lm(request)
+ assert isinstance(response, Response)
+ assert response.text == "typed"
+ assert lm.history[-1]["request"] is request
+ assert len(transport.requests) == 2
+
+
+async def test_async_native_cache_and_history(monkeypatch):
+ transport = native_transport(monkeypatch, [wire("async")])
+ lm = dspy.LM("openai/gpt-4o-mini")
+ try:
+ with track_usage() as usage:
+ assert await lm.acall("hello") == ["async"]
+ assert await lm.acall("hello") == ["async"]
+ assert len(transport.requests) == 1
+ assert len(lm.history) == 2
+ assert usage.get_total_tokens()[lm.model]["total_tokens"] == 3
+ finally:
+ await lm.aclose()
+
+
+def test_native_n_fans_out_but_bookkeeping_is_one_call(monkeypatch):
+ transport = native_transport(monkeypatch, [wire("first"), wire("second")])
+
+ class Trace(BaseCallback):
+ def __init__(self):
+ self.events = []
+
+ def on_lm_start(self, call_id, instance, inputs):
+ self.events.append(("start", call_id))
+
+ def on_lm_end(self, call_id, outputs, exception):
+ self.events.append(("end", call_id))
+
+ trace = Trace()
+ lm = dspy.LM("openai/gpt-4o-mini", callbacks=[trace])
+ with track_usage() as usage:
+ assert lm("hello", n=2) == ["first", "second"]
+ assert len(transport.requests) == 2
+ assert len(lm.history) == 1
+ assert trace.events[1] == ("end", trace.events[0][1])
+ assert len(trace.events) == 2
+ assert usage.get_total_tokens()[lm.model]["total_tokens"] == 6
+ assert lm("hello", n=2) == ["first", "second"]
+ assert len(transport.requests) == 2
+
+
+def test_native_retries_without_backend_fallback(monkeypatch):
+ import dspy.clients.execution as execution
+
+ transport = native_transport(monkeypatch, [wire(status=429), wire("recovered")])
+ delays = []
+ monkeypatch.setattr(execution.time, "sleep", delays.append)
+ lm = dspy.LM("openai/gpt-4o-mini", num_retries=1, cache=False)
+ assert lm("hello") == ["recovered"]
+ assert len(transport.requests) == 2
+ assert delays == [1]
+
+
+def test_provider_auth_error_does_not_switch_backend(monkeypatch):
+ native_transport(monkeypatch, [wire(status=401)])
+
+ def forbidden(*args, **kwargs):
+ pytest.fail("An authentication failure must not invoke LiteLLM")
+
+ monkeypatch.setattr(LiteLLMEngine, "complete_legacy", forbidden)
+ with pytest.raises(dspy.LMAuthError):
+ dspy.LM("openai/gpt-4o-mini", cache=False)("hello")
+
+
+def test_unrepresentable_ordinary_options_choose_compatibility_before_io(monkeypatch):
+ transport = native_transport(monkeypatch, [])
+ from dspy.clients.call_result import CallResult
+
+ calls = []
+
+ def complete(engine, lm, request, **context):
+ calls.append(request)
+ return CallResult(outputs=["compatible"], response_model=lm.model)
+
+ monkeypatch.setattr(LiteLLMEngine, "complete_legacy", complete)
+ assert dspy.LM("openai/gpt-4o-mini")("hello", prediction={"type": "content", "content": "hello"}) == ["compatible"]
+ assert calls[0]["prediction"]["content"] == "hello"
+ assert not transport.requests
+
+
+async def test_legacy_plugin_sync_async_preserve_forward_inputs():
+ from dspy.dsp.utils.utils import dotdict
+
+ class Plugin(dspy.BaseLM):
+ def forward(self, prompt=None, messages=None, **kwargs):
+ self.received = prompt, messages, kwargs
+ return dotdict(model=self.model, usage={"total_tokens": 1}, choices=[
+ dotdict(message=dotdict(content="ok"), finish_reason="stop")
+ ])
+
+ async def aforward(self, **kwargs):
+ return self.forward(**kwargs)
+
+ plugin = Plugin("custom")
+ with pytest.warns(DeprecationWarning, match="scheduled for removal in 3.5"):
+ assert plugin("hello") == ["ok"]
+ assert plugin.received == ("hello", None, {})
+ with pytest.warns(DeprecationWarning, match="scheduled for removal in 3.5"):
+ assert await plugin.acall("hello") == ["ok"]
+ assert len(plugin.history) == 2
+
+
+def test_new_native_cache_uses_restricted_plain_data(monkeypatch, tmp_path):
+ transport = native_transport(monkeypatch, [wire()])
+ dspy.configure_cache(enable_disk_cache=True, enable_memory_cache=False, disk_cache_dir=tmp_path, restrict_pickle=True)
+ try:
+ lm = dspy.LM("openai/gpt-4o-mini")
+ assert lm("hello") == ["hello"]
+ assert lm("hello") == ["hello"]
+ assert len(transport.requests) == 1
+ finally:
+ dspy.cache.disk_cache.close()
+
+
[email protected]("case_index", range(6))
+async def test_old_disk_cache_hits_before_native_routing(case_index, monkeypatch, tmp_path):
+ from tests.clients.test_lm_migration_compatibility import plain
+
+ fixtures = Path(__file__).parent / "fixtures" / "lm_3_3_0"
+ case = json.loads((fixtures / "manifest.json").read_text())["cases"][case_index]
+ with zipfile.ZipFile(fixtures / f"{case['name']}.zip") as archive:
+ archive.extractall(tmp_path)
+ dspy.configure_cache(enable_disk_cache=True, enable_memory_cache=False, disk_cache_dir=tmp_path)
+ import dspy.clients.execution as execution
+
+ def forbidden(*args, **kwargs):
+ pytest.fail("An old disk hit must not even initialize an engine")
+
+ monkeypatch.setattr(execution, "_engine", forbidden)
+ try:
+ lm = dspy.LM(**case["init"])
+ output = lm(**case["call"]) if case["mode"] == "sync" else await lm.acall(**case["call"])
+ assert plain(output) == case["outputs"]
+ assert lm.history[-1]["usage"] == {}
+ finally:
+ dspy.cache.disk_cache.close()
+
+
+async def test_native_streaming_reaches_existing_listener(monkeypatch):
+ payloads = [{"choices": [{"index": 0, "delta": {"content": text}, "finish_reason": None}]}
+ for text in ("[[ ## answer ## ]]\n", "native", "\n\n[[ ## completed ## ]]")]
+ payloads.append({"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
+ "usage": {"prompt_tokens": 2, "completion_tokens": 1, "total_tokens": 3}})
+ chunks = [f"data: {json.dumps(payload)}\n\n".encode() for payload in payloads]
+ chunks.append(b"data: [DONE]\n\n")
+ transport = native_transport(monkeypatch, [FakeResponse(status=200, body=b"", chunks=chunks)])
+ lm = dspy.LM("openai/gpt-4o-mini")
+ predict = dspy.Predict("question -> answer")
+ streamer = dspy.streamify(predict, stream_listeners=[dspy.streaming.StreamListener("answer")])
+ with dspy.context(lm=lm):
+ values = [value async for value in streamer(question="hello")]
+ cached = [value async for value in streamer(question="hello")]
+ assert values[-1].answer == "native"
+ assert "".join(value.chunk for value in values if isinstance(value, dspy.streaming.StreamResponse)) == "native"
+ assert len(cached) == 1 and cached[0].answer == "native"
+ assert len(transport.requests) == 1
+
+
+async def test_partial_stream_is_not_retried_and_reports_callback_error():
+ from dspy.lm15 import StreamDeltaEvent, StreamStartEvent, TextDelta
+
+ class Engine:
+ def __init__(self):
+ self.calls = 0
+
+ def complete(self, request):
+ raise AssertionError("Expected a streaming call")
+
+ def stream(self, request):
+ self.calls += 1
+ yield StreamStartEvent(model=request.model)
+ yield StreamDeltaEvent(TextDelta("partial"))
+ raise dspy.LMTransportError("stream broke")
+
+ class Sink:
+ async def send(self, chunk):
+ pass
+
+ class Trace(BaseCallback):
+ error = None
+
+ def on_lm_end(self, call_id, outputs, exception):
+ self.error = exception
+
+ engine = Engine()
+ trace = Trace()
+ lm = dspy.LM("custom", engine=engine, callbacks=[trace], num_retries=3)
+ import anyio
+
+ with dspy.context(send_stream=Sink()):
+ with pytest.raises(dspy.LMTransportError, match="stream broke"):
+ await anyio.to_thread.run_sync(lambda: lm("hello"))
+ assert engine.calls == 1
+ assert isinstance(trace.error, dspy.LMTransportError)
+ assert lm.history == []
+
+
+def test_dummy_uses_canonical_engine_without_changing_script_behavior():
+ from dspy.utils.dummies import DummyLM
+
+ lm = DummyLM([{"answer": "one"}, {"answer": "two"}])
+ with dspy.context(lm=lm):
+ predict = dspy.Predict("question -> answer")
+ assert predict(question="same").answer == "one"
+ assert predict(question="same").answer == "two"
+ assert len(lm.history) == 2
diff --git a/tests/clients/test_lm_message_deprecation.py b/tests/clients/test_lm_message_deprecation.py
new file mode 100644
index 000000000..47d278851
--- /dev/null
+++ b/tests/clients/test_lm_message_deprecation.py
@@ -0,0 +1,358 @@
+"""Migration warnings target callers, not DSPy's temporary adapter internals."""
+
+import asyncio
+import inspect
+import warnings
+from pathlib import Path
+
+import pytest
+
+import dspy
+from dspy.adapters.baml_adapter import BAMLAdapter
+from dspy.clients.call_result import CallResult
+from dspy.clients.engines import AsyncLiteLLMEngine, LiteLLMEngine
+from dspy.lm15 import Message, Request, Response, Usage
+from dspy.utils.callback import BaseCallback
+from dspy.utils.dummies import DummyLM
+
+_GUIDE = "https://dspy.ai/community/normalized-lm-api-migration/"
+
+
+class Engine:
+ def __init__(self, text="ok"):
+ self.text = text
+ self.requests = []
+
+ def complete(self, request):
+ self.requests.append(request)
+ return Response(None, request.model, Message.assistant(self.text), "stop", Usage())
+
+
+class AsyncEngine:
+ def __init__(self, sync):
+ self.sync = sync
+
+ async def complete(self, request):
+ return self.sync.complete(request)
+
+
+def make_lm(text="ok", *, cache=False, callbacks=None):
+ engine = Engine(text)
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=cache, callbacks=callbacks)
+ return lm, engine
+
+
+def migration_warnings(recorded):
+ return [item for item in recorded if _GUIDE in str(item.message)]
+
+
[email protected]("asynchronous", [False, True])
[email protected]("callbacks", [False, True])
+async def test_dictionary_messages_warn_at_the_call_site_even_on_cache_hits(asynchronous, callbacks):
+ lm, engine = make_lm(cache=True, callbacks=[BaseCallback()] if callbacks else None)
+ messages = [{"role": "user", "content": "hello"}]
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ for _ in range(2):
+ if asynchronous:
+ line = inspect.currentframe().f_lineno + 1
+ result = await lm.acall(messages=messages)
+ else:
+ line = inspect.currentframe().f_lineno + 1
+ result = lm(messages=messages)
+ assert result == ["ok"]
+
+ found = migration_warnings(recorded)
+ assert len(found) == 2
+ assert all(item.category is DeprecationWarning for item in found)
+ assert all(Path(item.filename).resolve() == Path(__file__).resolve() for item in found)
+ assert all(item.lineno == line for item in found)
+ assert all("OpenAI-style message dictionaries" in str(item.message) for item in found)
+ assert all("removal in DSPy 3.5" in str(item.message) for item in found)
+ assert all("lm('hello') remains supported" in str(item.message) for item in found)
+ assert all(_GUIDE + "#migrating-openai-style-messages" in str(item.message) for item in found)
+ assert len(engine.requests) == 1
+ assert messages == [{"role": "user", "content": "hello"}]
+
+
[email protected]("asynchronous", [False, True])
+async def test_message_objects_cannot_bypass_the_deprecated_keyword(asynchronous):
+ lm, engine = make_lm()
+ # Canonical Message objects belong in Request.messages, not lm(messages=).
+ # Diagnose the keyword before any backend-specific object conversion.
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", DeprecationWarning)
+ with pytest.raises(DeprecationWarning, match="through messages="):
+ if asynchronous:
+ await lm.acall(messages=[Message.user("hello")])
+ else:
+ lm(messages=[Message.user("hello")])
+ assert not engine.requests
+
+
[email protected]("asynchronous", [False, True])
+async def test_prompt_and_canonical_request_calls_stay_quiet(asynchronous):
+ lm, engine = make_lm()
+ request = Request(model=lm.model, messages=(Message.user("hello"),))
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ if asynchronous:
+ assert await lm.acall("hello") == ["ok"]
+ response = await lm.acall(request)
+ else:
+ assert lm("hello") == ["ok"]
+ response = lm(request)
+ assert isinstance(response, Response)
+ assert response.text == "ok"
+ assert len(engine.requests) == 2
+ assert not migration_warnings(recorded)
+
+
[email protected]("asynchronous", [False, True])
[email protected]("adapter_kind,text", [
+ ("chat", "[[ ## answer ## ]]\nok"),
+ ("json", '{"answer":"ok"}'),
+ ("xml", "<answer>ok</answer>"),
+ ("baml", '{"answer":"ok"}'),
+ ("two_step", "[[ ## answer ## ]]\nok"),
+])
+async def test_builtin_adapter_calls_stay_quiet_without_hiding_later_user_calls(asynchronous, adapter_kind, text):
+ lm, engine = make_lm(text)
+ adapter = {
+ "chat": dspy.ChatAdapter(use_json_adapter_fallback=False),
+ "json": dspy.JSONAdapter(),
+ "xml": dspy.XMLAdapter(use_json_adapter_fallback=False),
+ "baml": BAMLAdapter(),
+ "two_step": dspy.TwoStepAdapter(lm),
+ }[adapter_kind]
+ program = dspy.Predict("question -> answer")
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with dspy.context(lm=lm, adapter=adapter):
+ result = await program.acall(question="hello") if asynchronous else program(question="hello")
+ assert result.answer == "ok"
+ assert not migration_warnings(recorded)
+ assert len(engine.requests) == (2 if adapter_kind == "two_step" else 1)
+
+ messages = [{"role": "user", "content": "another call"}]
+ if asynchronous:
+ await lm.acall(messages=messages)
+ else:
+ lm(messages=messages)
+ assert len(migration_warnings(recorded)) == 1
+
+
+def test_dummy_program_calls_stay_quiet():
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with dspy.context(lm=DummyLM([{"answer": "ok"}])):
+ assert dspy.Predict("question -> answer")(question="hello").answer == "ok"
+ assert not migration_warnings(recorded)
+
+
[email protected]("asynchronous", [False, True])
+async def test_streamified_programs_stay_quiet(asynchronous):
+ from dspy.lm15 import response_to_events
+
+ class StreamingEngine(Engine):
+ def stream(self, request):
+ return response_to_events(self.complete(request))
+
+ class AsyncStreamingEngine(AsyncEngine):
+ async def stream(self, request):
+ for event in self.sync.stream(request):
+ yield event
+
+ engine = StreamingEngine("[[ ## answer ## ]]\nok")
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncStreamingEngine(engine), cache=False)
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with dspy.context(lm=lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False)):
+ stream = dspy.streamify(dspy.Predict("question -> answer"), is_async_program=asynchronous)
+ values = [value async for value in stream(question="hello")]
+ assert values[-1].answer == "ok"
+ assert len(engine.requests) == 1
+ assert not migration_warnings(recorded)
+
+
+async def test_adapter_origin_does_not_leak_to_concurrent_user_call():
+ lm, engine = make_lm("[[ ## answer ## ]]\nok")
+ entered, release = asyncio.Event(), asyncio.Event()
+
+ class BlockingEngine(AsyncEngine):
+ async def complete(self, request):
+ if "[[ ## question ## ]]" in request.messages[-1].text:
+ entered.set()
+ await release.wait()
+ return await super().complete(request)
+
+ lm = lm.copy(async_engine=BlockingEngine(engine))
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with dspy.context(lm=lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False)):
+ task = asyncio.create_task(dspy.Predict("question -> answer").acall(question="hello"))
+ try:
+ await asyncio.wait_for(entered.wait(), 5)
+ await lm.acall(messages=[{"role": "user", "content": "user call"}])
+ finally:
+ release.set()
+ await asyncio.gather(task, return_exceptions=True)
+ assert task.result().answer == "ok"
+ found = migration_warnings(recorded)
+ assert len(found) == 1
+ assert "OpenAI-style message dictionaries" in str(found[0].message)
+
+
+def test_callback_call_on_another_lm_is_not_hidden_even_with_same_messages():
+ other, _ = make_lm()
+
+ class Callback(BaseCallback):
+ def on_lm_start(self, call_id, instance, inputs):
+ other(messages=inputs["messages"])
+
+ lm, _ = make_lm("[[ ## answer ## ]]\nok", callbacks=[Callback()])
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with dspy.context(lm=lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False)):
+ assert dspy.Predict("question -> answer")(question="hello").answer == "ok"
+ found = migration_warnings(recorded)
+ assert len(found) == 1
+ assert "OpenAI-style message dictionaries" in str(found[0].message)
+
+
+def test_nested_engine_call_with_different_messages_is_not_hidden():
+ lm, engine = make_lm("[[ ## answer ## ]]\nok")
+ complete = engine.complete
+ nested = False
+
+ def with_nested_call(request):
+ nonlocal nested
+ if not nested:
+ nested = True
+ lm(messages=[{"role": "user", "content": "nested user call"}])
+ return complete(request)
+
+ engine.complete = with_nested_call
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with dspy.context(lm=lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False)):
+ assert dspy.Predict("question -> answer")(question="hello").answer == "ok"
+ assert len(migration_warnings(recorded)) == 1
+ assert len(engine.requests) == 2
+
+
+def test_adapter_origin_is_restored_after_failure():
+ lm, engine = make_lm()
+
+ def fail(request):
+ raise dspy.LMAuthError("denied")
+
+ engine.complete = fail
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ with dspy.context(lm=lm, adapter=dspy.ChatAdapter(use_json_adapter_fallback=False)):
+ with pytest.raises(dspy.LMAuthError):
+ dspy.Predict("question -> answer")(question="hello")
+ assert not migration_warnings(recorded)
+ with pytest.raises(dspy.LMAuthError):
+ lm(messages=[{"role": "user", "content": "hello"}])
+ assert len(migration_warnings(recorded)) == 1
+
+
[email protected]("asynchronous", [False, True])
+async def test_message_warning_as_error_stops_before_engine_execution(asynchronous):
+ lm, engine = make_lm()
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", DeprecationWarning)
+ with pytest.raises(DeprecationWarning, match="OpenAI-style"):
+ if asynchronous:
+ await lm.acall(messages=[{"role": "user", "content": "hello"}])
+ else:
+ lm(messages=[{"role": "user", "content": "hello"}])
+ assert not engine.requests
+ assert not lm.history
+
+
[email protected]("asynchronous", [False, True])
+async def test_custom_legacy_shortcut_warns_but_canonical_method_does_not(asynchronous):
+ class ShortcutEngine(Engine):
+ def complete_legacy(self, lm, request, **context):
+ return CallResult(outputs=["ok"], response_model=lm.model)
+
+ class AsyncShortcutEngine(AsyncEngine):
+ async def complete_legacy(self, lm, request, **context):
+ return self.sync.complete_legacy(lm, request, **context)
+
+ engine = ShortcutEngine()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncShortcutEngine(engine), cache=False)
+ with pytest.warns(DeprecationWarning, match=r"complete_legacy\(\).*removal in DSPy 3.5"):
+ assert (await lm.acall("hello") if asynchronous else lm("hello")) == ["ok"]
+ request = Request(model=lm.model, messages=(Message.user("hello"),))
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ response = await lm.acall(request) if asynchronous else lm(request)
+ assert response.text == "ok"
+ assert not migration_warnings(recorded)
+
+
[email protected]("asynchronous", [False, True])
[email protected]("form", ["named", "explicit", "subclass"])
+async def test_litellm_backend_itself_is_not_deprecated(asynchronous, form, monkeypatch):
+ def complete(self, lm, request, **context):
+ return CallResult(outputs=["ok"], response_model=lm.model)
+
+ async def acomplete(self, lm, request, **context):
+ return complete(self, lm, request, **context)
+
+ monkeypatch.setattr(LiteLLMEngine, "complete_legacy", complete)
+ monkeypatch.setattr(AsyncLiteLLMEngine, "complete_legacy", acomplete)
+ class InheritedEngine(LiteLLMEngine):
+ pass
+
+ class AsyncInheritedEngine(AsyncLiteLLMEngine):
+ pass
+
+ if form == "named":
+ options = {"engine": "litellm"}
+ elif form == "subclass":
+ options = {"engine": InheritedEngine(), "async_engine": AsyncInheritedEngine()}
+ else:
+ options = {"engine": LiteLLMEngine(), "async_engine": AsyncLiteLLMEngine()}
+ lm = dspy.LM("custom", cache=False, **options)
+ with warnings.catch_warnings(record=True) as recorded:
+ warnings.simplefilter("always", DeprecationWarning)
+ assert (await lm.acall("hello") if asynchronous else lm("hello")) == ["ok"]
+ assert not migration_warnings(recorded)
+
+
[email protected]("asynchronous", [False, True])
+async def test_shortcut_warning_as_error_stops_before_execution(asynchronous):
+ class ShortcutEngine(Engine):
+ def complete_legacy(self, *args, **kwargs):
+ raise AssertionError("A warnings-as-errors policy must prevent execution")
+
+ class AsyncShortcutEngine(AsyncEngine):
+ async def complete_legacy(self, *args, **kwargs):
+ raise AssertionError("A warnings-as-errors policy must prevent execution")
+
+ engine = ShortcutEngine()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncShortcutEngine(engine), cache=False)
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", DeprecationWarning)
+ with pytest.raises(DeprecationWarning, match="complete_legacy"):
+ if asynchronous:
+ await lm.acall("hello")
+ else:
+ lm("hello")
+ assert not engine.requests
+ assert not lm.history
diff --git a/tests/clients/test_lm_migration_compatibility.py b/tests/clients/test_lm_migration_compatibility.py
new file mode 100644
index 000000000..726aea6d5
--- /dev/null
+++ b/tests/clients/test_lm_migration_compatibility.py
@@ -0,0 +1,185 @@
+"""Public behavior captured before replacing the LM backend; no provider calls."""
+
+import copy
+import importlib
+import json
+import zipfile
+from pathlib import Path
+
+import litellm
+import pytest
+from pydantic import BaseModel
+
+import dspy
+from dspy.clients.cache import Cache
+from dspy.utils.callback import BaseCallback
+from dspy.utils.usage_tracker import track_usage
+
+FIXTURES = Path(__file__).parent / "fixtures" / "lm_3_3_0"
+MANIFEST = json.loads((FIXTURES / "manifest.json").read_text())
+CASES = MANIFEST["cases"]
+
+
+def plain(value):
+ # Read fields, not SDK serializers: older pickled LiteLLM models can carry
+ # deferred Pydantic serializers even though their public fields are usable.
+ if isinstance(value, BaseModel):
+ value = dict(value)
+ if isinstance(value, dict):
+ return {key: plain(item) for key, item in value.items()}
+ if isinstance(value, (list, tuple)):
+ return [plain(item) for item in value]
+ return value
+
+
+class Trace(BaseCallback):
+ def __init__(self):
+ self.events = []
+ self.errors = []
+
+ def on_lm_start(self, call_id, instance, inputs):
+ self.events.append(("start", call_id))
+
+ def on_lm_end(self, call_id, outputs, exception):
+ self.events.append(("end", call_id))
+ self.errors.append(exception)
+
+
+def response_for(case):
+ response = litellm.ModelResponse(**copy.deepcopy(case["provider_response"]))
+ response._hidden_params["response_cost"] = 0.01
+ return response
+
+
+def backend(monkeypatch):
+ """Only replace the transport boundary, never LM.forward or the cache."""
+ calls = []
+ state = {"case": CASES[0], "forbid": False}
+
+ def complete(**kwargs):
+ if state["forbid"]:
+ pytest.fail("A saved cache hit must not call the provider")
+ calls.append(copy.deepcopy(kwargs))
+ return response_for(state["case"])
+
+ async def acomplete(**kwargs):
+ return complete(**kwargs)
+
+ module = importlib.import_module("dspy.clients.lm")
+ provider = module._get_litellm()
+ monkeypatch.setattr(provider, "completion", complete)
+ monkeypatch.setattr(provider, "acompletion", acomplete)
+ return calls, state
+
+
[email protected]("case", CASES, ids=lambda case: case["name"])
[email protected]("saved", [False, True], ids=["cold", "3.3-disk-hit"])
+async def test_calls_match_3_3_outputs_keys_and_bookkeeping(case, saved, backend, monkeypatch, tmp_path):
+ calls, state = backend
+ state.update(case=case, forbid=saved)
+ if saved:
+ # Trusted, repository-owned fixtures only. Extract into a disposable
+ # directory: reading a disk cache updates SQLite metadata.
+ with zipfile.ZipFile(FIXTURES / f"{case['name']}.zip") as archive:
+ archive.extractall(tmp_path)
+ cache = Cache(True, not saved, str(tmp_path))
+ monkeypatch.setattr(dspy, "cache", cache)
+ trace = Trace()
+ lm = dspy.LM(engine="litellm", **case["init"], callbacks=[trace])
+ try:
+ with track_usage() as usage:
+ outputs = lm(**case["call"]) if case["mode"] == "sync" else await lm.acall(**case["call"])
+ expected = "warm" if saved else "cold"
+ assert plain(outputs) == case["outputs"]
+ assert usage.get_total_tokens() == case[f"{expected}_usage"]
+ assert len(lm.history) == 1
+ history = lm.history[0]
+ assert plain({key: history[key] for key in MANIFEST["history_keys"]}) == case[f"{expected}_history"]
+ assert history["response"] is not None
+ assert history["timestamp"] and history["uuid"]
+ assert len(calls) == (0 if saved else 1)
+ assert trace.events[0][0] == "start" and trace.events[1][0] == "end"
+ assert len(trace.events) == 2 and trace.events[0][1] == trace.events[1][1]
+ assert trace.errors == [None]
+ if not saved:
+ assert list(cache.memory_cache) == [case["cache_key"]]
+ if case["name"].startswith("rich_n"):
+ assert len(outputs) == 2
+ assert outputs[0]["tool_calls"][0].function.name == "weather"
+ finally:
+ cache.disk_cache.close()
+
+
[email protected]("mode", ["sync", "async"])
+async def test_cache_controls_and_usage_are_per_call(mode, backend):
+ calls, _ = backend
+ trace = Trace()
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", temperature=0.7, max_tokens=32, callbacks=[trace])
+
+ async def invoke(**kwargs):
+ return lm("Say hello.", **kwargs) if mode == "sync" else await lm.acall("Say hello.", **kwargs)
+
+ with track_usage() as usage:
+ first = await invoke()
+ assert await invoke() == first
+ assert len(calls) == 1
+ await invoke(cache=False)
+ assert len(calls) == 2
+ await invoke()
+ assert len(calls) == 2
+ await invoke(rollout_id=11)
+ await invoke(rollout_id=11)
+ assert len(calls) == 3
+ await invoke(rollout_id=12)
+ assert len(calls) == 4
+ assert usage.get_total_tokens()[lm.model]["total_tokens"] == 4 * 20
+ assert len(lm.history) == 7
+ assert len(trace.events) == 14
+ assert trace.errors == [None] * 7
+ for start, end in zip(trace.events[::2], trace.events[1::2]):
+ assert start[0] == "start" and end == ("end", start[1])
+ assert all("rollout_id" not in call for call in calls)
+
+
[email protected]("override", [{"n": 2}, {"temperature": 0.8}, {"max_tokens": 33}])
+def test_generation_changes_do_not_reuse_the_original_cache_entry(override, backend):
+ calls, _ = backend
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", temperature=0.7, max_tokens=32)
+ lm("Say hello.")
+ lm("Say hello.", **override)
+ lm("Say hello.", **override)
+ assert len(calls) == 2
+
+
[email protected]("mode", ["sync", "async"])
+async def test_legacy_plugin_works_through_chat_adapter(mode):
+ class LegacyLM(dspy.BaseLM):
+ forward_contract = "legacy"
+
+ def forward(self, prompt=None, messages=None, **kwargs):
+ self.received = (prompt, messages, kwargs)
+ return litellm.ModelResponse(
+ model=self.model,
+ choices=[{"message": {"role": "assistant", "content":
+ "[[ ## answer ## ]]\nyes\n\n[[ ## completed ## ]]"}, "finish_reason": "stop"}],
+ usage={"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
+ )
+
+ async def aforward(self, prompt=None, messages=None, **kwargs):
+ return self.forward(prompt, messages, **kwargs)
+
+ lm = LegacyLM("legacy-test")
+ adapter = dspy.ChatAdapter()
+ kwargs = dict(lm=lm, lm_kwargs={"temperature": 0.4}, signature=dspy.Signature("question -> answer"),
+ demos=[], inputs={"question": "Can you answer?"})
+ result = adapter(**kwargs) if mode == "sync" else await adapter.acall(**kwargs)
+ assert result == [{"answer": "yes"}]
+ prompt, messages, parameters = lm.received
+ assert prompt is None
+ assert messages[-1]["role"] == "user" and "Can you answer?" in messages[-1]["content"]
+ assert parameters["temperature"] == 0.4
+ assert len(lm.history) == 1
diff --git a/tests/clients/test_lm_schema_conversion.py b/tests/clients/test_lm_schema_conversion.py
new file mode 100644
index 000000000..f0fe7f019
--- /dev/null
+++ b/tests/clients/test_lm_schema_conversion.py
@@ -0,0 +1,48 @@
+"""Generated Pydantic schemas receive the same preparation on both LM paths."""
+
+import copy
+
+import pydantic
+import pytest
+
+import dspy
+from dspy.clients.execution import _canonical, prepare
+from dspy.clients.legacy_requests import chat_to_responses
+
+
[email protected]("asynchronous", [False, True])
+def test_generated_nested_schema_matches_legacy_preparation(asynchronous):
+ class Item(pydantic.BaseModel):
+ name: str
+
+ class Answer(pydantic.BaseModel):
+ items: list[Item]
+ alternative: Item | str
+
+ original = copy.deepcopy(Answer.model_json_schema())
+ lm = dspy.LM("openai/gpt-4.1-mini", model_type="responses", engine="lm15", cache=False)
+ call = prepare(lm, "Return an answer.", None, {"response_format": Answer}, asynchronous=asynchronous)
+ canonical = _canonical(call)
+ schema = canonical.config.response_format["schema"]
+ legacy = chat_to_responses(call.legacy)["text"]["format"]["schema"]
+
+ assert schema == legacy
+ assert schema["additionalProperties"] is False
+ assert schema["$defs"]["Item"]["additionalProperties"] is False
+ assert canonical.config.response_format["strict"] is True
+ assert Answer.model_json_schema() == original
+ assert "additionalProperties" not in original
+
+
[email protected]("additional_properties", [None, True, False])
+def test_raw_schema_is_not_rewritten(additional_properties):
+ schema = {"type": "object", "properties": {"word": {"type": "string"}}}
+ if additional_properties is not None:
+ schema["additionalProperties"] = additional_properties
+ format_ = {"type": "json_schema", "json_schema": {"name": "Answer", "schema": schema, "strict": True}}
+ original = copy.deepcopy(format_)
+ lm = dspy.LM("openai/gpt-4.1-mini", model_type="responses", engine="lm15", cache=False)
+ call = prepare(lm, "Return an answer.", None, {"response_format": format_})
+
+ assert _canonical(call).config.response_format["schema"] == original["json_schema"]["schema"]
+ assert format_ == original
diff --git a/tests/clients/test_native_metadata.py b/tests/clients/test_native_metadata.py
new file mode 100644
index 000000000..470b86002
--- /dev/null
+++ b/tests/clients/test_native_metadata.py
@@ -0,0 +1,147 @@
+"""Metadata loading, native capability decisions, and provider token accounting."""
+
+import json
+from unittest.mock import patch
+
+import pytest
+
+import dspy
+import dspy.clients.model_metadata as metadata
+from dspy.clients.costs import estimate_cost
+from dspy.lm15 import Config, Message, Request, Response, Usage
+
+
[email protected](autouse=True)
+def local_metadata(monkeypatch):
+ monkeypatch.setenv("LITELLM_LOCAL_MODEL_COST_MAP", "True")
+ monkeypatch.setattr(metadata, "_data", None)
+ monkeypatch.setattr(metadata, "_source", {})
+
+
+def test_snapshot_capabilities_do_not_import_litellm():
+ with patch("dspy.clients.lm._get_litellm", side_effect=AssertionError("LiteLLM accessed")):
+ lm = dspy.LM("openai/gpt-4o", engine="lm15")
+ assert lm.supports_function_calling
+ assert lm.supports_response_schema
+ assert "response_format" in lm.supported_params
+ assert not lm.supports_reasoning
+ claude = dspy.LM("anthropic/claude-sonnet-4-5", engine="lm15")
+ assert claude.supports_reasoning
+ assert claude.supports_response_schema
+ assert metadata.source_info()["is_env_forced"]
+
+
+def test_unknown_and_custom_capabilities():
+ assert not dspy.LM("openai/unlisted").supports_function_calling
+
+ class Engine:
+ supports_function_calling = True
+ supported_params = frozenset({"tools"})
+
+ def complete(self, request):
+ raise AssertionError("No generation during capability lookup")
+
+ lm = dspy.LM("custom", engine=Engine())
+ assert lm.supports_function_calling
+ assert lm.supported_params == {"tools"}
+
+
+def test_lm15_restriction_beats_catalog(monkeypatch):
+ monkeypatch.setattr(metadata, "_data", {"zai/glm-test": {
+ "litellm_provider": "zai", "supports_response_schema": True,
+ "supports_function_calling": True,
+ }})
+ lm = dspy.LM("zai:glm-test", engine="lm15")
+ assert not lm.supports_response_schema
+ assert "response_format" in lm.supported_params # JSON mode, not schema enforcement
+
+
+def test_missing_flags_inherit_but_false_and_prices_do_not(monkeypatch):
+ monkeypatch.setattr(metadata, "_data", {
+ "openai/test": {"supports_reasoning": False},
+ "test": {"litellm_provider": "openai", "supports_reasoning": True,
+ "supports_function_calling": True, "input_cost_per_token": 9},
+ })
+ info = metadata.model_info("openai-chat", "test")
+ assert info["supports_reasoning"] is False
+ assert info["supports_function_calling"] is True
+ assert "input_cost_per_token" not in info
+ info["supports_reasoning"] = True
+ assert metadata.model_info("openai-chat", "test")["supports_reasoning"] is False
+
+
+def test_remote_first_once_and_fallback(monkeypatch):
+ monkeypatch.delenv("LITELLM_LOCAL_MODEL_COST_MAP")
+ with patch.object(metadata.urllib.request, "urlopen", side_effect=OSError("secret")) as fetch:
+ metadata.model_info("openai-chat", "gpt-4o")
+ metadata.model_info("openai-chat", "gpt-4o")
+ assert fetch.call_count == 1
+ assert metadata.source_info()["source"] == "local"
+ assert metadata.source_info()["fallback_reason"] == "OSError"
+
+
+def test_valid_remote_map_is_adopted(monkeypatch):
+ import io
+
+ monkeypatch.delenv("LITELLM_LOCAL_MODEL_COST_MAP")
+ data = metadata._snapshot()
+ data["gpt-4o"]["supports_reasoning"] = True
+ with patch.object(metadata.urllib.request, "urlopen", return_value=io.BytesIO(json.dumps(data).encode())):
+ assert metadata.model_info("openai-chat", "gpt-4o")["supports_reasoning"]
+ assert metadata.source_info()["source"] == "remote"
+
+
+def answer(model, **usage):
+ return Response(id=None, model=model, message=Message.assistant("ok"),
+ finish_reason="stop", usage=Usage(**usage))
+
+
+def test_openai_cached_input_and_reasoning_are_not_double_counted():
+ response = answer("gpt-4o", input_tokens=100, output_tokens=30, cache_read_tokens=40, reasoning_tokens=10)
+ cost, details = estimate_cost(response, provider="openai-chat", requested_model=response.model)
+ assert cost == pytest.approx(60 * 2.5e-6 + 40 * 1.25e-6 + 30 * 1e-5)
+ assert details["kind"] == "estimate"
+
+
+def test_anthropic_disjoint_cache_counters():
+ response = answer("claude-sonnet-4-5", input_tokens=100, output_tokens=30,
+ cache_read_tokens=40, cache_write_tokens=20)
+ cost, _ = estimate_cost(response, provider="anthropic", requested_model=response.model)
+ assert cost == pytest.approx(100 * 3e-6 + 40 * 3e-7 + 20 * 3.75e-6 + 30 * 1.5e-5)
+
+
+def test_gemini_reasoning_is_additional_output():
+ response = answer("gemini-2.5-flash", input_tokens=100, output_tokens=30, reasoning_tokens=10)
+ cost, _ = estimate_cost(response, provider="gemini", requested_model=response.model)
+ assert cost == pytest.approx(100 * 3e-7 + 40 * 2.5e-6)
+
+
+def test_long_context_and_priority_rates(monkeypatch):
+ monkeypatch.setattr(metadata, "_data", {"openai/test": {
+ "input_cost_per_token_above_200k_tokens_priority": 2e-6,
+ "output_cost_per_token_above_200k_tokens_priority": 8e-6,
+ }})
+ response = answer("test", input_tokens=200001, output_tokens=30, reasoning_tokens=10)
+ request = Request(model="test", messages=(Message.user("hi"),), config=Config(service_tier="priority"))
+ cost, _ = estimate_cost(response, provider="openai-chat", requested_model="test", request=request)
+ assert cost == pytest.approx(200001 * 2e-6 + 30 * 8e-6)
+
+
+def test_unknown_price_and_malformed_metadata_do_not_fail_generation(monkeypatch):
+ response = answer("unknown", input_tokens=1, output_tokens=1)
+ assert estimate_cost(response, provider="openai-chat", requested_model="unknown")[0] is None
+ monkeypatch.setattr("dspy.clients.costs.model_info", lambda *args: {"broken": object()})
+ assert estimate_cost(response, provider="openai-chat", requested_model="unknown")[0] is None
+
+
+def test_cost_provenance_survives_cache_record():
+ from dspy.clients.call_result import CallResult, combine
+
+ response = answer("gpt-4o", input_tokens=1, output_tokens=1)
+ result = CallResult.native(response)
+ result.cost, result.cost_details = estimate_cost(response, provider="openai-chat", requested_model=response.model)
+ cached = CallResult.load(result.dump())
+ assert cached.cost == result.cost
+ assert cached.cost_details == result.cost_details
+ assert cached.usage == {}
+ assert combine([result, result], model_type="chat").cost == result.cost * 2
diff --git a/tests/clients/test_prompt_cache.py b/tests/clients/test_prompt_cache.py
new file mode 100644
index 000000000..0056d349b
--- /dev/null
+++ b/tests/clients/test_prompt_cache.py
@@ -0,0 +1,169 @@
+"""The prompt-cache bridge preserves lm15 objects without changing answer caching."""
+
+import json
+
+import pytest
+
+import dspy
+from dspy.clients.execution import _canonical, prepare
+from dspy.lm15 import CacheConfig, Config, Message, Request, Response, Usage
+
+
+class Engine:
+ def __init__(self):
+ self.requests = []
+
+ def complete(self, request):
+ self.requests.append(request)
+ return Response(None, request.model, Message.assistant("ok"), "stop", Usage(input_tokens=1, output_tokens=1))
+
+
+class AsyncEngine:
+ def __init__(self, sync):
+ self.sync = sync
+
+ async def complete(self, request):
+ return self.sync.complete(request)
+
+
[email protected]("asynchronous", [False, True])
+async def test_defaults_overrides_and_typed_requests(asynchronous):
+ engine = Engine()
+ stable = CacheConfig(prefix="stable")
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), cache=False, prompt_cache=stable)
+
+ async def call(*args, **kwargs):
+ return await lm.acall(*args, **kwargs) if asynchronous else lm(*args, **kwargs)
+
+ assert await call("hello") == ["ok"]
+ assert engine.requests[-1].config.cache == stable
+ override = CacheConfig(prefix="history")
+ await call("hello", prompt_cache=override)
+ assert engine.requests[-1].config.cache == override
+ await call("hello", prompt_cache=None)
+ assert engine.requests[-1].config.cache is None
+ assert lm.kwargs["prompt_cache"] == stable
+ request = Request(model=lm.model, messages=(Message.user("typed"),), config=Config())
+ assert isinstance(await call(request), Response)
+ assert engine.requests[-1].config.cache is None # typed requests own their complete config
+ with pytest.raises(TypeError, match=r"Request\.config"):
+ await call(request, prompt_cache=override)
+
+
+def test_copy_and_json_state_roundtrip():
+ config = CacheConfig(prefix="stable", retention="long", key="prefix")
+ lm = dspy.LM("openai/gpt-4o", engine="lm15", prompt_cache=config)
+ restored = dspy.BaseLM.load_state(json.loads(json.dumps(lm.dump_state())))
+ assert restored.kwargs["prompt_cache"] == config
+ assert lm.copy().kwargs["prompt_cache"] == config
+ assert "prompt_cache" not in lm.copy(prompt_cache=None).kwargs
+ assert lm.kwargs["prompt_cache"] == config
+ assert lm.copy(prompt_cache=CacheConfig(mode="off")).kwargs["prompt_cache"].mode == "off"
+
+
[email protected]("value", [True, "stable", {"prefix": "stable"}])
+def test_invalid_values_are_rejected(value):
+ with pytest.raises(TypeError, match="CacheConfig"):
+ dspy.LM("openai/gpt-4o", prompt_cache=value)
+ lm = dspy.LM("custom", engine=Engine(), cache=False)
+ with pytest.raises(TypeError, match="CacheConfig"):
+ lm("hello", prompt_cache=value)
+ with pytest.raises(TypeError, match="CacheConfig"):
+ lm.copy(prompt_cache=value)
+
+
+def test_answer_cache_keys_serialize_and_separate_prompt_policies():
+ lm = dspy.LM("openai/gpt-4o")
+ plain = prepare(lm, "hello", None, {}).key(lm, False)
+ disabled = prepare(lm, "hello", None, {"prompt_cache": None}).key(lm, False)
+ stable = prepare(lm, "hello", None, {"prompt_cache": CacheConfig(prefix="stable")}).key(lm, False)
+ history = prepare(lm, "hello", None, {"prompt_cache": CacheConfig(prefix="history")}).key(lm, False)
+ assert plain == disabled
+ assert dspy.cache.cache_key(stable) != dspy.cache.cache_key(history)
+ assert json.loads(json.dumps(stable))["prompt_cache"] == {"mode": "auto", "prefix": "stable"}
+
+
[email protected]("asynchronous", [False, True])
+async def test_answer_caching_still_works(asynchronous):
+ engine = Engine()
+ lm = dspy.LM("custom", engine=engine, async_engine=AsyncEngine(engine), prompt_cache=CacheConfig(prefix="stable"))
+ for _ in range(2):
+ result = await lm.acall("hello") if asynchronous else lm("hello")
+ assert result == ["ok"]
+ assert len(engine.requests) == 1
+ assert lm.history[-1]["usage"] == {}
+
+
[email protected]("options", [
+ {"engine": "litellm"}, {"engine": "auto", "headers": {"x-test": "yes"}},
+ {"engine": "auto", "model_type": "text"},
+])
+def test_no_compatibility_fallback(options, monkeypatch):
+ from dspy.clients.engines import LiteLLMEngine
+
+ def forbidden(*args, **kwargs):
+ raise AssertionError("prompt_cache must not reach LiteLLM")
+
+ monkeypatch.setattr(LiteLLMEngine, "complete_legacy", forbidden)
+ lm = dspy.LM("openai/gpt-4o", cache=False, prompt_cache=CacheConfig(prefix="stable"), **options)
+ with pytest.raises(dspy.LMUnsupportedFeatureError):
+ lm("hello")
+
+
+def test_unrepresentable_input_does_not_fallback_with_prompt_cache(monkeypatch):
+ from dspy.clients.engines import LiteLLMEngine
+
+ def forbidden(*args, **kwargs):
+ raise AssertionError("Compatibility fallback attempted")
+
+ monkeypatch.setattr(LiteLLMEngine, "complete_legacy", forbidden)
+ lm = dspy.LM("openai/gpt-4o", cache=False, prompt_cache=CacheConfig(prefix="stable"))
+ with pytest.raises(dspy.LMUnsupportedFeatureError):
+ lm("hello", prediction={"type": "content", "content": "hello"})
+
+
+def test_provider_cache_options_cannot_conflict():
+ lm = dspy.LM("openai/gpt-4o", prompt_cache=CacheConfig(prefix="stable"))
+ call = prepare(lm, "hello", None, {"prompt_cache_key": "other"})
+ with pytest.raises(dspy.LMUnsupportedFeatureError, match="Do not combine"):
+ _canonical(call)
+
+
+def test_none_is_not_forwarded_to_litellm(monkeypatch):
+ from dspy.clients.call_result import CallResult
+ from dspy.clients.engines import LiteLLMEngine
+
+ def complete(self, lm, request, **context):
+ assert "prompt_cache" not in request
+ return CallResult(outputs=["ok"], response_model=lm.model)
+
+ monkeypatch.setattr(LiteLLMEngine, "complete_legacy", complete)
+ assert dspy.LM("openai/gpt-4o", engine="litellm", cache=False)("hello", prompt_cache=None) == ["ok"]
+
+
+def test_native_anthropic_marks_system_prefix(monkeypatch):
+ import dspy.clients.execution as execution
+ from dspy._vendor.lm15.testing import FakeResponse, FakeTransport
+ from dspy.lm15 import RouterConfig
+
+ transport = FakeTransport([FakeResponse(status=200, body=json.dumps({
+ "id": "answer", "model": "claude-haiku-4-5", "stop_reason": "end_turn",
+ "content": [{"type": "text", "text": "ok"}],
+ "usage": {"input_tokens": 1, "output_tokens": 1, "cache_creation_input_tokens": 100},
+ }).encode())])
+ monkeypatch.setattr(execution, "RouterConfig", lambda **kwargs: RouterConfig(**kwargs, transport=transport))
+ lm = dspy.LM("anthropic/claude-haiku-4-5", engine="lm15", api_key="fake", cache=False,
+ prompt_cache=CacheConfig(prefix="stable"))
+ try:
+ assert lm(messages=[{"role": "system", "content": "stable instructions"},
+ {"role": "user", "content": "changing input"}]) == ["ok"]
+ payload = json.loads(transport.requests[0].body)
+ assert payload["system"] == [{"type": "text", "text": "stable instructions",
+ "cache_control": {"type": "ephemeral"}}]
+ assert "prompt_cache" not in payload
+ assert "cache_control" not in payload["messages"][0]["content"][0]
+ assert lm.history[-1]["usage"]["prompt_tokens_details"]["cache_creation_tokens"] == 100
+ finally:
+ lm.close()
diff --git a/tests/core/test_types.py b/tests/core/test_types.py
index 60d767971..dc0b368b8 100644
--- a/tests/core/test_types.py
+++ b/tests/core/test_types.py
@@ -1,496 +1,28 @@
-import json
+"""The retired experimental vocabulary must not remain as misleading aliases."""
-import pydantic
-import pytest
-
-from dspy.core.types import (
- LMAudioPart,
- LMBinaryPart,
- LMConfig,
- LMDocumentPart,
- LMHistoryEntry,
- LMImagePart,
- LMMessage,
- LMOutput,
- LMOutputBuilder,
- LMPromptCacheConfig,
- LMReasoningConfig,
- LMRequest,
- LMRequestPatch,
- LMResponse,
- LMStreamDeltaEvent,
- LMTextDelta,
- LMTextPart,
- LMThinkingDelta,
- LMThinkingPart,
- LMToolCallDelta,
- LMToolCallPart,
- LMToolChoice,
- LMToolResultPart,
- LMUsage,
- LMVideoPart,
- User,
-)
-
-
-def _history_entry(message: LMMessage) -> LMHistoryEntry:
- return LMHistoryEntry(
- request=LMRequest(model="model", messages=[message]),
- response=LMResponse.from_text("ok"),
- timestamp="timestamp",
- uuid="uuid",
- )
-
-
-def test_message_content_and_tool_calls_normalize_for_dspy_history_surface():
- message = LMMessage(
- role="assistant",
- content="Use search.",
- tool_calls=[
- {
- "id": "call_1",
- "function": {"name": "search", "arguments": '{"query": "dspy"}'},
- }
- ],
- )
-
- assert message.text == "Use search."
- assert [part for part in message.parts if isinstance(part, LMToolCallPart)] == [
- LMToolCallPart(id="call_1", name="search", args={"query": "dspy"})
- ]
- assert _history_entry(message).messages == [
- {
- "role": "assistant",
- "content": "Use search.",
- "tool_calls": [
- {
- "type": "function",
- "function": {"name": "search", "arguments": '{"query": "dspy"}'},
- "id": "call_1",
- }
- ],
- }
- ]
-
-
-def test_tool_result_content_none_normalizes_to_empty_parts():
- assert LMToolResultPart(content=None).content == []
-
- message = LMMessage(role="tool", content=None, tool_call_id="call_1", name="search")
- result = message.parts[0]
-
- assert isinstance(result, LMToolResultPart)
- assert result.call_id == "call_1"
- assert result.name == "search"
- assert result.content == []
- assert _history_entry(message).messages == [
- {"role": "tool", "content": "", "tool_call_id": "call_1", "name": "search"}
- ]
-
-
-def test_audio_content_accepts_url_and_history_preserves_url():
- message = LMMessage(
- role="user",
- content=[
- {
- "type": "input_audio",
- "input_audio": {"url": "https://example.com/audio.wav", "format": "wav"},
- }
- ],
- )
-
- audio = message.parts[0]
- assert isinstance(audio, LMAudioPart)
- assert audio.url == "https://example.com/audio.wav"
- assert _history_entry(message).messages[0]["content"][0]["input_audio"] == {
- "format": "wav",
- "url": "https://example.com/audio.wav",
- }
-
-
-def test_audio_content_defaults_null_format_to_wav():
- message = LMMessage(
- role="user",
- content=[
- {
- "type": "input_audio",
- "input_audio": {"url": "https://example.com/audio.wav", "format": None},
- }
- ],
- )
-
- audio = message.parts[0]
-
- assert isinstance(audio, LMAudioPart)
- assert audio.media_type == "audio/wav"
-
-
-def test_image_content_requires_mapping_with_url():
- with pytest.raises(TypeError, match="Image content block"):
- LMMessage(role="user", content=[{"type": "image_url", "image_url": "https://example.com/image.png"}])
-
- with pytest.raises(ValueError, match="requires url"):
- LMMessage(role="user", content=[{"type": "image_url", "image_url": {}}])
-
-
-def test_video_data_round_trips_through_history_messages():
- message = User(LMVideoPart(data="YWJj", media_type="video/mp4"))
- content = _history_entry(message).messages[0]["content"][0]
- round_tripped = LMMessage(role="user", content=[content]).parts[0]
-
- assert content == {
- "type": "video",
- "video": {"media_type": "video/mp4", "data": "data:video/mp4;base64,YWJj"},
- }
- assert isinstance(round_tripped, LMVideoPart)
- assert round_tripped.data == "YWJj"
- assert round_tripped.media_type == "video/mp4"
-
-
-def test_tool_dicts_coerce_and_emit_strict_and_extras():
- from dspy.clients.openai_format import tool_to_openai, tool_to_openai_responses
-
- nested = LMRequest.from_call(
- model="openai/gpt-5-nano",
- messages=[{"role": "user", "content": "hi"}],
- tools=[
- {
- "type": "function",
- "function": {"name": "f", "parameters": {}, "strict": True, "vendor_key": 1},
- }
- ],
- ).tools[0]
- assert nested.strict is True
- assert nested.provider_data == {"vendor_key": 1}
- assert tool_to_openai(nested) == {
- "type": "function",
- "function": {"name": "f", "parameters": {}, "strict": True, "vendor_key": 1},
- }
- assert tool_to_openai_responses(nested) == {
- "type": "function",
- "name": "f",
- "parameters": {},
- "strict": True,
- "vendor_key": 1,
- }
-
- flat = LMRequest.from_call(
- model="openai/gpt-5-nano",
- messages=[{"role": "user", "content": "hi"}],
- tools=[{"type": "function", "name": "f", "parameters": {}, "strict": False, "custom_field": "x"}],
- ).tools[0]
- assert flat.strict is False
- assert flat.provider_data == {"custom_field": "x"}
-
-
-def test_parsed_tool_call_replays_with_call_id_not_item_id():
- from dspy.clients.openai_format import assistant_tool_call_to_openai, tool_call_to_responses_input
-
- message = LMMessage(
- role="assistant",
- content=None,
- tool_calls=[
- {
- "id": "fc_1",
- "type": "function_call",
- "name": "get_weather",
- "arguments": json.dumps({"city": "Paris"}),
- "call_id": "call_1",
- "status": "completed",
- }
- ],
- )
- part = message.parts[0]
- assert isinstance(part, LMToolCallPart)
- assert part.id == "call_1"
- assert part.name == "get_weather"
- assert part.args == {"city": "Paris"}
-
- part.provider_data = {"id": "fc_1", "call_id": "call_1", "status": "completed", "raw_arguments": "{}"}
-
- chat_call = assistant_tool_call_to_openai(part)
- assert chat_call["id"] == "call_1"
- assert "status" not in chat_call and "raw_arguments" not in chat_call
-
- responses_item = tool_call_to_responses_input(part)
- assert responses_item["call_id"] == "call_1"
- assert responses_item["arguments"] == json.dumps({"city": "Paris"})
-
-
-def test_tool_provider_extras_cannot_override_canonical_fields():
- from dspy.clients.openai_format import tool_to_openai, tool_to_openai_responses
-
- tool = LMRequest.from_call(
- model="openai/gpt-5-nano",
- messages=[{"role": "user", "content": "hi"}],
- tools=[
- {
- "type": "function",
- "name": "safe_name",
- "parameters": {"type": "object"},
- "strict": True,
- "provider_data": {
- "type": "web_search",
- "name": "overridden_name",
- "parameters": {"type": "string"},
- "strict": False,
- "vendor_key": 1,
- },
- }
- ],
- ).tools[0]
-
- assert tool_to_openai(tool) == {
- "type": "function",
- "function": {
- "name": "safe_name",
- "parameters": {"type": "object"},
- "strict": True,
- "vendor_key": 1,
- },
- }
- assert tool_to_openai_responses(tool) == {
- "type": "function",
- "name": "safe_name",
- "parameters": {"type": "object"},
- "strict": True,
- "vendor_key": 1,
- }
-
-
-def test_close_object_schemas_ignores_schema_shaped_defaults():
- from dspy.clients.openai_format import response_format_to_responses
-
- class WithDictDefault(pydantic.BaseModel):
- payload: dict = {"type": "object"}
-
- schema = response_format_to_responses(WithDictDefault)["schema"]
- assert schema["additionalProperties"] is False
- assert schema["properties"]["payload"]["default"] == {"type": "object"}
-
-
-def test_file_content_block_with_data_and_id_round_trips_losslessly_to_responses_format():
- from dspy.clients.openai_format import to_openai_responses_request
-
- message = LMMessage(
- role="user",
- content=[
- {
- "type": "file",
- "file": {
- "file_data": "data:application/pdf;base64,JVBERi0xLjQ=",
- "file_id": "file_123",
- "filename": "report.pdf",
- },
- }
- ],
- )
- part = message.parts[0]
-
- assert isinstance(part, LMBinaryPart)
- assert part.metadata["legacy_content_block"] == {
- "type": "file",
- "file": {
- "file_data": "data:application/pdf;base64,JVBERi0xLjQ=",
- "file_id": "file_123",
- "filename": "report.pdf",
- },
- }
-
- request = LMRequest(model="model", messages=[message])
- content = to_openai_responses_request(request)["input"][0]["content"]
-
- assert content == [
- {
- "type": "input_file",
- "file_data": "data:application/pdf;base64,JVBERi0xLjQ=",
- "file_id": "file_123",
- "filename": "report.pdf",
- }
- ]
-
-
-def test_document_source_url_stays_url_and_round_trips_through_history_messages():
- message = LMMessage(
- role="user",
- content=[
- {
- "type": "document",
- "source": "https://example.com/report.pdf",
- "title": "Report",
- }
- ],
- )
- document = message.parts[0]
- content = _history_entry(message).messages[0]["content"][0]
- round_tripped = LMMessage(role="user", content=[content]).parts[0]
-
- assert isinstance(document, LMDocumentPart)
- assert document.url == "https://example.com/report.pdf"
- assert document.data is None
- assert isinstance(round_tripped, LMDocumentPart)
- assert round_tripped.url == "https://example.com/report.pdf"
- assert round_tripped.data is None
-
-
-def test_config_extensions_flatten_when_converted_to_legacy_kwargs():
- config = LMConfig.from_kwargs(temperature=0.2, provider_flag=True)
- patch = LMRequestPatch(config=config)
- request = LMRequest.from_call(model="model", prompt="hi", temperature=0.2, provider_flag=True)
- entry = LMHistoryEntry(
- request=request,
- response=LMResponse.from_text("ok"),
- timestamp="timestamp",
- uuid="uuid",
- )
-
- assert patch.as_lm_kwargs() == {"provider_flag": True, "temperature": 0.2}
- assert entry.kwargs == {"provider_flag": True, "temperature": 0.2}
-
-
-def test_lm_kwargs_aliases_normalize_for_existing_dspy_lm_callers():
- config = LMConfig.from_kwargs(
- reasoning={"effort": "low", "summary": "auto"},
- reasoning_effort="high",
- tool_choice={"mode": "auto"},
- parallel_tool_calls=False,
- cache=True,
- rollout_id=7,
- prompt_cache=True,
- prompt_cache_key="prompt-cache",
- provider_flag=True,
- )
-
- assert config.reasoning.effort == "high"
- assert config.reasoning.summary == "auto"
- assert config.tool_choice.mode == "auto"
- assert config.tool_choice.parallel is False
- assert config.cache.enabled is True
- assert config.cache.rollout_id == 7
- assert config.prompt_cache.enabled is True
- assert config.prompt_cache.key == "prompt-cache"
- assert config.extensions == {"provider_flag": True}
-
-
-def test_nested_config_aliases_remain_supported_for_existing_interfaces():
- reasoning = LMReasoningConfig.from_value({"reasoning_effort": "high"})
- tool_choice = LMToolChoice.from_value({"parallel_tool_calls": False})
- prompt_cache = LMPromptCacheConfig.from_value({"prompt_cache_key": "prompt-cache"})
-
- assert reasoning.effort == "high"
- assert tool_choice.mode == "auto"
- assert tool_choice.parallel is False
- assert prompt_cache.key == "prompt-cache"
-
-
-def test_usage_normalizes_existing_user_visible_token_aliases():
- provider_usage = LMUsage(prompt_tokens=1, completion_tokens=2)
- canonical_usage = LMUsage(input_tokens=1, output_tokens=2)
-
- assert provider_usage.input_tokens == 1
- assert provider_usage.output_tokens == 2
- assert provider_usage.total_tokens == 3
- assert canonical_usage.prompt_tokens == 1
- assert canonical_usage.completion_tokens == 2
- assert canonical_usage.total_tokens == 3
-
-
-def test_default_config_does_not_serialize_empty_stop_sequences():
- request = LMRequest.from_call(model="model", prompt="hi")
- entry = LMHistoryEntry(
- request=request,
- response=LMResponse.from_text("ok"),
- timestamp="timestamp",
- uuid="uuid",
- )
-
- assert request.config.stop is None
- assert entry.kwargs == {}
-
-
-def test_history_entry_exposes_existing_dict_style_properties():
- message = User("hi")
- request = LMRequest.from_call(model="model", messages=[message], temperature=0.2)
- response = LMResponse.from_text("ok", model="response-model", usage={"input_tokens": 1}, cost=0.5)
- entry = LMHistoryEntry(request=request, response=response, timestamp="timestamp", uuid="uuid")
-
- assert entry.model == "model"
- assert entry.prompt == "hi"
- assert entry.messages == [{"role": "user", "content": "hi"}]
- assert entry.outputs == ["ok"]
- assert entry.usage["input_tokens"] == 1
- assert entry.cost == 0.5
- assert entry.kwargs == {"temperature": 0.2}
- assert entry.response_model == "response-model"
- assert entry["messages"] == [{"role": "user", "content": "hi"}]
- assert entry["kwargs"] == {"temperature": 0.2}
-
-
-def test_response_rejects_empty_outputs():
- with pytest.raises(pydantic.ValidationError):
- LMResponse(model="model", outputs=[])
-
-
-def test_response_to_outputs_preserves_multipart_values():
- image = LMImagePart(url="https://example.com/image.png")
- response = LMResponse(
- outputs=[
- LMOutput(parts=[LMTextPart(text="part one"), LMTextPart(text="part two")]),
- LMOutput(parts=[LMTextPart(text="caption"), image]),
- ]
- )
-
- assert response.to_outputs() == [["part one", "part two"], ["caption", image]]
-
-
-def test_output_to_value_preserves_redacted_thinking_part():
- thinking = LMThinkingPart(text="hidden", redacted=True)
- output = LMOutput(parts=[thinking])
-
- assert output.to_value() == [thinking]
-
-
-def test_stream_event_indices_must_be_non_negative():
- with pytest.raises(pydantic.ValidationError):
- LMStreamDeltaEvent(output_index=-1, part_index=0, delta=LMTextDelta(text="x"))
-
- with pytest.raises(pydantic.ValidationError):
- LMStreamDeltaEvent(output_index=0, part_index=-1, delta=LMTextDelta(text="x"))
-
-
-def test_stream_builder_rejects_sparse_output_indices():
- builder = LMOutputBuilder()
- builder.apply(LMStreamDeltaEvent(output_index=2, part_index=0, delta=LMTextDelta(text="third")))
-
- with pytest.raises(ValueError, match="output indices"):
- builder.to_response()
+import importlib
+import pytest
-def test_stream_builder_rejects_sparse_part_indices():
- builder = LMOutputBuilder()
- builder.apply(LMStreamDeltaEvent(output_index=0, part_index=1, delta=LMTextDelta(text="second")))
+import dspy
+from dspy.lm15 import Message, Request, Response
- with pytest.raises(ValueError, match="part indices"):
- builder.to_response()
+def test_old_types_import_explains_migration():
+ with pytest.raises(ImportError, match="dspy.lm15"):
+ importlib.import_module("dspy.core.types")
-def test_stream_builder_rejects_delta_type_changes():
- builder = LMOutputBuilder()
- builder.apply(LMStreamDeltaEvent(output_index=0, part_index=0, delta=LMTextDelta(text="text")))
- with pytest.raises(ValueError, match="thinking delta"):
- builder.apply(LMStreamDeltaEvent(output_index=0, part_index=0, delta=LMThinkingDelta(text="thought")))
+def test_old_public_type_names_are_removed():
+ for name in ("LMRequest", "LMResponse", "LMMessage", "LMConfig", "System", "User", "Assistant", "Developer", "ToolCall", "ToolResult"):
+ assert not hasattr(dspy, name)
-def test_stream_builder_rejects_incomplete_tool_call_arguments():
- builder = LMOutputBuilder()
- builder.apply(
- LMStreamDeltaEvent(
- output_index=0,
- part_index=0,
- delta=LMToolCallDelta(id="call_1", name="search", args_delta='{"query": '),
- )
- )
+def test_bundled_types_are_not_dspy_reimplementations():
+ from dspy._vendor import lm15
- with pytest.raises(ValueError, match="tool-call arguments"):
- builder.to_response()
+ assert Request is lm15.Request
+ assert Response is lm15.Response
+ request = Request(model="example", system="Instructions", messages=(Message.user("Hello"),))
+ assert request.messages[0].text == "Hello"
+ assert request.system == "Instructions"
diff --git a/tests/evaluate/test_evaluate.py b/tests/evaluate/test_evaluate.py
index db0688aac..bb90e471e 100644
--- a/tests/evaluate/test_evaluate.py
+++ b/tests/evaluate/test_evaluate.py
@@ -130,40 +130,58 @@ def test_multithread_evaluate_call():
def test_multi_thread_evaluate_call_cancelled(monkeypatch):
- # slow LM that sleeps for 1 second before returning the answer
- class SlowLM(DummyLM):
- def __call__(self, *args, **kwargs):
- import time
+ from dspy.utils import parallelizer
- time.sleep(1)
- return super().__call__(*args, **kwargs)
+ started = threading.Barrier(3, timeout=10) # two workers and the main thread
+ release = threading.Event()
+ worker_finished = threading.Event()
+ pools = []
+ running = []
+ original_pool = parallelizer.ThreadPoolExecutor
- dspy.configure(lm=SlowLM({"What is 1+1?": {"answer": "2"}, "What is 2+2?": {"answer": "4"}}))
+ def make_pool(*args, **kwargs):
+ pool = original_pool(*args, **kwargs)
+ pools.append(pool)
+ return pool
+ class SlowLM(DummyLM):
+ def __call__(self, *args, **kwargs):
+ try:
+ started.wait()
+ if not release.wait(10):
+ raise TimeoutError("Test did not release the evaluation worker")
+ return super().__call__(*args, **kwargs)
+ finally:
+ worker_finished.set()
+
+ def interrupt_when_running(futures, **kwargs):
+ running.extend(futures)
+ started.wait()
+ # Deliver a real SIGINT with the evaluation handler installed, without
+ # a timer thread that could signal an unrelated test later.
+ signal.raise_signal(signal.SIGINT)
+ pytest.fail("SIGINT did not interrupt evaluation")
+
+ monkeypatch.setattr(parallelizer, "ThreadPoolExecutor", make_pool)
+ monkeypatch.setattr(parallelizer, "wait", interrupt_when_running)
+ dspy.configure(lm=SlowLM({"What is 1+1?": {"answer": "2"}, "What is 2+2?": {"answer": "4"}}))
devset = [new_example("What is 1+1?", "2"), new_example("What is 2+2?", "4")]
program = Predict("question -> answer")
- assert program(question="What is 1+1?").answer == "2"
-
- # spawn a thread that will sleep for .1 seconds then send a KeyboardInterrupt
- def sleep_then_interrupt():
- import time
-
- time.sleep(0.1)
- import os
-
- os.kill(os.getpid(), signal.SIGINT)
-
- input_thread = threading.Thread(target=sleep_then_interrupt)
- input_thread.start()
-
- with pytest.raises(KeyboardInterrupt):
- ev = Evaluate(
- devset=devset,
- metric=answer_exact_match,
- display_progress=False,
- num_threads=2,
- )
- ev(program)
+ ev = Evaluate(devset=devset, metric=answer_exact_match, display_progress=False, num_threads=2)
+ try:
+ with pytest.raises(KeyboardInterrupt):
+ ev(program)
+ # Cancellation must return while the LM calls are still blocked.
+ assert not worker_finished.is_set()
+ assert len(running) == 2 and all(future.running() for future in running)
+ finally:
+ started.abort()
+ release.set()
+ # Production intentionally uses shutdown(wait=False). The test owns
+ # these workers and must drain them before fixtures/patches are reset.
+ for pool in pools:
+ pool.shutdown(wait=True, cancel_futures=True)
+ assert all(future.done() for future in running)
def test_evaluate_call_wrong_answer():
diff --git a/tests/predict/test_chain_of_thought.py b/tests/predict/test_chain_of_thought.py
index a8dbb70ba..c19100dd5 100644
--- a/tests/predict/test_chain_of_thought.py
+++ b/tests/predict/test_chain_of_thought.py
@@ -31,7 +31,7 @@ async def test_async_chain_of_thought():
def test_chain_of_thought_with_native_reasoning():
"""Test ChainOfThought with a model that supports native reasoning, but using manual fields."""
- lm = dspy.LM(model="anthropic/claude-3-7-sonnet-20250219", cache=False)
+ lm = dspy.LM(engine="litellm", model="anthropic/claude-3-7-sonnet-20250219", cache=False)
dspy.settings.configure(lm=lm)
with mock.patch("litellm.completion") as mock_completion:
@@ -60,7 +60,7 @@ def test_chain_of_thought_with_native_reasoning():
def test_chain_of_thought_with_manual_reasoning():
"""Test ChainOfThought with manual reasoning where LM doesn't support native reasoning."""
- lm = dspy.LM(model="openai/gpt-4o-mini")
+ lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
dspy.settings.configure(lm=lm)
with mock.patch("litellm.completion") as mock_completion:
diff --git a/tests/predict/test_predict.py b/tests/predict/test_predict.py
index 65e61504d..00a41b98c 100644
--- a/tests/predict/test_predict.py
+++ b/tests/predict/test_predict.py
@@ -333,7 +333,7 @@ def test_load_ignores_serialized_endpoint_override_by_default(tmp_path, endpoint
file_path = tmp_path / "model.json"
override_url = "http://override.local/v1"
original_predict = dspy.Predict("q->a")
- original_predict.lm = dspy.LM(model="openai/gpt-4o-mini")
+ original_predict.lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
original_predict.save(file_path)
with open(file_path, "rb") as f:
@@ -357,7 +357,7 @@ def test_load_allows_serialized_endpoint_override_with_opt_in(tmp_path, endpoint
file_path = tmp_path / "model.json"
override_url = "http://override.local/v1"
original_predict = dspy.Predict("q->a")
- original_predict.lm = dspy.LM(model="openai/gpt-4o-mini")
+ original_predict.lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
original_predict.save(file_path)
with open(file_path, "rb") as f:
@@ -379,7 +379,7 @@ def test_load_allows_serialized_endpoint_override_with_opt_in(tmp_path, endpoint
def test_load_state_ignores_serialized_endpoint_override_by_default(endpoint_override_key):
override_url = "http://override.local/v1"
original_predict = dspy.Predict("q->a")
- original_predict.lm = dspy.LM(model="openai/gpt-4o-mini")
+ original_predict.lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
saved_state = copy.deepcopy(original_predict.dump_state())
saved_state["lm"][endpoint_override_key] = override_url
@@ -397,7 +397,7 @@ def test_load_state_ignores_serialized_endpoint_override_by_default(endpoint_ove
def test_load_state_allows_serialized_endpoint_override_with_opt_in(endpoint_override_key):
override_url = "http://override.local/v1"
original_predict = dspy.Predict("q->a")
- original_predict.lm = dspy.LM(model="openai/gpt-4o-mini")
+ original_predict.lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
saved_state = copy.deepcopy(original_predict.dump_state())
saved_state["lm"][endpoint_override_key] = override_url
@@ -413,7 +413,7 @@ def test_load_state_allows_serialized_endpoint_override_with_opt_in(endpoint_ove
def test_load_state_ignores_serialized_model_list_endpoint_override_by_default():
override_url = "http://override.local/v1"
original_predict = dspy.Predict("q->a")
- original_predict.lm = dspy.LM(model="openai/gpt-4o-mini")
+ original_predict.lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
saved_state = copy.deepcopy(original_predict.dump_state())
saved_state["lm"]["model_list"] = [
{
@@ -440,7 +440,7 @@ def test_load_prevents_serialized_endpoint_override_reaching_litellm(tmp_path, e
file_path = tmp_path / "model.json"
override_url = "http://override.local/v1"
original_predict = dspy.Predict("q->a")
- original_predict.lm = dspy.LM(model="openai/gpt-4o-mini")
+ original_predict.lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
original_predict.save(file_path)
with open(file_path, "rb") as f:
@@ -470,7 +470,7 @@ def test_load_blocks_serialized_model_list_unless_opted_in(tmp_path):
file_path = tmp_path / "model.json"
override_url = "http://override.local/v1"
original_predict = dspy.Predict("q->a")
- original_predict.lm = dspy.LM(model="openai/gpt-4o-mini")
+ original_predict.lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini")
original_predict.save(file_path)
with open(file_path, "rb") as f:
@@ -518,7 +518,7 @@ def test_load_uses_env_api_key_without_honoring_serialized_endpoint_override(tmp
env_api_key = "sk-live-test-secret"
original_predict = dspy.Predict("q->a")
- original_predict.lm = dspy.LM(model="openai/gpt-4o-mini", model_type="text")
+ original_predict.lm = dspy.LM(engine="litellm", model="openai/gpt-4o-mini", model_type="text")
original_predict.save(file_path)
with open(file_path, "rb") as f:
@@ -795,7 +795,7 @@ def test_call_predict_with_chat_history(adapter_type):
def test_lm_usage():
program = Predict("question -> answer")
- dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", cache=False), track_usage=True)
+ dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), track_usage=True)
with patch(
"dspy.clients.lm.litellm_completion",
return_value=ModelResponse(
@@ -816,7 +816,7 @@ def test_lm_usage_with_parallel():
time.sleep(0.5)
return program(question=question)
- dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", cache=False), track_usage=True)
+ dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), track_usage=True)
with patch(
"dspy.clients.lm.litellm_completion",
return_value=ModelResponse(
@@ -848,7 +848,7 @@ async def test_lm_usage_with_async():
program.aforward = types.MethodType(patched_aforward, program)
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), track_usage=True):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), track_usage=True):
with patch(
"litellm.acompletion",
return_value=ModelResponse(
@@ -962,7 +962,7 @@ async def test_async_predict():
def test_predicted_outputs_piped_from_predict_to_lm_call():
program = Predict("question -> answer")
- dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))
+ dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm"))
with patch("litellm.completion") as mock_completion:
program(
diff --git a/tests/predict/test_rlm.py b/tests/predict/test_rlm.py
index 09b607374..b6e50b2a9 100644
--- a/tests/predict/test_rlm.py
+++ b/tests/predict/test_rlm.py
@@ -300,7 +300,7 @@ class TestRLMInitialization:
tools = RLM("context -> answer", sub_lm=MagicMock(return_value="untyped response"))._make_llm_tools()
- with pytest.raises(TypeError, match="Sub-LM must return dspy.LMResponse or a non-empty list"):
+ with pytest.raises(TypeError, match="Sub-LM must return dspy.lm15.Response or a non-empty list"):
tools["llm_query"]("test prompt")
def test_llm_query_reports_textless_response_type(self):
@@ -444,14 +444,12 @@ class TestRLMInterpreterLifecycle:
pass
class CapturingLM(dspy.BaseLM):
- forward_contract = "typed_lm"
-
def __init__(self):
super().__init__("snapshot-model", temperature=0.0, max_tokens=1000, cache=False)
- self.request = None
+ self.messages = None
- def forward(self, request):
- self.request = request
+ def forward(self, prompt=None, messages=None, **kwargs):
+ self.messages = messages
raise StopLMCall
lm = CapturingLM()
@@ -464,10 +462,13 @@ class TestRLMInterpreterLifecycle:
iteration="1/20",
)
- request = lm.request.model_dump_json(indent=2).encode()
- snapshot = Path(__file__).with_name("snapshots") / "rlm_python_interpreter_lm_request.json"
+ import json
- assert request == snapshot.read_bytes().removesuffix(b"\n")
+ snapshot = Path(__file__).with_name("snapshots") / "rlm_python_interpreter_lm_request.json"
+ recorded = json.loads(snapshot.read_text())
+ expected = [{"role": message["role"], "content": "".join(part["text"] for part in message["parts"])}
+ for message in recorded["messages"]]
+ assert lm.messages == expected
def test_interpreter_remains_available_as_signature_input(self):
factory = MockInterpreterFactory(responses=[FinalOutput({"answer": "CPython"})])
diff --git a/tests/primitives/test_base_module.py b/tests/primitives/test_base_module.py
index fb538f906..f8401df2b 100644
--- a/tests/primitives/test_base_module.py
+++ b/tests/primitives/test_base_module.py
@@ -357,7 +357,7 @@ async def test_usage_tracker_async_parallel():
program.acall(question="What is the capital of France?"),
]
with dspy.context(
- lm=dspy.LM("openai/gpt-4o-mini", cache=False), track_usage=True, adapter=dspy.JSONAdapter()
+ lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), track_usage=True, adapter=dspy.JSONAdapter()
):
results = await asyncio.gather(*coroutines)
@@ -404,7 +404,7 @@ def test_module_history():
],
model="openai/gpt-4o-mini",
)
- dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter())
program = MyProgram()
program(question="What is the capital of France?")
@@ -452,7 +452,7 @@ def test_module_history_with_concurrency():
choices=[Choices(message=Message(content="{'reasoning': 'N/A', 'answer': 'Holy crab!'}"))],
model="openai/gpt-4o-mini",
)
- dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter())
program = MyProgram()
parallelizer = dspy.Parallel()
@@ -486,7 +486,7 @@ async def test_module_history_async():
model="openai/gpt-4o-mini",
)
program = MyProgram()
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()):
await program.acall(question="What is the capital of France?")
# Second call only call the submodule.
@@ -503,7 +503,7 @@ async def test_module_history_async():
assert program.history[0]["outputs"] == ["{'reasoning': 'Paris is the capital of France', 'answer': 'Paris'}"]
with dspy.context(
- disable_history=True, lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()
+ disable_history=True, lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()
):
await program.acall(question="What is the capital of France?")
@@ -513,7 +513,7 @@ async def test_module_history_async():
assert len(program.cot.predict.history) == 2
with dspy.context(
- disable_history=False, lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()
+ disable_history=False, lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()
):
await program.acall(question="What is the capital of France?")
# History is recorded again when history is enabled.
diff --git a/tests/primitives/test_python_interpreter.py b/tests/primitives/test_python_interpreter.py
index 860fa1142..172eb5f70 100644
--- a/tests/primitives/test_python_interpreter.py
+++ b/tests/primitives/test_python_interpreter.py
@@ -224,17 +224,41 @@ def test_enable_write_flag(tmp_path):
def test_enable_net_flag():
- test_url = "https://example.com"
-
- with PythonInterpreter(enable_network_access=None) as interpreter:
- code = f"import js\nresp = await js.fetch({test_url!r})\nresp.status"
- with pytest.raises(CodeInterpreterError, match="PythonError"):
- interpreter.execute(code)
+ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+ from threading import Thread
+
+ requests = []
+
+ class Handler(BaseHTTPRequestHandler):
+ def do_GET(self):
+ requests.append(self.path)
+ self.send_response(200)
+ self.send_header("Content-Length", "2")
+ self.end_headers()
+ self.wfile.write(b"ok")
+
+ def log_message(self, *args):
+ pass
+
+ server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
+ thread = Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ address = f"127.0.0.1:{server.server_port}"
+ code = f"import js\nresp = await js.fetch('http://{address}/')\nawait resp.text()\nresp.status"
+ try:
+ with PythonInterpreter(enable_network_access=None) as interpreter:
+ with pytest.raises(CodeInterpreterError):
+ interpreter.execute(code)
+ assert requests == [], "Denied requests must not reach the server"
- with PythonInterpreter(enable_network_access=["example.com"]) as interpreter:
- code = f"import js\nresp = await js.fetch({test_url!r})\nresp.status"
- result = interpreter.execute(code)
- assert int(result) == 200, "Network access is permitted with enable_network_access"
+ with PythonInterpreter(enable_network_access=[address]) as interpreter:
+ result = interpreter.execute(code)
+ assert int(result) == 200, "Network access is permitted with enable_network_access"
+ assert requests == ["/"]
+ finally:
+ server.shutdown()
+ server.server_close()
+ thread.join()
def test_interpreter_security_filesystem_access(tmp_path):
diff --git a/tests/streaming/test_streaming.py b/tests/streaming/test_streaming.py
index 829a63c76..cd3904f72 100644
--- a/tests/streaming/test_streaming.py
+++ b/tests/streaming/test_streaming.py
@@ -19,7 +19,7 @@ from dspy.streaming import StatusMessage, StatusMessageProvider, StreamResponse,
async def test_streamify_yields_expected_response_chunks(litellm_test_server):
api_base, _ = litellm_test_server
lm = dspy.LM(
- model="openai/dspy-test-model",
+ engine="litellm", model="openai/dspy-test-model",
api_base=api_base,
api_key="fakekey",
cache=True,
@@ -51,12 +51,14 @@ async def test_streamify_yields_expected_response_chunks(litellm_test_server):
async def test_streaming_response_yields_expected_response_chunks(litellm_test_server):
api_base, _ = litellm_test_server
lm = dspy.LM(
- model="openai/dspy-test-model",
+ engine="litellm", model="openai/dspy-test-model",
api_base=api_base,
api_key="fakekey",
cache=False,
)
- with dspy.context(lm=lm):
+ # The server fixture speaks JSON. This encoding test must not depend on
+ # replaying a response with another adapter after its chunks were emitted.
+ with dspy.context(lm=lm, adapter=dspy.JSONAdapter()):
class TestSignature(dspy.Signature):
input_text: str = dspy.InputField()
@@ -341,7 +343,7 @@ async def test_streaming_handles_space_correctly():
)
with mock.patch("litellm.acompletion", side_effect=gpt_4o_mini_stream):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
output = program(question="What is the capital of France?")
all_chunks = []
async for value in output:
@@ -499,7 +501,7 @@ async def test_stream_listener_returns_correct_chunk_chat_adapter():
dspy.streaming.StreamListener(signature_field_name="judgement"),
],
)
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False)):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)):
output = program(question="why did a chicken cross the kitchen?")
all_chunks = []
async for value in output:
@@ -606,7 +608,7 @@ async def test_stream_listener_returns_correct_chunk_json_adapter():
dspy.streaming.StreamListener(signature_field_name="judgement"),
],
)
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()):
output = program(question="why did a chicken cross the kitchen?")
all_chunks = []
async for value in output:
@@ -702,7 +704,7 @@ async def test_stream_listener_returns_correct_chunk_chat_adapter_untokenized_st
dspy.streaming.StreamListener(signature_field_name="judgement"),
],
)
- with dspy.context(lm=dspy.LM("gemini/gemini-2.5-flash", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("gemini/gemini-2.5-flash", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
output = program(question="why did a chicken cross the kitchen?")
all_chunks = []
async for value in output:
@@ -771,7 +773,7 @@ async def test_stream_listener_missing_completion_marker_chat_adapter():
dspy.streaming.StreamListener(signature_field_name="answer"),
],
)
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
output = program(question="Test question")
all_chunks = []
final_prediction = None
@@ -835,7 +837,7 @@ async def test_stream_listener_returns_correct_chunk_json_adapter_untokenized_st
dspy.streaming.StreamListener(signature_field_name="judgement"),
],
)
- with dspy.context(lm=dspy.LM("gemini/gemini-2.5-flash", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM("gemini/gemini-2.5-flash", engine="litellm", cache=False), adapter=dspy.JSONAdapter()):
output = program(question="why did a chicken cross the kitchen?")
all_chunks = []
async for value in output:
@@ -868,7 +870,7 @@ async def test_status_message_non_blocking():
program = dspy.streamify(MyProgram(), status_message_provider=StatusMessageProvider())
with mock.patch("litellm.acompletion", new_callable=AsyncMock, side_effect=[dummy_tool]):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False)):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)):
output = program(question="why did a chicken cross the kitchen?")
timestamps = []
async for value in output:
@@ -896,7 +898,7 @@ async def test_status_message_non_blocking_async_program():
program = dspy.streamify(MyProgram(), status_message_provider=StatusMessageProvider(), is_async_program=True)
with mock.patch("litellm.acompletion", new_callable=AsyncMock, side_effect=[dummy_tool]):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False)):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)):
output = program(question="why did a chicken cross the kitchen?")
timestamps = []
async for value in output:
@@ -952,7 +954,7 @@ async def test_stream_listener_allow_reuse():
return stream_generators.pop(0)() # return new async generator instance
with mock.patch("litellm.acompletion", side_effect=completion_side_effect):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False)):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)):
output = program(question="why did a chicken cross the kitchen?")
all_chunks = []
async for value in output:
@@ -1018,7 +1020,7 @@ async def test_stream_listener_returns_correct_chunk_xml_adapter():
dspy.streaming.StreamListener(signature_field_name="judgement"),
],
)
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.XMLAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.XMLAdapter()):
output = program(question="why did a chicken cross the kitchen?")
all_chunks = []
async for value in output:
@@ -1107,7 +1109,7 @@ async def test_streaming_allows_custom_streamable_type():
with mock.patch("litellm.acompletion", side_effect=stream):
with dspy.context(
- lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.ChatAdapter(native_response_types=[CustomType])
+ lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.ChatAdapter(native_response_types=[CustomType])
):
output = program(question="why did a chicken cross the kitchen?")
all_chunks = []
@@ -1198,7 +1200,7 @@ async def test_streaming_with_citations():
docs = [Document(data="Water boils at 100°C at standard pressure.", title="Physics Facts")]
with dspy.context(
- lm=dspy.LM("anthropic/claude-3-5-sonnet-20241022", cache=False),
+ lm=dspy.LM("anthropic/claude-3-5-sonnet-20241022", engine="litellm", cache=False),
adapter=dspy.ChatAdapter(native_response_types=[Citations]),
):
output = program(documents=docs, question="What temperature does water boil?")
@@ -1286,7 +1288,7 @@ async def test_chat_adapter_simple_pydantic_streaming():
)
with mock.patch("litellm.acompletion", side_effect=chat_stream):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
output = program(question="Say hello")
chunks = []
async for value in output:
@@ -1335,7 +1337,7 @@ async def test_chat_adapter_with_generic_type_annotation():
)
with mock.patch("litellm.acompletion", side_effect=chat_stream):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
output = program(question="Say hello")
chunks = []
async for value in output:
@@ -1385,7 +1387,7 @@ async def test_chat_adapter_nested_pydantic_streaming():
)
with mock.patch("litellm.acompletion", side_effect=nested_stream):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
output = program(question="Generate nested response")
chunks = []
async for value in output:
@@ -1441,7 +1443,7 @@ async def test_chat_adapter_mixed_fields_streaming():
)
with mock.patch("litellm.acompletion", side_effect=mixed_stream):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
output = program(question="Generate mixed response")
summary_chunks = []
details_chunks = []
@@ -1493,7 +1495,7 @@ async def test_json_adapter_simple_pydantic_streaming():
)
with mock.patch("litellm.acompletion", side_effect=json_stream):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()):
output = program(question="Say hello in JSON")
chunks = []
async for value in output:
@@ -1544,7 +1546,7 @@ async def test_json_adapter_bracket_balance_detection():
)
with mock.patch("litellm.acompletion", side_effect=complex_json_stream):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()):
output = program(question="Generate complex JSON")
chunks = []
async for value in output:
@@ -1597,7 +1599,7 @@ async def test_json_adapter_multiple_fields_detection():
)
with mock.patch("litellm.acompletion", side_effect=multi_field_stream):
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()):
output = program(question="Generate two responses")
first_chunks = []
second_chunks = []
@@ -1754,7 +1756,7 @@ async def test_streaming_reasoning_model():
],
)
with dspy.context(
- lm=dspy.LM("anthropic/claude-3-7-sonnet-20250219", cache=False),
+ lm=dspy.LM("anthropic/claude-3-7-sonnet-20250219", engine="litellm", cache=False),
adapter=dspy.ChatAdapter(native_response_types=[dspy.Reasoning]),
):
output = program(question="Why did a chicken cross the kitchen?")
@@ -1844,7 +1846,7 @@ async def test_stream_listener_empty_last_chunk_chat_adapter():
dspy.streaming.StreamListener(signature_field_name="answer"),
],
)
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
output = program(question="Why did the chicken cross the kitchen?")
all_chunks = []
async for value in output:
@@ -1899,7 +1901,7 @@ async def test_stream_listener_empty_last_chunk_json_adapter():
dspy.streaming.StreamListener(signature_field_name="answer"),
],
)
- with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()):
output = program(question="Why did the chicken cross the kitchen?")
all_chunks = []
async for value in output:
@@ -2040,7 +2042,7 @@ async def test_streaming_reasoning_fallback():
],
)
with dspy.context(
- lm=dspy.LM("openai/gpt-4o-mini", cache=False),
+ lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False),
adapter=dspy.ChatAdapter(),
):
output = program(question="Why did a chicken cross the kitchen?")
diff --git a/tests/teleprompt/test_bootstrap_trace.py b/tests/teleprompt/test_bootstrap_trace.py
index 89ffd64a1..d325de13e 100644
--- a/tests/teleprompt/test_bootstrap_trace.py
+++ b/tests/teleprompt/test_bootstrap_trace.py
@@ -35,45 +35,21 @@ def test_bootstrap_trace_data():
return example.number == prediction.number
# Configure dspy
- dspy.configure(lm=dspy.LM(model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter())
+ dspy.configure(lm=dspy.LM(engine="litellm", model="openai/gpt-4o-mini", cache=False), adapter=dspy.JSONAdapter())
- # Mock litellm completion responses
- # 4 successful responses and 1 that will trigger AdapterParseError
- successful_responses = [
+ # One reply per example: JSON parsing failures do not restart execution in
+ # another format. Keep four successes and one captured parsing failure.
+ responses = [
ModelResponse(
- choices=[Choices(message=Message(content='```json\n{"number": 1}\n```'))],
+ choices=[Choices(message=Message(content=(
+ "This is an invalid JSON!" if number is None else f'```json\n{{"number": {number}}}\n```'
+ )))],
model="openai/gpt-4o-mini",
- ),
- ModelResponse(
- choices=[Choices(message=Message(content='```json\n{"number": 2}\n```'))],
- model="openai/gpt-4o-mini",
- ),
- ModelResponse(
- choices=[Choices(message=Message(content='```json\n{"number": 3}\n```'))],
- model="openai/gpt-4o-mini",
- ),
- ModelResponse(
- choices=[Choices(message=Message(content='```json\n{"number": 4}\n```'))],
- model="openai/gpt-4o-mini",
- ),
+ )
+ for number in (1, 2, None, 4, 5)
]
- # Create a side effect that will trigger AdapterParseError on the 3rd call (index 2)
- def completion_side_effect(*args, **kwargs):
- call_count = completion_side_effect.call_count
- completion_side_effect.call_count += 1
-
- if call_count in (2, 3):
- # Return malformed responses for both structured-output mode and JSON-mode fallback.
- return ModelResponse(
- choices=[Choices(message=Message(content="This is an invalid JSON!"))],
- model="openai/gpt-4o-mini",
- )
- return successful_responses[call_count if call_count < 2 else call_count - 2]
-
- completion_side_effect.call_count = 0
-
- with mock.patch("litellm.completion", side_effect=completion_side_effect):
+ with mock.patch("litellm.completion", side_effect=responses) as completion:
# Call bootstrap_trace_data
results = bootstrap_trace_data(
program=program,
@@ -84,7 +60,8 @@ def test_bootstrap_trace_data():
capture_failed_parses=True,
)
- # Verify results
+ # Verify results, including the absence of a hidden JSON-mode replay.
+ assert completion.call_count == 5
assert len(results) == 5, f"Expected 5 results, got {len(results)}"
# Count successful and failed predictions
diff --git a/tests/test_vendored_lm15.py b/tests/test_vendored_lm15.py
index 1c1ddeb07..f226fb0f1 100644
--- a/tests/test_vendored_lm15.py
+++ b/tests/test_vendored_lm15.py
@@ -6,6 +6,8 @@ import subprocess
import sys
from pathlib import Path
+import pytest
+
def test_public_exports_are_the_vendored_objects():
import dspy
@@ -53,6 +55,30 @@ def test_vendored_lm15_carries_provenance_and_license():
assert "MIT" in (pkg.parent / "lm15-LICENSE").read_text()
+def test_engine_error_boundaries_and_stream_guard_ship_in_wheel():
+ import dspy
+ from dspy.clients.engines.stream_guard import checked_stream
+ from dspy.lm15 import StreamAssemblyError, StreamDeltaEvent, StreamStartEvent, TextDelta
+
+ class BrokenEngine:
+ calls = 0
+
+ def complete(self, request):
+ self.calls += 1
+ raise RuntimeError("network invariant failed")
+
+ engine = BrokenEngine()
+ lm = dspy.LM("custom", engine=engine, cache=False, num_retries=3)
+ with pytest.raises(dspy.LMUnexpectedError) as caught:
+ lm("hello")
+ assert isinstance(caught.value.__cause__, RuntimeError)
+ assert engine.calls == 1
+ assert issubclass(dspy.LMLockTimeoutError, dspy.LMError)
+ assert issubclass(dspy.LMStreamAssemblyError, dspy.LMUnexpectedError)
+ with pytest.raises(StreamAssemblyError, match="without a completion event"):
+ list(checked_stream(iter([StreamStartEvent(), StreamDeltaEvent(TextDelta("partial"))])))
+
+
def test_version_ignores_an_unrelated_installed_distribution(tmp_path):
metadata = tmp_path / "lm15-7.8.9.dist-info"
metadata.mkdir()
diff --git a/tests/utils/test_parallelizer_interrupt.py b/tests/utils/test_parallelizer_interrupt.py
new file mode 100644
index 000000000..b152565ca
--- /dev/null
+++ b/tests/utils/test_parallelizer_interrupt.py
@@ -0,0 +1,24 @@
+"""Signal dispositions are values as well as callables."""
+
+import signal
+
+import pytest
+
+from dspy.utils import parallelizer
+
+
[email protected]("original", [signal.SIG_DFL, signal.SIG_IGN])
+def test_interrupt_cancels_and_restores_sentinel_handler(monkeypatch, original):
+ installed = []
+ monkeypatch.setattr(parallelizer.signal, "getsignal", lambda signum: original)
+ monkeypatch.setattr(parallelizer.signal, "signal", lambda signum, handler: installed.append(handler))
+
+ def interrupt(*args, **kwargs):
+ installed[-1](signal.SIGINT, None)
+
+ monkeypatch.setattr(parallelizer, "wait", interrupt)
+ executor = parallelizer.ParallelExecutor(num_threads=2, disable_progress_bar=True)
+ with pytest.raises(KeyboardInterrupt):
+ executor.execute(lambda value: value, [1, 2])
+ assert executor.cancel_jobs.is_set()
+ assert installed[-1] is original
diff --git a/tests/utils/test_settings.py b/tests/utils/test_settings.py
index 71ee2c08c..25a46241c 100644
--- a/tests/utils/test_settings.py
+++ b/tests/utils/test_settings.py
@@ -55,14 +55,14 @@ def test_dspy_context_parallel():
def test_dspy_context_with_dspy_parallel():
- dspy.configure(lm=dspy.LM("openai/gpt-4o", cache=False), adapter=dspy.ChatAdapter())
+ dspy.configure(lm=dspy.LM("openai/gpt-4o", engine="litellm", cache=False), adapter=dspy.ChatAdapter())
class MyModule(dspy.Module):
def __init__(self):
self.predict = dspy.Predict("question -> answer")
def forward(self, question: str) -> str:
- lm = dspy.LM("openai/gpt-4o-mini", cache=False) if "France" in question else dspy.settings.lm
+ lm = dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False) if "France" in question else dspy.settings.lm
with dspy.context(lm=lm):
time.sleep(1)
assert dspy.settings.lm.model == lm.model
@@ -104,9 +104,9 @@ async def test_dspy_context_with_async_task_group():
async def aforward(self, question: str) -> str:
lm = (
- dspy.LM("openai/gpt-4o-mini", cache=False)
+ dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False)
if "France" in question
- else dspy.LM("openai/gpt-4o", cache=False)
+ else dspy.LM("openai/gpt-4o", engine="litellm", cache=False)
)
with dspy.context(lm=lm, trace=[]):
await asyncio.sleep(1)
@@ -117,7 +117,7 @@ async def test_dspy_context_with_async_task_group():
module = MyModule()
- with dspy.context(lm=dspy.LM("openai/gpt-4.1", cache=False), adapter=dspy.ChatAdapter()):
+ with dspy.context(lm=dspy.LM("openai/gpt-4.1", engine="litellm", cache=False), adapter=dspy.ChatAdapter()):
with mock.patch("litellm.acompletion") as mock_completion:
mock_completion.return_value = ModelResponse(
choices=[Choices(message=Message(content="[[ ## answer ## ]]\nParis"))],
@@ -140,14 +140,18 @@ async def test_dspy_context_with_async_task_group():
assert results[2].answer == "Paris"
assert results[3].answer == "Paris"
- # Verify mock was called correctly
+ # Concurrent calls can arrive in any order. Check the association,
+ # not just model counts, so leaked task-local settings still fail.
assert mock_completion.call_count == 4
- # France question uses gpt-4o-mini
- assert mock_completion.call_args_list[0].kwargs["model"] == "openai/gpt-4o-mini"
- assert mock_completion.call_args_list[1].kwargs["model"] == "openai/gpt-4o-mini"
- # Germany question uses gpt-4o
- assert mock_completion.call_args_list[2].kwargs["model"] == "openai/gpt-4o"
- assert mock_completion.call_args_list[3].kwargs["model"] == "openai/gpt-4o"
+ seen = {"France": 0, "Germany": 0}
+ for call in mock_completion.call_args_list:
+ content = call.kwargs["messages"][-1]["content"]
+ country = "France" if "France" in content else "Germany"
+ assert country in content
+ expected = "openai/gpt-4o-mini" if country == "France" else "openai/gpt-4o"
+ assert call.kwargs["model"] == expected
+ seen[country] += 1
+ assert seen == {"France": 2, "Germany": 2}
# The main thread is not affected by the context
assert dspy.settings.lm.model == "openai/gpt-4.1"