DSPy: The framework for programming—not prompting—language models
Python
fix(dspy): use native reasoning with a plan fallback in CoT and RLM (#10643)
Isaac Miller1af633fecbb2
diff --git a/docs/docs/api/modules/ChainOfThought.md b/docs/docs/api/modules/ChainOfThought.mdindex 21a1fad94..338b1ac52 100644--- a/docs/docs/api/modules/ChainOfThought.md+++ b/docs/docs/api/modules/ChainOfThought.md@@ -1,5 +1,11 @@# dspy.ChainOfThought+`ChainOfThought` defaults to `dspy.Reasoning`: it enables native reasoning on supported LMs without requesting a reasoning field in the answer. On other LMs, it requests a short `plan` before the task outputs and returns it as `result.reasoning`. If your signature already has a `plan` field, DSPy uses an unused name such as `plan_2` for this fallback. Native reasoning defaults to `reasoning_effort="low"`; configure `reasoning_effort` on the LM to override it. An explicit `None` or `"none"` selects the plan fallback instead. Models that always reason may still use native reasoning internally.++The public output is still `result.reasoning`. It contains a `dspy.Reasoning` value, or `None` if the provider does not return a native reasoning summary. This is not a guaranteed explanation of the answer. Use `str(result.reasoning)` when present to get text. Existing demonstrations keep their `reasoning` keys.++Pass `rationale_field_type=str` to retain an explicit string-valued reasoning field, or supply a custom `rationale_field` to control its description. These explicit fields do not disable reasoning already configured on the LM. No separate reasoning-cache setting is required.+<!-- START_API_REF -->::: dspy.ChainOfThoughthandler: pythondiff --git a/docs/docs/diving-deeper/rlm.md b/docs/docs/diving-deeper/rlm.mdindex 19018e1dc..a14d0a73b 100644--- a/docs/docs/diving-deeper/rlm.md+++ b/docs/docs/diving-deeper/rlm.md@@ -16,7 +16,9 @@ A normal `Predict` call puts the whole context into the prompt, so every token c### 3. The LLM drives an iterative REPL loop-`dspy.RLM` is not a single call. `forward()` runs up to `max_iters` turns. On each turn the internal `generate_action` predictor sees the variable metadata and the prior REPL history, then emits reasoning plus one Python code block. The code runs, its output joins the `REPLHistory`, and the next turn begins. State persists across turns because the whole run shares one interpreter session. This is why the instructions push the model to explore first and take small steps: each turn is meant to build on what the last one printed.+`dspy.RLM` is not a single call. `forward()` runs up to `max_iters` turns. On each turn the internal `generate_action` predictor sees the variable metadata and the prior REPL history, then emits one Python code block. It uses native reasoning on supported LMs, or requests a short `plan` before the code otherwise. That plan is returned under the existing `reasoning` key. The code runs, its output joins the `REPLHistory`, and the next turn begins. State persists across turns because the whole run shares one interpreter session. This is why the instructions push the model to explore first and take small steps: each turn is meant to build on what the last one printed.++Reasoning uses the same defaults as `ChainOfThought` and respects the LM's `reasoning_effort`. No separate reasoning-cache configuration is needed. REPL history carries text, code, and execution results; it does not replay provider-native reasoning state across iterations.### 4. Built-in `llm_query` tools give the loop its recursion@@ -101,7 +103,7 @@ The base class for inputs that need custom loading. Implement `sandbox_setup`, `### Inspecting the trajectory**`Prediction` fields: your output fields, `trajectory`, `final_reasoning`**-`forward()` returns a `Prediction` carrying the signature’s output fields plus two debugging fields. `trajectory` is a list of `{reasoning, code, output}` dicts, one per turn. `final_reasoning` is the model’s reasoning on the closing step. Read `trajectory` to see exactly what code ran and why.+`forward()` returns a `Prediction` carrying the signature’s output fields plus two debugging fields. `trajectory` is a list of `{reasoning, code, output}` dicts, one per turn. `reasoning` and `final_reasoning` remain strings: they contain the returned native summary or fallback plan, or an empty string when no summary is available. Read `trajectory` to see what code ran and what it returned; a reasoning explanation is not guaranteed.**`RLM.tools`**A property returning the user-provided tools as a name-to-`Tool` dict, excluding the built-in `llm_query` and `llm_query_batched`. Use it to confirm what the model can call.diff --git a/dspy/adapters/base.py b/dspy/adapters/base.pyindex 421cc3acb..274e80635 100644--- a/dspy/adapters/base.py+++ b/dspy/adapters/base.py@@ -15,6 +15,7 @@ from dspy.adapters.utils import apply_output_field_defaults, serialize_for_jsonfrom dspy.clients._deprecation import adapter_message_callfrom dspy.clients.base_lm import BaseLMfrom dspy.clients.capabilities import with_capability_planning+from dspy.dsp.utils.settings import settingsfrom dspy.signatures.field import InputFieldfrom dspy.signatures.signature import Signaturefrom dspy.utils.callback import BaseCallback, with_callbacks@@ -88,6 +89,7 @@ class Adapter:# stacked PR should replace this with an explicit `_AdapterPlan` that# records deleted fields, native tools, native output fields, inserted# messages/parts, and LM config patches.+ original_signature = signatureif not self.use_native_function_calling:for key in ("tools", "tool_choice", "parallel_tool_calls"):lm_kwargs.pop(key, None)@@ -130,6 +132,16 @@ class Adapter:):signature = field.annotation.adapt_to_native_lm_feature(signature, name, lm, lm_kwargs)+ # Only surviving default reasoning fields need the explicit plan fallback.+ for name in signature.output_fields:+ fallback_name = Reasoning._fallback_field_name(original_signature, name)+ if fallback_name != name:+ signature = signature.with_updated_fields(name, __dspy_reasoning_field=name)+ signature = Signature(+ {fallback_name if key == name else key: field for key, field in signature.fields.items()},+ signature.instructions,+ )+return signaturedef _call_postprocess(@@ -169,6 +181,11 @@ class Adapter:message="The LM returned an empty or null response.",)+ for name, field in processed_signature.output_fields.items():+ if original_name := field.json_schema_extra.get("__dspy_reasoning_field"):+ if name in value:+ value[original_name] = value.pop(name)+# Fields removed for native features are absent from the processed parsevalue = apply_output_field_defaults(original_signature, value)for field_name in original_signature.output_fields:@@ -187,6 +204,10 @@ class Adapter:isinstance(field.annotation, type)and field.annotation in self.native_response_typesand issubclass(field.annotation, Type)+ and not (+ field.annotation is Reasoning+ and Reasoning._fallback_field_name(original_signature, name) in processed_signature.output_fields+ )):parsed_value = field.annotation.parse_lm_response(output)if parsed_value is not None:@@ -228,6 +249,10 @@ class Adapter:messages = self.format(processed_signature, demos, inputs)if lm_kwargs.get("parallel_tool_calls") is not None:lm_kwargs.setdefault("tool_choice", "auto")+ if settings.send_stream and settings.stream_listeners:+ from dspy.streaming.messages import _StreamSignature, sync_send_to_stream++ sync_send_to_stream(settings.send_stream, _StreamSignature(id(settings.caller_predict), processed_signature))# TODO(3.5): build Request and parse Response directly; remove this marker.with adapter_message_call(lm, messages):outputs = lm(messages=messages, **lm_kwargs)@@ -246,6 +271,10 @@ class Adapter:messages = self.format(processed_signature, demos, inputs)if lm_kwargs.get("parallel_tool_calls") is not None:lm_kwargs.setdefault("tool_choice", "auto")+ if settings.send_stream and settings.stream_listeners:+ from dspy.streaming.messages import _StreamSignature++ await settings.send_stream.send(_StreamSignature(id(settings.caller_predict), processed_signature))# TODO(3.5): share the canonical request/response boundary with __call__.with adapter_message_call(lm, messages):outputs = await lm.acall(messages=messages, **lm_kwargs)@@ -297,6 +326,20 @@ class Adapter:A list of multiturn messages as expected by the LM."""inputs_copy = dict(inputs)+ fallback_names = {+ field.json_schema_extra["__dspy_reasoning_field"]: name+ for name, field in signature.output_fields.items()+ if "__dspy_reasoning_field" in field.json_schema_extra+ }+ if fallback_names:+ def remap(values):+ return {fallback_names.get(key, key): value for key, value in values.items()}++ demos = [remap(demo) for demo in demos]+ for name, field in signature.input_fields.items():+ if field.annotation is History and name in inputs_copy:+ history = inputs_copy[name]+ inputs_copy[name] = history.model_copy(update={"messages": [remap(m) for m in history.messages]})# If the signature and inputs have conversation history, we need to format the conversation history and# remove the history field from the signature.diff --git a/dspy/adapters/json_adapter.py b/dspy/adapters/json_adapter.pyindex d665be0e7..57eb2f4d0 100644--- a/dspy/adapters/json_adapter.py+++ b/dspy/adapters/json_adapter.py@@ -8,6 +8,7 @@ import regexfrom pydantic.fields import FieldInfofrom dspy.adapters.chat_adapter import ChatAdapter, FieldInfoWithName+from dspy.adapters.types.reasoning import Reasoningfrom dspy.adapters.types.tool import ToolCallsfrom dspy.adapters.utils import (apply_output_field_defaults,@@ -18,7 +19,6 @@ from dspy.adapters.utils import (translate_field_type,)from dspy.clients.base_lm import BaseLM-from dspy.clients.capabilities import with_capability_planningfrom dspy.signatures.signature import Signature, SignatureMetafrom dspy.utils.callback import BaseCallbackfrom dspy.utils.exceptions import AdapterParseError@@ -68,29 +68,16 @@ class JSONAdapter(ChatAdapter):format_ = {"type": "json_object"}lm_kwargs["response_format"] = format_- @with_capability_planning- def __call__(+ def _call_preprocess(self,lm: BaseLM,lm_kwargs: dict[str, Any],signature: type[Signature],- demos: list[dict[str, Any]],inputs: dict[str, Any],- ) -> list[dict[str, Any]]:+ ) -> type[Signature]:+ signature = super()._call_preprocess(lm, lm_kwargs, signature, inputs)self._prepare_response_format(lm, lm_kwargs, signature)- return super().__call__(lm, lm_kwargs, signature, demos, inputs)-- @with_capability_planning- async def acall(- self,- lm: BaseLM,- lm_kwargs: dict[str, Any],- signature: type[Signature],- demos: list[dict[str, Any]],- inputs: dict[str, Any],- ) -> list[dict[str, Any]]:- self._prepare_response_format(lm, lm_kwargs, signature)- return await super().acall(lm, lm_kwargs, signature, demos, inputs)+ return signaturedef format_field_structure(self, signature: type[Signature]) -> str:parts = []@@ -232,7 +219,8 @@ def _get_structured_outputs_response_format(fields = {}for name, field in signature.output_fields.items():- annotation = field.annotation+ # Reasoning is rendered and parsed as text, not a {content: ...} object.+ annotation = str if field.annotation is Reasoning else field.annotationif use_native_function_calling and annotation == ToolCalls:# Skip ToolCalls field if native function calling is enabled.continuediff --git a/dspy/adapters/types/reasoning.py b/dspy/adapters/types/reasoning.pyindex c4fa3de74..c09591c0c 100644--- a/dspy/adapters/types/reasoning.py+++ b/dspy/adapters/types/reasoning.py@@ -24,6 +24,20 @@ class Reasoning(Type):def format(self):return f"{self.content}"+ @staticmethod+ def _fallback_field_name(signature: type["Signature"], field_name: str) -> str:+ """Resolve the default CoT/RLM wire name without colliding with task fields."""+ field = signature.output_fields[field_name]+ name = field.json_schema_extra.get("__dspy_reasoning_fallback") if field.annotation is Reasoning else None+ if not name:+ return field_name+ candidate = name+ suffix = 2+ while candidate in signature.fields:+ candidate = f"{name}_{suffix}"+ suffix += 1+ return candidate+@pydantic.model_validator(mode="before")@classmethoddef validate_input(cls, data: Any):@@ -60,18 +74,11 @@ class Reasoning(Type):# reasoning effort is set in `lm_kwargs` or `lm.kwargs`.reasoning_effort = "low"- if reasoning_effort is None or not lm.supports_reasoning:- # If users explicitly set `reasoning_effort` to None or the LM doesn't support reasoning, we don't enable- # native reasoning.- return signature-- if "gpt-5" in lm.model and lm.model_type == "chat":- # There is a caveat of Litellm as 1.79.0 that when using the chat completion API on GPT-5 family models,- # the reasoning content is not available in the response. As a workaround, we don't enable the native- # reasoning feature for GPT-5 family models when using the chat completion API.- # Litellm issue: https://github.com/BerriAI/litellm/issues/14748+ if reasoning_effort in (None, "none") or not lm.supports_reasoning:+ # Respect explicit disabling as well as models without native reasoning.return signature+ # Native reasoning need not expose a summary (including GPT-5 chat).lm_kwargs["reasoning_effort"] = reasoning_effort# Delete the reasoning field from the signature to use the native reasoning feature.return signature.delete(field_name)diff --git a/dspy/predict/chain_of_thought.py b/dspy/predict/chain_of_thought.pyindex bf6cbb426..cf07abb19 100644--- a/dspy/predict/chain_of_thought.py+++ b/dspy/predict/chain_of_thought.py@@ -10,18 +10,19 @@ from dspy.signatures.signature import Signature, ensure_signatureclass ChainOfThought(Module):- """A module that reasons step by step before producing the output.+ """A module that reasons before producing the output.- Extends a given signature with an additional ``reasoning`` output field that- the language model fills in before answering. This chain-of-thought process- improves accuracy on tasks that require multi-step logic or explanation.+ Uses native reasoning when supported, otherwise requests a short plan before+ answering. The result exposes either as ``reasoning``; native reasoning may+ be ``None`` when the provider does not return a summary.Args:signature: The DSPy signature defining the module's inputs and outputs.rationale_field: An optional custom ``FieldInfo`` for the reasoning field.- If not provided, a default field with description ``${reasoning}`` is used.- rationale_field_type: The type annotation for the reasoning field. Defaults- to ``str``. Ignored if ``rationale_field`` is provided.+ Its annotation and description are preserved.+ rationale_field_type: Defaults to ``dspy.Reasoning``. Pass ``str`` to+ request an explicit reasoning field instead. Ignored if+ ``rationale_field`` is provided.**config: Additional keyword arguments forwarded to the underlying``dspy.Predict`` module.@@ -37,15 +38,19 @@ class ChainOfThought(Module):self,signature: str | type[Signature],rationale_field: FieldInfo | None = None,- rationale_field_type: type = str,+ rationale_field_type: type | None = None,**config: dict[str, Any],):super().__init__()signature = ensure_signature(signature)- desc = "${reasoning}"+ rationale_field_type = rationale_field_type or dspy.Reasoning+ use_default = rationale_field is None and rationale_field_type is dspy.Reasoning+ desc = "A short plan for solving the task." if use_default else "${reasoning}"rationale_field_type = rationale_field.annotation if rationale_field else rationale_field_typerationale_field = rationale_field if rationale_field else dspy.OutputField(desc=desc)extended_signature = signature.prepend(name="reasoning", field=rationale_field, type_=rationale_field_type)+ if use_default:+ extended_signature = extended_signature.with_updated_fields("reasoning", __dspy_reasoning_fallback="plan")self.predict = dspy.Predict(extended_signature, **config)def forward(self, **kwargs):diff --git a/dspy/predict/multi_chain_comparison.py b/dspy/predict/multi_chain_comparison.pyindex eaab57673..cf677b4b2 100644--- a/dspy/predict/multi_chain_comparison.py+++ b/dspy/predict/multi_chain_comparison.py@@ -36,10 +36,11 @@ class MultiChainComparison(Module):attempts = []for c in completions:- rationale = c.get("rationale", c.get("reasoning")).strip().split("\n")[0].strip()+ rationale = str(c.get("rationale", c.get("reasoning")) or "").strip().split("\n")[0].strip()answer = str(c[self.last_key]).strip().split("\n")[0].strip()attempts.append(- f"«I'm trying to {rationale} I'm not sure but my prediction is {answer}»",+ f"«I'm trying to {rationale} I'm not sure but my prediction is {answer}»"+ if rationale else f"«My prediction is {answer}»",)assert (diff --git a/dspy/predict/rlm.py b/dspy/predict/rlm.pyindex cb0b2d85b..d42581987 100644--- a/dspy/predict/rlm.py+++ b/dspy/predict/rlm.py@@ -419,7 +419,8 @@ class RLM(Module):.append("variables_info", dspy.InputField(desc="Metadata about the variables available in the REPL"), type_=str).append("repl_history", dspy.InputField(desc="Previous REPL code executions and their outputs"), type_=REPLHistory).append("iteration", dspy.InputField(desc="Current iteration number (1-indexed) out of max_iters"), type_=str)- .append("reasoning", dspy.OutputField(desc="Think step-by-step: what do you know? What remains? Plan your next action."), type_=str)+ .append("reasoning", dspy.OutputField(desc="A short plan for the next action."), type_=dspy.Reasoning)+ .with_updated_fields("reasoning", __dspy_reasoning_fallback="plan").append("code", dspy.OutputField(desc="Python code to execute. Use markdown code block format: ```python\\n<code>\\n```"), type_=str))@@ -734,25 +735,27 @@ class RLM(Module):Returns:Prediction if FINAL was called successfully, else updated REPLHistory"""+ # Providers may omit a native reasoning summary; trajectories stay string-valued.+ reasoning = str(pred.reasoning) if pred.reasoning is not None else ""# Handle error strings from caught exceptionsif isinstance(result, str) and result.startswith("[Error]"):output = self._format_output(result)- return history.append(reasoning=pred.reasoning, code=code, output=output)+ return history.append(reasoning=reasoning, code=code, output=output)# Handle FINAL outputif isinstance(result, FinalOutput):parsed_outputs, error = self._process_final_output(result, output_field_names)if error:- return history.append(reasoning=pred.reasoning, code=code, output=error)+ return history.append(reasoning=reasoning, code=code, output=error)final_history = history.append(- reasoning=pred.reasoning, code=code, output=f"FINAL: {parsed_outputs}"+ reasoning=reasoning, code=code, output=f"FINAL: {parsed_outputs}")return Prediction(**parsed_outputs,trajectory=[e.model_dump() for e in final_history],- final_reasoning=pred.reasoning,+ final_reasoning=reasoning,)# Format non-final result as output@@ -764,7 +767,7 @@ class RLM(Module):output = self._format_output(output)if self.verbose:logger.info(REPLEntry.format_output(output, self.max_output_chars))- return history.append(reasoning=pred.reasoning, code=code, output=output)+ return history.append(reasoning=reasoning, code=code, output=output)def _execute_code(self,diff --git a/dspy/streaming/messages.py b/dspy/streaming/messages.pyindex 19e1f915d..97d4429ef 100644--- a/dspy/streaming/messages.py+++ b/dspy/streaming/messages.py@@ -1,15 +1,26 @@import asyncioimport concurrent.futuresfrom dataclasses import dataclass-from typing import Any+from typing import TYPE_CHECKING, Anyfrom dspy.dsp.utils.settings import settingsfrom dspy.utils.callback import BaseCallbackfrom dspy.utils.lazy_import import require+if TYPE_CHECKING:+ from dspy.signatures.signature import Signature+anyio = require("anyio")+@dataclass+class _StreamSignature:+ """The adapter's render signature, queued before that call's response chunks."""++ predict_id: int+ signature: type["Signature"]++@dataclassclass StreamResponse:predict_name: strdiff --git a/dspy/streaming/streamify.py b/dspy/streaming/streamify.pyindex 1a5aae5f8..044204156 100644--- a/dspy/streaming/streamify.py+++ b/dspy/streaming/streamify.py@@ -10,7 +10,7 @@ import orjsonfrom dspy.dsp.utils.settings import settingsfrom dspy.primitives.prediction import Prediction-from dspy.streaming.messages import StatusMessage, StatusMessageProvider, StatusStreamingCallback+from dspy.streaming.messages import StatusMessage, StatusMessageProvider, StatusStreamingCallback, _StreamSignaturefrom dspy.streaming.streaming_listener import StreamListener, find_predictor_for_stream_listenersfrom dspy.utils.asyncify import asyncifyfrom dspy.utils.lazy_import import require@@ -219,7 +219,10 @@ def streamify(tg.start_soon(generator, args, kwargs, send_stream)async for value in receive_stream:- if _is_litellm_model_response_stream(value):+ if isinstance(value, _StreamSignature):+ for listener in predict_id_to_listener.get(value.predict_id, []):+ listener._set_signature(value.signature)+ elif _is_litellm_model_response_stream(value):if len(predict_id_to_listener) == 0:# No listeners are configured, yield the chunk directly for backwards compatibility.yield valuediff --git a/dspy/streaming/streaming_listener.py b/dspy/streaming/streaming_listener.pyindex 7d5650e54..9f171f99b 100644--- a/dspy/streaming/streaming_listener.py+++ b/dspy/streaming/streaming_listener.py@@ -9,6 +9,7 @@ from typing import TYPE_CHECKING, Anyfrom dspy.adapters.chat_adapter import ChatAdapterfrom dspy.adapters.json_adapter import JSONAdapterfrom dspy.adapters.types import Type+from dspy.adapters.types.reasoning import Reasoningfrom dspy.adapters.xml_adapter import XMLAdapterfrom dspy.dsp.utils.settings import settingsfrom dspy.streaming.messages import StreamResponse@@ -45,6 +46,8 @@ class StreamListener:hurt the performance because the same stream chunk is sent to multiple listeners."""self.signature_field_name = signature_field_name+ self._wire_field_name = signature_field_name+ self._use_native_reasoning = Trueself.predict = predictself.predict_name = predict_name@@ -118,6 +121,19 @@ class StreamListener:return False+ def _set_signature(self, signature):+ # Use the adapter's decision, after per-call overrides and native adaptation.+ self._use_native_reasoning = True+ for name, field in signature.output_fields.items():+ if field.json_schema_extra.get("__dspy_reasoning_field", name) == self.signature_field_name:+ self._use_native_reasoning = False+ self._wire_field_name = name+ self.adapter_identifiers["ChatAdapter"]["start_identifier"] = f"[[ ## {name} ## ]]"+ self.adapter_identifiers["JSONAdapter"]["start_identifier"] = f'"{name}":'+ self.adapter_identifiers["XMLAdapter"]["start_identifier"] = f"<{name}>"+ self.adapter_identifiers["XMLAdapter"]["end_identifier"] = re.compile(rf"</{name}>")+ break+def receive(self, chunk: ModelResponseStream):adapter_name = settings.adapter.__class__.__name__ if settings.adapter else "ChatAdapter"if adapter_name not in self.adapter_identifiers:@@ -149,6 +165,7 @@ class StreamListener:and inspect.isclass(self._output_type)and issubclass(self._output_type, Type)and self._output_type.is_streamable()+ and (self._output_type is not Reasoning or self._use_native_reasoning)):if parsed_chunk := self._output_type.parse_stream_chunk(chunk):return StreamResponse(@@ -279,7 +296,7 @@ class StreamListener:keys = list(parsed.keys())next_field_name = Nonefor key in keys:- if key != self.signature_field_name:+ if key != self._wire_field_name:next_field_name = keybreak@@ -340,7 +357,7 @@ class StreamListener:if isinstance(settings.adapter, JSONAdapter):return last_tokenselif isinstance(settings.adapter, XMLAdapter):- boundary_index = last_tokens.find(f"</{self.signature_field_name}>")+ boundary_index = last_tokens.find(f"</{self._wire_field_name}>")if boundary_index == -1:boundary_index = len(last_tokens)return last_tokens[:boundary_index]diff --git a/tests/adapters/test_reasoning.py b/tests/adapters/test_reasoning.pyindex 848d324e9..50433a0f2 100644--- a/tests/adapters/test_reasoning.py+++ b/tests/adapters/test_reasoning.py@@ -93,7 +93,7 @@ def test_reasoning_with_chain_of_thought():lm = DummyLM([{"reasoning": "Let me think step by step", "answer": "42"}])dspy.configure(lm=lm)- cot = dspy.ChainOfThought("question -> answer")+ cot = dspy.ChainOfThought("question -> answer", rationale_field_type=str)result = cot(question="What is the answer?")# Test that we can use string methods on result.reasoningdiff --git a/tests/callback/test_callback.py b/tests/callback/test_callback.pyindex fb60e9546..020f21d50 100644--- a/tests/callback/test_callback.py+++ b/tests/callback/test_callback.py@@ -239,7 +239,7 @@ def test_multiple_callbacks():def test_callback_complex_module():callback = MyCallback()dspy.configure(- lm=DummyLM({"How are you?": {"answer": "test output", "reasoning": "No more responses"}}),+ lm=DummyLM({"How are you?": {"answer": "test output", "plan": "No more responses"}}),callbacks=[callback],)@@ -271,7 +271,7 @@ def test_callback_complex_module():async def test_callback_async_module():callback = MyCallback()with dspy.context(- lm=DummyLM({"How are you?": {"answer": "test output", "reasoning": "No more responses"}}),+ lm=DummyLM({"How are you?": {"answer": "test output", "plan": "No more responses"}}),callbacks=[callback],):cot = dspy.ChainOfThought("question -> answer", n=3)diff --git a/tests/evaluate/test_auto_evaluation.py b/tests/evaluate/test_auto_evaluation.pyindex 4643f6cd6..b3ec8b847 100644--- a/tests/evaluate/test_auto_evaluation.py+++ b/tests/evaluate/test_auto_evaluation.py@@ -6,12 +6,12 @@ from dspy.utils.dummies import DummyLMdef test_semantic_f1_returns_prediction_without_trace():# Configure with a dummy LM that returns precision and recall values- # ChainOfThought adds a "reasoning" field to the output+ # ChainOfThought requests a "plan" field from non-reasoning models.dspy.configure(lm=DummyLM([{- "reasoning": "Comparing the responses",+ "plan": "Comparing the responses","precision": 1.0,"recall": 1.0,}@@ -38,7 +38,7 @@ def test_semantic_f1_returns_prediction_with_trace():lm=DummyLM([{- "reasoning": "Comparing the responses",+ "plan": "Comparing the responses","precision": 1.0,"recall": 1.0,}@@ -65,7 +65,7 @@ def test_semantic_f1_score_value():lm=DummyLM([{- "reasoning": "Comparing the responses",+ "plan": "Comparing the responses","precision": 0.8,"recall": 0.6,}@@ -92,14 +92,14 @@ def test_complete_and_grounded_returns_prediction_without_trace():lm=DummyLM([{- "reasoning": "Analyzing completeness",+ "plan": "Analyzing completeness","ground_truth_key_ideas": "the answer is 2","system_response_key_ideas": "the answer is 2","discussion": "both match","completeness": 1.0,},{- "reasoning": "Analyzing groundedness",+ "plan": "Analyzing groundedness","system_response_claims": "1+1=2","discussion": "supported by context","groundedness": 1.0,@@ -127,14 +127,14 @@ def test_complete_and_grounded_returns_prediction_with_trace():lm=DummyLM([{- "reasoning": "Analyzing completeness",+ "plan": "Analyzing completeness","ground_truth_key_ideas": "the answer is 2","system_response_key_ideas": "the answer is 2","discussion": "both match","completeness": 0.9,},{- "reasoning": "Analyzing groundedness",+ "plan": "Analyzing groundedness","system_response_claims": "1+1=2","discussion": "supported by context","groundedness": 0.8,@@ -162,12 +162,12 @@ def test_semantic_f1_prediction_can_be_compared():lm=DummyLM([{- "reasoning": "Comparing first response",+ "plan": "Comparing first response","precision": 0.8,"recall": 0.6,},{- "reasoning": "Comparing second response",+ "plan": "Comparing second response","precision": 0.9,"recall": 0.7,},diff --git a/tests/flex/test_flex_interpreter.py b/tests/flex/test_flex_interpreter.pyindex 3c98f83b9..67daf685d 100644--- a/tests/flex/test_flex_interpreter.py+++ b/tests/flex/test_flex_interpreter.py@@ -236,7 +236,7 @@ def test_call_runs_predictor_via_host_lm() -> None:flex = _bridged_flex()inv = flex._bridge.invocation()inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})- dspy.configure(lm=DummyLM([{"reasoning": "two doubled is four", "result": "4"}]))+ dspy.configure(lm=DummyLM([{"plan": "two doubled is four", "result": "4"}]))fields = inv.call("solve", {"value": 2})assert fields["result"] == 4assert "reasoning" in fields@@ -416,7 +416,7 @@ def test_max_predictor_calls_is_plumbed_to_bridge() -> None:def test_predictor_call_budget_is_enforced() -> None:# The budget lives in the per-forward _Invocation, so every forward starts fresh.flex = _bridged_flex_with(max_predictor_calls=2)- dspy.configure(lm=DummyLM([{"reasoning": "r", "result": str(i)} for i in range(3)]))+ dspy.configure(lm=DummyLM([{"plan": "r", "result": str(i)} for i in range(3)]))inv = flex._bridge.invocation()inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})inv.call("solve", {"value": 1}) # 1st@@ -432,7 +432,7 @@ def test_predictor_call_budget_is_enforced() -> None:def test_predictor_call_budget_can_be_disabled() -> None:flex = _bridged_flex_with(max_predictor_calls=None)assert flex._bridge._max_predictor_calls is None- dspy.configure(lm=DummyLM([{"reasoning": "r", "result": str(i)} for i in range(5)]))+ dspy.configure(lm=DummyLM([{"plan": "r", "result": str(i)} for i in range(5)]))inv = flex._bridge.invocation()inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})for _ in range(5): # no cap -> all calls go through@@ -611,7 +611,7 @@ COT_GLUE_MODULE = textwrap.dedent(@deno_requireddef test_optimized_cot_runs_in_sandbox_with_lm_bridged() -> None:- dspy.configure(lm=DummyLM([{"reasoning": "double of 21 is 42", "result": "the result is 42"}]))+ dspy.configure(lm=DummyLM([{"plan": "double of 21 is 42", "result": "the result is 42"}]))flex = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter())flex._bind_code(COT_GLUE_MODULE)# forward() runs entirely in the sandbox; self.solve(...) bridges to the host DummyLM, and the@@ -672,7 +672,7 @@ def test_forward_runs_on_a_fresh_instance_each_call() -> None:@deno_requireddef test_bridged_program_survives_save_load(tmp_path) -> None:- dspy.configure(lm=DummyLM([{"reasoning": "double of 21 is 42", "result": "the result is 42"}]))+ dspy.configure(lm=DummyLM([{"plan": "double of 21 is 42", "result": "the result is 42"}]))program = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter())path = tmp_path / "prog.json"program._bind_code(COT_GLUE_MODULE)@@ -730,7 +730,7 @@ RUNAWAY_MODULE = textwrap.dedent(@deno_requireddef test_runaway_glue_is_stopped_by_budget_in_sandbox() -> None:- dspy.configure(lm=DummyLM([{"reasoning": "r", "result": "1"} for _ in range(50)]))+ dspy.configure(lm=DummyLM([{"plan": "r", "result": "1"} for _ in range(50)]))flex = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter(), max_predictor_calls=3)flex._bind_code(RUNAWAY_MODULE)with pytest.raises(Exception) as excinfo: # the budget error propagates out of the sandbox@@ -814,7 +814,7 @@ def test_bridged_codeact_runs_in_flex_sandbox() -> None:lm=DummyLM([{"generated_code": "```python\nprint(shout('hello'))\n```", "finished": True},- {"reasoning": "the code printed HELLO", "out": "HELLO"},+ {"plan": "the code printed HELLO", "out": "HELLO"},]))@@ -854,7 +854,7 @@ def test_bridged_react_runs_in_flex_sandbox() -> None:[{"next_thought": "shout it", "next_tool_name": "shout", "next_tool_args": {"text": "hello"}},{"next_thought": "done", "next_tool_name": "finish", "next_tool_args": {}},- {"reasoning": "the tool returned HELLO", "out": "HELLO"},+ {"plan": "the tool returned HELLO", "out": "HELLO"},]))@@ -935,8 +935,8 @@ def test_nested_rlm_tool_provenance() -> None:dspy.configure(lm=DummyLM([- {"reasoning": "probe here, then spawn the sub-agent", "code": outer_code},- {"reasoning": "probe from the sub-agent", "code": inner_code},+ {"plan": "probe here, then spawn the sub-agent", "code": outer_code},+ {"plan": "probe from the sub-agent", "code": inner_code},]))diff --git a/tests/predict/snapshots/rlm_python_interpreter_lm_request.json b/tests/predict/snapshots/rlm_python_interpreter_lm_request.jsonindex 39aff8322..faa655546 100644--- a/tests/predict/snapshots/rlm_python_interpreter_lm_request.json+++ b/tests/predict/snapshots/rlm_python_interpreter_lm_request.json@@ -7,7 +7,7 @@{"type": "text","metadata": {},- "text": "Your input fields are:\n1. `variables_info` (str): Metadata about the variables available in the REPL\n2. `repl_history` (REPLHistory): Previous REPL code executions and their outputs\n3. `iteration` (str): Current iteration number (1-indexed) out of max_iters\nYour output fields are:\n1. `reasoning` (str): Think step-by-step: what do you know? What remains? Plan your next action.\n2. `code` (str): Python code to execute. Use markdown code block format: ```python\\n<code>\\n```\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## variables_info ## ]]\n{variables_info}\n\n[[ ## repl_history ## ]]\n{repl_history}\n\n[[ ## iteration ## ]]\n{iteration}\n\n[[ ## reasoning ## ]]\n{reasoning}\n\n[[ ## code ## ]]\n{code}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `query`, produce the fields `answer`.\n \n You are tasked with producing the following outputs given the inputs `query`:\n - {answer}\n \n You have access to a Python REPL environment. Write Python code and it will be executed. You will see the output, then write more code based on what you learned. This is an iterative process.\n \n Execution environment:\n Python runs in Pyodide/WebAssembly. State persists across executions, but subprocesses and native extensions are unavailable. Python standard libraries such as re, json, collections, and math are available. Host filesystem, environment, and network access require explicit permission.\n \n Sub-agents (dspy):\n You may `import dspy` and build sub-agents in the REPL for subtasks that need structured inputs/outputs.\n - `dspy.Predict(\"question -> answer\")(question=...)` or `dspy.ChainOfThought(...)` - single-step sub-agents.\n - `dspy.ReActV2(\"question -> answer\", tools=[...])(question=...)` - a multi-step tool-using sub-agent.\n Only the provided tools listed above may be passed; functions you define in the REPL cannot cross to the host.\n - `dspy.RLM(\"context, query -> answer\")(context=..., query=...)` - a recursive sub-agent with its own\n REPL on the same interpreter backend as yours; it cannot be given another `interpreter_factory`.\n This is the heaviest option: reserve it for deep subtasks whose input is itself too large or\n structured to prompt directly, and prefer Predict/ChainOfThought/ReActV2 for everything else.\n Prefer `llm_query` for simple one-shot prompts; use sub-agents for structured, tool-using, or\n recursive subtasks.\n \n Available:\n - Variables: `query` (your input data)\n - `llm_query(prompt)` - query a sub-LLM (~500K char capacity) for semantic analysis\n - `llm_query_batched(prompts)` - query multiple prompts concurrently (much faster for multiple queries)\n - `print()` - ALWAYS print to see results\n - `SUBMIT(answer)` - submit final output when done\n \n IMPORTANT: This is ITERATIVE. Each code block you write will execute, you'll see the output, then you decide what to do next. Do NOT try to solve everything in one step.\n \n 1. EXPLORE FIRST - Look at your data before processing it. Print samples, check types/lengths, understand the structure.\n 2. ITERATE - Write small code snippets, observe outputs, then decide next steps. State persists between iterations.\n 3. VERIFY BEFORE SUBMITTING - If results seem wrong (zeros, empty, unexpected), reconsider your approach.\n 4. USE llm_query FOR SEMANTICS - String matching finds WHERE things are; llm_query understands WHAT things mean.\n 5. MINIMIZE RETYPING (INPUTS & OUTPUTS) - When values are long, precise, or error-prone (IDs, numbers, code, quotes), re-access them via variables and parse/compute in code instead of retyping. Use small, targeted prints to sanity-check, but avoid manual copying when variables can carry the exact value.\n 6. SUBMIT ONLY AFTER SEEING OUTPUTS - SUBMIT ends the current run immediately. If you need to inspect printed output, run it in one step, review the result, then call SUBMIT in a later step.\n \n You have max 50 sub-LLM calls. When done, call SUBMIT() with your output."+ "text": "Your input fields are:\n1. `variables_info` (str): Metadata about the variables available in the REPL\n2. `repl_history` (REPLHistory): Previous REPL code executions and their outputs\n3. `iteration` (str): Current iteration number (1-indexed) out of max_iters\nYour output fields are:\n1. `plan` (str): A short plan for the next action.\n2. `code` (str): Python code to execute. Use markdown code block format: ```python\\n<code>\\n```\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## variables_info ## ]]\n{variables_info}\n\n[[ ## repl_history ## ]]\n{repl_history}\n\n[[ ## iteration ## ]]\n{iteration}\n\n[[ ## plan ## ]]\n{plan}\n\n[[ ## code ## ]]\n{code}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `query`, produce the fields `answer`.\n \n You are tasked with producing the following outputs given the inputs `query`:\n - {answer}\n \n You have access to a Python REPL environment. Write Python code and it will be executed. You will see the output, then write more code based on what you learned. This is an iterative process.\n \n Execution environment:\n Python runs in Pyodide/WebAssembly. State persists across executions, but subprocesses and native extensions are unavailable. Python standard libraries such as re, json, collections, and math are available. Host filesystem, environment, and network access require explicit permission.\n \n Sub-agents (dspy):\n You may `import dspy` and build sub-agents in the REPL for subtasks that need structured inputs/outputs.\n - `dspy.Predict(\"question -> answer\")(question=...)` or `dspy.ChainOfThought(...)` - single-step sub-agents.\n - `dspy.ReActV2(\"question -> answer\", tools=[...])(question=...)` - a multi-step tool-using sub-agent.\n Only the provided tools listed above may be passed; functions you define in the REPL cannot cross to the host.\n - `dspy.RLM(\"context, query -> answer\")(context=..., query=...)` - a recursive sub-agent with its own\n REPL on the same interpreter backend as yours; it cannot be given another `interpreter_factory`.\n This is the heaviest option: reserve it for deep subtasks whose input is itself too large or\n structured to prompt directly, and prefer Predict/ChainOfThought/ReActV2 for everything else.\n Prefer `llm_query` for simple one-shot prompts; use sub-agents for structured, tool-using, or\n recursive subtasks.\n \n Available:\n - Variables: `query` (your input data)\n - `llm_query(prompt)` - query a sub-LLM (~500K char capacity) for semantic analysis\n - `llm_query_batched(prompts)` - query multiple prompts concurrently (much faster for multiple queries)\n - `print()` - ALWAYS print to see results\n - `SUBMIT(answer)` - submit final output when done\n \n IMPORTANT: This is ITERATIVE. Each code block you write will execute, you'll see the output, then you decide what to do next. Do NOT try to solve everything in one step.\n \n 1. EXPLORE FIRST - Look at your data before processing it. Print samples, check types/lengths, understand the structure.\n 2. ITERATE - Write small code snippets, observe outputs, then decide next steps. State persists between iterations.\n 3. VERIFY BEFORE SUBMITTING - If results seem wrong (zeros, empty, unexpected), reconsider your approach.\n 4. USE llm_query FOR SEMANTICS - String matching finds WHERE things are; llm_query understands WHAT things mean.\n 5. MINIMIZE RETYPING (INPUTS & OUTPUTS) - When values are long, precise, or error-prone (IDs, numbers, code, quotes), re-access them via variables and parse/compute in code instead of retyping. Use small, targeted prints to sanity-check, but avoid manual copying when variables can carry the exact value.\n 6. SUBMIT ONLY AFTER SEEING OUTPUTS - SUBMIT ends the current run immediately. If you need to inspect printed output, run it in one step, review the result, then call SUBMIT in a later step.\n \n You have max 50 sub-LLM calls. When done, call SUBMIT() with your output."}],"name": null,@@ -19,7 +19,7 @@{"type": "text","metadata": {},- "text": "[[ ## variables_info ## ]]\n«query: str»\n\n[[ ## repl_history ## ]]\nYou have not interacted with the REPL environment yet.\n\n[[ ## iteration ## ]]\n1/20\n\nRespond with the corresponding output fields, starting with the field `[[ ## reasoning ## ]]`, then `[[ ## code ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."+ "text": "[[ ## variables_info ## ]]\n«query: str»\n\n[[ ## repl_history ## ]]\nYou have not interacted with the REPL environment yet.\n\n[[ ## iteration ## ]]\n1/20\n\nRespond with the corresponding output fields, starting with the field `[[ ## plan ## ]]` (must be formatted as a valid Python str), then `[[ ## code ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`."}],"name": null,diff --git a/tests/predict/test_chain_of_thought.py b/tests/predict/test_chain_of_thought.pyindex c19100dd5..a4b09a28b 100644--- a/tests/predict/test_chain_of_thought.py+++ b/tests/predict/test_chain_of_thought.py@@ -9,7 +9,7 @@ from dspy.utils import DummyLMdef test_initialization_with_string_signature():- lm = DummyLM([{"reasoning": "find the number after 1", "answer": "2"}])+ lm = DummyLM([{"plan": "find the number after 1", "answer": "2"}])dspy.configure(lm=lm)predict = ChainOfThought("question -> answer")assert list(predict.predict.signature.output_fields.keys()) == [@@ -21,7 +21,7 @@ def test_initialization_with_string_signature():@pytest.mark.asyncioasync def test_async_chain_of_thought():- lm = DummyLM([{"reasoning": "find the number after 1", "answer": "2"}])+ lm = DummyLM([{"plan": "find the number after 1", "answer": "2"}])with dspy.context(lm=lm):program = ChainOfThought("question -> answer")result = await program.acall(question="What is 1+1?")@@ -29,7 +29,7 @@ async def test_async_chain_of_thought():def test_chain_of_thought_with_native_reasoning():- """Test ChainOfThought with a model that supports native reasoning, but using manual fields."""+ """Native-capable models need no explicit reasoning or plan output."""lm = dspy.LM(engine="litellm", model="anthropic/claude-3-7-sonnet-20250219", cache=False)dspy.settings.configure(lm=lm)@@ -39,10 +39,8 @@ def test_chain_of_thought_with_native_reasoning():choices=[Choices(message=Message(- content=(- "[[ ## reasoning ## ]]\nStep-by-step thinking about the capital of France\n"- "[[ ## answer ## ]]\nParis\n[[ ## completion ## ]]"- )+ content="[[ ## answer ## ]]\nParis\n[[ ## completed ## ]]",+ reasoning_content="Paris is the capital of France.",),)],@@ -52,10 +50,13 @@ def test_chain_of_thought_with_native_reasoning():cot = ChainOfThought("question -> answer")result = cot(question="What is the capital of France?")assert result.answer == "Paris"- assert isinstance(result.reasoning, str)- assert result.reasoning == "Step-by-step thinking about the capital of France"+ assert isinstance(result.reasoning, dspy.Reasoning)+ assert result.reasoning == "Paris is the capital of France."- args, kwargs = mock_completion.call_args+ _args, kwargs = mock_completion.call_args+ assert kwargs["reasoning_effort"] == "low"+ assert "[[ ## reasoning ## ]]" not in str(kwargs["messages"])+ assert "[[ ## plan ## ]]" not in str(kwargs["messages"])def test_chain_of_thought_with_manual_reasoning():@@ -70,7 +71,7 @@ def test_chain_of_thought_with_manual_reasoning():reasoning="Step-by-step thinking about the capital of France",message=Message(content=(- "[[ ## reasoning ## ]]\nStep-by-step thinking about the capital of France\n"+ "[[ ## plan ## ]]\nRecall the capital of France.\n""[[ ## answer ## ]]\nParis\n[[ ## completion ## ]]")),@@ -82,4 +83,89 @@ def test_chain_of_thought_with_manual_reasoning():cot = ChainOfThought("question -> answer")result = cot(question="What is the capital of France?")assert result.answer == "Paris"- assert result.reasoning == "Step-by-step thinking about the capital of France"+ assert result.reasoning == "Recall the capital of France."++[email protected]("async_mode", [False, True])[email protected]("adapter_type", [dspy.ChatAdapter, dspy.JSONAdapter, dspy.XMLAdapter])[email protected]("mode", ["native", "omitted", "unsupported", "disabled", "disabled_string"])+async def test_default_reasoning_modes(mode, adapter_type, async_mode):+ class Model(DummyLM):+ @property+ def supports_reasoning(self):+ return mode != "unsupported"++ @property+ def supported_params(self):+ return ["response_format", "reasoning_effort"]++ @property+ def supports_response_schema(self):+ return True++ native = mode in ("native", "omitted")+ adapter = adapter_type()+ responses = [{"answer": "17"}, {"answer": "29"}]+ if not native:+ for response in responses:+ response["plan"] = "Use the supplied observations."+ # Some models return native thinking even when effort is disabled; preserve the explicit field.+ lm = Model(responses, reasoning=mode in ("native", "disabled", "disabled_string"), adapter=adapter)+ lm.kwargs["reasoning_effort"] = "high"+ program = ChainOfThought("question -> answer", n=2)+ config = {"reasoning_effort": None} if mode == "disabled" else {}+ if mode == "disabled_string":+ config = {"reasoning_effort": "none"}++ with dspy.context(lm=lm, adapter=adapter):+ result = await program.acall(question="Use observations", config=config) if async_mode else program(question="Use observations", config=config)++ expected = "Some reasoning" if mode == "native" else None if native else "Use the supplied observations."+ assert result.reasoning == expected+ assert result.completions.reasoning == [expected, expected]+ assert result.completions.answer == ["17", "29"]+ if adapter_type is dspy.JSONAdapter:+ schema = lm.history[0]["kwargs"]["response_format"].model_json_schema()+ assert set(schema["properties"]) == ({"answer"} if native else {"plan", "answer"})+ if not native:+ assert schema["properties"]["plan"]["type"] == "string"++[email protected]("adapter_type", [dspy.ChatAdapter, dspy.JSONAdapter, dspy.XMLAdapter])+def test_reasoning_fallback_preserves_demos_and_history(tmp_path, adapter_type):+ class Task(dspy.Signature):+ question: str = dspy.InputField()+ history: dspy.History = dspy.InputField()+ plan: str = dspy.OutputField()+ answer: str = dspy.OutputField()++ program = ChainOfThought(Task)+ program.predict.demos = [dspy.Example(question="Demo", reasoning="Demo reasoning", plan="Demo plan", answer="13")]+ program.save(tmp_path / "program.json")+ loaded = ChainOfThought(Task)+ loaded.load(tmp_path / "program.json")+ history = dspy.History(messages=[{"question": "Previous", "reasoning": "Previous reasoning", "plan": "Previous plan", "answer": "19"}])+ adapter = adapter_type()+ lm = DummyLM([{"plan_2": "New reasoning", "plan": "New plan", "answer": "23"}], adapter=adapter)+ with dspy.context(lm=lm, adapter=adapter):+ result = loaded(question="Current", history=history)+ assert result.reasoning == "New reasoning"+ assert result.plan == "New plan"+ messages = "\n".join(m["content"] for m in lm.history[0]["messages"])+ assert "plan_2" in messages+ assert "Demo reasoning" in messages and "Previous reasoning" in messages and "Previous plan" in messages+ assert loaded.predict.demos[0]["reasoning"] == "Demo reasoning"+ assert history.messages[0]["reasoning"] == "Previous reasoning"++[email protected]("custom", [False, True])+def test_explicit_rationale_is_not_renamed(custom):+ field = dspy.OutputField(desc="Custom working") if custom else None+ if field is not None:+ field.annotation = str+ program = ChainOfThought("question -> answer", rationale_field=field, rationale_field_type=str)+ with dspy.context(lm=DummyLM([{"reasoning": "Working", "answer": "31"}])):+ result = program(question="Question")+ assert result.reasoning == "Working"+ assert isinstance(result.reasoning, str)diff --git a/tests/predict/test_code_act.py b/tests/predict/test_code_act.pyindex 49fb61d35..1529d7e6a 100644--- a/tests/predict/test_code_act.py+++ b/tests/predict/test_code_act.py@@ -53,7 +53,7 @@ def test_codeact_code_generation(pooled_interpreter):"generated_code": "```python\nresult = add(1,1)\nprint(result)\n```","finished": True,},- {"reasoning": "Reason_B", "answer": "2"},+ {"plan": "Reason_B", "answer": "2"},])dspy.configure(lm=lm)@@ -85,7 +85,7 @@ def test_codeact_support_multiple_fields(pooled_interpreter):"generated_code": "```python\nresult = extract_maximum_minimum('2, 3, 5, 6')\nprint(result)\n```","finished": True,},- {"reasoning": "Reason_B", "maximum": "6", "minimum": "2"},+ {"plan": "Reason_B", "maximum": "6", "minimum": "2"},])dspy.configure(lm=lm)@@ -112,7 +112,7 @@ def test_codeact_code_parse_failure(pooled_interpreter):"generated_code": "```python\nresult = add(1,1)\nprint(result)\n```","finished": True,},- {"reasoning": "Reason_B", "answer": "2"},+ {"plan": "Reason_B", "answer": "2"},])dspy.configure(lm=lm)@@ -140,7 +140,7 @@ def test_codeact_code_execution_failure(pooled_interpreter):"generated_code": "```python\nresult = add(1,1)\nprint(result)\n```","finished": True,},- {"reasoning": "Reason_B", "answer": "2"},+ {"plan": "Reason_B", "answer": "2"},])dspy.configure(lm=lm)diff --git a/tests/predict/test_multi_chain_comparison.py b/tests/predict/test_multi_chain_comparison.pyindex 294e28db3..2ec8ae5bc 100644--- a/tests/predict/test_multi_chain_comparison.py+++ b/tests/predict/test_multi_chain_comparison.py@@ -38,3 +38,20 @@ def test_basic_example():assert final_pred.rationale == "my rationale"assert final_pred.answer == "blue"+++def test_comparing_native_completions_without_summaries():+ class NativeLM(DummyLM):+ @property+ def supports_reasoning(self):+ return True++ lm = NativeLM([{"answer": "blue"}, {"answer": "green"}, {"rationale": "Choose blue", "answer": "blue"}])+ with dspy.context(lm=lm):+ candidates = dspy.ChainOfThought(BasicQA, n=2)(question="Sky color?").completions+ assert candidates.reasoning == [None, None]+ result = dspy.MultiChainComparison(BasicQA, M=2)(candidates, question="Sky color?")+ assert result.answer == "blue"+ prompt = lm.history[-1]["messages"][-1]["content"]+ assert "blue" in prompt and "green" in prompt+ assert "None" not in promptdiff --git a/tests/predict/test_program_of_thought.py b/tests/predict/test_program_of_thought.pyindex 5828bd384..5d247b435 100644--- a/tests/predict/test_program_of_thought.py+++ b/tests/predict/test_program_of_thought.py@@ -52,10 +52,10 @@ def test_pot_code_generation(pooled_interpreter):lm = DummyLM([{- "reasoning": "Reason_A",+ "plan": "Reason_A","generated_code": "```python\nresult = 1+1\nSUBMIT({'answer': result})\n```",},- {"reasoning": "Reason_B", "answer": "2"},+ {"plan": "Reason_B", "answer": "2"},])dspy.configure(lm=lm)@@ -69,8 +69,8 @@ def test_pot_code_generation(pooled_interpreter):def test_old_style_pot(pooled_interpreter):lm = DummyLM([- {"reasoning": "Reason_A", "generated_code": "```python\nresult = 1+1\n```"},- {"reasoning": "Reason_B", "answer": "2"},+ {"plan": "Reason_A", "generated_code": "```python\nresult = 1+1\n```"},+ {"plan": "Reason_B", "answer": "2"},])dspy.configure(lm=lm)@@ -90,10 +90,10 @@ def test_pot_support_multiple_fields(pooled_interpreter):lm = DummyLM([{- "reasoning": "Reason_A",+ "plan": "Reason_A","generated_code": "```python\nmaximum = 6\nminimum = 2\nSUBMIT({'maximum': maximum, 'minimum': minimum})\n```",},- {"reasoning": "Reason_B", "maximum": "6", "minimum": "2"},+ {"plan": "Reason_B", "maximum": "6", "minimum": "2"},])dspy.configure(lm=lm)@@ -108,14 +108,14 @@ def test_pot_code_generation_with_one_error(pooled_interpreter):lm = DummyLM([{- "reasoning": "Reason_A",+ "plan": "Reason_A","generated_code": "```python\nresult = 1+0/0\nSUBMIT({'answer': result})\n```",},{- "reasoning": "Reason_B",+ "plan": "Reason_B","generated_code": "```python\nresult = 1+1\nSUBMIT({'answer': result})\n```",},- {"reasoning": "Reason_C", "answer": "2"},+ {"plan": "Reason_C", "answer": "2"},])dspy.configure(lm=lm)@@ -240,7 +240,7 @@ def test_pot_code_generation_persistent_errors():lm = DummyLM([{- "reasoning": "Reason_A",+ "plan": "Reason_A","generated_code": "```python\nresult = 1+0/0\nSUBMIT({'answer': result})\n```",},]@@ -257,7 +257,7 @@ def test_pot_code_parse_error():max_iters = 3lm = DummyLM([- {"reasoning": "Reason_A", "generated_code": "```python\ninvalid=python=code\n```"},+ {"plan": "Reason_A", "generated_code": "```python\ninvalid=python=code\n```"},]* max_iters)diff --git a/tests/predict/test_react.py b/tests/predict/test_react.pyindex 97cc2d828..4d180364f 100644--- a/tests/predict/test_react.py+++ b/tests/predict/test_react.py@@ -39,7 +39,7 @@ def test_tool_observation_preserves_custom_type():"next_tool_name": "finish","next_tool_args": {},},- {"reasoning": "image ready", "answer": "done"},+ {"plan": "image ready", "answer": "done"},],adapter=adapter,)@@ -96,7 +96,7 @@ def test_tool_calling_with_pydantic_args():"next_tool_args": {},},{- "reasoning": "This is a very rigorous reasoning process, trust me bro!",+ "plan": "This is a very rigorous reasoning process, trust me bro!","invitation_letter": "It's my honor to invite Alice to the Science Fair event on Friday.",},]@@ -165,7 +165,7 @@ def test_react_with_tools_skips_native_response_issubclass_for_generic_alias(mon"next_tool_args": {},},{- "reasoning": "I fetched the user profile and can answer the request.",+ "plan": "I fetched the user profile and can answer the request.","process_result": "Resolved Adam's request.",},]@@ -189,7 +189,7 @@ def test_tool_calling_without_typehint():[{"next_thought": "I need to add two numbers.", "next_tool_name": "foo", "next_tool_args": {"a": 1, "b": 2}},{"next_thought": "I have the sum, now I can finish.", "next_tool_name": "finish", "next_tool_args": {}},- {"reasoning": "I added the numbers successfully", "c": 3},+ {"plan": "I added the numbers successfully", "c": 3},])dspy.configure(lm=lm)@@ -359,7 +359,7 @@ def test_error_retry():"next_tool_args": {"a": 1, "b": 2},},# (The model *would* succeed on the 3rd turn, but max_iters=2 stops earlier.)- {"reasoning": "I added the numbers successfully", "c": 3},+ {"plan": "I added the numbers successfully", "c": 3},])dspy.configure(lm=lm)@@ -399,7 +399,7 @@ def test_tool_error_observation_format():"next_tool_name": "failing_tool","next_tool_args": {},},- {"reasoning": "The tool failed.", "answer": "n/a"},+ {"plan": "The tool failed.", "answer": "n/a"},])dspy.configure(lm=lm)@@ -453,7 +453,7 @@ async def test_async_tool_calling_with_pydantic_args():"next_tool_args": {},},{- "reasoning": "This is a very rigorous reasoning process, trust me bro!",+ "plan": "This is a very rigorous reasoning process, trust me bro!","invitation_letter": "It's my honor to invite Alice to the Science Fair event on Friday.",},]@@ -509,7 +509,7 @@ async def test_async_error_retry():"next_tool_args": {"a": 1, "b": 2},},# (The model *would* succeed on the 3rd turn, but max_iters=2 stops earlier.)- {"reasoning": "I added the numbers successfully", "c": 3},+ {"plan": "I added the numbers successfully", "c": 3},])with dspy.context(lm=lm):diff --git a/tests/predict/test_rlm.py b/tests/predict/test_rlm.pyindex 36fb94986..28d0a2533 100644--- a/tests/predict/test_rlm.py+++ b/tests/predict/test_rlm.py@@ -129,6 +129,33 @@ class TestMockInterpreter:# ============================================================================[email protected]("async_mode", [False, True])[email protected]("mode", ["native", "omitted", "unsupported"])+async def test_default_reasoning_trajectory(mode, async_mode):+ class Model(DummyLM):+ @property+ def supports_reasoning(self):+ return mode != "unsupported"++ responses = [{"code": "print(13)"}, {"code": "print(missing)"}, {"code": "SUBMIT(13)"}]+ if mode == "unsupported":+ for response in responses:+ response["plan"] = "Check the value before submitting."+ lm = Model(responses, reasoning=mode == "native")+ factory = MockInterpreterFactory(responses=[+ "13", CodeExecutionError("name 'missing' is not defined"), FinalOutput({"answer": 13}),+ ])+ program = RLM("query -> answer: int", max_iters=3, interpreter_factory=factory)+ with dspy.context(lm=lm):+ result = await program.acall(query="Return thirteen") if async_mode else program(query="Return thirteen")+ expected = "Some reasoning" if mode == "native" else "" if mode == "omitted" else responses[0]["plan"]+ assert result.answer == 13+ assert result.final_reasoning == expected+ assert len(result.trajectory) == 3+ assert all(entry["reasoning"] == expected and isinstance(entry["reasoning"], str) for entry in result.trajectory)++class TestRLMInitialization:"""Tests for RLM module initialization."""@@ -1488,7 +1515,7 @@ class TestRLMWithDummyLM:"""Test full RLM pipeline: DummyLM -> RLM -> PythonInterpreter -> result."""configured = MockInterpreterFactory()with dummy_lm_context([- {"reasoning": "I need to compute 2 + 3", "code": "result = 2 + 3\nSUBMIT(result)"},+ {"plan": "I need to compute 2 + 3", "code": "result = 2 + 3\nSUBMIT(result)"},]):rlm = RLM("query -> answer: int", max_iters=3)with dspy.context(interpreter_factory=configured):@@ -1501,8 +1528,8 @@ class TestRLMWithDummyLM:def test_multi_turn_computation_e2e(self):"""Test RLM with multiple turns before SUBMIT."""with dummy_lm_context([- {"reasoning": "First explore the data", "code": "x = 10\nprint(f'x = {x}')"},- {"reasoning": "Now compute and return", "code": "y = x * 2\nSUBMIT(y)"},+ {"plan": "First explore the data", "code": "x = 10\nprint(f'x = {x}')"},+ {"plan": "Now compute and return", "code": "y = x * 2\nSUBMIT(y)"},]):rlm = RLM("query -> answer: int", max_iters=5)result = rlm.forward(query="Double ten", interpreter_factory=PythonInterpreter)@@ -1513,7 +1540,7 @@ class TestRLMWithDummyLM:def test_with_input_variables_e2e(self):"""Test RLM with input variables passed to sandbox."""with dummy_lm_context([- {"reasoning": "Sum the numbers in the list", "code": "SUBMIT(sum(numbers))"},+ {"plan": "Sum the numbers in the list", "code": "SUBMIT(sum(numbers))"},]):rlm = RLM("numbers: list[int] -> total: int", max_iters=3)result = rlm.forward(numbers=[1, 2, 3, 4, 5], interpreter_factory=PythonInterpreter)@@ -1526,7 +1553,7 @@ class TestRLMWithDummyLM:return {"apple": "red", "banana": "yellow"}.get(key, "unknown")with dummy_lm_context([- {"reasoning": "Look up the color of apple", "code": 'color = lookup(key="apple")\nSUBMIT(color)'},+ {"plan": "Look up the color of apple", "code": 'color = lookup(key="apple")\nSUBMIT(color)'},]):rlm = RLM("fruit -> color: str", max_iters=3, tools=[lookup])result = rlm.forward(fruit="apple", interpreter_factory=PythonInterpreter)@@ -1567,7 +1594,7 @@ class TestRLMWithDummyLM:with dummy_lm_context([{- "reasoning": "Call the tool",+ "plan": "Call the tool","code": 'result = score_payload({"value": 3})\nSUBMIT(result)',},]):@@ -1585,7 +1612,7 @@ class TestRLMWithDummyLM:async def test_aforward_simple_computation_e2e(self):"""Test aforward() full pipeline: DummyLM -> RLM -> PythonInterpreter -> result."""with dummy_lm_context([- {"reasoning": "I need to compute 2 + 3", "code": "result = 2 + 3\nSUBMIT(result)"},+ {"plan": "I need to compute 2 + 3", "code": "result = 2 + 3\nSUBMIT(result)"},]):rlm = RLM("query -> answer: int", max_iters=3)result = await rlm.aforward(query="What is 2 + 3?")@@ -1597,8 +1624,8 @@ class TestRLMWithDummyLM:async def test_aforward_multi_turn_e2e(self):"""Test aforward() with multiple turns before SUBMIT."""with dummy_lm_context([- {"reasoning": "First explore the data", "code": "x = 10\nprint(f'x = {x}')"},- {"reasoning": "Now compute and return", "code": "y = x * 2\nSUBMIT(y)"},+ {"plan": "First explore the data", "code": "x = 10\nprint(f'x = {x}')"},+ {"plan": "Now compute and return", "code": "y = x * 2\nSUBMIT(y)"},]):rlm = RLM("query -> answer: int", max_iters=5)result = await rlm.aforward(query="Double ten")@@ -1610,7 +1637,7 @@ class TestRLMWithDummyLM:async def test_aforward_with_input_variables_e2e(self):"""Test aforward() with input variables passed to sandbox."""with dummy_lm_context([- {"reasoning": "Sum the numbers in the list", "code": "SUBMIT(sum(numbers))"},+ {"plan": "Sum the numbers in the list", "code": "SUBMIT(sum(numbers))"},]):rlm = RLM("numbers: list[int] -> total: int", max_iters=3)result = await rlm.aforward(numbers=[1, 2, 3, 4, 5])diff --git a/tests/primitives/test_base_module.py b/tests/primitives/test_base_module.pyindex 9d7ff19a6..c10bb1129 100644--- a/tests/primitives/test_base_module.py+++ b/tests/primitives/test_base_module.py@@ -106,7 +106,7 @@ def test_save_and_load_with_pkl(tmp_path):trainset = [dspy.Example(**example).with_inputs("current_date", "target_date") for example in trainset]dspy.configure(- lm=DummyLM([{"date_diff": "1", "reasoning": "n/a"}, {"date_diff": "2", "reasoning": "n/a"}] * 10)+ lm=DummyLM([{"date_diff": "1", "plan": "n/a"}, {"date_diff": "2", "plan": "n/a"}] * 10))cot = dspy.ChainOfThought(MySignature)@@ -400,7 +400,7 @@ def test_module_history():with patch("litellm.completion") as mock_completion:mock_completion.return_value = ModelResponse(choices=[- Choices(message=Message(content="{'reasoning': 'Paris is the capital of France', 'answer': 'Paris'}"))+ Choices(message=Message(content="{'plan': 'Paris is the capital of France', 'answer': 'Paris'}"))],model="openai/gpt-4o-mini",)@@ -419,7 +419,7 @@ def test_module_history():# The same history entity is shared across all the ancestor callers to reduce memory usage.assert id(program.history[0]) == id(program.cot.history[0])- assert program.history[0]["outputs"] == ["{'reasoning': 'Paris is the capital of France', 'answer': 'Paris'}"]+ assert program.history[0]["outputs"] == ["{'plan': 'Paris is the capital of France', 'answer': 'Paris'}"]dspy.configure(disable_history=True)@@ -449,7 +449,7 @@ def test_module_history_with_concurrency():with patch("litellm.completion") as mock_completion:mock_completion.return_value = ModelResponse(- choices=[Choices(message=Message(content="{'reasoning': 'N/A', 'answer': 'Holy crab!'}"))],+ choices=[Choices(message=Message(content="{'plan': 'N/A', 'answer': 'Holy crab!'}"))],model="openai/gpt-4o-mini",)dspy.configure(lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter())@@ -481,7 +481,7 @@ async def test_module_history_async():with patch("litellm.acompletion") as mock_completion:mock_completion.return_value = ModelResponse(choices=[- Choices(message=Message(content="{'reasoning': 'Paris is the capital of France', 'answer': 'Paris'}"))+ Choices(message=Message(content="{'plan': 'Paris is the capital of France', 'answer': 'Paris'}"))],model="openai/gpt-4o-mini",)@@ -500,7 +500,7 @@ async def test_module_history_async():# The same history entity is shared across all the ancestor callers to reduce memory usage.assert id(program.history[0]) == id(program.cot.history[0])- assert program.history[0]["outputs"] == ["{'reasoning': 'Paris is the capital of France', 'answer': 'Paris'}"]+ assert program.history[0]["outputs"] == ["{'plan': 'Paris is the capital of France', 'answer': 'Paris'}"]with dspy.context(disable_history=True, lm=dspy.LM("openai/gpt-4o-mini", engine="litellm", cache=False), adapter=dspy.JSONAdapter()diff --git a/tests/streaming/test_streaming.py b/tests/streaming/test_streaming.pyindex 1233ae7df..b3c86c8fb 100644--- a/tests/streaming/test_streaming.py+++ b/tests/streaming/test_streaming.py@@ -2150,6 +2150,54 @@ async def test_async_streamify_raises_the_public_error_itself():pass[email protected]("adapter_type", [dspy.ChatAdapter, dspy.JSONAdapter, dspy.XMLAdapter])[email protected]("mode", ["native", "unsupported", None, "none"])+async def test_default_cot_fallback_streams_reasoning(adapter_type, mode):+ texts = {+ dspy.ChatAdapter: "[[ ## plan ## ]]\nCount the signed items.\n\n[[ ## answer ## ]]\n17\n\n[[ ## completed ## ]]",+ dspy.JSONAdapter: '{"plan": "Count the signed items.", "answer": "17"}',+ dspy.XMLAdapter: "<plan>Count the signed items.</plan><answer>17</answer>",+ }+ if mode == "native":+ texts = {+ dspy.ChatAdapter: "[[ ## answer ## ]]\n17\n\n[[ ## completed ## ]]",+ dspy.JSONAdapter: '{"answer": "17"}',+ dspy.XMLAdapter: "<answer>17</answer>",+ }++ async def stream(*args, **kwargs):+ text = texts[adapter_type]+ for offset in range(0, len(text), 4):+ # Providers can return thinking even when disabled, including alongside content.+ yield ModelResponseStream(choices=[StreamingChoices(delta=Delta(+ content=text[offset:offset + 4], reasoning_content="Native thinking." if offset == 0 else None,+ ))])++ model = "openai/gpt-4o-mini" if mode == "unsupported" else "openai/gpt-5"+ lm = dspy.LM(model, engine="litellm", cache=False, max_tokens=16000)+ lm.kwargs["reasoning_effort"] = "high"+ config = {"reasoning_effort": mode} if mode in (None, "none") else {}+ program = dspy.streamify(+ dspy.ChainOfThought("question -> answer"),+ stream_listeners=[dspy.streaming.StreamListener("reasoning")],+ )+ with mock.patch("litellm.acompletion", side_effect=stream), dspy.context(lm=lm, adapter=adapter_type()):+ values = [value async for value in program(question="Count items", config=config)]+ chunks = [value for value in values if isinstance(value, StreamResponse)]+ assert chunks and all(c.signature_field_name == "reasoning" for c in chunks)+ expected = "Native thinking." if mode == "native" else "Count the signed items."+ streamed = "".join(c.chunk for c in chunks).strip()+ if adapter_type is dspy.JSONAdapter and mode != "native":+ # JSON listeners emit raw JSON fragments, not decoded strings.+ assert f'"{expected}"' in streamed+ assert "Native thinking." not in streamed+ else:+ assert streamed == expected+ assert values[-1].reasoning == expected+ assert values[-1].answer == "17"++def test_streamify_keeps_a_group_of_several_failures():import builtinsdiff --git a/tests/teleprompt/test_copro_optimizer.py b/tests/teleprompt/test_copro_optimizer.pyindex 397f2c262..c67ba3440 100644--- a/tests/teleprompt/test_copro_optimizer.py+++ b/tests/teleprompt/test_copro_optimizer.py@@ -118,13 +118,13 @@ def test_optimization_and_output_verification():lm = DummyLM([{"proposed_instruction": "Optimized Prompt", "proposed_prefix_for_output_field": "Optimized Prefix"},- {"reasoning": "france", "output": "Paris"},- {"reasoning": "france", "output": "Paris"},- {"reasoning": "france", "output": "Paris"},- {"reasoning": "france", "output": "Paris"},- {"reasoning": "france", "output": "Paris"},- {"reasoning": "france", "output": "Paris"},- {"reasoning": "france", "output": "Paris"},+ {"plan": "france", "output": "Paris"},+ {"plan": "france", "output": "Paris"},+ {"plan": "france", "output": "Paris"},+ {"plan": "france", "output": "Paris"},+ {"plan": "france", "output": "Paris"},+ {"plan": "france", "output": "Paris"},+ {"plan": "france", "output": "Paris"},])dspy.configure(lm=lm)