DSPy: The framework for programming—not prompting—language models
Python
fix(predict): isolate interpreters per invocation (#10022)
Isaac Miller623360078f5d
diff --git a/docs/docs/api/modules/RLM.md b/docs/docs/api/modules/RLM.mdindex cf2250b3e..d873caec7 100644--- a/docs/docs/api/modules/RLM.md+++ b/docs/docs/api/modules/RLM.md@@ -115,7 +115,7 @@ $20,000,000| `verbose` | `bool` | `False` | Log detailed execution info || `tools` | `list[Union[Callable, dspy.Tool]]` | `None` | Additional tool functions callable from interpreter code || `sub_lm` | `dspy.LM` | `None` | LM for sub-queries. Defaults to `dspy.settings.lm`. Use a cheaper model here. |-| `interpreter` | `CodeInterpreter` | `None` | Custom interpreter. Defaults to `PythonInterpreter` (Deno/Pyodide WASM). |+| `interpreter_factory` | `Callable[[], CodeInterpreter]` | `PythonInterpreter` | Creates one interpreter per invocation. RLM shuts down each returned interpreter. |## Built-in Tools@@ -187,6 +187,24 @@ rlm = dspy.RLM()```+### Configuring the Interpreter++Pass the interpreter class directly when it needs no arguments. For configured construction, use any zero-argument callable, such as `functools.partial`:++```python+from functools import partial++rlm = dspy.RLM(+ "context, query -> answer",+ interpreter_factory=partial(+ dspy.PythonInterpreter,+ enable_network_access=["example.com"],+ ),+)+```++RLM creates and shuts down one interpreter from this factory per invocation. It adds invocation-scoped tools to the returned interpreter's mutable `tools` dictionary, so remote sandboxes need a `CodeInterpreter` adapter that supports that protocol. To reuse a caller-owned interpreter, pass it as the first positional argument when calling the module: `rlm(interpreter, context=data, query=query)`. RLM updates its tools and output metadata but does not shut down or restore it. Reuse is supported only for sequential calls to the same RLM instance; use the factory path for concurrency.+### Custom Sandbox-Serializable InputsFor inputs that should be loaded into the sandbox differently from normal Python values, subclass `dspy.SandboxSerializable`. RLM detects these inputs, sends their serialized payload into the interpreter, runs their setup code, and exposes the reconstructed value under the original input name.@@ -216,7 +234,7 @@ import asynciorlm = dspy.RLM("context, query -> answer")async def process():- result = await rlm.aforward(context=data, query="Summarize this")+ result = await rlm.acall(context=data, query="Summarize this")return result.answeranswer = asyncio.run(process())@@ -247,7 +265,7 @@ RLM returns a `Prediction` with:RLM is marked as experimental. The API may change in future releases.!!! note "Thread Safety"- RLM instances are not thread-safe when using a custom interpreter. Create separate instances for concurrent use, or use the default `PythonInterpreter` which creates a fresh instance per `forward()` call.+ `interpreter_factory` may be called concurrently and must return a fresh interpreter each time. An interpreter passed as the first positional argument to `rlm(...)` or `rlm.acall(...)` is caller-owned and may be reused only for sequential calls to the same RLM instance. `PythonInterpreter` must also stay on the thread where it was first used.!!! note "Interpreter Requirements"The default `PythonInterpreter` requires [Deno](https://deno.land/) to be installed for the Pyodide WASM sandbox.diff --git a/docs/docs/diving-deeper/built-in-module-variants.md b/docs/docs/diving-deeper/built-in-module-variants.mdindex 83de88ccf..31ee74138 100644--- a/docs/docs/diving-deeper/built-in-module-variants.md+++ b/docs/docs/diving-deeper/built-in-module-variants.md@@ -26,7 +26,7 @@ The signature `reward_fn(args, pred) -> float` mirrors the inference-time metric### 5. `ProgramOfThought` and `CodeAct` ship with a Python interpreter-Both rely on `PythonInterpreter`, which runs LM-generated code in a sandbox via Deno’s WASM runtime. Deno isolation is the reason: the LM’s code runs in a process with no filesystem or network access by default, so executing untrusted output is bounded. Installing Deno is a hard dependency for both modules; the constructor raises if Deno isn’t available.+Both rely on `PythonInterpreter`, which runs LM-generated code in a sandbox via Deno’s WASM runtime. Deno isolation is the reason: the LM’s code runs in a process with no filesystem or network access by default, so executing untrusted output is bounded. Each invocation gets a fresh interpreter, while retries and iterations within that invocation share its state.### 6. `CodeAct` is `ReAct` plus a code sandbox@@ -74,13 +74,15 @@ The constructor mutates the input signature: it prepends one output field (a synFor tasks where the answer is best computed, not narrated.-**`dspy.ProgramOfThought(signature, max_iters=3, interpreter=None)`**-Holds three internal `ChainOfThought` predictors: `code_generate` produces Python, `code_regenerate` rewrites it after an execution error, and `generate_output` extracts the declared output fields from the run’s printed result. The forward loop asks `code_generate` for code, runs it through the `PythonInterpreter`, and on error feeds the error message back to `code_regenerate` for up to `max_iters` rounds. Once execution succeeds, `generate_output` produces the signature’s output fields. If `max_iters` is exhausted, the module raises.+Each module accepts an `interpreter_factory` that is called once per invocation; DSPy shuts down the returned interpreter even when the invocation raises. Passing an interpreter as the first positional argument when calling the module, such as `program(interpreter, **inputs)`, instead uses that caller-owned instance without shutting it down. Caller-owned reuse is sequential; use the factory path for concurrent invocations. A `PythonInterpreter` override must also stay on the thread where it was first used.-**`dspy.CodeAct(signature, tools, max_iters=5, interpreter=None)`**-Multiple inheritance from `ReAct` and `ProgramOfThought`. Tools must be plain `def` functions, not callable objects — the module reads `inspect.getsource(tool.func)` and injects each definition into the sandbox at the start of every `forward`. Each iteration: an inner `codeact` predictor produces Python plus a `finished` boolean; the interpreter runs the code; the trajectory dict gains a `generated_code_i` and `code_output_i` (or `observation_i` on parse/execution error). The loop exits when the LM sets `finished=True` or `max_iters` is reached. A `ChainOfThought` extractor then reads the trajectory and produces the declared outputs.+**`dspy.ProgramOfThought(signature, max_iters=3, interpreter_factory=PythonInterpreter)`**+Holds three internal `ChainOfThought` predictors: `code_generate` produces Python, `code_regenerate` rewrites it after a recoverable execution error, and `generate_output` extracts the declared output fields from the run’s printed result. The forward loop asks `code_generate` for code, runs it through the `PythonInterpreter`, and feeds `CodeExecutionError` or `SyntaxError` back to `code_regenerate` for up to `max_iters` rounds. A terminal `CodeInterpreterError` propagates immediately. Once execution succeeds, `generate_output` produces the signature’s output fields. If `max_iters` is exhausted, the module raises.-**`dspy.RLM(signature, max_iters=20, max_llm_calls=50, max_output_chars=10_000, verbose=False, tools=None, sub_lm=None, interpreter=None)`**+**`dspy.CodeAct(signature, tools, max_iters=5, interpreter_factory=PythonInterpreter)`**+Multiple inheritance from `ReAct` and `ProgramOfThought`. Tools must be plain `def` functions, not callable objects — the module reads `inspect.getsource(tool.func)` and injects each definition into the sandbox at the start of every `forward`. Each iteration: an inner `codeact` predictor produces Python plus a `finished` boolean; the interpreter runs the code; the trajectory dict gains a `generated_code_i` and `code_output_i` (or `observation_i` on a parse or recoverable execution error). Terminal interpreter failures propagate. The loop exits when the LM sets `finished=True` or `max_iters` is reached. A `ChainOfThought` extractor then reads the trajectory and produces the declared outputs.++**`dspy.RLM(signature, max_iters=20, max_llm_calls=50, max_output_chars=10_000, verbose=False, tools=None, sub_lm=None, interpreter_factory=PythonInterpreter)`**Experimental. A REPL-style code agent that exposes two built-in tools — `llm_query` and `llm_query_batched` — so generated code can call a separate `sub_lm` mid-execution. A shared counter across iterations enforces `max_llm_calls`; tool names are validated as Python identifiers; `SandboxSerializable` inputs encode into the sandbox so large contexts don’t have to be re-marshalled each turn. If the loop ends without an explicit submission, the extractor pass produces the final outputs from the trajectory.### Running modules in paralleldiff --git a/docs/docs/diving-deeper/rlm.md b/docs/docs/diving-deeper/rlm.mdindex 995c3f8a8..e1ca0bcf5 100644--- a/docs/docs/diving-deeper/rlm.md+++ b/docs/docs/diving-deeper/rlm.md@@ -32,7 +32,7 @@ The model steering the loop and the model answering snippets need not be the sam### 7. Generated code runs in a pluggable sandboxed interpreter-The model’s code is untrusted, so it runs in a sandbox. The default `PythonInterpreter` executes Python in a Deno and Pyodide WASM runtime with no filesystem or network access, created fresh per `forward()` and shut down afterward. You can pass any `CodeInterpreter` through `interpreter=`, such as a mock for tests or a remote sandbox like E2B or Modal for heavier work.+The model’s code is untrusted, so it runs in a sandbox. The default `PythonInterpreter` executes Python in a Deno and Pyodide WASM runtime with no filesystem or network access. Each `forward()` creates one interpreter, uses it for the whole REPL loop, and shuts it down afterward. Supply `interpreter_factory=` to configure another `CodeInterpreter`, such as an adapter for a remote sandbox.### 8. `SandboxSerializable` loads large inputs into the sandbox once@@ -54,11 +54,11 @@ The class carries the `@experimental` decorator. The hard parts are still settli### Defining and running an RLM-**`dspy.RLM(signature, max_iters=20, max_llm_calls=50, max_output_chars=10_000, verbose=False, tools=None, sub_lm=None, interpreter=None)`**+**`dspy.RLM(signature, max_iters=20, max_llm_calls=50, max_output_chars=10_000, verbose=False, tools=None, sub_lm=None, interpreter_factory=PythonInterpreter)`**The constructor parses the signature and builds the two internal predictors, formatting your task instructions, input names, and output-field types into the action prompt and creating a separate extract signature for the fallback. The budgets and the sandbox configuration are all fixed here, so one instance carries one configuration.-**`forward(**inputs)` / `aforward(**inputs)` / `__call__`**-`forward()` validates the inputs against the signature, builds the variable list, opens the interpreter, and runs the loop. Each turn asks the action predictor for code, runs it, and appends the result to history until the model submits or the loop reaches `max_iters`. `aforward()` is the async twin and uses `acall` on the predictors. Both return a `Prediction`.+**`__call__([interpreter], **inputs)` / `acall([interpreter], **inputs)`**+The public call validates the inputs against the signature, builds the variable list, opens the interpreter, and runs the loop. Each turn asks the action predictor for code, runs it, and appends the result to history until the model submits or the loop reaches `max_iters`. `acall()` is the async twin and uses `acall` on the predictors. Both return a `Prediction`. An interpreter passed as the first positional argument is caller-owned: RLM updates its execution context but does not shut it down.### Programming the loop with built-in tools@@ -89,8 +89,11 @@ The model for `llm_query` and `llm_query_batched`. Left unset it falls back to `**`tools=[...]`**A list of plain functions or `dspy.Tool` objects. RLM normalizes each to a `Tool`, rejects names that aren’t valid identifiers or that collide with the built-ins, and documents their signatures in the action prompt. The model calls them as ordinary Python inside its code.-**`interpreter=...`**-Any `CodeInterpreter`. RLM injects the query tools and output-field metadata into it on each `forward()`, so a custom interpreter still gets `llm_query` and typed submission. Reach for this to run a remote sandbox or a test mock.+**`interpreter_factory=...`**+A zero-argument callable that returns a fresh `CodeInterpreter` for one invocation. RLM may call the factory concurrently, and it always shuts down the returned interpreter. A class such as `PythonInterpreter` is already a factory; use `functools.partial` or a callable provider object when construction needs configuration. RLM adds invocation-scoped tools to the returned interpreter's mutable `tools` dictionary, so remote sandboxes need a `CodeInterpreter` adapter that supports that protocol.++**`__call__(interpreter, **inputs)` / `acall(interpreter, **inputs)`**+An escape hatch for a caller-owned interpreter, supplied as the first positional argument. RLM mutates its `tools` dictionary and, when supported, its output-field metadata, but does not shut down or restore the instance. Reuse is supported only for sequential calls to the same RLM instance, so retained variables and tool registrations stay within one program and trust boundary. Use `interpreter_factory` for concurrent invocations. A `PythonInterpreter` override must also stay on the thread where it was first used.**`dspy.SandboxSerializable`**The base class for inputs that need custom loading. Implement `sandbox_setup`, `to_sandbox`, `sandbox_assignment`, and `rlm_preview`. It also defines a Pydantic schema hook, so a subclass can be a typed field in a signature, as in `data: DataFrame = dspy.InputField()`.diff --git a/dspy/predict/code_act.py b/dspy/predict/code_act.pyindex 1773c8da1..f53023993 100644--- a/dspy/predict/code_act.py+++ b/dspy/predict/code_act.py@@ -6,6 +6,7 @@ import dspyfrom dspy.adapters.types.tool import Toolfrom dspy.predict.program_of_thought import ProgramOfThoughtfrom dspy.predict.react import ReAct+from dspy.primitives.code_interpreter import CodeInterpreter, _validate_interpreter_factoryfrom dspy.primitives.python_interpreter import PythonInterpreterfrom dspy.signatures.signature import Signature, ensure_signature@@ -16,7 +17,13 @@ class CodeAct(ReAct, ProgramOfThought):CodeAct is a module that utilizes the Code Interpreter and predefined tools to solve the problem."""- def __init__(self, signature: str | type[Signature], tools: list[Callable], max_iters: int = 5, interpreter: PythonInterpreter | None = None):+ def __init__(+ self,+ signature: str | type[Signature],+ tools: list[Callable],+ max_iters: int = 5,+ interpreter_factory: Callable[[], CodeInterpreter] = PythonInterpreter,+ ):"""Initializes the CodeAct class with the specified model, temperature, and max tokens.@@ -24,7 +31,8 @@ class CodeAct(ReAct, ProgramOfThought):signature (Union[str, Type[Signature]]): The signature of the module.tools (list[Callable]): The tool callables to be used. CodeAct only accepts functions and not callable objects.max_iters (int): The maximum number of iterations to generate the answer.- interpreter: PythonInterpreter instance to use. If None, a new one is instantiated.+ interpreter_factory: Zero-argument callable that creates an interpreter for each forward pass. The+ callable may be invoked concurrently, and DSPy shuts down each interpreter it returns.Examples:```pythonfrom dspy.predict import CodeAct@@ -37,6 +45,7 @@ class CodeAct(ReAct, ProgramOfThought):act(n=5) # 120```"""+ _validate_interpreter_factory(interpreter_factory)self.signature = ensure_signature(signature)self.max_iters = max_itersself.history = []@@ -65,8 +74,7 @@ class CodeAct(ReAct, ProgramOfThought):self.tools: dict[str, Tool] = toolsself.codeact = dspy.Predict(codeact_signature)self.extractor = dspy.ChainOfThought(extract_signature)- # It will raises exception when dspy cannot find available deno instance by now.- self.interpreter = interpreter or PythonInterpreter()+ self._interpreter_factory = interpreter_factorydef _build_instructions(self, signature, tools):instructions = [f"{signature.instructions}\n"] if signature.instructions else []@@ -87,33 +95,46 @@ class CodeAct(ReAct, ProgramOfThought):return instructions- def forward(self, **kwargs):- # Define the tool functions in the interpreter- for tool in self.tools.values():- self.interpreter(inspect.getsource(tool.func))+ def forward(self, interpreter: CodeInterpreter | None = None, /, **kwargs):+ """Run the program with a fresh interpreter or a caller-owned override.- trajectory = {}- max_iters = kwargs.pop("max_iters", self.max_iters)- for idx in range(max_iters):- code_data = self.codeact(trajectory=trajectory, **kwargs)- output = None- code, error = self._parse_code(code_data)-- if error:- trajectory[f"observation_{idx}"] = f"Failed to parse the generated code: {error}"- continue-- trajectory[f"generated_code_{idx}"] = code- output, error = self._execute_code(code)-- if not error:- trajectory[f"code_output_{idx}"] = output- else:- trajectory[f"observation_{idx}"] = f"Failed to execute the generated code: {error}"-- if code_data.finished:- break+ Args:+ interpreter: Optional caller-owned interpreter, passed positionally. The caller must shut it down.+ **kwargs: Input values matching the signature's input fields.- extract = self._call_with_potential_trajectory_truncation(self.extractor, trajectory, **kwargs)- self.interpreter.shutdown()- return dspy.Prediction(trajectory=trajectory, **extract)+ Raises:+ CodeInterpreterError: If interpreter setup, process, or protocol fails.+ """+ if "interpreter" in kwargs and "interpreter" not in self.signature.input_fields:+ raise TypeError(+ "To use a caller-owned interpreter, pass it as the first positional argument when calling the module."+ )+ with self._interpreter_context(interpreter) as interpreter:+ # Define the tool functions in the interpreter+ for tool in self.tools.values():+ interpreter.execute(inspect.getsource(tool.func))++ trajectory = {}+ max_iters = kwargs.pop("max_iters", self.max_iters)+ for idx in range(max_iters):+ code_data = self.codeact(trajectory=trajectory, **kwargs)+ output = None+ code, error = self._parse_code(code_data)++ if error:+ trajectory[f"observation_{idx}"] = f"Failed to parse the generated code: {error}"+ continue++ trajectory[f"generated_code_{idx}"] = code+ output, error = self._execute_code(code, interpreter)++ if not error:+ trajectory[f"code_output_{idx}"] = output+ else:+ trajectory[f"observation_{idx}"] = f"Failed to execute the generated code: {error}"++ if code_data.finished:+ break++ extract = self._call_with_potential_trajectory_truncation(self.extractor, trajectory, **kwargs)+ return dspy.Prediction(trajectory=trajectory, **extract)diff --git a/dspy/predict/program_of_thought.py b/dspy/predict/program_of_thought.pyindex 24b4d668a..3b29ea008 100644--- a/dspy/predict/program_of_thought.py+++ b/dspy/predict/program_of_thought.py@@ -1,9 +1,18 @@import jsonimport loggingimport re+from collections.abc import Callable, Iterator+from contextlib import contextmanagerimport dspy-from dspy.primitives.code_interpreter import FinalOutput+from dspy.primitives.code_interpreter import (+ CodeExecutionError,+ CodeInterpreter,+ FinalOutput,+ _create_interpreter,+ _validate_interpreter,+ _validate_interpreter_factory,+)from dspy.primitives.module import Modulefrom dspy.primitives.python_interpreter import PythonInterpreterfrom dspy.signatures.signature import Signature, ensure_signature@@ -27,14 +36,21 @@ class ProgramOfThought(Module):```"""- def __init__(self, signature: str | type[Signature], max_iters: int = 3, interpreter: PythonInterpreter | None = None):+ def __init__(+ self,+ signature: str | type[Signature],+ max_iters: int = 3,+ interpreter_factory: Callable[[], CodeInterpreter] = PythonInterpreter,+ ):"""Args:signature: The signature of the module.max_iters: The maximum number of iterations to retry code generation and execution.- interpreter: PythonInterpreter instance to use. If None, a new one is instantiated.+ interpreter_factory: Zero-argument callable that creates an interpreter for each forward pass. The+ callable may be invoked concurrently, and DSPy shuts down each interpreter it returns."""super().__init__()+ _validate_interpreter_factory(interpreter_factory)self.signature = signature = ensure_signature(signature)self.max_iters = max_iters@@ -59,8 +75,21 @@ class ProgramOfThought(Module):self._generate_instruction("answer"),),)- # It will raises exception when dspy cannot find available deno instance by now.- self.interpreter = interpreter or PythonInterpreter()+ self._interpreter_factory = interpreter_factory++ @contextmanager+ def _interpreter_context(self, interpreter: CodeInterpreter | None) -> Iterator[CodeInterpreter]:+ """Yield a caller-owned interpreter or manage a factory-created one."""+ if interpreter is not None:+ _validate_interpreter(interpreter)+ yield interpreter+ return++ interpreter = _create_interpreter(self._interpreter_factory)+ try:+ yield interpreter+ finally:+ interpreter.shutdown()def _generate_signature(self, mode):signature_dict = dict(self.input_fields)@@ -137,44 +166,55 @@ class ProgramOfThought(Module):code_block += "\n" + last_line_match.group(1)return code_block, None- def _execute_code(self, code):+ def _execute_code(self, code: str, interpreter: CodeInterpreter):"""- Execute the code using PythonInterpreter and return the output or error.+ Execute the code using the current interpreter and return the output or error."""if not code:return None, "Error: Empty code before execution."try:- result = self.interpreter.execute(code)+ result = interpreter.execute(code)if isinstance(result, FinalOutput):result = result.output# Since it's more complex structure now, just blindly use json to represents all.output = json.dumps(result)return output, None- except Exception as e:+ except (CodeExecutionError, SyntaxError) as e:return None, str(e)- def forward(self, **kwargs):- input_kwargs = {field_name: kwargs[field_name] for field_name in self.input_fields}- code_data = self.code_generate(**input_kwargs)- output = None- code, error = self._parse_code(code_data)- if not error:- output, error = self._execute_code(code)- hop = 1- # Retrying code generation and execution until no error or reach max_iters- while error is not None:- logger.error(f"Error in code execution: {error}")- if hop == self.max_iters:- self.interpreter.shutdown()- raise RuntimeError(f"Max hops reached. Failed to run ProgramOfThought: {error}")- input_kwargs.update({"previous_code": code, "error": error})- code_data = self.code_regenerate(**input_kwargs)+ def forward(self, interpreter: CodeInterpreter | None = None, /, **kwargs):+ """Run the program with a fresh interpreter or a caller-owned override.++ Args:+ interpreter: Optional caller-owned interpreter, passed positionally. The caller must shut it down.+ **kwargs: Input values matching the signature's input fields.++ Raises:+ CodeInterpreterError: If interpreter setup, process, or protocol fails.+ """+ if "interpreter" in kwargs and "interpreter" not in self.input_fields:+ raise TypeError(+ "To use a caller-owned interpreter, pass it as the first positional argument when calling the module."+ )+ with self._interpreter_context(interpreter) as interpreter:+ input_kwargs = {field_name: kwargs[field_name] for field_name in self.input_fields}+ code_data = self.code_generate(**input_kwargs)+ output = Nonecode, error = self._parse_code(code_data)if not error:- output, error = self._execute_code(code)- hop += 1- input_kwargs.update({"final_generated_code": code, "code_output": output})- output_gen_result = self.generate_output(**input_kwargs)- self.interpreter.shutdown()- return output_gen_result+ output, error = self._execute_code(code, interpreter)+ hop = 1+ # Retrying code generation and execution until no error or reach max_iters+ while error is not None:+ logger.error(f"Error in code execution: {error}")+ if hop == self.max_iters:+ raise RuntimeError(f"Max hops reached. Failed to run ProgramOfThought: {error}")+ input_kwargs.update({"previous_code": code, "error": error})+ code_data = self.code_regenerate(**input_kwargs)+ code, error = self._parse_code(code_data)+ if not error:+ output, error = self._execute_code(code, interpreter)+ hop += 1+ input_kwargs.update({"final_generated_code": code, "code_output": output})+ return self.generate_output(**input_kwargs)diff --git a/dspy/predict/rlm.py b/dspy/predict/rlm.pyindex 8343d742e..7febd2bd9 100644--- a/dspy/predict/rlm.py+++ b/dspy/predict/rlm.py@@ -23,7 +23,15 @@ import pydanticimport dspyfrom dspy.adapters.types.tool import Toolfrom dspy.adapters.utils import parse_value, translate_field_type-from dspy.primitives.code_interpreter import SIMPLE_TYPES, CodeExecutionError, CodeInterpreter, FinalOutput+from dspy.primitives.code_interpreter import (+ SIMPLE_TYPES,+ CodeExecutionError,+ CodeInterpreter,+ FinalOutput,+ _create_interpreter,+ _validate_interpreter,+ _validate_interpreter_factory,+)from dspy.primitives.module import Modulefrom dspy.primitives.prediction import Predictionfrom dspy.primitives.python_interpreter import PythonInterpreter@@ -109,12 +117,12 @@ class RLM(Module):through code execution. The LLM writes Python code to examine data, callsub-LLMs for semantic analysis, and build up answers iteratively.- The default interpreter is PythonInterpreter (Deno/Pyodide/WASM), but you- can provide any CodeInterpreter implementation (e.g., MockInterpreter, or write a custom one using E2B or Modal).-- Note: RLM instances are not thread-safe when using a custom interpreter.- Create separate RLM instances for concurrent use, or use the default- PythonInterpreter which creates a fresh instance per forward() call.+ The default interpreter is PythonInterpreter (Deno/Pyodide/WASM), but+ ``interpreter_factory`` can create another CodeInterpreter implementation,+ such as an adapter for a remote sandbox. RLM updates the interpreter's+ mutable ``tools`` dictionary with invocation-scoped tools before execution.+ A caller-owned interpreter may be reused sequentially with the same RLM+ instance, but must not be shared by overlapping invocations.Examples:```python@@ -134,7 +142,7 @@ class RLM(Module):verbose: bool = False,tools: list[Callable] | None = None,sub_lm: dspy.LM | None = None,- interpreter: CodeInterpreter | None = None,+ interpreter_factory: Callable[[], CodeInterpreter] = PythonInterpreter,):"""Args:@@ -148,16 +156,19 @@ class RLM(Module):Built-in tools: llm_query(prompt), llm_query_batched(prompts).sub_lm: LM for llm_query/llm_query_batched. Defaults to dspy.settings.lm.Allows using a different (e.g., cheaper) model for sub-queries.- interpreter: CodeInterpreter implementation to use. Defaults to PythonInterpreter.+ interpreter_factory: Zero-argument callable that creates an interpreter for each forward pass. The+ callable may be invoked concurrently, and DSPy shuts down each interpreter it returns. RLM updates+ the returned interpreter's mutable ``tools`` dictionary before execution."""super().__init__()+ _validate_interpreter_factory(interpreter_factory)self.signature = ensure_signature(signature)self.max_iters = max_itersself.max_llm_calls = max_llm_callsself.max_output_chars = max_output_charsself.verbose = verboseself.sub_lm = sub_lm- self._interpreter = interpreter+ self._interpreter_factory = interpreter_factoryself._user_tools = self._normalize_tools(tools)self._validate_tools(self._user_tools)@@ -198,7 +209,7 @@ class RLM(Module):def _validate_tools(self, tools: dict[str, Tool]) -> None:"""Validate user-provided tools have valid names."""- for name, tool in tools.items():+ for name in tools:if not name.isidentifier():raise ValueError(f"Invalid tool name '{name}': must be a valid Python identifier")if name in self._RESERVED_TOOL_NAMES:@@ -376,6 +387,10 @@ class RLM(Module):def _validate_inputs(self, input_args: dict[str, Any]) -> None:"""Raise ValueError if required input fields are missing."""+ if "interpreter" in input_args and "interpreter" not in self.signature.input_fields:+ raise TypeError(+ "To use a caller-owned interpreter, pass it as the first positional argument when calling the module."+ )missing = set(self.signature.input_fields.keys()) - set(input_args.keys())if missing:raise ValueError(f"Missing required inputs: {sorted(missing)}")@@ -448,20 +463,24 @@ class RLM(Module):interpreter._tools_registered = False@contextmanager- def _interpreter_context(self, execution_tools: dict[str, Callable]) -> Iterator[CodeInterpreter]:- """Yield interpreter, creating PythonInterpreter if none provided at init."""- if self._interpreter is not None:- self._inject_execution_context(self._interpreter, execution_tools)- yield self._interpreter- else:- repl = PythonInterpreter(- tools=execution_tools,- output_fields=self._get_output_fields_info(),- )- try:- yield repl- finally:- repl.shutdown()+ def _interpreter_context(+ self,+ execution_tools: dict[str, Callable],+ interpreter: CodeInterpreter | None,+ ) -> Iterator[CodeInterpreter]:+ """Yield a caller-owned interpreter or manage a factory-created one."""+ if interpreter is not None:+ _validate_interpreter(interpreter)+ self._inject_execution_context(interpreter, execution_tools)+ yield interpreter+ return++ interpreter = _create_interpreter(self._interpreter_factory)+ try:+ self._inject_execution_context(interpreter, execution_tools)+ yield interpreter+ finally:+ interpreter.shutdown()# =========================================================================# Execution Core@@ -625,11 +644,14 @@ class RLM(Module):# Public Interface# =========================================================================- def forward(self, **input_args) -> Prediction:+ def forward(self, interpreter: CodeInterpreter | None = None, /, **input_args) -> Prediction:"""Execute RLM to produce outputs from the given inputs.Args:- **input_args: Input values matching the signature's input fields+ interpreter: Optional caller-owned interpreter, passed positionally. RLM injects invocation tools and+ output metadata into it but does not shut it down. Reuse is supported only for sequential calls to+ this RLM instance.+ **input_args: Input values matching the signature's input fields.Returns:Prediction with output field(s) from the signature and 'trajectory' for debugging@@ -644,7 +666,7 @@ class RLM(Module):execution_tools = self._prepare_execution_tools()variables = self._build_variables(**input_args)- with self._interpreter_context(execution_tools) as repl:+ with self._interpreter_context(execution_tools, interpreter) as repl:regular_args = self._prepare_serializable_vars(input_args, repl)history: REPLHistory = REPLHistory(max_output_chars=self.max_output_chars)@@ -711,11 +733,14 @@ class RLM(Module):result = self._execute_code(repl, code, input_args)return self._process_execution_result(pred, code, result, history, output_field_names)- async def aforward(self, **input_args) -> Prediction:+ async def aforward(self, interpreter: CodeInterpreter | None = None, /, **input_args) -> Prediction:"""Async version of forward(). Execute RLM to produce outputs.Args:- **input_args: Input values matching the signature's input fields+ interpreter: Optional caller-owned interpreter, passed positionally. RLM injects invocation tools and+ output metadata into it but does not shut it down. Reuse is supported only for sequential calls to+ this RLM instance.+ **input_args: Input values matching the signature's input fields.Returns:Prediction with output field(s) from the signature and 'trajectory' for debugging@@ -730,7 +755,7 @@ class RLM(Module):execution_tools = self._prepare_execution_tools()variables = self._build_variables(**input_args)- with self._interpreter_context(execution_tools) as repl:+ with self._interpreter_context(execution_tools, interpreter) as repl:regular_args = self._prepare_serializable_vars(input_args, repl)history = REPLHistory(max_output_chars=self.max_output_chars)diff --git a/dspy/primitives/code_interpreter.py b/dspy/primitives/code_interpreter.pyindex 16eb3e454..ce43810ae 100644--- a/dspy/primitives/code_interpreter.py+++ b/dspy/primitives/code_interpreter.py@@ -141,3 +141,35 @@ class CodeInterpreter(Protocol):A new instance should be created for a fresh session."""...+++def _validate_interpreter_factory(factory: Any) -> None:+ """Validate the configured provider without invoking it."""+ if not isinstance(factory, type) and isinstance(factory, CodeInterpreter):+ raise TypeError(+ "interpreter_factory received an object that already implements CodeInterpreter, so its ownership "+ "is ambiguous. Pass an existing interpreter as the first positional argument when calling the module. "+ "If this object also creates interpreters, pass a dedicated zero-argument creation callable instead."+ )+ if not callable(factory):+ raise TypeError(+ "interpreter_factory must be a zero-argument callable that creates a CodeInterpreter, "+ f"not {type(factory).__name__}."+ )+++def _create_interpreter(factory: Callable[[], CodeInterpreter]) -> CodeInterpreter:+ """Create an interpreter and validate the factory's return value."""+ interpreter = factory()+ if not isinstance(interpreter, CodeInterpreter):+ raise TypeError(+ "interpreter_factory must return a CodeInterpreter, "+ f"not {type(interpreter).__name__}."+ )+ return interpreter+++def _validate_interpreter(interpreter: Any) -> None:+ """Validate a caller-owned interpreter."""+ if not isinstance(interpreter, CodeInterpreter):+ raise TypeError(f"interpreter must implement CodeInterpreter, not {type(interpreter).__name__}.")diff --git a/tests/mock_interpreter.py b/tests/mock_interpreter.pyindex 255f0282f..9c9c6def8 100644--- a/tests/mock_interpreter.py+++ b/tests/mock_interpreter.py@@ -12,7 +12,7 @@ from typing import Any, Callablefrom dspy.primitives.code_interpreter import CodeInterpreterError, FinalOutput-__all__ = ["MockInterpreter"]+__all__ = ["MockInterpreter", "MockInterpreterFactory"]class MockInterpreter:@@ -130,3 +130,21 @@ class MockInterpreter:) -> Any:"""Shorthand for execute()."""return self.execute(code, variables)+++class MockInterpreterFactory:+ """Callable test provider that records every interpreter it creates."""++ def __init__(+ self,+ responses: list[str | FinalOutput | Exception] | None = None,+ execute_fn: Callable[[str, dict[str, Any]], Any] | None = None,+ ) -> None:+ self.responses = list(responses) if responses else []+ self.execute_fn = execute_fn+ self.instances: list[MockInterpreter] = []++ def __call__(self) -> MockInterpreter:+ interpreter = MockInterpreter(responses=self.responses, execute_fn=self.execute_fn)+ self.instances.append(interpreter)+ return interpreterdiff --git a/tests/predict/test_code_act.py b/tests/predict/test_code_act.pyindex a268d9d85..a9df7ecdd 100644--- a/tests/predict/test_code_act.py+++ b/tests/predict/test_code_act.py@@ -1,9 +1,15 @@+import threading+from unittest.mock import Mock+import pytestimport dspyfrom dspy import Signature+from dspy.evaluate.metrics import answer_exact_matchfrom dspy.predict import CodeAct+from dspy.primitives.code_interpreter import CodeInterpreterErrorfrom dspy.utils import DummyLM+from tests.mock_interpreter import MockInterpreter, MockInterpreterFactorypytestmark = pytest.mark.deno@@ -12,6 +18,19 @@ class BasicQA(Signature):question = dspy.InputField()answer = dspy.OutputField(desc="often between 1 and 5 words")++class StaticPredictor:+ def __init__(self, **fields):+ self.fields = fields++ def __call__(self, **kwargs):+ return dspy.Prediction(**self.fields)+++class RaisingPredictor:+ def __call__(self, **kwargs):+ raise ValueError("unexpected extractor failure")+def add(a: float, b: float) -> float:"add two numbers"return a + b@@ -35,7 +54,6 @@ def test_codeact_code_generation():"code_output_0": '"2\\n"',"generated_code_0": "result = add(1,1)\nprint(result)",}- assert program.interpreter.deno_process is Noneclass ExtremumFinder(Signature):@@ -67,7 +85,6 @@ def test_codeact_support_multiple_fields():"code_output_0": '"{\'maximum\': 6.0, \'minimum\': 2.0}\\n"',"generated_code_0": "result = extract_maximum_minimum('2, 3, 5, 6')\nprint(result)",}- assert program.interpreter.deno_process is Nonedef test_codeact_code_parse_failure():@@ -96,7 +113,6 @@ def test_codeact_code_parse_failure():"generated_code_1": "result = add(1,1)\nprint(result)","code_output_1": '"2\\n"',}- assert program.interpreter.deno_process is Nonedef test_codeact_code_execution_failure():@@ -125,7 +141,131 @@ def test_codeact_code_execution_failure():"generated_code_1": "result = add(1,1)\nprint(result)","code_output_1": '"2\\n"',}- assert program.interpreter.deno_process is None+++def test_codeact_evaluate_creates_one_interpreter_per_example():+ tool_registration_barrier = threading.Barrier(4)++ def execute(code, variables):+ if code.startswith("def add"):+ tool_registration_barrier.wait(timeout=30)+ return ""+ return "2\n"++ factory = MockInterpreterFactory(execute_fn=execute)+ program = CodeAct(BasicQA, tools=[add], interpreter_factory=factory)+ program.codeact = StaticPredictor(generated_code="print(add(1, 1))", finished=True)+ program.extractor = StaticPredictor(answer="2")+ devset = [+ dspy.Example(question=f"What is 1+1? ({index})", answer="2").with_inputs("question")+ for index in range(4)+ ]++ result = dspy.Evaluate(+ devset=devset,+ metric=answer_exact_match,+ num_threads=4,+ display_progress=False,+ )(program)++ assert result.score == 100.0+ assert len(factory.instances) == 4+ assert len({id(interpreter) for interpreter in factory.instances}) == 4+ for interpreter in factory.instances:+ assert interpreter.call_count == 2+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ interpreter.execute("print('closed')")+++def test_codeact_factory_creates_fresh_interpreter_per_sequential_call():+ factory = MockInterpreterFactory(responses=["", "2\n"])+ program = CodeAct(BasicQA, tools=[add], interpreter_factory=factory)+ program.codeact = StaticPredictor(generated_code="print(add(1, 1))", finished=True)+ program.extractor = StaticPredictor(answer="2")++ first = program(question="What is 1+1?")+ second = program(question="What is 1+1 again?")++ assert first.answer == second.answer == "2"+ assert len(factory.instances) == 2+ assert factory.instances[0] is not factory.instances[1]+ for interpreter in factory.instances:+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ interpreter.execute("print('closed')")+++def test_codeact_allows_interpreter_as_signature_input():+ factory = MockInterpreterFactory(responses=["", "CPython\n"])+ program = CodeAct("interpreter -> answer", tools=[add], interpreter_factory=factory)+ program.codeact = Mock(+ return_value=dspy.Prediction(generated_code="print(interpreter)", finished=True)+ )+ program.extractor = StaticPredictor(answer="CPython")++ result = program(interpreter="CPython")++ assert result.answer == "CPython"+ assert program.codeact.call_count == 1+ assert program.codeact.call_args.kwargs["interpreter"] == "CPython"+++def test_codeact_rejects_keyword_interpreter_override():+ factory = MockInterpreterFactory()+ program = CodeAct(BasicQA, tools=[add], interpreter_factory=factory)++ with pytest.raises(TypeError, match="first positional argument"):+ program(question="What is 1+1?", interpreter=MockInterpreter())++ assert factory.instances == []+++def test_codeact_rejects_removed_constructor_interpreter_keyword():+ with pytest.raises(TypeError, match="unexpected keyword argument 'interpreter'"):+ CodeAct(BasicQA, tools=[add], interpreter=MockInterpreter())+++def test_codeact_does_not_shutdown_caller_owned_interpreter():+ factory = MockInterpreterFactory()+ program = CodeAct(BasicQA, tools=[add], interpreter_factory=factory)+ program.codeact = StaticPredictor(generated_code="print(add(1, 1))", finished=True)+ program.extractor = StaticPredictor(answer="2")+ interpreter = MockInterpreter(responses=["", "2\n"])++ try:+ result = program(interpreter, question="What is 1+1?")++ assert result.answer == "2"+ assert factory.instances == []+ assert interpreter.execute("print('still open')") == ""+ finally:+ interpreter.shutdown()+++def test_codeact_shuts_down_factory_interpreter_when_extractor_raises():+ factory = MockInterpreterFactory(responses=["", "2\n"])+ program = CodeAct(BasicQA, tools=[add], interpreter_factory=factory)+ program.codeact = StaticPredictor(generated_code="print(add(1, 1))", finished=True)+ program.extractor = RaisingPredictor()++ with pytest.raises(ValueError, match="unexpected extractor failure"):+ program(question="What is 1+1?")++ assert len(factory.instances) == 1+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ factory.instances[0].execute("print('closed')")+++def test_codeact_propagates_terminal_interpreter_failure_and_shuts_down():+ factory = MockInterpreterFactory(responses=["", CodeInterpreterError("protocol corrupt")])+ program = CodeAct(BasicQA, tools=[add], interpreter_factory=factory)+ program.codeact = StaticPredictor(generated_code="print(add(1, 1))", finished=True)++ with pytest.raises(CodeInterpreterError, match="protocol corrupt"):+ program(question="What is 1+1?")++ assert len(factory.instances) == 1+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ factory.instances[0].execute("print('closed')")class CustomTool:@@ -133,5 +273,5 @@ class CustomTool:return a + bdef test_codeact_tool_validation():- with pytest.raises(ValueError, match="CodeAct only accepts functions and not callable objects."):+ with pytest.raises(ValueError, match=r"CodeAct only accepts functions and not callable objects."):CodeAct(BasicQA, tools=[CustomTool()])diff --git a/tests/predict/test_program_of_thought.py b/tests/predict/test_program_of_thought.pyindex 525c1d492..d02ee5dbd 100644--- a/tests/predict/test_program_of_thought.py+++ b/tests/predict/test_program_of_thought.py@@ -1,10 +1,15 @@-from unittest.mock import patch+import threading+from unittest.mock import Mock, patchimport pytestimport dspyfrom dspy import ProgramOfThought, Signature+from dspy.evaluate.metrics import answer_exact_match+from dspy.primitives.code_interpreter import CodeInterpreterError, FinalOutput+from dspy.primitives.python_interpreter import PythonInterpreterfrom dspy.utils import DummyLM+from tests.mock_interpreter import MockInterpreter, MockInterpreterFactoryclass BasicQA(Signature):@@ -12,6 +17,28 @@ class BasicQA(Signature):answer = dspy.OutputField(desc="often between 1 and 5 words")+class StaticPredictor:+ def __init__(self, **fields):+ self.fields = fields++ def __call__(self, **kwargs):+ return dspy.Prediction(**self.fields)+++class RecordingPythonInterpreterFactory:+ def __init__(self, parties: int):+ self.instances = []+ self._lock = threading.Lock()+ self._barrier = threading.Barrier(parties)++ def __call__(self):+ interpreter = PythonInterpreter()+ with self._lock:+ self.instances.append(interpreter)+ self._barrier.wait(timeout=30)+ return interpreter++@pytest.mark.denodef test_pot_code_generation():lm = DummyLM(@@ -27,7 +54,6 @@ def test_pot_code_generation():pot = ProgramOfThought(BasicQA)res = pot(question="What is 1+1?")assert res.answer == "2"- assert pot.interpreter.deno_process is None# This test ensures the old finetuned saved models still work@@ -43,7 +69,6 @@ def test_old_style_pot():pot = ProgramOfThought(BasicQA)res = pot(question="What is 1+1?")assert res.answer == "2"- assert pot.interpreter.deno_process is Noneclass ExtremumFinder(Signature):@@ -68,7 +93,6 @@ def test_pot_support_multiple_fields():res = pot(input_list="2, 3, 5, 6")assert res.maximum == "6"assert res.minimum == "2"- assert pot.interpreter.deno_process is None@pytest.mark.deno@@ -90,7 +114,117 @@ def test_pot_code_generation_with_one_error():pot = ProgramOfThought(BasicQA)res = pot(question="What is 1+1?")assert res.answer == "2"- assert pot.interpreter.deno_process is None+++def test_pot_evaluate_creates_one_interpreter_per_example():+ factory = RecordingPythonInterpreterFactory(parties=4)+ pot = ProgramOfThought(BasicQA, interpreter_factory=factory)+ pot.code_generate = StaticPredictor(generated_code="SUBMIT({'answer': 2})")+ pot.generate_output = StaticPredictor(answer="2")+ devset = [+ dspy.Example(question=f"What is 1+1? ({index})", answer="2").with_inputs("question")+ for index in range(4)+ ]++ result = dspy.Evaluate(+ devset=devset,+ metric=answer_exact_match,+ num_threads=4,+ display_progress=False,+ )(pot)++ assert result.score == 100.0+ assert len(factory.instances) == 4+ assert len({id(interpreter) for interpreter in factory.instances}) == 4+ assert all(interpreter.deno_process is None for interpreter in factory.instances)+++def test_pot_factory_creates_fresh_interpreter_per_sequential_call():+ factory = MockInterpreterFactory(responses=[FinalOutput({"answer": "2"})])+ pot = ProgramOfThought(BasicQA, interpreter_factory=factory)+ pot.code_generate = StaticPredictor(generated_code="SUBMIT({'answer': 2})")+ pot.generate_output = StaticPredictor(answer="2")++ first = pot(question="What is 1+1?")+ second = pot(question="What is 1+1 again?")++ assert first.answer == second.answer == "2"+ assert len(factory.instances) == 2+ assert factory.instances[0] is not factory.instances[1]+ for interpreter in factory.instances:+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ interpreter.execute("print('closed')")+++def test_pot_allows_interpreter_as_signature_input():+ factory = MockInterpreterFactory(responses=[FinalOutput({"answer": "CPython"})])+ pot = ProgramOfThought("interpreter -> answer", interpreter_factory=factory)+ pot.code_generate = Mock(return_value=dspy.Prediction(generated_code="SUBMIT({'answer': interpreter})"))+ pot.generate_output = StaticPredictor(answer="CPython")++ result = pot(interpreter="CPython")++ assert result.answer == "CPython"+ pot.code_generate.assert_called_once_with(interpreter="CPython")+++def test_pot_rejects_keyword_interpreter_override():+ factory = MockInterpreterFactory()+ pot = ProgramOfThought(BasicQA, interpreter_factory=factory)++ with pytest.raises(TypeError, match="first positional argument"):+ pot(question="What is 1+1?", interpreter=MockInterpreter())++ assert factory.instances == []+++def test_pot_rejects_removed_constructor_interpreter_keyword():+ with pytest.raises(TypeError, match="unexpected keyword argument 'interpreter'"):+ ProgramOfThought(BasicQA, interpreter=MockInterpreter())+++def test_pot_does_not_shutdown_caller_owned_interpreter():+ factory = MockInterpreterFactory()+ pot = ProgramOfThought(BasicQA, interpreter_factory=factory)+ pot.code_generate = StaticPredictor(generated_code="SUBMIT({'answer': 2})")+ pot.generate_output = StaticPredictor(answer="2")+ interpreter = MockInterpreter(responses=[FinalOutput({"answer": "2"})])++ try:+ result = pot(interpreter, question="What is 1+1?")++ assert result.answer == "2"+ assert factory.instances == []+ assert interpreter.execute("print('still open')") == ""+ finally:+ interpreter.shutdown()+++def test_pot_shuts_down_factory_interpreter_when_execution_raises():+ factory = MockInterpreterFactory(responses=[ValueError("unexpected interpreter failure")])+ pot = ProgramOfThought(BasicQA, interpreter_factory=factory)+ pot.code_generate = StaticPredictor(generated_code="raise ValueError")++ with pytest.raises(ValueError, match="unexpected interpreter failure"):+ pot(question="What is 1+1?")++ assert len(factory.instances) == 1+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ factory.instances[0].execute("print('closed')")+++def test_pot_propagates_terminal_interpreter_failure_and_shuts_down():+ factory = MockInterpreterFactory(responses=[CodeInterpreterError("protocol corrupt")])+ pot = ProgramOfThought(BasicQA, interpreter_factory=factory)+ pot.code_generate = StaticPredictor(generated_code="print('test')")++ with pytest.raises(CodeInterpreterError, match="protocol corrupt"):+ pot(question="What is 1+1?")++ assert len(factory.instances) == 1+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ factory.instances[0].execute("print('closed')")@pytest.mark.deno@@ -108,7 +242,7 @@ def test_pot_code_generation_persistent_errors():dspy.configure(lm=lm)pot = ProgramOfThought(BasicQA, max_iters=max_iters)- with pytest.raises(RuntimeError, match="Max hops reached. Failed to run ProgramOfThought: ZeroDivisionError:"):+ with pytest.raises(RuntimeError, match=r"Max hops reached. Failed to run ProgramOfThought: ZeroDivisionError:"):pot(question="What is 1+1?")@@ -125,7 +259,7 @@ def test_pot_code_parse_error():with (patch("dspy.predict.program_of_thought.ProgramOfThought._execute_code") as mock_execute_code,pytest.raises(- RuntimeError, match="Max hops reached. Failed to run ProgramOfThought: Error: Code format is not correct."+ RuntimeError, match=r"Max hops reached. Failed to run ProgramOfThought: Error: Code format is not correct."),):pot(question="What is 1+1?")diff --git a/tests/predict/test_rlm.py b/tests/predict/test_rlm.pyindex 6cc5247ea..bf3e68084 100644--- a/tests/predict/test_rlm.py+++ b/tests/predict/test_rlm.py@@ -18,7 +18,7 @@ from dspy.primitives.prediction import Predictionfrom dspy.primitives.python_interpreter import PythonInterpreterfrom dspy.primitives.repl_types import REPLEntry, REPLHistory, REPLVariablefrom dspy.primitives.sandbox_serializable import SandboxSerializable-from tests.mock_interpreter import MockInterpreter+from tests.mock_interpreter import MockInterpreter, MockInterpreterFactory# ============================================================================# Test Helpers and Factories@@ -192,32 +192,54 @@ class TestRLMInitialization:rlm = RLM("context -> answer")assert rlm.max_llm_calls == 50assert rlm.sub_lm is None- assert rlm._interpreter is None+ assert rlm._interpreter_factory is PythonInterpreter# Test custom values- mock = MockInterpreter()mock_lm = dspy.LM("openai/gpt-4o-mini")- rlm = RLM("context -> answer", max_llm_calls=100, sub_lm=mock_lm, interpreter=mock)+ rlm = RLM(+ "context -> answer",+ max_llm_calls=100,+ sub_lm=mock_lm,+ interpreter_factory=MockInterpreter,+ )assert rlm.max_llm_calls == 100assert rlm.sub_lm is mock_lm- assert rlm._interpreter is mock+ assert rlm._interpreter_factory is MockInterpreterdef test_forward_validates_required_inputs(self):"""Test that forward() raises ValueError for missing required inputs."""- mock = MockInterpreter(responses=["result"])-# Single missing input- rlm = RLM("context, query -> answer", max_iters=3, interpreter=mock)+ rlm = RLM("context, query -> answer", max_iters=3)with pytest.raises(ValueError, match="Missing required input"):rlm.forward(context="some context") # Missing 'query'# Multiple missing inputs - all should be reported- rlm = RLM("a, b, c -> answer", max_iters=3, interpreter=mock)+ rlm = RLM("a, b, c -> answer", max_iters=3)with pytest.raises(ValueError) as exc_info:rlm.forward(a="only a") # Missing 'b' and 'c'assert "b" in str(exc_info.value)assert "c" in str(exc_info.value)+ def test_interpreter_instance_is_rejected_as_factory(self):+ with pytest.raises(TypeError, match="first positional argument when calling the module"):+ RLM("context -> answer", interpreter_factory=MockInterpreter())++ def test_constructor_interpreter_keyword_is_removed(self):+ with pytest.raises(TypeError, match="unexpected keyword argument 'interpreter'"):+ RLM("context -> answer", interpreter=MockInterpreter())++ def test_factory_return_value_is_validated(self):+ rlm = RLM("query -> answer", interpreter_factory=lambda: None)++ with pytest.raises(TypeError, match="interpreter_factory must return a CodeInterpreter, not NoneType"):+ rlm(query="test")++ def test_keyword_interpreter_override_has_clear_error(self):+ rlm = RLM("query -> answer")++ with pytest.raises(TypeError, match="first positional argument"):+ rlm(query="test", interpreter=MockInterpreter())+def test_batched_query_errors_have_clear_markers(self):"""Test that errors in llm_query_batched are prefixed with [ERROR]."""from unittest.mock import MagicMock@@ -295,6 +317,80 @@ class TestRLMInitialization:tools["llm_query"](prompt="one more")+class TestRLMInterpreterLifecycle:+ def test_interpreter_remains_available_as_signature_input(self):+ factory = MockInterpreterFactory(responses=[FinalOutput({"answer": "CPython"})])+ rlm = RLM("interpreter -> answer", max_iters=1, interpreter_factory=factory)+ rlm.generate_action = make_mock_predictor([+ {"reasoning": "Return input", "code": "SUBMIT(interpreter)"},+ ])++ result = rlm(interpreter="CPython")++ assert result.answer == "CPython"+ assert factory.instances[0].call_history[0][1] == {"interpreter": "CPython"}++ @pytest.mark.asyncio+ async def test_factory_creates_and_shuts_down_one_interpreter_per_call(self):+ factory = MockInterpreterFactory(responses=[FinalOutput({"answer": "42"})])+ rlm = RLM("query -> answer", max_iters=1, interpreter_factory=factory)+ rlm.generate_action = make_mock_predictor([+ {"reasoning": "Return answer", "code": 'SUBMIT("42")'},+ ])++ sync_result = rlm(query="sync")+ async_result = await rlm.acall(query="async")++ assert sync_result.answer == "42"+ assert async_result.answer == "42"+ assert len(factory.instances) == 2+ assert factory.instances[0] is not factory.instances[1]+ for interpreter in factory.instances:+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ interpreter.execute("print('closed')")++ @pytest.mark.asyncio+ async def test_caller_owned_interpreter_can_be_reused_across_sequential_calls(self):+ factory = MockInterpreterFactory()+ interpreter = MockInterpreter(+ responses=[+ FinalOutput({"answer": "first"}),+ FinalOutput({"answer": "second"}),+ ]+ )+ rlm = RLM("query -> answer", max_iters=1, interpreter_factory=factory)+ rlm.generate_action = make_mock_predictor([+ {"reasoning": "Return answer", "code": "SUBMIT(answer)"},+ ])++ try:+ sync_result = rlm(interpreter, query="sync")+ async_result = await rlm.acall(interpreter, query="async")++ assert sync_result.answer == "first"+ assert async_result.answer == "second"+ assert factory.instances == []+ assert interpreter.execute("print('still open')") == ""+ finally:+ interpreter.shutdown()++ def test_factory_interpreter_is_shutdown_when_prediction_raises(self):+ class RaisingPredictor:+ def __call__(self, **kwargs):+ raise ValueError("unexpected predictor failure")++ factory = MockInterpreterFactory()+ rlm = RLM("query -> answer", max_iters=1, interpreter_factory=factory)+ rlm.generate_action = RaisingPredictor()++ with pytest.raises(ValueError, match="unexpected predictor failure"):+ rlm(query="test")++ assert len(factory.instances) == 1+ with pytest.raises(CodeInterpreterError, match="shutdown"):+ factory.instances[0].execute("print('closed')")++class TestRLMCodeFenceParsing:"""Tests for robust fenced-code extraction."""@@ -537,12 +633,12 @@ class TestRLMCallMethod:def test_call_is_alias_for_forward(self):"""Test that __call__ is an alias for forward()."""mock = MockInterpreter(responses=[FinalOutput({"answer": "42"})])- rlm = RLM("query -> answer", max_iters=3, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=3)rlm.generate_action = make_mock_predictor([{"reasoning": "Return answer", "code": 'SUBMIT("42")'},])- result = rlm(query="What is the answer?")+ result = rlm(mock, query="What is the answer?")assert result.answer == "42"@@ -556,7 +652,7 @@ class TestRLMMaxIterationsFallback:"still exploring...","more exploring...",])- rlm = RLM("query -> answer", max_iters=3, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=3)rlm.generate_action = make_mock_predictor([{"reasoning": "Explore 1", "code": "print('exploring')"},{"reasoning": "Explore 2", "code": "print('exploring')"},@@ -567,7 +663,7 @@ class TestRLMMaxIterationsFallback:{"answer": "extracted_answer"},])- result = rlm.forward(query="test")+ result = rlm.forward(mock, query="test")assert result.answer == "extracted_answer"assert result.final_reasoning == "Extract forced final output"@@ -584,13 +680,13 @@ class TestRLMToolExceptions:CodeExecutionError("RuntimeError: Tool failed!"),FinalOutput({"answer": "recovered"}),])- rlm = RLM("query -> answer", max_iters=5, interpreter=mock, tools=[failing_tool])+ rlm = RLM("query -> answer", max_iters=5, tools=[failing_tool])rlm.generate_action = make_mock_predictor([{"reasoning": "Call tool", "code": "failing_tool()"},{"reasoning": "Recover", "code": 'SUBMIT("recovered")'},])- result = rlm.forward(query="test")+ result = rlm.forward(mock, query="test")assert result.answer == "recovered"def test_runtime_error_history_uses_stripped_code(self):@@ -599,13 +695,13 @@ class TestRLMToolExceptions:CodeExecutionError("NameError: name 'x' is not defined"),FinalOutput({"answer": "recovered"}),])- rlm = RLM("query -> answer", max_iters=5, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=5)rlm.generate_action = make_mock_predictor([{"reasoning": "Will fail", "code": "```python\nprint(x)\n```"},{"reasoning": "Recover", "code": 'SUBMIT("recovered")'},])- result = rlm.forward(query="test")+ result = rlm.forward(mock, query="test")assert result.answer == "recovered"first_step = result.trajectory[0]assert first_step["code"] == "print(x)"@@ -616,13 +712,13 @@ class TestRLMToolExceptions:SyntaxError("invalid syntax"),FinalOutput({"answer": "recovered"}),])- rlm = RLM("query -> answer", max_iters=5, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=5)rlm.generate_action = make_mock_predictor([{"reasoning": "Bad code", "code": "```python\ndef incomplete(\n```"},{"reasoning": "Recover", "code": 'SUBMIT("recovered")'},])- result = rlm.forward(query="test")+ result = rlm.forward(mock, query="test")assert result.answer == "recovered"assert result.trajectory[0]["output"].startswith("[Error] invalid syntax")@@ -631,13 +727,13 @@ class TestRLMToolExceptions:mock = MockInterpreter(responses=[FinalOutput({"answer": "recovered"}),])- rlm = RLM("query -> answer", max_iters=5, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=5)rlm.generate_action = make_mock_predictor([{"reasoning": "Wrong language", "code": "```bash\nls -la\n```"},{"reasoning": "Recover", "code": 'SUBMIT("recovered")'},])- result = rlm.forward(query="test")+ result = rlm.forward(mock, query="test")assert result.answer == "recovered"assert result.trajectory[0]["output"].startswith("[Error]")@@ -649,14 +745,14 @@ class TestRLMToolExceptions:raise CodeInterpreterError("protocol corrupt")mock = MockInterpreter(execute_fn=fail_generated_code)- rlm = RLM("query -> answer", max_iters=1, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=1)rlm.generate_action = make_mock_predictor([{"reasoning": "Try code", "code": "print('test')"},])rlm.extract = make_mock_predictor([{"answer": "hallucinated"}])with pytest.raises(CodeInterpreterError, match="protocol corrupt"):- rlm.forward(query="test")+ rlm.forward(mock, query="test")@pytest.mark.asyncioasync def test_interpreter_failure_propagates_async(self):@@ -667,14 +763,14 @@ class TestRLMToolExceptions:raise CodeInterpreterError("protocol corrupt")mock = MockInterpreter(execute_fn=fail_generated_code)- rlm = RLM("query -> answer", max_iters=1, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=1)rlm.generate_action = make_mock_predictor([{"reasoning": "Try code", "code": "print('test')"},])rlm.extract = make_mock_predictor([{"answer": "hallucinated"}])with pytest.raises(CodeInterpreterError, match="protocol corrupt"):- await rlm.aforward(query="test")+ await rlm.aforward(mock, query="test")class TestRLMDynamicSignature:@@ -916,24 +1012,24 @@ class TestRLMAsyncMock:async def test_aforward_basic(self):"""Test aforward() returns Prediction with expected output (MockInterpreter)."""mock = MockInterpreter(responses=[FinalOutput({"answer": "42"})])- rlm = RLM("query -> answer", max_iters=3, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=3)rlm.generate_action = make_mock_predictor([{"reasoning": "Return answer", "code": 'SUBMIT("42")'},])- result = await rlm.aforward(query="What is the answer?")+ result = await rlm.aforward(mock, query="What is the answer?")assert result.answer == "42"@pytest.mark.asyncioasync def test_aforward_int_output_mock(self):"""Test aforward() returns int when signature expects int (MockInterpreter)."""mock = MockInterpreter(responses=[FinalOutput({"count": 42})])- rlm = RLM("query -> count: int", max_iters=3, interpreter=mock)+ rlm = RLM("query -> count: int", max_iters=3)rlm.generate_action = make_mock_predictor([{"reasoning": "Return count", "code": "SUBMIT(42)"},])- result = await rlm.aforward(query="count items")+ result = await rlm.aforward(mock, query="count items")assert result.count == 42assert isinstance(result.count, int)@@ -944,13 +1040,13 @@ class TestRLMAsyncMock:"explored data",FinalOutput({"answer": "done"}),])- rlm = RLM("query -> answer", max_iters=5, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=5)rlm.generate_action = make_mock_predictor([{"reasoning": "Explore first", "code": "print('exploring')"},{"reasoning": "Now finish", "code": 'SUBMIT("done")'},])- result = await rlm.aforward(query="test")+ result = await rlm.aforward(mock, query="test")assert result.answer == "done"@@ -967,12 +1063,12 @@ class TestRLMTypeCoercionMock:def test_type_coercion(self, output_field, output_type, final_value, code, expected):"""Test RLM type coercion for various types (MockInterpreter)."""mock = MockInterpreter(responses=[FinalOutput({output_field: final_value})])- rlm = RLM(f"query -> {output_field}: {output_type}", max_iters=3, interpreter=mock)+ rlm = RLM(f"query -> {output_field}: {output_type}", max_iters=3)rlm.generate_action = make_mock_predictor([{"reasoning": "Return value", "code": code},])- result = rlm.forward(query="test")+ result = rlm.forward(mock, query="test")assert getattr(result, output_field) == expecteddef test_type_error_retries(self):@@ -981,13 +1077,13 @@ class TestRLMTypeCoercionMock:FinalOutput({"answer": "maybe"}), # Invalid for LiteralFinalOutput({"answer": "yes"}), # Valid])- rlm = RLM("query -> answer: Literal['yes', 'no']", max_iters=5, interpreter=mock)+ rlm = RLM("query -> answer: Literal['yes', 'no']", max_iters=5)rlm.generate_action = make_mock_predictor([{"reasoning": "Try maybe", "code": 'SUBMIT("maybe")'},{"reasoning": "Try yes", "code": 'SUBMIT("yes")'},])- result = rlm.forward(query="is it yes?")+ result = rlm.forward(mock, query="is it yes?")assert result.answer == "yes"@@ -1325,7 +1421,7 @@ class TestPrepareSerializableVars:def test_separates_serializable_from_regular(self):"""Serializable values are injected; regular values are returned."""mock = MockInterpreter(responses=["", FinalOutput({"answer": "42"})])- rlm = RLM("data, query -> answer", max_iters=3, interpreter=mock)+ rlm = RLM("data, query -> answer", max_iters=3)stub = _StubSerializable("payload")@@ -1347,7 +1443,7 @@ class TestPrepareSerializableVars:def test_no_serializable_returns_all(self):"""When no SandboxSerializable values exist, all args are returned."""mock = MockInterpreter(responses=[FinalOutput({"answer": "42"})])- rlm = RLM("query -> answer", max_iters=3, interpreter=mock)+ rlm = RLM("query -> answer", max_iters=3)rlm._inject_execution_context(mock, rlm._prepare_execution_tools())regular = rlm._prepare_serializable_vars({"query": "hello"}, mock)@@ -1358,7 +1454,7 @@ class TestPrepareSerializableVars:def test_binary_payload_uses_base64_transport(self):"""Non-UTF8 bytes should be transported via base64 and decoded in sandbox code."""mock = MockInterpreter(responses=[""])- rlm = RLM("data, query -> answer", interpreter=mock)+ rlm = RLM("data, query -> answer")payload = _BinarySerializable()rlm._inject_execution_context(mock, rlm._prepare_execution_tools())@@ -1377,7 +1473,7 @@ class TestPrepareSerializableVars:contract is: code stays small, payload travels as a named variable."""mock = MockInterpreter(responses=[""])- rlm = RLM("data, query -> answer", interpreter=mock)+ rlm = RLM("data, query -> answer")large_text = "x" * (2 * 1024 * 1024) # 2 MB UTF-8 payload@@ -1410,13 +1506,13 @@ class TestPrepareSerializableVars:"", # setup execution for _prepare_serializable_varsFinalOutput({"answer": "done"}),])- rlm = RLM("data, query -> answer", max_iters=3, interpreter=mock)+ rlm = RLM("data, query -> answer", max_iters=3)rlm.generate_action = make_mock_predictor([{"reasoning": "Done", "code": 'SUBMIT("done")'},])stub = _StubSerializable("test_payload")- result = rlm.forward(data=stub, query="test")+ result = rlm.forward(mock, data=stub, query="test")assert result.answer == "done"# First call should be the serializable setup, second should be the iteration@@ -1445,7 +1541,7 @@ class TestLargeSerializableRoundTrip:return f"LargeText({len(large_text)} chars)"with PythonInterpreter(tools={}) as interp:- rlm = RLM("data -> answer", interpreter=interp)+ rlm = RLM("data -> answer")rlm._inject_execution_context(interp, rlm._prepare_execution_tools())rlm._prepare_serializable_vars({"data": _LargeText()}, interp)result = interp.execute("print(len(data)); print(data[:6])")