Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

fix(rlm)!: accept interpreter factories at call time (#10493)

Isaac Miller430af8d8722f
diff --git a/docs/docs/api/modules/RLM.md b/docs/docs/api/modules/RLM.md
index 5e71fbaf5..30cc3cdc7 100644
--- a/docs/docs/api/modules/RLM.md
+++ b/docs/docs/api/modules/RLM.md
@@ -205,7 +205,16 @@ One call puts the same interpreter behind every code-executing module in the pro
dspy.configure(interpreter_factory=MyInterpreter)
```
-RLM creates and shuts down one interpreter from this factory per invocation. It adds invocation-scoped tools to the returned interpreter's mutable `tools` dictionary, so remote sandboxes need a `CodeInterpreter` adapter that supports that protocol. To reuse a caller-owned interpreter, pass it as the first positional argument when calling the module: `rlm(interpreter, context=data, query=query)`. RLM updates its tools and output metadata but does not shut down or restore it. Reuse is supported only for sequential calls to the same RLM instance; use the factory path for concurrency.
+RLM creates and shuts down one interpreter from this factory per invocation. It adds invocation-scoped tools to the returned interpreter's mutable `tools` dictionary, so remote sandboxes need a `CodeInterpreter` adapter that supports that protocol.
+
+To override the factory for a single invocation, pass a zero-argument callable via the `interpreter_factory` keyword:
+
+```python
+result = rlm(context=data, query=query, interpreter_factory=MyInterpreter)
+result = await rlm.acall(context=data, query=query, interpreter_factory=MyInterpreter)
+```
+
+The call-time factory takes precedence over constructor and global/context factories, including when explicitly passing `PythonInterpreter`. RLM calls it once and shuts down the returned interpreter on success or failure. The factory must return a fresh interpreter each time. Passing a live interpreter at call time is no longer supported; migrate `rlm(interpreter, ...)` to `rlm(..., interpreter_factory=factory)`. The option is keyword-only, and `interpreter_factory` is reserved for runtime configuration rather than signature inputs.
If the factory exposes an `execution_instructions` string, RLM adds it to the action predictor's task instructions,
which DSPy adapters place in the system prompt. Optimizers such as GEPA may therefore adapt the execution guidance
@@ -274,7 +283,7 @@ RLM returns a `Prediction` with:
RLM is marked as experimental. The API may change in future releases.
!!! note "Thread Safety"
- `interpreter_factory` may be called concurrently and must return a fresh interpreter each time. An interpreter passed as the first positional argument to `rlm(...)` or `rlm.acall(...)` is caller-owned and may be reused only for sequential calls to the same RLM instance. `PythonInterpreter` must also stay on the thread where it was first used.
+ Constructor and call-time factories may be called concurrently and must return a fresh interpreter each time. RLM retains that interpreter throughout one invocation and shuts it down afterward. `PythonInterpreter` must stay on the thread where it was first used.
!!! note "Interpreter Requirements"
RLM defaults to `PythonInterpreter`, which requires [Deno](https://deno.land/) to be installed for the Pyodide WASM sandbox.
diff --git a/docs/docs/diving-deeper/rlm.md b/docs/docs/diving-deeper/rlm.md
index 97526215b..993dc7206 100644
--- a/docs/docs/diving-deeper/rlm.md
+++ b/docs/docs/diving-deeper/rlm.md
@@ -57,8 +57,8 @@ The class carries the `@experimental` decorator. The hard parts are still settli
**`dspy.RLM(signature, max_iters=20, max_llm_calls=50, max_output_chars=10_000, verbose=False, tools=None, sub_lm=None, interpreter_factory=PythonInterpreter)`**
The constructor parses the signature and builds the two internal predictors, formatting your task instructions, input names, and output-field types into the action prompt and creating a separate extract signature for the fallback. The budgets and the sandbox configuration are all fixed here, so one instance carries one configuration.
-**`__call__([interpreter], **inputs)` / `acall([interpreter], **inputs)`**
-The public call validates the inputs against the signature, builds the variable list, opens the interpreter, and runs the loop. Each turn asks the action predictor for code, runs it, and appends the result to history until the model submits or the loop reaches `max_iters`. `acall()` is the async twin and uses `acall` on the predictors. Both return a `Prediction`. An interpreter passed as the first positional argument is caller-owned: RLM updates its execution context but does not shut it down.
+**`__call__(**inputs, interpreter_factory=...)` / `acall(**inputs, interpreter_factory=...)`**
+The public call validates the inputs against the signature, builds the variable list, opens the interpreter, and runs the loop. Each turn asks the action predictor for code, runs it, and appends the result to history until the model submits or the loop reaches `max_iters`. `acall()` is the async twin and uses `acall` on the predictors. Both return a `Prediction`. An optional zero-argument factory passed via `interpreter_factory=` overrides the constructor and configured factories for this invocation. RLM creates one interpreter and shuts it down on success or failure.
### Programming the loop with built-in tools
@@ -92,8 +92,8 @@ A list of plain functions or `dspy.Tool` objects. RLM normalizes each to a `Tool
**`interpreter_factory=...`**
A zero-argument callable that returns a fresh `CodeInterpreter` for one invocation. RLM may call the factory concurrently, and it always shuts down the returned interpreter. A class such as `PythonInterpreter` or `LocalInterpreter` is already a factory; use `functools.partial` or a callable provider object when construction needs configuration. RLM adds invocation-scoped tools to the returned interpreter's mutable `tools` dictionary, so remote sandboxes need a `CodeInterpreter` adapter that supports that protocol. `dspy.configure(interpreter_factory=...)` replaces the default on each `forward()`, so `dspy.context(interpreter_factory=...)` scopes the choice. If the active factory exposes an `execution_instructions` string, RLM refreshes it for each action call so the prompt matches the runtime executing the generated code.
-**`__call__(interpreter, **inputs)` / `acall(interpreter, **inputs)`**
-An escape hatch for a caller-owned interpreter, supplied as the first positional argument. RLM mutates its `tools` dictionary and, when supported, its output-field metadata, but does not shut down or restore the instance. Reuse is supported only for sequential calls to the same RLM instance, so retained variables and tool registrations stay within one program and trust boundary. Use `interpreter_factory` for concurrent invocations. A `PythonInterpreter` override must also stay on the thread where it was first used.
+**`__call__(**inputs, interpreter_factory=...)` / `acall(**inputs, interpreter_factory=...)`**
+A per-invocation factory override, supplied by keyword. For example, `rlm(query=query, interpreter_factory=dspy.PythonInterpreter)` explicitly chooses PythonInterpreter even when another factory is configured. The factory must return a fresh interpreter; RLM injects tools and output metadata and always shuts it down. Its `execution_instructions` metadata controls the runtime guidance for that invocation without changing the shared predictor. Live interpreter instances are no longer accepted at call time. The factory option is keyword-only, and `interpreter_factory` is reserved for runtime configuration rather than signature inputs.
**`dspy.SandboxSerializable`**
The base class for inputs that need custom loading. Implement `sandbox_setup`, `to_sandbox`, `sandbox_assignment`, and `rlm_preview`. It also defines a Pydantic schema hook, so a subclass can be a typed field in a signature, as in `data: DataFrame = dspy.InputField()`.
diff --git a/dspy/predict/rlm.py b/dspy/predict/rlm.py
index 8b4db265e..7b0e20e06 100644
--- a/dspy/predict/rlm.py
+++ b/dspy/predict/rlm.py
@@ -33,7 +33,6 @@ from dspy.primitives.code_interpreter import (
CodeExecutionError,
CodeInterpreter,
FinalOutput,
- _create_interpreter,
_validate_interpreter,
_validate_interpreter_factory,
resolve_interpreter_factory,
@@ -127,9 +126,9 @@ class RLM(Module):
``dspy.configure(interpreter_factory=...)`` replaces that default. Either route
accepts an adapter for a remote sandbox.
RLM updates the interpreter's mutable ``tools`` dictionary with
- invocation-scoped tools before execution. A caller-owned interpreter may be
- reused sequentially with the same RLM instance, but must not be shared by
- overlapping invocations.
+ invocation-scoped tools before execution. Pass a zero-argument factory via
+ ``interpreter_factory=`` at call time to override the runtime for one invocation.
+ RLM shuts down every interpreter it creates.
Examples:
```python
@@ -237,7 +236,7 @@ class RLM(Module):
return normalized
def _validate_namespace(self, tools: dict[str, Tool]) -> None:
- """Validate names owned by the RLM result and sandbox APIs."""
+ """Validate names owned by the RLM call, result, and sandbox APIs."""
for name in tools:
if not name.isidentifier() or keyword.iskeyword(name):
raise ValueError(f"Invalid tool name '{name}': must be a valid Python identifier and not a keyword")
@@ -245,6 +244,8 @@ class RLM(Module):
raise ValueError(f"Tool name '{name}' conflicts with built-in sandbox function")
input_names = set(self.signature.input_fields)
+ if "interpreter_factory" in input_names:
+ raise ValueError("'interpreter_factory' is reserved for RLM runtime configuration, not a signature input.")
reserved_inputs = sorted(input_names & self._RESERVED_SANDBOX_NAMES)
if reserved_inputs:
raise ValueError(f"Input fields conflict with built-in sandbox functions: {reserved_inputs}")
@@ -422,9 +423,8 @@ class RLM(Module):
raise TypeError("interpreter_factory.execution_instructions must be a string")
return execution_instructions
- def _action_signature_for_current_factory(self) -> type[Signature]:
+ def _action_signature_for_current_factory(self, factory: Callable[[], CodeInterpreter]) -> type[Signature]:
"""Return an action signature whose runtime guidance matches the active interpreter."""
- factory = resolve_interpreter_factory(self._interpreter_factory)
execution_instructions = self._get_execution_instructions(factory)
# getattr, because generate_action may have been replaced by a predictor that
# carries no signature of its own.
@@ -483,10 +483,6 @@ class RLM(Module):
def _validate_inputs(self, input_args: dict[str, Any]) -> None:
"""Validate call-time arguments against the signature's input namespace."""
- if "interpreter" in input_args and "interpreter" not in self.signature.input_fields:
- raise TypeError(
- "To use a caller-owned interpreter, pass it as the first positional argument when calling the module."
- )
input_names = set(self.signature.input_fields)
unexpected = set(input_args) - input_names
if unexpected:
@@ -579,16 +575,12 @@ class RLM(Module):
def _interpreter_context(
self,
execution_tools: dict[str, Callable],
- interpreter: CodeInterpreter | None,
+ factory: Callable[[], CodeInterpreter],
) -> Iterator[CodeInterpreter]:
- """Yield a caller-owned interpreter or manage a factory-created one."""
- if interpreter is not None:
- _validate_interpreter(interpreter)
- self._inject_execution_context(interpreter, execution_tools)
- yield interpreter
- return
-
- interpreter = _create_interpreter(self._interpreter_factory)
+ """Create and close one interpreter for this invocation."""
+ _validate_interpreter_factory(factory)
+ interpreter = factory()
+ _validate_interpreter(interpreter)
try:
self._inject_execution_context(interpreter, execution_tools)
yield interpreter
@@ -730,13 +722,14 @@ class RLM(Module):
iteration: int,
input_args: dict[str, Any],
output_field_names: list[str],
+ interpreter_factory: Callable[[], CodeInterpreter],
) -> Prediction | REPLHistory:
"""Execute one iteration. Returns Prediction if done, else updated REPLHistory."""
variables_info = [variable.format() for variable in variables]
# A per-call signature, not a mutation of generate_action.signature, keeps a
# dspy.context override local to this invocation.
action = self.generate_action(
- signature=self._action_signature_for_current_factory(),
+ signature=self._action_signature_for_current_factory(interpreter_factory),
variables_info=variables_info,
repl_history=history,
iteration=f"{iteration + 1}/{self.max_iters}",
@@ -760,13 +753,13 @@ class RLM(Module):
# Public Interface
# =========================================================================
- def forward(self, interpreter: CodeInterpreter | None = None, /, **input_args) -> Prediction:
+ def forward(self, *, interpreter_factory: Callable[[], CodeInterpreter] | None = None, **input_args) -> Prediction:
"""Execute RLM to produce outputs from the given inputs.
Args:
- interpreter: Optional caller-owned interpreter, passed positionally. RLM injects invocation tools and
- output metadata into it but does not shut it down. Reuse is supported only for sequential calls to
- this RLM instance.
+ interpreter_factory: Optional zero-argument factory, passed by keyword. Overrides the constructor
+ and configured factories for this invocation. Must return a fresh interpreter; RLM injects tools
+ and output metadata and shuts it down on exit, including failures.
**input_args: Input values matching the signature's input fields.
Returns:
@@ -777,18 +770,20 @@ class RLM(Module):
CodeInterpreterError: If interpreter setup, process, or protocol fails
"""
self._validate_inputs(input_args)
+ if interpreter_factory is None:
+ interpreter_factory = resolve_interpreter_factory(self._interpreter_factory)
output_field_names = list(self.signature.output_fields.keys())
execution_tools = self._prepare_execution_tools()
variables = self._build_variables(**input_args)
- with self._interpreter_context(execution_tools, interpreter) as repl:
+ with self._interpreter_context(execution_tools, interpreter_factory) as repl:
regular_args = self._prepare_serializable_vars(input_args, repl)
history: REPLHistory = REPLHistory(max_output_chars=self.max_output_chars)
for iteration in range(self.max_iters):
result: Prediction | REPLHistory = self._execute_iteration(
- repl, variables, history, iteration, regular_args, output_field_names
+ repl, variables, history, iteration, regular_args, output_field_names, interpreter_factory
)
if isinstance(result, Prediction):
return result
@@ -826,11 +821,12 @@ class RLM(Module):
iteration: int,
input_args: dict[str, Any],
output_field_names: list[str],
+ interpreter_factory: Callable[[], CodeInterpreter],
) -> Prediction | REPLHistory:
"""Async version: Execute one iteration."""
variables_info = [variable.format() for variable in variables]
pred = await self.generate_action.acall(
- signature=self._action_signature_for_current_factory(),
+ signature=self._action_signature_for_current_factory(interpreter_factory),
variables_info=variables_info,
repl_history=history,
iteration=f"{iteration + 1}/{self.max_iters}",
@@ -850,13 +846,13 @@ class RLM(Module):
result = self._execute_code(repl, code, input_args)
return self._process_execution_result(pred, code, result, history, output_field_names)
- async def aforward(self, interpreter: CodeInterpreter | None = None, /, **input_args) -> Prediction:
+ async def aforward(self, *, interpreter_factory: Callable[[], CodeInterpreter] | None = None, **input_args) -> Prediction:
"""Async version of forward(). Execute RLM to produce outputs.
Args:
- interpreter: Optional caller-owned interpreter, passed positionally. RLM injects invocation tools and
- output metadata into it but does not shut it down. Reuse is supported only for sequential calls to
- this RLM instance.
+ interpreter_factory: Optional zero-argument factory, passed by keyword. Overrides the constructor
+ and configured factories for this invocation. Must return a fresh interpreter; RLM injects tools
+ and output metadata and shuts it down on exit, including failures.
**input_args: Input values matching the signature's input fields.
Returns:
@@ -867,18 +863,20 @@ class RLM(Module):
CodeInterpreterError: If interpreter setup, process, or protocol fails
"""
self._validate_inputs(input_args)
+ if interpreter_factory is None:
+ interpreter_factory = resolve_interpreter_factory(self._interpreter_factory)
output_field_names = list(self.signature.output_fields.keys())
execution_tools = self._prepare_execution_tools()
variables = self._build_variables(**input_args)
- with self._interpreter_context(execution_tools, interpreter) as repl:
+ with self._interpreter_context(execution_tools, interpreter_factory) as repl:
regular_args = self._prepare_serializable_vars(input_args, repl)
history = REPLHistory(max_output_chars=self.max_output_chars)
for iteration in range(self.max_iters):
result = await self._aexecute_iteration(
- repl, variables, history, iteration, regular_args, output_field_names
+ repl, variables, history, iteration, regular_args, output_field_names, interpreter_factory
)
if isinstance(result, Prediction):
return result
diff --git a/dspy/primitives/code_interpreter.py b/dspy/primitives/code_interpreter.py
index 0197240c9..827c9627b 100644
--- a/dspy/primitives/code_interpreter.py
+++ b/dspy/primitives/code_interpreter.py
@@ -159,7 +159,7 @@ def _validate_interpreter_factory(factory: Any, name: str = "interpreter_factory
if not isinstance(factory, type) and isinstance(factory, CodeInterpreter):
raise TypeError(
f"{name} received an object that already implements CodeInterpreter, so its ownership "
- "is ambiguous. Pass an existing interpreter as the first positional argument when calling the module. "
+ "is ambiguous. "
"If this object also creates interpreters, pass a dedicated zero-argument creation callable instead."
)
if not callable(factory):
diff --git a/tests/predict/test_rlm.py b/tests/predict/test_rlm.py
index 704a3dccb..a570f7e64 100644
--- a/tests/predict/test_rlm.py
+++ b/tests/predict/test_rlm.py
@@ -266,7 +266,7 @@ class TestRLMInitialization:
assert "c" in str(exc_info.value)
def test_interpreter_instance_is_rejected_as_factory(self):
- with pytest.raises(TypeError, match="first positional argument when calling the module"):
+ with pytest.raises(TypeError, match="already implements CodeInterpreter"):
RLM("context -> answer", interpreter_factory=MockInterpreter())
def test_constructor_interpreter_keyword_is_removed(self):
@@ -276,14 +276,17 @@ class TestRLMInitialization:
def test_factory_return_value_is_validated(self):
rlm = RLM("query -> answer", interpreter_factory=lambda: None)
- with pytest.raises(TypeError, match="interpreter_factory must return a CodeInterpreter, not NoneType"):
+ with pytest.raises(TypeError, match="interpreter must implement CodeInterpreter, not NoneType"):
rlm(query="test")
- def test_keyword_interpreter_override_has_clear_error(self):
+ @pytest.mark.asyncio
+ async def test_positional_interpreter_factory_is_rejected(self):
rlm = RLM("query -> answer")
- with pytest.raises(TypeError, match="first positional argument"):
- rlm(query="test", interpreter=MockInterpreter())
+ with pytest.raises(TypeError):
+ rlm(MockInterpreterFactory(), query="test")
+ with pytest.raises(TypeError):
+ await rlm.acall(MockInterpreterFactory(), query="test")
def test_llm_query_returns_legacy_response_text(self):
from dspy.utils.dummies import DummyLM
@@ -510,29 +513,83 @@ class TestRLMInterpreterLifecycle:
interpreter.execute("print('closed')")
@pytest.mark.asyncio
- async def test_caller_owned_interpreter_can_be_reused_across_sequential_calls(self):
+ async def test_call_time_factory_overrides_constructor_and_context_without_mutating_module(self):
+ constructor = MockInterpreterFactory(responses=[FinalOutput({"answer": "constructor"})])
+ configured = MockInterpreterFactory()
+ factory = MockInterpreterFactory(responses=["exploring", FinalOutput({"answer": "override"})])
+ factory.execution_instructions = "Call-time runtime guidance."
+ rlm = RLM("query -> answer", max_iters=2, interpreter_factory=constructor)
+ signatures = []
+
+ class Action:
+ def __call__(self, **kwargs):
+ signatures.append(kwargs["signature"].instructions)
+ return dspy.Prediction(reasoning="Return answer", code="SUBMIT(answer)")
+
+ async def acall(self, **kwargs):
+ return self(**kwargs)
+
+ rlm.generate_action = Action()
+ original_signature = rlm._action_signature.instructions
+ with dspy.context(interpreter_factory=configured):
+ sync_result = rlm(query="sync", interpreter_factory=factory)
+ async_result = await rlm.acall(query="async", interpreter_factory=factory)
+ assert rlm(query="default").answer == "constructor"
+
+ assert sync_result.answer == async_result.answer == "override"
+ assert configured.instances == []
+ assert len(constructor.instances) == 1
+ assert len(factory.instances) == 2
+ assert factory.instances[0] is not factory.instances[1]
+ assert all("Call-time runtime guidance." in signature for signature in signatures[:4])
+ assert "Call-time runtime guidance." not in signatures[4]
+ assert rlm._action_signature.instructions == original_signature
+ for interpreter in factory.instances:
+ assert interpreter.call_count == 2
+ with pytest.raises(CodeInterpreterError, match="shutdown"):
+ interpreter.execute("print('closed')")
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize("factory", [MockInterpreter(), 123, lambda: None])
+ async def test_invalid_call_time_factory_is_rejected(self, factory):
+ rlm = RLM("query -> answer")
+ with pytest.raises(TypeError):
+ rlm(query="sync", interpreter_factory=factory)
+ with pytest.raises(TypeError):
+ await rlm.acall(query="async", interpreter_factory=factory)
+
+ def test_interpreter_factory_is_reserved_as_signature_input(self):
+ with pytest.raises(ValueError, match="'interpreter_factory' is reserved"):
+ RLM("interpreter_factory -> answer")
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize("phase", ["setup", "prediction", "extraction", "cancellation"])
+ async def test_call_time_factory_cleanup_on_failure(self, phase, monkeypatch):
+ import asyncio
+
+ error = asyncio.CancelledError() if phase == "cancellation" else ValueError(phase)
+
+ class RaisingPredictor:
+ async def acall(self, **kwargs):
+ raise error
+
factory = MockInterpreterFactory()
- interpreter = MockInterpreter(
- responses=[
- FinalOutput({"answer": "first"}),
- FinalOutput({"answer": "second"}),
- ]
- )
- rlm = RLM("query -> answer", max_iters=1, interpreter_factory=factory)
- rlm.generate_action = make_mock_predictor([
- {"reasoning": "Return answer", "code": "SUBMIT(answer)"},
- ])
+ rlm = RLM("query -> answer", max_iters=0 if phase == "extraction" else 1)
+ rlm.generate_action = RaisingPredictor()
+ rlm.extract = RaisingPredictor()
+ if phase == "setup":
+ def fail_setup(*args):
+ raise error
- try:
- sync_result = rlm(interpreter, query="sync")
- async_result = await rlm.acall(interpreter, query="async")
+ monkeypatch.setattr(rlm, "_inject_execution_context", fail_setup)
- assert sync_result.answer == "first"
- assert async_result.answer == "second"
- assert factory.instances == []
- assert interpreter.execute("print('still open')") == ""
- finally:
- interpreter.shutdown()
+ with pytest.raises(type(error)) as exc:
+ await rlm.acall(query="test", interpreter_factory=factory)
+
+ assert exc.value is error
+ assert len(factory.instances) == 1
+ with pytest.raises(CodeInterpreterError, match="shutdown"):
+ factory.instances[0].execute("print('closed')")
def test_factory_interpreter_is_shutdown_when_prediction_raises(self):
class RaisingPredictor:
@@ -816,7 +873,7 @@ class TestRLMCallMethod:
{"reasoning": "Return answer", "code": 'SUBMIT("42")'},
])
- result = rlm(mock, query="What is the answer?")
+ result = rlm(query="What is the answer?", interpreter_factory=lambda: mock)
assert result.answer == "42"
@@ -841,7 +898,7 @@ class TestRLMMaxIterationsFallback:
{"answer": "extracted_answer"},
])
- result = rlm.forward(mock, query="test")
+ result = rlm.forward(query="test", interpreter_factory=lambda: mock)
assert result.answer == "extracted_answer"
assert result.final_reasoning == "Extract forced final output"
@@ -864,7 +921,7 @@ class TestRLMToolExceptions:
{"reasoning": "Recover", "code": 'SUBMIT("recovered")'},
])
- result = rlm.forward(mock, query="test")
+ result = rlm.forward(query="test", interpreter_factory=lambda: mock)
assert result.answer == "recovered"
def test_runtime_error_history_uses_stripped_code(self):
@@ -879,7 +936,7 @@ class TestRLMToolExceptions:
{"reasoning": "Recover", "code": 'SUBMIT("recovered")'},
])
- result = rlm.forward(mock, query="test")
+ result = rlm.forward(query="test", interpreter_factory=lambda: mock)
assert result.answer == "recovered"
first_step = result.trajectory[0]
assert first_step["code"] == "print(x)"
@@ -896,7 +953,7 @@ class TestRLMToolExceptions:
{"reasoning": "Recover", "code": 'SUBMIT("recovered")'},
])
- result = rlm.forward(mock, query="test")
+ result = rlm.forward(query="test", interpreter_factory=lambda: mock)
assert result.answer == "recovered"
assert result.trajectory[0]["output"].startswith("[Error] invalid syntax")
@@ -911,7 +968,7 @@ class TestRLMToolExceptions:
{"reasoning": "Recover", "code": 'SUBMIT("recovered")'},
])
- result = rlm.forward(mock, query="test")
+ result = rlm.forward(query="test", interpreter_factory=lambda: mock)
assert result.answer == "recovered"
assert result.trajectory[0]["output"].startswith("[Error]")
@@ -930,7 +987,8 @@ class TestRLMToolExceptions:
rlm.extract = make_mock_predictor([{"answer": "hallucinated"}])
with pytest.raises(CodeInterpreterError, match="protocol corrupt"):
- rlm.forward(mock, query="test")
+ rlm.forward(query="test", interpreter_factory=lambda: mock)
+ assert mock._shutdown
@pytest.mark.asyncio
async def test_interpreter_failure_propagates_async(self):
@@ -948,7 +1006,8 @@ class TestRLMToolExceptions:
rlm.extract = make_mock_predictor([{"answer": "hallucinated"}])
with pytest.raises(CodeInterpreterError, match="protocol corrupt"):
- await rlm.aforward(mock, query="test")
+ await rlm.aforward(query="test", interpreter_factory=lambda: mock)
+ assert mock._shutdown
class TestRLMDynamicSignature:
@@ -1208,7 +1267,7 @@ class TestRLMAsyncMock:
{"reasoning": "Return answer", "code": 'SUBMIT("42")'},
])
- result = await rlm.aforward(mock, query="What is the answer?")
+ result = await rlm.aforward(query="What is the answer?", interpreter_factory=lambda: mock)
assert result.answer == "42"
@pytest.mark.asyncio
@@ -1220,7 +1279,7 @@ class TestRLMAsyncMock:
{"reasoning": "Return count", "code": "SUBMIT(42)"},
])
- result = await rlm.aforward(mock, query="count items")
+ result = await rlm.aforward(query="count items", interpreter_factory=lambda: mock)
assert result.count == 42
assert isinstance(result.count, int)
@@ -1237,7 +1296,7 @@ class TestRLMAsyncMock:
{"reasoning": "Now finish", "code": 'SUBMIT("done")'},
])
- result = await rlm.aforward(mock, query="test")
+ result = await rlm.aforward(query="test", interpreter_factory=lambda: mock)
assert result.answer == "done"
@@ -1259,7 +1318,7 @@ class TestRLMTypeCoercionMock:
{"reasoning": "Return value", "code": code},
])
- result = rlm.forward(mock, query="test")
+ result = rlm.forward(query="test", interpreter_factory=lambda: mock)
assert getattr(result, output_field) == expected
def test_type_error_retries(self):
@@ -1274,7 +1333,7 @@ class TestRLMTypeCoercionMock:
{"reasoning": "Try yes", "code": 'SUBMIT("yes")'},
])
- result = rlm.forward(mock, query="is it yes?")
+ result = rlm.forward(query="is it yes?", interpreter_factory=lambda: mock)
assert result.answer == "yes"
@@ -1299,25 +1358,25 @@ class TestRLMTypeCoercion:
("data", "dict[str, str]", 'SUBMIT({"key": "value"})', {"key": "value"}, dict),
("answer", "Literal['yes', 'no']", 'SUBMIT("yes")', "yes", str),
])
- def test_type_coercion(self, output_field, output_type, code, expected, expected_type, pooled_interpreter):
+ def test_type_coercion(self, output_field, output_type, code, expected, expected_type):
"""Test RLM type coercion for various types with PythonInterpreter."""
rlm = RLM(f"query -> {output_field}: {output_type}", max_iters=3)
rlm.generate_action = make_mock_predictor([
{"reasoning": "Return value", "code": code},
])
- result = rlm.forward(pooled_interpreter, query="test")
+ result = rlm.forward(query="test", interpreter_factory=PythonInterpreter)
assert getattr(result, output_field) == expected
assert isinstance(getattr(result, output_field), expected_type)
- def test_submit_extracts_typed_value(self, pooled_interpreter):
+ def test_submit_extracts_typed_value(self):
"""Test RLM SUBMIT correctly extracts typed value."""
rlm = RLM("query -> count: int", max_iters=3)
rlm.generate_action = make_mock_predictor([
{"reasoning": "Compute and return", "code": "result = 42\nSUBMIT(result)"},
])
- result = rlm.forward(pooled_interpreter, query="count items")
+ result = rlm.forward(query="count items", interpreter_factory=PythonInterpreter)
assert result.count == 42
assert isinstance(result.count, int)
@@ -1334,42 +1393,42 @@ class TestRLMMultipleOutputs:
Tests SUBMIT() calling patterns with multi-output signatures.
"""
- def test_multi_output_final_kwargs(self, pooled_interpreter):
+ def test_multi_output_final_kwargs(self):
"""SUBMIT(field1=val1, field2=val2) with keyword args."""
rlm = RLM("query -> name: str, count: int", max_iters=3)
rlm.generate_action = make_mock_predictor([
{"reasoning": "Return both outputs", "code": 'SUBMIT(name="alice", count=5)'},
])
- result = rlm.forward(pooled_interpreter, query="test")
+ result = rlm.forward(query="test", interpreter_factory=PythonInterpreter)
assert result.name == "alice"
assert result.count == 5
assert isinstance(result.count, int)
- def test_multi_output_final_positional(self, pooled_interpreter):
+ def test_multi_output_final_positional(self):
"""SUBMIT(val1, val2) with positional args mapped to field order."""
rlm = RLM("query -> name: str, count: int", max_iters=3)
rlm.generate_action = make_mock_predictor([
{"reasoning": "Return both outputs positionally", "code": 'SUBMIT("bob", 10)'},
])
- result = rlm.forward(pooled_interpreter, query="test")
+ result = rlm.forward(query="test", interpreter_factory=PythonInterpreter)
assert result.name == "bob"
assert result.count == 10
- def test_multi_output_three_fields(self, pooled_interpreter):
+ def test_multi_output_three_fields(self):
"""Signature with 3+ output fields of different types."""
rlm = RLM("query -> name: str, age: int, active: bool", max_iters=3)
rlm.generate_action = make_mock_predictor([
{"reasoning": "Return all three", "code": 'SUBMIT(name="carol", age=30, active=True)'},
])
- result = rlm.forward(pooled_interpreter, query="test")
+ result = rlm.forward(query="test", interpreter_factory=PythonInterpreter)
assert result.name == "carol"
assert result.age == 30
assert result.active is True
- def test_multi_output_final_missing_field_errors(self, pooled_interpreter):
+ def test_multi_output_final_missing_field_errors(self):
"""SUBMIT() with missing field should return error in output."""
rlm = RLM("query -> name: str, count: int", max_iters=3)
rlm.generate_action = make_mock_predictor([
@@ -1378,29 +1437,29 @@ class TestRLMMultipleOutputs:
])
# RLM should retry after getting error for missing field
- result = rlm.forward(pooled_interpreter, query="test")
+ result = rlm.forward(query="test", interpreter_factory=PythonInterpreter)
assert result.name == "alice"
assert result.count == 5
- def test_multi_output_submit_vars(self, pooled_interpreter):
+ def test_multi_output_submit_vars(self):
"""SUBMIT can pass variables directly for multiple outputs."""
rlm = RLM("query -> name: str, count: int", max_iters=3)
rlm.generate_action = make_mock_predictor([
{"reasoning": "Use SUBMIT", "code": 'n = "dave"\nc = 15\nSUBMIT(n, c)'},
])
- result = rlm.forward(pooled_interpreter, query="test")
+ result = rlm.forward(query="test", interpreter_factory=PythonInterpreter)
assert result.name == "dave"
assert result.count == 15
- def test_multi_output_type_coercion(self, pooled_interpreter):
+ def test_multi_output_type_coercion(self):
"""Each output field is coerced to its declared type."""
rlm = RLM("query -> count: int, ratio: float, flag: bool", max_iters=3)
rlm.generate_action = make_mock_predictor([
{"reasoning": "Return mixed types", "code": "SUBMIT(count=42, ratio=3.14, flag=True)"},
])
- result = rlm.forward(pooled_interpreter, query="test")
+ result = rlm.forward(query="test", interpreter_factory=PythonInterpreter)
assert result.count == 42
assert isinstance(result.count, int)
assert result.ratio == 3.14
@@ -1422,40 +1481,43 @@ class TestRLMWithDummyLM:
typed output_fields for SUBMIT based on the signature.
"""
- def test_simple_computation_e2e(self, pooled_interpreter):
+ def test_simple_computation_e2e(self):
"""Test full RLM pipeline: DummyLM -> RLM -> PythonInterpreter -> result."""
+ configured = MockInterpreterFactory()
with dummy_lm_context([
{"reasoning": "I need to compute 2 + 3", "code": "result = 2 + 3\nSUBMIT(result)"},
]):
rlm = RLM("query -> answer: int", max_iters=3)
- result = rlm.forward(pooled_interpreter, query="What is 2 + 3?")
+ with dspy.context(interpreter_factory=configured):
+ result = rlm(query="What is 2 + 3?", interpreter_factory=PythonInterpreter)
assert result.answer == 5
assert isinstance(result.answer, int)
+ assert configured.instances == []
- def test_multi_turn_computation_e2e(self, pooled_interpreter):
+ def test_multi_turn_computation_e2e(self):
"""Test RLM with multiple turns before SUBMIT."""
with dummy_lm_context([
{"reasoning": "First explore the data", "code": "x = 10\nprint(f'x = {x}')"},
{"reasoning": "Now compute and return", "code": "y = x * 2\nSUBMIT(y)"},
]):
rlm = RLM("query -> answer: int", max_iters=5)
- result = rlm.forward(pooled_interpreter, query="Double ten")
+ result = rlm.forward(query="Double ten", interpreter_factory=PythonInterpreter)
assert result.answer == 20
assert len(result.trajectory) == 2
- def test_with_input_variables_e2e(self, pooled_interpreter):
+ def test_with_input_variables_e2e(self):
"""Test RLM with input variables passed to sandbox."""
with dummy_lm_context([
{"reasoning": "Sum the numbers in the list", "code": "SUBMIT(sum(numbers))"},
]):
rlm = RLM("numbers: list[int] -> total: int", max_iters=3)
- result = rlm.forward(pooled_interpreter, numbers=[1, 2, 3, 4, 5])
+ result = rlm.forward(numbers=[1, 2, 3, 4, 5], interpreter_factory=PythonInterpreter)
assert result.total == 15
- def test_with_tool_e2e(self, pooled_interpreter):
+ def test_with_tool_e2e(self):
"""Test RLM calling a host-side tool through the sandbox."""
def lookup(key: str) -> str:
return {"apple": "red", "banana": "yellow"}.get(key, "unknown")
@@ -1464,11 +1526,11 @@ class TestRLMWithDummyLM:
{"reasoning": "Look up the color of apple", "code": 'color = lookup(key="apple")\nSUBMIT(color)'},
]):
rlm = RLM("fruit -> color: str", max_iters=3, tools=[lookup])
- result = rlm.forward(pooled_interpreter, fruit="apple")
+ result = rlm.forward(fruit="apple", interpreter_factory=PythonInterpreter)
assert result.color == "red"
- def test_dspy_tool_execution_semantics_e2e(self, pooled_interpreter):
+ def test_dspy_tool_execution_semantics_e2e(self):
import inspect
from pydantic import BaseModel
@@ -1507,7 +1569,7 @@ class TestRLMWithDummyLM:
},
]):
with dspy.context(callbacks=[Recorder()]):
- result = rlm.forward(pooled_interpreter, query="test")
+ result = rlm.forward(query="test", interpreter_factory=PythonInterpreter)
assert result.answer == 6
assert len(received) == 1
@@ -1751,7 +1813,7 @@ class TestPrepareSerializableVars:
])
stub = _StubSerializable("test_payload")
- result = rlm.forward(mock, data=stub, query="test")
+ result = rlm.forward(data=stub, query="test", interpreter_factory=lambda: mock)
assert result.answer == "done"
# First call should be the serializable setup, second should be the iteration