Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

feat(RLM/Flex): Implement sub-dspy interpreters (#10478)

Michael I.ba3f9198efe5
diff --git a/docs/docs/diving-deeper/flex.md b/docs/docs/diving-deeper/flex.md
index 4d8a41b8a..c90561e1d 100644
--- a/docs/docs/diving-deeper/flex.md
+++ b/docs/docs/diving-deeper/flex.md
@@ -90,7 +90,7 @@ The optimizer-authored code does not run against the real `dspy` package. Inside
**Available**
- `dspy.Module` — the base class the generated source subclasses. `__init__` assigns predictors; `forward` runs in the sandbox.
-- `dspy.Predict`, `dspy.ChainOfThought`, `dspy.ReAct`, `dspy.ReActV2`, `dspy.RLM` — constructed in the sandbox but *built on the host*, where the real LM calls happen. Constructor kwargs cross as JSON.
+- `dspy.Predict`, `dspy.ChainOfThought`, `dspy.ReAct`, `dspy.ReActV2`, `dspy.RLM` — constructed in the sandbox but *built on the host*, where the real LM calls happen. Constructor kwargs cross as JSON; generation parameters (`temperature`, `max_tokens`, `n`, …) apply to the bridged calls, while LM routing and credential options (`api_base`, `api_key`, headers, `model`) are rejected by the host.
- `dspy.Signature("inputs -> outputs", "instructions")` — the string form only, used to give an inner predictor its instructions. It is a marker the host turns back into a real `Signature`, not the class: it has no methods, no `with_instructions()`, and no `dspy.InputField` / `dspy.OutputField` class form.
- `dspy.Prediction(**fields)` — the return value of `forward`. Holds fields; it is not the host `Prediction` type.
- `dspy.Tool(func)` — a pass-through wrapper. Only tools you passed to `dspy.Flex(tools=...)` can be handed to a bridged sub-predictor; they are already in scope by name, so wrapping is optional.
@@ -104,7 +104,7 @@ The optimizer-authored code does not run against the real `dspy` package. Inside
- Class-based signatures and typed field declarations.
- Host objects generally: values cross as JSON, so a tool you define inside `forward` cannot be passed to a bridged sub-predictor, and a predictor field that isn't JSON-serializable raises rather than silently degrading.
-Anything outside this surface fails when the candidate runs, and GEPA scores it at the failure score — a missing name costs a search step rather than crashing the run. The shim lives in `dspy/predict/flex/_sandbox_shim.py` and the proposer-facing version of this list is `dspy/predict/flex/primitives_doc.py`.
+Anything outside this surface fails when the candidate runs, and GEPA scores it at the failure score — a missing name costs a search step rather than crashing the run. The shim lives in `dspy/primitives/_facade_shim.py` and the proposer-facing version of this list is `dspy/predict/flex/primitives_doc.py`.
### Saving and loading
diff --git a/docs/docs/diving-deeper/rlm.md b/docs/docs/diving-deeper/rlm.md
index 993dc7206..19018e1dc 100644
--- a/docs/docs/diving-deeper/rlm.md
+++ b/docs/docs/diving-deeper/rlm.md
@@ -93,7 +93,7 @@ A list of plain functions or `dspy.Tool` objects. RLM normalizes each to a `Tool
A zero-argument callable that returns a fresh `CodeInterpreter` for one invocation. RLM may call the factory concurrently, and it always shuts down the returned interpreter. A class such as `PythonInterpreter` or `LocalInterpreter` is already a factory; use `functools.partial` or a callable provider object when construction needs configuration. RLM adds invocation-scoped tools to the returned interpreter's mutable `tools` dictionary, so remote sandboxes need a `CodeInterpreter` adapter that supports that protocol. `dspy.configure(interpreter_factory=...)` replaces the default on each `forward()`, so `dspy.context(interpreter_factory=...)` scopes the choice. If the active factory exposes an `execution_instructions` string, RLM refreshes it for each action call so the prompt matches the runtime executing the generated code.
**`__call__(**inputs, interpreter_factory=...)` / `acall(**inputs, interpreter_factory=...)`**
-A per-invocation factory override, supplied by keyword. For example, `rlm(query=query, interpreter_factory=dspy.PythonInterpreter)` explicitly chooses PythonInterpreter even when another factory is configured. The factory must return a fresh interpreter; RLM injects tools and output metadata and always shuts it down. Its `execution_instructions` metadata controls the runtime guidance for that invocation without changing the shared predictor. Live interpreter instances are no longer accepted at call time. The factory option is keyword-only, and `interpreter_factory` is reserved for runtime configuration rather than signature inputs.
+A per-invocation factory override, supplied by keyword. For example, `rlm(query=query, interpreter_factory=dspy.PythonInterpreter)` explicitly chooses PythonInterpreter even when another factory is configured. The factory must return a fresh interpreter; RLM injects tools and output metadata and always shuts it down. Its `execution_instructions` metadata controls the runtime guidance for that invocation without changing the shared predictor. Sub-agents built in the sandbox run their own code on this factory too. Live interpreter instances are no longer accepted at call time. The factory option is keyword-only, and `interpreter_factory` is reserved for runtime configuration rather than signature inputs.
**`dspy.SandboxSerializable`**
The base class for inputs that need custom loading. Implement `sandbox_setup`, `to_sandbox`, `sandbox_assignment`, and `rlm_preview`. It also defines a Pydantic schema hook, so a subclass can be a typed field in a signature, as in `data: DataFrame = dspy.InputField()`.
diff --git a/dspy/predict/flex/bridge.py b/dspy/predict/flex/bridge.py
index ea163839d..d1975ceb3 100644
--- a/dspy/predict/flex/bridge.py
+++ b/dspy/predict/flex/bridge.py
@@ -5,61 +5,33 @@ every ``forward``:
1. creates a fresh interpreter from the Flex's ``interpreter_factory``, or from
``dspy.settings.interpreter_factory`` when Flex uses its default (``BridgeRuntime.forward``);
-2. injects the sandbox-side shim (``_sandbox_shim.py``), which fakes a tiny ``dspy`` module whose
- predictor constructors and calls are proxies;
+2. installs the sandbox dspy facade (``dspy.primitives.facade``) over the Flex's tools, interpreter
+ factory, predictor-call budget, and custom types;
3. executes ``module_src`` and drives its ``forward`` with the call's inputs.
-Only three things cross the sandbox boundary, all as JSON over the interpreter's tool-call
-protocol (``CodeInterpreter.tools``):
-
-- ``__dspy_construct__``: the shim asks the host to build a real predictor (``_Invocation.construct``).
- The host constructs it from the serialized signature/kwargs and returns a string handle.
-- ``__dspy_call__``: the shim runs a predictor by handle (``_Invocation.call``); the host makes the
- real LM call and returns the prediction's fields as JSON.
-- user tools passed to ``dspy.Flex(tools=...)``, registered by name so sandbox code can call them
- directly; the callables themselves stay on the host.
-
-All per-forward state, including the constructed predictors, the predictor-call budget, custom-type
-originals, and the last LM infrastructure error, lives in a ``_Invocation`` created for that
-forward, never on the shared ``Flex``, so concurrent forwards (e.g. threaded evaluation) are
-isolated. The final ``dspy.Prediction`` is parsed against the Flex signature's declared output
-types on the way out (``BridgeRuntime._to_prediction``).
+User tools passed to ``dspy.Flex(tools=...)`` are registered by name so sandbox code can call them
+directly; the callables themselves stay on the host. The final ``dspy.Prediction`` is parsed against
+the Flex signature's declared output types on the way out (``BridgeRuntime._to_prediction``).
"""
from __future__ import annotations
import ast
import functools
-import inspect
import json
import logging
-from pathlib import Path
from typing import Any, Callable
from pydantic import TypeAdapter
-from pydantic_core import PydanticSerializationError, to_jsonable_python
import dspy
-from dspy import CodeInterpreterError
from dspy.adapters.types.base_type import Type as _CustomType
from dspy.adapters.utils import annotation_allows_none, parse_value
-from dspy.primitives.code_interpreter import _create_interpreter
-from dspy.signatures.signature import make_signature
-from dspy.utils.exceptions import LMError
+from dspy.primitives.code_interpreter import CodeInterpreterError, _create_interpreter
+from dspy.primitives.facade import FacadeInvocation, restore_custom_types
logger = logging.getLogger(__name__)
-# Tool names the shim uses to call the host; must match the names registered in BridgeRuntime.
-CONSTRUCT_TOOL = "__dspy_construct__"
-CALL_TOOL = "__dspy_call__"
-
-# Predictors the sandbox shim may construct and the host builds
-BRIDGEABLE_KINDS = ("Predict", "ChainOfThought", "RLM", "CodeAct", "ProgramOfThought", "ReAct", "ReActV2")
-# The shim's dspy.Signature(...) emits this marker so the host can rebuild a Signature.
-SIGNATURE_MARKER = "__dspy_sig__"
-# The shim passes tools by name (callables can't cross the JSON boundary); the host resolves the name
-# back to the real tool object passed to dspy.Flex(tools=...).
-TOOL_MARKER = "__dspy_tool__"
# Variable/identifier names used in the per-forward driver code (namespaced to avoid clashing with
# whatever the optimizer-authored module uses).
_INPUTS_VAR = "__dspy_flex_inputs"
@@ -68,10 +40,6 @@ _OUT_VAR = "__dspy_flex_out"
_JSON_VAR = "__dspy_flex_json"
-# The sandbox-side dspy shim, injected as text into each per-forward interpreter.
-SHIM_SETUP = (Path(__file__).parent / "_sandbox_shim.py").read_text(encoding="utf-8")
-
-
def parse_module_class_name(module_src: str) -> str:
"""Return the name of the generated ``dspy.Module`` subclass in ``module_src``."""
tree = ast.parse(module_src)
@@ -85,53 +53,6 @@ def parse_module_class_name(module_src: str) -> str:
return chosen[0].name
-def _accepts_interpreter_factory(cls: type) -> bool:
- """True if ``cls.__init__`` takes an ``interpreter_factory`` parameter."""
- try:
- return "interpreter_factory" in inspect.signature(cls.__init__).parameters
- except (TypeError, ValueError):
- return False
-
-
-def _resolve_signature(signature: Any, custom_types: dict[str, type] | None = None) -> Any:
- """Turn a shim signature payload back into something a host predictor accepts."""
- if isinstance(signature, dict) and signature.get(SIGNATURE_MARKER):
- # marker always carries a string signature; make_signature applies instructions if given
- return make_signature(signature["signature"], signature.get("instructions"), custom_types=custom_types)
- if isinstance(signature, str):
- return make_signature(signature, custom_types=custom_types)
- return signature
-
-
-def _jsonable(value: Any) -> Any:
- """Coerce a predictor output field to a JSON-serializable value."""
- try:
- return to_jsonable_python(value)
- except PydanticSerializationError:
- return value
-
-
-def prediction_to_fields(pred: Any) -> dict[str, Any]:
- """Serialise a host ``dspy.Prediction`` (or dict) to a JSON-able field dict for the sandbox."""
- store = getattr(pred, "_store", None)
- if store is None:
- if isinstance(pred, dict):
- store = pred
- else:
- raise CodeInterpreterError(
- f"A bridged predictor must return a dspy.Prediction; got {type(pred).__name__}"
- )
- fields = {k: _jsonable(v) for k, v in dict(store).items()}
- try:
- json.dumps(fields)
- except TypeError as e:
- raise CodeInterpreterError(
- "A bridged predictor returned a field that cannot cross the sandbox boundary "
- f"(must be JSON-serializable): {e}"
- ) from e
- return fields
-
-
def _tool_entrypoint(tool: Any) -> Callable[..., Any]:
"""A callable for the interpreter's tool registry."""
func = getattr(tool, "func", None)
@@ -150,7 +71,7 @@ def _restoring_entrypoint(fn: Callable[..., Any], originals: dict[str, Any]) ->
@functools.wraps(fn)
def entrypoint(**kwargs: Any) -> Any:
- return fn(**{k: _restore_custom_types(v, originals) for k, v in kwargs.items()})
+ return fn(**{k: restore_custom_types(v, originals) for k, v in kwargs.items()})
return entrypoint
@@ -167,66 +88,12 @@ def _collect_custom_type_originals(value: Any, out: dict[str, Any]) -> None:
_collect_custom_type_originals(v, out)
-def _restore_custom_types(value: Any, originals: dict[str, Any]) -> Any:
- """Substitute serialized custom-type strings back with the original host objects."""
- if isinstance(value, str) and value in originals:
- return originals[value]
- if isinstance(value, dict):
- return {k: _restore_custom_types(v, originals) for k, v in value.items()}
- if isinstance(value, list):
- return [_restore_custom_types(v, originals) for v in value]
- return value
-
-
-class _Invocation:
- """Per-forward bridge state: the predictors this forward constructed (keyed by the sandbox
- attribute name), its predictor-call budget, and the original custom-type objects to restore
- on bridged calls. Each forward gets its own ``_Invocation``."""
-
- def __init__(self, runtime: BridgeRuntime, originals: dict[str, Any]) -> None:
- self._runtime = runtime
- self._originals = originals
- self._predictors: dict[str, Any] = {}
- self._calls = 0
- self._lm_error: tuple[LMError, str] | None = None
-
- def construct(self, kind: str, signature: Any, attr_name: str, kwargs: dict[str, Any] | None = None) -> str:
- self._lm_error = None
- if kind not in BRIDGEABLE_KINDS:
- raise CodeInterpreterError(
- f"dspy.{kind} is not supported inside a sandboxed dspy.Flex yet "
- f"(bridgeable: {', '.join(BRIDGEABLE_KINDS)})"
- )
- self._predictors[attr_name] = self._runtime._build_predictor(kind, signature, kwargs)
- return attr_name
-
- def call(self, handle: str, inputs: dict[str, Any] | None = None) -> dict[str, Any]:
- self._lm_error = None
- self._calls += 1
- budget = self._runtime._max_predictor_calls
- if budget is not None and self._calls > budget:
- raise CodeInterpreterError(
- f"Sandboxed dspy.Flex forward exceeded its predictor-call budget "
- f"({budget}). Raise max_predictor_calls if this is expected."
- )
- predictor = self._predictors.get(handle)
- if predictor is None:
- raise CodeInterpreterError(f"Unknown predictor handle: {handle!r}")
- restored = {k: _restore_custom_types(v, self._originals) for k, v in (inputs or {}).items()}
- try:
- return prediction_to_fields(predictor(**restored))
- except LMError as e:
- tag = f"[dspy bridge lm-error #{self._calls}]"
- self._lm_error = (e, tag)
- raise CodeInterpreterError(f"{tag} {type(e).__name__}: {e}") from e
-
-
class BridgeRuntime:
"""Host side of the dspy.Flex bridge: runs the bound ``module_src`` for one ``Flex``.
Each ``forward`` creates a fresh interpreter from the factory, executes the shim and the
bound source, runs the module instance, and shuts the interpreter down. Predictors are built
- host-side but held by that forward's ``_Invocation`` — never set on the ``Flex`` — so nothing
+ host-side but held by that forward's ``FacadeInvocation`` — never set on the ``Flex`` — so nothing
outlives the call and parallel forwards stay isolated. ``_max_predictor_calls`` caps bridged
predictor calls per forward.
@@ -249,20 +116,30 @@ class BridgeRuntime:
self._module_src = module_src
self._class_name = parse_module_class_name(module_src)
+ def invocation(self, originals: dict[str, Any] | None = None) -> FacadeInvocation:
+ """A fresh per-forward invocation over tools, factory, budget, and custom types."""
+ ctx = self._flex._flex_ctx
+ return FacadeInvocation(
+ ctx.context_names(),
+ self._factory,
+ self._max_predictor_calls,
+ custom_types=ctx.custom_types(),
+ originals=originals,
+ )
+
def forward(self, inputs: dict[str, Any]) -> Any:
originals: dict[str, Any] = {}
for v in inputs.values():
_collect_custom_type_originals(v, originals)
- invocation = _Invocation(self, originals)
+ invocation = self.invocation(originals)
interp = _create_interpreter(self._factory)
try:
- interp.tools.update({CONSTRUCT_TOOL: invocation.construct, CALL_TOOL: invocation.call})
# User tools callable by name in the sandbox.
interp.tools.update(
{name: _restoring_entrypoint(fn, originals) for name, fn in self._tool_callables().items()}
)
- interp.execute(SHIM_SETUP)
+ invocation.install(interp)
interp.execute(self._module_src) # defines the class in the sandbox
code = (
f"{_INSTANCE_VAR} = {self._class_name}()\n"
@@ -330,38 +207,3 @@ class BridgeRuntime:
def _tool_callables(self) -> dict[str, Callable[..., Any]]:
"""User tools as ``name -> callable`` to register so sandbox code can call them."""
return {name: _tool_entrypoint(tool) for name, tool in self._flex._flex_ctx.context_names().items()}
-
- def _resolve_tool(self, name: str) -> Any:
- ctx = self._flex._flex_ctx.context_names()
- if name not in ctx:
- raise CodeInterpreterError(
- f"Sandboxed code referenced tool {name!r}, which was not passed to dspy.Flex(tools=...). "
- "Tools authored inside the generated module cannot be handed to a bridged sub-predictor."
- )
- return ctx[name]
-
- def _decode_tools(self, value: Any) -> Any:
- """Turn the shim's tool name-markers back into the real host tool objects."""
- if isinstance(value, dict) and TOOL_MARKER in value:
- return self._resolve_tool(value[TOOL_MARKER])
- if isinstance(value, list):
- return [self._decode_tools(v) for v in value]
- if isinstance(value, dict):
- return {k: self._decode_tools(v) for k, v in value.items()}
- return value
-
- def _sub_interpreter_factory(self) -> Any:
- return self._factory
-
- def _build_predictor(self, kind: str, signature: Any, kwargs: dict[str, Any] | None) -> Any:
- cls = getattr(dspy, kind)
- extra = {k: self._decode_tools(v) for k, v in (kwargs or {}).items()}
- # A code-executing sub-predictor should run its inner code in the backend chosen for Flex, so
- # hand it the Flex interpreter factory (it makes and tears down a fresh interpreter per forward).
- # The sandbox code can't set this itself, since a live interpreter can't cross the boundary.
- # The resolver lets a configured factory override the default for Flex and its
- # code-executing sub-predictors alike.
- if "interpreter_factory" not in extra and _accepts_interpreter_factory(cls):
- factory = self._sub_interpreter_factory()
- extra["interpreter_factory"] = factory
- return cls(_resolve_signature(signature, self._flex._flex_ctx.custom_types()), **extra)
diff --git a/dspy/predict/flex/ctx.py b/dspy/predict/flex/ctx.py
index e5bc3120e..31d808695 100644
--- a/dspy/predict/flex/ctx.py
+++ b/dspy/predict/flex/ctx.py
@@ -6,6 +6,8 @@ import typing
from dataclasses import dataclass, field
from typing import Any, get_args, get_origin
+from dspy.primitives.facade import is_reserved_sandbox_name
+
@dataclass
class FlexContext:
@@ -25,10 +27,10 @@ class FlexContext:
f"referenced by name in the generated code — but got {name!r}. Use a `def` "
f"function or `dspy.Tool(func, name='my_tool')`."
)
- if name == "dspy" or name.startswith(("_dspy", "__dspy")):
+ if is_reserved_sandbox_name(name):
raise ValueError(
- f"Tool name {name!r} is reserved: 'dspy' and names starting with '_dspy' belong "
- f"to the dspy.Flex sandbox. Rename it, e.g. `dspy.Tool(func, name='my_tool')`."
+ f"Tool name {name!r} is reserved: 'dspy' and names starting with '_dspy' belong to the "
+ "sandbox. Rename it, e.g. `dspy.Tool(func, name='my_tool')`."
)
out[name] = tool
return out
diff --git a/dspy/predict/rlm.py b/dspy/predict/rlm.py
index 7b0e20e06..cc9bf813e 100644
--- a/dspy/predict/rlm.py
+++ b/dspy/predict/rlm.py
@@ -37,6 +37,7 @@ from dspy.primitives.code_interpreter import (
_validate_interpreter_factory,
resolve_interpreter_factory,
)
+from dspy.primitives.facade import CALL_TOOL, CONSTRUCT_TOOL, FacadeInvocation, is_reserved_sandbox_name
from dspy.primitives.module import Module
from dspy.primitives.prediction import Prediction
from dspy.primitives.python_interpreter import PythonInterpreter
@@ -77,9 +78,41 @@ IMPORTANT: This is ITERATIVE. Each code block you write will execute, you'll see
You have max {max_llm_calls} sub-LLM calls. When done, call SUBMIT() with your output."""
+# Appended to the interpreter rules whenever RLM installs the sandbox dspy facade (every factory-made interpreter).
+SUB_AGENT_INSTRUCTIONS = """
+Sub-agents (dspy):
+You may `import dspy` and build sub-agents in the REPL for subtasks that need structured inputs/outputs.
+- `dspy.Predict("question -> answer")(question=...)` or `dspy.ChainOfThought(...)` - single-step sub-agents.
+- `dspy.ReActV2("question -> answer", tools=[...])(question=...)` - a multi-step tool-using sub-agent.
+ Only the provided tools listed above may be passed; functions you define in the REPL cannot cross to the host.
+- `dspy.RLM("context, query -> answer")(context=..., query=...)` - a recursive sub-agent with its own
+ REPL on the same interpreter backend as yours; it cannot be given another `interpreter_factory`.
+ This is the heaviest option: reserve it for deep subtasks whose input is itself too large or
+ structured to prompt directly, and prefer Predict/ChainOfThought/ReActV2 for everything else.
+Prefer `llm_query` for simple one-shot prompts; use sub-agents for structured, tool-using, or
+recursive subtasks.
+"""
_PYTHON_FENCE_LANGS = {"python", "py", "python3", "py3", ""}
+class _LLMCallBudget:
+ """Per-forward ceiling on sub-LLM calls, shared by every host path generated code can reach."""
+
+ def __init__(self, limit: int):
+ self._limit = limit
+ self._count = 0
+ self._lock = threading.Lock()
+
+ def reserve(self, n: int = 1) -> None:
+ with self._lock:
+ if self._count + n > self._limit:
+ raise RuntimeError(
+ f"LLM call limit exceeded: {self._count} + {n} > {self._limit}. "
+ f"Use Python code for aggregation instead of making more LLM calls."
+ )
+ self._count += n
+
+
def _strip_code_fences(code: str) -> str:
"""Extract Python code from markdown fences, or return as-is if no fences."""
code = code.strip()
@@ -155,17 +188,19 @@ class RLM(Module):
signature: Defines inputs and outputs. String like "context, query -> answer"
or a Signature class.
max_iters: Maximum REPL interaction iterations.
- max_llm_calls: Maximum sub-LLM calls (llm_query/llm_query_batched) per execution.
+ max_llm_calls: Maximum sub-LLM calls (llm_query/llm_query_batched and sub-agent LM calls) per execution.
max_output_chars: Maximum characters to include from REPL output.
verbose: Whether to log detailed execution info.
tools: List of tool functions or dspy.Tool objects callable from interpreter code.
Built-in tools: llm_query(prompt), llm_query_batched(prompts).
- sub_lm: LM for llm_query/llm_query_batched. Defaults to dspy.settings.lm.
+ sub_lm: LM for llm_query/llm_query_batched and sub-agents. Defaults to dspy.settings.lm.
Allows using a different (e.g., cheaper) model for sub-queries.
interpreter_factory: Zero-argument callable that creates an interpreter for each forward pass. The
callable may be invoked concurrently, and DSPy shuts down each interpreter it returns. RLM updates
the returned interpreter's mutable ``tools`` dictionary before execution. The callable may expose
- an ``execution_instructions`` string describing its runtime for the action prompt. RLM applies the
+ an ``execution_instructions`` string describing its runtime for the action prompt. RLM installs the
+ sandbox dspy facade into every interpreter it creates, so the runtime must be able to host it
+ (see ``CodeInterpreter``). RLM applies the
active factory's instructions to each action call, so ``dspy.context`` can switch runtimes
without changing the shared predictor signature. Defaults to ``dspy.PythonInterpreter``;
``dspy.configure(interpreter_factory=...)`` replaces the default.
@@ -236,17 +271,20 @@ class RLM(Module):
return normalized
def _validate_namespace(self, tools: dict[str, Tool]) -> None:
- """Validate names owned by the RLM call, result, and sandbox APIs."""
+ """Validate names owned by the RLM call, result, sandbox APIs, and the sandbox dspy facade."""
+ def is_reserved(name: str) -> bool:
+ return name in self._RESERVED_SANDBOX_NAMES or is_reserved_sandbox_name(name)
+
for name in tools:
if not name.isidentifier() or keyword.iskeyword(name):
raise ValueError(f"Invalid tool name '{name}': must be a valid Python identifier and not a keyword")
- if name in self._RESERVED_SANDBOX_NAMES:
+ if is_reserved(name):
raise ValueError(f"Tool name '{name}' conflicts with built-in sandbox function")
input_names = set(self.signature.input_fields)
if "interpreter_factory" in input_names:
raise ValueError("'interpreter_factory' is reserved for RLM runtime configuration, not a signature input.")
- reserved_inputs = sorted(input_names & self._RESERVED_SANDBOX_NAMES)
+ reserved_inputs = sorted(name for name in input_names if is_reserved(name))
if reserved_inputs:
raise ValueError(f"Input fields conflict with built-in sandbox functions: {reserved_inputs}")
@@ -279,21 +317,11 @@ class RLM(Module):
return "\n".join(lines)
- def _make_llm_tools(self, max_workers: int = 8) -> dict[str, Callable]:
- """Create llm_query and llm_query_batched tools with a fresh call counter."""
- state = {"call_count": 0}
- lock = threading.Lock()
+ def _make_llm_tools(self, budget: _LLMCallBudget | None = None, max_workers: int = 8) -> dict[str, Callable]:
+ """Create llm_query and llm_query_batched tools drawing on ``budget`` (fresh by default)."""
+ budget = budget if budget is not None else _LLMCallBudget(self.max_llm_calls)
lm = self.sub_lm
- def _check_and_increment(n: int = 1) -> None:
- with lock:
- if state["call_count"] + n > self.max_llm_calls:
- raise RuntimeError(
- f"LLM call limit exceeded: {state['call_count']} + {n} > {self.max_llm_calls}. "
- f"Use Python code for aggregation instead of making more LLM calls."
- )
- state["call_count"] += n
-
def _query_lm(prompt: str) -> str:
target_lm = lm if lm is not None else dspy.settings.lm
if target_lm is None:
@@ -320,14 +348,14 @@ class RLM(Module):
"""Query the LLM with a prompt string."""
if not prompt:
raise ValueError("prompt cannot be empty")
- _check_and_increment(1)
+ budget.reserve(1)
return _query_lm(prompt)
def llm_query_batched(prompts: list[str]) -> list[str]:
"""Query prompts concurrently, isolating LM failures while propagating contract errors."""
if not prompts:
return []
- _check_and_increment(len(prompts))
+ budget.reserve(len(prompts))
results: dict[int, str] = {}
with ThreadPoolExecutor(max_workers=max_workers) as executor:
@@ -377,7 +405,10 @@ class RLM(Module):
factory = resolve_interpreter_factory(self._interpreter_factory)
execution_instructions = self._get_execution_instructions(factory)
self._initial_execution_instructions = execution_instructions
- self._initial_interpreter_rules = self._format_interpreter_rules(execution_instructions)
+ self._initial_sub_agent_rules = SUB_AGENT_INSTRUCTIONS
+ self._initial_interpreter_rules = self._format_interpreter_rules(
+ execution_instructions, self._initial_sub_agent_rules
+ )
interpreter_rules = self._initial_interpreter_rules
action_sig = (
@@ -413,8 +444,9 @@ class RLM(Module):
return action_sig, extract_sig
@staticmethod
- def _format_interpreter_rules(execution_instructions: str) -> str:
- return f"\nExecution environment:\n{execution_instructions}\n" if execution_instructions else ""
+ def _format_interpreter_rules(execution_instructions: str, sub_agent_rules: str) -> str:
+ rules = f"\nExecution environment:\n{execution_instructions}\n" if execution_instructions else ""
+ return rules + sub_agent_rules
@staticmethod
def _get_execution_instructions(factory: Callable[[], CodeInterpreter]) -> str:
@@ -423,8 +455,13 @@ class RLM(Module):
raise TypeError("interpreter_factory.execution_instructions must be a string")
return execution_instructions
- def _action_signature_for_current_factory(self, factory: Callable[[], CodeInterpreter]) -> type[Signature]:
- """Return an action signature whose runtime guidance matches the active interpreter."""
+ def _action_signature_for_current_factory(
+ self, factory: Callable[[], CodeInterpreter], sub_agent_rules: str
+ ) -> type[Signature]:
+ """Return an action signature whose runtime guidance matches the active interpreter.
+
+ ``sub_agent_rules`` is the sub-agent guidance for this invocation's interpreter (see ``_setup_facade``).
+ """
execution_instructions = self._get_execution_instructions(factory)
# getattr, because generate_action may have been replaced by a predictor that
# carries no signature of its own.
@@ -432,12 +469,15 @@ class RLM(Module):
# Same runtime as at construction: no derived signature per iteration, and an
# optimizer's revised instructions stay untouched.
- if execution_instructions == self._initial_execution_instructions:
+ if (
+ execution_instructions == self._initial_execution_instructions
+ and sub_agent_rules == self._initial_sub_agent_rules
+ ):
return current_signature
# Replace only DSPy's own fragment, so an override cannot stack on stale guidance.
current_instructions = current_signature.instructions
- current_rules = self._format_interpreter_rules(execution_instructions)
+ current_rules = self._format_interpreter_rules(execution_instructions, sub_agent_rules)
if self._initial_interpreter_rules and self._initial_interpreter_rules in current_instructions:
instructions = current_instructions.replace(self._initial_interpreter_rules, current_rules, 1)
elif current_rules:
@@ -534,6 +574,28 @@ class RLM(Module):
return regular_args
+ def _setup_facade(
+ self, repl: CodeInterpreter, budget: _LLMCallBudget, interpreter_factory: Callable[[], CodeInterpreter]
+ ) -> str:
+ """Install the sandbox dspy facade into this invocation's interpreter; return its sub-agent guidance.
+
+ Nested code-executing sub-agents get their interpreters from ``interpreter_factory``, the factory this
+ invocation runs on. The facade resolves ``sub_lm`` (else ``dspy.settings.lm``) per call, like
+ ``llm_query``, and charges ``max_llm_calls``.
+ """
+ invocation = FacadeInvocation(
+ self._user_tools, interpreter_factory, None, lm=self.sub_lm, reserve=budget.reserve
+ )
+ try:
+ invocation.install(repl)
+ except CodeExecutionError as e:
+ # The runtime cannot host the shim (e.g. it runs code in the host's memory): no sub-agents this call.
+ for name in (CONSTRUCT_TOOL, CALL_TOOL):
+ repl.tools.pop(name, None)
+ logger.warning("RLM sub-agents are unavailable on %s: %s", type(repl).__name__, e)
+ return ""
+ return SUB_AGENT_INSTRUCTIONS
+
# =========================================================================
# CodeInterpreter Lifecycle
# =========================================================================
@@ -551,9 +613,10 @@ class RLM(Module):
invoke.__signature__ = inspect.signature(tool.func)
return invoke
- def _prepare_execution_tools(self) -> dict[str, Callable]:
- """Create fresh LLM tools and merge with user-provided tools."""
- execution_tools = self._make_llm_tools()
+ def _prepare_execution_tools(self, budget: _LLMCallBudget | None = None) -> dict[str, Callable]:
+ """Create the LLM tools on ``budget`` (fresh by default) and merge with user-provided tools."""
+ budget = budget if budget is not None else _LLMCallBudget(self.max_llm_calls)
+ execution_tools = self._make_llm_tools(budget)
execution_tools.update({name: self._make_interpreter_tool(tool) for name, tool in self._user_tools.items()})
return execution_tools
@@ -723,13 +786,14 @@ class RLM(Module):
input_args: dict[str, Any],
output_field_names: list[str],
interpreter_factory: Callable[[], CodeInterpreter],
+ sub_agent_rules: str,
) -> Prediction | REPLHistory:
"""Execute one iteration. Returns Prediction if done, else updated REPLHistory."""
variables_info = [variable.format() for variable in variables]
# A per-call signature, not a mutation of generate_action.signature, keeps a
# dspy.context override local to this invocation.
action = self.generate_action(
- signature=self._action_signature_for_current_factory(interpreter_factory),
+ signature=self._action_signature_for_current_factory(interpreter_factory, sub_agent_rules),
variables_info=variables_info,
repl_history=history,
iteration=f"{iteration + 1}/{self.max_iters}",
@@ -759,7 +823,8 @@ class RLM(Module):
Args:
interpreter_factory: Optional zero-argument factory, passed by keyword. Overrides the constructor
and configured factories for this invocation. Must return a fresh interpreter; RLM injects tools
- and output metadata and shuts it down on exit, including failures.
+ and output metadata and shuts it down on exit, including failures. Sub-agents built in the
+ sandbox run their own code on this factory too.
**input_args: Input values matching the signature's input fields.
Returns:
@@ -774,16 +839,19 @@ class RLM(Module):
interpreter_factory = resolve_interpreter_factory(self._interpreter_factory)
output_field_names = list(self.signature.output_fields.keys())
- execution_tools = self._prepare_execution_tools()
+ budget = _LLMCallBudget(self.max_llm_calls)
+ execution_tools = self._prepare_execution_tools(budget)
variables = self._build_variables(**input_args)
with self._interpreter_context(execution_tools, interpreter_factory) as repl:
+ sub_agent_rules = self._setup_facade(repl, budget, interpreter_factory)
regular_args = self._prepare_serializable_vars(input_args, repl)
history: REPLHistory = REPLHistory(max_output_chars=self.max_output_chars)
for iteration in range(self.max_iters):
result: Prediction | REPLHistory = self._execute_iteration(
- repl, variables, history, iteration, regular_args, output_field_names, interpreter_factory
+ repl, variables, history, iteration, regular_args, output_field_names, interpreter_factory,
+ sub_agent_rules,
)
if isinstance(result, Prediction):
return result
@@ -822,11 +890,12 @@ class RLM(Module):
input_args: dict[str, Any],
output_field_names: list[str],
interpreter_factory: Callable[[], CodeInterpreter],
+ sub_agent_rules: str,
) -> Prediction | REPLHistory:
"""Async version: Execute one iteration."""
variables_info = [variable.format() for variable in variables]
pred = await self.generate_action.acall(
- signature=self._action_signature_for_current_factory(interpreter_factory),
+ signature=self._action_signature_for_current_factory(interpreter_factory, sub_agent_rules),
variables_info=variables_info,
repl_history=history,
iteration=f"{iteration + 1}/{self.max_iters}",
@@ -852,7 +921,8 @@ class RLM(Module):
Args:
interpreter_factory: Optional zero-argument factory, passed by keyword. Overrides the constructor
and configured factories for this invocation. Must return a fresh interpreter; RLM injects tools
- and output metadata and shuts it down on exit, including failures.
+ and output metadata and shuts it down on exit, including failures. Sub-agents built in the
+ sandbox run their own code on this factory too.
**input_args: Input values matching the signature's input fields.
Returns:
@@ -867,16 +937,19 @@ class RLM(Module):
interpreter_factory = resolve_interpreter_factory(self._interpreter_factory)
output_field_names = list(self.signature.output_fields.keys())
- execution_tools = self._prepare_execution_tools()
+ budget = _LLMCallBudget(self.max_llm_calls)
+ execution_tools = self._prepare_execution_tools(budget)
variables = self._build_variables(**input_args)
with self._interpreter_context(execution_tools, interpreter_factory) as repl:
+ sub_agent_rules = self._setup_facade(repl, budget, interpreter_factory)
regular_args = self._prepare_serializable_vars(input_args, repl)
history = REPLHistory(max_output_chars=self.max_output_chars)
for iteration in range(self.max_iters):
result = await self._aexecute_iteration(
- repl, variables, history, iteration, regular_args, output_field_names, interpreter_factory
+ repl, variables, history, iteration, regular_args, output_field_names, interpreter_factory,
+ sub_agent_rules,
)
if isinstance(result, Prediction):
return result
diff --git a/dspy/predict/flex/_sandbox_shim.py b/dspy/primitives/_facade_shim.py
similarity index 65%
rename from dspy/predict/flex/_sandbox_shim.py
rename to dspy/primitives/_facade_shim.py
index f25631dff..f6417a1de 100644
--- a/dspy/predict/flex/_sandbox_shim.py
+++ b/dspy/primitives/_facade_shim.py
@@ -1,7 +1,7 @@
-"""Sandbox side of the dspy.Flex bridge: a stand-in ``dspy`` module for optimizer-authored code.
+"""Sandbox side of the dspy facade: a stand-in ``dspy`` module for sandboxed code.
-``BridgeRuntime.forward`` executes this source in each per-forward interpreter before the bound
-``module_src``, so the generated module runs sandboxed while predictors are built and called on
+Consumers such as ``dspy.Flex`` and ``dspy.RLM`` execute this source in the interpreter after
+registering the host tools, so sandboxed code builds and calls predictors that actually run on
the host:
- ``dspy.Predict`` and other allowed modules return a ``_DspyPending``; construction waits for attribute assignment,
@@ -13,8 +13,8 @@ the host:
markers the host resolves: tools by name (``__dspy_tool__``), ``dspy.Signature(...)`` results
as ``__dspy_sig__`` payloads.
-Every name here is ``_dspy``-prefixed to stay clear of generated code; ``FlexContext`` rejects
-user tool names in that namespace.
+Every name here is ``_dspy``-prefixed to stay clear of sandboxed code; consumers reject user tool
+names in that namespace.
"""
import sys as _dspy_sys
@@ -57,6 +57,9 @@ class _DspyProxy:
return _DspyPrediction(**(_out or {}))
+_dspy_anon_count = 0
+
+
class _DspyPending:
"""Returned by a shim constructor before the attribute name is known (captured in __setattr__)."""
@@ -64,6 +67,21 @@ class _DspyPending:
self.kind = _kind
self.sig = _sig
self.kwargs = _kwargs
+ self.proxy = None
+
+ def __call__(self, **_inputs):
+ if self.proxy is None:
+ global _dspy_anon_count
+ _dspy_anon_count += 1
+ _handle = _dspy_host(
+ "__dspy_construct__",
+ kind=self.kind,
+ signature=self.sig,
+ attr_name="_dspy_anon_" + str(_dspy_anon_count),
+ kwargs=self.kwargs,
+ )
+ self.proxy = _DspyProxy(_handle)
+ return self.proxy(**_inputs)
class _DspyModule:
@@ -86,10 +104,19 @@ class _DspyModule:
return self.forward(**_kw)
+def _dspy_tool_name(_v):
+ # Host tools are referenced by the sandbox global they are bound to: backends may bind them as
+ # anonymous proxies, so the callable's own __name__ is only a fallback.
+ for _k, _x in globals().items():
+ if _x is _v and not _k.startswith("_"):
+ return _k
+ return getattr(_v, "__name__", type(_v).__name__)
+
+
def _dspy_enc(_v):
- # Tool references (e.g. tools=[shout]) are sandbox functions; send them to the host by name.
- if callable(_v) and hasattr(_v, "__name__"):
- return {"__dspy_tool__": _v.__name__}
+ # Tool references (e.g. tools=[shout]) are sandbox callables; send them to the host by name.
+ if callable(_v):
+ return {"__dspy_tool__": _dspy_tool_name(_v)}
if isinstance(_v, (list, tuple)):
return [_dspy_enc(_x) for _x in _v]
if isinstance(_v, dict):
@@ -122,6 +149,13 @@ for _k in ("Predict", "ChainOfThought", "RLM", "CodeAct", "ProgramOfThought", "R
dspy = _dspy
# Register as the importable ``dspy`` only inside the sandbox, where the registered host tools are
-# present in globals().
+# present in globals(). A sandbox whose ``dspy`` is the host's own module image (the host process or a
+# fork of it) would hand generated code the host's memory, so the facade refuses it.
if "__dspy_construct__" in globals():
+ _dspy_host_facade = getattr(_dspy_sys, "modules", {}).get("dspy.primitives.facade")
+ if getattr(_dspy_host_facade, "_HOST_PROCESS_TOKEN", None) == __dspy_host_token: # noqa: F821 - host-injected
+ raise RuntimeError(
+ "This interpreter runs code in the host's memory (the host process or a fork of it); "
+ "the dspy facade needs an isolated interpreter."
+ )
_dspy_sys.modules["dspy"] = _dspy
diff --git a/dspy/primitives/code_interpreter.py b/dspy/primitives/code_interpreter.py
index 827c9627b..8b06e453a 100644
--- a/dspy/primitives/code_interpreter.py
+++ b/dspy/primitives/code_interpreter.py
@@ -80,6 +80,12 @@ class CodeInterpreter(Protocol):
Pooling:
For interpreter pooling, call start() to pre-warm instances, then
distribute execute() calls across the pool.
+
+ Sandbox dspy facade:
+ dspy.Flex and dspy.RLM install a ``dspy`` shim whose predictors are built and
+ run on the host, so registered tools must be callable as globals in executed
+ code and state must persist across execute() calls. The shim refuses an
+ interpreter that runs code in the host's memory (the host process or a fork).
"""
@property
diff --git a/dspy/primitives/facade.py b/dspy/primitives/facade.py
new file mode 100644
index 000000000..d24c20fcf
--- /dev/null
+++ b/dspy/primitives/facade.py
@@ -0,0 +1,299 @@
+"""Host side of the sandbox dspy facade.
+
+Sandboxed code (a ``dspy.Flex`` module or a ``dspy.RLM`` REPL block) gets a stand-in ``dspy``
+module, ``_facade_shim.py``, whose predictor constructors and calls are proxies. Only JSON crosses
+the sandbox boundary, over the interpreter's tool-call protocol (``CodeInterpreter.tools``):
+
+- ``__dspy_construct__``: the shim asks the host to build a real predictor
+ (``FacadeInvocation.construct``) from a serialized signature/kwargs and gets a string handle.
+- ``__dspy_call__``: the shim runs a predictor by handle (``FacadeInvocation.call``); the host
+ makes the real LM call and returns the prediction's fields as JSON.
+
+Callables and signatures travel as markers the host resolves: tools by name (``__dspy_tool__``)
+against the host module's tools, ``dspy.Signature(...)`` results as ``__dspy_sig__`` payloads.
+All per-forward state (constructed predictors, the predictor-call budget, custom-type originals,
+the last LM infrastructure error) lives in a ``FacadeInvocation`` created for that forward.
+
+Every LM call made for sandboxed code runs on a ``SandboxLM`` over the host LM, which admits only
+generation parameters (``SANDBOX_LM_KWARGS``) and meters the calls.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import inspect
+import json
+import secrets
+from pathlib import Path
+from typing import Any, Callable
+
+from pydantic_core import PydanticSerializationError, to_jsonable_python
+
+import dspy
+from dspy.clients.base_lm import BaseLM
+from dspy.primitives.code_interpreter import CodeInterpreterError
+from dspy.signatures.signature import make_signature
+from dspy.utils.exceptions import LMError
+
+# Tool names the shim uses to call the host; registered by FacadeInvocation.install.
+CONSTRUCT_TOOL = "__dspy_construct__"
+CALL_TOOL = "__dspy_call__"
+
+# Predictors the sandbox shim may construct and the host builds
+BRIDGEABLE_KINDS = ("Predict", "ChainOfThought", "RLM", "CodeAct", "ProgramOfThought", "ReAct", "ReActV2")
+# The shim's dspy.Signature(...) emits this marker so the host can rebuild a Signature.
+SIGNATURE_MARKER = "__dspy_sig__"
+# The shim passes tools by name (callables can't cross the JSON boundary); the host resolves the name
+# against the host module's tools.
+TOOL_MARKER = "__dspy_tool__"
+
+# The sandbox-side dspy shim, injected as text into each interpreter.
+SHIM_SETUP = (Path(__file__).parent / "_facade_shim.py").read_text(encoding="utf-8")
+# Shipped to the shim so it can refuse a sandbox that is the host's own memory image (host process or fork).
+HOST_TOKEN_VAR = "__dspy_host_token"
+_HOST_PROCESS_TOKEN = secrets.token_hex(16)
+
+
+def is_reserved_sandbox_name(name: str) -> bool:
+ """Names the facade owns in the sandbox namespace: ``dspy`` and the ``_dspy``/``__dspy`` prefixes
+ of its internals and host-injected variables."""
+ return name == "dspy" or name.startswith(("_dspy", "__dspy"))
+
+
+def _accepts_interpreter_factory(cls: type) -> bool:
+ """True if ``cls.__init__`` takes an ``interpreter_factory`` parameter."""
+ try:
+ return "interpreter_factory" in inspect.signature(cls.__init__).parameters
+ except (TypeError, ValueError):
+ return False
+
+
+def _resolve_signature(signature: Any, custom_types: dict[str, type] | None = None) -> Any:
+ """Turn a shim signature payload back into something a host predictor accepts."""
+ if isinstance(signature, dict) and signature.get(SIGNATURE_MARKER):
+ # marker always carries a string signature; make_signature applies instructions if given
+ return make_signature(signature["signature"], signature.get("instructions"), custom_types=custom_types)
+ if isinstance(signature, str):
+ return make_signature(signature, custom_types=custom_types)
+ return signature
+
+
+def _jsonable(value: Any) -> Any:
+ """Coerce a predictor output field to a JSON-serializable value."""
+ try:
+ return to_jsonable_python(value)
+ except PydanticSerializationError:
+ return value
+
+
+def prediction_to_fields(pred: Any) -> dict[str, Any]:
+ """Serialise a host ``dspy.Prediction`` (or dict) to a JSON-able field dict for the sandbox."""
+ store = getattr(pred, "_store", None)
+ if store is None:
+ if isinstance(pred, dict):
+ store = pred
+ else:
+ raise CodeInterpreterError(
+ f"A bridged predictor must return a dspy.Prediction; got {type(pred).__name__}"
+ )
+ fields = {k: _jsonable(v) for k, v in dict(store).items()}
+ try:
+ json.dumps(fields)
+ except TypeError as e:
+ raise CodeInterpreterError(
+ "A bridged predictor returned a field that cannot cross the sandbox boundary "
+ f"(must be JSON-serializable): {e}"
+ ) from e
+ return fields
+
+
+def restore_custom_types(value: Any, originals: dict[str, Any]) -> Any:
+ """Substitute serialized custom-type strings back with the original host objects."""
+ if isinstance(value, str) and value in originals:
+ return originals[value]
+ if isinstance(value, dict):
+ return {k: restore_custom_types(v, originals) for k, v in value.items()}
+ if isinstance(value, list):
+ return [restore_custom_types(v, originals) for v in value]
+ return value
+
+
+# Generation parameters sandboxed code may set per LM call; routing, credential, and transport options are not.
+SANDBOX_LM_KWARGS = frozenset({
+ "temperature", "top_p", "max_tokens", "max_completion_tokens", "stop", "n", "seed",
+ "logprobs", "top_logprobs", "response_format",
+ "reasoning", "reasoning_effort",
+ "tools", "tool_choice", "parallel_tool_calls",
+ "cache", "rollout_id", "prompt_cache", "prompt_cache_key",
+})
+_LM_CALL_INPUTS = frozenset({"prompt", "messages", "request"})
+
+
+class SandboxLM(BaseLM):
+ """The host LM as sandboxed code may call it: only ``SANDBOX_LM_KWARGS`` per call, and ``reserve(n)`` before
+ each, one slot per requested completion."""
+
+ def __init__(self, lm: Any, reserve: Callable[[int], None] | None = None) -> None:
+ # lm may be any callable RLM accepts as sub_lm, not only a BaseLM. Adapters and Predict read the
+ # generation defaults off the proxy; credentials and routing stay on the wrapped LM.
+ super().__init__(model=getattr(lm, "model", None), model_type=getattr(lm, "model_type", "chat"))
+ self.kwargs = {k: v for k, v in (getattr(lm, "kwargs", None) or {}).items() if k in SANDBOX_LM_KWARGS}
+ self._lm = lm
+ self._reserve = reserve
+
+ @staticmethod
+ def _check(kwargs: dict[str, Any]) -> None:
+ rejected = sorted(kwargs.keys() - _LM_CALL_INPUTS - SANDBOX_LM_KWARGS)
+ if rejected:
+ raise TypeError(
+ f"Sandboxed code may not set LM option(s) {rejected}; allowed per call: {sorted(SANDBOX_LM_KWARGS)}."
+ )
+
+ def _admit(self, kwargs: dict[str, Any]) -> None:
+ self._check(kwargs)
+ if self._reserve is not None:
+ # `n` completions cost `n` calls, so it cannot multiply the budget.
+ self._reserve(max(1, int(kwargs.get("n", self.kwargs.get("n")) or 1)))
+
+ def __call__(self, *args: Any, **kwargs: Any) -> Any:
+ self._admit(kwargs)
+ return self._lm(*args, **kwargs)
+
+ async def acall(self, *args: Any, **kwargs: Any) -> Any:
+ self._admit(kwargs)
+ return await self._lm.acall(*args, **kwargs)
+
+ def forward(self, *args: Any, **kwargs: Any) -> Any:
+ self._admit(kwargs)
+ return self._lm.forward(*args, **kwargs)
+
+ async def aforward(self, *args: Any, **kwargs: Any) -> Any:
+ self._admit(kwargs)
+ return await self._lm.aforward(*args, **kwargs)
+
+ def copy(self, **kwargs: Any) -> SandboxLM:
+ self._check(kwargs)
+ return SandboxLM(self._lm.copy(**kwargs), self._reserve)
+
+ # Adapters choose structured outputs and native tool calling by these; answer for the wrapped model.
+ @property
+ def supports_function_calling(self) -> bool:
+ return getattr(self._lm, "supports_function_calling", False)
+
+ @property
+ def supports_reasoning(self) -> bool:
+ return getattr(self._lm, "supports_reasoning", False)
+
+ @property
+ def supports_response_schema(self) -> bool:
+ return getattr(self._lm, "supports_response_schema", False)
+
+ @property
+ def supported_params(self) -> set[str]:
+ return getattr(self._lm, "supported_params", set())
+
+
+class FacadeInvocation:
+ """Per-forward host side of the sandbox dspy facade.
+
+ Builds the predictors the shim asks for (keyed by the sandbox attribute name) from the host
+ module's tools and interpreter factory, runs them by handle under a predictor-call budget on a
+ ``SandboxLM``, and restores custom-type inputs. Each forward gets its own ``FacadeInvocation``.
+ ``lm`` defaults to ``dspy.settings.lm`` at call time; ``reserve`` meters every LM call.
+ """
+
+ def __init__(
+ self,
+ tools: dict[str, Any],
+ interpreter_factory: Callable[[], Any],
+ max_predictor_calls: int | None,
+ *,
+ custom_types: dict[str, type] | None = None,
+ lm: Any = None,
+ reserve: Callable[[int], None] | None = None,
+ originals: dict[str, Any] | None = None,
+ ) -> None:
+ self._tools = tools
+ self._interpreter_factory = interpreter_factory
+ self._max_predictor_calls = max_predictor_calls
+ self._custom_types = custom_types
+ self._lm = lm
+ self._reserve = reserve
+ self._originals = originals or {}
+ self._predictors: dict[str, Any] = {}
+ self._calls = 0
+ self._lm_error: tuple[LMError, str] | None = None
+
+ def install(self, interpreter: Any) -> None:
+ """Register the tools the shim calls, then install the ``dspy`` facade in the sandbox."""
+ interpreter.tools.update({CONSTRUCT_TOOL: self.construct, CALL_TOOL: self.call})
+ interpreter.execute(SHIM_SETUP, variables={HOST_TOKEN_VAR: _HOST_PROCESS_TOKEN})
+
+ def construct(self, kind: str, signature: Any, attr_name: str, kwargs: dict[str, Any] | None = None) -> str:
+ self._lm_error = None
+ if kind not in BRIDGEABLE_KINDS:
+ raise CodeInterpreterError(
+ f"dspy.{kind} is not supported through the sandbox dspy bridge yet "
+ f"(bridgeable: {', '.join(BRIDGEABLE_KINDS)})"
+ )
+ self._predictors[attr_name] = self._build_predictor(kind, signature, kwargs)
+ return attr_name
+
+ def call(self, handle: str, inputs: dict[str, Any] | None = None) -> dict[str, Any]:
+ self._lm_error = None
+ self._calls += 1
+ budget = self._max_predictor_calls
+ if budget is not None and self._calls > budget:
+ raise CodeInterpreterError(
+ f"Sandboxed forward exceeded its predictor-call budget ({budget}). "
+ "Raise max_predictor_calls if this is expected."
+ )
+ predictor = self._predictors.get(handle)
+ if predictor is None:
+ raise CodeInterpreterError(f"Unknown predictor handle: {handle!r}")
+ restored = {k: restore_custom_types(v, self._originals) for k, v in (inputs or {}).items()}
+ try:
+ with self._lm_scope():
+ return prediction_to_fields(predictor(**restored))
+ except LMError as e:
+ tag = f"[dspy bridge lm-error #{self._calls}]"
+ self._lm_error = (e, tag)
+ raise CodeInterpreterError(f"{tag} {type(e).__name__}: {e}") from e
+
+ def _lm_scope(self) -> contextlib.AbstractContextManager[Any]:
+ """dspy.context(lm=SandboxLM) for one bridged call; a scoped override, so nested predictors use it too."""
+ lm = self._lm if self._lm is not None else dspy.settings.lm
+ if lm is None:
+ return contextlib.nullcontext()
+ if isinstance(lm, SandboxLM) and self._reserve is None:
+ return dspy.context(lm=lm)
+ return dspy.context(lm=SandboxLM(lm, self._reserve))
+
+ def _build_predictor(self, kind: str, signature: Any, kwargs: dict[str, Any] | None) -> Any:
+ cls = getattr(dspy, kind)
+ if "interpreter_factory" in (kwargs or {}):
+ raise CodeInterpreterError(
+ f"dspy.{kind} built in the sandbox runs on the host module's interpreter; "
+ "sandboxed code cannot choose its interpreter_factory."
+ )
+ extra = {k: self._decode_tools(v) for k, v in (kwargs or {}).items()}
+ # A code-executing sub-predictor always runs its inner code on the host module's backend.
+ if _accepts_interpreter_factory(cls):
+ extra["interpreter_factory"] = self._interpreter_factory
+ return cls(_resolve_signature(signature, self._custom_types), **extra)
+
+ def _decode_tools(self, value: Any) -> Any:
+ """Turn the shim's tool name-markers back into the real host tool objects."""
+ if isinstance(value, dict) and TOOL_MARKER in value:
+ name = value[TOOL_MARKER]
+ if name not in self._tools:
+ raise CodeInterpreterError(
+ f"Sandboxed code referenced tool {name!r}, which was not provided to the host module's "
+ "tools. Functions authored inside the sandbox cannot be handed to a bridged sub-predictor."
+ )
+ return self._tools[name]
+ if isinstance(value, list):
+ return [self._decode_tools(v) for v in value]
+ if isinstance(value, dict):
+ return {k: self._decode_tools(v) for k, v in value.items()}
+ return value
diff --git a/pyproject.toml b/pyproject.toml
index fd920dc36..a49e298f6 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -72,6 +72,11 @@ test_extras = [
"numpy>=1.26.0",
]
+[dependency-groups]
+dev = [
+ "dspy-interpreters @ git+https://github.com/cmpnd-ai/dspy-interpreters",
+]
+
[tool.setuptools.packages.find]
where = ["."]
include = ["dspy", "dspy.*"]
diff --git a/tests/conftest.py b/tests/conftest.py
index 5c005dc61..e8c68db93 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -5,6 +5,11 @@ from typing import Any
import pytest
+# Read LiteLLM's bundled model catalog instead of fetching its live copy from GitHub, so capability
+# lookups do not change under the suite when upstream edits or retires an entry. Set before anything
+# imports litellm; tests of the remote path unset it with monkeypatch.
+os.environ.setdefault("LITELLM_LOCAL_MODEL_COST_MAP", "True")
+
from tests.test_utils.server import ( # noqa: F401
_litellm_test_server,
litellm_test_server,
diff --git a/tests/flex/test_flex_interpreter.py b/tests/flex/test_flex_interpreter.py
index 76da1e92c..86fb4a7e9 100644
--- a/tests/flex/test_flex_interpreter.py
+++ b/tests/flex/test_flex_interpreter.py
@@ -26,6 +26,7 @@ import pytest
import dspy
from dspy.predict.flex import Flex, bridge
+from dspy.primitives import facade
from dspy.primitives.code_interpreter import (
CodeExecutionError,
CodeInterpreterError,
@@ -125,7 +126,7 @@ def _bridged_flex_with(**kwargs):
def test_construct_builds_real_predictor_in_per_forward_registry() -> None:
flex = _bridged_flex()
- inv = bridge._Invocation(flex._bridge, {})
+ inv = flex._bridge.invocation()
handle = inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})
assert handle == "solve"
assert isinstance(inv._predictors["solve"], dspy.ChainOfThought)
@@ -139,7 +140,7 @@ def test_construct_names_cannot_clobber_the_flex() -> None:
# Construction writes to the per-forward registry, never setattr on the host Flex, so a
# generated predictor named like a Flex attribute is just a handle — nothing to corrupt.
flex = _bridged_flex()
- inv = bridge._Invocation(flex._bridge, {})
+ inv = flex._bridge.invocation()
for name in ("lm", "forward", "module_src", "_bridge"):
assert inv.construct("Predict", "value: int -> result: int", name, {}) == name
assert flex.lm is None # host attributes untouched
@@ -153,8 +154,8 @@ def test_parallel_forwards_cannot_overwrite_each_others_predictors() -> None:
# attached to the shared Flex, A would run B's predictor; each invocation owning its own
# registry keeps them isolated.
flex = _bridged_flex()
- a = bridge._Invocation(flex._bridge, {})
- b = bridge._Invocation(flex._bridge, {})
+ a = flex._bridge.invocation()
+ b = flex._bridge.invocation()
a.construct("Predict", "value: int -> result: int", "solve", {})
b.construct("ChainOfThought", "question: str -> answer: str", "solve", {}) # same handle
assert a._predictors["solve"] is not b._predictors["solve"]
@@ -166,10 +167,10 @@ def test_parallel_forwards_cannot_overwrite_each_others_predictors() -> None:
def test_react_kinds_are_bridgeable_and_construct_on_host() -> None:
# ReAct/ReActV2 are tool-calling agents: they run NO inner code (so take no interpreter) and call
# named tools on the host. They just need whitelisting; the tools must be Flex-level to resolve.
- assert "ReAct" in bridge.BRIDGEABLE_KINDS and "ReActV2" in bridge.BRIDGEABLE_KINDS
+ assert "ReAct" in facade.BRIDGEABLE_KINDS and "ReActV2" in facade.BRIDGEABLE_KINDS
flex = Flex(ShoutSig, tools=[shout], interpreter_factory=lambda: MockInterpreter())
- inv = bridge._Invocation(flex._bridge, {})
- tools_payload = [{bridge.TOOL_MARKER: "shout"}]
+ inv = flex._bridge.invocation()
+ tools_payload = [{facade.TOOL_MARKER: "shout"}]
inv.construct("ReAct", "text: str -> out: str", "agent", {"tools": tools_payload})
agent = inv._predictors["agent"]
assert isinstance(agent, dspy.ReAct)
@@ -187,8 +188,8 @@ def test_reactv2_constructs_and_runs_through_the_bridge() -> None:
return f"found {query}"
flex = Flex(ShoutSig, tools=[lookup], interpreter_factory=lambda: MockInterpreter())
- inv = bridge._Invocation(flex._bridge, {})
- inv.construct("ReActV2", "question: str -> answer: str", "agent", {"tools": [{bridge.TOOL_MARKER: "lookup"}]})
+ inv = flex._bridge.invocation()
+ inv.construct("ReActV2", "question: str -> answer: str", "agent", {"tools": [{facade.TOOL_MARKER: "lookup"}]})
agent = inv._predictors["agent"]
assert isinstance(agent, dspy.ReActV2)
assert not hasattr(agent, "interpreter") # not a code-executing predictor
@@ -214,9 +215,25 @@ def test_reactv2_constructs_and_runs_through_the_bridge() -> None:
assert tool_sequence == ["lookup", "submit"]
+def test_bridged_rlm_inherits_the_flex_factory() -> None:
+ # A bridged sub-RLM runs on the Flex factory, and sandboxed code cannot pick another.
+ def factory() -> MockInterpreter:
+ return MockInterpreter()
+
+ flex = Flex(Doubler, interpreter_factory=factory)
+ inv = flex._bridge.invocation()
+ inv.construct("RLM", "value: int -> result: int", "rlm", {})
+
+ sub_rlm = inv._predictors["rlm"]
+ assert sub_rlm._interpreter_factory is factory
+ assert "Sub-agents (dspy)" in sub_rlm.generate_action.signature.instructions
+ with pytest.raises(CodeInterpreterError, match="cannot choose its interpreter_factory"):
+ inv.construct("RLM", "value: int -> result: int", "chosen", {"interpreter_factory": "LocalInterpreter"})
+
+
def test_call_runs_predictor_via_host_lm() -> None:
flex = _bridged_flex()
- inv = bridge._Invocation(flex._bridge, {})
+ inv = flex._bridge.invocation()
inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})
dspy.configure(lm=DummyLM([{"reasoning": "two doubled is four", "result": "4"}]))
fields = inv.call("solve", {"value": 2})
@@ -224,23 +241,34 @@ def test_call_runs_predictor_via_host_lm() -> None:
assert "reasoning" in fields
+def test_bridged_predictors_cannot_set_lm_routing_or_credentials() -> None:
+ flex = _bridged_flex()
+ inv = flex._bridge.invocation()
+ inv.construct("Predict", "value: int -> result: int", "solve", {"api_base": "http://evil.example", "api_key": "sk-x"})
+ dspy.configure(lm=DummyLM([{"result": "4"}]))
+ with pytest.raises(TypeError, match=r"may not set LM option\(s\) \['api_base', 'api_key'\]"):
+ inv.call("solve", {"value": 2})
+ inv.construct("Predict", "value: int -> result: int", "tuned", {"temperature": 0.7})
+ assert inv.call("tuned", {"value": 2}) == {"result": 4}
+
+
def test_call_unknown_handle_raises() -> None:
flex = _bridged_flex()
with pytest.raises(CodeInterpreterError):
- bridge._Invocation(flex._bridge, {}).call("nope", {"value": 1})
+ flex._bridge.invocation().call("nope", {"value": 1})
def test_tool_name_markers_resolve_to_real_tools() -> None:
# The shim passes tools to a bridged sub-predictor by name; the host resolves them back.
flex = Flex(Doubler, tools=[shout], interpreter_factory=lambda: MockInterpreter())
- assert flex._bridge._decode_tools([{"__dspy_tool__": "shout"}]) == [shout]
+ assert flex._bridge.invocation()._decode_tools([{"__dspy_tool__": "shout"}]) == [shout]
with pytest.raises(CodeInterpreterError):
- flex._bridge._decode_tools({"__dspy_tool__": "not_a_tool"}) # module-authored / unknown tool
+ flex._bridge.invocation()._decode_tools({"__dspy_tool__": "not_a_tool"}) # module-authored / unknown tool
def test_interpreter_is_not_serialized() -> None:
flex = _bridged_flex()
- inv = bridge._Invocation(flex._bridge, {})
+ inv = flex._bridge.invocation()
inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})
state = flex.dump_state()
assert state.get("module_src") # the code persists
@@ -267,9 +295,9 @@ def test_parse_module_class_name_prefers_forward() -> None:
def test_prediction_to_fields_rejects_non_jsonable() -> None:
- assert bridge.prediction_to_fields(dspy.Prediction(a=1, b="x")) == {"a": 1, "b": "x"}
+ assert facade.prediction_to_fields(dspy.Prediction(a=1, b="x")) == {"a": 1, "b": "x"}
with pytest.raises(CodeInterpreterError):
- bridge.prediction_to_fields(dspy.Prediction(a=object()))
+ facade.prediction_to_fields(dspy.Prediction(a=object()))
def test_prediction_to_fields_serializes_json_mode_values() -> None:
@@ -283,7 +311,7 @@ def test_prediction_to_fields_serializes_json_mode_values() -> None:
when: datetime.datetime
color: Color
- fields = bridge.prediction_to_fields(
+ fields = facade.prediction_to_fields(
dspy.Prediction(
event=Event(when=datetime.datetime(2026, 1, 1), color=Color.RED),
when=datetime.datetime(2026, 1, 2),
@@ -303,22 +331,22 @@ def test_prediction_to_fields_serializes_json_mode_values() -> None:
def test_shim_reaches_host_only_via_registered_tools() -> None:
# The bridge must not depend on Deno/Pyodide internals — only on the CodeInterpreter.tools
# contract — so any backend (a local microVM, gVisor, ...) can drive it.
- shim = bridge.SHIM_SETUP
+ shim = facade.SHIM_SETUP
assert "pyodide" not in shim.lower()
assert "_js_tool_call" not in shim
assert "run_sync" not in shim
# It does call the registered host tools by name.
- assert bridge.CONSTRUCT_TOOL in shim
- assert bridge.CALL_TOOL in shim
+ assert facade.CONSTRUCT_TOOL in shim
+ assert facade.CALL_TOOL in shim
def test_shim_file_literals_match_bridge_constants() -> None:
# The shim lives in a sibling .py file with literal protocol strings (no interpolation); guard
# against drift between those literals and bridge.py's host-side constants.
- shim = bridge.SHIM_SETUP
- for token in (bridge.CONSTRUCT_TOOL, bridge.CALL_TOOL, bridge.TOOL_MARKER, bridge.SIGNATURE_MARKER):
+ shim = facade.SHIM_SETUP
+ for token in (facade.CONSTRUCT_TOOL, facade.CALL_TOOL, facade.TOOL_MARKER, facade.SIGNATURE_MARKER):
assert token in shim, f"{token!r} missing from the shim file"
- for kind in bridge.BRIDGEABLE_KINDS:
+ for kind in facade.BRIDGEABLE_KINDS:
assert kind in shim, f"bridgeable kind {kind!r} missing from the shim file"
@@ -331,14 +359,14 @@ def test_predictor_call_budget_is_enforced() -> None:
# The budget lives in the per-forward _Invocation, so every forward starts fresh.
flex = _bridged_flex_with(max_predictor_calls=2)
dspy.configure(lm=DummyLM([{"reasoning": "r", "result": str(i)} for i in range(3)]))
- inv = bridge._Invocation(flex._bridge, {})
+ inv = flex._bridge.invocation()
inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})
inv.call("solve", {"value": 1}) # 1st
inv.call("solve", {"value": 1}) # 2nd
- with pytest.raises(CodeInterpreterError, match="budget"):
+ with pytest.raises(CodeInterpreterError, match=r"budget.*Raise max_predictor_calls if this is expected"):
inv.call("solve", {"value": 1}) # 3rd exceeds the cap
# A new forward gets a new budget — and its own registry, so it constructs anew.
- fresh = bridge._Invocation(flex._bridge, {})
+ fresh = flex._bridge.invocation()
fresh.construct("ChainOfThought", "value: int -> result: int", "solve", {})
fresh.call("solve", {"value": 1})
@@ -347,7 +375,7 @@ def test_predictor_call_budget_can_be_disabled() -> None:
flex = _bridged_flex_with(max_predictor_calls=None)
assert flex._bridge._max_predictor_calls is None
dspy.configure(lm=DummyLM([{"reasoning": "r", "result": str(i)} for i in range(5)]))
- inv = bridge._Invocation(flex._bridge, {})
+ inv = flex._bridge.invocation()
inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})
for _ in range(5): # no cap -> all calls go through
inv.call("solve", {"value": 1})
@@ -402,10 +430,10 @@ def test_bridged_lm_failure_keeps_its_type_across_the_boundary() -> None:
# typed error, not as a CodeExecutionError that merely mentions the type name — so it can't
# be mistaken for (or absorbed as) a broken code candidate.
def drive(tools):
- tools[bridge.CONSTRUCT_TOOL](
+ tools[facade.CONSTRUCT_TOOL](
kind="Predict", signature="value: int -> result: int", attr_name="solve", kwargs={}
)
- tools[bridge.CALL_TOOL](handle="solve", inputs={"value": 2})
+ tools[facade.CALL_TOOL](handle="solve", inputs={"value": 2})
flex = _boundary_flex(drive)
dspy.configure(lm=_RateLimitedLM([]))
@@ -420,14 +448,14 @@ def test_recovered_lm_failure_does_not_hijack_a_later_crash() -> None:
# callback), a subsequent crash is its own bug: the stale LMError must not repaint a
# candidate error as an infrastructure one.
def drive(tools):
- tools[bridge.CONSTRUCT_TOOL](
+ tools[facade.CONSTRUCT_TOOL](
kind="Predict", signature="value: int -> result: int", attr_name="solve", kwargs={}
)
try:
- tools[bridge.CALL_TOOL](handle="solve", inputs={"value": 2})
+ tools[facade.CALL_TOOL](handle="solve", inputs={"value": 2})
except Exception:
pass # the generated code swallows the flattened boundary error and falls back
- tools[bridge.CONSTRUCT_TOOL](kind="Predict", signature="q -> a", attr_name="fallback", kwargs={})
+ tools[facade.CONSTRUCT_TOOL](kind="Predict", signature="q -> a", attr_name="fallback", kwargs={})
raise NameError("name 'oops' is not defined")
flex = _boundary_flex(drive)
@@ -440,11 +468,11 @@ def test_recovered_lm_failure_does_not_hijack_a_local_later_crash() -> None:
# Same recovery, but with NO bridge call between it and the crash — the host can't observe
# the recovery, so only the tag correlation tells the stale LMError apart from the local bug.
def drive(tools):
- tools[bridge.CONSTRUCT_TOOL](
+ tools[facade.CONSTRUCT_TOOL](
kind="Predict", signature="value: int -> result: int", attr_name="solve", kwargs={}
)
try:
- tools[bridge.CALL_TOOL](handle="solve", inputs={"value": 2})
+ tools[facade.CALL_TOOL](handle="solve", inputs={"value": 2})
except Exception:
pass # the generated code recovers from the provider failure
raise NameError("later sandbox bug")
@@ -466,21 +494,21 @@ def test_recovered_lm_failure_does_not_hijack_a_local_later_crash() -> None:
def test_accepts_interpreter_factory_only_for_code_executing_kinds() -> None:
# Introspection-based, not a hardcoded list, so it tracks the real constructors.
- assert bridge._accepts_interpreter_factory(dspy.CodeAct)
- assert bridge._accepts_interpreter_factory(dspy.ProgramOfThought)
- assert bridge._accepts_interpreter_factory(dspy.RLM) # type: ignore[arg-type] # dspy re-exports RLM oddly
+ assert facade._accepts_interpreter_factory(dspy.CodeAct)
+ assert facade._accepts_interpreter_factory(dspy.ProgramOfThought)
+ assert facade._accepts_interpreter_factory(dspy.RLM) # type: ignore[arg-type] # dspy re-exports RLM oddly
# Pure-LM and tool-calling predictors run no inner code, so no interpreter factory is injected.
- assert not bridge._accepts_interpreter_factory(dspy.Predict)
- assert not bridge._accepts_interpreter_factory(dspy.ChainOfThought)
- assert not bridge._accepts_interpreter_factory(dspy.ReAct)
- assert not bridge._accepts_interpreter_factory(dspy.ReActV2) # type: ignore[arg-type] # re-export quirk
+ assert not facade._accepts_interpreter_factory(dspy.Predict)
+ assert not facade._accepts_interpreter_factory(dspy.ChainOfThought)
+ assert not facade._accepts_interpreter_factory(dspy.ReAct)
+ assert not facade._accepts_interpreter_factory(dspy.ReActV2) # type: ignore[arg-type] # re-export quirk
def test_bridged_codeact_inherits_flex_interpreter_backend() -> None:
# CodeAct requires function tools; `shout` is a Flex-level tool, resolvable on the host.
flex = Flex(ShoutSig, tools=[shout], interpreter_factory=lambda: MockInterpreter())
- inv = bridge._Invocation(flex._bridge, {})
- tools_payload = [{bridge.TOOL_MARKER: "shout"}] # how the shim passes a tool by name
+ inv = flex._bridge.invocation()
+ tools_payload = [{facade.TOOL_MARKER: "shout"}] # how the shim passes a tool by name
inv.construct("CodeAct", "text: str -> out: str", "act", {"tools": tools_payload})
# The sub-predictor makes its per-forward interpreter from the Flex factory (the configured
# backend), not a default PythonInterpreter.
@@ -490,13 +518,11 @@ def test_bridged_codeact_inherits_flex_interpreter_backend() -> None:
assert isinstance(act._interpreter_factory(), MockInterpreter)
-def test_sub_interpreter_factory_makes_a_fresh_interpreter_per_call() -> None:
+def test_code_executing_sub_predictors_get_the_flex_interpreter_factory() -> None:
flex = _bridged_flex() # a real factory (lambda: MockInterpreter())
- factory = flex._bridge._sub_interpreter_factory()
- assert factory is flex._bridge._factory # the sub-predictor gets the Flex factory itself
- a, b = factory(), factory()
- assert isinstance(a, MockInterpreter) and isinstance(b, MockInterpreter)
- assert a is not b # so each forward is isolated in its own interpreter
+ inv = flex._bridge.invocation()
+ inv.construct("RLM", "value: int -> result: int", "inner", {})
+ assert inv._predictors["inner"]._interpreter_factory is flex._bridge._factory
# =============================================================================
diff --git a/tests/flex/test_flex_output_types.py b/tests/flex/test_flex_output_types.py
index fbea00efa..4257c7c95 100644
--- a/tests/flex/test_flex_output_types.py
+++ b/tests/flex/test_flex_output_types.py
@@ -27,7 +27,7 @@ import pydantic
import pytest
import dspy
-from dspy.predict.flex import Flex, bridge
+from dspy.predict.flex import Flex
from dspy.primitives.code_interpreter import CodeInterpreterError
from dspy.teleprompt.bootstrap_trace import FailedPrediction
from dspy.teleprompt.gepa.gepa_utils import DspyAdapter
@@ -126,7 +126,7 @@ def test_custom_type_resolves_when_it_is_not_on_the_calling_stack() -> None:
assert out.contact == _offstack_types.Contact(name="Ada", age=36)
# And the host-built sub-predictor resolved `Contact` to the module's class, not a lookalike:
- inv = bridge._Invocation(program._bridge, {})
+ inv = program._bridge.invocation()
inv.construct("Predict", "text: str -> contact: Contact", "p", {})
assert inv._predictors["p"].signature.output_fields["contact"].annotation is _offstack_types.Contact
@@ -150,7 +150,7 @@ def test_sub_signature_naming_a_custom_type_resolves_on_the_host() -> None:
program._bind_code(source)
dspy.configure(lm=DummyLM([{"person": json.dumps({"name": "Ada", "age": 36})}] * 2))
program(text="Ada is 36") # would raise `Unknown name: Person` if resolution failed
- inv = bridge._Invocation(program._bridge, {})
+ inv = program._bridge.invocation()
inv.construct("Predict", "text: str -> person: Person", "p", {})
assert inv._predictors["p"].signature.output_fields["person"].annotation is Person
diff --git a/tests/flex/test_tools.py b/tests/flex/test_tools.py
index 33b0da6a9..d158c8f86 100644
--- a/tests/flex/test_tools.py
+++ b/tests/flex/test_tools.py
@@ -20,9 +20,9 @@ import pytest
import dspy
from dspy.predict.flex import Flex
-from dspy.predict.flex.bridge import SHIM_SETUP
from dspy.predict.flex.primitives_doc import PRIMITIVES_CATALOG
from dspy.primitives.code_interpreter import CodeInterpreterError
+from dspy.primitives.facade import SHIM_SETUP
from dspy.teleprompt.gepa.gepa_flex_utils import CodeProposalSignature
from dspy.teleprompt.gepa.gepa_utils import DspyAdapter
from dspy.utils.dummies import DummyLM
diff --git a/tests/mock_interpreter.py b/tests/mock_interpreter.py
index 9c9c6def8..80d124d20 100644
--- a/tests/mock_interpreter.py
+++ b/tests/mock_interpreter.py
@@ -15,6 +15,12 @@ from dspy.primitives.code_interpreter import CodeInterpreterError, FinalOutput
__all__ = ["MockInterpreter", "MockInterpreterFactory"]
+def _is_facade_setup(code: str) -> bool:
+ from dspy.primitives.facade import SHIM_SETUP
+
+ return code == SHIM_SETUP
+
+
class MockInterpreter:
"""Mock interpreter that returns scripted responses.
@@ -62,6 +68,8 @@ class MockInterpreter:
self.tools = tools or {}
self.call_count = 0
self.call_history: list[tuple[str, dict[str, Any]]] = []
+ # Sandbox dspy facade installs (RLM and Flex set one up per interpreter); kept out of the script.
+ self.setup_history: list[tuple[str, dict[str, Any]]] = []
self._shutdown = False
def start(self) -> None:
@@ -89,6 +97,10 @@ class MockInterpreter:
raise CodeInterpreterError("MockInterpreter has been shutdown")
variables = variables or {}
+ if self.execute_fn is None and _is_facade_setup(code):
+ self.setup_history.append((code, variables))
+ return None
+
self.call_history.append((code, variables))
self.call_count += 1
diff --git a/tests/predict/snapshots/rlm_python_interpreter_lm_request.json b/tests/predict/snapshots/rlm_python_interpreter_lm_request.json
index 1baed7178..39aff8322 100644
--- a/tests/predict/snapshots/rlm_python_interpreter_lm_request.json
+++ b/tests/predict/snapshots/rlm_python_interpreter_lm_request.json
@@ -7,7 +7,7 @@
{
"type": "text",
"metadata": {},
- "text": "Your input fields are:\n1. `variables_info` (str): Metadata about the variables available in the REPL\n2. `repl_history` (REPLHistory): Previous REPL code executions and their outputs\n3. `iteration` (str): Current iteration number (1-indexed) out of max_iters\nYour output fields are:\n1. `reasoning` (str): Think step-by-step: what do you know? What remains? Plan your next action.\n2. `code` (str): Python code to execute. Use markdown code block format: ```python\\n<code>\\n```\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## variables_info ## ]]\n{variables_info}\n\n[[ ## repl_history ## ]]\n{repl_history}\n\n[[ ## iteration ## ]]\n{iteration}\n\n[[ ## reasoning ## ]]\n{reasoning}\n\n[[ ## code ## ]]\n{code}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `query`, produce the fields `answer`.\n \n You are tasked with producing the following outputs given the inputs `query`:\n - {answer}\n \n You have access to a Python REPL environment. Write Python code and it will be executed. You will see the output, then write more code based on what you learned. This is an iterative process.\n \n Execution environment:\n Python runs in Pyodide/WebAssembly. State persists across executions, but subprocesses and native extensions are unavailable. Python standard libraries such as re, json, collections, and math are available. Host filesystem, environment, and network access require explicit permission.\n \n Available:\n - Variables: `query` (your input data)\n - `llm_query(prompt)` - query a sub-LLM (~500K char capacity) for semantic analysis\n - `llm_query_batched(prompts)` - query multiple prompts concurrently (much faster for multiple queries)\n - `print()` - ALWAYS print to see results\n - `SUBMIT(answer)` - submit final output when done\n \n IMPORTANT: This is ITERATIVE. Each code block you write will execute, you'll see the output, then you decide what to do next. Do NOT try to solve everything in one step.\n \n 1. EXPLORE FIRST - Look at your data before processing it. Print samples, check types/lengths, understand the structure.\n 2. ITERATE - Write small code snippets, observe outputs, then decide next steps. State persists between iterations.\n 3. VERIFY BEFORE SUBMITTING - If results seem wrong (zeros, empty, unexpected), reconsider your approach.\n 4. USE llm_query FOR SEMANTICS - String matching finds WHERE things are; llm_query understands WHAT things mean.\n 5. MINIMIZE RETYPING (INPUTS & OUTPUTS) - When values are long, precise, or error-prone (IDs, numbers, code, quotes), re-access them via variables and parse/compute in code instead of retyping. Use small, targeted prints to sanity-check, but avoid manual copying when variables can carry the exact value.\n 6. SUBMIT ONLY AFTER SEEING OUTPUTS - SUBMIT ends the current run immediately. If you need to inspect printed output, run it in one step, review the result, then call SUBMIT in a later step.\n \n You have max 50 sub-LLM calls. When done, call SUBMIT() with your output."
+ "text": "Your input fields are:\n1. `variables_info` (str): Metadata about the variables available in the REPL\n2. `repl_history` (REPLHistory): Previous REPL code executions and their outputs\n3. `iteration` (str): Current iteration number (1-indexed) out of max_iters\nYour output fields are:\n1. `reasoning` (str): Think step-by-step: what do you know? What remains? Plan your next action.\n2. `code` (str): Python code to execute. Use markdown code block format: ```python\\n<code>\\n```\nAll interactions will be structured in the following way, with the appropriate values filled in.\n\n[[ ## variables_info ## ]]\n{variables_info}\n\n[[ ## repl_history ## ]]\n{repl_history}\n\n[[ ## iteration ## ]]\n{iteration}\n\n[[ ## reasoning ## ]]\n{reasoning}\n\n[[ ## code ## ]]\n{code}\n\n[[ ## completed ## ]]\nIn adhering to this structure, your objective is: \n Given the fields `query`, produce the fields `answer`.\n \n You are tasked with producing the following outputs given the inputs `query`:\n - {answer}\n \n You have access to a Python REPL environment. Write Python code and it will be executed. You will see the output, then write more code based on what you learned. This is an iterative process.\n \n Execution environment:\n Python runs in Pyodide/WebAssembly. State persists across executions, but subprocesses and native extensions are unavailable. Python standard libraries such as re, json, collections, and math are available. Host filesystem, environment, and network access require explicit permission.\n \n Sub-agents (dspy):\n You may `import dspy` and build sub-agents in the REPL for subtasks that need structured inputs/outputs.\n - `dspy.Predict(\"question -> answer\")(question=...)` or `dspy.ChainOfThought(...)` - single-step sub-agents.\n - `dspy.ReActV2(\"question -> answer\", tools=[...])(question=...)` - a multi-step tool-using sub-agent.\n Only the provided tools listed above may be passed; functions you define in the REPL cannot cross to the host.\n - `dspy.RLM(\"context, query -> answer\")(context=..., query=...)` - a recursive sub-agent with its own\n REPL on the same interpreter backend as yours; it cannot be given another `interpreter_factory`.\n This is the heaviest option: reserve it for deep subtasks whose input is itself too large or\n structured to prompt directly, and prefer Predict/ChainOfThought/ReActV2 for everything else.\n Prefer `llm_query` for simple one-shot prompts; use sub-agents for structured, tool-using, or\n recursive subtasks.\n \n Available:\n - Variables: `query` (your input data)\n - `llm_query(prompt)` - query a sub-LLM (~500K char capacity) for semantic analysis\n - `llm_query_batched(prompts)` - query multiple prompts concurrently (much faster for multiple queries)\n - `print()` - ALWAYS print to see results\n - `SUBMIT(answer)` - submit final output when done\n \n IMPORTANT: This is ITERATIVE. Each code block you write will execute, you'll see the output, then you decide what to do next. Do NOT try to solve everything in one step.\n \n 1. EXPLORE FIRST - Look at your data before processing it. Print samples, check types/lengths, understand the structure.\n 2. ITERATE - Write small code snippets, observe outputs, then decide next steps. State persists between iterations.\n 3. VERIFY BEFORE SUBMITTING - If results seem wrong (zeros, empty, unexpected), reconsider your approach.\n 4. USE llm_query FOR SEMANTICS - String matching finds WHERE things are; llm_query understands WHAT things mean.\n 5. MINIMIZE RETYPING (INPUTS & OUTPUTS) - When values are long, precise, or error-prone (IDs, numbers, code, quotes), re-access them via variables and parse/compute in code instead of retyping. Use small, targeted prints to sanity-check, but avoid manual copying when variables can carry the exact value.\n 6. SUBMIT ONLY AFTER SEEING OUTPUTS - SUBMIT ends the current run immediately. If you need to inspect printed output, run it in one step, review the result, then call SUBMIT in a later step.\n \n You have max 50 sub-LLM calls. When done, call SUBMIT() with your output."
}
],
"name": null,
diff --git a/tests/predict/test_rlm.py b/tests/predict/test_rlm.py
index a570f7e64..718508d73 100644
--- a/tests/predict/test_rlm.py
+++ b/tests/predict/test_rlm.py
@@ -7,14 +7,16 @@ Test organization:
"""
import base64
-from contextlib import contextmanager
+import io
+import sys
+from contextlib import contextmanager, redirect_stdout
from pathlib import Path
import pytest
import dspy
from dspy.adapters.types.tool import Tool
-from dspy.predict.rlm import RLM, _strip_code_fences
+from dspy.predict.rlm import RLM, _LLMCallBudget, _strip_code_fences
from dspy.primitives.code_interpreter import (
CodeExecutionError,
CodeInterpreterError,
@@ -25,6 +27,7 @@ from dspy.primitives.prediction import Prediction
from dspy.primitives.python_interpreter import PythonInterpreter
from dspy.primitives.repl_types import REPLEntry, REPLHistory, REPLVariable
from dspy.primitives.sandbox_serializable import SandboxSerializable
+from dspy.utils.dummies import DummyLM
from tests.mock_interpreter import MockInterpreter, MockInterpreterFactory
# ============================================================================
@@ -1721,7 +1724,7 @@ class TestPrepareSerializableVars:
def test_separates_serializable_from_regular(self):
"""Serializable values are injected; regular values are returned."""
- mock = MockInterpreter(responses=["", FinalOutput({"answer": "42"})])
+ mock = MockInterpreter(responses=[FinalOutput({"answer": "42"})])
rlm = RLM("data, query -> answer", max_iters=3)
stub = _StubSerializable("payload")
@@ -1851,5 +1854,320 @@ class TestLargeSerializableRoundTrip:
assert "abc123" in result
+# ============================================================================
+# Sub-agents (the sandbox dspy facade)
+# ============================================================================
+
+
+class _Submit(Exception): # noqa: N818 - control-flow signal, not an error
+ def __init__(self, outputs: dict):
+ self.outputs = outputs
+
+
+class _InProcessInterpreter:
+ """Minimal interpreter for tests that executes in the host process, like a real in-process backend."""
+
+ def __init__(self):
+ self.tools = {}
+ self.output_fields = None
+ self._namespace = {}
+
+ def start(self) -> None:
+ pass
+
+ def execute(self, code: str, variables: dict | None = None):
+ self._namespace.update(self.tools)
+ self._namespace.update(variables or {})
+ output_names = [field["name"] for field in self.output_fields or []]
+
+ def SUBMIT(*args, **kwargs): # noqa: N802 - sandbox API name
+ outputs = dict(zip(output_names, args, strict=False))
+ outputs.update(kwargs)
+ raise _Submit(outputs)
+
+ self._namespace["SUBMIT"] = SUBMIT
+ buffer = io.StringIO()
+ try:
+ with redirect_stdout(buffer):
+ exec(code, self._namespace)
+ except _Submit as submit:
+ return FinalOutput(submit.outputs)
+ except Exception as e:
+ raise CodeExecutionError(f"{type(e).__name__}: {e}") from e
+ return buffer.getvalue()
+
+ def shutdown(self) -> None:
+ pass
+
+
+def _recording_predictor(code: str):
+ """Mock action predictor that records the signature RLM hands each call."""
+ seen = []
+
+ class Recording:
+ def __call__(self, signature=None, **kwargs):
+ seen.append(signature)
+ return Prediction(reasoning="Done", code=code)
+
+ async def acall(self, signature=None, **kwargs):
+ return self(signature=signature, **kwargs)
+
+ return Recording(), seen
+
+
+def _facade_invocation(rlm: RLM, interpreter_factory=None):
+ """The host side of the facade RLM installs into an invocation's interpreter."""
+ from dspy.primitives import facade
+
+ interpreter = MockInterpreter()
+ factory = interpreter_factory or resolve_interpreter_factory(rlm._interpreter_factory)
+ rlm._setup_facade(interpreter, _LLMCallBudget(rlm.max_llm_calls), factory)
+ return interpreter.tools[facade.CONSTRUCT_TOOL].__self__
+
+
+class TestRLMSubAgents:
+ def test_prompt_offers_sub_agents(self):
+ instructions = RLM("query -> answer", interpreter_factory=MockInterpreterFactory()).generate_action.signature.instructions
+
+ assert "Sub-agents (dspy)" in instructions
+ assert 'dspy.RLM("context, query -> answer")' in instructions
+ assert "cannot cross to the host" in instructions
+ assert "interpreter_factory=" not in instructions
+ assert "dspy.configure" not in instructions
+
+ @pytest.mark.parametrize("use_async", [False, True])
+ def test_facade_is_installed_into_every_factory_interpreter(self, use_async):
+ import asyncio
+
+ from dspy.primitives import facade
+
+ factory = MockInterpreterFactory()
+ rlm = RLM("query -> answer", max_iters=1, interpreter_factory=factory)
+ rlm.generate_action, seen = _recording_predictor('SUBMIT("42")')
+ factory.responses = [FinalOutput({"answer": "42"})]
+
+ result = asyncio.run(rlm.acall(query="q")) if use_async else rlm(query="q")
+
+ assert result.answer == "42"
+ interpreter = factory.instances[0]
+ assert [code for code, _ in interpreter.setup_history] == [facade.SHIM_SETUP]
+ assert {facade.CONSTRUCT_TOOL, facade.CALL_TOOL} <= interpreter.tools.keys()
+ assert "Sub-agents (dspy)" in seen[0].instructions
+
+ @pytest.mark.parametrize("use_async", [False, True])
+ def test_call_time_factory_hosts_the_facade_and_backs_nested_sub_agents(self, use_async):
+ import asyncio
+
+ from dspy.primitives import facade
+
+ configured, override = MockInterpreterFactory(), MockInterpreterFactory([FinalOutput({"answer": "42"})])
+ rlm = RLM("query -> answer", max_iters=1, interpreter_factory=configured)
+ rlm.generate_action, seen = _recording_predictor('SUBMIT("42")')
+
+ call = rlm.acall(query="q", interpreter_factory=override) if use_async else None
+ result = asyncio.run(call) if use_async else rlm(query="q", interpreter_factory=override)
+
+ assert result.answer == "42"
+ assert configured.instances == []
+ interpreter = override.instances[0]
+ assert [code for code, _ in interpreter.setup_history] == [facade.SHIM_SETUP]
+ assert "Sub-agents (dspy)" in seen[0].instructions
+ # A nested code-executing sub-agent runs on the factory this invocation runs on.
+ invocation = interpreter.tools[facade.CONSTRUCT_TOOL].__self__
+ invocation.construct("RLM", "q -> a", "nested")
+ assert invocation._predictors["nested"]._interpreter_factory is override
+
+ def test_interpreter_in_the_host_memory_runs_without_sub_agents(self, caplog):
+ # The shim refuses it rather than replace the host's `dspy` module; RLM carries on without sub-agents.
+ rlm = RLM("query -> answer", max_iters=1, interpreter_factory=_InProcessInterpreter)
+ rlm.generate_action, seen = _recording_predictor('SUBMIT("42")')
+
+ with caplog.at_level("WARNING", logger="dspy.predict.rlm"):
+ assert rlm(query="q").answer == "42"
+
+ assert sys.modules["dspy"] is dspy
+ assert "Sub-agents (dspy)" not in seen[0].instructions
+ assert "host's memory" in caplog.text
+
+ def test_facade_reserves_sandbox_names(self):
+ with pytest.raises(ValueError, match="conflict"):
+ RLM("dspy -> answer")
+ # Shim internals and host-injected variables live under the _dspy/__dspy prefixes.
+ for reserved in ("_dspy_host", "__dspy_code", "__dspy_construct__"):
+ with pytest.raises(ValueError, match="conflicts with built-in sandbox function"):
+ RLM("query -> answer", tools=[dspy.Tool(lambda: "", name=reserved)])
+
+ # The factory is the host's to choose, so its old marker name is an ordinary tool name.
+ def dspy_interpreter_factory() -> str:
+ """Unrelated user tool."""
+ return ""
+
+ RLM("query -> answer", tools=[dspy_interpreter_factory])
+
+ def test_facade_builds_and_runs_predictors_on_host(self):
+ invocation = _facade_invocation(RLM("query -> answer"))
+
+ handle = invocation.construct("Predict", "question -> answer", "sub")
+ dspy.configure(lm=DummyLM([{"answer": "bridged"}]))
+ fields = invocation.call(handle, {"question": "ping"})
+
+ assert fields["answer"] == "bridged"
+ with pytest.raises(CodeInterpreterError, match="not supported through the sandbox dspy bridge"):
+ invocation.construct("Flex", "q -> a", "nope")
+ with pytest.raises(CodeInterpreterError, match="Unknown predictor handle"):
+ invocation.call("missing")
+
+ def test_facade_nested_rlm_gets_the_rlm_interpreter_factory(self):
+ factory = MockInterpreterFactory()
+ invocation = _facade_invocation(RLM("query -> answer", interpreter_factory=factory))
+
+ invocation.construct("RLM", "q -> a", "nested")
+ assert invocation._predictors["nested"]._interpreter_factory is factory
+
+ @pytest.mark.parametrize("requested", ["__dspy_interpreter_factory__", "LocalInterpreter", None])
+ def test_facade_sub_agents_cannot_choose_their_interpreter(self, requested):
+ invocation = _facade_invocation(RLM("query -> answer"))
+
+ for kind in ("RLM", "CodeAct", "ProgramOfThought", "Predict"):
+ with pytest.raises(CodeInterpreterError, match="cannot choose its interpreter_factory"):
+ invocation.construct(kind, "q -> a", "chosen", {"interpreter_factory": requested})
+ assert "chosen" not in invocation._predictors
+
+ def test_facade_tool_markers_resolve_only_provided_tools(self):
+ from dspy.primitives import facade
+
+ rlm = RLM("query -> answer", tools=[echo_tool])
+ invocation = _facade_invocation(rlm)
+
+ assert invocation._decode_tools({facade.TOOL_MARKER: "echo_tool"}) is rlm._user_tools["echo_tool"]
+ with pytest.raises(CodeInterpreterError, match="cannot be handed to a bridged sub-predictor"):
+ invocation._decode_tools({facade.TOOL_MARKER: "repl_defined"})
+
+ def test_facade_lm_calls_draw_on_max_llm_calls(self):
+ # Bridged predictors are charged per LM call on the forward's budget, not per predictor call.
+ invocation = _facade_invocation(RLM("query -> answer", max_llm_calls=1))
+ dspy.configure(lm=DummyLM([{"answer": "first"}, {"answer": "second"}]))
+
+ handle = invocation.construct("Predict", "question -> answer", "sub")
+ assert invocation.call(handle, {"question": "ping"})["answer"] == "first"
+ with pytest.raises(RuntimeError, match=r"LLM call limit exceeded: 1 \+ 1 > 1"):
+ invocation.call(handle, {"question": "again"})
+
+ def test_facade_n_completions_draw_n_calls(self):
+ invocation = _facade_invocation(RLM("query -> answer", max_llm_calls=2))
+ dspy.configure(lm=DummyLM([{"answer": "x"}] * 4))
+
+ handle = invocation.construct("Predict", "question -> answer", "sub", {"n": 3})
+ with pytest.raises(RuntimeError, match=r"LLM call limit exceeded: 0 \+ 3 > 2"):
+ invocation.call(handle, {"question": "ping"})
+
+ def test_facade_resolves_the_lm_per_call_like_llm_query(self):
+ # With no sub_lm, the facade serves dspy.settings.lm as it is when the sub-agent runs.
+ invocation = _facade_invocation(RLM("query -> answer"))
+ handle = invocation.construct("Predict", "question -> answer", "sub")
+
+ with dspy.context(lm=DummyLM([{"answer": "late lm"}])):
+ assert invocation.call(handle, {"question": "ping"})["answer"] == "late lm"
+
+ def test_facade_serves_sub_lm_as_a_scoped_override(self):
+ # Any BaseLM works as sub_lm (predictors run host-side), and a scoped override (not set_lm)
+ # reaches a bridged RLM's own llm_query too.
+ sub_lm = DummyLM([{"answer": "from sub_lm"}] * 4)
+ invocation = _facade_invocation(RLM("query -> answer", sub_lm=sub_lm))
+
+ handle = invocation.construct("Predict", "question -> answer", "sub")
+ assert invocation.call(handle, {"question": "ping"})["answer"] == "from sub_lm"
+
+ handle = invocation.construct("RLM", "question -> answer", "nested")
+ nested = invocation._predictors["nested"]
+ assert nested.generate_action.lm is None # not set_lm'd
+ seen = {}
+
+ def probe(**kwargs):
+ seen["lm"] = dspy.settings.lm
+ raise _Submit({"answer": "done"})
+
+ nested.forward = probe
+ with pytest.raises(_Submit):
+ invocation.call(handle, {"question": "ping"})
+
+ assert isinstance(seen["lm"], dspy.BaseLM)
+ assert seen["lm"]._lm is sub_lm
+
+ def test_facade_predictors_cannot_set_lm_routing_or_credentials(self):
+ invocation = _facade_invocation(RLM("query -> answer", sub_lm=DummyLM([{"answer": "ok"}] * 2)))
+
+ handle = invocation.construct("Predict", "question -> answer", "sub", {"api_base": "http://evil.example"})
+ with pytest.raises(TypeError, match=r"may not set LM option\(s\) \['api_base'\]"):
+ invocation.call(handle, {"question": "ping"})
+ handle = invocation.construct("Predict", "question -> answer", "plain", {"temperature": 0.7})
+ with pytest.raises(TypeError, match=r"may not set LM option\(s\) \['api_key', 'extra_headers'\]"):
+ invocation.call(handle, {"question": "ping", "config": {"api_key": "sk-x", "extra_headers": {"X": "y"}}})
+ assert invocation.call(handle, {"question": "ping"})["answer"] == "ok"
+
+
+class TestRLMFacadeDeno:
+ def test_facade_runs_on_the_default_sandbox(self):
+ # The facade needs no dspy inside the sandbox.
+ rlm = RLM("query -> answer", max_iters=2, interpreter_factory=PythonInterpreter)
+ rlm.generate_action = make_mock_predictor([
+ {
+ "reasoning": "Run a sub-agent through the facade",
+ "code": (
+ "sub = dspy.Predict('question -> answer')\n"
+ "res = sub(question='ping')\n"
+ "print(res.answer)"
+ ),
+ },
+ {"reasoning": "Submit", "code": "SUBMIT(res.answer)"},
+ ])
+
+ with dummy_lm_context([{"answer": "facade on pyodide"}]):
+ result = rlm(query="q")
+
+ assert result.answer == "facade on pyodide"
+
+ def test_facade_lm_calls_share_the_llm_query_budget_on_the_default_sandbox(self):
+ rlm = RLM("query -> answer", max_iters=2, max_llm_calls=1, interpreter_factory=PythonInterpreter)
+ rlm.generate_action = make_mock_predictor([
+ {
+ "reasoning": "Spend the budget with llm_query, then try a bridged sub-agent",
+ "code": (
+ "first = llm_query('a')\n"
+ "try:\n"
+ " dspy.Predict('question -> answer')(question='b')\n"
+ "except Exception as e:\n"
+ " print('sub-agent:', e)\n"
+ ),
+ },
+ {"reasoning": "Submit", "code": "SUBMIT(first)"},
+ ])
+
+ with dummy_lm_context([{"answer": "one"}, {"answer": "two"}]):
+ result = rlm(query="q")
+
+ assert "LLM call limit exceeded: 1 + 1 > 1" in result.trajectory[0]["output"]
+ assert "one" in result.answer
+
+ def test_facade_survives_serializable_inputs_on_the_default_sandbox(self):
+ # PythonInterpreter registers tools once, so the facade must be installed before serializable setup.
+ rlm = RLM("data, query -> answer", max_iters=2, interpreter_factory=PythonInterpreter)
+ rlm.generate_action = make_mock_predictor([
+ {
+ "reasoning": "Run a sub-agent through the facade",
+ "code": "res = dspy.Predict('question -> answer')(question='ping')\nprint(res.answer)",
+ },
+ {"reasoning": "Submit", "code": "SUBMIT(res.answer)"},
+ ])
+
+ with dummy_lm_context([{"answer": "facade with serializable input"}]):
+ result = rlm(data=_StubSerializable("payload"), query="q")
+
+ # The block's own output: a dead facade would fall back to extract and still yield the answer.
+ assert "facade with serializable input" in result.trajectory[0]["output"]
+ assert len(result.trajectory) == 2
+
+
if __name__ == "__main__":
pytest.main([__file__, "-v"])
diff --git a/tests/primitives/test_dspy_interpreters_conformance.py b/tests/primitives/test_dspy_interpreters_conformance.py
new file mode 100644
index 000000000..3c14032f7
--- /dev/null
+++ b/tests/primitives/test_dspy_interpreters_conformance.py
@@ -0,0 +1,111 @@
+"""Conformance against dspy-interpreters, the third-party CodeInterpreter backends (a test-only
+dependency, declared in the ``dev`` dependency group).
+
+RLM installs the sandbox dspy facade into every interpreter it creates. On real backends: an isolated
+one runs bridged sub-agents, one that executes in the host's memory runs RLM without them, and the
+library's own consumer checks pass.
+"""
+
+import pytest
+
+import dspy
+from dspy.predict.rlm import RLM
+from dspy.primitives.prediction import Prediction
+from dspy.utils.dummies import DummyLM
+
+dspy_interpreters = pytest.importorskip("dspy_interpreters")
+
+InProcessInterpreter = dspy_interpreters.InProcessInterpreter
+SubprocessInterpreter = dspy_interpreters.SubprocessInterpreter
+BACKENDS = ["InProcessInterpreter", "SubprocessInterpreter"]
+
+
+def make_scripted_predictor(responses: list[dict]):
+ class ScriptedPredictor:
+ def __init__(self):
+ self.idx = 0
+
+ def __call__(self, **kwargs):
+ response = responses[self.idx % len(responses)]
+ self.idx += 1
+ return Prediction(**response)
+
+ return ScriptedPredictor()
+
+
[email protected]("backend_name", BACKENDS)
+def test_backends_satisfy_protocol(backend_name):
+ interpreter = getattr(dspy_interpreters, backend_name)()
+ try:
+ assert isinstance(interpreter, dspy.CodeInterpreter)
+ finally:
+ interpreter.shutdown()
+
+
+def test_in_process_backend_runs_rlm_without_sub_agents(caplog):
+ seen = []
+
+ class Recording:
+ def __call__(self, signature=None, **kwargs):
+ seen.append(signature.instructions)
+ return Prediction(reasoning="Submit", code='SUBMIT("ok")')
+
+ rlm = RLM("query -> answer", max_iters=1, interpreter_factory=InProcessInterpreter)
+ rlm.generate_action = Recording()
+ with caplog.at_level("WARNING", logger="dspy.predict.rlm"):
+ assert rlm(query="q").answer == "ok"
+
+ assert "Sub-agents (dspy)" not in seen[0]
+ assert "sub-agents are unavailable on InProcessInterpreter" in caplog.text
+
+
[email protected]("backend_name", BACKENDS)
+def test_library_consumer_checks_pass(backend_name):
+ backend = getattr(dspy_interpreters, backend_name)
+ dspy_interpreters.check_interpreter(backend).raise_for_failures()
+ dspy_interpreters.check_rlm(backend).raise_for_failures()
+
+
+def test_flex_facade_needs_an_isolated_backend():
+ dspy_interpreters.check_flex_facade(SubprocessInterpreter).raise_for_failures()
+ report = dspy_interpreters.check_flex_facade(InProcessInterpreter)
+ assert not report.passed and "host's memory" in report.results[0].detail
+
+
+def test_isolated_backend_runs_bridged_sub_agents_with_host_tools():
+ # Regression: worker backends bind host tools as "<lambda>" proxies; the shim names them by global.
+ calls = []
+
+ def echo(text: str) -> str:
+ """Echo the text."""
+ calls.append(text)
+ return text
+
+ rlm = RLM("query -> answer", max_iters=2, tools=[echo], interpreter_factory=SubprocessInterpreter)
+ rlm.generate_action = make_scripted_predictor([
+ {
+ "reasoning": "Bridged ReActV2 with a host tool",
+ "code": (
+ "import dspy\n"
+ "agent = dspy.ReActV2('question -> answer', tools=[echo])\n"
+ "res = agent(question='hi')\n"
+ "print(res.answer)"
+ ),
+ },
+ {"reasoning": "Submit", "code": "SUBMIT(res.answer)"},
+ ])
+ lm = DummyLM([
+ {
+ "next_thought": "echo it",
+ "tool_calls": dspy.ToolCalls.from_dict_list([{"name": "echo", "args": {"text": "hi"}}]),
+ },
+ {
+ "next_thought": "done",
+ "tool_calls": dspy.ToolCalls.from_dict_list([{"name": "submit", "args": {"answer": "echoed hi"}}]),
+ },
+ ])
+ with dspy.context(lm=lm, adapter=dspy.ChatAdapter()):
+ result = rlm(query="q")
+
+ assert result.answer == "echoed hi"
+ assert calls == ["hi"]
diff --git a/tests/primitives/test_facade.py b/tests/primitives/test_facade.py
new file mode 100644
index 000000000..113ad66b6
--- /dev/null
+++ b/tests/primitives/test_facade.py
@@ -0,0 +1,104 @@
+"""SandboxLM: the host LM as sandboxed code may call it."""
+
+import pytest
+
+from dspy.primitives.facade import SANDBOX_LM_KWARGS, SandboxLM
+from dspy.utils.dummies import DummyLM
+
+
+class _CapableLM(DummyLM):
+ supports_function_calling = True
+ supports_response_schema = True
+ supported_params = frozenset({"temperature", "response_format", "tools"})
+
+
+def _lm():
+ return DummyLM([{"answer": "a"}, {"answer": "b"}])
+
+
+def test_generation_parameters_pass_and_the_wrapped_lm_does_the_work():
+ lm = _lm()
+ outputs = SandboxLM(lm)(messages=[{"role": "user", "content": "hi"}], temperature=0.3, max_tokens=5)
+ assert outputs == ["[[ ## answer ## ]]\na"]
+ assert len(lm.history) == 1
+
+
+ "options",
+ [{"api_base": "http://evil.example"}, {"api_key": "sk-x"}, {"extra_headers": {"X": "y"}}, {"model": "gpt-5-pro"}],
+)
+def test_routing_credential_and_transport_options_are_rejected_before_any_request(options):
+ lm = _lm()
+ sandbox = SandboxLM(lm)
+ with pytest.raises(TypeError, match="may not set LM option"):
+ sandbox("prompt", temperature=0.1, **options)
+ with pytest.raises(TypeError, match="may not set LM option"):
+ sandbox.copy(**options)
+ assert lm.history == []
+ assert not options.keys() & SANDBOX_LM_KWARGS
+
+
+async def test_acall_applies_the_same_allowlist():
+ lm = _lm()
+ with pytest.raises(TypeError, match=r"may not set LM option\(s\) \['api_key'\]"):
+ await SandboxLM(lm).acall("prompt", api_key="sk-x")
+ assert lm.history == []
+ assert await SandboxLM(lm).acall("prompt", temperature=0.2) == ["[[ ## answer ## ]]\na"]
+
+
+def test_reserve_meters_admitted_calls_only():
+ reserved = []
+ sandbox = SandboxLM(_lm(), reserve=reserved.append)
+ sandbox("one")
+ with pytest.raises(TypeError):
+ sandbox("two", api_key="sk-x")
+ sandbox.copy(temperature=0.9)("three")
+ sandbox.forward("four")
+ with pytest.raises(TypeError):
+ sandbox.forward("five", api_base="http://evil.example")
+ assert reserved == [1, 1, 1]
+
+
+def test_reserve_charges_one_slot_per_requested_completion():
+ # `n` asks for n completions in one request; it must not multiply max_llm_calls.
+ class _StubLM:
+ model = "stub"
+
+ def __init__(self):
+ self.kwargs = {"n": 4, "api_key": "sk-host"}
+
+ def __call__(self, *args, **kwargs):
+ return ["x"] * kwargs.get("n", self.kwargs["n"])
+
+ reserved = []
+ sandbox = SandboxLM(_StubLM(), reserve=reserved.append)
+ assert len(sandbox("per-call", n=64)) == 64
+ sandbox("mirrored default")
+ sandbox("explicit one", n=1)
+ assert reserved == [64, 4, 1]
+
+
+def test_proxy_mirrors_identity_kwargs_and_capabilities_of_the_wrapped_lm():
+ lm = _CapableLM([{"answer": "a"}])
+ lm.kwargs.update(temperature=0.7, api_key="sk-secret", api_base="http://internal.example")
+ sandbox = SandboxLM(lm)
+ assert (sandbox.model, sandbox.model_type) == (lm.model, lm.model_type)
+ assert sandbox.kwargs["temperature"] == 0.7
+ assert not {"api_key", "api_base"} & sandbox.kwargs.keys() # credentials stay on the wrapped LM
+ assert sandbox.supports_function_calling and sandbox.supports_response_schema and not sandbox.supports_reasoning
+ assert sandbox.supported_params == {"temperature", "response_format", "tools"}
+ assert not SandboxLM(_lm()).supports_function_calling
+
+
+def test_wraps_a_callable_only_lm():
+ # RLM accepts any callable as sub_lm; the facade is installed by default, so the proxy must too.
+ class PlainLM:
+ def __call__(self, prompt):
+ return ["plain:" + prompt]
+
+ sandbox = SandboxLM(PlainLM())
+ assert sandbox("hi") == ["plain:hi"]
+ assert (sandbox.model, sandbox.kwargs, sandbox.supports_function_calling, sandbox.supported_params) == (
+ None, {}, False, set()
+ )
diff --git a/tests/primitives/test_interpreter_factory_config.py b/tests/primitives/test_interpreter_factory_config.py
index ce6288e97..5ece39728 100644
--- a/tests/primitives/test_interpreter_factory_config.py
+++ b/tests/primitives/test_interpreter_factory_config.py
@@ -220,5 +220,4 @@ def test_flex_lets_a_sub_predictor_resolve_the_configured_factory_itself():
dspy.configure(interpreter_factory=MockInterpreterFactory())
flex = Flex(Doubler)
- assert flex._bridge._sub_interpreter_factory() is PythonInterpreter
- assert flex._bridge._build_predictor("RLM", "query -> answer", {})._interpreter_factory is PythonInterpreter
+ assert flex._bridge.invocation()._build_predictor("RLM", "query -> answer", {})._interpreter_factory is PythonInterpreter
diff --git a/uv.lock b/uv.lock
index fd0ad1176..49915f771 100644
--- a/uv.lock
+++ b/uv.lock
@@ -946,6 +946,11 @@ weaviate = [
{ name = "weaviate-client" },
]
+[package.dev-dependencies]
+dev = [
+ { name = "dspy-interpreters" },
+]
+
[package.metadata]
requires-dist = [
{ name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.18.0,<1.0.0" },
@@ -993,6 +998,17 @@ requires-dist = [
]
provides-extras = ["anthropic", "typesafe", "weaviate", "mcp", "deno", "langchain", "optuna", "numpy", "litellm", "dev", "test-extras"]
+[package.metadata.requires-dev]
+dev = [{ name = "dspy-interpreters", git = "https://github.com/cmpnd-ai/dspy-interpreters" }]
+
+[[package]]
+name = "dspy-interpreters"
+version = "0.1.0"
+source = { git = "https://github.com/cmpnd-ai/dspy-interpreters#f05f3bdca9b2fc543bc3f8512b887e36b90c7e04" }
+dependencies = [
+ { name = "dspy" },
+]
+
[[package]]
name = "email-validator"
version = "2.3.0"