DSPy: The framework for programming—not prompting—language models
Python
feat(dspy.Flex) + GEPA extension (#10047)
Michael I.e4e97aae29b8
diff --git a/docs/docs/api/modules/Flex.md b/docs/docs/api/modules/Flex.mdnew file mode 100644index 000000000..826ff5045--- /dev/null+++ b/docs/docs/api/modules/Flex.md@@ -0,0 +1,179 @@+# dspy.Flex++`Flex` is a DSPy module whose implementation is *optimizable code* rather than a fixed prompt. You construct it from a signature, and it defaults to a thin baseline over that signature. What makes it different is what an optimizer is allowed to do with it: instead of only rewriting instructions, `dspy.GEPA` can rewrite the module's entire source — splitting the task into multiple predictors, folding deterministic steps into plain Python, and authoring its own helper functions. Being a `Flex` is what tells GEPA that the module's code is an optimizable parameter.++## When to Use Flex++Reach for `Flex` when you'd rather have the optimizer discover the program's structure than hand-write it. That's the case when:++- The best decomposition is **unknown or worth searching** — you have a metric and a dataset to judge candidate structures against.+- Parts of the task are **deterministic** and shouldn't cost an LM call — arithmetic, parsing, lookups, normalization.+- You want the optimizer to **trade accuracy against cost** — e.g. rewarding programs that answer clear cases in code and reserve the LM for genuinely hard ones.++## Basic Usage++```python+import dspy++dspy.configure(lm=dspy.LM("openai/gpt-5"))++# Construct Flex from a signature, like any module.+solve = dspy.Flex("invoice: str -> total_cents: int")++# Runs the baseline (a single dspy.Predict).+result = solve(invoice="2 widgets @ $3.50, shipping $1.00")+print(result.total_cents)+```++Out of the box, `solve` is just a `dspy.Predict` over the signature, wrapped in a module (with `tools`, it starts as a `dspy.RLM` instead — see [Tools](#tools)). The point of `Flex` is what happens when you optimize it (see [Optimizing with GEPA](#optimizing-with-gepa)): GEPA can replace that baseline with, say, a predictor that only extracts quantities and unit prices, and a line of Python that multiplies and sums them.++The generated code always runs in a sandbox (`interpreter_factory` defaults to `dspy.PythonInterpreter`), so the example above needs [Deno](https://deno.land/) installed — see [Sandboxed Execution](#sandboxed-execution).++## How Optimization Works++`dspy.GEPA` discovers `Flex` submodules by type. When GEPA compiles a program containing one or more `Flex` submodules, it treats each one as a **code component**: rather than proposing a new instruction string, its reflection model proposes a new *whole module source*, guided by the signature, any available tools, and your metric's feedback on failing examples. GEPA binds the candidate source, evaluates it, and keeps it if it advances the Pareto frontier — the same search GEPA runs for prompts, applied to code.++A broken candidate can't crash the optimization run. If the reflection model emits source that fails to bind, GEPA scores that candidate as a failure and moves on, rather than aborting the optimization.++## Optimizing with GEPA++You optimize a `Flex` the same way you optimize any DSPy program — hand it to `dspy.GEPA` with a metric and a trainset:++```python+import dspy++dspy.configure(lm=dspy.LM("openai/gpt-5-mini")) # runs the program++def metric(gold, pred, trace=None, pred_name=None, pred_trace=None):+ correct = getattr(pred, "total_cents", None) == gold.total_cents+ fb = "Correct." if correct else (+ f"Wrong total: got {getattr(pred, 'total_cents', None)}, expected {gold.total_cents}. "+ "Have the LM extract line items, then sum them in Python."+ )+ return dspy.Prediction(score=1.0 if correct else 0.0, feedback=fb)++solve = dspy.Flex("invoice: str -> total_cents: int")++optimized = dspy.GEPA(+ metric=metric,+ reflection_lm=dspy.LM("openai/gpt-5", temperature=1.0, max_tokens=8000),+ max_metric_calls=60,+).compile(solve, trainset=trainset, valset=valset)++print(optimized.module_src) # the discovered program+```++The `metric` returns a `dspy.Prediction(score=..., feedback=...)` — a scalar plus natural-language feedback that GEPA reflects on to revise the module. For how to write an effective feedback metric, see [Implementing Feedback Metrics](../optimizers/GEPA/overview.md#implementing-feedback-metrics) in the GEPA guide and the [dspy.GEPA tutorials](../../tutorials/gepa_ai_program/index.md).++### Rewarding leaner programs with a trace-aware metric++A common goal with `Flex` is to push work out of the LM and into deterministic code. To optimize for that, your metric needs to see *how* an answer was produced, not just whether it was right. Declare a `program_trace` parameter and GEPA will pass the execution trace to the metric at scoring time, letting you penalize LM calls:++```python+LLM_CALL_PENALTY = 0.15++def metric(gold, pred, trace=None, pred_name=None, pred_trace=None, program_trace=None):+ correct = getattr(pred, "total_cents", None) == gold.total_cents+ n_calls = len(program_trace) if program_trace else 0+ score = max(0.0, (1.0 if correct else 0.0) - LLM_CALL_PENALTY * n_calls)+ fb = f"{'Correct' if correct else 'Wrong'} — used {n_calls} LM call(s). Settle clear cases in Python."+ return dspy.Prediction(score=score, feedback=fb)+```++The `program_trace` parameter is opt-in *by declaration*: only metrics that name it receive the trace. Keep the penalty small relative to correctness, so a decomposition has to *hold* accuracy to win.++## Sandboxed Execution++`Flex` always runs its generated code in a sandbox — never in the host Python process. `interpreter_factory` defaults to `dspy.PythonInterpreter` (Deno/Pyodide) and must be a **zero-argument factory** returning a fresh `CodeInterpreter`; a bare instance is not accepted, so parallel evaluations receive isolated sessions. The factory is called once per sandbox session, including separate sessions requested by nested code-executing modules. The code is authored by the reflection model, so isolating it keeps it from running with your host's full permissions. With the default interpreter, optimizer-authored control flow, string work, arithmetic, and supported imports run inside the sandbox, and only provided-tool calls, predictor construction, and predictor calls bridge back to the host, which makes the real LM calls.++Because the default builds a `PythonInterpreter`, *running* a `Flex` needs [Deno](https://deno.land/) installed; without it, the call raises.++```python+solve = dspy.Flex(+ "invoice: str -> total_cents: int",+ interpreter_factory=lambda: dspy.PythonInterpreter(), # the default; swap in your own CodeInterpreter factory here+)+```++Each call owns and shuts down every interpreter session it creates, so a `Flex` holds no live sessions between calls.++## Tools++Pass `tools` and the baseline starts as a `dspy.RLM` instead of a `dspy.Predict`.++```python+def lookup_sku(code: str) -> dict:+ """Look up a product by SKU."""+ return catalog[code]++solve = dspy.Flex("order: str -> total_cents: int", tools=[lookup_sku])+```++The optimizer can then wire your tools into `dspy.RLM(..., tools=[...])` / `dspy.ReAct(..., tools=[...])`, or call them directly from `forward`.++## Saving and Loading++A `Flex` serializes its `module_src`, so saving and loading a program restores the optimized code:++```python+optimized.save("solver.json")++restored = dspy.Flex("invoice: str -> total_cents: int")+restored.load("solver.json") # rebinds the saved module_src+```++The interpreter is a **runtime dependency and is not serialized**. Reconstructing with `dspy.Flex(signature)` restores the default sandbox automatically; if you optimized with a custom `interpreter_factory`, pass the same one when you reconstruct the module before calling `load`.++## Constructor Parameters++| Parameter | Type | Default | Description |+|-----------|------|---------|-------------|+| `signature` | `str \| Signature` | required | Declares the module's inputs and outputs (e.g. `"invoice -> total_cents: int"`). |+| `tools` | `list[Callable \| dspy.Tool]` | `None` | Tools the generated code may call. With tools, the baseline is a `dspy.RLM`; without, a `dspy.Predict`. |+| `interpreter_factory` | `Callable[[], CodeInterpreter]` | `PythonInterpreter` | Zero-arg factory returning a fresh `CodeInterpreter` for each sandbox session; defaults to `dspy.PythonInterpreter` (needs Deno). A bare interpreter instance is not accepted. Supported Python and libraries are interpreter-dependent. |+| `max_predictor_calls` | `int \| None` | `100` | Maximum number of predictor calls the generated code can make in one `forward` — a guard against runaway loops. `None` removes the limit. |++## Notes++!!! warning "Experimental"+ `Flex` is marked experimental. The API and the optimization behavior may change between releases; pin a version if you depend on it.++!!! note "Interpreter Requirements"+ `Flex` always runs generated code in a sandbox (`interpreter_factory` defaults to `dspy.PythonInterpreter`), which requires [Deno](https://deno.land/) for its Pyodide WASM sandbox — see the [RLM page](RLM.md#deno-installation) for installation notes.++## API Reference++<!-- START_API_REF -->+::: dspy.Flex+ handler: python+ options:+ members:+ - __init__+ - __call__+ - forward+ - module_src+ - signature+ - deepcopy+ - dump_state+ - get_lm+ - inspect_history+ - load+ - load_state+ - named_parameters+ - named_predictors+ - named_sub_modules+ - parameters+ - predictors+ - reset+ - reset_copy+ - save+ - set_lm+ show_source: true+ show_root_heading: true+ heading_level: 3+ docstring_style: google+ show_root_full_path: true+ show_object_full_path: false+ separate_signature: false+ inherited_members: true+<!-- END_API_REF -->diff --git a/docs/docs/api/modules/index.md b/docs/docs/api/modules/index.mdindex 70646b958..abb0e34b5 100644--- a/docs/docs/api/modules/index.md+++ b/docs/docs/api/modules/index.md@@ -6,6 +6,7 @@ API reference for DSPy modules. Select a page below.- [BestOfN](BestOfN.md)- [ChainOfThought](ChainOfThought.md)- [CodeAct](CodeAct.md)+- [Flex](Flex.md)- [Module](Module.md)- [MultiChainComparison](MultiChainComparison.md)- [Parallel](Parallel.md)diff --git a/docs/docs/diving-deeper/built-in-module-variants.md b/docs/docs/diving-deeper/built-in-module-variants.mdindex 31ee74138..e42d3fc40 100644--- a/docs/docs/diving-deeper/built-in-module-variants.md+++ b/docs/docs/diving-deeper/built-in-module-variants.md@@ -2,9 +2,9 @@## Intent-`Predict`, `ChainOfThought`, and `ReAct` cover most programs, but DSPy ships a handful of other modules for situations where one LM call is not enough, or where reasoning needs a Python runtime, or where you want to fan out across examples. This page collects those modules, groups them by what they’re for, and gives selection guidance so you know which one to reach for when.+`Predict`, `ChainOfThought`, and `ReAct` cover most programs, but DSPy ships a handful of other modules for situations where one LM call is not enough, or where reasoning needs a Python runtime, or where you want to fan out across examples, or where the module's structure itself is worth learning rather than hand-writing. This page collects those modules, groups them by what they’re for, and gives selection guidance so you know which one to reach for when.-Read this when a plain `Predict` or `ChainOfThought` isn’t getting you there and you’re trying to decide between sampling more, comparing drafts, executing code, or running the same module in parallel.+Read this when a plain `Predict` or `ChainOfThought` isn’t getting you there and you’re trying to decide between sampling more, comparing drafts, executing code, running the same module in parallel, or letting an optimizer discover the structure.## Design decisions@@ -48,6 +48,10 @@ A plain function — no LM call, no signature, no `dspy.Module`. It tallies compThe class is decorated with `@experimental` and the interface is still in flux. It composes a code sandbox with built-in `llm_query` / `llm_query_batched` tools that let generated code call a separate sub-LM mid-execution. The mental model is a Python REPL the LM drives, with another LM available as a callable inside. Useful, but the boundary conditions — max call counts, sandbox lifetime, error recovery — are still being worked out.+### 11. `Flex` puts the module's code into the optimization search space++Every other DSPy module fixes its structure at construction time; `Flex` doesn't. It holds its implementation as source code (`module_src`) and marks that code as the optimizable parameter, so `dspy.GEPA` rewrites its *entire implementation* — how many predictors, which primitives, what runs in Python instead of an LM — against your metric, rather than only tuning instructions. Construct it from a signature like any module (it begins as a one-call `Predict` baseline, or `RLM` when given tools), then let optimization discover the decomposition. Also experimental. It has its own deep dive: [Flex: Optimizable module code](flex.md).+## API walkthroughGrouped by what you’re trying to do.@@ -101,4 +105,5 @@ Wraps `ParallelExecutor` and submits each `(module, example)` pair to a thread p- [Modules: composing your own](modules.md) — every variant here is a `dspy.Module` (except `Parallel` and `majority`), so the composition rules apply.- [Tools, ReAct, and MCP](tools-react-and-mcp.md) — `CodeAct` and `RLM` use the same tool-wrapping machinery as `ReAct`.- [RLM: exploring large contexts with code](rlm.md) — the deep dive on the experimental REPL-driven module summarized above.+- [Flex: Optimizable module code](flex.md) — the deep dive on the code-optimizable Flex module summarized above.- [Settings and `context()`](settings-and-context.md) — how `Parallel` and `Module.batch` snapshot the active overrides into each worker.diff --git a/docs/docs/diving-deeper/flex.md b/docs/docs/diving-deeper/flex.mdnew file mode 100644index 000000000..e9fddbc3a--- /dev/null+++ b/docs/docs/diving-deeper/flex.md@@ -0,0 +1,119 @@+# Flex: Optimizable module code++## Intent++`dspy.Flex` is for tasks where you don't know the right *shape* of the solution up front. Every other module fixes its structure when you construct it — `Predict` is one call, `ChainOfThought` is one call with reasoning, `ReAct` is a tool loop — and optimization only tunes the prompt around that fixed structure. `Flex` moves the structure itself into the search space. You give it a signature; it starts as a `dspy.Predict` or `dspy.RLM` baseline; and `dspy.GEPA` rewrites its entire implementation — how many predictors, which primitives, what runs in Python instead of an LM — against your metric. Reach for it when the best decomposition is something you'd rather discover and optimize.++## Design decisions++### 1. A Flex is a module whose source code is the optimizable parameter++An ordinary module's tunable surface is its predictors' instructions. A `Flex`'s tunable surface is a whole `dspy.Module` subclass, held as a source string and exposed as `module_src`. That class has the usual two methods: `__init__` constructs whatever predictors it needs, and `forward` calls them and returns a `dspy.Prediction`. Optimizing the module means replacing that source with a better version — different predictors, different control flow, more or less Python. The prompt is no longer the unit of optimization; the program code is.++### 2. It drops into any signature and starts from a simple baseline++You construct `Flex` from any `dspy.Signature`, and it's immediately runnable. With no tools, its baseline source is a single `dspy.Predict` over the whole signature; with tools, a `dspy.RLM` so the baseline can call them. The baseline is the simplest thing that works as the starting point of a search.++### 3. GEPA discovers `Flex` by type and optimizes code instead of text++When GEPA compiles a program, it enumerates the `Flex` submodules and splits its work: each `Flex` becomes a **code component**, every other predictor stays an **instruction component**. Code components are seeded with their current `module_src` and evolved by a dedicated code proposer; instruction components are seeded with their current instructions and evolved by GEPA's usual instruction proposer. A custom `instruction_proposer` replaces the instruction proposer only; code components stay on the code proposer.++### 4. The code proposer reflects on whole-program behavior++Instruction optimization in GEPA is per-predictor: it looks at one predictor's inputs, outputs, and feedback. Code optimization can't work that way, because the predictors *are* part of what's being rewritten — they may not exist in the next candidate. So a code component reflects on the whole program's I/O instead: the module's inputs, its final prediction, and the metric's feedback per example. The proposer is given the signature, any available tools, a catalog of allowed primitives, the current source, and a batch of failing examples, and asked to return a full revised module class.++### 5. Predictors inside a Flex are owned by its code, not tuned in parallel++When you mix a `Flex` with ordinary modules in one program, GEPA optimizes the `Flex`'s code and the other predictors' instructions — but never the instructions of predictors that live *inside* the `Flex`. Those predictors are constructed by the current `module_src` and will be replaced wholesale by the next code candidate, so tuning their instructions would be optimizing something that may be overwritten.++`Flex` is a subclass of `Parameter` and `Module` (like `dspy.Predict`), so a parent program's `named_parameters()` yields it as one leaf and never recurses into it. A parent's `named_predictors()`, `predictors()`, `set_lm()`, and demo/instruction optimizers like `BootstrapFewShot` don't see the `Flex`'s internals. The `Flex` reports the same about itself: `flex.named_predictors()` is empty — its update unit is its code, not the predictors the code constructs. A `Flex` can carry its own LM (`flex.set_lm(...)`), which `forward` applies as the ambient LM for the whole call; otherwise its bridged predictor calls resolve the ambient LM at call time (`dspy.configure(lm=...)`, or a caller's `dspy.context(lm=...)`). `reset_copy()` resets a `Flex` the way it resets a `Predict`: the LM is cleared while `module_src` is kept, just as tuned instructions are.++### 6. A broken candidate scores as a failure, not a crash++The reflection model authors code, and code can be wrong. A candidate that doesn't parse raises when GEPA binds it to build the program for evaluation; `Flex` optimization catches that, logs it, and scores the whole batch at the failure score, letting the search continue and simply not select the broken candidate. Code that parses but breaks when it runs — an import that doesn't resolve, an edge case that throws on some inputs — fails per example at `forward` instead: each crashed example is scored at the failure score in its own slot, by example index, so the surviving scores stay aligned to the batch and GEPA's per-instance bookkeeping stays intact. Either way, the optimization is robust to the reflection model's mistakes by construction.++### 7. Generated code always runs in an interpreter, never in-process++`Flex` runs `module_src` in a sandbox: `interpreter_factory` defaults to `dspy.PythonInterpreter` (Deno/Pyodide) and must be a *zero-argument factory* — a bare instance or `None` is rejected. Since the code is authored by the reflection model, it never gets the host's full permissions: everything it does stays in the sandbox except provided-tool calls, predictor construction, and predictor calls, which bridge back to the host. The factory creates a fresh interpreter for each sandbox session; a forward owns an outer session and nested code-executing modules may request separate sessions. Source portability between custom interpreters is not guaranteed. `max_predictor_calls` caps how many predictor calls the generated code can make in one `forward`.++### 8. The declared output types are enforced at the sandbox boundary++Everything the generated code returns crosses back as JSON, so a field declared `int`, `list[str]`, or a pydantic model would arrive as a bare string or dict. `Flex` parses each declared output field against its annotation on the way out, so a `Flex` returns the same types as a `dspy.Predict` over the same signature and stays substitutable for one. Two things follow. Type names have to survive the round trip in *both* directions: the baseline renders custom types into its signature string (`text: str -> person: Person`) and the host resolves those names from the `Flex`'s own signature, because a signature crosses the boundary as text and dspy's usual caller-frame type lookup runs inside dspy, where your module is out of scope. And a candidate whose output doesn't match — wrong shape, or a declared field missing entirely — raises a `CodeInterpreterError` naming the field, which GEPA scores at the failure score for that example rather than handing the metric a `Prediction` that breaks somewhere else. An annotation that can't round-trip through a signature string at all (a `Callable`, say) is emitted untyped rather than rendered into source that wouldn't bind.++### 9. The code is state; the interpreter is a runtime dependency++A saved `Flex` is `{"module_src": ..., "lm": ...}` — the code, plus any LM set directly on the module. The internal predictors are not saved: they are derived from the code, and each `forward` reconstructs them from the bound source. The interpreter is a live runtime resource and is not serialized; `save(path, save_program=True)` cloudpickles the program with the bridge excluded, and loading rebuilds it. Reconstructing with `dspy.Flex(signature)` restores the default sandbox, so you only re-supply the `interpreter_factory` before `load` if you optimized with a custom one.++### 10. Flex is experimental and the interface is in flux++The class carries the `@experimental` decorator. Treat the API and serialization format as subject to change between minor releases and pin a version if you depend on it.++## API walkthrough++### Defining and running a Flex++**`dspy.Flex(signature, *, tools=None, interpreter_factory=PythonInterpreter, max_predictor_calls=100)`**+Parses the signature and binds the baseline source — a single `dspy.Predict` over the signature, or a `dspy.RLM` when `tools` are given. Validates `interpreter_factory` (a zero-arg factory, defaulting to `dspy.PythonInterpreter`), and sets up the sandbox bridge.++**`__call__(**inputs)` / `forward(**inputs)`**+Runs the currently bound source inside the interpreter, bridging predictor calls back to the host. Returns a `dspy.Prediction` over the signature's output fields, and accepts keyword inputs only.++**`module_src`**+A read-only property holding the current implementation as source — one `dspy.Module` subclass. This is the value GEPA reads as the seed and overwrites with each accepted candidate.++### Optimizing with GEPA++**`dspy.GEPA(metric=..., reflection_lm=..., ...).compile(flex_program, trainset=..., valset=...)`**+Compiling a program that contains one or more `Flex` submodules optimizes each `Flex`'s code and every non-flex predictor's instructions together, under one budget. Returns a new program whose `module_src` (per flex submodule) is the best code found. The auto-budget counts each flex submodule as one component alongside the instruction predictors.++**The metric's `feedback`**+As with any GEPA metric, the feedback string goes into the prompt handed to the proposer — here, the *code* proposer. Feedback that diagnoses *why* an output was wrong and hints at structure steers rewrites far better than a bare score.++**Trace-aware metrics (`program_trace`)**+Add `program_trace=None` as a sixth parameter to your metric and GEPA passes the execution trace at scoring time, so you can score against how the answer was produced. For instance, you may use `len(program_trace)` as an LM-call count to fold a small per-call penalty into the score.++### Tools and sandboxing++**`tools=[...]`**+Plain functions or `dspy.Tool` instances, referenced by name in the generated code, so each name must be a valid Python identifier. Providing tools makes the baseline a `dspy.RLM` and tells the code proposer they are in scope — it can wire them into `dspy.RLM`/`dspy.ReAct`, call them directly, or supplement them with its own inline helpers.++**`interpreter_factory=...`**+Defaults to `dspy.PythonInterpreter` (sandboxed, needs Deno). Must be a zero-argument callable returning a fresh `CodeInterpreter` for each sandbox session; parallel evaluations and nested code-executing modules can therefore receive isolated sessions. As in `dspy.RLM`, a bare interpreter instance is not accepted. This low-level hook does not guarantee source or standard-library portability between different interpreters.++**`max_predictor_calls`**+The maximum number of predictor calls the generated code can make in one `forward`. It guards against runaway loops. `None` removes the limit.++### What the generated code can use++The optimizer-authored code does not run against the real `dspy` package. Inside the sandbox, `dspy` is a small shim whose job is to hand predictor construction and predictor calls back to the host — so what it exposes is narrower than the library, and it is the same surface the code proposer is told about.++**Available**++- `dspy.Module` — the base class the generated source subclasses. `__init__` assigns predictors; `forward` runs in the sandbox.+- `dspy.Predict`, `dspy.ChainOfThought`, `dspy.ReAct`, `dspy.ReActV2`, `dspy.RLM`, `dspy.CodeAct`, `dspy.ProgramOfThought` — constructed in the sandbox but *built on the host*, where the real LM calls happen. Constructor kwargs cross as JSON.+- `dspy.Signature("inputs -> outputs", "instructions")` — the string form only, used to give an inner predictor its instructions. It is a marker the host turns back into a real `Signature`, not the class: it has no methods, no `with_instructions()`, and no `dspy.InputField` / `dspy.OutputField` class form.+- `dspy.Prediction(**fields)` — the return value of `forward`. Holds fields; it is not the host `Prediction` type.+- `dspy.Tool(func)` — a pass-through wrapper. Only tools you passed to `dspy.Flex(tools=...)` can be handed to a bridged sub-predictor; they are already in scope by name, so wrapping is optional.+- The Python standard library, imported *inside* `forward` (the interpreter's own sandboxing still applies — `dspy.PythonInterpreter` has no filesystem or network access by default).++**Not available**++- Adapters (`dspy.ChatAdapter`, `dspy.JSONAdapter`, …). The host formats and parses every bridged call; the generated code never sees a prompt or a raw completion.+- `dspy.settings`, `dspy.context`, `dspy.configure`, `dspy.LM`. The LM is whatever the host has configured; generated code cannot select or reconfigure one.+- `dspy.Example`, `dspy.Evaluate`, the optimizers, retrievers, and `dspy.Flex` itself — no nesting.+- Class-based signatures and typed field declarations.+- Host objects generally: values cross as JSON, so a tool you define inside `forward` cannot be passed to a bridged sub-predictor, and a predictor field that isn't JSON-serializable raises rather than silently degrading.++Anything outside this surface fails when the candidate runs, and GEPA scores it at the failure score — a missing name costs a search step rather than crashing the run. The shim lives in `dspy/predict/flex/_sandbox_shim.py` and the proposer-facing version of this list is `dspy/predict/flex/primitives_doc.py`.++### Saving and loading++**`save(path)` / `load(path)` / `dump_state()` / `load_state(state)`**+`module_src` travels in the serialized state (along with any LM set on the module), so loading restores the optimized code; the predictors are rebuilt from it on each `forward`. The interpreter is not serialized; reconstructing with `dspy.Flex(signature)` restores the default sandbox, so re-supply it in the constructor before `load` only if you used a custom `interpreter_factory`.++## Cross-links++- [`dspy.Flex` API reference](../api/modules/Flex.md) — constructor table, examples, and the full method list.+- [Built-in module variants](built-in-module-variants.md) — where `Flex` sits among the other non-`Predict` modules.+- [GEPA in depth](gepa-in-depth.md) — the reflective optimizer that drives `Flex`'s code search.+- [RLM: exploring large contexts with code](rlm.md) — the module `Flex` uses as its tool-enabled baseline.diff --git a/docs/mkdocs.yml b/docs/mkdocs.ymlindex 0c2121b14..9b20f9c42 100644--- a/docs/mkdocs.yml+++ b/docs/mkdocs.yml@@ -28,6 +28,7 @@ nav:- "Modules: composing your own": diving-deeper/modules.md- Built-in module variants: diving-deeper/built-in-module-variants.md- "RLM: exploring large contexts with code": diving-deeper/rlm.md+ - "Flex: optimizable module code": diving-deeper/flex.md- "Tools, ReAct, and MCP": diving-deeper/tools-react-and-mcp.md- Metrics and evaluation: diving-deeper/metrics-and-evaluation.md- "Optimizers: choosing one": diving-deeper/choosing-an-optimizer.md@@ -128,6 +129,7 @@ nav:- BestOfN: api/modules/BestOfN.md- ChainOfThought: api/modules/ChainOfThought.md- CodeAct: api/modules/CodeAct.md+ - Flex: api/modules/Flex.md- Module: api/modules/Module.md- MultiChainComparison: api/modules/MultiChainComparison.md- Parallel: api/modules/Parallel.md@@ -243,6 +245,7 @@ plugins:- mkdocstrings:handlers:python:+ paths: [".."]options:docstring_style: googleshow_source: truediff --git a/dspy/__init__.py b/dspy/__init__.pyindex 1525ef4a6..3b29e08ed 100644--- a/dspy/__init__.py+++ b/dspy/__init__.py@@ -3,6 +3,7 @@ from dspy.primitives import *from dspy.retrievers import *from dspy.signatures import *from dspy.teleprompt import *+from dspy.predict.flex import Flexfrom dspy.evaluate import Evaluate # isort: skipfrom dspy.clients import * # isort: skipdiff --git a/dspy/predict/flex/__init__.py b/dspy/predict/flex/__init__.pynew file mode 100644index 000000000..c0a52f6c6--- /dev/null+++ b/dspy/predict/flex/__init__.py@@ -0,0 +1,3 @@+from dspy.predict.flex.flex import Flex++__all__ = ["Flex"]diff --git a/dspy/predict/flex/_sandbox_shim.py b/dspy/predict/flex/_sandbox_shim.pynew file mode 100644index 000000000..f25631dff--- /dev/null+++ b/dspy/predict/flex/_sandbox_shim.py@@ -0,0 +1,127 @@+"""Sandbox side of the dspy.Flex bridge: a stand-in ``dspy`` module for optimizer-authored code.++``BridgeRuntime.forward`` executes this source in each per-forward interpreter before the bound+``module_src``, so the generated module runs sandboxed while predictors are built and called on+the host:++- ``dspy.Predict`` and other allowed modules return a ``_DspyPending``; construction waits for attribute assignment,+ because the attribute name is the predictor's host-side handle.+- ``_DspyModule.__setattr__`` has the host build the real predictor (``__dspy_construct__``) and+ binds a ``_DspyProxy`` in its place. Calling the proxy runs the predictor (``__dspy_call__``)+ and wraps the returned output fields in a ``_DspyPrediction``.+- Callables and signatures cannot cross the JSON boundary as themselves, so they travel as+ markers the host resolves: tools by name (``__dspy_tool__``), ``dspy.Signature(...)`` results+ as ``__dspy_sig__`` payloads.++Every name here is ``_dspy``-prefixed to stay clear of generated code; ``FlexContext`` rejects+user tool names in that namespace.+"""++import sys as _dspy_sys+import types as _dspy_types+++def _dspy_host(_fn, **_kw):+ # Call a registered host tool by name (the CodeInterpreter.tools contract) and return its result.+ return globals()[_fn](**_kw)+++class _DspyPrediction:+ """Sandbox-side stand-in for dspy.Prediction; just holds output fields."""++ def __init__(self, **_fields):+ object.__setattr__(self, "_fields", dict(_fields))++ def __getattr__(self, _name):+ _f = object.__getattribute__(self, "_fields")+ if _name in _f:+ return _f[_name]+ raise AttributeError(_name)++ def __getitem__(self, _name):+ return object.__getattribute__(self, "_fields")[_name]++ def __repr__(self):+ return "Prediction(" + repr(object.__getattribute__(self, "_fields")) + ")"+++class _DspyProxy:+ """Sandbox-side handle to a host predictor. Calling it runs the real predictor on the host."""++ def __init__(self, _handle):+ object.__setattr__(self, "_handle", _handle)++ def __call__(self, **_inputs):+ _h = object.__getattribute__(self, "_handle")+ _out = _dspy_host("__dspy_call__", handle=_h, inputs=_inputs)+ return _DspyPrediction(**(_out or {}))+++class _DspyPending:+ """Returned by a shim constructor before the attribute name is known (captured in __setattr__)."""++ def __init__(self, _kind, _sig, _kwargs):+ self.kind = _kind+ self.sig = _sig+ self.kwargs = _kwargs+++class _DspyModule:+ def __init__(self, *_a, **_k):+ pass++ def __setattr__(self, _name, _value):+ if isinstance(_value, _DspyPending):+ _h = _dspy_host(+ "__dspy_construct__",+ kind=_value.kind,+ signature=_value.sig,+ attr_name=_name,+ kwargs=_value.kwargs,+ )+ _value = _DspyProxy(_h)+ object.__setattr__(self, _name, _value)++ def __call__(self, **_kw):+ return self.forward(**_kw)+++def _dspy_enc(_v):+ # Tool references (e.g. tools=[shout]) are sandbox functions; send them to the host by name.+ if callable(_v) and hasattr(_v, "__name__"):+ return {"__dspy_tool__": _v.__name__}+ if isinstance(_v, (list, tuple)):+ return [_dspy_enc(_x) for _x in _v]+ if isinstance(_v, dict):+ return {_k: _dspy_enc(_x) for _k, _x in _v.items()}+ return _v+++def _dspy_make_ctor(_kind):+ def _ctor(signature=None, **_kwargs):+ return _DspyPending(_kind, signature, {_k: _dspy_enc(_v) for _k, _v in _kwargs.items()})++ return _ctor+++def _dspy_signature(signature, instructions=None, **_kw):+ return {"__dspy_sig__": True, "signature": signature, "instructions": instructions}+++def _dspy_tool(func, **_kw):+ return func+++_dspy = _dspy_types.ModuleType("dspy")+_dspy.Module = _DspyModule+_dspy.Prediction = _DspyPrediction+_dspy.Signature = _dspy_signature+_dspy.Tool = _dspy_tool+for _k in ("Predict", "ChainOfThought", "RLM", "CodeAct", "ProgramOfThought", "ReAct", "ReActV2"):+ setattr(_dspy, _k, _dspy_make_ctor(_k))+dspy = _dspy++# Register as the importable ``dspy`` only inside the sandbox, where the registered host tools are+# present in globals().+if "__dspy_construct__" in globals():+ _dspy_sys.modules["dspy"] = _dspydiff --git a/dspy/predict/flex/bridge.py b/dspy/predict/flex/bridge.pynew file mode 100644index 000000000..bc8bb59a0--- /dev/null+++ b/dspy/predict/flex/bridge.py@@ -0,0 +1,373 @@+"""Host side of the dspy.Flex sandbox bridge.++A ``Flex`` never executes its optimizer-authored ``module_src`` in the host process. Instead,+every ``forward``:++1. creates a fresh interpreter from the Flex's ``interpreter_factory`` (``BridgeRuntime.forward``);+2. injects the sandbox-side shim (``_sandbox_shim.py``), which fakes a tiny ``dspy`` module whose+ predictor constructors and calls are proxies;+3. executes ``module_src`` and drives its ``forward`` with the call's inputs.++Only three things cross the sandbox boundary, all as JSON over the interpreter's tool-call+protocol (``CodeInterpreter.tools``):++- ``__dspy_construct__``: the shim asks the host to build a real predictor (``_Invocation.construct``).+ The host constructs it from the serialized signature/kwargs and returns a string handle.+- ``__dspy_call__``: the shim runs a predictor by handle (``_Invocation.call``); the host makes the+ real LM call and returns the prediction's fields as JSON.+- user tools passed to ``dspy.Flex(tools=...)``, registered by name so sandbox code can call them+ directly; the callables themselves stay on the host.++All per-forward state, including the constructed predictors, the predictor-call budget, custom-type+originals, and the last LM infrastructure error, lives in a ``_Invocation`` created for that+forward, never on the shared ``Flex``, so concurrent forwards (e.g. threaded evaluation) are+isolated. The final ``dspy.Prediction`` is parsed against the Flex signature's declared output+types on the way out (``BridgeRuntime._to_prediction``).+"""++from __future__ import annotations++import ast+import functools+import inspect+import json+import logging+import types+from pathlib import Path+from typing import Any, Callable, Union, get_args, get_origin++from pydantic import TypeAdapter+from pydantic_core import PydanticSerializationError, to_jsonable_python++import dspy+from dspy import CodeInterpreterError+from dspy.adapters.types.base_type import Type as _CustomType+from dspy.adapters.utils import parse_value+from dspy.primitives.code_interpreter import _create_interpreter+from dspy.signatures.signature import make_signature+from dspy.utils.exceptions import LMError++logger = logging.getLogger(__name__)++# Tool names the shim uses to call the host; must match the names registered in BridgeRuntime.+CONSTRUCT_TOOL = "__dspy_construct__"+CALL_TOOL = "__dspy_call__"++# Predictors the sandbox shim may construct and the host builds+BRIDGEABLE_KINDS = ("Predict", "ChainOfThought", "RLM", "CodeAct", "ProgramOfThought", "ReAct", "ReActV2")+# The shim's dspy.Signature(...) emits this marker so the host can rebuild a Signature.+SIGNATURE_MARKER = "__dspy_sig__"+# The shim passes tools by name (callables can't cross the JSON boundary); the host resolves the name+# back to the real tool object passed to dspy.Flex(tools=...).+TOOL_MARKER = "__dspy_tool__"+# Variable/identifier names used in the per-forward driver code (namespaced to avoid clashing with+# whatever the optimizer-authored module uses).+_INPUTS_VAR = "__dspy_flex_inputs"+_INSTANCE_VAR = "__dspy_flex_instance"+_OUT_VAR = "__dspy_flex_out"+_JSON_VAR = "__dspy_flex_json"+++# The sandbox-side dspy shim, injected as text into each per-forward interpreter.+SHIM_SETUP = (Path(__file__).parent / "_sandbox_shim.py").read_text(encoding="utf-8")+++def parse_module_class_name(module_src: str) -> str:+ """Return the name of the generated ``dspy.Module`` subclass in ``module_src``."""+ tree = ast.parse(module_src)+ classes = [node for node in tree.body if isinstance(node, ast.ClassDef)]+ with_forward = [+ c for c in classes if any(isinstance(b, ast.FunctionDef) and b.name == "forward" for b in c.body)+ ]+ chosen = with_forward or classes+ if not chosen:+ raise CodeInterpreterError("module_src must define a dspy.Module subclass with a `forward` method")+ return chosen[0].name+++def _accepts_interpreter_factory(cls: type) -> bool:+ """True if ``cls.__init__`` takes an ``interpreter_factory`` parameter."""+ try:+ return "interpreter_factory" in inspect.signature(cls.__init__).parameters+ except (TypeError, ValueError):+ return False+++def _resolve_signature(signature: Any, custom_types: dict[str, type] | None = None) -> Any:+ """Turn a shim signature payload back into something a host predictor accepts."""+ if isinstance(signature, dict) and signature.get(SIGNATURE_MARKER):+ # marker always carries a string signature; make_signature applies instructions if given+ return make_signature(signature["signature"], signature.get("instructions"), custom_types=custom_types)+ if isinstance(signature, str):+ return make_signature(signature, custom_types=custom_types)+ return signature+++def _jsonable(value: Any) -> Any:+ """Coerce a predictor output field to a JSON-serializable value."""+ try:+ return to_jsonable_python(value)+ except PydanticSerializationError:+ return value+++def prediction_to_fields(pred: Any) -> dict[str, Any]:+ """Serialise a host ``dspy.Prediction`` (or dict) to a JSON-able field dict for the sandbox."""+ store = getattr(pred, "_store", None)+ if store is None:+ if isinstance(pred, dict):+ store = pred+ else:+ raise CodeInterpreterError(+ f"A bridged predictor must return a dspy.Prediction; got {type(pred).__name__}"+ )+ fields = {k: _jsonable(v) for k, v in dict(store).items()}+ try:+ json.dumps(fields)+ except TypeError as e:+ raise CodeInterpreterError(+ "A bridged predictor returned a field that cannot cross the sandbox boundary "+ f"(must be JSON-serializable): {e}"+ ) from e+ return fields+++def _tool_entrypoint(tool: Any) -> Callable[..., Any]:+ """A callable for the interpreter's tool registry."""+ func = getattr(tool, "func", None)+ if func is None:+ return tool++ @functools.wraps(func)+ def entrypoint(**kwargs: Any) -> Any:+ return tool(**kwargs)++ return entrypoint+++def _restoring_entrypoint(fn: Callable[..., Any], originals: dict[str, Any]) -> Callable[..., Any]:+ """Wrap a tool entrypoint so serialized custom-type inputs arrive as the original objects."""++ @functools.wraps(fn)+ def entrypoint(**kwargs: Any) -> Any:+ return fn(**{k: _restore_custom_types(v, originals) for k, v in kwargs.items()})++ return entrypoint+++def _collect_custom_type_originals(value: Any, out: dict[str, Any]) -> None:+ """Record custom-type instances by their serialized string, recursing into containers."""+ if isinstance(value, _CustomType):+ out[value.serialize_model()] = value+ elif isinstance(value, dict):+ for v in value.values():+ _collect_custom_type_originals(v, out)+ elif isinstance(value, (list, tuple)):+ for v in value:+ _collect_custom_type_originals(v, out)+++def _restore_custom_types(value: Any, originals: dict[str, Any]) -> Any:+ """Substitute serialized custom-type strings back with the original host objects."""+ if isinstance(value, str) and value in originals:+ return originals[value]+ if isinstance(value, dict):+ return {k: _restore_custom_types(v, originals) for k, v in value.items()}+ if isinstance(value, list):+ return [_restore_custom_types(v, originals) for v in value]+ return value+++def _is_field_optional(annotation: Any) -> bool:+ """True if the field annotation is optional, a union admitting None."""+ return annotation is type(None) or (+ get_origin(annotation) in (Union, types.UnionType) and type(None) in get_args(annotation)+ )+++class _Invocation:+ """Per-forward bridge state: the predictors this forward constructed (keyed by the sandbox+ attribute name), its predictor-call budget, and the original custom-type objects to restore+ on bridged calls. Each forward gets its own ``_Invocation``."""++ def __init__(self, runtime: BridgeRuntime, originals: dict[str, Any]) -> None:+ self._runtime = runtime+ self._originals = originals+ self._predictors: dict[str, Any] = {}+ self._calls = 0+ self._lm_error: tuple[LMError, str] | None = None++ def construct(self, kind: str, signature: Any, attr_name: str, kwargs: dict[str, Any] | None = None) -> str:+ self._lm_error = None+ if kind not in BRIDGEABLE_KINDS:+ raise CodeInterpreterError(+ f"dspy.{kind} is not supported inside a sandboxed dspy.Flex yet "+ f"(bridgeable: {', '.join(BRIDGEABLE_KINDS)})"+ )+ self._predictors[attr_name] = self._runtime._build_predictor(kind, signature, kwargs)+ return attr_name++ def call(self, handle: str, inputs: dict[str, Any] | None = None) -> dict[str, Any]:+ self._lm_error = None+ self._calls += 1+ budget = self._runtime._max_predictor_calls+ if budget is not None and self._calls > budget:+ raise CodeInterpreterError(+ f"Sandboxed dspy.Flex forward exceeded its predictor-call budget "+ f"({budget}). Raise max_predictor_calls if this is expected."+ )+ predictor = self._predictors.get(handle)+ if predictor is None:+ raise CodeInterpreterError(f"Unknown predictor handle: {handle!r}")+ restored = {k: _restore_custom_types(v, self._originals) for k, v in (inputs or {}).items()}+ try:+ return prediction_to_fields(predictor(**restored))+ except LMError as e:+ tag = f"[dspy bridge lm-error #{self._calls}]"+ self._lm_error = (e, tag)+ raise CodeInterpreterError(f"{tag} {type(e).__name__}: {e}") from e+++class BridgeRuntime:+ """Host side of the dspy.Flex bridge: runs the bound ``module_src`` for one ``Flex``.++ Each ``forward`` creates a fresh interpreter from the factory, executes the shim and the+ bound source, runs the module instance, and shuts the interpreter down. Predictors are built+ host-side but held by that forward's ``_Invocation`` — never set on the ``Flex`` — so nothing+ outlives the call and parallel forwards stay isolated. ``_max_predictor_calls`` caps bridged+ predictor calls per forward.++ User ``tools`` are registered with the interpreter, callable by name from the sandbox, and+ resolved by name when handed to a bridged sub-predictor; functions defined inside the+ generated module stay in the sandbox. A code-executing sub-predictor (RLM, CodeAct,+ ProgramOfThought) receives the same interpreter factory, so its inner code runs on the+ backend chosen for the Flex.+ """++ def __init__(self, flex: Any, factory: Callable[[], Any], max_predictor_calls: int | None = 100) -> None:+ self._flex = flex+ self._factory = factory+ self._max_predictor_calls = max_predictor_calls+ self._module_src: str | None = None+ self._class_name: str | None = None++ def bind(self, module_src: str) -> None:+ """Record new source; it runs at the next ``forward``."""+ self._module_src = module_src+ self._class_name = parse_module_class_name(module_src)++ def forward(self, inputs: dict[str, Any]) -> Any:+ originals: dict[str, Any] = {}+ for v in inputs.values():+ _collect_custom_type_originals(v, originals)++ invocation = _Invocation(self, originals)+ interp = _create_interpreter(self._factory)+ try:+ interp.tools.update({CONSTRUCT_TOOL: invocation.construct, CALL_TOOL: invocation.call})+ # User tools callable by name in the sandbox.+ interp.tools.update(+ {name: _restoring_entrypoint(fn, originals) for name, fn in self._tool_callables().items()}+ )+ interp.execute(SHIM_SETUP)+ interp.execute(self._module_src) # defines the class in the sandbox+ code = (+ f"{_INSTANCE_VAR} = {self._class_name}()\n"+ f"{_OUT_VAR} = {_INSTANCE_VAR}.forward(**{_INPUTS_VAR})\n"+ f"import json as {_JSON_VAR}\n"+ f"{_JSON_VAR}.dumps({_OUT_VAR}._fields if hasattr({_OUT_VAR}, '_fields') else {_OUT_VAR})"+ )+ result = interp.execute(code, variables={_INPUTS_VAR: dict(inputs)})+ except CodeInterpreterError as e:+ if invocation._lm_error is not None:+ lm_error, tag = invocation._lm_error+ if tag in str(e):+ raise lm_error from e+ raise+ finally:+ try:+ interp.shutdown()+ except Exception:+ logger.warning("dspy.Flex: interpreter.shutdown() raised after forward", exc_info=True)+ if not isinstance(result, str) or not result:+ raise CodeInterpreterError(+ "Sandboxed forward returned no serializable result; the generated forward must return "+ f"a dspy.Prediction (got {result!r})"+ )+ return self._to_prediction(json.loads(result))++ def _to_prediction(self, fields: dict[str, Any]) -> Any:+ signature = self._flex.signature+ out = dict(fields)+ filled: set[str] = set()+ missing: list[str] = []+ for name, field in signature.output_fields.items():+ if name in out:+ continue+ if not field.is_required():+ out[name] = field.get_default(call_default_factory=True)+ filled.add(name)+ elif _is_field_optional(field.annotation):+ out[name] = None+ filled.add(name)+ else:+ missing.append(name)+ if missing:+ raise CodeInterpreterError(+ f"Sandboxed forward returned a dspy.Prediction missing required output field(s) "+ f"{missing}; the signature declares {list(signature.output_fields)}."+ )+ for name, field in signature.output_fields.items():+ if name in filled:+ continue+ try:+ if out[name] is None:+ out[name] = TypeAdapter(field.annotation).validate_python(None)+ else:+ out[name] = parse_value(out[name], field.annotation)+ except Exception as e:+ raise CodeInterpreterError(+ f"Sandboxed forward returned {out[name]!r} for output field {name!r}, which is "+ f"not a valid {field.annotation}: {e}"+ ) from e+ return dspy.Prediction(**out)++ # -- host-side bridge callbacks --------------++ def _tool_callables(self) -> dict[str, Callable[..., Any]]:+ """User tools as ``name -> callable`` to register so sandbox code can call them."""+ return {name: _tool_entrypoint(tool) for name, tool in self._flex._flex_ctx.context_names().items()}++ def _resolve_tool(self, name: str) -> Any:+ ctx = self._flex._flex_ctx.context_names()+ if name not in ctx:+ raise CodeInterpreterError(+ f"Sandboxed code referenced tool {name!r}, which was not passed to dspy.Flex(tools=...). "+ "Tools authored inside the generated module cannot be handed to a bridged sub-predictor."+ )+ return ctx[name]++ def _decode_tools(self, value: Any) -> Any:+ """Turn the shim's tool name-markers back into the real host tool objects."""+ if isinstance(value, dict) and TOOL_MARKER in value:+ return self._resolve_tool(value[TOOL_MARKER])+ if isinstance(value, list):+ return [self._decode_tools(v) for v in value]+ if isinstance(value, dict):+ return {k: self._decode_tools(v) for k, v in value.items()}+ return value++ def _sub_interpreter_factory(self) -> Any:+ return self._factory++ def _build_predictor(self, kind: str, signature: Any, kwargs: dict[str, Any] | None) -> Any:+ cls = getattr(dspy, kind)+ extra = {k: self._decode_tools(v) for k, v in (kwargs or {}).items()}+ # A code-executing sub-predictor should run its inner code in the backend chosen for Flex, so+ # hand it the Flex interpreter factory (it makes and tears down a fresh interpreter per forward).+ # The sandbox code can't set this itself, since a live interpreter can't cross the boundary.+ if "interpreter_factory" not in extra and _accepts_interpreter_factory(cls):+ factory = self._sub_interpreter_factory()+ if factory is not None:+ extra["interpreter_factory"] = factory+ return cls(_resolve_signature(signature, self._flex._flex_ctx.custom_types()), **extra)diff --git a/dspy/predict/flex/ctx.py b/dspy/predict/flex/ctx.pynew file mode 100644index 000000000..d998967f0--- /dev/null+++ b/dspy/predict/flex/ctx.py@@ -0,0 +1,176 @@+from __future__ import annotations++import ast+import types+import typing+from dataclasses import dataclass, field+from typing import Any, get_args, get_origin+++@dataclass+class FlexContext:+ """Signature + tools that the baseline and code-optimization prompts render from."""++ signature_cls: type+ tools: list[Any] = field(default_factory=list)++ def context_names(self) -> dict[str, Any]:+ """Names to inject into the exec globals for the generated code."""+ out: dict[str, Any] = {}+ for tool in self.tools:+ name = getattr(tool, "name", None) or getattr(tool, "__name__", None)+ if not name or not name.isidentifier():+ raise ValueError(+ f"Tool {tool!r} needs a name that is a valid Python identifier — it is "+ f"referenced by name in the generated code — but got {name!r}. Use a `def` "+ f"function or `dspy.Tool(func, name='my_tool')`."+ )+ if name == "dspy" or name.startswith(("_dspy", "__dspy")):+ raise ValueError(+ f"Tool name {name!r} is reserved: 'dspy' and names starting with '_dspy' belong "+ f"to the dspy.Flex sandbox. Rename it, e.g. `dspy.Tool(func, name='my_tool')`."+ )+ out[name] = tool+ return out++ def render_signature_spec(self) -> str:+ cls = self.signature_cls+ name = getattr(cls, "__name__", "AnonymousSignature")+ objective = getattr(cls, "instructions", "") or ""++ def _fmt(fields_dict: dict[str, Any]) -> str:+ lines: list[str] = []+ for fname, finfo in fields_dict.items():+ type_str = _type_name(finfo.annotation)+ extra = finfo.json_schema_extra or {}+ desc = extra.get("desc", "")+ line = f" - {fname}: {type_str}"+ if desc and not desc.startswith("${"):+ line += f" -- {desc}"+ lines.append(line)+ return "\n".join(lines) if lines else " (none)"++ return (+ f"Signature: {name}\n"+ f"Objective (docstring): {objective}\n"+ f"Input fields:\n{_fmt(cls.input_fields)}\n"+ f"Output fields:\n{_fmt(cls.output_fields)}\n"+ )++ def render_signature_string(self) -> str:+ """Render a parseable ``"in: T, in2 -> out: T2"`` string for the baseline."""+ cls = self.signature_cls+ custom = self.custom_types()++ def _render(fields_dict: dict[str, Any]) -> str:+ parts: list[str] = []+ for fname, finfo in fields_dict.items():+ try:+ parts.append(f"{fname}: {render_annotation(finfo.annotation, custom)}")+ except ValueError:+ parts.append(fname)+ return ", ".join(parts)++ return f"{_render(cls.input_fields)} -> {_render(cls.output_fields)}"++ def custom_types(self) -> dict[str, type]:+ cls = self.signature_cls+ out: dict[str, type] = {}+ for finfo in {**cls.input_fields, **cls.output_fields}.values():+ _collect_custom_types(finfo.annotation, out)+ return out++ def render_context_blurb(self, sandboxed: bool = False) -> str:+ parts: list[str] = []+ if self.tools:+ tool_lines: list[str] = []+ for tool in self.tools:+ tname = getattr(tool, "name", None) or getattr(tool, "__name__", "?")+ tdesc = getattr(tool, "desc", None) or getattr(tool, "__doc__", "") or ""+ tdesc = tdesc.strip().splitlines()[0] if tdesc else ""+ tool_lines.append(f" - {tname}: {tdesc}")+ parts.append("Available tools (in scope by name):\n" + "\n".join(tool_lines))+ if sandboxed:+ parts.append(+ "This module runs in a sandbox: only the tools listed above may be passed to "+ "dspy.ReAct/dspy.RLM (their functions live on the host). A function you define inside "+ "the module can be called directly in forward, but cannot be handed to those predictors."+ )+ return "\n\n".join(parts) if parts else "(no extra context)"+++def _type_name(t: Any) -> str:+ if t is None:+ return "None"+ name = getattr(t, "__name__", None)+ if name:+ return name+ return str(t).replace("typing.", "")+++def _collect_custom_types(annotation: Any, out: dict[str, type]) -> None:+ """Add every non-builtin type named anywhere in ``annotation`` to ``out``, keyed by name."""+ for arg in get_args(annotation):+ _collect_custom_types(arg, out)+ if get_origin(annotation) is not None or annotation is type(None):+ return+ name = getattr(annotation, "__name__", None)+ if isinstance(annotation, type) and name and name.isidentifier() and annotation.__module__ != "builtins":+ out[name] = annotation+++_BUILTIN_TYPES = (int, str, float, bool, list, tuple, dict, set, frozenset, complex, bytes, bytearray)+++def render_annotation(annotation: Any, custom_types: dict[str, type] | None = None) -> str:+ """Render ``annotation`` as a signature-string type token, the inverse of dspy's+ ``_parse_type_node``."""+ return ast.unparse(_render_type_node(annotation, custom_types or {}))+++def _render_type_node(annotation: Any, custom_types: dict[str, type]) -> ast.expr:+ """Build the annotation AST for ``annotation``, emitting only nodes ``_parse_type_node`` handles."""+ if annotation is type(None):+ return ast.Constant(value=None)++ origin = get_origin(annotation)+ if origin is None:+ name = getattr(annotation, "__name__", None)+ if annotation in _BUILTIN_TYPES:+ return ast.Name(id=annotation.__name__)+ if annotation is Any or (name and typing.__dict__.get(name) is annotation):+ return ast.Name(id=name or "Any")+ if isinstance(annotation, type) and name and custom_types.get(name) is annotation:+ return ast.Name(id=name)+ raise ValueError(+ f"Annotation {annotation!r} has no name the signature-string grammar can resolve; "+ "pass it via custom_types to make it renderable."+ )++ args = get_args(annotation)+ if origin in (typing.Union, types.UnionType):+ node = _render_type_node(args[0], custom_types)+ for arg in args[1:]:+ node = ast.BinOp(left=node, op=ast.BitOr(), right=_render_type_node(arg, custom_types))+ return node+ if origin is typing.Literal:+ if not all(v is None or isinstance(v, (str, int, bool, bytes)) for v in args):+ raise ValueError(f"Literal values in {annotation!r} are not signature-string constants.")+ elts: list[ast.expr] = [ast.Constant(value=v) for v in args]+ return ast.Subscript(value=ast.Name(id="Literal"), slice=ast.Tuple(elts=elts) if len(elts) > 1 else elts[0])++ base = _render_type_node(origin, custom_types)+ elts = [ast.Constant(value=...) if a is Ellipsis else _render_type_node(a, custom_types) for a in args]+ return ast.Subscript(value=base, slice=ast.Tuple(elts=elts) if len(elts) > 1 else elts[0])+++def _strip_code_fences(s: str) -> str:+ s = (s or "").strip()+ if s.startswith("```"):+ nl = s.find("\n")+ if nl != -1:+ s = s[nl + 1 :]+ if s.endswith("```"):+ s = s[:-3]+ # Normalize tabs so exec doesn't choke on mixed tabs/spaces from the LM's output.+ return s.strip().expandtabs(4)diff --git a/dspy/predict/flex/flex.py b/dspy/predict/flex/flex.pynew file mode 100644index 000000000..554ddb19f--- /dev/null+++ b/dspy/predict/flex/flex.py@@ -0,0 +1,168 @@+from __future__ import annotations++from typing import Any, Callable++from dspy.clients.base_lm import BaseLM+from dspy.dsp.utils.settings import settings+from dspy.predict.flex.bridge import BridgeRuntime+from dspy.predict.flex.ctx import FlexContext+from dspy.predict.parameter import Parameter+from dspy.predict.predict import _sanitize_lm_state+from dspy.primitives.code_interpreter import CodeInterpreter, _validate_interpreter_factory+from dspy.primitives.module import Module+from dspy.primitives.python_interpreter import PythonInterpreter+from dspy.signatures.signature import Signature, ensure_signature+from dspy.utils.annotation import experimental+++@experimental(version="3.3.0")+class Flex(Module, Parameter):+ """A module whose implementation is optimizable code, not just a prompt.++ Construct it like any module (``dspy.Flex(MySignature)``). It starts as a baseline that delegates+ to a single ``dspy.Predict`` over the signature — or ``dspy.RLM`` when ``tools`` are given, so the+ baseline can call them. ``dspy.GEPA`` recognizes ``Flex`` instances by type and rewrites their+ source — a single ``dspy.Module`` subclass, exposed as ``module_src`` — into decomposed predictors+ plus plain Python instead of only tuning instructions.++ The optimizer-authored code runs inside an interpreter. ``Flex`` never+ runs it in the host Python process. ``interpreter_factory`` defaults to ``dspy.PythonInterpreter``+ (Deno/Pyodide) and must be a zero-argument callable returning a new ``CodeInterpreter``.+ Flex may validate or lower source and install its guest shim before execution; a custom+ interpreter therefore defines the Python and standard-library subset available to that source.+ The optimizer-authored glue runs isolated; only provided-tool calls, predictor construction,+ and predictor calls bridge back to the host, which makes the real LM calls.++ Args:+ signature: A ``dspy.Signature`` class or string declaring inputs/outputs.+ tools: ``dspy.Tool`` instances or named callables.+ interpreter_factory: Zero-argument callable returning a fresh ``CodeInterpreter`` for each+ sandbox session. Defaults to ``dspy.PythonInterpreter`` (sandbox, requires Deno).+ max_predictor_calls: Maximum number of predictor calls the optimizer-authored code can+ make per ``forward`. ``None`` removes the limit.+ """++ def __init__(+ self,+ signature: Any,+ *,+ tools: list[Any] | None = None,+ interpreter_factory: Callable[[], CodeInterpreter] = PythonInterpreter,+ max_predictor_calls: int | None = 100,+ ):+ super().__init__()++ self._signature_cls = ensure_signature(signature)+ self._name = getattr(self._signature_cls, "__name__", None) or "Flex"+ self._flex_ctx = FlexContext(signature_cls=self._signature_cls, tools=list(tools or []))++ self._module_src: str | None = None+ self.lm = None++ _validate_interpreter_factory(interpreter_factory)+ self._interpreter_factory = interpreter_factory+ self._max_predictor_calls = max_predictor_calls++ self._rebuild_bridge()+ self._bind_code(self._baseline_src())++ def _rebuild_bridge(self) -> None:+ self._bridge: BridgeRuntime = BridgeRuntime(self, self._interpreter_factory, self._max_predictor_calls)++ def __getstate__(self) -> dict[str, Any]:+ state = super().__getstate__()+ state.pop("_bridge", None)+ return state++ def __setstate__(self, state: dict[str, Any]) -> None:+ super().__setstate__(state)+ self._rebuild_bridge()+ if self._module_src is not None:+ self._bridge.bind(self._module_src)++ @property+ def signature(self) -> type[Signature]:+ return self._signature_cls++ @property+ def module_src(self) -> str | None:+ return self._module_src++ def dump_state(self, json_mode: bool = True) -> dict[str, Any]:+ return {+ "module_src": self._module_src,+ "lm": self.lm.dump_state() if self.lm else None,+ }++ def load_state(self, state: dict[str, Any], *, allow_unsafe_lm_state: bool = False) -> None:+ module_src = state.get("module_src")+ if module_src:+ self._bind_code(module_src)+ lm_state = state.get("lm")+ if lm_state:+ sanitized = _sanitize_lm_state(lm_state, allow_unsafe_lm_state)+ self.lm = (+ BaseLM.load_state(sanitized, allow_custom_lm_class=allow_unsafe_lm_state) if sanitized else None+ )+ else:+ self.lm = None++ def reset(self) -> None:+ """Clear the LM; ``module_src`` is kept, and predictors are rebuilt from it each forward."""+ self.lm = None++ def named_predictors(self):+ """A Flex's update unit is only its ``module_src``."""+ return []++ def set_lm(self, lm) -> None:+ self.lm = lm++ def get_lm(self):+ return self.lm++ def __repr__(self):+ return f"{self.__class__.__name__}({self._name})"++ def _baseline_src(self) -> str:+ """Starting source: one predictor over the whole signature, wrapped in a ``dspy.Module``.++ Uses ``dspy.RLM`` when tools are provided; otherwise a single ``dspy.Predict``.+ """+ cls: type[Signature] = self._signature_cls+ sig_str = self._flex_ctx.render_signature_string()+ returns = ", ".join(f"{name}=result.{name}" for name in cls.output_fields)+ instructions = (getattr(cls, "instructions", "") or "").strip()+ sig_arg = f"dspy.Signature({sig_str!r}, {instructions!r})" if instructions else repr(sig_str)+ tool_names = list(self._flex_ctx.context_names())+ if tool_names:+ attr, ctor = "rlm", f"dspy.RLM({sig_arg}, tools=[{', '.join(tool_names)}])"+ else:+ attr, ctor = "predict", f"dspy.Predict({sig_arg})"+ return (+ f"class {self._class_name()}(dspy.Module):\n"+ " def __init__(self):\n"+ " super().__init__()\n"+ f" self.{attr} = {ctor}\n"+ "\n"+ " def forward(self, **inputs):\n"+ f" result = self.{attr}(**inputs)\n"+ f" return dspy.Prediction({returns})"+ )++ def _class_name(self) -> str:+ base = self._name if (self._name and self._name.isidentifier()) else "Flex"+ return f"{base}Module"++ def _bind_code(self, module_src: str) -> None:+ self._bridge.bind(module_src)+ self._module_src = module_src++ def forward(self, *args: Any, **kwargs: Any) -> Any:+ """Run the bound ``forward`` inside the interpreter."""+ if args:+ raise TypeError("dspy.Flex accepts keyword inputs only")+ if self.lm is not None:+ with settings.context(lm=self.lm):+ return self._bridge.forward(kwargs)+ return self._bridge.forward(kwargs)diff --git a/dspy/predict/flex/primitives_doc.py b/dspy/predict/flex/primitives_doc.pynew file mode 100644index 000000000..7d9dde778--- /dev/null+++ b/dspy/predict/flex/primitives_doc.py@@ -0,0 +1,245 @@+PRIMITIVES_CATALOG: str = """\+You are authoring ONE Python source string: a single `dspy.Module` subclass that+implements the user's signature. It has exactly two methods:++1) `def __init__(self):` — calls `super().__init__()` and assigns each predictor it+ needs to an attribute, e.g. `self.classify = dspy.Predict("text -> label")`.+ Each predictor is a constructed `dspy.Predict` / `dspy.ChainOfThought` /+ `dspy.ReAct` / `dspy.RLM` over a sub-signature string. Define a predictor ONLY+ for a step that genuinely needs a language model. If the task needs no LM at+ all, define no predictors (an `__init__` with just `super().__init__()`).++2) `def forward(self, **inputs):` — returns `dspy.Prediction(<output_fields...>)`+ matching the declared signature. The body may call the predictors+ (`self.<name>(...)`) AND/OR run arbitrary deterministic Python — parsing,+ arithmetic, regex, string and data-structure manipulation, control flow, and+ small helper functions defined *inside* `forward`. A module may be fully+ deterministic (no predictors), fully LM-driven, or a mix. Prefer plain Python+ for anything that doesn't require an LM's judgment.++The whole class is one optimizable unit — `__init__` and `forward` are rewritten+together, so a predictor renamed in one place must be updated in the other.++Skeleton (the shape of every module you write):++```+class SolveModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.extract = dspy.Predict("invoice: str -> qty: int, unit_price_cents: int")++ def forward(self, **inputs):+ e = self.extract(invoice=inputs["invoice"])+ return dspy.Prediction(total_cents=int(e.qty) * int(e.unit_price_cents))+```++`dspy` is already in scope — do NOT add `import` statements at module scope (import+stdlib modules like `re` *inside* `forward` if you need them). Python and standard-library+availability is defined by this Flex's configured interpreter; candidates are evaluated in that+interpreter and should not assume host-Python access.++Available DSPy primitives:++- `dspy.Predict("a, b -> c, d")`, one LM call. `result = self.p(a=x, b=y)` returns a+ `dspy.Prediction` object; the declared outputs are attributes on it:+ `result.c`, `result.d`. Signature strings use snake_case names.+ TYPE the fields whenever you know the type — `"text: str -> count: int, tags: list[str]"` —+ a typed output is parsed and coerced to that type, while an untyped one comes back as a bare+ string you must cast yourself.+ Give any predictor task-specific INSTRUCTIONS by constructing it over a+ `dspy.Signature("a, b -> c, d", "natural-language instructions ...")` instead of a bare+ string.+ You may optimize these string instructions. See "Writing and refining instructions"+ below.++- `dspy.ChainOfThought("a -> b")`, like Predict but adds an implicit `reasoning`+ output field; useful when the answer benefits from explicit step-by-step thought.++- `dspy.ReAct(signature_str, tools=[tool1, tool2])`, multi-step reasoning + tool+ use. `tools` is a list of plain Python callables or `dspy.Tool` instances.++- `dspy.RLM("context, query -> output")`, Recursive Language Model: the LLM writes+ Python code in a sandboxed REPL to programmatically explore very large or+ structured inputs (long documents, big JSON blobs, tables) and build up an answer+ iteratively. Use when one of the input fields is too large or too structured for a+ single direct prompt. The defaults are sensible; only pass a kwarg to override one.+ Optional kwargs: `max_iters` (REPL steps, default 20), `max_llm_calls` (sub-LLM+ calls, default 50), `sub_lm` (cheaper LM for sub-queries), `tools` (extra callables+ for the REPL).++- Tools for `dspy.ReAct` / `dspy.RLM` come from two places:+ (1) any tools listed in the available context, in scope by name — ONLY these provided tools may+ be handed to a sub-predictor via `dspy.ReAct(..., tools=[...])` / `dspy.RLM(..., tools=[...])`;+ and (2) helpers you AUTHOR yourself — when a sub-step needs a capability the provided tools don't+ cover (deterministic lookups, parsing/formatting, calculators, validators, retrieval helpers,+ etc.), define a plain function (with a docstring and type hints) nested inside `forward` and CALL+ IT DIRECTLY. This code runs in a sandbox, so an authored helper cannot be handed to a bridged+ sub-predictor — keep authored helpers to direct calls in `forward`.++- `dspy.Tool(func)`, wraps a callable as a tool (for `ReAct`/`RLM`); usually you can pass the+ bare function and it is wrapped for you.++- `dspy.Prediction(field_a=value_a, field_b=value_b)`, construct the final return+ value. The keyword names MUST match the parent signature's output_fields+ exactly, and each `value_*` must be a plain value of the declared output+ type — never a `dspy.Prediction` you got back from a predictor call.++The list above is the WHOLE of `dspy` available here. These do NOT exist in this+module, and referencing one makes the code fail to bind: adapters+(`dspy.ChatAdapter`, `dspy.JSONAdapter`), `dspy.settings`, `dspy.context`,+`dspy.configure`, `dspy.LM`, `dspy.Example`, `dspy.Evaluate`, the optimizers,+retrievers, `dspy.Flex` itself, and class-based signatures with+`dspy.InputField` / `dspy.OutputField` (use the string form,+`dspy.Signature("a -> b", "instructions")`). The host owns prompting, parsing,+and the LM; this source only declares predictors and the Python wiring them+together. The code also runs sandboxed, with no host filesystem or network, unless otherwise+provided via a host tool.++Rules you MUST follow:++- Define every predictor in `__init__` as `self.<name> = dspy.Predict(...)` and call+ it in `forward` as `self.<name>(...)`.+- `forward` runs on a fresh instance every call: do not rely on `self` attributes+ carrying state across calls (caches, counters). Cross-call state, if truly needed,+ belongs in a provided host tool.++- Give predictors INSTRUCTIONS, don't just rely on field names. When the feedback shows+ the model needs guidance, construct the predictor over+ `dspy.Signature("inputs -> outputs", "instructions")` and refine those instructions from the+ failing examples (see "Writing and refining instructions"). These instructions are optimized+ ONLY through this source, so improving them is as important as changing the code.++- Do NOT access dunder attributes (anything matching `__*__`). Use the public API.++- Every `while` loop must be bounded, either with an explicit `break`, or rewrite+ to `for _ in range(N):`. Pick a small `N` (e.g. 5).++- Always return `dspy.Prediction(...)` from `forward`. Never raise from `forward`+ unless the input itself is malformed.++- Use predictors only where an LM is genuinely needed (extraction, classification,+ generation, open-ended reasoning); do everything else — math, parsing, formatting,+ validation, control flow — in plain Python. When no step needs an LM, write a+ pure-Python `forward` and define no predictors in `__init__`.++- Define any helper functions NESTED inside `forward` (not at module scope), and+ add no module-scope code other than the single class definition, so the persisted+ module file and the live module execute identically.++- Keep the design composable: prefer two short predictors with clear signatures+ over one all-knowing prompt.++- Do not hardcode example outputs or magic constants from the task description.++- Pick the right primitive for each step. Rough guide:+ * `dspy.Predict` for direct, single-call transformations.+ * `dspy.ChainOfThought` when the answer benefits from explicit step-by-step+ reasoning.+ * `dspy.ReAct` when external tools must be called during reasoning.+ * `dspy.RLM` when an input field is a large blob (long document, big JSON,+ multi-table data) that doesn't fit naturally in a single prompt — RLM+ lets the LLM explore it iteratively via a sandboxed REPL.++Writing and refining instructions:++The predictors you define are prompts, and a predictor's natural-language instructions — the+second argument to `dspy.Signature("inputs -> outputs", "instructions")` — are a PRIMARY thing+you optimize here, exactly like the code structure. Because these predictors live inside this+dspy.Flex module, this source is the ONLY place their instructions get optimized, so rewriting+them well matters as much as picking the right primitive. Don't leave a predictor on a bare+signature string once the feedback shows the model needs guidance.++When you revise, mine the failing examples and feedback the way a dedicated instruction+optimizer would:++- Read every failing input, output, and its feedback, and diagnose WHY each one failed.+- Categorize what the feedback is telling you:+ * Error patterns — recurring mistakes to explicitly prevent.+ * Success patterns — what worked and should be preserved/reinforced.+ * Domain-knowledge gaps — task-specific facts, definitions, or conventions the model+ lacked; state them IN the instruction, since the model won't otherwise have them.+ * Task-specific guidance — required output format, edge cases, and constraints.+- Fold those findings into each affected predictor's instructions: a clear task definition,+ the domain knowledge it needs, explicit rules that prevent the observed errors, the exact+ output format required, and precise, actionable language. Keep instructions specific to the+ task but NEVER paste in example answers or magic constants from the data.++Rule of thumb: fix a predictor's INSTRUCTIONS when a failure is about WHAT the model should do+or know; change the CODE structure (which predictors, how they're wired, what runs in plain+Python) when the failure is about HOW the steps fit together. The best revision often does both.++Common patterns (study these before writing `forward`):++1) Unwrap every predictor output. A predictor call returns a `dspy.Prediction`+ whose declared output fields are attributes on it. Always read those+ attributes before composing further calls or the final return.++ ```+ class TotalModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.parse = dspy.Predict("invoice -> qty: int, unit_price: float")+ self.shipping = dspy.Predict("invoice -> shipping_dollars: float")++ # RIGHT — pull fields off each predictor result, then build the final Prediction+ def forward(self, **inputs):+ parsed = self.parse(invoice=inputs["invoice"])+ ship = self.shipping(invoice=inputs["invoice"])+ cents = round((parsed.qty * parsed.unit_price + ship.shipping_dollars) * 100)+ return dspy.Prediction(total_cents=int(cents))++ # WRONG — passing the whole inner Prediction as the field value, producing+ # Prediction(total_cents=Prediction(total_cents=...)) at runtime:+ # return dspy.Prediction(total_cents=self.parse(invoice=inputs["invoice"]))+ ```++2) Coerce to the declared output type. If the parent signature declares+ `total_cents: int`, cast in Python (`int(...)`, `float(...)`, `str(...)`,+ `bool(...)`, list comprehensions for `list[T]`) before returning. LM+ string outputs do NOT auto-cast to the declared type.++3) Compose predictor outputs into the next call by passing the unwrapped+ attribute, never the whole Prediction. `self.next(x=prev.x)` — not+ `self.next(x=prev)`.++4) Do arithmetic, sorting, filtering, and aggregation in plain Python in+ `forward()`. Use the LM to extract / classify / generate; let Python do+ the math. This is more reliable and easier to debug than asking the LM+ to do both at once.++5) A fully-deterministic task needs no LM: define no predictors and implement+ `forward` in plain Python, with any helpers nested inside it.++ ```+ class SlugifyModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ def slugify(text):+ cleaned = "".join(c.lower() if c.isalnum() else "-" for c in text)+ return "-".join(part for part in cleaned.split("-") if part)++ return dspy.Prediction(slug=slugify(inputs["title"]))+ ```++6) Put task guidance in a predictor's INSTRUCTIONS, not only in the field names, and refine+ them from feedback. Give the signature a second argument spelling out the rules the failures+ revealed (definitions, output format, error-prevention), without pasting in example answers.++ ```+ class ClassifyModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.classify = dspy.Predict(dspy.Signature(+ "ticket: str -> category: str",+ "Classify the support ticket into exactly one of: billing, bug, feature, other. "+ "Choose 'bug' only for a defect in existing behavior; a request for new behavior "+ "is 'feature'. Reply with the lowercase category word and nothing else."+ ))++ def forward(self, **inputs):+ out = self.classify(ticket=inputs["ticket"])+ return dspy.Prediction(category=out.category.strip().lower())+ ```+"""diff --git a/dspy/predict/refine.py b/dspy/predict/refine.pyindex bc4aca45e..da49c7ff6 100644--- a/dspy/predict/refine.py+++ b/dspy/predict/refine.py@@ -120,7 +120,7 @@ class Refine(Module):class WrapperAdapter(adapter.__class__):def __call__(self, lm, lm_kwargs, signature, demos, inputs):- inputs["hint_"] = advice.get(signature2name[signature], "N/A") # noqa: B023+ inputs["hint_"] = advice.get(signature2name.get(signature), "N/A") # noqa: B023signature = signature.append("hint_", InputField(desc="A hint to the module from an earlier run"))diff --git a/dspy/primitives/base_module.py b/dspy/primitives/base_module.pyindex ab2214a69..2d2dc57b5 100644--- a/dspy/primitives/base_module.py+++ b/dspy/primitives/base_module.py@@ -157,11 +157,11 @@ class BaseModule:return {name: param.dump_state(json_mode=json_mode) for name, param in self.named_parameters()}def load_state(self, state, *, allow_unsafe_lm_state=False):- from dspy.predict.predict import Predict+ from dspy import Moduledef _apply(module):for name, param in module.named_parameters():- if isinstance(param, Predict):+ if isinstance(param, Module):param.load_state(state[name], allow_unsafe_lm_state=allow_unsafe_lm_state)else:param.load_state(state[name])diff --git a/dspy/primitives/code_interpreter.py b/dspy/primitives/code_interpreter.pyindex ce43810ae..6e44a19eb 100644--- a/dspy/primitives/code_interpreter.py+++ b/dspy/primitives/code_interpreter.py@@ -74,11 +74,13 @@ class CodeInterpreter(Protocol):"""@property- def tools(self) -> dict[str, Callable[..., str]]:+ def tools(self) -> dict[str, Callable[..., Any]]:"""Tools available for interpreter code to call.Tools are host-side functions that can be invoked from within the- interpreter. Each tool accepts keyword arguments and returns a string.+ interpreter. Each tool accepts keyword arguments. Return values must+ satisfy the boundary supported by the interpreter; Flex tools must+ return JSON-compatible values.Implementations should accept tools via constructor and expose themthrough this property.diff --git a/dspy/primitives/module.py b/dspy/primitives/module.pyindex c6f6f70b2..10f092393 100644--- a/dspy/primitives/module.py+++ b/dspy/primitives/module.py@@ -179,8 +179,8 @@ class Module(BaseModule, metaclass=ProgramMeta):def set_lm(self, lm):"""Set the language model for all predictors in this module.- This method recursively sets the language model for all Predict- instances contained within this module.+ This method recursively sets the language model on every leaf module+ in this module.Args:lm: The language model instance to use for all predictors.@@ -191,14 +191,15 @@ class Module(BaseModule, metaclass=ProgramMeta):>>> program = dspy.Predict("question -> answer")>>> program.set_lm(lm)"""- for _, param in self.named_predictors():- param.lm = lm+ for _, param in self.named_parameters():+ if isinstance(param, Module):+ param.lm = lmdef get_lm(self):"""Get the language model used by this module's predictors.- Returns the language model if all predictors use the same LM.- Raises an error if multiple different LMs are in use.+ Returns the language model if every module leaf uses the same LM. Raises an error if multiple+ different LMs are in use.Returns:The language model instance used by this module's predictors.@@ -207,7 +208,7 @@ class Module(BaseModule, metaclass=ProgramMeta):ValueError: If multiple different language models are beingused by the predictors in this module."""- all_used_lms = [param.lm for _, param in self.named_predictors()]+ all_used_lms = [param.lm for _, param in self.named_parameters() if isinstance(param, Module)]if len(set(all_used_lms)) == 1:return all_used_lms[0]diff --git a/dspy/primitives/python_interpreter.py b/dspy/primitives/python_interpreter.pyindex 66061b634..b7c90e510 100644--- a/dspy/primitives/python_interpreter.py+++ b/dspy/primitives/python_interpreter.py@@ -21,7 +21,7 @@ from os import PathLikefrom typing import Any, Callable, NoReturnfrom pydantic import BaseModel-from pydantic_core import PydanticSerializationError+from pydantic_core import PydanticSerializationError, to_jsonable_pythonfrom dspy.primitives.code_interpreter import SIMPLE_TYPES, CodeExecutionError, CodeInterpreterError, FinalOutput@@ -112,8 +112,11 @@ def _make_jsonable(value: Any) -> Any:Handles Pydantic BaseModel, dataclasses, and namedtuples so that``json.dumps()`` can serialize tool results without falling back to- ``str()``.+ ``str()``. Anything else JSON-mode serializable (datetimes, enums, UUIDs,+ sets, ...) is converted the way it would serialize into a JSON body."""+ if value is None or isinstance(value, (str, int, float, bool)):+ return valueif isinstance(value, BaseModel):return _make_jsonable(_dump_pydantic(value))if dataclasses.is_dataclass(value) and not isinstance(value, type):@@ -129,7 +132,10 @@ def _make_jsonable(value: Any) -> Any:return [_make_jsonable(v) for v in value]if isinstance(value, tuple):return tuple(_make_jsonable(v) for v in value)- return value+ try:+ return to_jsonable_python(value)+ except PydanticSerializationError:+ return valueclass PythonInterpreter:@@ -528,7 +534,11 @@ class PythonInterpreter:except TypeError:return [self._to_json_compatible(v) for v in value]else:- raise CodeInterpreterError(f"Unsupported value type: {type(value).__name__}")+ try:+ coerced = to_jsonable_python(value)+ except PydanticSerializationError:+ raise CodeInterpreterError(f"Unsupported value type: {type(value).__name__}") from None+ return self._to_json_compatible(coerced)def _inject_variables(self, code: str, variables: dict[str, Any]) -> str:"""Insert Python assignments for each variable at the top of the code."""@@ -595,7 +605,11 @@ class PythonInterpreter:items = ", ".join(self._serialize_value(item) for item in sorted_items)return f"[{items}]"else:- raise CodeInterpreterError(f"Unsupported value type: {type(value).__name__}")+ try:+ coerced = to_jsonable_python(value)+ except PydanticSerializationError:+ raise CodeInterpreterError(f"Unsupported value type: {type(value).__name__}") from None+ return self._serialize_value(coerced)def _inject_large_var(self, name: str, value: str) -> None:"""Inject a large variable via the virtual filesystem."""diff --git a/dspy/signatures/signature.py b/dspy/signatures/signature.pyindex 90f552de4..31a876c59 100644--- a/dspy/signatures/signature.py+++ b/dspy/signatures/signature.py@@ -630,7 +630,7 @@ def make_signature(# Default prompt when no instructions are providedif instructions is None:- sig = Signature(signature, "") # Simple way to parse input/output fields+ sig = Signature(signature, "", custom_types=custom_types) # Simple way to parse input/output fieldsinstructions = _default_instructions(sig)return create_model(diff --git a/dspy/teleprompt/bootstrap_trace.py b/dspy/teleprompt/bootstrap_trace.pyindex 2f0cc60cd..84837f49b 100644--- a/dspy/teleprompt/bootstrap_trace.py+++ b/dspy/teleprompt/bootstrap_trace.py@@ -8,7 +8,7 @@ from dspy.evaluate.evaluate import Evaluatefrom dspy.primitives.example import Examplefrom dspy.primitives.module import Modulefrom dspy.primitives.prediction import Prediction-from dspy.utils.exceptions import AdapterParseError+from dspy.utils.exceptions import AdapterParseError, LMErrorlogger = logging.getLogger(__name__)@@ -38,6 +38,7 @@ def bootstrap_trace_data(format_failure_score: float = -1,log_format_failures: bool = False,callback_metadata: dict[str, Any] | None = None,+ capture_crashes: bool = False,) -> list[TraceData]:# Return a list of dicts with the following keys: example_ind, example, prediction, trace, and score# (if metric != None)@@ -107,6 +108,14 @@ def bootstrap_trace_data()return failed_pred, trace+ except LMError:+ raise+ except Exception as e:+ if not capture_crashes:+ raise+ logger.warning("Program raised on an example; capturing it as a failed prediction: %s: %s",+ type(e).__name__, e)+ return FailedPrediction(completion_text=f"{type(e).__name__}: {e}"), dspy.settings.trace.copy()program.forward = MethodType(patched_forward, program)diff --git a/dspy/teleprompt/gepa/gepa.py b/dspy/teleprompt/gepa/gepa.pyindex 0f2eaf9d5..f774a5f5a 100644--- a/dspy/teleprompt/gepa/gepa.py+++ b/dspy/teleprompt/gepa/gepa.py@@ -33,6 +33,7 @@ class GEPAFeedbackMetric(Protocol):trace: Optional["DSPyTrace"],pred_name: str | None,pred_trace: Optional["DSPyTrace"],+ program_trace: Optional["DSPyTrace"] = None,) -> Union[float, "ScoreWithFeedback"]:"""This function is called with the following arguments:@@ -42,6 +43,10 @@ class GEPAFeedbackMetric(Protocol):- pred_name: Optional. The name of the target predictor currently being optimized by GEPA, for whichthe feedback is being requested.- pred_trace: Optional. The trace of the target predictor's execution GEPA is seeking feedback for.+ - program_trace: Optional. The full execution trace of the program, supplied at scoring time when a+ `dspy.Flex` submodule is being optimized. Declare this parameter to score against how an answer was+ produced (e.g. `len(program_trace)` as an LM-call count), rather than only whether it was correct.+ Unlike `trace`, it is populated during candidate *scoring*.Note the `pred_name` and `pred_trace` arguments. During optimization, GEPA will call the metric to obtainfeedback for individual predictors being optimized. GEPA provides the name of the predictor in `pred_name`@@ -112,11 +117,19 @@ class DspyGEPAResult:for val_id in self.per_val_instance_best_candidates}+ @staticmethod+ def _candidate_components(cand: Module) -> dict[str, str]:+ """The candidate's optimized components. It can be either instruction text per predictor, or+ the full `module_src` of each `dspy.Flex` submodule under its parameter path."""+ from dspy.teleprompt.gepa.gepa_flex_utils import enumerate_flex_submodules++ components = {name: pred.signature.instructions for name, pred in cand.named_predictors()}+ for path, flex in enumerate_flex_submodules(cand).items():+ components[path] = flex.module_src+ return components+def to_dict(self) -> dict[str, Any]:- cands = [- {name: pred.signature.instructions for name, pred in cand.named_predictors()}- for cand in self.candidates- ]+ cands = [self._candidate_components(cand) for cand in self.candidates]return dict(candidates=cands,@@ -174,6 +187,7 @@ class GEPA(Teleprompter):trace: Optional[DSPyTrace] = None,pred_name: Optional[str] = None,pred_trace: Optional[DSPyTrace] = None,+ program_trace: Optional[DSPyTrace] = None,) -> float | ScoreWithFeedback:\"""This function is called with the following arguments:@@ -183,6 +197,9 @@ class GEPA(Teleprompter):- pred_name: Optional. The name of the target predictor currently being optimized by GEPA, for whichthe feedback is being requested.- pred_trace: Optional. The trace of the target predictor's execution GEPA is seeking feedback for.+ - program_trace: Optional. The program's execution trace, supplied at scoring time when a `dspy.Flex`+ submodule is being optimized. Declare it to score against how the answer was produced (e.g. an+ LM-call penalty). Defaults to None.Note the `pred_name` and `pred_trace` arguments. During optimization, GEPA will call the metric to obtainfeedback for individual predictors being optimized. GEPA provides the name of the predictor in `pred_name`@@ -492,14 +509,20 @@ class GEPA(Teleprompter):"""from gepa import GEPAResult, optimize+ from dspy.teleprompt.gepa.gepa_flex_utils import enumerate_flex_submodulesfrom dspy.teleprompt.gepa.gepa_utils import DspyAdapter, LoggerAdapterassert trainset is not None and len(trainset) > 0, "Trainset must be provided and non-empty"assert teacher is None, "Teacher is not supported in DspyGEPA yet."+ # dspy.Flex submodules get their code optimized, not just their instructions.+ flex_submodules = enumerate_flex_submodules(student)+ instruction_predictors = list(student.named_predictors())++ num_components = len(instruction_predictors) + len(flex_submodules)if self.auto is not None:self.max_metric_calls = self.auto_budget(- num_preds=len(student.predictors()),+ num_preds=max(num_components, 1),num_candidates=AUTO_RUN_SETTINGS[self.auto]["n"],valset_size=len(valset) if valset is not None else len(trainset),)@@ -554,7 +577,7 @@ class GEPA(Teleprompter):return feedback_fn- feedback_map = {k: feedback_fn_creator(k, v) for k, v in student.named_predictors()}+ feedback_map = {k: feedback_fn_creator(k, v) for k, v in instruction_predictors}# Build the DSPy adapter that encapsulates evaluation, trace capture, feedback extraction, and instruction proposaladapter = DspyAdapter(@@ -571,8 +594,11 @@ class GEPA(Teleprompter):reflection_minibatch_size=self.reflection_minibatch_size,)- # Build the seed candidate: map each predictor name to its current instruction- seed_candidate = {name: pred.signature.instructions for name, pred in student.named_predictors()}+ # Seed candidate: instruction text per non-flex predictor, plus the current module_src of+ # each dspy.Flex submodule as its code component.+ seed_candidate = {name: pred.signature.instructions for name, pred in instruction_predictors}+ for path, flex in flex_submodules.items():+ seed_candidate[path] = flex.module_srcgepa_result: GEPAResult = optimize(seed_candidate=seed_candidate,diff --git a/dspy/teleprompt/gepa/gepa_flex_utils.py b/dspy/teleprompt/gepa/gepa_flex_utils.pynew file mode 100644index 000000000..fe55e6db1--- /dev/null+++ b/dspy/teleprompt/gepa/gepa_flex_utils.py@@ -0,0 +1,255 @@+from __future__ import annotations++import inspect+import logging+from typing import Any++from gepa import EvaluationBatch++import dspy+from dspy.predict.flex import Flex+from dspy.predict.flex.ctx import _strip_code_fences+from dspy.predict.flex.primitives_doc import PRIMITIVES_CATALOG+from dspy.primitives import Prediction+from dspy.teleprompt import bootstrap_trace as bootstrap_trace_module+from dspy.teleprompt.bootstrap_trace import FailedPrediction+from dspy.utils.exceptions import LMError++logger = logging.getLogger(__name__)++def enumerate_flex_submodules(root) -> dict[str, Any]:+ """Map parameter path -> module for every code-optimizable (``dspy.Flex``) submodule."""+ return {name: sub for name, sub in root.named_parameters() if isinstance(sub, Flex)}+++def flex_task_context(student_module) -> tuple[dict[str, str], dict[str, str]]:+ """Per flex submodule path, the ``(task_description, available_context)`` shown to the code+ proposer, keyed by the submodule's parameter path (its candidate key).+ """+ task_descriptions: dict[str, str] = {}+ context_blurbs: dict[str, str] = {}+ for path, sub in enumerate_flex_submodules(student_module).items():+ ctx = getattr(sub, "_flex_ctx", None)+ if ctx is not None and hasattr(ctx, "render_signature_spec"):+ task_descriptions[path] = ctx.render_signature_spec()+ context_blurbs[path] = ctx.render_context_blurb(sandboxed=True)+ return task_descriptions, context_blurbs+++def _render_prediction(prediction: Any) -> Any:+ if isinstance(prediction, FailedPrediction):+ return f"FailedPrediction: {prediction.completion_text}"+ if isinstance(prediction, Prediction):+ try:+ return {k: str(v) for k, v in prediction.items()}+ except Exception:+ pass+ return str(prediction)+++def _format_failures(records: list[dict[str, Any]]) -> str:+ if not records:+ return "(no failing examples available)"+ chunks: list[str] = []+ for i, rec in enumerate(records):+ chunks.append(+ f"=== Example {i} ===\n"+ f"Inputs:\n{rec.get('Inputs')!r}\n"+ f"Generated Outputs:\n{rec.get('Generated Outputs')!r}\n"+ f"Feedback:\n{rec.get('Feedback')}"+ )+ return "\n\n".join(chunks)+++class CodeProposalSignature(dspy.Signature):+ """Revise the full source code of a dspy.Flex submodule.++ You receive the submodule's task description (its Signature), the available context (any tools+ and style notes), the catalog of allowed primitives, the module's current source, and a batch+ of failing examples with feedback. Produce a revised source that fixes the observed failures+ and follows the catalog.++ The source is ONE ``dspy.Module`` subclass with two coupled methods, and you MUST output the+ entire, internally-consistent class:+ 1. ``def __init__(self):`` calling ``super().__init__()`` and assigning the predictors it+ needs. Pick the simplest primitive that fits each step: ``dspy.Predict("...")`` for a+ direct call (the common default), ``dspy.ChainOfThought("...")`` when explicit reasoning+ helps, and ``dspy.RLM`` / ``dspy.ReAct`` when the step must call tools or explore a+ large/structured input. Assign no predictors at all if the task needs no LM.+ 2. ``def forward(self, **inputs):`` that calls those predictors as ``self.<name>`` and+ returns ``dspy.Prediction(<output fields>=...)``.+ Because ``forward`` calls predictors by name, never rename a predictor in one place without+ updating the other.++ Tools are OPTIONAL — use them only when a step genuinely needs one; many good modules are just+ a ``dspy.Predict`` or two plus plain Python. When you do use tools, they come from two places.+ (1) Any listed in ``available_context`` are in scope by name — wire the useful ones into+ ``dspy.RLM(..., tools=[...])`` / ``dspy.ReAct(..., tools=[...])`` or call them directly+ (reference them by the exact names; do not import or redefine them). If ``available_context``+ is '(no extra context)', no tools were provided — don't reference any.+ (2) AUTHOR your own: when a sub-step needs a capability the provided tools don't cover, define a+ documented helper nested inside ``forward`` and call it directly. Authored helpers live in this+ source, so they are optimized and persisted exactly like the rest of the code. They run in the+ sandbox and cannot be handed to a bridged sub-predictor, so only the provided tools may be wired+ into ``dspy.RLM``/``dspy.ReAct`` via ``tools=[...]``.++ Optimize the predictors' INSTRUCTIONS, not just the code structure. Each predictor's+ natural-language instructions live in this source — construct a predictor over+ ``dspy.Signature("inputs -> outputs", "instructions")`` and refine those instructions from the+ failing examples and feedback (add a clear task definition, domain knowledge the model lacked,+ the required output format, and rules that prevent the observed errors). These predictors are+ inside a dspy.Flex module, so this source is the ONLY place their prompts get optimized. See the primitives+ catalog's "Writing and refining instructions" section for how. Fix instructions when a failure is about+ WHAT the model should do or know; change the structure when it is about HOW steps are wired.+ """++ task_description: str = dspy.InputField(+ desc="The submodule's Signature: name, objective, input and output fields."+ )+ available_context: str = dspy.InputField(+ desc="Tools (in scope by name) and style notes available to the module. May be "+ "'(no extra context)'."+ )+ primitives_catalog: str = dspy.InputField(+ desc="Catalog of allowed primitives and conventions the revised code must follow."+ )+ current_source: str = dspy.InputField(+ desc="The module's current full source: one dspy.Module subclass (its __init__ and forward)."+ )+ failures: str = dspy.InputField(+ desc="A batch of failing examples and feedback. Diagnose them and revise the module to fix them."+ )+ revised_source: str = dspy.OutputField(+ desc="The full revised module source: one `dspy.Module` subclass with `__init__` (predictors, "+ 'including any refined `dspy.Signature(..., "instructions")`) and `forward`.'+ )+++def propose_code(+ code_keys: list[str],+ candidate: dict[str, str],+ reflective_dataset: dict[str, list[dict[str, Any]]],+ task_descriptions: dict[str, str],+ context_blurbs: dict[str, str],+ reflection_lm,+) -> dict[str, str]:+ """Propose a revised ``module_src`` for each code component; keep the original on failure."""+ results: dict[str, str] = {}+ proposer = dspy.Predict(CodeProposalSignature)+ with dspy.context(lm=reflection_lm):+ for ckey in code_keys:+ try:+ out = proposer(+ task_description=task_descriptions.get(ckey, ckey),+ available_context=context_blurbs.get(ckey, "(no extra context)"),+ primitives_catalog=PRIMITIVES_CATALOG,+ current_source=candidate[ckey],+ failures=_format_failures(reflective_dataset.get(ckey, [])),+ )+ results[ckey] = _strip_code_fences(out.revised_source)+ except LMError:+ raise+ except Exception as e:+ logger.warning("Code proposer failed on %s (%s); keeping original.", ckey, e)+ results[ckey] = candidate[ckey]+ return results+++def rebind_flex_code(program, candidate: dict[str, str]) -> None:+ """Rebind ``module_src`` on each ``dspy.Flex`` submodule whose path is in the candidate.++ A candidate that does not parse raises here (``DspyAdapter.evaluate`` catches that and scores+ the batch as a failure); one that parses but breaks at runtime fails per example during+ evaluation, since the code first runs at ``forward``.+ """+ for path, flex in enumerate_flex_submodules(program).items():+ if path in candidate:+ flex._bind_code(candidate[path])+++def code_reflective_records(eval_batch) -> list[dict[str, Any]]:+ """Module-level reflective records for a code component (whole-program I/O).++ Unlike instruction components (per-predictor traces), a flex submodule's code is reflected on+ using the whole program's inputs/outputs/feedback per example.+ """+ records: list[dict[str, Any]] = []+ for data in eval_batch.trajectories or []:+ example = data["example"]+ prediction = data["prediction"]+ raw_score = data.get("score")+ if hasattr(raw_score, "score"):+ feedback = raw_score.get("feedback") or ""+ score_val = raw_score["score"]+ else:+ feedback = ""+ score_val = raw_score+ records.append(+ {+ "Inputs": {k: str(v) for k, v in example.inputs().items()},+ "Generated Outputs": _render_prediction(prediction),+ "Feedback": feedback or f"This trajectory got a score of {score_val}.",+ }+ )+ return records+++def _metric_scoring_kwargs(metric_fn, program_trace) -> dict[str, Any]:+ """Keyword args for the optional GEPA-contract params ``metric_fn`` declares."""+ try:+ declared = inspect.signature(metric_fn).parameters.keys()+ except (TypeError, ValueError):+ return {}+ values = {"trace": None, "pred_name": None, "pred_trace": None, "program_trace": program_trace}+ return {name: value for name, value in values.items() if name in declared}+++def evaluate_with_trace(+ program,+ batch,+ *,+ metric_fn,+ num_threads,+ failure_score,+ callback_metadata,+ capture_traces,+) -> EvaluationBatch:+ """Trace-capturing evaluation used when a flex submodule is present.++ A metric that declares a ``program_trace`` parameter receives the execution trace at+ scoring time (e.g. to penalize LM calls and reward deterministic code); its ``trace``+ argument stays None, preserving the eval-mode semantics of non-Flex GEPA scoring.+ """+ trajs = bootstrap_trace_module.bootstrap_trace_data(+ program=program,+ dataset=batch,+ metric=None, # capture traces only; we score with the trace just below+ num_threads=num_threads,+ raise_on_error=False,+ capture_failed_parses=True,+ capture_crashes=True,+ failure_score=failure_score,+ format_failure_score=failure_score,+ callback_metadata=callback_metadata,+ )+ outputs: list[Any] = [None] * len(batch)+ scores: list[float] = [failure_score] * len(batch)+ for t in trajs:+ pred = t["prediction"]+ outputs[t["example_ind"]] = pred+ if isinstance(pred, FailedPrediction):+ result = failure_score+ else:+ try:+ result = metric_fn(t["example"], pred, **_metric_scoring_kwargs(metric_fn, t["trace"]))+ except Exception as e:+ logger.warning(+ "Metric raised while scoring an example; scoring it at failure_score: %s: %s",+ type(e).__name__,+ e,+ )+ result = failure_score+ t["score"] = result # make_reflective_dataset reads this for the (trace-aware) feedback+ score = result["score"] if hasattr(result, "score") else result+ scores[t["example_ind"]] = failure_score if score is None else score+ return EvaluationBatch(outputs=outputs, scores=scores, trajectories=trajs if capture_traces else None)diff --git a/dspy/teleprompt/gepa/gepa_utils.py b/dspy/teleprompt/gepa/gepa_utils.pyindex dae7157fe..e88f969ef 100644--- a/dspy/teleprompt/gepa/gepa_utils.py+++ b/dspy/teleprompt/gepa/gepa_utils.py@@ -12,7 +12,16 @@ from dspy.adapters.types import Historyfrom dspy.adapters.types.base_type import Typefrom dspy.evaluate import Evaluatefrom dspy.primitives import Example, Prediction+from dspy.primitives.code_interpreter import CodeInterpreterErrorfrom dspy.teleprompt.bootstrap_trace import FailedPrediction, TraceData+from dspy.teleprompt.gepa.gepa_flex_utils import (+ code_reflective_records,+ enumerate_flex_submodules,+ evaluate_with_trace,+ flex_task_context,+ propose_code,+ rebind_flex_code,+)logger = logging.getLogger(__name__)@@ -100,6 +109,12 @@ class DspyAdapter(GEPAAdapter[Example, TraceData, Prediction]):self.custom_instruction_proposer = custom_instruction_proposerself.warn_on_score_mismatch = warn_on_score_mismatchself.reflection_minibatch_size = reflection_minibatch_size+ self._warned_custom_proposer_skips_code = False++ # dspy.Flex code components are keyed by the submodule's parameter path.+ self._flex_paths = set(enumerate_flex_submodules(student_module))+ # Task description + available context shown to the dspy.Flex code proposer, per submodule.+ self._flex_task_descriptions, self._flex_context_blurbs = flex_task_context(student_module)def propose_new_texts(self,@@ -108,16 +123,40 @@ class DspyAdapter(GEPAAdapter[Example, TraceData, Prediction]):components_to_update: list[str],) -> dict[str, str]:reflection_lm = self.reflection_lm or dspy.settings.lm- # If custom proposer provided, override everything with custom proposer++ # dspy.Flex code components are rewritten by the code proposer.+ results: dict[str, str] = {}+ code_keys = [c for c in components_to_update if c in self._flex_paths]+ if code_keys:+ results.update(+ propose_code(+ code_keys, candidate, reflective_dataset,+ self._flex_task_descriptions, self._flex_context_blurbs, reflection_lm,+ )+ )+ components_to_update = [c for c in components_to_update if c not in self._flex_paths]+ if not components_to_update:+ return results++ # A custom proposer overrides the default *instruction* proposer only.if self.custom_instruction_proposer:+ if code_keys and not self._warned_custom_proposer_skips_code:+ logger.warning(+ "A custom instruction_proposer is set, but %d dspy.Flex code component(s) are "+ "being optimized. The custom proposer handles instruction components only; the "+ "built-in code proposer rewrites the flex source.",+ len(code_keys),+ )+ self._warned_custom_proposer_skips_code = Truewith dspy.context(lm=reflection_lm):- return self.custom_instruction_proposer(- candidate=candidate,- reflective_dataset=reflective_dataset,- components_to_update=components_to_update,+ results.update(+ self.custom_instruction_proposer(+ candidate=candidate,+ reflective_dataset=reflective_dataset,+ components_to_update=components_to_update,+ ))-- results: dict[str, str] = {}+ return resultswith dspy.context(lm=reflection_lm):for name in components_to_update:@@ -136,6 +175,14 @@ class DspyAdapter(GEPAAdapter[Example, TraceData, Prediction]):def build_program(self, candidate: dict[str, str]):new_prog = self.student.deepcopy()+ # Rebind code for any dspy.Flex submodules in the candidate (see gepa_flex_utils). A+ # candidate that doesn't parse raises here (evaluate() catches it and scores the batch as+ # a failure); runtime breakage surfaces per example when the code first runs at forward.+ rebind_flex_code(new_prog, candidate)++ # Apply instruction updates. Code components are keyed by a Flex's parameter path, which+ # never names a predictor, and predictors inside a Flex were excluded from the candidate,+ # so both are left untouched.for name, pred in new_prog.named_predictors():if name in candidate:pred.signature = pred.signature.with_instructions(candidate[name])@@ -143,13 +190,39 @@ class DspyAdapter(GEPAAdapter[Example, TraceData, Prediction]):return new_progdef evaluate(self, batch, candidate, capture_traces=False):- program = self.build_program(candidate)+ try:+ program = self.build_program(candidate)+ except (SyntaxError, CodeInterpreterError) as e:+ # A code candidate that fails to bind scores as a failure rather than+ # crashing the run. Only source-level failures are caught. Anything else+ # (e.g., an LM provider or rate-limit error) is not the candidate's fault and must+ # propagate.+ logger.warning("Candidate failed to build (%s); scoring the batch as failures.", e)+ return EvaluationBatch(+ outputs=[None] * len(batch),+ scores=[self.failure_score] * len(batch),+ trajectories=[] if capture_traces else None,+ )callback_metadata = ({"metric_key": "eval_full"}if self.reflection_minibatch_size is None or len(batch) > self.reflection_minibatch_sizeelse {"disable_logging": True})+ # When a dspy.Flex submodule is present, capture the execution trace at scoring time so a+ # metric that declares a `program_trace` parameter can score against it (e.g. an LM-call+ # penalty).+ if self._flex_paths:+ return evaluate_with_trace(+ program,+ batch,+ metric_fn=self.metric_fn,+ num_threads=self.num_threads,+ failure_score=self.failure_score,+ callback_metadata=callback_metadata,+ capture_traces=capture_traces,+ )+if capture_traces:# bootstrap_trace_data-like flow with trace capturefrom dspy.teleprompt import bootstrap_trace as bootstrap_trace_module@@ -203,6 +276,14 @@ class DspyAdapter(GEPAAdapter[Example, TraceData, Prediction]):ret_d: dict[str, list[ReflectiveExample]] = {}for pred_name in components_to_update:+ # Code components (dspy.Flex submodules) reflect on whole-program I/O, not per-predictor+ # traces.+ if pred_name in self._flex_paths:+ recs = code_reflective_records(eval_batch)+ if recs:+ ret_d[pred_name] = recs+ continue+# Find the predictor objectmodule = Nonefor name, m in program.named_predictors():diff --git a/dspy/teleprompt/simba_utils.py b/dspy/teleprompt/simba_utils.pyindex a47c00c82..ed811ea8a 100644--- a/dspy/teleprompt/simba_utils.py+++ b/dspy/teleprompt/simba_utils.py@@ -91,7 +91,9 @@ def append_a_demo(demo_input_field_maxlen):_inputs[k] = f"{str(v)[:demo_input_field_maxlen]}\n\t\t... <TRUNCATED FOR BREVITY>"demo = dspy.Example(augmented=True, **_inputs, **_outputs)- name = predictor2name[id(predictor)]+ name = predictor2name.get(id(predictor))+ if name is None:+ continuename2demo[name] = demo # keep the last demo for each predictorfor name, demo in name2demo.items():predictor = name2predictor[name]@@ -130,10 +132,12 @@ def append_a_rule(bucket, system, **kwargs):better_trajectory = [{"module_name": predictor2name[id(p)], "inputs": i, "outputs": dict(o)}for p, i, o in good["trace"]+ if id(p) in predictor2name]worse_trajectory = [{"module_name": predictor2name[id(p)], "inputs": i, "outputs": dict(o)}for p, i, o in bad["trace"]+ if id(p) in predictor2name]kwargs = {diff --git a/tests/flex/_offstack_types.py b/tests/flex/_offstack_types.pynew file mode 100644index 000000000..7631af300--- /dev/null+++ b/tests/flex/_offstack_types.py@@ -0,0 +1,26 @@+"""A signature with a custom output type, defined away from the test module that uses it.++dspy resolves a type named in a signature string by walking caller frames for the bare name. Test+modules normally define their types at module scope, which puts them on the stack and hides that+lookup's limits. Importing *this* module as a module object — `from tests.flex import+_offstack_types` — keeps `Contact` off every frame, which is the condition `bridge._resolve_signature`+exists for, and the condition any program written as `import myapp.models as models` is in.+"""++from __future__ import annotations++import pydantic++import dspy+++class Contact(pydantic.BaseModel):+ name: str+ age: int+++class ExtractContact(dspy.Signature):+ """Extract the contact described by the text."""++ text: str = dspy.InputField()+ contact: Contact = dspy.OutputField()diff --git a/tests/flex/test_flex_binding.py b/tests/flex/test_flex_binding.pynew file mode 100644index 000000000..e497b3f75--- /dev/null+++ b/tests/flex/test_flex_binding.py@@ -0,0 +1,323 @@+"""dspy.Flex binds a module's source (``module_src``) and makes it usable like any dspy.Module.++Binding a source — the deterministic baseline at construction, or a GEPA-optimized decomposition —+records it host-side (a syntax check, no interpreter); each forward runs it in a fresh sandbox,+constructing its predictors into that forward's own bridge invocation (nothing is attached to the+module, so parallel forwards can't overwrite each other's predictors). Everything+round-trips through Module.save / load so the generated code can be persisted and rerun. Tests+that run predictors need Deno; pure ``module_src`` checks use a MockInterpreter instead.+(The baseline's RLM-vs-Predict shape is covered in test_tools.py; GEPA's code rewriting in+test_flex_gepa.py.)+"""++from __future__ import annotations++import shutil+import textwrap++import pytest++import dspy+from dspy.predict.flex.flex import Flex+from dspy.utils.dummies import DummyLM+from tests.mock_interpreter import MockInterpreter++deno_required = pytest.mark.skipif(shutil.which("deno") is None, reason="Deno is not installed")++# A plain dspy.Predict module we bind and run with a DummyLM, standing in for a GEPA-optimized+# decomposition — end-to-end forward tests bind this instead of the construction baseline.+ECHO_MODULE = textwrap.dedent("""+ class EchoModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.echo = dspy.Predict("q -> a")++ def forward(self, q):+ out = self.echo(q=q)+ return dspy.Prediction(a=out.a)+""").strip()+++class Echo(dspy.Signature):+ """Echo the question as the answer."""++ q: str = dspy.InputField()+ a: str = dspy.OutputField()+++@deno_required+def test_forward_attaches_nothing_to_the_module() -> None:+ # A forward's predictors live in that forward's bridge invocation, never on the shared Flex:+ # parallel forwards (e.g. Evaluate(num_threads>1)) run on one module object, and attached+ # predictors would let one forward overwrite another's. The module's surface stays clean,+ # and named_predictors() reports no tunable units — the Flex's update unit is its module_src.+ dspy.configure(lm=DummyLM([{"a": "hi"}]))+ program = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ program(q="hello")+ assert not hasattr(program, "predict")+ assert "predict" not in [n for n, _ in program.named_parameters()]+ assert program.named_predictors() == []+++def test_construction_writes_nothing_to_disk(tmp_path, monkeypatch) -> None:+ # The implementation lives in memory; nothing is written until you call save(). Only module_src is+ # inspected here, so a MockInterpreter keeps it Deno-free.+ monkeypatch.chdir(tmp_path)+ program = Flex(Echo, interpreter_factory=lambda: MockInterpreter())+ assert program.module_src is not None+ assert "dspy.Predict(" in program.module_src+ # Flex itself persists nothing — no module file written until save() is called.+ # (dspy's global `.dspy_cache` may appear; that's unrelated to Flex persistence.)+ assert not list(tmp_path.glob("*.py"))+ assert not list(tmp_path.glob("*.json"))+++@deno_required+def test_save_load_roundtrips_generated_code(tmp_path) -> None:+ """Module.save/load carries the generated code (module_src), like instruction state."""+ path = tmp_path / "program.json"+ program = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(ECHO_MODULE) # stand in for a GEPA-optimized decomposition+ program.save(path)++ # A fresh Flex starts on the Predict baseline...+ reloaded = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ assert "self.echo" not in reloaded.module_src+ # ...and load() rebinds the saved code (no interpreter or LM needed: the code runs, and+ # constructs its predictors, at each forward).+ reloaded.load(path)+ assert "self.echo" in reloaded.module_src+ assert not hasattr(reloaded, "echo")+++@deno_required+def test_nested_flex_state_roundtrips_inside_a_program(tmp_path) -> None:+ """A Flex nested in a larger program persists like a top-level one.++ The parent's state must carry the sub-module's `module_src`, not just its predictors' state:+ those predictors are *created by* that source, so without it a reload has nothing to attach+ them to and the optimized code is lost.+ """+ path = tmp_path / "program.json"++ class Program(dspy.Module):+ def __init__(self) -> None:+ super().__init__()+ self.sibling = dspy.Predict("x -> y")+ self.flex = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ program = Program()+ program.flex._bind_code(ECHO_MODULE) # stand in for a GEPA-optimized decomposition+ program.sibling.signature = program.sibling.signature.with_instructions("tuned sibling")++ state = program.dump_state()+ assert state["flex"]["module_src"] == ECHO_MODULE # the code, not just its predictors+ assert "sibling" in state # a plain predictor stays a flat top-level entry+ # The Flex's internals belong to its own state; the parent must not dump them separately.+ assert not any(key.startswith("flex.") for key in state)++ program.save(path)++ reloaded = Program()+ assert "self.echo" not in reloaded.flex.module_src # a fresh program is on the baseline+ reloaded.load(path)++ assert reloaded.flex.module_src == ECHO_MODULE+ assert reloaded.sibling.signature.instructions == "tuned sibling"+++def test_flex_is_a_parameter_leaf_in_parent_programs() -> None:+ # Saving, GEPA discovery, and reset_copy all walk named_parameters(); a Flex appears there+ # as one state-bearing leaf (like dspy.Predict), so nothing inside it is enumerated — or+ # optimized — behind the code's back.+ class Program(dspy.Module):+ def __init__(self) -> None:+ super().__init__()+ self.sibling = dspy.Predict("x -> y")+ self.flex = Flex(Echo, interpreter_factory=lambda: MockInterpreter())++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ program = Program()+ # Even a predictor set directly on the Flex (the bridge itself never attaches any) stays+ # behind the leaf.+ program.flex.echo = dspy.Predict("q -> a")++ params = dict(program.named_parameters())+ assert isinstance(params["flex"], Flex)+ assert not any(name.startswith("flex.") for name in params)+ # The internals are not tunable units from any vantage point; the parent sees one leaf.+ assert program.flex.named_predictors() == []+ assert [n for n, _ in program.named_predictors()] == ["sibling"]+++def test_parent_set_lm_and_get_lm_are_symmetric() -> None:+ # set_lm writes to every LM-bearing leaf (Predicts and Flexes); get_lm reads the same set —+ # so set-then-get is an identity, a flex-only program has a unique LM rather than an error,+ # and a divergent flex LM is reported instead of silently ignored.+ class Program(dspy.Module):+ def __init__(self) -> None:+ super().__init__()+ self.sibling = dspy.Predict("x -> y")+ self.flex = Flex(Echo, interpreter_factory=MockInterpreter)++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ program = Program()+ lm = dspy.LM("openai/gpt-4o-mini")+ program.set_lm(lm)+ assert program.sibling.lm is lm+ assert program.flex.get_lm() is lm+ assert program.get_lm() is lm # read-your-writes++ program.flex.lm = dspy.LM("openai/gpt-4.1")+ with pytest.raises(ValueError, match="Multiple LMs"):+ program.get_lm()++ class OnlyFlex(dspy.Module):+ def __init__(self) -> None:+ super().__init__()+ self.flex = Flex(Echo, interpreter_factory=MockInterpreter)++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ only = OnlyFlex()+ only.set_lm(lm)+ assert only.get_lm() is lm # previously raised "Multiple LMs" on the empty predictor list+++def test_set_lm_is_stored_and_survives_rebind_and_save(tmp_path) -> None:+ # set_lm stores one module-level LM (one home), applied via context at forward time — so it+ # survives rebinds and save/load, unlike writing .lm onto per-forward internals that the+ # next forward rebuilds from scratch.+ flex = Flex(Echo, interpreter_factory=MockInterpreter)+ lm = dspy.LM("openai/gpt-4o-mini")+ flex.set_lm(lm)+ assert flex.get_lm() is lm++ flex._bind_code(ECHO_MODULE) # a rebind must not lose the module-level LM+ assert flex.get_lm() is lm++ path = tmp_path / "flex.json"+ flex.save(path)+ reloaded = Flex(Echo, interpreter_factory=MockInterpreter)+ reloaded.load(path)+ assert reloaded.get_lm() is not None+ assert reloaded.get_lm().model == lm.model++ # Like Predict: loading a state without an LM clears any previously set one.+ reloaded.load_state({"module_src": ECHO_MODULE, "lm": None})+ assert reloaded.get_lm() is None++ flex.set_lm(lm)+ assert flex.reset_copy().get_lm() is None # reset clears the LM, like Predict.reset+++@deno_required+def test_context_lm_reaches_bridged_calls() -> None:+ # A Flex holds no LM of its own (its state is only module_src); like any module, its+ # predictor calls resolve the ambient LM — including a caller's dspy.context, since the+ # bridged calls run host-side in the calling thread.+ program = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(ECHO_MODULE)+ with dspy.context(lm=DummyLM([{"a": "from-context-lm"}])):+ assert program(q="hello").a == "from-context-lm"+++def test_reset_copy_keeps_optimized_code_and_clears_the_lm() -> None:+ # reset_copy() calls .reset() on every parameter leaf. Predict.reset() clears demos/lm but+ # keeps tuned instructions; a Flex resets the same way — the LM is cleared, the optimized+ # module_src is kept — so optimizer chains (e.g. GEPA then BootstrapFewShot, which compiles+ # a reset copy) don't silently discard the discovered code. There is no other internal state:+ # predictors are rebuilt from the source inside each forward's bridge invocation.+ class Program(dspy.Module):+ def __init__(self) -> None:+ super().__init__()+ self.flex = Flex(Echo, interpreter_factory=lambda: MockInterpreter())++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ program = Program()+ program.flex._bind_code(ECHO_MODULE) # stand in for a GEPA-optimized decomposition+ program.flex.set_lm(dspy.LM("openai/gpt-4o-mini"))++ fresh = program.reset_copy()+ assert fresh.flex.module_src == ECHO_MODULE # the optimized code survives, like instructions+ assert fresh.flex.get_lm() is None # the LM is cleared, like Predict.reset+ assert program.flex.get_lm() is not None # the original is untouched+++def test_state_layout_unchanged_without_a_flex() -> None:+ # Programs with no Flex keep the flat `{param_path: state}` layout, so existing saved state+ # still loads.+ class Plain(dspy.Module):+ def __init__(self) -> None:+ super().__init__()+ self.first = dspy.Predict("x -> y")+ self.second = dspy.Predict("y -> z")++ def forward(self, **kwargs):+ return self.second(**self.first(**kwargs))++ assert sorted(Plain().dump_state()) == ["first", "second"]+++def test_save_program_roundtrips_via_cloudpickle(tmp_path) -> None:+ # save_program=True cloudpickles the live program object. The bridge (sessions, locks,+ # thread-locals) is excluded from the pickle and rebuilt session-less on load.+ class Program(dspy.Module):+ def __init__(self) -> None:+ super().__init__()+ self.sibling = dspy.Predict("x -> y")+ self.flex = Flex(Echo, interpreter_factory=MockInterpreter)++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ program = Program()+ program.flex._bind_code(ECHO_MODULE)+ assert "_bridge" not in program.flex.__getstate__()+ program.save(tmp_path, save_program=True)++ loaded = dspy.load(str(tmp_path), allow_pickle=True)+ assert loaded.flex.module_src == ECHO_MODULE+ assert loaded.flex._bridge._flex is loaded.flex+ assert loaded.flex._bridge._module_src == ECHO_MODULE+++@deno_required+def test_save_program_end_to_end_forward(tmp_path) -> None:+ """save_program=True with a live Deno session: reload elsewhere and run it."""+ program = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(ECHO_MODULE)+ program.save(tmp_path, save_program=True)++ dspy.configure(lm=DummyLM([{"a": "echoed-back"}]))+ loaded = dspy.load(str(tmp_path), allow_pickle=True)+ assert loaded.module_src == ECHO_MODULE+ result = loaded(q="hello")+ assert result.a == "echoed-back"+++@deno_required+def test_end_to_end_forward_after_save_load(tmp_path) -> None:+ """End-to-end: bind a plain dspy.Predict class, save it, reload into a fresh Flex, run it."""+ path = tmp_path / "program.json"+ program = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(ECHO_MODULE)+ program.save(path)++ dspy.configure(lm=DummyLM([{"a": "echoed-back"}]))+ reloaded = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ reloaded.load(path)+ assert "self.echo" in reloaded.module_src+ result = reloaded(q="hello")+ assert isinstance(result, dspy.Prediction)+ assert result.a == "echoed-back"diff --git a/tests/flex/test_flex_gepa.py b/tests/flex/test_flex_gepa.pynew file mode 100644index 000000000..861d9344f--- /dev/null+++ b/tests/flex/test_flex_gepa.py@@ -0,0 +1,541 @@+"""Tests for dspy.Flex code optimization inside dspy.GEPA.++Covers the behavior:+- a freshly constructed dspy.Flex binds a deterministic, LM-free baseline (a dspy.Module subclass;+ a single dspy.Predict when no tools are given),+- code-optimizable submodules are discovered by type (dspy.Flex), not by a duck-typed marker,+- GEPA's seed candidate mixes per-submodule *code* components (each a full `module_src`) with+ *instruction* components for non-flex predictors, excluding predictors that live inside a Flex,+- the adapter rebinds Flex code from a candidate and routes code components through the+ code proposer,+- GEPA requires a non-empty trainset.+"""++from __future__ import annotations++import shutil+import textwrap++import pytest++import dspy+from dspy.teleprompt.bootstrap_trace import FailedPrediction+from dspy.teleprompt.gepa.gepa import DspyGEPAResult+from dspy.teleprompt.gepa.gepa_flex_utils import enumerate_flex_submodules+from dspy.teleprompt.gepa.gepa_utils import DspyAdapter+from dspy.utils.dummies import DummyLM+from dspy.utils.exceptions import LMRateLimitError++deno_required = pytest.mark.skipif(shutil.which("deno") is None, reason="Deno is not installed")++# A plain dspy.Predict module class that binds without an LM (no RLM interpreter needed).+SIMPLE_MODULE = textwrap.dedent(+ """+ class EchoModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.p = dspy.Predict("q -> a")++ def forward(self, **inputs):+ r = self.p(q=inputs["q"])+ return dspy.Prediction(a=r.a)+ """+).strip()+++class Echo(dspy.Signature):+ """Echo the question as the answer."""++ q: str = dspy.InputField()+ a: str = dspy.OutputField()+++def _metric(gold, pred, trace=None, pred_name=None, pred_trace=None):+ return 1.0 if getattr(pred, "a", None) == gold.a else 0.0+++# --- baseline + discovery ----------------------------------------------------+++def test_flex_baseline_is_predict_and_lm_free() -> None:+ # No LM configured: constructing a Flex must not make any LM call — with no tools it binds the+ # deterministic dspy.Predict baseline (a dspy.Module subclass).+ program = dspy.Flex(Echo)+ assert "dspy.Predict(" in program.module_src+ assert "q: str -> a: str" in program.module_src # typed signature string+ assert "result.a" in program.module_src # unwraps the declared output+ assert "class EchoModule(dspy.Module)" in program.module_src+++def test_only_flex_instances_are_code_optimizable() -> None:+ """Discovery is by type. A module that merely looks flex-shaped is not rewritten: GEPA would+ hand it optimizer-authored source with nowhere safe to run it."""++ class LookAlike(dspy.Module):+ def _bind_code(self, src):+ raise AssertionError("must never be treated as a code component")++ def forward(self, **kwargs):+ return dspy.Prediction(a="x")++ class Prog(dspy.Module):+ def __init__(self):+ super().__init__()+ self.real = dspy.Flex(Echo)+ self.fake = LookAlike()++ def forward(self, **kwargs):+ return self.real(**kwargs)++ assert set(enumerate_flex_submodules(Prog())) == {"real"}+++def test_enumerate_finds_top_level_and_nested_flex() -> None:+ top = dspy.Flex(Echo)+ assert set(enumerate_flex_submodules(top)) == {"self"}++ class Prog(dspy.Module):+ def __init__(self):+ super().__init__()+ self.flex = dspy.Flex(Echo)+ self.sibling = dspy.Predict("x -> y")++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ prog = Prog()+ flex_paths = enumerate_flex_submodules(prog)+ assert set(flex_paths) == {"flex"} # the regular Predict is not code-optimizable+++# --- exclusion of Flex-internal predictors -----------------------------------+++def test_flex_internal_predictors_excluded_from_instruction_components() -> None:+ class Prog(dspy.Module):+ def __init__(self):+ super().__init__()+ self.flex = dspy.Flex(Echo)+ self.sibling = dspy.Predict("x -> y")++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ prog = Prog()+ # Only the sibling Predict gets instruction optimization; the Flex's internal predictors are+ # owned by its code and never reported as tunable units — from the parent (Parameter leaf) or+ # from the Flex itself (named_predictors() is empty) — so no exclusion logic is needed.+ assert [n for n, _ in prog.named_predictors()] == ["sibling"]+ assert prog.flex.named_predictors() == []+ assert set(enumerate_flex_submodules(prog)) == {"flex"}+++# --- adapter: build_program rebinds code, applies instructions ---------------+++def test_build_program_rebinds_flex_code() -> None:+ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=_metric, feedback_map={})++ candidate = {"self": SIMPLE_MODULE}+ rebuilt = adapter.build_program(candidate)++ assert 'dspy.Predict("q -> a")' in rebuilt.module_src+ assert "self.p" in rebuilt.module_src+ assert "dspy.RLM(" not in rebuilt.module_src # baseline replaced+ assert not hasattr(rebuilt, "p") # predictors are per-forward artifacts, never set on the Flex+++# --- adapter: selection eval passes the trace to a flex metric ---------------+++@deno_required+def test_selection_eval_passes_program_trace_to_declaring_metric() -> None:+ """The selection eval (capture_traces=False) must pass the execution trace to a metric that+ declares `program_trace`, so a trace-dependent score (e.g. an LLM-call penalty that rewards+ deterministic code) actually drives candidate selection."""+ seen: dict[str, object] = {}++ def trace_aware_metric(gold, pred, trace=None, pred_name=None, pred_trace=None, program_trace=None):+ seen["trace"] = trace+ seen["n_calls"] = len(program_trace) if program_trace is not None else None+ return 1.0++ student = dspy.Flex(Echo) # a flex submodule is present+ adapter = DspyAdapter(student_module=student, metric_fn=trace_aware_metric, feedback_map={})+ candidate = {"self": SIMPLE_MODULE} # one dspy.Predict -> one traced call+ ex = dspy.Example(q="hi", a="hi").with_inputs("q")++ dspy.configure(lm=DummyLM([{"a": "hi"}]))+ adapter.evaluate([ex], candidate, capture_traces=False) # the selection path++ assert seen.get("n_calls") == 1 # the metric received the trace, with the one call+ assert seen.get("trace") is None # eval-mode semantics of the `trace` argument are preserved+++@deno_required+def test_selection_eval_keeps_vanilla_semantics_for_legacy_metric() -> None:+ seen: dict[str, object] = {}++ def legacy_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):+ seen["trace"] = trace+ if trace is not None:+ return getattr(pred, "a", None) == gold.a # bootstrapping mode: strict bool+ return 0.75 # eval mode: continuous score++ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=legacy_metric, feedback_map={})+ candidate = {"self": SIMPLE_MODULE}+ ex = dspy.Example(q="hi", a="hi").with_inputs("q")++ dspy.configure(lm=DummyLM([{"a": "hi"}]))+ batch = adapter.evaluate([ex], candidate, capture_traces=False)++ assert seen.get("trace") is None # never fed the trace through the bootstrapping channel+ assert batch.scores == [0.75]+++@deno_required+def test_selection_eval_binds_metric_with_required_contract_params() -> None:+ """A metric written to the full GEPAFeedbackMetric signature with `trace`/`pred_name`/+ `pred_trace` REQUIRED (no defaults) must still bind at flex scoring time: flex passes those+ declared parameters as None rather than dropping them, so the metric doesn't raise TypeError."""+ seen: dict[str, object] = {}++ def strict_metric(gold, pred, trace, pred_name, pred_trace, program_trace=None):+ seen["trace"] = trace+ seen["pred_name"] = pred_name+ seen["pred_trace"] = pred_trace+ seen["n_calls"] = len(program_trace) if program_trace is not None else None+ return 1.0++ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=strict_metric, feedback_map={})+ candidate = {"self": SIMPLE_MODULE}+ ex = dspy.Example(q="hi", a="hi").with_inputs("q")++ dspy.configure(lm=DummyLM([{"a": "hi"}]))+ batch = adapter.evaluate([ex], candidate, capture_traces=False)++ assert batch.scores == [1.0] # bound and scored instead of raising TypeError+ assert seen.get("trace") is None and seen.get("pred_name") is None and seen.get("pred_trace") is None+ assert seen.get("n_calls") == 1 # program_trace still delivered to a metric that declares it+++# --- adapter: scores stay aligned to the batch when examples crash ------------++CRASHY_MODULE = textwrap.dedent(+ """+ class CrashyModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ if inputs["q"] == "boom":+ raise RuntimeError("runtime crash on this input")+ return dspy.Prediction(a=inputs["q"])+ """+).strip()+++@deno_required+def test_evaluate_scores_stay_aligned_when_an_example_crashes() -> None:+ """A code candidate that binds fine but raises at runtime on one input must still score+ one-per-example, aligned by position — the gepa engine pairs scores with example ids+ positionally, so a short list would credit example N+1's score to example N (and crash its+ state bookkeeping). The crashed slot carries a FailedPrediction with the error."""++ def exact_match(gold, pred, trace=None, pred_name=None, pred_trace=None):+ return 1.0 if getattr(pred, "a", None) == gold.a else 0.0++ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=exact_match, feedback_map={})+ candidate = {"self": CRASHY_MODULE}+ batch = [+ dspy.Example(q="hi", a="hi").with_inputs("q"),+ dspy.Example(q="boom", a="boom").with_inputs("q"), # raises inside forward+ dspy.Example(q="yo", a="yo").with_inputs("q"),+ ]++ result = adapter.evaluate(batch, candidate, capture_traces=False)++ assert result.scores == [1.0, 0.0, 1.0] # crashed example scores as a failure, in place+ assert isinstance(result.outputs[1], FailedPrediction) # the crash is captured, not dropped+ assert "runtime crash on this input" in result.outputs[1].completion_text+ assert getattr(result.outputs[2], "a", None) == "yo" # later examples keep their own results+++def test_crashed_examples_still_reach_code_reflective_records() -> None:+ """An example whose forward raised must reach the code proposer's reflective records with+ the failing input and the error — for a code candidate the crash IS the feedback. If it+ were dropped (bootstrap_trace_data's default), GEPA would score the slot as a failure yet+ reflection would never learn which input broke or how, and would keep proposing code with+ the same bug."""++ class Prog(dspy.Module):+ def __init__(self):+ super().__init__()+ self.flex = dspy.Flex(Echo) # a flex submodule routes evaluation through the trace path++ def forward(self, **kwargs):+ if kwargs["q"] == "boom":+ raise RuntimeError("runtime crash on this input")+ return dspy.Prediction(a=kwargs["q"])++ adapter = DspyAdapter(student_module=Prog(), metric_fn=_metric, feedback_map={})+ candidate = {"flex": SIMPLE_MODULE}+ batch = [+ dspy.Example(q="hi", a="hi").with_inputs("q"),+ dspy.Example(q="boom", a="boom").with_inputs("q"), # raises inside forward+ ]++ eval_batch = adapter.evaluate(batch, candidate, capture_traces=True)+ assert eval_batch.scores == [1.0, 0.0]++ recs = adapter.make_reflective_dataset(candidate, eval_batch, ["flex"])["flex"]+ assert len(recs) == 2 # the crashed example is not silently dropped from reflection+ crashed = recs[1]+ assert crashed["Inputs"] == {"q": "boom"} # the proposer sees which input broke+ assert "RuntimeError: runtime crash on this input" in str(crashed["Generated Outputs"]) # and how+++# --- adapter: which build failures are absorbed ------------------------------+++ ("label", "source"),+ [+ ("syntax error", "class Broken(dspy.Module)\n def forward(self): pass"),+ ("no module class", "x = 1"),+ (+ "raises while constructing predictors",+ "class Broken(dspy.Module):\n"+ " def __init__(self):\n"+ " super().__init__()\n"+ " self.p = dspy.Nope('q -> a')\n"+ " def forward(self, **inputs):\n"+ " return dspy.Prediction(a='x')",+ ),+ ],+)+def test_a_candidate_that_cannot_bind_scores_as_a_failure(label, source) -> None:+ # Every way a candidate's own source can break binding — parsing it, running it, or the host+ # rejecting what it asks for — is the search's problem to score, not to crash on.+ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=_metric, feedback_map={}, failure_score=-1.0)+ ex = dspy.Example(q="hi", a="hi").with_inputs("q")++ result = adapter.evaluate([ex, ex], {"self": source}, capture_traces=False)++ assert result.scores == [-1.0, -1.0], label+ # No usable prediction either way: a bind failure yields None (the batch never ran); a+ # source that binds but breaks at forward yields a FailedPrediction carrying the error.+ assert all(o is None or isinstance(o, FailedPrediction) for o in result.outputs), label+++def test_an_infrastructure_error_during_build_is_not_scored_as_a_failure(monkeypatch) -> None:+ """Only source-level failures are absorbed. An LM provider or rate-limit error is not the+ candidate's fault — swallowing it would silently score a whole run at the failure score and+ hand back an 'optimized' program chosen from garbage."""++ class RateLimitError(Exception):+ pass++ def boom(program, candidate):+ raise RateLimitError("429 from the provider")++ monkeypatch.setattr("dspy.teleprompt.gepa.gepa_utils.rebind_flex_code", boom)++ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=_metric, feedback_map={})+ ex = dspy.Example(q="hi", a="hi").with_inputs("q")++ with pytest.raises(RateLimitError):+ adapter.evaluate([ex], {"self": SIMPLE_MODULE}, capture_traces=False)+++# --- adapter: propose routes code keys to the code proposer ------------------+++def test_propose_new_texts_uses_code_proposer_for_code_keys() -> None:+ reflection = DummyLM([{"revised_source": SIMPLE_MODULE}])+ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=_metric, feedback_map={}, reflection_lm=reflection)+ ckey = "self"+ candidate = {ckey: student.module_src}+ reflective = {ckey: [{"Inputs": {"q": "x"}, "Generated Outputs": "wrong", "Feedback": "bad"}]}++ out = adapter.propose_new_texts(candidate, reflective, [ckey])+ assert 'dspy.Predict("q -> a")' in out[ckey]+++def test_code_proposer_infra_error_propagates() -> None:+ """An LM infrastructure failure during code proposal is not the candidate's fault. Keeping+ the original source would make every iteration 'succeed' with an unchanged candidate and+ silently burn the optimization budget; propagating matches the instruction-proposer paths+ (and the build-time policy in DspyAdapter.evaluate)."""++ class DownLM(DummyLM):+ def forward(self, *args, **kwargs):+ raise LMRateLimitError("429 from the provider", model="dummy")++ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=_metric, feedback_map={}, reflection_lm=DownLM([]))+ candidate = {"self": student.module_src}+ reflective = {"self": [{"Inputs": {"q": "x"}, "Generated Outputs": "wrong", "Feedback": "bad"}]}++ with pytest.raises(LMRateLimitError):+ adapter.propose_new_texts(candidate, reflective, ["self"])+++def test_code_proposer_bad_proposal_keeps_original() -> None:+ """A proposal-level failure that is not LM infrastructure (e.g. reflection output the+ proposal machinery can't handle) keeps that component's original source instead of+ crashing the run."""++ class GarbledLM(DummyLM):+ def forward(self, *args, **kwargs):+ raise RuntimeError("reflection output could not be handled")++ student = dspy.Flex(Echo)+ adapter = DspyAdapter(student_module=student, metric_fn=_metric, feedback_map={}, reflection_lm=GarbledLM([]))+ original = student.module_src+ candidate = {"self": original}+ reflective = {"self": [{"Inputs": {"q": "x"}, "Generated Outputs": "wrong", "Feedback": "bad"}]}++ out = adapter.propose_new_texts(candidate, reflective, ["self"])+ assert out == {"self": original}+++# --- GEPA.compile: seed mixes code + instruction components ------------------+++class _FakeResult:+ def __init__(self, best_candidate):+ self.best_candidate = best_candidate+ self.val_aggregate_scores = []+++def test_gepa_seed_mixes_code_and_instruction_components(monkeypatch) -> None:+ class Prog(dspy.Module):+ def __init__(self):+ super().__init__()+ self.flex = dspy.Flex(Echo)+ self.sibling = dspy.Predict("x -> y")++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ prog = Prog()+ captured = {}++ def fake_optimize(**kwargs):+ captured["seed"] = kwargs["seed_candidate"]+ return _FakeResult(best_candidate=kwargs["seed_candidate"])++ monkeypatch.setattr("gepa.optimize", fake_optimize)++ optimizer = dspy.GEPA(metric=_metric, reflection_lm=DummyLM([]), max_metric_calls=10)+ ex = dspy.Example(q="x", a="y", x="x", y="y").with_inputs("q", "x")+ optimizer.compile(prog, trainset=[ex], valset=[ex])++ seed = captured["seed"]+ code_key = "flex"+ assert code_key in seed # one code component for the whole Flex+ assert "class " in seed[code_key] # the value is the full module class source+ assert "sibling" in seed # the non-flex predictor's instruction+ # The Flex's internal predictors are NOT separate instruction components.+ assert not any(k.startswith("flex.") for k in seed)+++# --- DspyGEPAResult.to_dict: candidates keep their code components -----------+++def test_result_to_dict_keeps_flex_code_components() -> None:+ """to_dict() must serialize the same components GEPA optimized: the full module_src per Flex+ under its parameter path, and no entries for the Flex's internal predictors (whose+ instructions are written by that code, not optimized on their own)."""++ class Prog(dspy.Module):+ def __init__(self):+ super().__init__()+ self.flex = dspy.Flex(Echo)+ self.sibling = dspy.Predict("x -> y")++ def forward(self, **kwargs):+ return self.flex(**kwargs)++ prog = Prog()+ prog.flex._bind_code(SIMPLE_MODULE)++ result = DspyGEPAResult(+ candidates=[prog],+ parents=[[None]],+ val_aggregate_scores=[1.0],+ val_subscores=[{0: 1.0}],+ per_val_instance_best_candidates={0: {0}},+ discovery_eval_counts=[0],+ )+ (candidate,) = result.to_dict()["candidates"]++ code_key = "flex"+ assert candidate[code_key] == SIMPLE_MODULE # the optimized code survives serialization+ assert candidate["sibling"] == prog.sibling.signature.instructions+ # The Flex's internal predictors are not components; reporting them would misstate what was+ # optimized and silently stand in for the dropped code.+ assert not any(k.startswith("flex.") for k in candidate)+++# --- GEPA.compile: a non-empty trainset is required ------------------------------------+++def test_gepa_flex_module_requires_trainset() -> None:+ # A Flex with no trainset must raise, like any other module — GEPA needs examples to+ # score candidates against.+ student = dspy.Flex(Echo) # in-memory; baseline Predict+ optimizer = dspy.GEPA(metric=_metric, reflection_lm=DummyLM([]), max_metric_calls=10)+ with pytest.raises(AssertionError, match=r"[Tt]rainset"):+ optimizer.compile(student, trainset=[])+++def test_gepa_plain_module_requires_trainset() -> None:+ class Plain(dspy.Module):+ def __init__(self):+ super().__init__()+ self.p = dspy.Predict("q -> a")++ def forward(self, **kwargs):+ return self.p(**kwargs)++ optimizer = dspy.GEPA(metric=_metric, reflection_lm=DummyLM([]), max_metric_calls=10)+ with pytest.raises(AssertionError, match=r"[Tt]rainset"):+ optimizer.compile(Plain(), trainset=[])+++def test_metric_exception_scores_failure_instead_of_aborting() -> None:+ """Parity with non-Flex GEPA scoring: there the metric runs inside Evaluate's executor, so a+ raising metric costs that example failure_score. The flex path calls the metric directly and+ must convert the exception the same way rather than aborting the whole optimization."""+ from dspy.teleprompt.gepa.gepa_flex_utils import evaluate_with_trace++ class Echo2(dspy.Module):+ def forward(self, **kwargs):+ return dspy.Prediction(a=kwargs["q"])++ def metric(example, pred, trace=None):+ if example.q == "bad":+ raise RuntimeError("metric bug on this example")+ return 1.0++ batch = [dspy.Example(q="good").with_inputs("q"), dspy.Example(q="bad").with_inputs("q")]+ result = evaluate_with_trace(+ Echo2(),+ batch,+ metric_fn=metric,+ num_threads=1,+ failure_score=0.0,+ callback_metadata=None,+ capture_traces=True,+ )+ assert result.scores == [1.0, 0.0] # the raising example scores failure_score, the rest still count+ assert result.outputs[0].a == "good"diff --git a/tests/flex/test_flex_gepa_seed.py b/tests/flex/test_flex_gepa_seed.pynew file mode 100644index 000000000..edca43bcc--- /dev/null+++ b/tests/flex/test_flex_gepa_seed.py@@ -0,0 +1,61 @@+"""GEPA re-optimization seeds from the module's CURRENT source (incl. manual edits).++When a user edits a flexed module (or re-runs GEPA on a previously-optimized one) and+optimizes again, GEPA must build on the current implementation rather than starting over.+``GEPA.compile`` constructs each flex submodule's seed candidate as ``flex.module_src`` (see+gepa.py), reading the live source. These LM-free tests lock in that the current/edited source+is what gets seeded. A MockInterpreter factory keeps them Deno-free — only ``module_src`` is read,+not executed.+"""++from __future__ import annotations++import dspy+from dspy.predict.flex.flex import Flex+from dspy.teleprompt.gepa.gepa_flex_utils import enumerate_flex_submodules+from tests.mock_interpreter import MockInterpreter++EDITED_MODULE = (+ "class EchoModule(dspy.Module):\n"+ " def __init__(self):\n"+ " super().__init__()\n"+ ' self.echo = dspy.Predict("q -> a")\n'+ "\n"+ " def forward(self, q):\n"+ " out = self.echo(q=q)\n"+ " return dspy.Prediction(a=out.a.upper())"+)+++class Echo(dspy.Signature):+ """Echo the question as the answer."""++ q: str = dspy.InputField()+ a: str = dspy.OutputField()+++def test_gepa_discovers_edited_submodule_in_a_program() -> None:+ """A manual edit (or a prior GEPA run's output) is what gets seeded next time: mirror GEPA's+ discovery — enumerate_flex_submodules + the per-submodule seed value (flex.module_src)."""+ flex = Flex(Echo, interpreter_factory=lambda: MockInterpreter())+ flex._bind_code(EDITED_MODULE) # an in-session tweak, or the result of a prior GEPA run++ class Program(dspy.Module):+ def __init__(self) -> None:+ super().__init__()+ self.classifier = flex++ def forward(self, **kwargs):+ return self.classifier(**kwargs)++ program = Program()+ submodules = enumerate_flex_submodules(program)+ assert submodules, "the flex submodule should be discoverable by GEPA"++ seed_candidate: dict[str, str] = {}+ for sub_path, sub in submodules.items():+ seed_candidate[sub_path] = sub.module_src # the exact GEPA seed expression++ # Every seeded candidate carries the edit forward into the next GEPA run.+ assert seed_candidate+ assert all("out.a.upper()" in code for code in seed_candidate.values())diff --git a/tests/flex/test_flex_interpreter.py b/tests/flex/test_flex_interpreter.pynew file mode 100644index 000000000..0aa6a1b60--- /dev/null+++ b/tests/flex/test_flex_interpreter.py@@ -0,0 +1,854 @@+"""dspy.Flex(interpreter_factory=...) runs generated code inside a sandbox via the run-in-interpreter bridge.++Two layers of coverage:++* Unit (no Deno): the interpreter_factory is validated; the host-side bridge callbacks+ (``_Invocation.construct``/``.call``) build real predictors on the host, keep them in a+ per-forward registry (never on the Flex, so parallel forwards can't clobber each other), and+ run via the configured LM. These exercise everything the sandbox would drive, without Deno.+* Integration (Deno required, skipped otherwise): an optimized CoT-style module — with optimizer+ authored glue like ``import re`` — actually runs inside a ``dspy.PythonInterpreter`` with the LM call+ bridged back to the host, and adversarial file access is contained to the sandbox.+"""++from __future__ import annotations++import datetime+import enum+import inspect+import json+import shutil+import sys+import textwrap++import pydantic+import pytest++import dspy+from dspy.predict.flex import Flex, bridge+from dspy.primitives.code_interpreter import CodeExecutionError, CodeInterpreterError+from dspy.utils.dummies import DummyLM+from dspy.utils.exceptions import LMError, LMRateLimitError+from tests.mock_interpreter import MockInterpreter++deno_required = pytest.mark.skipif(shutil.which("deno") is None, reason="Deno is not installed")+++class Doubler(dspy.Signature):+ """Return double the input value."""++ value: int = dspy.InputField()+ result: int = dspy.OutputField()+++def shout(text: str) -> str:+ """Uppercase helper used to exercise user tools through the bridge."""+ return text.upper()+++class ShoutSig(dspy.Signature):+ """Uppercase the input text."""++ text: str = dspy.InputField()+ out: str = dspy.OutputField()+++# =============================================================================+# Normalization / construction (no Deno)+# =============================================================================+++def test_default_interpreter_factory_is_python_interpreter() -> None:+ # Like dspy.RLM, interpreter_factory defaults to dspy.PythonInterpreter (a class used as a+ # zero-arg factory). Inspect the constructor default so this needs no Deno.+ default = inspect.signature(Flex.__init__).parameters["interpreter_factory"].default+ assert default is dspy.PythonInterpreter+++@deno_required+def test_default_is_sandbox() -> None:+ # With no interpreter_factory argument, Flex builds a PythonInterpreter sandbox (needs Deno).+ flex = Flex(Doubler)+ assert flex._bridge is not None+ assert flex._interpreter_factory is dspy.PythonInterpreter+ assert isinstance(flex._interpreter_factory(), dspy.PythonInterpreter)+++def test_bare_instance_is_rejected() -> None:+ # There is no in-process mode and no shared-instance mode: like dspy.RLM, the constructor takes a+ # factory, not a live interpreter. Passing an instance is a TypeError.+ with pytest.raises(TypeError):+ Flex(Doubler, interpreter_factory=MockInterpreter()) # type: ignore[arg-type]+++def test_none_interpreter_factory_is_rejected() -> None:+ # None used to mean "run in-process"; that path is gone, so None is now invalid.+ with pytest.raises(TypeError):+ Flex(Doubler, interpreter_factory=None) # type: ignore[arg-type]+++def test_factory_is_used_as_is() -> None:+ flex = Flex(Doubler, interpreter_factory=lambda: MockInterpreter())+ assert flex._bridge is not None+ assert callable(flex._interpreter_factory)+++def test_bad_interpreter_type_raises() -> None:+ with pytest.raises(TypeError):+ Flex(Doubler, interpreter_factory=123) # type: ignore[arg-type]+++def test_tools_plus_interpreter_is_supported() -> None:+ flex = Flex(Doubler, tools=[shout], interpreter_factory=lambda: MockInterpreter())+ assert flex._bridge is not None+ # the user tool is registered (name -> callable) so sandbox glue can call it by name+ assert flex._bridge._tool_callables()["shout"] is shout+++# =============================================================================+# Host bridge callbacks (no Deno) — simulate what the sandbox drives+# =============================================================================+++def _bridged_flex():+ # MockInterpreter doesn't execute Python, so eager init attaches nothing; we drive the host+ # bridge callbacks directly, exactly as the sandbox shim would.+ return Flex(Doubler, interpreter_factory=lambda: MockInterpreter())+++def _bridged_flex_with(**kwargs):+ return Flex(Doubler, interpreter_factory=lambda: MockInterpreter(), **kwargs)+++def test_construct_builds_real_predictor_in_per_forward_registry() -> None:+ flex = _bridged_flex()+ inv = bridge._Invocation(flex._bridge, {})+ handle = inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})+ assert handle == "solve"+ assert isinstance(inv._predictors["solve"], dspy.ChainOfThought)+ # Nothing is set on the Flex itself: the module's surface stays clean, and+ # named_predictors() reports no tunable units — internals belong to the code.+ assert not hasattr(flex, "solve")+ assert flex.named_predictors() == []+++def test_construct_names_cannot_clobber_the_flex() -> None:+ # Construction writes to the per-forward registry, never setattr on the host Flex, so a+ # generated predictor named like a Flex attribute is just a handle — nothing to corrupt.+ flex = _bridged_flex()+ inv = bridge._Invocation(flex._bridge, {})+ for name in ("lm", "forward", "module_src", "_bridge"):+ assert inv.construct("Predict", "value: int -> result: int", name, {}) == name+ assert flex.lm is None # host attributes untouched+ assert callable(flex.forward)+ assert isinstance(flex._bridge, bridge.BridgeRuntime)+++def test_parallel_forwards_cannot_overwrite_each_others_predictors() -> None:+ # The regression the per-forward registry prevents: two forwards on one Flex (e.g. under+ # Evaluate(num_threads>1)) interleave as construct(A), construct(B), call(A). With predictors+ # attached to the shared Flex, A would run B's predictor; each invocation owning its own+ # registry keeps them isolated.+ flex = _bridged_flex()+ a = bridge._Invocation(flex._bridge, {})+ b = bridge._Invocation(flex._bridge, {})+ a.construct("Predict", "value: int -> result: int", "solve", {})+ b.construct("ChainOfThought", "question: str -> answer: str", "solve", {}) # same handle+ assert a._predictors["solve"] is not b._predictors["solve"]+ dspy.configure(lm=DummyLM([{"result": "4"}]))+ fields = a.call("solve", {"value": 2})+ assert fields == {"result": 4} # A's own predictor and signature, not B's+++def test_react_kinds_are_bridgeable_and_construct_on_host() -> None:+ # ReAct/ReActV2 are tool-calling agents: they run NO inner code (so take no interpreter) and call+ # named tools on the host. They just need whitelisting; the tools must be Flex-level to resolve.+ assert "ReAct" in bridge.BRIDGEABLE_KINDS and "ReActV2" in bridge.BRIDGEABLE_KINDS+ flex = Flex(ShoutSig, tools=[shout], interpreter_factory=lambda: MockInterpreter())+ inv = bridge._Invocation(flex._bridge, {})+ tools_payload = [{bridge.TOOL_MARKER: "shout"}]+ inv.construct("ReAct", "text: str -> out: str", "agent", {"tools": tools_payload})+ agent = inv._predictors["agent"]+ assert isinstance(agent, dspy.ReAct)+ # No interpreter is injected into a non-code-executing predictor.+ assert not hasattr(agent, "interpreter")+++def test_call_runs_predictor_via_host_lm() -> None:+ flex = _bridged_flex()+ inv = bridge._Invocation(flex._bridge, {})+ inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})+ dspy.configure(lm=DummyLM([{"reasoning": "two doubled is four", "result": "4"}]))+ fields = inv.call("solve", {"value": 2})+ assert fields["result"] == 4+ assert "reasoning" in fields+++def test_call_unknown_handle_raises() -> None:+ flex = _bridged_flex()+ with pytest.raises(CodeInterpreterError):+ bridge._Invocation(flex._bridge, {}).call("nope", {"value": 1})+++def test_tool_name_markers_resolve_to_real_tools() -> None:+ # The shim passes tools to a bridged sub-predictor by name; the host resolves them back.+ flex = Flex(Doubler, tools=[shout], interpreter_factory=lambda: MockInterpreter())+ assert flex._bridge._decode_tools([{"__dspy_tool__": "shout"}]) == [shout]+ with pytest.raises(CodeInterpreterError):+ flex._bridge._decode_tools({"__dspy_tool__": "not_a_tool"}) # module-authored / unknown tool+++def test_interpreter_is_not_serialized() -> None:+ flex = _bridged_flex()+ inv = bridge._Invocation(flex._bridge, {})+ inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})+ state = flex.dump_state()+ assert state.get("module_src") # the code persists+ assert "interpreter" not in json.dumps(state) # the runtime sandbox does not+++# =============================================================================+# bridge.py helpers (no Deno)+# =============================================================================+++def test_parse_module_class_name_prefers_forward() -> None:+ src = textwrap.dedent(+ """+ class Helper:+ pass++ class MyModule(dspy.Module):+ def forward(self, **inputs):+ return dspy.Prediction()+ """+ )+ assert bridge.parse_module_class_name(src) == "MyModule"+++def test_prediction_to_fields_rejects_non_jsonable() -> None:+ assert bridge.prediction_to_fields(dspy.Prediction(a=1, b="x")) == {"a": 1, "b": "x"}+ with pytest.raises(CodeInterpreterError):+ bridge.prediction_to_fields(dspy.Prediction(a=object()))+++def test_prediction_to_fields_serializes_json_mode_values() -> None:+ # Everything dspy.Predict can output must cross the boundary: datetimes, enums, and pydantic+ # models holding them serialize in JSON mode (python-mode model_dump used to leave them as+ # live objects, and Flex rejected outputs a plain Predict returns and parses just fine).+ class Color(enum.Enum):+ RED = "red"++ class Event(pydantic.BaseModel):+ when: datetime.datetime+ color: Color++ fields = bridge.prediction_to_fields(+ dspy.Prediction(+ event=Event(when=datetime.datetime(2026, 1, 1), color=Color.RED),+ when=datetime.datetime(2026, 1, 2),+ color=Color.RED,+ )+ )+ assert fields["event"] == {"when": "2026-01-01T00:00:00", "color": "red"}+ assert fields["when"] == "2026-01-02T00:00:00"+ assert fields["color"] == "red"+++# =============================================================================+# Backend-agnostic shim + predictor-call budget (Phase 2, no Deno)+# =============================================================================+++def test_shim_reaches_host_only_via_registered_tools() -> None:+ # The bridge must not depend on Deno/Pyodide internals — only on the CodeInterpreter.tools+ # contract — so any backend (a local microVM, gVisor, ...) can drive it.+ shim = bridge.SHIM_SETUP+ assert "pyodide" not in shim.lower()+ assert "_js_tool_call" not in shim+ assert "run_sync" not in shim+ # It does call the registered host tools by name.+ assert bridge.CONSTRUCT_TOOL in shim+ assert bridge.CALL_TOOL in shim+++def test_shim_file_literals_match_bridge_constants() -> None:+ # The shim lives in a sibling .py file with literal protocol strings (no interpolation); guard+ # against drift between those literals and bridge.py's host-side constants.+ shim = bridge.SHIM_SETUP+ for token in (bridge.CONSTRUCT_TOOL, bridge.CALL_TOOL, bridge.TOOL_MARKER, bridge.SIGNATURE_MARKER):+ assert token in shim, f"{token!r} missing from the shim file"+ for kind in bridge.BRIDGEABLE_KINDS:+ assert kind in shim, f"bridgeable kind {kind!r} missing from the shim file"+++def test_max_predictor_calls_is_plumbed_to_bridge() -> None:+ flex = _bridged_flex_with(max_predictor_calls=7)+ assert flex._bridge._max_predictor_calls == 7+++def test_predictor_call_budget_is_enforced() -> None:+ # The budget lives in the per-forward _Invocation, so every forward starts fresh.+ flex = _bridged_flex_with(max_predictor_calls=2)+ dspy.configure(lm=DummyLM([{"reasoning": "r", "result": str(i)} for i in range(3)]))+ inv = bridge._Invocation(flex._bridge, {})+ inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})+ inv.call("solve", {"value": 1}) # 1st+ inv.call("solve", {"value": 1}) # 2nd+ with pytest.raises(CodeInterpreterError, match="budget"):+ inv.call("solve", {"value": 1}) # 3rd exceeds the cap+ # A new forward gets a new budget — and its own registry, so it constructs anew.+ fresh = bridge._Invocation(flex._bridge, {})+ fresh.construct("ChainOfThought", "value: int -> result: int", "solve", {})+ fresh.call("solve", {"value": 1})+++def test_predictor_call_budget_can_be_disabled() -> None:+ flex = _bridged_flex_with(max_predictor_calls=None)+ assert flex._bridge._max_predictor_calls is None+ dspy.configure(lm=DummyLM([{"reasoning": "r", "result": str(i)} for i in range(5)]))+ inv = bridge._Invocation(flex._bridge, {})+ inv.construct("ChainOfThought", "value: int -> result: int", "solve", {})+ for _ in range(5): # no cap -> all calls go through+ inv.call("solve", {"value": 1})+++# =============================================================================+# LM infrastructure errors keep their type across the boundary (no Deno)+# =============================================================================+# The JSON-RPC boundary flattens a host-tool exception into a message string, so a bridged+# predictor's provider failure would surface as a generic CodeExecutionError. The bridge stashes+# the typed error on the _Invocation and stamps a tag into the message that crosses the boundary;+# forward() re-raises the typed error only when the failure that killed the run carries that tag.+# Infrastructure failures are not the generated code's fault, and callers discriminate by type+# (e.g. catching dspy.LMError to retry) — but a failure the generated code recovered from must+# not repaint a later, unrelated crash as an infrastructure error.+++class _RateLimitedLM(DummyLM):+ """A host LM whose provider is down: every call raises a typed LM infrastructure error."""++ def forward(self, *args, **kwargs):+ raise LMRateLimitError("429 from the provider", model="dummy")+++def _boundary_flex(drive):+ """A Flex whose MockInterpreter drives the bridge callbacks the way the sandbox shim would.++ ``drive(tools)`` plays the generated forward. Any exception it lets escape is flattened to a+ ``CodeExecutionError`` carrying only the type name in its message, exactly like the real+ boundary (``PythonInterpreter._handle_tool_call`` + the execute error path).+ """+ interps: list[MockInterpreter] = []++ def execute_fn(code, variables):+ if not variables:+ return "" # the shim-setup and class-definition steps+ try:+ drive(interps[-1].tools)+ except Exception as e:+ raise CodeExecutionError(f"{type(e).__name__}: {e}") from None+ return ""++ def factory():+ interps.append(MockInterpreter(execute_fn=execute_fn))+ return interps[-1]++ return Flex(Doubler, interpreter_factory=factory)+++def test_bridged_lm_failure_keeps_its_type_across_the_boundary() -> None:+ # An uncaught provider failure inside the generated forward reaches the caller as the real+ # typed error, not as a CodeExecutionError that merely mentions the type name — so it can't+ # be mistaken for (or absorbed as) a broken code candidate.+ def drive(tools):+ tools[bridge.CONSTRUCT_TOOL](+ kind="Predict", signature="value: int -> result: int", attr_name="solve", kwargs={}+ )+ tools[bridge.CALL_TOOL](handle="solve", inputs={"value": 2})++ flex = _boundary_flex(drive)+ dspy.configure(lm=_RateLimitedLM([]))+ with pytest.raises(LMRateLimitError) as err:+ flex(value=2)+ assert isinstance(err.value, LMError) # catchable by the family callers actually handle+ assert isinstance(err.value.__cause__, CodeExecutionError) # the sandbox error stays attached+++def test_recovered_lm_failure_does_not_hijack_a_later_crash() -> None:+ # If the generated code catches the LM failure and keeps going (proven by a later bridge+ # callback), a subsequent crash is its own bug: the stale LMError must not repaint a+ # candidate error as an infrastructure one.+ def drive(tools):+ tools[bridge.CONSTRUCT_TOOL](+ kind="Predict", signature="value: int -> result: int", attr_name="solve", kwargs={}+ )+ try:+ tools[bridge.CALL_TOOL](handle="solve", inputs={"value": 2})+ except Exception:+ pass # the generated code swallows the flattened boundary error and falls back+ tools[bridge.CONSTRUCT_TOOL](kind="Predict", signature="q -> a", attr_name="fallback", kwargs={})+ raise NameError("name 'oops' is not defined")++ flex = _boundary_flex(drive)+ dspy.configure(lm=_RateLimitedLM([]))+ with pytest.raises(CodeExecutionError, match="NameError"):+ flex(value=2)+++def test_recovered_lm_failure_does_not_hijack_a_local_later_crash() -> None:+ # Same recovery, but with NO bridge call between it and the crash — the host can't observe+ # the recovery, so only the tag correlation tells the stale LMError apart from the local bug.+ def drive(tools):+ tools[bridge.CONSTRUCT_TOOL](+ kind="Predict", signature="value: int -> result: int", attr_name="solve", kwargs={}+ )+ try:+ tools[bridge.CALL_TOOL](handle="solve", inputs={"value": 2})+ except Exception:+ pass # the generated code recovers from the provider failure+ raise NameError("later sandbox bug")++ flex = _boundary_flex(drive)+ dspy.configure(lm=_RateLimitedLM([]))+ with pytest.raises(CodeExecutionError, match="NameError"):+ flex(value=2)+++# =============================================================================+# Code-executing sub-predictors inherit the Flex sandbox backend (no Deno)+# =============================================================================+# A bridged CodeAct/ProgramOfThought/RLM runs its OWN inner code. It must run that code in the+# sandbox backend chosen for Flex (via the Flex interpreter factory), not a separate default+# Deno/Pyodide sandbox — otherwise sandboxing the Flex glue but not the code its sub-predictors+# execute would be a hole.+++def test_accepts_interpreter_factory_only_for_code_executing_kinds() -> None:+ # Introspection-based, not a hardcoded list, so it tracks the real constructors.+ assert bridge._accepts_interpreter_factory(dspy.CodeAct)+ assert bridge._accepts_interpreter_factory(dspy.ProgramOfThought)+ assert bridge._accepts_interpreter_factory(dspy.RLM) # type: ignore[arg-type] # dspy re-exports RLM oddly+ # Pure-LM and tool-calling predictors run no inner code, so no interpreter factory is injected.+ assert not bridge._accepts_interpreter_factory(dspy.Predict)+ assert not bridge._accepts_interpreter_factory(dspy.ChainOfThought)+ assert not bridge._accepts_interpreter_factory(dspy.ReAct)+ assert not bridge._accepts_interpreter_factory(dspy.ReActV2) # type: ignore[arg-type] # re-export quirk+++def test_bridged_codeact_inherits_flex_interpreter_backend() -> None:+ # CodeAct requires function tools; `shout` is a Flex-level tool, resolvable on the host.+ flex = Flex(ShoutSig, tools=[shout], interpreter_factory=lambda: MockInterpreter())+ inv = bridge._Invocation(flex._bridge, {})+ tools_payload = [{bridge.TOOL_MARKER: "shout"}] # how the shim passes a tool by name+ inv.construct("CodeAct", "text: str -> out: str", "act", {"tools": tools_payload})+ # The sub-predictor makes its per-forward interpreter from the Flex factory (the configured+ # backend), not a default PythonInterpreter.+ act = inv._predictors["act"]+ assert isinstance(act, dspy.CodeAct)+ assert act._interpreter_factory is flex._bridge._factory+ assert isinstance(act._interpreter_factory(), MockInterpreter)+++def test_sub_interpreter_factory_makes_a_fresh_interpreter_per_call() -> None:+ flex = _bridged_flex() # a real factory (lambda: MockInterpreter())+ factory = flex._bridge._sub_interpreter_factory()+ assert factory is flex._bridge._factory # the sub-predictor gets the Flex factory itself+ a, b = factory(), factory()+ assert isinstance(a, MockInterpreter) and isinstance(b, MockInterpreter)+ assert a is not b # so each forward is isolated in its own interpreter+++# =============================================================================+# Integration (Deno) — the real run-in-interpreter bridge+# =============================================================================++# An optimized CoT module with optimizer-authored glue: the ChainOfThought call must bridge to the+# host, while the import/regex/int parsing run in the sandbox.+COT_GLUE_MODULE = textwrap.dedent(+ """+ class DoublerModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.solve = dspy.ChainOfThought("value: int -> result: str")++ def forward(self, **inputs):+ r = self.solve(value=inputs["value"])+ try:+ answer = int(r.result)+ except (ValueError, TypeError):+ import re+ m = re.search(r"-?\\d+", str(r.result))+ answer = int(m.group()) if m else 0+ return dspy.Prediction(result=answer)+ """+).strip()+++@deno_required+def test_optimized_cot_runs_in_sandbox_with_lm_bridged() -> None:+ dspy.configure(lm=DummyLM([{"reasoning": "double of 21 is 42", "result": "the result is 42"}]))+ flex = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter())+ flex._bind_code(COT_GLUE_MODULE)+ # forward() runs entirely in the sandbox; self.solve(...) bridges to the host DummyLM, and the+ # `import re` / int-parsing glue executes inside Pyodide.+ out = flex(value=21)+ assert isinstance(out, dspy.Prediction)+ assert out.result == 42+ # The predictor lived in the forward's invocation, not on the module: nothing is attached,+ # so parallel forwards on one Flex cannot overwrite each other's predictors.+ assert not hasattr(flex, "solve")+++# Adversarial module: tries to read a host file. In the sandbox this hits Pyodide's virtual FS, which+# has no such file, so it is contained — the host file is never exposed.+def _evil_read_module(host_path: str) -> str:+ return textwrap.dedent(+ f"""+ class DoublerModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ try:+ with open({host_path!r}) as f:+ leaked = f.read()+ except Exception as e:+ leaked = "BLOCKED:" + type(e).__name__+ return dspy.Prediction(result=leaked)+ """+ ).strip()+++@deno_required+def test_forward_runs_on_a_fresh_instance_each_call() -> None:+ """Each forward instantiates the module anew in the sandbox — instance state does not carry+ across calls. This is deliberate: GEPA scores every rollout under these semantics, so+ deployment must match; it also keeps scores independent of evaluation order and of which+ interpreter serves the call. The primitives catalog tells the proposer as much.+ """+ counter_src = textwrap.dedent(+ """+ class DoublerModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.calls = 0++ def forward(self, value):+ self.calls += 1+ return dspy.Prediction(result=self.calls)+ """+ ).strip()++ program = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(counter_src)+ assert program(value=1).result == 1+ assert program(value=1).result == 1 # not 2: a persistent instance would diverge from scoring+++@deno_required+def test_bridged_program_survives_save_load(tmp_path) -> None:+ dspy.configure(lm=DummyLM([{"reasoning": "double of 21 is 42", "result": "the result is 42"}]))+ program = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter())+ path = tmp_path / "prog.json"+ program._bind_code(COT_GLUE_MODULE)+ program.save(path)++ # The interpreter is a runtime dependency (like the LM): the saved code persists, the sandbox does not.+ assert "interpreter" not in path.read_text()++ # Reconstruct with a fresh interpreter factory and load the saved code.+ reloaded = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter())+ reloaded.load(path)+ assert reloaded.module_src and "self.solve" in reloaded.module_src+ out = reloaded(value=21) # still runs in the sandbox, LM bridged to the host+ assert out.result == 42+++# A module whose glue calls a user-provided tool directly (no LM): proves Flex tools are callable+# from inside the sandbox via the host bridge.+TOOL_GLUE_MODULE = textwrap.dedent(+ """+ class ShoutModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ return dspy.Prediction(out=shout(inputs["text"]))+ """+).strip()+++@deno_required+def test_user_tool_is_callable_from_sandbox_glue() -> None:+ flex = Flex(ShoutSig, tools=[shout], interpreter_factory=lambda: dspy.PythonInterpreter())+ flex._bind_code(TOOL_GLUE_MODULE)+ # shout() runs on the host (bridged from the sandbox); no LM involved.+ assert flex(text="hello").out == "HELLO"+++RUNAWAY_MODULE = textwrap.dedent(+ """+ class DoublerModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.solve = dspy.ChainOfThought("value: int -> result: str")++ def forward(self, **inputs):+ n = 0+ for _ in range(100): # runaway glue: would drive 100 host LM calls+ self.solve(value=inputs["value"])+ n += 1+ return dspy.Prediction(result=n)+ """+).strip()+++@deno_required+def test_runaway_glue_is_stopped_by_budget_in_sandbox() -> None:+ dspy.configure(lm=DummyLM([{"reasoning": "r", "result": "1"} for _ in range(50)]))+ flex = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter(), max_predictor_calls=3)+ flex._bind_code(RUNAWAY_MODULE)+ with pytest.raises(Exception) as excinfo: # the budget error propagates out of the sandbox+ flex(value=1)+ assert "budget" in str(excinfo.value).lower()+++# The real-boundary counterparts of the fake-boundary LM-error tests above: the tag call() stamps+# into the boundary message must survive the actual Deno/Pyodide JSON-RPC round trip.+LM_FAILURE_MODULE = textwrap.dedent(+ """+ class DoublerModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.solve = dspy.Predict("value: int -> result: int")++ def forward(self, **inputs):+ r = self.solve(value=inputs["value"])+ return dspy.Prediction(result=r.result)+ """+).strip()++RECOVERED_LM_FAILURE_MODULE = textwrap.dedent(+ """+ class DoublerModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.solve = dspy.Predict("value: int -> result: int")++ def forward(self, **inputs):+ try:+ r = self.solve(value=inputs["value"])+ except Exception:+ pass # recover from the provider failure...+ return oops # ...then crash locally, with no bridge call in between (NameError)+ """+).strip()+++@deno_required+def test_lm_failure_type_survives_the_real_sandbox_boundary() -> None:+ flex = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter())+ flex._bind_code(LM_FAILURE_MODULE)+ dspy.configure(lm=_RateLimitedLM([]))+ with pytest.raises(LMRateLimitError):+ flex(value=2)+++@deno_required+def test_recovered_lm_failure_does_not_hijack_through_the_real_sandbox() -> None:+ flex = Flex(Doubler, interpreter_factory=lambda: dspy.PythonInterpreter())+ flex._bind_code(RECOVERED_LM_FAILURE_MODULE)+ dspy.configure(lm=_RateLimitedLM([]))+ with pytest.raises(CodeInterpreterError) as err:+ flex(value=2)+ assert not isinstance(err.value, LMError) # the local NameError is not repainted as infra+ assert "NameError" in str(err.value) or "oops" in str(err.value)+++# A module that constructs and calls a dspy.CodeAct with a Flex-level tool. CodeAct runs its own+# LM-authored code in an interpreter; under the bridge that interpreter is a fresh instance from the+# Flex factory, so CodeAct's code runs in the SAME sandbox backend configured for Flex.+CODEACT_MODULE = textwrap.dedent(+ """+ class ShoutModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.act = dspy.CodeAct("text: str -> out: str", tools=[shout])++ def forward(self, **inputs):+ r = self.act(text=inputs["text"])+ return dspy.Prediction(out=r.out)+ """+).strip()+++@deno_required+def test_bridged_codeact_runs_in_flex_sandbox() -> None:+ # CodeAct generates code that calls the (Flex-level) `shout` tool, then extracts the answer.+ dspy.configure(+ lm=DummyLM(+ [+ {"generated_code": "```python\nprint(shout('hello'))\n```", "finished": True},+ {"reasoning": "the code printed HELLO", "out": "HELLO"},+ ]+ )+ )++ def backend_factory():+ return dspy.PythonInterpreter()++ flex = Flex(ShoutSig, tools=[shout], interpreter_factory=backend_factory)+ flex._bind_code(CODEACT_MODULE)+ out = flex(text="hello")+ assert out.out == "HELLO"+ # (That the sub-CodeAct builds its interpreter from the very factory Flex was given is+ # asserted in test_bridged_codeact_inherits_flex_interpreter_backend.)+++# A module that constructs and calls a dspy.ReAct with a Flex-level tool. ReAct runs no code of its+# own: the LM picks a tool, the host runs it, and the LM extracts the answer — all bridged from the+# sandbox where the module's glue runs.+REACT_MODULE = textwrap.dedent(+ """+ class ShoutModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.agent = dspy.ReAct("text: str -> out: str", tools=[shout])++ def forward(self, **inputs):+ r = self.agent(text=inputs["text"])+ return dspy.Prediction(out=r.out)+ """+).strip()+++@deno_required+def test_bridged_react_runs_in_flex_sandbox() -> None:+ dspy.configure(+ lm=DummyLM(+ [+ {"next_thought": "shout it", "next_tool_name": "shout", "next_tool_args": {"text": "hello"}},+ {"next_thought": "done", "next_tool_name": "finish", "next_tool_args": {}},+ {"reasoning": "the tool returned HELLO", "out": "HELLO"},+ ]+ )+ )+ flex = Flex(ShoutSig, tools=[shout], interpreter_factory=lambda: dspy.PythonInterpreter())+ flex._bind_code(REACT_MODULE)+ out = flex(text="hello")+ # The agent's glue ran in the sandbox; the `shout` tool call and every LM call bridged to the host.+ assert out.out == "HELLO"+++@deno_required+def test_adversarial_host_file_access_is_contained(tmp_path) -> None:+ secret = tmp_path / "host_secret.txt"+ secret.write_text("TOP-SECRET")+ assert secret.read_text() == "TOP-SECRET" # readable in-process on the host++ class Leak(dspy.Signature):+ value: int = dspy.InputField()+ result: str = dspy.OutputField()++ flex = Flex(Leak, interpreter_factory=lambda: dspy.PythonInterpreter())+ flex._bind_code(_evil_read_module(str(secret)))+ out = flex(value=1)+ # The sandbox could not reach the host file; the secret never crossed the boundary.+ assert "TOP-SECRET" not in str(out.result)+ assert str(out.result).startswith("BLOCKED")+++@deno_required+def test_nested_rlm_tool_provenance() -> None:+ """A GEPA-authored RLM spawns a second RLM through a host tool, and a host-provided tool is+ threaded down to both. Assert the trust boundary AND that each layer is a distinct interpreter:++ - each layer runs in a Pyodide sandbox (its ``sys.platform`` is 'emscripten', not the host's);+ - each layer is its OWN interpreter, not one shared sandbox. A global counter persists WITHIN+ an interpreter but resets in a fresh one, so the Flex glue counts 1 then 2 in interp A, while+ each nested agent sees 1 (a fresh namespace) — if they shared the glue's interpreter it would+ read 3, then 4;+ - the host-provided ``probe`` tool, wherever it is called from, executes in the HOST process.++ A code-executing agent's REPL has no ``dspy`` shim, so the second RLM is spawned via a host tool+ (``spawn_inner``) that constructs and runs it on the host, passing ``probe`` further down; only+ that sub-agent's ``out`` string crosses back, so its trajectory never touches the Flex bridge.+ """+ host_platform = sys.platform+ assert host_platform != "emscripten" # sanity: the test itself is the real host, not the sandbox+ record: dict[str, dict[str, str]] = {}+ host_seen: dict[str, str] = {}++ def probe(layer: str, platform: str, count: str) -> str:+ # Host-provided tool: runs in the host process. Records, per layer, the platform and the+ # per-interpreter counter value that layer reported, plus the platform `probe` itself runs in.+ record[layer] = {"platform": platform, "count": count}+ host_seen["platform"] = __import__("sys").platform+ return "ok"++ def spawn_inner(task: str) -> str:+ # Host tool that spawns a SECOND code-executing agent (a nested interpreter), passing the+ # host `probe` tool down to it. Runs on the host; only the inner RLM's `out` string is+ # returned, so nothing but a string crosses back.+ inner = dspy.RLM("task: str -> out: str", tools=[probe])+ return inner(task=task).out++ class ProbeSig(dspy.Signature):+ task: str = dspy.InputField()+ out: str = dspy.OutputField()++ # Each agent bumps a global counter and reports it; a fresh interpreter starts the count at 1.+ outer_code = (+ "```python\nimport sys\ng = globals()\ng['_n'] = g.get('_n', 0) + 1\n"+ "probe(layer='outer_rlm', platform=sys.platform, count=str(g['_n']))\n"+ "spawn_inner(task='go')\nSUBMIT(out='outer')\n```"+ )+ inner_code = (+ "```python\nimport sys\ng = globals()\ng['_n'] = g.get('_n', 0) + 1\n"+ "probe(layer='inner_rlm', platform=sys.platform, count=str(g['_n']))\nSUBMIT(out='inner')\n```"+ )+ dspy.configure(+ lm=DummyLM(+ [+ {"reasoning": "probe here, then spawn the sub-agent", "code": outer_code},+ {"reasoning": "probe from the sub-agent", "code": inner_code},+ ]+ )+ )++ module = textwrap.dedent(+ """+ class ProbeModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.rlm = dspy.RLM("task: str -> out: str", tools=[probe, spawn_inner])++ def forward(self, **inputs):+ import sys+ g = globals()+ g["_n"] = g.get("_n", 0) + 1 # interp A -> 1+ probe(layer="glue_1", platform=sys.platform, count=str(g["_n"]))+ g["_n"] = g.get("_n", 0) + 1 # same interp -> 2+ probe(layer="glue_2", platform=sys.platform, count=str(g["_n"]))+ self.rlm(task=inputs["task"])+ return dspy.Prediction(out="done")+ """+ ).strip()++ flex = Flex(ProbeSig, tools=[probe, spawn_inner], interpreter_factory=lambda: dspy.PythonInterpreter())+ flex._bind_code(module)+ flex(task="go")++ # Every layer ran inside a Pyodide sandbox, not the host process:+ for layer in ("glue_1", "glue_2", "outer_rlm", "inner_rlm"):+ assert record[layer]["platform"] == "emscripten", f"Failed at {layer}."+ # The counter persists WITHIN the Flex glue's interpreter (1 then 2)...+ assert record["glue_1"]["count"] == "1"+ assert record["glue_2"]["count"] == "2"+ # ...but each nested agent starts fresh at 1 — proving it is its OWN interpreter, not the glue's+ # (which would read 3, then 4) and not each other's:+ assert record["outer_rlm"]["count"] == "1"+ assert record["inner_rlm"]["count"] == "1"+ # The host-provided tool itself executed in the host process, wherever it was called from:+ assert host_seen["platform"] == host_platformdiff --git a/tests/flex/test_flex_output_types.py b/tests/flex/test_flex_output_types.pynew file mode 100644index 000000000..fbea00efa--- /dev/null+++ b/tests/flex/test_flex_output_types.py@@ -0,0 +1,462 @@+"""dspy.Flex honours its signature's declared output types.++Everything the generated code returns crosses the sandbox boundary as JSON, so a field declared+`int`, `list[str]`, or a pydantic model would otherwise come back to the caller as a bare string or+dict — unlike every other module over the same signature. Two halves are covered here:++* the *baseline source* names custom types (`text: str -> person: Person`), so the sub-predictor is+ actually asked for a Person and the host can resolve that name when it rebuilds the signature;+* the *return path* parses what comes back against the annotation, so a Flex returns the same types+ as a dspy.Predict would — and a candidate that returns the wrong shape raises a clear error+ instead of handing the metric a Prediction that breaks later.++That error is what GEPA catches: a code candidate whose output doesn't match the declared type is+dropped from trace capture and scored at the failure score, in place, like any other runtime fault.+"""++from __future__ import annotations++import enum+import json+import logging+import shutil+import textwrap+from typing import Callable++import pydantic+import pytest++import dspy+from dspy.predict.flex import Flex, bridge+from dspy.primitives.code_interpreter import CodeInterpreterError+from dspy.teleprompt.bootstrap_trace import FailedPrediction+from dspy.teleprompt.gepa.gepa_utils import DspyAdapter+from dspy.utils.dummies import DummyLM+from tests.flex import _offstack_types # imported as a module: `Contact` is never a name here+from tests.mock_interpreter import MockInterpreter++deno_required = pytest.mark.skipif(shutil.which("deno") is None, reason="Deno is not installed")+++class Person(pydantic.BaseModel):+ name: str+ age: int+++class Color(enum.Enum):+ RED = "red"+ BLUE = "blue"+++class Extract(dspy.Signature):+ """Extract the person described by the text."""++ text: str = dspy.InputField()+ person: Person = dspy.OutputField()+++class Roster(dspy.Signature):+ text: str = dspy.InputField()+ people: list[Person] = dspy.OutputField()+ color: Color = dspy.OutputField()+ count: int = dspy.OutputField()+++def _sandbox() -> dspy.PythonInterpreter:+ return dspy.PythonInterpreter()+++# =============================================================================+# The baseline source names the declared types (no Deno)+# =============================================================================+++def test_baseline_signature_string_names_custom_types() -> None:+ # The generated code can only refer to a type by name, so the rendered signature string has to+ # carry it: without `person: Person` the sub-predictor is asked for a plain string and the LM is+ # never shown the model's schema.+ program = Flex(Extract, interpreter_factory=MockInterpreter)+ assert "text: str -> person: Person" in program.module_src+++def test_baseline_signature_string_names_generic_and_enum_types() -> None:+ program = Flex(Roster, interpreter_factory=MockInterpreter)+ assert "people: list[Person]" in program.module_src+ assert "color: Color" in program.module_src+ assert "count: int" in program.module_src+++def test_baseline_omits_a_type_the_signature_parser_cannot_read_back() -> None:+ # An annotation the signature-string grammar cannot carry is emitted untyped rather than+ # rendered into source that would fail to bind.+ class Unrenderable(dspy.Signature):+ text: str = dspy.InputField()+ fn: Callable[[int], int] = dspy.OutputField()++ program = Flex(Unrenderable, interpreter_factory=MockInterpreter)+ assert "'text: str -> fn'" in program.module_src # the field is there, with no type annotation+++@deno_required+def test_custom_type_resolves_when_it_is_not_on_the_calling_stack() -> None:+ """The regression `bridge._resolve_signature` exists for.++ dspy resolves a type named in a signature string by walking caller frames for the bare name,+ so `from myapp.models import Contact` accidentally works while `import myapp.models as models`+ does not — the same program, written two ways, with only one of them binding. `Contact` lives+ in `_offstack_types` and is reached only through the module object, so nothing on the stack+ holds the name and the explicit `custom_types` handoff is what has to resolve it.+ """+ source = textwrap.dedent(+ """+ class M(dspy.Module):+ def __init__(self):+ super().__init__()+ self.p = dspy.Predict("text: str -> contact: Contact")++ def forward(self, **inputs):+ return dspy.Prediction(contact=self.p(text=inputs["text"]).contact)+ """+ ).strip()++ program = Flex(_offstack_types.ExtractContact, interpreter_factory=_sandbox)+ program._bind_code(source)+ dspy.configure(lm=DummyLM([{"contact": json.dumps({"name": "Ada", "age": 36})}] * 2))+ out = program(text="Ada is 36") # raised `Unknown name: Contact` before the handoff+ assert out.contact == _offstack_types.Contact(name="Ada", age=36)++ # And the host-built sub-predictor resolved `Contact` to the module's class, not a lookalike:+ inv = bridge._Invocation(program._bridge, {})+ inv.construct("Predict", "text: str -> contact: Contact", "p", {})+ assert inv._predictors["p"].signature.output_fields["contact"].annotation is _offstack_types.Contact+++@deno_required+def test_sub_signature_naming_a_custom_type_resolves_on_the_host() -> None:+ # The sandbox sends a signature as text; the host resolves `Person` from the Flex's own+ # signature, since dspy's usual caller-frame lookup runs inside dspy and cannot see it.+ source = textwrap.dedent(+ """+ class M(dspy.Module):+ def __init__(self):+ super().__init__()+ self.p = dspy.Predict("text: str -> person: Person")++ def forward(self, **inputs):+ return dspy.Prediction(person=self.p(text=inputs["text"]).person)+ """+ ).strip()+ program = Flex(Extract, interpreter_factory=_sandbox)+ program._bind_code(source)+ dspy.configure(lm=DummyLM([{"person": json.dumps({"name": "Ada", "age": 36})}] * 2))+ program(text="Ada is 36") # would raise `Unknown name: Person` if resolution failed+ inv = bridge._Invocation(program._bridge, {})+ inv.construct("Predict", "text: str -> person: Person", "p", {})+ assert inv._predictors["p"].signature.output_fields["person"].annotation is Person+++# =============================================================================+# The return path parses against the declared types (Deno)+# =============================================================================+++@deno_required+def test_pydantic_output_comes_back_as_the_model_like_predict_does() -> None:+ payload = json.dumps({"name": "Ada", "age": 36})+ dspy.configure(lm=DummyLM([{"person": payload}] * 2))++ baseline = dspy.Predict(Extract)(text="Ada is 36")+ program = Flex(Extract, interpreter_factory=_sandbox)+ flexed = program(text="Ada is 36")++ # The point of the parse: a Flex is substitutable for a Predict over the same signature.+ assert isinstance(baseline.person, Person)+ assert isinstance(flexed.person, Person)+ assert flexed.person == baseline.person == Person(name="Ada", age=36)+++@deno_required+def test_generic_enum_and_scalar_outputs_are_all_parsed() -> None:+ source = textwrap.dedent(+ """+ class M(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ # JSON is all that crosses the boundary: dicts for models, a bare string for the+ # enum, and the count arrives as a string unless it is parsed back.+ return dspy.Prediction(+ people=[{"name": "Ada", "age": 36}, {"name": "Bob", "age": 40}],+ color="blue",+ count="2",+ )+ """+ ).strip()+ program = Flex(Roster, interpreter_factory=_sandbox)+ program._bind_code(source)+ out = program(text="two people")++ assert out.people == [Person(name="Ada", age=36), Person(name="Bob", age=40)]+ assert out.color is Color.BLUE+ assert out.count == 2 and isinstance(out.count, int)+++@deno_required+def test_payload_that_does_not_match_the_declared_type_raises() -> None:+ source = textwrap.dedent(+ """+ class M(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ return dspy.Prediction(person="Ada, aged thirty-six")+ """+ ).strip()+ program = Flex(Extract, interpreter_factory=_sandbox)+ program._bind_code(source)+ with pytest.raises(CodeInterpreterError, match="person"):+ program(text="Ada is 36")+++@deno_required+def test_missing_declared_output_field_raises() -> None:+ source = textwrap.dedent(+ """+ class M(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ return dspy.Prediction(summary="Ada, 36") # not the declared output field+ """+ ).strip()+ program = Flex(Extract, interpreter_factory=_sandbox)+ program._bind_code(source)+ with pytest.raises(CodeInterpreterError, match=r"missing required output field\(s\) \['person'\]"):+ program(text="Ada is 36")+++# =============================================================================+# How GEPA catches it+# =============================================================================+++@deno_required+def test_gepa_scores_a_type_mismatch_as_a_failure_in_place() -> None:+ """A candidate that returns the wrong type for a declared field is a bad candidate, not a+ crashed run: the example it fails on takes the failure score in its own slot, so the search+ keeps going and simply doesn't select it."""++ def exact_match(gold, pred, trace=None, pred_name=None, pred_trace=None):+ return 1.0 if getattr(pred, "person", None) == gold.person else 0.0++ source = textwrap.dedent(+ """+ class M(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ if inputs["text"] == "bad":+ return dspy.Prediction(person="not a person")+ return dspy.Prediction(person={"name": inputs["text"], "age": 1})+ """+ ).strip()++ student = Flex(Extract, interpreter_factory=_sandbox)+ adapter = DspyAdapter(student_module=student, metric_fn=exact_match, feedback_map={})+ batch = [+ dspy.Example(text="Ada", person=Person(name="Ada", age=1)).with_inputs("text"),+ dspy.Example(text="bad", person=Person(name="bad", age=1)).with_inputs("text"),+ dspy.Example(text="Bob", person=Person(name="Bob", age=1)).with_inputs("text"),+ ]++ result = adapter.evaluate(batch, {"self": source}, capture_traces=False)++ assert result.scores == [1.0, 0.0, 1.0] # the type error is scored, not raised+ assert isinstance(result.outputs[1], FailedPrediction)+ assert "not a valid" in result.outputs[1].completion_text+ assert result.outputs[0].person == Person(name="Ada", age=1) # and it is a real model, parsed+++@deno_required+def test_custom_type_inputs_are_restored_across_the_bridge(caplog) -> None:+ """A custom-type input (dspy.Image) crosses into the sandbox as its serialized string; the+ bridge hands the original object back to the host predictor, so Predict sees a real Image —+ no spurious "Type mismatch" warning, and adapters get the object rather than a tagged string.+ """+ class Caption(dspy.Signature):+ image: dspy.Image = dspy.InputField()+ a: str = dspy.OutputField()++ dspy.configure(lm=DummyLM([{"a": "a cat"}]))+ flex = Flex(Caption, interpreter_factory=lambda: dspy.PythonInterpreter())+ with caplog.at_level(logging.WARNING, logger="dspy.predict.predict"):+ out = flex(image=dspy.Image(url="https://example.com/x.png"))++ assert out.a == "a cat"+ mismatch_warnings = [r.getMessage() for r in caplog.records if "Type mismatch" in r.getMessage()]+ assert mismatch_warnings == []+++# =============================================================================+# Optional / defaulted output fields: presence is required, None only where admitted+# =============================================================================+++def test_optional_output_field_accepts_explicit_none() -> None:+ """Optionality applies to the value, not the key: an explicit null stays None."""+ class MaybeAnswer(dspy.Signature):+ q: str = dspy.InputField()+ a: str | None = dspy.OutputField()++ program = Flex(MaybeAnswer, interpreter_factory=MockInterpreter)+ assert program._bridge._to_prediction({"a": None}).a is None+++def test_optional_output_field_may_be_omitted() -> None:+ """The Flex exit is a module boundary, not an LM response: the signature's own declarations+ apply, so an Optional field the generated code omits comes back as None."""+ class MaybeAnswer(dspy.Signature):+ q: str = dspy.InputField()+ a: str | None = dspy.OutputField()+ b: str = dspy.OutputField()++ program = Flex(MaybeAnswer, interpreter_factory=MockInterpreter)+ pred = program._bridge._to_prediction({"b": "x"})+ assert pred.a is None+ assert pred.b == "x"+++def test_output_field_default_fills_a_missing_field() -> None:+ """A field declared with a default falls back to it when the generated code omits it."""+ class Defaulted(dspy.Signature):+ q: str = dspy.InputField()+ a: str = dspy.OutputField(default="fallback")++ program = Flex(Defaulted, interpreter_factory=MockInterpreter)+ assert program._bridge._to_prediction({}).a == "fallback"+++def test_complex_output_field_defaults_fill_host_side_per_forward() -> None:+ """Defaults fill on the host, after the JSON crossing: any object works, regardless of the+ sandbox's boundary, and each forward gets a fresh copy (pydantic deep-copies plain defaults;+ factories are called per fill) — mutating one prediction cannot poison later forwards."""+ class Complex(dspy.Signature):+ q: str = dspy.InputField()+ person: Person = dspy.OutputField(default=Person(name="unknown", age=0))+ tags: list[str] = dspy.OutputField(default_factory=lambda: ["fresh"])++ program = Flex(Complex, interpreter_factory=MockInterpreter)+ first = program._bridge._to_prediction({})+ assert first.person == Person(name="unknown", age=0)+ first.person.name = "MUTATED"+ first.tags.append("MUTATED")++ second = program._bridge._to_prediction({})+ assert second.person == Person(name="unknown", age=0) # fresh copy, unaffected+ assert second.tags == ["fresh"] # factory re-invoked per fill+++def test_missing_required_output_field_still_raises() -> None:+ """No default and not Optional means required: omitting it is the candidate's error."""+ class Strict(dspy.Signature):+ q: str = dspy.InputField()+ a: str = dspy.OutputField()++ program = Flex(Strict, interpreter_factory=MockInterpreter)+ with pytest.raises(CodeInterpreterError, match=r"missing required output field\(s\) \['a'\]"):+ program._bridge._to_prediction({})+++def test_none_for_a_non_optional_output_field_is_rejected() -> None:+ """A JSON null for `a: str` must not be stringified into "None" — a value the generated+ program never produced (natively the caller would see None, and an adapter can only ever+ yield text). The bridge surfaces the type mismatch instead."""+ class Strict(dspy.Signature):+ q: str = dspy.InputField()+ a: str = dspy.OutputField()++ program = Flex(Strict, interpreter_factory=MockInterpreter)+ with pytest.raises(CodeInterpreterError, match=r"returned None for output field 'a'"):+ program._bridge._to_prediction({"a": None})+++@deno_required+def test_optional_none_round_trips_through_the_sandbox() -> None:+ """End to end: a generated forward returning an explicit None crosses as JSON null and the+ caller receives a real None, not a "None" string."""+ class MaybeAnswer(dspy.Signature):+ q: str = dspy.InputField()+ a: str | None = dspy.OutputField()++ source = textwrap.dedent(+ """+ class MaybeModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ return dspy.Prediction(a=None)+ """+ ).strip()++ program = Flex(MaybeAnswer, interpreter_factory=_sandbox)+ program._bind_code(source)+ assert program(q="x").a is None+++# =============================================================================+# render_annotation: the signature parser's inverse (no Deno)+# =============================================================================+++def test_render_annotation_round_trips_through_the_signature_parser() -> None:+ """``render_annotation`` mirrors dspy's ``_parse_type_node``: for every annotation it+ accepts, the emitted token parses back to the identical annotation under the same+ ``custom_types``. This round trip is the renderer's correctness contract."""+ from typing import Any, Literal, Optional++ from dspy.predict.flex.ctx import render_annotation+ from dspy.signatures.signature import make_signature++ custom = {"Person": Person, "Color": Color}+ cases = [+ str,+ int,+ bool,+ Person,+ Color,+ list[Person],+ dict[str, int],+ dict[str, Any],+ tuple[int, ...],+ str | None,+ Optional[float], # noqa: UP045 — covers the typing.Union spelling, distinct origin from `float | None`+ str | int | None,+ Literal["a", "b"],+ Literal[1, 2],+ list[dict[str, list[int]]],+ Any,+ ]+ for annotation in cases:+ token = render_annotation(annotation, custom)+ probe = make_signature(f"x: {token} -> y", custom_types=custom)+ assert probe.input_fields["x"].annotation == annotation, f"{annotation!r} -> {token!r}"+++def test_render_annotation_rejects_what_the_grammar_cannot_carry() -> None:+ from dspy.predict.flex.ctx import render_annotation++ with pytest.raises(ValueError):+ render_annotation(Callable[[int], int])++ class NotShared(pydantic.BaseModel):+ x: int++ # A class the parser could not resolve by name (absent from custom_types) is refused at+ # render time instead of producing a token that fails to parse back.+ with pytest.raises(ValueError):+ render_annotation(NotShared)diff --git a/tests/flex/test_tools.py b/tests/flex/test_tools.pynew file mode 100644index 000000000..33b0da6a9--- /dev/null+++ b/tests/flex/test_tools.py@@ -0,0 +1,303 @@+"""dspy.Flex accepts tools, hands them to the underlying RLM baseline, exposes them to the+GEPA code proposer, and round-trips them through save/load.++Tools are callables/dspy.Tool objects — they are NOT serialized. Like the signature, they are+re-provided at construction; the saved ``module_src`` references them by name and resolves+against the reconstructed Flex's context at bind time.++Flex runs its code through an interpreter (no in-process mode), so tests that only inspect+``module_src`` / context use a MockInterpreter (Deno-free), while tests that bind-and-run+generated code are marked ``deno_required``.+"""++from __future__ import annotations++import re+import shutil+import textwrap++import pytest++import dspy+from dspy.predict.flex import Flex+from dspy.predict.flex.bridge import SHIM_SETUP+from dspy.predict.flex.primitives_doc import PRIMITIVES_CATALOG+from dspy.primitives.code_interpreter import CodeInterpreterError+from dspy.teleprompt.gepa.gepa_flex_utils import CodeProposalSignature+from dspy.teleprompt.gepa.gepa_utils import DspyAdapter+from dspy.utils.dummies import DummyLM+from tests.mock_interpreter import MockInterpreter++deno_required = pytest.mark.skipif(shutil.which("deno") is None, reason="Deno is not installed")+++def shout(text: str) -> str:+ """Return the text in uppercase."""+ return text.upper()+++class Echo(dspy.Signature):+ """Echo the question as the answer."""++ q: str = dspy.InputField()+ a: str = dspy.OutputField()+++# A tool-using module with NO predictors: forward calls the in-scope `shout` tool directly,+# so it runs LM-free (stands in for code GEPA might produce that wires a tool in).+TOOL_MODULE = textwrap.dedent("""+ class EchoModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, q):+ return dspy.Prediction(a=shout(q))+""").strip()+++def test_baseline_is_rlm_with_tools() -> None:+ # With tools, the baseline delegates to dspy.RLM (which can call them via its REPL), not Predict,+ # and the tools are wired into its constructor by name.+ src = Flex(Echo, tools=[shout], interpreter_factory=lambda: MockInterpreter()).module_src+ assert "dspy.RLM(" in src+ assert "tools=[shout]" in src+ assert "dspy.Predict(" not in src+++def test_reserved_bridge_tool_names_are_rejected() -> None:+ # The sandbox bridge registers its own callbacks (__dspy_construct__, __dspy_call__) alongside+ # user tools, and the shim owns `dspy`/`_dspy*` in the sandbox namespace; a user tool with a+ # reserved name would silently replace or be shadowed by them.+ for reserved in ("__dspy_call__", "_dspy_host", "dspy"):+ with pytest.raises(ValueError, match="reserved"):+ Flex(Echo, tools=[dspy.Tool(shout, name=reserved)], interpreter_factory=lambda: MockInterpreter())+++def test_baseline_is_predict_without_tools() -> None:+ # Without tools, the baseline is a single dspy.Predict (no RLM, no tools arg).+ src = Flex(Echo, interpreter_factory=lambda: MockInterpreter()).module_src+ assert "dspy.Predict(" in src+ assert "dspy.RLM(" not in src+ assert "tools=[" not in src+++@deno_required+def test_tool_is_in_scope_for_bound_code() -> None:+ program = Flex(Echo, tools=[shout], interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(TOOL_MODULE)+ # LM-free: forward only calls the tool (bridged from the sandbox to the host).+ assert program(q="hello").a == "HELLO"+++@deno_required+def test_save_load_roundtrips_tool_using_code(tmp_path) -> None:+ path = tmp_path / "program.json"+ program = Flex(Echo, tools=[shout], interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(TOOL_MODULE)+ program.save(path)++ # Reconstruct with the SAME tools (the architecture is re-provided in code, like the+ # signature), then load the saved code — the tool reference resolves and runs.+ reloaded = Flex(Echo, tools=[shout], interpreter_factory=lambda: dspy.PythonInterpreter())+ reloaded.load(path)+ assert "shout(q)" in reloaded.module_src+ assert reloaded(q="hi").a == "HI"+++@deno_required+def test_load_without_tools_cannot_resolve_them(tmp_path) -> None:+ # The baseline RLM is built with tools=[shout] in __init__; reconstructing without the+ # tool can't resolve the name in the sandbox. Loading succeeds (binding is a host-side+ # syntax check) — the failure surfaces when the code first runs (tools must be re-provided).+ path = tmp_path / "program.json"+ program = Flex(Echo, tools=[shout], interpreter_factory=lambda: dspy.PythonInterpreter())+ program.save(path)++ reloaded = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter()) # tools NOT re-provided+ reloaded.load(path)+ with pytest.raises(CodeInterpreterError):+ reloaded(q="hi")+++# The primitives catalog offers the proposer `dspy.Tool(func)`, so the sandbox shim has to define+# it — a name the catalog advertises but the shim lacks turns into an AttributeError at bind time,+# scoring an otherwise-fine candidate as a failure.+WRAPPED_TOOL_MODULE = textwrap.dedent("""+ class EchoModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.react = dspy.ReAct("q -> a", tools=[dspy.Tool(shout)])++ def forward(self, **inputs):+ return dspy.Prediction(a=shout(inputs["q"]))+""").strip()+++@deno_required+def test_dspy_tool_wrapper_is_available_in_the_sandbox() -> None:+ program = Flex(Echo, tools=[shout], interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(WRAPPED_TOOL_MODULE)+ # forward re-runs __init__ in the sandbox, so it only succeeds if the shim defines dspy.Tool+ # and the wrapper resolves to the same host tool a bare reference would (a failed resolution+ # raises out of the ReAct's construction).+ assert program(q="hi").a == "HI"+++def test_proposer_prompts_only_advertise_names_the_sandbox_defines() -> None:+ """Every `dspy.X(...)` the proposer is told to write must exist in the shim.++ The prompts and the shim are the two halves of one contract, edited in different files: a name+ advertised but not defined becomes an AttributeError at bind time, and GEPA scores the+ candidate as a failure with nothing pointing at the real cause.+ """+ # The shim only registers itself as the importable `dspy` inside the sandbox (where the host+ # bridge tools are in globals), so running it here just builds the module object to inspect.+ namespace: dict = {}+ exec(SHIM_SETUP, namespace)+ shim = namespace["dspy"]++ prompts = PRIMITIVES_CATALOG + (CodeProposalSignature.__doc__ or "")+ advertised = set(re.findall(r"dspy\.([A-Z]\w+)\(", prompts))+ assert advertised, "expected the prompts to name some primitives"+ missing = sorted(name for name in advertised if not hasattr(shim, name))+ assert not missing, f"advertised to the proposer but missing from the sandbox shim: {missing}"+++def test_gepa_proposer_is_told_about_tools() -> None:+ """GEPA can optimize tool usage because the proposer is shown the available tools."""++ def _metric(gold, pred, trace=None, pred_name=None, pred_trace=None):+ return 1.0++ student = Flex(Echo, tools=[shout], interpreter_factory=lambda: MockInterpreter())+ adapter = DspyAdapter(student_module=student, metric_fn=_metric, feedback_map={})+ assert "shout" in adapter._flex_context_blurbs["self"]+++# A module that AUTHORS its own helper (defined inside forward) and calls it directly — what GEPA is+# encouraged to do in a sandbox, where an authored function cannot be handed to a bridged+# sub-predictor. The helper lives in the source, so it persists with the code itself.+SELF_AUTHORED_TOOL_MODULE = textwrap.dedent("""+ class EchoModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ def upcase(text: str) -> str:+ "Uppercase the text."+ return text.upper()+ return dspy.Prediction(a=upcase(inputs["q"]))+""").strip()+++@deno_required+def test_self_authored_tool_persists_with_the_code(tmp_path) -> None:+ # Helpers GEPA writes itself live in module_src, so they persist exactly like the code (and like+ # instructions): reconstructing with ONLY the signature — no tools re-provided — rebuilds the+ # helper from the saved source and runs it.+ path = tmp_path / "program.json"+ flex = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter()) # no user-provided tools+ flex._bind_code(SELF_AUTHORED_TOOL_MODULE)+ assert "def upcase" in flex.module_src+ flex.save(path)++ reloaded = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter()) # still no tools+ reloaded.load(path)+ assert reloaded.module_src == SELF_AUTHORED_TOOL_MODULE+ assert reloaded(q="hi").a == "HI"+++# Instructions baked into the generated code, on an inner predictor.+INSTR_MODULE = textwrap.dedent("""+ class EchoModule(dspy.Module):+ def __init__(self):+ super().__init__()+ self.p = dspy.Predict(dspy.Signature("q -> a", "Always answer in uppercase."))++ def forward(self, q):+ return dspy.Prediction(a=self.p(q=q).a)+""").strip()+++@deno_required+def test_inner_predictor_instructions_persist(tmp_path) -> None:+ # The instructions GEPA writes into the code persist through save/load via module_src alone:+ # each forward rebuilds the predictor from the source, instructions included — observable in+ # the prompt the LM actually receives.+ path = tmp_path / "program.json"+ lm = DummyLM([{"a": "HI"}] * 4)+ dspy.configure(lm=lm)+ flex = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ flex._bind_code(INSTR_MODULE)+ flex(q="hi")+ assert "Always answer in uppercase." in str(lm.history[-1]["messages"])+ flex.save(path)++ reloaded = Flex(Echo, interpreter_factory=lambda: dspy.PythonInterpreter())+ reloaded.load(path)+ reloaded(q="hi")+ assert "Always answer in uppercase." in str(lm.history[-1]["messages"])+++def double(n: int) -> str:+ """Double the number."""+ return str(n * 2)+++@deno_required+def test_direct_tool_calls_go_through_the_tool_wrapper() -> None:+ # Tools keep their dspy.Tool wrapper when the sandbox calls them directly, so DSPy's argument+ # validation applies: n=3 works, n="3" is rejected loudly — the raw underlying function would+ # have silently returned "33".+ def src(arg: str) -> str:+ return textwrap.dedent(+ f"""+ class EchoModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, q):+ return dspy.Prediction(a=double(n={arg}))+ """+ ).strip()++ program = Flex(Echo, tools=[dspy.Tool(double)], interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(src("3"))+ assert program(q="x").a == "6"+ program._bind_code(src('"3"'))+ with pytest.raises(CodeInterpreterError, match="not of type 'integer'"):+ program(q="x")+++@deno_required+def test_direct_tool_calls_receive_restored_custom_type_inputs() -> None:+ """A custom-type input (dspy.Image) crosses into the sandbox as its serialized string; a+ direct tool call must hand the tool the original object, exactly as a bridged predictor+ receives it via ``_Invocation.call``."""+ seen: dict = {}++ def describe(img) -> str:+ """Describe an image."""+ seen["img"] = img+ return "a cat"++ class Caption(dspy.Signature):+ image: dspy.Image = dspy.InputField()+ a: str = dspy.OutputField()++ src = textwrap.dedent(+ """+ class CaptionModule(dspy.Module):+ def __init__(self):+ super().__init__()++ def forward(self, **inputs):+ return dspy.Prediction(a=describe(img=inputs["image"]))+ """+ ).strip()++ program = Flex(Caption, tools=[describe], interpreter_factory=lambda: dspy.PythonInterpreter())+ program._bind_code(src)+ image = dspy.Image(url="https://example.com/x.png")+ assert program(image=image).a == "a cat"+ assert seen["img"] is image # the original object, not the serialized tag stringdiff --git a/tests/primitives/test_python_interpreter.py b/tests/primitives/test_python_interpreter.pyindex 90a3a53fe..0bedbbfd6 100644--- a/tests/primitives/test_python_interpreter.py+++ b/tests/primitives/test_python_interpreter.py@@ -3,14 +3,17 @@ import dataclassesimport jsonimport osimport random+import threading+from collections import namedtuplefrom datetime import datetime+from enum import Enumfrom typing import NamedTupleimport pytestfrom pydantic import BaseModel, ConfigDictfrom dspy.primitives.code_interpreter import CodeExecutionError, CodeInterpreterError, FinalOutput-from dspy.primitives.python_interpreter import PythonInterpreter+from dspy.primitives.python_interpreter import LARGE_VAR_THRESHOLD, PythonInterpreter, _make_jsonableclass _Hit(BaseModel):@@ -340,6 +343,35 @@ def test_pydantic_json_values_are_compatible_with_large_variable_injection(monkeassert json.loads(interpreter._pending_large_vars["hit"]) == expected+def test_json_mode_values_cross_every_host_to_sandbox_path():+ """Bare datetimes/enums/sets (not wrapped in a model) serialize the way they would into a+ JSON body on all three host->sandbox paths: variable injection, large-var injection, and+ tool results — instead of raising "Unsupported value type" or str()-flattening the result."""++ class Color(Enum):+ RED = "red"++ value = {"t": datetime(2026, 1, 1), "c": Color.RED}+ interpreter = PythonInterpreter()+ assert interpreter._serialize_value(value) == "{'t': '2026-01-01T00:00:00', 'c': 'red'}"+ assert interpreter._to_json_compatible(value) == {"t": "2026-01-01T00:00:00", "c": "red"}+ jsonable = _make_jsonable({**value, "s": {3, 1}})+ assert jsonable["t"] == "2026-01-01T00:00:00" and jsonable["c"] == "red"+ assert sorted(jsonable["s"]) == [1, 3]+++def test_json_mode_coercion_preserves_existing_fallbacks():+ class P(NamedTuple):+ x: int+ y: int++ assert _make_jsonable(P(1, 2)) == {"x": 1, "y": 2} # namedtuples keep field names, not [1, 2]+ obj = object()+ assert _make_jsonable(obj) is obj # unknown values still take the json.dumps/str fallback+ with pytest.raises(CodeInterpreterError, match="Unsupported value type"):+ PythonInterpreter()._serialize_value(object()) # injection still rejects them loudly++def test_unserializable_pydantic_variable_raises_code_interpreter_error():"""Invalid host variables should fail before the interpreter process starts."""interpreter = PythonInterpreter()@@ -809,8 +841,6 @@ class _TypedPoint(NamedTuple):label: str-from collections import namedtuple-_Point = namedtuple("_Point", ["x", "y"])@@ -838,7 +868,6 @@ def test_tool_returns_collections_namedtuple(configure_pooled_interpreter):def test_tool_returns_dataclass_with_unserializable_field_falls_back(configure_pooled_interpreter):"""Dataclass with non-serializable fields should fall back to str() gracefully."""- import threading@dataclasses.dataclassclass _Holder:@@ -960,8 +989,6 @@ def test_extract_parameters_complex_types():def test_large_variable_injection(pooled_interpreter):"""Test that large strings are injected via filesystem to avoid Pyodide's FFI size limit."""- from dspy.primitives.python_interpreter import LARGE_VAR_THRESHOLD-# Create a string just over the thresholdlarge_data = "x" * (LARGE_VAR_THRESHOLD + 1024)@@ -972,8 +999,6 @@ def test_large_variable_injection(pooled_interpreter):def test_large_variable_content_integrity(pooled_interpreter):"""Test that large variable content is preserved exactly through filesystem injection."""- from dspy.primitives.python_interpreter import LARGE_VAR_THRESHOLD-# Create a string with recognizable pattern just over thresholdpattern = "ABCDEFGHIJ" * 100large_data = pattern * ((LARGE_VAR_THRESHOLD // len(pattern)) + 1)@@ -992,8 +1017,6 @@ last_100 = data[-100:]def test_mixed_small_and_large_variables(pooled_interpreter):"""Test that small and large variables can be used together."""- from dspy.primitives.python_interpreter import LARGE_VAR_THRESHOLD-small_var = "hello"large_var = "x" * (LARGE_VAR_THRESHOLD + 1024)@@ -1006,8 +1029,6 @@ def test_mixed_small_and_large_variables(pooled_interpreter):def test_multiple_large_variables(pooled_interpreter):"""Test that multiple large variables can be injected."""- from dspy.primitives.python_interpreter import LARGE_VAR_THRESHOLD-large_a = "a" * (LARGE_VAR_THRESHOLD + 100)large_b = "b" * (LARGE_VAR_THRESHOLD + 200)@@ -1019,8 +1040,6 @@ def test_multiple_large_variables(pooled_interpreter):def test_large_list_variable(pooled_interpreter):"""Test that large list variables are injected via filesystem and JSON parsed."""- from dspy.primitives.python_interpreter import LARGE_VAR_THRESHOLD-# Each element "x" serializes to ~3 chars, so divide threshold by 3num_elements = LARGE_VAR_THRESHOLD // 3large_list = ["x"] * num_elements@@ -1060,8 +1079,6 @@ def test_large_variable_threshold_boundary():The threshold applies to the serialized size, not the original value.For strings, serialization adds 2 bytes (quotes)."""- from dspy.primitives.python_interpreter import LARGE_VAR_THRESHOLD-# Serialized size at threshold - should use embedded (not filesystem)# Account for 2 bytes of quotes added by repr()at_threshold = "x" * (LARGE_VAR_THRESHOLD - 2)diff --git a/tests/signatures/test_custom_types.py b/tests/signatures/test_custom_types.pyindex 5393dc24d..9b02fd8e7 100644--- a/tests/signatures/test_custom_types.py+++ b/tests/signatures/test_custom_types.py@@ -117,3 +117,30 @@ def test_basic_custom_type_resolution():sig = Signature("input: CustomType -> output: str")assert sig.input_fields["input"].annotation == CustomType+++def test_explicit_custom_types_do_not_depend_on_the_caller_frame():+ """`custom_types` is documented as the reliable alternative to frame introspection, so it has+ to resolve a type whose *name* is nowhere on the call stack.++ `make_signature` re-parses the signature string a second time to build the default+ instructions, and used to drop `custom_types` on that pass, falling back to the caller-frame+ lookup. The same call then succeeded or raised `Unknown name` depending only on how the caller+ happened to import the type — `from myapp.models import Item` worked, `import myapp.models as+ models` did not.+ """+ # Named only as a string and a dict key: no frame anywhere binds `OffStackItem`.+ item = pydantic.create_model("OffStackItem", value=(str, ...))+ types = {"OffStackItem": item}++ sig = Signature("query: str -> found: OffStackItem", custom_types=types)+ assert sig.output_fields["found"].annotation is item+ assert sig.instructions == "Given the fields `query`, produce the fields `found`."++ nested = Signature("query: str -> found: list[OffStackItem]", custom_types=types)+ assert nested.output_fields["found"].annotation == list[item]++ # Supplying instructions skipped the re-parse entirely; both paths must agree.+ with_instructions = Signature("query: str -> found: OffStackItem", "Find it.", custom_types=types)+ assert with_instructions.output_fields["found"].annotation is item+ assert with_instructions.instructions == "Find it."diff --git a/tests/teleprompt/test_bootstrap_trace.py b/tests/teleprompt/test_bootstrap_trace.pyindex 062bd3cc8..89ffd64a1 100644--- a/tests/teleprompt/test_bootstrap_trace.py+++ b/tests/teleprompt/test_bootstrap_trace.py@@ -151,3 +151,30 @@ def test_bootstrap_trace_data_passes_callback_metadata(monkeypatch):)assert captured_metadata["value"] == {"disable_logging": True}+++def test_capture_crashes_does_not_capture_lm_errors():+ """``capture_crashes`` converts program bugs into FailedPrediction, but an LMError is+ infrastructure: repainting it as a code failure would hand the optimizer an invalid+ evaluation, so it must propagate to the evaluator's error handling instead."""+ from dspy.utils.exceptions import LMRateLimitError++ example = Example(q="x").with_inputs("q")++ class Bug(dspy.Module):+ def forward(self, **kwargs):+ raise RuntimeError("a real code failure")++ data = bootstrap_trace_data(Bug(), dataset=[example], num_threads=1, capture_crashes=True)+ assert isinstance(data[0]["prediction"], FailedPrediction)+ assert "RuntimeError" in data[0]["prediction"].completion_text++ class Flaky(dspy.Module):+ def forward(self, **kwargs):+ raise LMRateLimitError("429 from the provider")++ # raise_on_error=False mirrors the flex GEPA call site (gepa_flex_utils).+ data = bootstrap_trace_data(+ Flaky(), dataset=[example], num_threads=1, capture_crashes=True, raise_on_error=False+ )+ assert data == [] # handled by the evaluator as an error, never repainted as a FailedPrediction