Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Make BetterTogether compatible with all optimizers (#9149)

Dilara Soylubb110a0262f2
diff --git a/docs/docs/api/optimizers/BetterTogether.md b/docs/docs/api/optimizers/BetterTogether.md
index 33029b476..fc8712279 100644
--- a/docs/docs/api/optimizers/BetterTogether.md
+++ b/docs/docs/api/optimizers/BetterTogether.md
@@ -1,5 +1,7 @@
# dspy.BetterTogether
+**BetterTogether** is a meta-optimizer proposed in the paper [Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together](https://arxiv.org/abs/2407.10930) by Dilara Soylu, Christopher Potts, and Omar Khattab. It combines prompt optimization and weight optimization (fine-tuning) by applying them in a configurable sequence, allowing a student program to iteratively improve both its prompts and model parameters. The core insight is that prompt and weight optimization can complement each other: prompt optimization can potentially discover effective task decompositions and reasoning strategies, while weight optimization can specialize the model to execute these patterns more efficiently. Using these approaches together in sequences (e.g., prompt optimization then weight optimization) may allow each to build on the improvements made by the other.
+
<!-- START_API_REF -->
::: dspy.BetterTogether
handler: python
@@ -17,3 +19,139 @@
inherited_members: true
:::
<!-- END_API_REF -->
+
+## Usage Examples
+
+See BetterTogether usage tutorials in [BetterTogether Tutorials](../../../tutorials/bettertogether/index.md).
+
+## How BetterTogether Works
+
+BetterTogether executes optimizers in a configurable sequence, evaluating each intermediate result and returning the best performing program. Here's how it works:
+
+### 1. **Initialization with Custom Optimizers**
+
+When initialized, BetterTogether accepts any DSPy optimizers (Teleprompters) as keyword arguments. The keys become the optimizer names used in the strategy string:
+
+```python
+optimizer = BetterTogether(
+ metric=metric,
+ p=GEPA(...), # 'p' can be used in strategy
+ w=BootstrapFinetune(...) # 'w' can be used in strategy
+)
+```
+
+If no optimizers are provided, BetterTogether defaults to `BootstrapFewShotWithRandomSearch` (key: 'p') and `BootstrapFinetune` (key: 'w').
+
+### 2. **Strategy Execution**
+
+The strategy string defines the sequence of optimizers to apply. For example:
+
+```python
+compiled = optimizer.compile(
+ student,
+ trainset=trainset,
+ valset=valset,
+ strategy="p -> w -> p"
+)
+```
+
+This strategy `"p -> w -> p"` means:
+
+1. Run prompt optimizer ('p')
+2. Run weight optimizer ('w') on the result
+3. Run prompt optimizer ('p') again on the result
+
+At each step:
+- The trainset is shuffled (if `shuffle_trainset_between_steps=True`)
+- The optimizer is run on the current student program
+- The result is evaluated on the validation set
+- The candidate program and score are recorded
+
+Since BetterTogether is a meta-optimizer, any sequence of optimizers can be combined. The optimizer names in the strategy string correspond to the keyword arguments from initialization. For example, you can sequence different prompt optimizers (note: this illustrates BetterTogether's flexibility, not necessarily a recommended configuration):
+
+```python
+optimizer = BetterTogether(
+ metric=metric,
+ mipro=MIPROv2(metric=metric, auto="light"),
+ gepa=GEPA(metric=metric, auto="light")
+)
+
+compiled = optimizer.compile(
+ student,
+ trainset=trainset,
+ valset=valset,
+ strategy="mipro -> gepa -> mipro"
+)
+```
+
+### 3. **Validation and Program Selection**
+
+BetterTogether can use a validation set in three ways:
+
+- **Explicit valset**: If `valset` is provided, it's used for evaluation
+- **Auto-split**: If `valset_ratio > 0`, a portion of trainset is held out for validation
+- **No validation**: If both `valset` and `valset_ratio` are None/0, no validation occurs
+
+After all optimization steps complete, the best program is selected based on validation set availability:
+
+- **With validation**: The program with the **best score** is returned (with earlier programs winning ties)
+- **Without validation**: The **latest program** is returned
+
+If an optimization step fails:
+- The error is logged with full traceback
+- Optimization stops early
+- The best program found so far is returned
+- `flag_compilation_error_occurred` is set to `True`
+
+The returned program includes two additional attributes:
+
+- `candidate_programs`: List of all evaluated programs with their scores and strategies, sorted by score (best first). When an error occurs, this contains all successfully evaluated programs up to the point of failure.
+- `flag_compilation_error_occurred`: Boolean indicating if any optimization step failed during compilation.
+
+### 4. **Further Details**
+
+**Model Lifecycle Management**: For local models (like LocalLM), BetterTogether automatically launches models before first use, kills them after optimization completes, and relaunches them after fine-tuning when model names change. These operations are no-ops for API-based LMs but needed for local model serving.
+
+**Custom Compile Arguments**: You can pass custom compile arguments to specific optimizers using the `optimizer_compile_args` parameter:
+
+- **Override default arguments**: Pass custom trainset/valset/teacher to specific optimizers
+- **Customize per optimizer**: Each optimizer can have different compile arguments (e.g., `num_trials`, `max_bootstrapped_demos`)
+
+Note: The `student` argument cannot be included in `optimizer_compile_args` - BetterTogether manages the student program for all optimizers. See the `compile()` method docstring for detailed argument documentation.
+
+## Best Practices
+
+### When to Use BetterTogether
+
+BetterTogether is the right optimizer when:
+
+- **You want to squeeze every bit of performance**: Prompt optimization is often the best bang for buck, quickly discovering high-level strategies. When opportunity allows, adding weight optimization on top compounds these gains, yielding benefits that exceed either approach alone.
+- **You have fine-tuning capabilities**: Weight optimizers like `BootstrapFinetune` require LMs with a fine-tuning interface. Currently supported: `LocalProvider`, `DatabricksProvider`, and `OpenAIProvider`. You can extend the `Provider` class for custom use cases, or use BetterTogether to combine prompt optimizers only.
+
+The [Databricks case study](https://www.databricks.com/blog/building-state-art-enterprise-agents-90x-cheaper-automated-prompt-optimization) demonstrates this effectiveness. They evaluated on IE Bench—a comprehensive suite spanning enterprise domains (finance, legal, commerce, healthcare) with complex challenges: 100+ page documents, 70+ extraction fields, and hierarchical schemas. Using GPT-4.1:
+
+- **SFT alone**: +1.9 points over baseline
+- **GEPA alone**: +2.1 points over baseline (slightly exceeding SFT)
+- **GEPA + SFT (BetterTogether)**: +4.8 points over baseline
+
+This demonstrates that prompt optimization can match or surpass supervised fine-tuning, and combining these techniques yields strong compounding benefits.
+
+### Common Strategies and Optimizers
+
+Common strategies:
+
+- `"p -> w"`: Optimize prompts first, then fine-tune (simple and often effective)
+- `"p -> w -> p"`: Optimize prompts, fine-tune, then optimize prompts again (can build on fine-tuning improvements)
+- `"w -> p"`: Fine-tune first, then optimize prompts
+
+Example optimizer combinations:
+
+- **GEPA + BootstrapFinetune**: Prompt optimization with fine-tuning
+- **MIPROv2 + BootstrapFinetune**: Prompt optimization with fine-tuning
+- **Multiple prompt optimizers**: Alternate between different prompt optimization approaches (experimental)
+
+## Further Reading
+
+- [BetterTogether Paper: arxiv:2407.10930](https://arxiv.org/abs/2407.10930)
+- [Databricks Case Study](https://www.databricks.com/blog/building-state-art-enterprise-agents-90x-cheaper-automated-prompt-optimization) - Real-world application combining BetterTogether with GEPA
+- [DSPy Optimizers Overview](../../../learn/programming/optimizers.md)
diff --git a/docs/docs/learn/optimization/optimizers.md b/docs/docs/learn/optimization/optimizers.md
index 253807cf3..ad98a1970 100644
--- a/docs/docs/learn/optimization/optimizers.md
+++ b/docs/docs/learn/optimization/optimizers.md
@@ -73,6 +73,10 @@ This optimizer is used to fine-tune the underlying LLM(s).
10. [**`Ensemble`**](../../api/optimizers/Ensemble.md): Ensembles a set of DSPy programs and either uses the full set or randomly samples a subset into a single program.
+### Meta-Optimizers
+
+11. [**`BetterTogether`**](../../api/optimizers/BetterTogether.md): A meta-optimizer that combines prompt optimization and weight optimization (fine-tuning) in configurable sequences. Prompt optimization can discover effective task decompositions and reasoning strategies, while weight optimization can specialize the model to execute these patterns more efficiently. Using these approaches together in sequences (e.g., prompt → weight → prompt) may allow each to build on the improvements made by the other. Empirically, this approach often outperforms either strategy alone. Detailed tutorial available at [BetterTogether AIME Tutorial](../../tutorials/bettertogether_aime/index.ipynb).
+
## Which optimizer should I use?
diff --git a/dspy/teleprompt/bettertogether.py b/dspy/teleprompt/bettertogether.py
index d1154f9ae..c842df6f5 100644
--- a/dspy/teleprompt/bettertogether.py
+++ b/dspy/teleprompt/bettertogether.py
@@ -1,8 +1,10 @@
+import inspect
import logging
import random
-from typing import Callable
+from typing import Any, Callable
import dspy
+from dspy.evaluate import Evaluate
from dspy.primitives.example import Example
from dspy.primitives.module import Module
from dspy.teleprompt.bootstrap_finetune import (
@@ -11,160 +13,619 @@ from dspy.teleprompt.bootstrap_finetune import (
kill_lms,
launch_lms,
prepare_student,
+ prepare_teacher,
)
from dspy.teleprompt.random_search import BootstrapFewShotWithRandomSearch
from dspy.teleprompt.teleprompt import Teleprompter
+from dspy.teleprompt.utils import eval_candidate_program
logger = logging.getLogger(__name__)
+# ANSI color codes for terminal output
+YELLOW = "\033[93m"
+GREEN = "\033[92m"
+BLUE = "\033[94m"
+BOLD = "\033[1m"
+ENDC = "\033[0m"
+
class BetterTogether(Teleprompter):
+ """A meta-optimizer that combines prompt and weight optimization in configurable sequences.
+
+ BetterTogether is a meta-optimizer proposed in the paper
+ [Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together](https://arxiv.org/abs/2407.10930).
+ It combines prompt optimization and weight optimization (fine-tuning) by applying them in a
+ configurable sequence, allowing a student program to iteratively improve both its prompts and
+ model parameters.
+
+ The core insight is that prompt and weight optimization can complement each other: prompt
+ optimization can potentially discover effective task decompositions and reasoning strategies,
+ while weight optimization can specialize the model to execute these patterns more efficiently.
+ Using these approaches together in sequences (e.g., prompt optimization then weight optimization)
+ may allow each to build on the improvements made by the other. Empirically, this approach often
+ outperforms either strategy alone, even with state-of-the-art optimizers. For example, a
+ [Databricks case study](https://www.databricks.com/blog/building-state-art-enterprise-agents-90x-cheaper-automated-prompt-optimization)
+ shows that combining BetterTogether with GEPA and fine-tuning outperforms either approach
+ alone.
+
+ The optimizer is initialized with a metric and custom optimizers. For example, you can combine
+ GEPA for prompt optimization with BootstrapFinetune for weight optimization:
+ ``BetterTogether(metric=metric, p=GEPA(...), w=BootstrapFinetune(...))``. The ``compile()``
+ method takes a student program, trainset, and strategy string where strategy keys correspond
+ to the optimizer names from initialization. It executes each optimizer in the specified sequence.
+ When a validation set is provided, the best performing program is returned; otherwise, the
+ latest program is returned. Note: Weight optimizers like BootstrapFinetune require student
+ programs to have LMs explicitly set (not relying on global dspy.settings.lm), and BetterTogether
+ mirrors this requirement for simplicity. Therefore we call ``set_lm`` before compiling.
+
+ >>> from dspy.teleprompt import GEPA, BootstrapFinetune
+ >>>
+ >>> # Combine GEPA for prompt optimization with BootstrapFinetune for weight optimization
+ >>> optimizer = BetterTogether(
+ ... metric=metric,
+ ... p=GEPA(metric=metric, auto="medium"),
+ ... w=BootstrapFinetune(metric=metric)
+ ... )
+ >>>
+ >>> student.set_lm(lm)
+ >>> compiled = optimizer.compile(
+ ... student,
+ ... trainset=trainset,
+ ... valset=valset,
+ ... strategy="p -> w"
+ ... )
+
+ You can pass optimizer-specific arguments to each optimizer's ``compile()`` method using
+ ``optimizer_compile_args``. This allows you to customize each optimizer's behavior:
+
+ >>> from dspy.teleprompt import MIPROv2
+ >>>
+ >>> # Use MIPROv2 for prompt optimization with custom parameters
+ >>> optimizer = BetterTogether(
+ ... metric=metric,
+ ... p=MIPROv2(metric=metric),
+ ... w=BootstrapFinetune(metric=metric)
+ ... )
+ >>>
+ >>> student.set_lm(lm)
+ >>> compiled = optimizer.compile(
+ ... student,
+ ... trainset=trainset,
+ ... valset=valset,
+ ... strategy="p -> w",
+ ... optimizer_compile_args={
+ ... "p": {"num_trials": 10, "max_bootstrapped_demos": 8}, # Configure MIPROv2's compile arguments
+ ... }
+ ... )
+
+ Since BetterTogether is a meta-optimizer that can run arbitrary optimizers in sequence, any
+ sequence of optimizers can be combined together. The optimizer names used in the strategy string
+ correspond to the keyword arguments specified in the constructor. For example, different prompt
+ optimizers can be alternated multiple times (though note this is just an illustration of
+ BetterTogether's flexibility, not a recommended configuration):
+
+ >>> from dspy.teleprompt import MIPROv2, GEPA
+ >>>
+ >>> # Chain two optimizers three times: MIPROv2 -> GEPA -> MIPROv2
+ >>> optimizer = BetterTogether(
+ ... metric=metric,
+ ... mipro=MIPROv2(metric=metric, auto="light"),
+ ... gepa=GEPA(metric=metric, auto="light")
+ ... )
+ >>>
+ >>> student.set_lm(lm)
+ >>> compiled = optimizer.compile(
+ ... student,
+ ... trainset=trainset,
+ ... valset=valset,
+ ... strategy="mipro -> gepa -> mipro"
+ ... )
+
+ Note:
+ Output Attributes: The returned program includes two additional attributes:
+ ``candidate_programs`` and ``flag_compilation_error_occurred``. The ``candidate_programs``
+ attribute is a list of dicts, each containing 'program', 'score', and 'strategy' (e.g.,
+ '', 'p', 'p -> w', 'p -> w -> p'), sorted by descending score (similar to
+ ``dspy.MIPROv2.candidate_programs``). If any optimizer step fails,
+ ``flag_compilation_error_occurred`` is set to True and the best program found so far is
+ returned.
+
+ Model Lifecycle Management: BetterTogether automatically manages language model lifecycle
+ (launching, killing, and relaunching after fine-tuning), which are no-ops for most
+ API-based LMs. This is particularly important when using weight optimizers like
+ BootstrapFinetune with local providers (e.g., dspy.LocalProvider), as it handles model
+ initialization and cleanup between optimization steps.
+ """
STRAT_SEP = " -> "
- def __init__(self,
+ def __init__(
+ self,
metric: Callable,
- prompt_optimizer: Teleprompter | None = None,
- weight_optimizer: Teleprompter | None = None,
- seed: int | None = None,
- ):
- if not dspy.settings.experimental:
- raise ValueError("This is an experimental optimizer. Set `dspy.settings.experimental` to `True` to use it.")
-
- # TODO: Note that the BetterTogether optimizer is meaningful when
- # BootstrapFinetune uses a metric to filter the training data before
- # fine-tuning. However, one can also choose to run this optimizer with
- # a BootstrapFinetune without a metric, say, if there aren't labels
- # available for the training data. Should this be noted somewhere?
- # TODO: We should re-consider if the metric should be required.
- self.prompt_optimizer = prompt_optimizer if prompt_optimizer else BootstrapFewShotWithRandomSearch(metric=metric)
- self.weight_optimizer = weight_optimizer if weight_optimizer else BootstrapFinetune(metric=metric)
-
- is_supported_prompt = isinstance(self.prompt_optimizer, BootstrapFewShotWithRandomSearch)
- is_supported_weight = isinstance(self.weight_optimizer, BootstrapFinetune)
- if not is_supported_prompt or not is_supported_weight:
- raise ValueError(
- "The BetterTogether optimizer only supports the following optimizers for now: BootstrapFinetune, "
- "BootstrapFewShotWithRandomSearch."
- )
+ **optimizers: Teleprompter,
+ ):
+ """Initialize BetterTogether with a metric and custom optimizers.
- self.rng = random.Random(seed)
+ Args:
+ metric: Evaluation metric function for scoring programs. Should accept
+ ``(example, prediction, trace=None)`` and return a numeric score (higher is better).
+ This metric is used to evaluate candidate programs during optimization and is passed
+ to the default optimizers if no custom optimizers are provided.
+ **optimizers: Custom optimizers as keyword arguments, where keys become the optimizer
+ names used in the strategy string. For example, ``p=GEPA(...), w=BootstrapFinetune(...)``
+ makes 'p' and 'w' available for use in strategies like ``"p -> w"``. If not provided,
+ defaults to ``p=BootstrapFewShotWithRandomSearch(metric=metric)`` and
+ ``w=BootstrapFinetune(metric=metric)``. Any DSPy Teleprompter can be used.
+
+ Example:
+ >>> # Use custom optimizers
+ >>> from dspy.teleprompt import GEPA, BootstrapFinetune
+ >>> optimizer = BetterTogether(
+ ... metric=metric,
+ ... p=GEPA(metric=metric, auto="medium"),
+ ... w=BootstrapFinetune(metric=metric)
+ ... )
+ >>>
+ >>> # Use default optimizers
+ >>> optimizer = BetterTogether(metric=metric)
+ """
+ self.metric = metric
+
+ if not optimizers:
+ logger.info(
+ "No optimizers provided. Using defaults: "
+ "BootstrapFewShotWithRandomSearch (p) and BootstrapFinetune (w). "
+ "You can use the letters p and w to specify the compile strategy. "
+ "For example, to run weight optimization after prompt optimization, use strategy='p -> w'."
+ )
+ optimizers = {
+ "p": BootstrapFewShotWithRandomSearch(metric=metric),
+ "w": BootstrapFinetune(metric=metric),
+ }
+ for key, optimizer in optimizers.items():
+ if not isinstance(optimizer, Teleprompter):
+ raise TypeError(
+ f"Optimizer '{key}' must be a Teleprompter, "
+ f"got {type(optimizer).__name__}"
+ )
+ self.optimizers: dict[str, Teleprompter] = optimizers
def compile(
self,
student: Module,
+ *,
trainset: list[Example],
+ teacher: Module | list[Module] | None = None,
+ valset: list[Example] | None = None,
+ # often specified in init in other optimizers
+ num_threads: int | None = None,
+ max_errors: int | None = None,
+ provide_traceback: bool | None = None,
+ seed: int | None = None,
+ # specific to BetterTogether
+ valset_ratio: float = 0.1,
+ shuffle_trainset_between_steps: bool = True,
strategy: str = "p -> w -> p",
- valset_ratio = 0.1,
+ optimizer_compile_args: dict[str, dict[str, Any]] | None = None,
) -> Module:
- # TODO: We could record acc on a different valset to pick the best
- # strategy within the provided strategy
- logger.info("Validating the strategy")
- parsed_strategy = strategy.lower().split(self.STRAT_SEP)
+ """Compile and optimize a student program using a sequence of optimization strategies.
- if not all(s in ["p", "w"] for s in parsed_strategy):
- raise ValueError(
- f"The strategy should be a sequence of 'p' and 'w' separated by '{self.STRAT_SEP}', but "
- f"found: {strategy}"
- )
+ Executes the optimizers specified in the strategy string sequentially, evaluating each
+ intermediate result and returning the best performing program.
+
+ Args:
+ student: DSPy program to optimize. All predictors must have language models assigned.
+ program.set_lm(lm) can be used to assign a language model to all modules of a
+ program.
+ trainset: Training examples for optimization. Each optimizer receives the full trainset
+ (or a shuffled version if ``shuffle_trainset_between_steps=True``).
+ teacher: Optional teacher module(s) for bootstrapping. Can be a single module or list.
+ Passed to optimizers.
+ valset: Validation set for evaluating optimization steps. If not provided, a portion of
+ trainset is held out (controlled by ``valset_ratio``). If both ``valset`` and
+ ``valset_ratio`` are None/0, no validation occurs and the latest program is returned.
+ num_threads: Number of parallel evaluation threads. Default is None, which means sequential evaluation.
+ max_errors: Maximum errors to tolerate during evaluation. Defaults to
+ ``dspy.settings.max_errors``.
+ provide_traceback: Whether to show detailed tracebacks for evaluation errors.
+ seed: Random seed for reproducibility. Controls trainset shuffling and evaluation sampling.
+ valset_ratio: Fraction of trainset to hold out as validation (range [0, 1)). For example,
+ 0.1 holds out 10%. Set to 0 to skip validation. Default is 0.1.
+ shuffle_trainset_between_steps: Whether to shuffle trainset before each optimization step.
+ Helps prevent overfitting to example ordering. Default is True.
+ strategy: Sequence of optimizers to apply, separated by ``" -> "``. Each element must be
+ a key from the optimizers provided in ``__init__``. For example, ``"p -> w -> p"``
+ applies prompt optimization, then weight optimization, then prompt optimization again.
+ Default is ``"p -> w -> p"``.
+ optimizer_compile_args: Optional dict mapping optimizer keys to their ``compile()``
+ arguments. If trainset, valset, or teacher are provided in the dict for a specific
+ optimizer, they override the defaults from BetterTogether's compile method. For example:
+ ``{"p": {"num_trials": 10}, "w": {"trainset": custom_trainset}}``. This is useful to
+ override the default compile arguments for specific optimizers. The ``student`` argument
+ cannot be included in optimizer_compile_args; BetterTogether's compile method manages
+ the student reference for all optimizers.
+
+ Returns:
+ Optimized student program with two additional attributes:
- logger.info("Preparing the student program...")
- # TODO: Prepare student returns student.reset_copy(), which is what gets
- # optimized. We should make this clear in the doc comments.
+ - ``candidate_programs``: List of dicts with 'program', 'score', and 'strategy' keys,
+ sorted by score (best first). Contains all evaluated programs including the baseline.
+ - ``flag_compilation_error_occurred``: Boolean indicating if any optimization step failed.
+
+ Raises:
+ ValueError: If trainset is empty, valset_ratio not in [0, 1), strategy is empty or
+ contains invalid optimizer keys, or optimizer_compile_args contains invalid arguments.
+ TypeError: If optimizer_compile_args contains a 'student' key (not allowed).
+
+ Example:
+ >>> optimizer = BetterTogether(
+ ... metric=metric,
+ ... p=GEPA(metric=metric),
+ ... w=BootstrapFinetune(metric=metric)
+ ... )
+ >>> student.set_lm(lm)
+ >>> compiled = optimizer.compile(
+ ... student,
+ ... trainset=trainset,
+ ... valset=valset,
+ ... strategy="p -> w"
+ ... )
+ >>> print(f"Best score: {compiled.candidate_programs[0]['score']}")
+ """
+ logger.info(f"\n{BOLD}==> BETTERTOGETHER COMPILATION STARTED <=={ENDC}")
+ logger.info(f"{BLUE}Strategy:{ENDC} {strategy}")
+ logger.info(f"{BLUE}Trainset size:{ENDC} {len(trainset)}")
+ logger.info(f"{BLUE}Validation ratio:{ENDC} {valset_ratio if valset is None else 'using provided valset'}")
+
+ student, teacher = self._prepare_student_and_teacher(student, teacher)
+ trainset, valset = self._prepare_trainset_and_valset(trainset, valset, valset_ratio)
+ effective_max_errors = max_errors if max_errors is not None else dspy.settings.max_errors
+ parsed_strategy = self._prepare_strategy(strategy)
+ optimizer_compile_args = self._prepare_optimizer_compile_args(optimizer_compile_args, teacher)
+
+ student = self._run_strategies(
+ student,
+ trainset,
+ teacher,
+ valset,
+ num_threads,
+ effective_max_errors,
+ provide_traceback,
+ seed,
+ parsed_strategy,
+ shuffle_trainset_between_steps,
+ optimizer_compile_args,
+ )
+
+ logger.info(f"\n{BOLD}{GREEN}==> BETTERTOGETHER COMPILATION COMPLETE <=={ENDC}")
+ logger.info(f"{GREEN}Best score achieved:{ENDC} {student.candidate_programs[0]['score']}")
+ logger.info(f"{GREEN}Best strategy:{ENDC} {student.candidate_programs[0]['strategy'] or 'original (no optimization)'}")
+
+ student._compiled = True
+ return student
+
+ def _prepare_student_and_teacher(
+ self, student: Module, teacher: Module | list[Module] | None
+ ) -> tuple[Module, list[Module] | None]:
student = prepare_student(student)
all_predictors_have_lms(student)
- # Make a shallow copy of the trainset, so that we don't change the order
- # of the examples in the original trainset
- trainset = trainset[:]
- logger.info("Compiling the student program...")
- student = self._run_strategies(parsed_strategy, student, trainset, valset_ratio)
+ if not teacher:
+ return student, None
+ teacher = [teacher] if not isinstance(teacher, list) else teacher
+ teacher = [prepare_teacher(student, t) for t in teacher]
- logger.info("BetterTogether has finished compiling the student program")
- return student
+ return student, teacher
+
+ def _prepare_trainset_and_valset(
+ self, trainset: list[Example], valset: list[Example] | None, valset_ratio: float
+ ) -> tuple[list[Example], list[Example] | None]:
+ if not trainset:
+ raise ValueError("trainset cannot be empty")
+
+ if valset_ratio < 0 or valset_ratio >= 1:
+ raise ValueError(f"valset_ratio must be in range [0, 1), got {valset_ratio}")
+
+ trainset = trainset[:] # shallow copy to avoid modifying the original
+
+ if valset:
+ logger.info(f"{BLUE}Using provided validation set ({len(valset)} examples). Ignoring valset_ratio.{ENDC}")
+ return trainset, valset
+
+ if valset_ratio == 0:
+ logger.info(f"{YELLOW}No validation set provided and valset_ratio=0. No validation set created.{ENDC}")
+ return trainset, None
+
+ logger.info(f"{BLUE}Sampling {valset_ratio:.1%} of trainset as validation set.{ENDC}")
+ num_val_examples = int(valset_ratio * len(trainset))
+ valset = trainset[:num_val_examples]
+ trainset = trainset[num_val_examples:]
+ logger.info(f"{BLUE}Created validation set: {len(valset)} examples. Training set: {len(trainset)} examples.{ENDC}")
+
+ return trainset, valset
+
+ def _prepare_strategy(self, strategy: str) -> list[str]:
+ if not strategy or not strategy.strip():
+ raise ValueError("strategy cannot be empty")
+
+ parsed_strategy = strategy.split(self.STRAT_SEP)
+
+ invalid_steps = [s for s in parsed_strategy if s not in self.optimizers]
+ if invalid_steps:
+ raise ValueError(
+ f"Strategy contains invalid optimizer keys: {invalid_steps}. "
+ f"Valid keys are: {list(self.optimizers.keys())}"
+ )
+
+ return parsed_strategy
+
+ def _prepare_optimizer_compile_args(
+ self,
+ optimizer_compile_args: dict[str, dict[str, Any]] | None,
+ teacher: list[Module] | None,
+ ) -> dict[str, dict[str, Any]]:
+ logger.info(f"{BLUE}Validating optimizer compile arguments...{ENDC}")
+
+ # Validate user-provided compile args
+ if not optimizer_compile_args:
+ return {}
+
+ for optimizer_key, compile_args in optimizer_compile_args.items():
+ if optimizer_key not in self.optimizers:
+ raise ValueError(
+ f"Invalid optimizer key '{optimizer_key}'. "
+ f"Valid keys are: {list(self.optimizers.keys())}"
+ )
+ optimizer = self.optimizers[optimizer_key]
+ self._validate_compile_args(optimizer, optimizer_key, compile_args)
+
+ # Special checks
+ if optimizer.__class__.__name__ == "GEPA":
+ # GEPA accepts a teacher argument, but raises an error if it's set.
+ if teacher is not None:
+ raise ValueError(
+ "GEPA does not accept a teacher argument. Please remove the teacher argument."
+ )
+
+ return optimizer_compile_args
+
+ def _validate_compile_args(
+ self, optimizer: Teleprompter, optimizer_key: str, compile_args: dict[str, Any]
+ ) -> None:
+ if "student" in compile_args:
+ raise ValueError(
+ f"'student' is not allowed in optimizer_compile_args for optimizer '{optimizer_key}'. "
+ "The same student is used throughout compilation."
+ )
+
+ valid_params = inspect.signature(optimizer.compile).parameters
+ invalid_args = set(compile_args.keys()) - set(valid_params.keys())
+ if invalid_args:
+ raise ValueError(
+ f"Invalid compile arguments for optimizer '{optimizer_key}': {sorted(invalid_args)}. "
+ f"{optimizer.__class__.__name__}.compile() accepts: {list(valid_params.keys())}"
+ )
- def _run_strategies(self, parsed_strategy, student, trainset, valset_ratio) -> Module:
- # Keep track of all the partial strategies/programs in parsed_strategy
- # "" corresponds to the initial student program
+ def _run_strategies(
+ self,
+ student: Module,
+ trainset: list[Example],
+ teacher: list[Module] | None,
+ valset: list[Example] | None,
+ num_threads: int | None,
+ effective_max_errors: int | None,
+ provide_traceback: bool | None,
+ seed: int | None,
+ parsed_strategy: list[str],
+ shuffle_trainset_between_steps: bool,
+ optimizer_args: dict[str, dict[str, Any]],
+ ) -> Module:
+ rng = random.Random(seed)
candidate_programs = []
- candidate_programs.append(("", student))
- launched_flag = False
+ flag_lms_launched = False
+ flag_compilation_error_occurred = False
+
+ # Evaluate original program
+ logger.info(f"\n{BOLD}==> BASELINE EVALUATION <=={ENDC}")
+ logger.info("Evaluating original program (no optimization applied)")
+
+ launch_lms(student)
+ flag_lms_launched = True
+
+ score = self._evaluate_on_valset(student, valset, rng, num_threads, effective_max_errors, provide_traceback)
+ self._add_candidate(candidate_programs, student, strategy="", score=score)
+ logger.info(f"{YELLOW}Baseline score:{ENDC} {score}")
+ # Apply each optimization step
for ind, step_code in enumerate(parsed_strategy):
current_strategy = self.STRAT_SEP.join(parsed_strategy[:ind + 1])
- logger.info(
- f"\n########## Step {ind + 1} of {len(parsed_strategy)} - Strategy "
- f"'{current_strategy}' ##########"
- )
+ optimizer = self.optimizers[step_code]
+
+ logger.info(f"\n{BOLD}==> STEP {ind + 1}/{len(parsed_strategy)}: {optimizer.__class__.__name__.upper()} <=={ENDC}")
+ logger.info(f"{BLUE}Current strategy:{ENDC} '{current_strategy}'")
+ logger.info(f"{BLUE}Optimizer:{ENDC} {optimizer.__class__.__name__}")
+
+ try:
+ if shuffle_trainset_between_steps:
+ logger.info(f"{BLUE}Shuffling trainset...{ENDC}")
+ rng.shuffle(trainset)
+
+ # Run optimizer, evaluate, and record results
+ compile_args = optimizer_args.get(step_code, {})
+ student, score, is_new_best, lms_relaunched = self._run_and_evaluate_step(
+ optimizer, student, teacher, trainset, valset, compile_args,
+ candidate_programs, current_strategy, rng,
+ num_threads, effective_max_errors, provide_traceback
+ )
+
+ if lms_relaunched:
+ flag_lms_launched = True
- logger.info("Shuffling the trainset...")
- self.rng.shuffle(trainset)
- if not launched_flag:
- launch_lms(student)
- launched_flag = True
-
- # TODO: Should we reset or just deepcopy? How does resetting affect
- # the predictor LMs?
- student = student.deepcopy()
- student._compiled = False
- if step_code == "p":
- student = self._compile_prompt_optimizer(student, trainset, valset_ratio)
- elif step_code == "w":
- student = self._compile_weight_optimizer(student, trainset)
- launched_flag = False
-
- # Record the program corresponding to the current strategy
- candidate_programs.append((current_strategy, student))
-
- if launched_flag:
+ # Log score
+ if is_new_best:
+ logger.info(f"{GREEN}New best score!{ENDC} {score} (strategy: '{current_strategy}')")
+ else:
+ logger.info(f"{YELLOW}Score after optimization:{ENDC} {score}")
+
+ except Exception as e:
+ flag_compilation_error_occurred = True
+ logger.error(
+ f"{YELLOW}Step {ind + 1}/{len(parsed_strategy)} failed with error: {type(e).__name__}: {e}{ENDC}"
+ )
+ logger.error(
+ f"{YELLOW}Stopping optimization early. Returning best program found so far from {len(candidate_programs)} candidate(s).{ENDC}"
+ )
+ logger.error(f"{YELLOW}Traceback:{ENDC}", exc_info=True)
+ break
+
+ # Cleanup and finalize
+ if flag_lms_launched:
kill_lms(student)
- student.candidate_programs = candidate_programs
- return student
+ # Sort candidates by score (best first), with earlier programs winning ties
+ candidate_programs_with_idx = [(i, cp) for i, cp in enumerate(candidate_programs)]
+ candidate_programs_with_idx.sort(
+ key=lambda x: (x[1]["score"] if x[1]["score"] is not None else float("-inf"), -x[0]),
+ reverse=True
+ )
+ candidate_programs = [cp for _, cp in candidate_programs_with_idx]
- def _compile_prompt_optimizer(self, student, trainset, valset_ratio) -> Module:
- logger.info("Preparing for prompt optimization...")
-
- # Sampling a validation set from the trainset for the prompt optimizer
- # We drop the hints for prompt optimization
- trainset = [x.with_inputs(*list(set(x.inputs().keys()) - {"hint"})) for x in trainset]
- num_val = int(valset_ratio * len(trainset))
- prompt_valset = trainset[:num_val]
- prompt_trainset = trainset[num_val:]
-
- # TODO: To make this optimizer general, we need to ensure that all the
- # prompt optimizers are accepting a valset or encode a way to check if
- # a valset should be passed to an optimizer's compile method.
- # TODO: We should ensure that the prompt optimizers in DSPy respect the
- # predictor.lm attributes. In particular,
- # BootstrapFewShotWithRandomSearch seems to be resetting these. We are
- # manually re-setting the LMs here to circumvent this issue, but we
- # should consider addressing it in BFRS.
- logger.info("Compiling the prompt optimizer...")
- pred_lms = [pred.lm for pred in student.predictors()]
- student = self.prompt_optimizer.compile(student, trainset=prompt_trainset, valset=prompt_valset)
- for pred, lm in zip(student.predictors(), pred_lms, strict=False):
- pred.lm = lm
+ # Select best program based on valset availability
+ if valset is None or len(valset) == 0:
+ # No valset: return the latest program (last in original order)
+ best_program = candidate_programs_with_idx[-1][1]
+ else:
+ # Valset provided: return highest score (first after sorting)
+ best_program = candidate_programs[0]
- return student
+ # Attach sorted candidate programs and error flag to the best program
+ best_student = best_program["program"]
+ best_student.candidate_programs = candidate_programs
+ best_student.flag_compilation_error_occurred = flag_compilation_error_occurred
+
+ logger.info(f"\n{BOLD}==> OPTIMIZATION SUMMARY <=={ENDC}")
+ logger.info(f"{GREEN}Best score:{ENDC} {best_program['score']}")
+ strategy_display = best_program["strategy"] if best_program["strategy"] else "original (no optimization)"
+ logger.info(f"{GREEN}Best strategy:{ENDC} {strategy_display}")
+ logger.info(f"{BLUE}Total candidates evaluated:{ENDC} {len(candidate_programs)}")
+
+ return best_student
- def _compile_weight_optimizer(self, student, trainset) -> Module:
- logger.info("Preparing for weight optimization...")
+ def _run_and_evaluate_step(
+ self,
+ optimizer: Teleprompter,
+ student: Module,
+ teacher: list[Module] | None,
+ trainset: list[Example],
+ valset: list[Example] | None,
+ compile_args: dict[str, Any],
+ candidate_programs: list,
+ current_strategy: str,
+ rng: random.Random,
+ num_threads: int | None,
+ effective_max_errors: int | None,
+ provide_traceback: bool | None,
+ ) -> tuple[Module, float | None, bool, bool]:
+ """Run optimizer, evaluate result, and record candidate program.
- # Saving the LMs before compiling the weight optimizer
- original_lms = [pred.lm for pred in student.predictors()]
+ Returns:
+ student: Optimized student program
+ score: Validation score (None if no valset)
+ is_new_best: Whether this score is the best so far
+ lms_relaunched: Whether LMs were relaunched (for flag tracking)
+ """
+ # Save LMs before optimization
+ pred_lms_before = [pred.lm for pred in student.predictors()]
- # TODO: To make this optimizer general, we need to ensure that all the
- # prompt optimizers are accepting a valset or encode a way to check if
- # a valset should be passed to an optimizer's compile.
- logger.info("Compiling the weight optimizer...")
- student = self.weight_optimizer.compile(student, trainset=trainset)
+ # Run optimizer
+ student._compiled = False
+ logger.info(f"{BLUE}Running {optimizer.__class__.__name__} with {len(trainset)} training examples...{ENDC}")
- # Updating the train kwargs for the new LMs. This is needed because the
- # train_kwargs of the optimizer is configured for the original LMs.
- new_lms = [pred.lm for pred in student.predictors()]
- for original_lm, new_lm in zip(original_lms, new_lms, strict=False):
- original_params = self.weight_optimizer.train_kwargs[original_lm]
- self.weight_optimizer.train_kwargs[new_lm] = original_params
+ # Build compile args with standard parameters, filtering to only what the optimizer accepts
+ potential_args = {"trainset": trainset, "teacher": teacher, "valset": valset, **compile_args}
+ sig = inspect.signature(optimizer.compile)
+ accepted_params = set(sig.parameters.keys())
+ filtered_compile_args = {k: v for k, v in potential_args.items() if k in accepted_params}
- return student
+ student = optimizer.compile(student, **filtered_compile_args)
+
+ # Restore LMs if optimizer incorrectly reset them
+ # TODO: Some optimizers like BootstrapFewShotWithRandomSearch reset predictor LMs during compilation,
+ # which breaks weight optimizers as they require each predictor to have an LM. We should ensure that
+ # all optimizers respect the assigned LMs of programs and do not override them. Until then, we restore
+ # the original LMs here.
+ if not all_predictors_have_lms(student):
+ logger.warning(
+ f"{YELLOW}Warning: {optimizer.__class__.__name__} incorrectly reset predictor LMs. "
+ f"Restoring to original LMs.{ENDC}"
+ )
+ for pred, lm in zip(student.predictors(), pred_lms_before, strict=False):
+ pred.lm = lm
+
+ # Relaunch LMs if models changed (e.g., after fine-tuning)
+ # Weight optimizers create new model instances and kill the old ones before returning.
+ # We detect this via model name changes and relaunch the new models.
+ # Note: launch() and kill() are no-ops for most API-based LMs; these mainly affect local
+ # LMs that need launch or clean up routines.
+ lms_relaunched = False
+ if self._models_changed(student, pred_lms_before):
+ launch_lms(student)
+ lms_relaunched = True
+
+ # Evaluate optimized program
+ score = self._evaluate_on_valset(student, valset, rng, num_threads, effective_max_errors, provide_traceback)
+ self._add_candidate(candidate_programs, student, current_strategy, score)
+
+ # Check if this is the best score so far
+ valid_scores = [cp["score"] for cp in candidate_programs if cp["score"] is not None]
+ best_score_so_far = max(valid_scores) if valid_scores else float("-inf")
+ is_new_best = score is not None and score >= best_score_so_far
+
+ return student, score, is_new_best, lms_relaunched
+
+ def _models_changed(self, student: Module, pred_lms_before: list) -> bool:
+ """Check if model names changed after optimization (e.g., fine-tuning)."""
+ pred_lms_after = [pred.lm for pred in student.predictors()]
+ model_names_before = [lm.model if lm else None for lm in pred_lms_before]
+ model_names_after = [lm.model if lm else None for lm in pred_lms_after]
+ return model_names_before != model_names_after
+
+ def _add_candidate(
+ self,
+ candidate_programs: list,
+ student: Module,
+ strategy: str,
+ score: float,
+ ) -> None:
+ """Add a candidate program to the list."""
+ candidate_programs.append({
+ "score": score,
+ "program": student.deepcopy(),
+ "strategy": strategy,
+ })
+
+ def _evaluate_on_valset(
+ self,
+ program: Module,
+ valset: list[Example] | None,
+ rng: random.Random,
+ num_threads: int | None,
+ effective_max_errors: int | None,
+ provide_traceback: bool | None,
+ ) -> float | None:
+ if valset is None or len(valset) == 0:
+ logger.info(f"{YELLOW}No validation set provided. Skipping evaluation.{ENDC}")
+ return None
+
+ logger.info(f"{BLUE}Evaluating on {len(valset)} validation examples...{ENDC}")
+ evaluate = Evaluate(
+ devset=valset,
+ metric=self.metric,
+ num_threads=num_threads,
+ max_errors=effective_max_errors,
+ display_table=False,
+ display_progress=True,
+ provide_traceback=provide_traceback,
+ )
+ eval_result = eval_candidate_program(len(valset), valset, program, evaluate, rng)
+ return eval_result.score
diff --git a/tests/teleprompt/test_bettertogether.py b/tests/teleprompt/test_bettertogether.py
new file mode 100644
index 000000000..8f4ee4988
--- /dev/null
+++ b/tests/teleprompt/test_bettertogether.py
@@ -0,0 +1,744 @@
+"""BetterTogether optimizer tests.
+
+Most of the code in this test file was LLM-generated but has been verified
+to correctly test the BetterTogether optimizer functionality.
+"""
+from unittest.mock import Mock, patch
+
+import pytest
+
+import dspy
+from dspy import Example
+from dspy.predict import Predict
+from dspy.teleprompt import BetterTogether, BootstrapFewShotWithRandomSearch, BootstrapFinetune
+from dspy.teleprompt.teleprompt import Teleprompter
+from dspy.utils.dummies import DummyLM
+
+
+# Define a simple metric function for testing
+def simple_metric(example, prediction, trace=None):
+ return 1.0 if example.output == prediction.output else 0.0
+
+
+examples = [
+ Example(input="What is the oldest known human-made monument?", output="Göbekli Tepe in southeastern Turkiye, dating back to around 9600 BCE").with_inputs("input"),
+ Example(input="Why can't fish fall in love?", output="Because love is in the air").with_inputs("input"),
+ Example(input="What would bring world peace?", output="8 billion people meeting for a tea party in my backyard").with_inputs("input"),
+]
+trainset = examples[:2]
+valset = [examples[2]]
+
+
+class SimpleModule(dspy.Module):
+ def __init__(self, signature):
+ super().__init__()
+ self.predictor = Predict(signature)
+
+ def forward(self, **kwargs):
+ return self.predictor(**kwargs)
+
+
+# ============================================================================
+# Reusable Mock Optimizers
+# ============================================================================
+
+class SimpleOptimizer(Teleprompter):
+ """A simple optimizer that returns the student unchanged."""
+ def compile(self, student, **kwargs):
+ return student
+
+
+class MarkedOptimizer(Teleprompter):
+ """An optimizer that marks the program with a specific identifier."""
+ def __init__(self, marker):
+ self.marker = marker
+
+ def compile(self, student, **kwargs):
+ prog = SimpleModule("input -> output")
+ prog.marker = self.marker
+ return prog
+
+
+class CapturingOptimizer(Teleprompter):
+ """An optimizer that captures the kwargs it receives."""
+ def __init__(self):
+ self.received_kwargs = {}
+
+ def compile(self, student, trainset=None, valset=None, teacher=None,
+ num_trials=None, max_bootstrapped_demos=None, **kwargs):
+ self.received_kwargs = {
+ "trainset": trainset,
+ "valset": valset,
+ "teacher": teacher,
+ "num_trials": num_trials,
+ "max_bootstrapped_demos": max_bootstrapped_demos,
+ **kwargs
+ }
+ return student
+
+
+# ============================================================================
+# Pytest Fixtures
+# ============================================================================
+
+def student_with_lm():
+ """Create a student module with a DummyLM."""
+ student = SimpleModule("input -> output")
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+ return student
+
+
+def mock_bt_dependencies():
+ """Mock the common BetterTogether dependencies."""
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval, \
+ patch("dspy.teleprompt.bettertogether.launch_lms") as mock_launch, \
+ patch("dspy.teleprompt.bettertogether.kill_lms") as mock_kill:
+ mock_eval.return_value = Mock(score=0.8)
+ yield mock_eval, mock_launch, mock_kill
+
+
+# ============================================================================
+# Tests
+# ============================================================================
+
+def test_bettertogether_import():
+ """Sanity check: Test that BetterTogether can be imported."""
+ assert BetterTogether is not None, "Failed to import BetterTogether"
+
+
+def test_bettertogether_initialization_default():
+ """Test BetterTogether initialization with default optimizers."""
+ optimizer = BetterTogether(metric=simple_metric)
+
+ assert optimizer.metric == simple_metric, "Metric not correctly initialized"
+ assert "p" in optimizer.optimizers, "Default 'p' optimizer not created"
+ assert "w" in optimizer.optimizers, "Default 'w' optimizer not created"
+ assert isinstance(optimizer.optimizers["p"], BootstrapFewShotWithRandomSearch), \
+ "Default 'p' should be BootstrapFewShotWithRandomSearch"
+ assert isinstance(optimizer.optimizers["w"], BootstrapFinetune), \
+ "Default 'w' should be BootstrapFinetune"
+
+
+def test_bettertogether_initialization_custom():
+ """Test BetterTogether initialization with custom optimizers."""
+ custom_p = BootstrapFewShotWithRandomSearch(metric=simple_metric)
+ custom_w = BootstrapFinetune(metric=simple_metric)
+
+ optimizer = BetterTogether(
+ metric=simple_metric,
+ p=custom_p,
+ w=custom_w
+ )
+
+ assert optimizer.optimizers["p"] is custom_p, "Custom 'p' optimizer not set"
+ assert optimizer.optimizers["w"] is custom_w, "Custom 'w' optimizer not set"
+
+
+def test_bettertogether_initialization_invalid_optimizer():
+ """Test that BetterTogether rejects non-Teleprompter optimizers."""
+ try:
+ optimizer = BetterTogether(
+ metric=simple_metric,
+ p="not_a_teleprompter" # Invalid type
+ )
+ assert False, "Should have raised TypeError for invalid optimizer"
+ except TypeError as e:
+ assert "must be a Teleprompter" in str(e)
+
+
+def test_strategy_validation():
+ """Test strategy validation: valid, invalid, and empty strategies."""
+ optimizer = BetterTogether(metric=simple_metric)
+
+ # Valid strategies should parse without errors
+ valid_strategies = ["p", "w", "p -> w", "w -> p", "p -> w -> p"]
+ for strategy in valid_strategies:
+ parsed = optimizer._prepare_strategy(strategy)
+ assert parsed is not None, f"Failed to parse valid strategy: {strategy}"
+
+ # Invalid strategies should raise ValueError
+ with pytest.raises(ValueError, match="invalid optimizer keys"):
+ optimizer._prepare_strategy("p -> x -> w")
+
+ with pytest.raises(ValueError, match="cannot be empty"):
+ optimizer._prepare_strategy("")
+
+
+def test_compile_basic():
+ """Test basic compilation with mocked optimizers."""
+ from dspy.teleprompt.teleprompt import Teleprompter
+
+ student = SimpleModule("input -> output")
+
+ lm = DummyLM([{"output": "blue"}, {"output": "4"}])
+ student.set_lm(lm)
+
+ # Create a mock Teleprompter that returns the student
+ class MockTeleprompter(Teleprompter):
+ def __init__(self):
+ self.compile_called = False
+
+ def compile(self, student, **kwargs):
+ self.compile_called = True
+ return student
+
+ mock_p = MockTeleprompter()
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p)
+
+ # Mock evaluation to avoid actually running the metric
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ mock_eval.return_value = Mock(score=0.8)
+
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ compiled = optimizer.compile(
+ student,
+ trainset=trainset,
+ valset=valset,
+ strategy="p"
+ )
+
+ assert compiled is not None, "Compilation returned None"
+ assert hasattr(compiled, "candidate_programs"), "Missing candidate_programs attribute"
+ assert hasattr(compiled, "flag_compilation_error_occurred"), "Missing flag_compilation_error_occurred attribute"
+ assert mock_p.compile_called, "Mock optimizer compile was not called"
+
+
+def test_trainset_validation():
+ """Test that empty trainset is rejected."""
+ optimizer = BetterTogether(metric=simple_metric)
+ student = SimpleModule("input -> output")
+
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+
+ try:
+ optimizer.compile(student, trainset=[], valset=valset)
+ assert False, "Should have raised ValueError for empty trainset"
+ except ValueError as e:
+ assert "cannot be empty" in str(e).lower()
+
+
+def test_valset_ratio_validation():
+ """Test that invalid valset_ratio is rejected."""
+ optimizer = BetterTogether(metric=simple_metric)
+ student = SimpleModule("input -> output")
+
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+
+ # Test valset_ratio >= 1
+ try:
+ optimizer.compile(student, trainset=trainset, valset_ratio=1.0)
+ assert False, "Should have raised ValueError for valset_ratio >= 1"
+ except ValueError as e:
+ assert "must be in range [0, 1)" in str(e)
+
+ # Test valset_ratio < 0
+ try:
+ optimizer.compile(student, trainset=trainset, valset_ratio=-0.1)
+ assert False, "Should have raised ValueError for valset_ratio < 0"
+ except ValueError as e:
+ assert "must be in range [0, 1)" in str(e)
+
+
+def test_optimizer_compile_args_validation():
+ """Test that optimizer_compile_args is validated correctly."""
+ optimizer = BetterTogether(metric=simple_metric)
+
+ # Test invalid optimizer key
+ try:
+ optimizer._prepare_optimizer_compile_args(
+ {"invalid_key": {"num_trials": 10}},
+ teacher=None
+ )
+ assert False, "Should have raised ValueError for invalid optimizer key"
+ except ValueError as e:
+ assert "invalid optimizer key" in str(e).lower()
+
+
+def test_student_in_optimizer_compile_args():
+ """Test that 'student' in optimizer_compile_args is rejected."""
+ optimizer = BetterTogether(metric=simple_metric)
+
+ try:
+ optimizer._validate_compile_args(
+ optimizer.optimizers["p"],
+ "p",
+ {"student": SimpleModule("input -> output")}
+ )
+ assert False, "Should have raised ValueError for 'student' in compile_args"
+ except ValueError as e:
+ assert "student" in str(e).lower()
+ assert "not allowed" in str(e).lower()
+
+
+def test_compile_args_passed_to_optimizer(student_with_lm, mock_bt_dependencies):
+ """Test that optimizer_compile_args are correctly passed to optimizers."""
+ mock_eval, _, _ = mock_bt_dependencies
+ mock_eval.return_value = Mock(score=0.9)
+
+ mock_p = CapturingOptimizer()
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p)
+
+ # Define custom compile args for optimizer 'p'
+ custom_args = {"num_trials": 20, "max_bootstrapped_demos": 8}
+
+ optimizer.compile(
+ student_with_lm,
+ trainset=trainset,
+ valset=valset,
+ strategy="p",
+ optimizer_compile_args={"p": custom_args}
+ )
+
+ # Verify the custom args were passed to the optimizer
+ assert mock_p.received_kwargs is not None, "Optimizer compile was not called"
+ assert "num_trials" in mock_p.received_kwargs, "num_trials not passed to optimizer"
+ assert mock_p.received_kwargs["num_trials"] == 20, "num_trials value incorrect"
+ assert "max_bootstrapped_demos" in mock_p.received_kwargs, "max_bootstrapped_demos not passed"
+ assert mock_p.received_kwargs["max_bootstrapped_demos"] == 8, "max_bootstrapped_demos value incorrect"
+
+
+def test_compile_args_multi_optimizer_strategy():
+ """Test that different optimizers in a strategy receive their respective compile_args."""
+ from dspy.teleprompt.teleprompt import Teleprompter
+
+ student = SimpleModule("input -> output")
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+
+ # Create mock Teleprompters that capture their compile kwargs
+ class PromptOptimizer(Teleprompter):
+ def __init__(self):
+ self.received_kwargs = {}
+
+ def compile(self, student, trainset=None, num_trials=None, **kwargs):
+ self.received_kwargs = {
+ "trainset": trainset,
+ "num_trials": num_trials,
+ **kwargs
+ }
+ return student
+
+ class WeightOptimizer(Teleprompter):
+ def __init__(self):
+ self.received_kwargs = {}
+
+ def compile(self, student, trainset=None, num_batches=None, **kwargs):
+ self.received_kwargs = {
+ "trainset": trainset,
+ "num_batches": num_batches,
+ **kwargs
+ }
+ return student
+
+ mock_p = PromptOptimizer()
+ mock_w = WeightOptimizer()
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p, w=mock_w)
+
+ # Define different compile args for each optimizer
+ compile_args = {
+ "p": {"num_trials": 10},
+ "w": {"num_batches": 5}
+ }
+
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ mock_eval.return_value = Mock(score=0.85)
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ with patch.object(optimizer, "_models_changed", return_value=False):
+ optimizer.compile(
+ student,
+ trainset=trainset,
+ valset=valset,
+ strategy="p -> w",
+ optimizer_compile_args=compile_args
+ )
+
+ # Verify each optimizer received its specific args
+ assert mock_p.received_kwargs is not None, "Optimizer 'p' compile was not called"
+ assert "num_trials" in mock_p.received_kwargs, "num_trials not passed to optimizer 'p'"
+ assert mock_p.received_kwargs["num_trials"] == 10, "num_trials value incorrect for 'p'"
+ assert mock_p.received_kwargs.get("num_batches") is None, "Optimizer 'p' should not receive 'w' args"
+
+ assert mock_w.received_kwargs is not None, "Optimizer 'w' compile was not called"
+ assert "num_batches" in mock_w.received_kwargs, "num_batches not passed to optimizer 'w'"
+ assert mock_w.received_kwargs["num_batches"] == 5, "num_batches value incorrect for 'w'"
+ assert mock_w.received_kwargs.get("num_trials") is None, "Optimizer 'w' should not receive 'p' args"
+
+
+def test_compile_args_override_global_params():
+ """Test that optimizer_compile_args override global trainset/valset/teacher parameters."""
+ from dspy.teleprompt.teleprompt import Teleprompter
+
+ student = SimpleModule("input -> output")
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+
+ # Create a mock Teleprompter that captures compile kwargs
+ class CapturingTeleprompter(Teleprompter):
+ def __init__(self):
+ self.received_kwargs = {}
+
+ def compile(self, student, trainset=None, valset=None, teacher=None, **kwargs):
+ self.received_kwargs = {
+ "trainset": trainset,
+ "valset": valset,
+ "teacher": teacher,
+ **kwargs
+ }
+ return student
+
+ mock_p = CapturingTeleprompter()
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p)
+
+ # Create override values
+ override_trainset = [examples[2]] # Different from global trainset
+ override_valset = [examples[0]] # Different from global valset
+ override_teacher = SimpleModule("input -> output")
+
+ # Pass global values to compile, but override them in optimizer_compile_args
+ compile_args = {
+ "p": {
+ "trainset": override_trainset,
+ "valset": override_valset,
+ "teacher": override_teacher,
+ }
+ }
+
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ mock_eval.return_value = Mock(score=0.9)
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ optimizer.compile(
+ student,
+ trainset=trainset, # Global trainset (examples[:2])
+ valset=valset, # Global valset (examples[2])
+ teacher=None, # Global teacher (None)
+ strategy="p",
+ optimizer_compile_args=compile_args
+ )
+
+ # Verify the optimizer received the override values, not the global ones
+ assert mock_p.received_kwargs["trainset"] == override_trainset, \
+ "Optimizer should receive override trainset from compile_args"
+ assert mock_p.received_kwargs["valset"] == override_valset, \
+ "Optimizer should receive override valset from compile_args"
+ assert mock_p.received_kwargs["teacher"] is override_teacher, \
+ "Optimizer should receive override teacher from compile_args"
+
+ # Verify they're different from the global values
+ assert mock_p.received_kwargs["trainset"] != trainset, \
+ "Override trainset should differ from global trainset"
+ assert mock_p.received_kwargs["valset"] != valset, \
+ "Override valset should differ from global valset"
+
+
+def test_trainset_shuffling_between_steps():
+ """Test that trainset is shuffled between steps when shuffle_trainset_between_steps=True."""
+ from dspy.teleprompt.teleprompt import Teleprompter
+
+ student = SimpleModule("input -> output")
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+
+ # Create mock optimizers that capture the trainset they receive
+ trainsets_received = []
+
+ class TrainsetCapturingOptimizer(Teleprompter):
+ def compile(self, student, trainset=None, **kwargs):
+ trainsets_received.append(trainset)
+ return student
+
+ mock_p = TrainsetCapturingOptimizer()
+ mock_w = TrainsetCapturingOptimizer()
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p, w=mock_w)
+
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ mock_eval.return_value = Mock(score=0.8)
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ with patch.object(optimizer, "_models_changed", return_value=False):
+ optimizer.compile(
+ student,
+ trainset=trainset,
+ valset=valset,
+ strategy="p -> w",
+ shuffle_trainset_between_steps=True
+ )
+
+ # Verify trainset was shuffled between steps
+ assert len(trainsets_received) == 2, "Should have received trainset twice (for p and w)"
+ trainset_p = trainsets_received[0]
+ trainset_w = trainsets_received[1]
+
+ # Both should have same examples but potentially different order
+ assert len(trainset_p) == len(trainset_w), "Trainsets should have same length"
+ # With shuffling enabled and only 2 examples, there's a 50% chance they're in different order
+ # We can't reliably test order difference with small dataset, but we can verify they contain same examples
+ assert set(id(ex) for ex in trainset_p) == set(id(ex) for ex in trainset_w), \
+ "Trainsets should contain the same example objects"
+
+
+def test_strategy_execution_order():
+ """Test that strategy steps are executed in order and programs are passed correctly."""
+ from dspy.teleprompt.teleprompt import Teleprompter
+
+ student = SimpleModule("input -> output")
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+
+ # Track execution order and what program each optimizer receives
+ execution_log = []
+
+ class LoggingOptimizer(Teleprompter):
+ def __init__(self, name):
+ self.name = name
+
+ def compile(self, student, **kwargs):
+ # Create a new student with a marker to track the optimization path
+ optimized = SimpleModule("input -> output")
+ if not hasattr(student, "optimization_path"):
+ optimized.optimization_path = [self.name]
+ else:
+ optimized.optimization_path = student.optimization_path + [self.name]
+ execution_log.append((self.name, optimized.optimization_path.copy()))
+ return optimized
+
+ mock_p = LoggingOptimizer("p")
+ mock_w = LoggingOptimizer("w")
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p, w=mock_w)
+
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ mock_eval.return_value = Mock(score=0.85)
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ with patch.object(optimizer, "_models_changed", return_value=False):
+ result = optimizer.compile(
+ student,
+ trainset=trainset,
+ valset=valset,
+ strategy="p -> w -> p"
+ )
+
+ # Verify execution order
+ assert len(execution_log) == 3, "Should have executed 3 optimization steps"
+ assert execution_log[0] == ("p", ["p"]), "First step should be 'p'"
+ assert execution_log[1] == ("w", ["p", "w"]), "Second step should be 'w' receiving output from 'p'"
+ assert execution_log[2] == ("p", ["p", "w", "p"]), "Third step should be 'p' receiving output from 'w'"
+
+
+def test_lm_lifecycle_management():
+ """Test that launch_lms and kill_lms are called appropriately between steps."""
+ from dspy.teleprompt.teleprompt import Teleprompter
+
+ student = SimpleModule("input -> output")
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+
+ class SimpleOptimizer(Teleprompter):
+ def compile(self, student, **kwargs):
+ return student
+
+ mock_p = SimpleOptimizer()
+ mock_w = SimpleOptimizer()
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p, w=mock_w)
+
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ mock_eval.return_value = Mock(score=0.8)
+ with patch("dspy.teleprompt.bettertogether.launch_lms") as mock_launch:
+ with patch("dspy.teleprompt.bettertogether.kill_lms") as mock_kill:
+ with patch.object(optimizer, "_models_changed", return_value=True):
+ optimizer.compile(
+ student,
+ trainset=trainset,
+ valset=valset,
+ strategy="p -> w"
+ )
+
+ # Verify launch and kill were called
+ # When models change (which we mocked to return True), launch should be called
+ assert mock_launch.called, "launch_lms should be called when models change"
+ assert mock_kill.called, "kill_lms should be called when models change"
+
+
+def test_error_handling_returns_best_program():
+ """Test that if a step fails, the best program found so far is still returned."""
+ from dspy.teleprompt.teleprompt import Teleprompter
+
+ student = SimpleModule("input -> output")
+ lm = DummyLM([{"output": "test"}])
+ student.set_lm(lm)
+
+ # Create optimizers where the second one will fail
+ class SuccessfulOptimizer(Teleprompter):
+ def compile(self, student, **kwargs):
+ optimized = SimpleModule("input -> output")
+ optimized.step_name = "p_success"
+ return optimized
+
+ class FailingOptimizer(Teleprompter):
+ def compile(self, student, **kwargs):
+ raise RuntimeError("Intentional failure for testing")
+
+ mock_p = SuccessfulOptimizer()
+ mock_w = FailingOptimizer()
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p, w=mock_w)
+
+ # First call succeeds with score 0.7, second call (to failing optimizer) fails
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ mock_eval.side_effect = [
+ Mock(score=0.5), # Baseline
+ Mock(score=0.7), # After p (success)
+ ]
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ with patch.object(optimizer, "_models_changed", return_value=False):
+ result = optimizer.compile(
+ student,
+ trainset=trainset,
+ valset=valset,
+ strategy="p -> w"
+ )
+
+ # Verify a program was returned despite the failure
+ assert result is not None, "Should return a program even if a step fails"
+ assert hasattr(result, "flag_compilation_error_occurred"), "Should have error flag"
+ assert result.flag_compilation_error_occurred is True, "Error flag should be True"
+ assert hasattr(result, "candidate_programs"), "Should have candidate_programs"
+ assert len(result.candidate_programs) > 0, "Should have at least one candidate program"
+
+
[email protected]("test_valset,expected_marker,test_description", [
+ (valset, "p_optimized", "With valset: returns best score (p), not latest (w)"),
+ (None, "w_optimized", "Without valset: returns latest program (w)"),
+])
+def test_program_selection(student_with_lm, test_valset, expected_marker, test_description):
+ """Test program selection logic with and without validation set."""
+ mock_p = MarkedOptimizer("p_optimized")
+ mock_w = MarkedOptimizer("w_optimized")
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p, w=mock_w)
+
+ # Set up scores: baseline=0.5, p=0.9 (best), w=0.7
+ # When test_valset is provided, best score wins; when None, latest wins
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ if test_valset is not None:
+ mock_eval.side_effect = [
+ Mock(score=0.5), # Baseline
+ Mock(score=0.9), # After p (best score)
+ Mock(score=0.7), # After w (lower than p)
+ ]
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ with patch.object(optimizer, "_models_changed", return_value=False):
+ result = optimizer.compile(
+ student_with_lm,
+ trainset=trainset,
+ valset=test_valset,
+ strategy="p -> w"
+ )
+
+ # Verify the correct program was returned based on valset presence
+ assert hasattr(result, "marker"), "Result should have marker"
+ assert result.marker == expected_marker, test_description
+
+
+def test_candidate_programs_structure(student_with_lm):
+ """Test that candidate_programs has the correct structure and content."""
+ mock_p = MarkedOptimizer("p")
+ mock_w = MarkedOptimizer("w")
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p, w=mock_w)
+
+ # Set up scores: baseline=0.5, p=0.8, w=0.9 (best)
+ with patch("dspy.teleprompt.bettertogether.eval_candidate_program") as mock_eval:
+ mock_eval.side_effect = [
+ Mock(score=0.5), # Baseline
+ Mock(score=0.8), # After p
+ Mock(score=0.9), # After w (best)
+ ]
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ with patch.object(optimizer, "_models_changed", return_value=False):
+ result = optimizer.compile(
+ student_with_lm,
+ trainset=trainset,
+ valset=valset,
+ strategy="p -> w"
+ )
+
+ # Verify candidate_programs exists and has correct structure
+ assert hasattr(result, "candidate_programs"), "Result should have candidate_programs attribute"
+ candidates = result.candidate_programs
+
+ # Should have 3 candidates: baseline, p, w
+ assert len(candidates) == 3, f"Should have 3 candidates, got {len(candidates)}"
+
+ # Each candidate should have the required keys
+ for i, candidate in enumerate(candidates):
+ assert "score" in candidate, f"Candidate {i} missing 'score' key"
+ assert "program" in candidate, f"Candidate {i} missing 'program' key"
+ assert "strategy" in candidate, f"Candidate {i} missing 'strategy' key"
+ assert isinstance(candidate["score"], (int, float)), f"Candidate {i} score should be numeric"
+ assert isinstance(candidate["program"], dspy.Module), f"Candidate {i} program should be a Module"
+ assert isinstance(candidate["strategy"], (str, type(None))), f"Candidate {i} strategy should be str or None"
+
+ # Candidates should be sorted by score (best first)
+ scores = [c["score"] for c in candidates]
+ assert scores == sorted(scores, reverse=True), "Candidates should be sorted by score (descending)"
+
+ # Verify the best candidate is first
+ assert candidates[0]["score"] == 0.9, "Best candidate should have score 0.9"
+ assert candidates[0]["program"].marker == "w", "Best candidate should be from optimizer 'w'"
+
+ # Verify baseline candidate
+ baseline = [c for c in candidates if c["strategy"] is None or c["strategy"] == ""]
+ assert len(baseline) == 1, "Should have exactly one baseline candidate"
+ assert baseline[0]["score"] == 0.5, "Baseline should have score 0.5"
+
+
+def test_empty_valset_handling(student_with_lm):
+ """Test behavior when valset is an empty list vs None."""
+ # Test with empty list []
+ mock_p = MarkedOptimizer("optimized")
+ optimizer = BetterTogether(metric=simple_metric, p=mock_p)
+
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ with patch.object(optimizer, "_models_changed", return_value=False):
+ result = optimizer.compile(
+ student_with_lm,
+ trainset=trainset,
+ valset=[], # Empty list (not None)
+ strategy="p"
+ )
+
+ # With empty valset, should return latest program (same behavior as valset=None)
+ assert hasattr(result, "marker"), "Result should have marker"
+ assert result.marker == "optimized", "Should return the latest program when valset is empty list"
+ assert hasattr(result, "candidate_programs"), "Should have candidate_programs"
+
+ # Test with None - create fresh student and optimizer
+ student2 = SimpleModule("input -> output")
+ lm = DummyLM([{"output": "test"}])
+ student2.set_lm(lm)
+
+ mock_p2 = MarkedOptimizer("optimized")
+ optimizer2 = BetterTogether(metric=simple_metric, p=mock_p2)
+
+ with patch("dspy.teleprompt.bettertogether.launch_lms"):
+ with patch("dspy.teleprompt.bettertogether.kill_lms"):
+ with patch.object(optimizer2, "_models_changed", return_value=False):
+ result2 = optimizer2.compile(
+ student2,
+ trainset=trainset,
+ valset=None, # Explicit None
+ strategy="p"
+ )
+
+ # Both should behave the same way
+ assert hasattr(result2, "marker"), "Result2 should have marker"
+ assert result2.marker == "optimized", "Should return the latest program when valset is None"