DSPy: The framework for programming—not prompting—language models
Python
fixing eval set use and associated documentation
klopsahlonga0d9d83dcbac
diff --git a/docs/docs/deep-dive/optimizers/miprov2.md b/docs/docs/deep-dive/optimizers/miprov2.mdindex b2e8b03bc..1ef388ea1 100644--- a/docs/docs/deep-dive/optimizers/miprov2.md+++ b/docs/docs/deep-dive/optimizers/miprov2.md@@ -69,6 +69,7 @@ evaluate(program, devset=devset[:])Now we have the baseline pipeline ready to use, so let's try using the `MIPROv2` optimizer to improve our pipeline's performance!### Optimizing with `MIPROv2`+Here we show how to import, initialize, and compile our program with optimized few-shot examples and instructions using `MIPROv2`.```python# Import the optimizer@@ -107,6 +108,7 @@ evaluate(optimized_program, devset=devset[:])### Optimizing instructions only with `MIPROv2` (0-Shot)+In some cases, we may want to only optimize the instruction, rather than including few-shot examples in the prompt. The code below demonstrates how this can be done using `MIPROv2`. Note that the key difference involves setting `max_labeled_demos` and `max_bootstrapped_demos` to zero.```python# Import optimizerfrom dspy.teleprompt import MIPROv2@@ -114,7 +116,7 @@ from dspy.teleprompt import MIPROv2# Initialize optimizerteleprompter = MIPROv2(metric=gsm8k_metric,- num_candidates=15,+ num_candidates=7,init_temperature=0.5,verbose=False,num_threads=4,@@ -128,6 +130,8 @@ zeroshot_optimized_program = teleprompter.compile(max_bootstrapped_demos=0, # setting demos to 0 for 0-shot optimizationmax_labeled_demos=0,num_trials=15,+ minibatch_size=25,+ minibatch_full_eval_steps=10,minibatch=False,requires_permission_to_run=False,)@@ -143,32 +147,40 @@ evaluate(zeroshot_optimized_program, devset=devset[:])### Initialization Parameters-- **prompt_model**: Model used for prompt generation. Defaults to the model configured in `dspy.settings`.-- **task_model**: Model used for task execution. Defaults to the model configured in `dspy.settings`.-- **metric**: The evaluation metric used to optimize the task model.-- **num_candidates**: Number of candidate prompts to generate and evaluate. Default is 10.-- **num_threads**: Threads to use for evaluation.-- **max_errors**: Maximum errors during an evaluation run that can be made before throwing an Exception.-- **teacher_settings**: Settings to use for the teacher model that bootstraps few-shot examples.-- **init_temperature**: The initial temperature for prompt generation, influencing creativity. Default is 1.0.-- **verbose**: Enables printing intermediate steps and information.-- **track_stats**: Tracks optimization process statistics if set to True.-- **log_dir**: Directory for saving logs and output data.-- **metric_threshold**: A metric threshold is used if we only want to keep bootstrapped few-shot exampels that exceed some threshold of performance. Defaults to None.--### Compile Method Specific Parameters--- **student**: The base program to optimize.-- **trainset**: Training dataset for model training.-- **num_batches**: Number of optimization trials to run. Default is 30. Note that if you are using mini-batching, you might want to increase **num_batches**. If not, you might want to lower this.-- **max_bootstrapped_demos**: Maximum number of bootstrapped demonstrations used for few-shot learning. Default is 4.-- **max_labeled_demos**: Maximum number of labeled demonstrations used for few-shot learning. Default is 16.-- **seed**: Seed for random number generator for reproducibility. Default is 9.-- **minibatch**: Flag to enable minibatching during evaluation. Default is True.-- **minibatch_size**: Size of minibatches for evaluations. Default is 25.0.-- **minibatch_full_eval_steps**: Number of minibatch steps before performing a full evaluation.-- **program_aware_proposer**: Flag to enable summarizing a reflexive view of the code for your LM program . Default is True.-- **requires_permission_to_run**: Flag to require user confirmation before running optimizations. Default is True.+| Parameter | Type | Default | Description |+|----------------------|--------------|------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|+| `metric` | `dspy.metric` | N/A - Required | The evaluation metric used to optimize the task model. |+| `prompt_model` | `dspy.LM` | LM specified in `dspy.settings` | Model used for prompt generation. |+| `task_model` | `dspy.LM` | LM specified in `dspy.settings` | Model used for task execution. |+| `num_candidates` | `int` | `10` | Number of candidate instructions & few-shot examples to generate and evaluate for each predictor. If `num_candidates=10`, this means for a 2 module LM program we'll be optimizing over 10 candidates x 2 modules x 2 variables (few-shot ex. and instructions for each module)= 40 total variables. Therfore, if we increase `num_candidates`, we will probably want to increase `num_trials` as well (see Compile parameters). |+| `num_threads` | `int` | `6` | Threads to use for evaluation. |+| `max_errors` | `int` | `10` | Maximum errors during an evaluation run that can be made before throwing an Exception. |+| `teacher_settings` | `dict` | `{}` | Settings to use for the teacher model that bootstraps few-shot examples. An example dict would be `{lm=<dspy.LM object>}`. If your LM program with your default model is struggling to bootstrap any examples, it could be worth using a more powerful teacher model for bootstrapping. |+| `init_temperature` | `float` | `1.0` | The initial temperature for prompt generation, influencing creativity. |+| `verbose` | `bool` | `False` | Enables printing intermediate steps and information. |+| `track_stats` | `bool` | `True` | Logs relevant information through the optimization process if set to True. |+| `metric_threshold` | `float` | `None` | A metric threshold is used if we only want to keep bootstrapped few-shot examples that exceed some threshold of performance. |++### Compile Parameters++| Parameter | Type | Default | Description |+|----------------------------|----------|---------|----------------------------------------------------------------------------------------------------------|+| `student` | `dspy.Module` | N/A (Required) | The base program to optimize. |+| `trainset` | `List[dspy.Example]` | N/A (Required) | Training dataset which is used to bootstrap few-shot examples and instructions. If a separate `valset` is not specified, 80% of this training set will also be used as a validation set for evaluating new candidate prompts. |+| `valset` | `List[dspy.Example]` | Defaults to 80% of trainset | Dataset which is used to evaluate candidate prompts. We recommend using somewhere between 50-500 examples for optimization. |+| `num_trials` | `int` | `30` | Number of optimization trials to run. When `minibatch` is set to `True`, this represents the number of minibatch trials that will be run on batches of size `minibatch_size`. When minibatch is set to `False`, each trial uses a full evaluation on the training set. In both cases, we recommend setting `num_trials` to a *minimum* of .75 x # modules in program x # variables per module (2 if few-shot examples & instructions will both be optimized, 1 in the 0-shot case). |+| `minibatch` | `bool` | `True` | Flag to enable evaluating over minibatches of data (instead of the full validation set) for evaluation each trial. |+| `minibatch_size` | `int` | `25.0` | Size of minibatches for evaluations. |+| `minibatch_full_eval_steps` | `int` | `10` | When minibatching is enabled, a full evaluation on the validation set will be carried out every `minibatch_full_eval_steps` on the top averaging set of prompts (according to their average score on the minibatch trials).+| `max_bootstrapped_demos` | `int` | `4` | Maximum number of bootstrapped demonstrations to generate and include in the prompt. |+| `max_labeled_demos` | `int` | `16` | Maximum number of labeled demonstrations to generate and include in the prompt. Note that these differ from bootstrapped examples because they are just inputs & outputs sampled directly from the training set and do not have bootstrapped intermediate steps. |+| `seed` | `int` | `9` | Seed for reproducibility. | |+| `program_aware_proposer` | `bool` | `True` | Flag to enable summarizing a reflexive view of the code for your LM program. |+| `data_aware_proposer` | `bool` | `True` | Flag to enable summarizing your training dataset. |+| `view_data_batch_size` | `int` | `10` | Number of data examples to look at a time when generating the summary. |+| `tip_aware_proposer` | `bool` | `True` | Flag to enable using a randomly selected tip for instruction generation. |+| `fewshot_aware_proposer` | `bool` | `True` | Flag to enable using generated few-shot examples for instruction proposal. |+| `requires_permission_to_run` | `bool` | `True` | Flag to require user confirmation before running optimizations. |## How `MIPROv2` works@@ -177,7 +189,7 @@ At a high level, `MIPROv2` works by creating both few-shot examples and new instThese steps are broken down in more detail below:1) **Bootstrap Few-Shot Examples**: The same bootstrapping technique used in `BootstrapFewshotWithRandomSearch` is used to create few-shot examples. This works by randomly sampling examples from your training set, which are then run through your LM program. If the output from the program is correct for this example, it is kept as a valid few-shot example candidate. Otherwise, we try another example until we've curated the specified amount of few-shot example candidates. This step creates `num_candidates` sets of `max_bootstrapped_demos` bootstrapped examples and `max_labeled_demos` basic examples sampled from the training set.2) **Propose Instruction Candidates**. Next, we propose instruction candidates for each predictor in the program. This is done using another LM program as a proposer, which bootstraps & summarizes relevant information about the task to generate high quality instructions. Specifically, the instruction proposer includes (1) a generated summary of properties of the training dataset, (2) a generated summary of your LM program's code and the specific predictor that an instruction is being generated for, (3) the previously bootstrapped few-shot examples to show reference inputs / outputs for a given predictor and (4) a randomly sampled tip for generation (i.e. "be creative", "be concise", etc.) to help explore the feature space of potential instructions.-3. **Find an Optimized Combination of Few-Shot Examples & Instructions**. Finally, now that we've created these few-shot examples and instructions, we use Bayesian Optimization to choose which set of these would work best for each predictor in our program. This works by running a series of `num_batches` trials, where a new set of prompts are evaluated over our training set at each trial. This helps the Bayesian Optimizer learn which combination of prompts work best over time. If `minibatch` is set to `True` (which it is by default), then the new set of prompts are only evaluated on a minibatch of size `minibatch_size` at each trial which generally allows for more efficient exploration / exploitation. The best averaging set of prompts is then evalauted on the full training set every `minibatch_full_eval_steps` get a less noisey performance benchmark. At the end of the optimization process, the LM program with the set of prompts that performed best on the full training set is returned.+3. **Find an Optimized Combination of Few-Shot Examples & Instructions**. Finally, now that we've created these few-shot examples and instructions, we use Bayesian Optimization to choose which set of these would work best for each predictor in our program. This works by running a series of `num_batches` trials, where a new set of prompts are evaluated over our validation set at each trial. This helps the Bayesian Optimizer learn which combination of prompts work best over time. If `minibatch` is set to `True` (which it is by default), then the new set of prompts are only evaluated on a minibatch of size `minibatch_size` at each trial which generally allows for more efficient exploration / exploitation. The best averaging set of prompts is then evalauted on the full validation set every `minibatch_full_eval_steps` get a less noisey performance benchmark. At the end of the optimization process, the LM program with the set of prompts that performed best on the full validation set is returned.For those interested in more details, more information on `MIPROv2` along with a study on `MIPROv2` compared with other DSPy optimizers can be found in [this paper](https://arxiv.org/abs/2406.11695).\ No newline at end of filediff --git a/dspy/teleprompt/mipro_optimizer_v2.py b/dspy/teleprompt/mipro_optimizer_v2.pyindex dd040f64c..f92460304 100644--- a/dspy/teleprompt/mipro_optimizer_v2.py+++ b/dspy/teleprompt/mipro_optimizer_v2.py@@ -23,14 +23,8 @@ from dspy.teleprompt.utils import (save_candidate_program,save_file_to_log_dir,set_signature,- # setup_logging,)-try:- import wandb-except ImportError:- wandb = None-"""USAGE SUGGESTIONS:@@ -42,7 +36,7 @@ from dspy.teleprompt import MIPROv2teleprompter = MIPROv2(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=10, init_temperature=1.0)kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=0)compiled_prompt_opt = teleprompter.compile(program, trainset=trainset[:TRAIN_NUM], num_trials=100, max_bootstrapped_demos=3, max_labeled_demos=5)-eval_score = evaluate(compiled_prompt_opt, devset=evalset[:EVAL_NUM], **kwargs)+eval_score = evaluate(compiled_prompt_opt, devset=valset[:EVAL_NUM], **kwargs)```Note that this teleprompter takes in the following parameters:@@ -115,11 +109,14 @@ class MIPROv2(Teleprompter):max_labeled_demos=16,seed=9,minibatch=True,- program_aware_proposer=True,- requires_permission_to_run=True,- view_data_batch_size=10,minibatch_size=25,minibatch_full_eval_steps=10,+ program_aware_proposer=True,+ data_aware_proposer=True,+ view_data_batch_size=10,+ tip_aware_proposer=True,+ fewshot_aware_proposer=True,+ requires_permission_to_run=True,):# Define ANSI escape codes for colorsYELLOW = "\033[93m"@@ -138,7 +135,7 @@ class MIPROv2(Teleprompter):if len(trainset) < 2:raise ValueError("Trainset must have at least 2 examples if no valset specified, or at least 1 example with external validation set.")- valset_size = min(300, max(1, int(len(trainset) * 0.25))) # 25% of trainset, capped at 300+ valset_size = min(500, max(1, int(len(trainset) * 0.80))) # 80% of trainset, capped at 300cutoff = len(trainset) - valset_sizevalset = trainset[cutoff:]trainset = trainset[:cutoff]@@ -222,7 +219,7 @@ class MIPROv2(Teleprompter):# Set up program and evaluation functionprogram = student.deepcopy()evaluate = Evaluate(- devset=trainset,+ devset=valset,metric=self.metric,num_threads=self.num_threads,max_errors=self.max_errors,@@ -267,8 +264,8 @@ class MIPROv2(Teleprompter):# Generate N candidate instructions# Setup our proposer- print(f"\n==> STEP 2: PROPOSE INSTRUCTION CANDIDATES <==")- print("In this step, by default we will use the few-shot examples from the previous step, a generated dataset summary, and a summary of the program code, and a randomly selected prompting tip to propose instructions.")+ print("\n==> STEP 2: PROPOSE INSTRUCTION CANDIDATES <==")+ print("In this step, by default we will use the few-shot examples from the previous step, a generated dataset summary, a summary of the program code, and a randomly selected prompting tip to propose instructions.")proposer = GroundedProposer(program=program,@@ -277,12 +274,16 @@ class MIPROv2(Teleprompter):# program_code_string=self.program_code_string,view_data_batch_size=view_data_batch_size,program_aware=program_aware_proposer,+ use_dataset_summary=data_aware_proposer,+ use_task_demos=fewshot_aware_proposer,+ use_tip=tip_aware_proposer,+ set_tip_randomly=tip_aware_proposer,use_instruct_history=False,set_history_randomly=False,verbose = self.verbose,)- print(f"\nProposing instructions...\n")+ print("\nProposing instructions...\n")instruction_candidates = proposer.propose_instructions_for_program(trainset=trainset,program=program,@@ -303,18 +304,16 @@ class MIPROv2(Teleprompter):demo_candidates = None# Initialize variables to track during the optimization process- best_score = float("-inf")best_scoring_trial = 0- best_program = Nonetrial_logs = {}total_eval_calls = 0param_score_dict = defaultdict(list) # Dictionaries of paramater combinations we've tried, and their associated scoresfully_evaled_param_combos = {} # List of the parameter combinations we've done full evals of# Evaluate the default program- if self.verbose: print("Evaluating the default program...\n")+ print("Evaluating the default program...\n")default_score = eval_candidate_program(len(valset), valset, program, evaluate)- if self.verbose: print(f"Default program score: {default_score}\n")+ print(f"Default program score: {default_score}\n")best_score = default_scorebest_program = program.deepcopy()@@ -326,7 +325,7 @@ class MIPROv2(Teleprompter):instruction_candidates,demo_candidates,evaluate,- trainset,+ valset,):def objective(trial):nonlocal best_program, best_score, best_scoring_trial, trial_logs, total_eval_calls # Allow access to the outer variables@@ -397,11 +396,10 @@ class MIPROv2(Teleprompter):trial_logs[trial.number+1]["num_eval_calls"] = 0# Evaluate the candidate program with relevant batch size- evalset = trainset if minibatch else valset- batch_size = minibatch_size if minibatch else len(evalset)+ batch_size = minibatch_size if minibatch else len(valset)score = eval_candidate_program(- batch_size, evalset, candidate_program,+ batch_size, valset, candidate_program,evaluate,)@@ -409,7 +407,7 @@ class MIPROv2(Teleprompter):if self.verbose:print("Full trace of prompts in use on an example...")get_task_model_history_for_full_example(- candidate_program, self.task_model, trainset, evaluate,+ candidate_program, self.task_model, valset, evaluate,)# Log relevant information@@ -418,7 +416,7 @@ class MIPROv2(Teleprompter):(score, candidate_program),)trial_logs[trial.number+1]["num_eval_calls"] = batch_size- trial_logs[trial.number+1]["full_eval"] = batch_size >= len(trainset)+ trial_logs[trial.number+1]["full_eval"] = batch_size >= len(valset)trial_logs[trial.number+1]["score"] = scoretrial_logs[trial.number+1]["pruned"] = Falsetotal_eval_calls += trial_logs[trial.number+1]["num_eval_calls"]@@ -464,7 +462,7 @@ class MIPROv2(Teleprompter):# Log relevant informationfully_evaled_param_combos[combo_key] = {"program":highest_mean_program, "score": full_val_score}- total_eval_calls += len(trainset)+ total_eval_calls += len(valset)trial_logs[trial.number+1]["total_eval_calls_so_far"] = total_eval_callstrial_logs[trial.number+1]["full_eval"] = Truetrial_logs[trial.number+1]["program_path"] = save_candidate_program(@@ -489,10 +487,10 @@ class MIPROv2(Teleprompter):# Run the trialoptuna.logging.set_verbosity(optuna.logging.WARNING)- print(f"\n==> STEP 3: FINDING OPTIMAL PROMPT PARAMETERS <==")+ print(f"==> STEP 3: FINDING OPTIMAL PROMPT PARAMETERS <==")print("In this step, we will evaluate the program over a series of trials with different combinations of instructions and few-shot examples to find the optimal combination. Bayesian Optimization will be used for this search process.\n")objective_function = create_objective(- program, instruction_candidates, demo_candidates, evaluate, trainset,+ program, instruction_candidates, demo_candidates, evaluate, valset,)sampler = optuna.samplers.TPESampler(seed=seed, multivariate=True)study = optuna.create_study(direction="maximize", sampler=sampler)