Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

a few additional fixes to proposer, documentation, and formatting of mipro optimizer code

klopsahlong2d456c4a362a
diff --git a/docs/docs/deep-dive/optimizers/miprov2.md b/docs/docs/deep-dive/optimizers/miprov2.md
index 0d1ccd06b..759f003f1 100644
--- a/docs/docs/deep-dive/optimizers/miprov2.md
+++ b/docs/docs/deep-dive/optimizers/miprov2.md
@@ -218,7 +218,7 @@ At a high level, `MIPROv2` works by creating both few-shot examples and new inst
These steps are broken down in more detail below:
1) **Bootstrap Few-Shot Examples**: The same bootstrapping technique used in `BootstrapFewshotWithRandomSearch` is used to create few-shot examples. This works by randomly sampling examples from your training set, which are then run through your LM program. If the output from the program is correct for this example, it is kept as a valid few-shot example candidate. Otherwise, we try another example until we've curated the specified amount of few-shot example candidates. This step creates `num_candidates` sets of `max_bootstrapped_demos` bootstrapped examples and `max_labeled_demos` basic examples sampled from the training set.
2) **Propose Instruction Candidates**. Next, we propose instruction candidates for each predictor in the program. This is done using another LM program as a proposer, which bootstraps & summarizes relevant information about the task to generate high quality instructions. Specifically, the instruction proposer includes (1) a generated summary of properties of the training dataset, (2) a generated summary of your LM program's code and the specific predictor that an instruction is being generated for, (3) the previously bootstrapped few-shot examples to show reference inputs / outputs for a given predictor and (4) a randomly sampled tip for generation (i.e. "be creative", "be concise", etc.) to help explore the feature space of potential instructions.
-3. **Find an Optimized Combination of Few-Shot Examples & Instructions**. Finally, now that we've created these few-shot examples and instructions, we use Bayesian Optimization to choose which set of these would work best for each predictor in our program. This works by running a series of `num_batches` trials, where a new set of prompts are evaluated over our validation set at each trial. This helps the Bayesian Optimizer learn which combination of prompts work best over time. If `minibatch` is set to `True` (which it is by default), then the new set of prompts are only evaluated on a minibatch of size `minibatch_size` at each trial which generally allows for more efficient exploration / exploitation. The best averaging set of prompts is then evalauted on the full validation set every `minibatch_full_eval_steps` get a less noisey performance benchmark. At the end of the optimization process, the LM program with the set of prompts that performed best on the full validation set is returned.
+3. **Find an Optimized Combination of Few-Shot Examples & Instructions**. Finally, now that we've created these few-shot examples and instructions, we use Bayesian Optimization to choose which set of these would work best for each predictor in our program. This works by running a series of `num_trials` trials, where a new set of prompts are evaluated over our validation set at each trial. This helps the Bayesian Optimizer learn which combination of prompts work best over time. If `minibatch` is set to `True` (which it is by default), then the new set of prompts are only evaluated on a minibatch of size `minibatch_size` at each trial which generally allows for more efficient exploration / exploitation. The best averaging set of prompts is then evalauted on the full validation set every `minibatch_full_eval_steps` get a less noisey performance benchmark. At the end of the optimization process, the LM program with the set of prompts that performed best on the full validation set is returned.
For those interested in more details, more information on `MIPROv2` along with a study on `MIPROv2` compared with other DSPy optimizers can be found in [this paper](https://arxiv.org/abs/2406.11695).
\ No newline at end of file
diff --git a/dspy/propose/grounded_proposer.py b/dspy/propose/grounded_proposer.py
index 740197f6f..584ed5ee2 100644
--- a/dspy/propose/grounded_proposer.py
+++ b/dspy/propose/grounded_proposer.py
@@ -187,6 +187,7 @@ class GenerateModuleInstruction(dspy.Module):
# Summarize the program
program_description = "Not available"
module_code = "Not provided"
+ module_description = "Not provided"
if self.program_aware:
try:
program_description = strip_prefix(
@@ -209,18 +210,18 @@ class GenerateModuleInstruction(dspy.Module):
outputs.append(field_name)
module_code = f"{program.predictors()[pred_i].__class__.__name__}({', '.join(inputs)}) -> {', '.join(outputs)}"
+
+ module_description = self.describe_module(
+ program_code=self.program_code_string,
+ program_description=program_description,
+ program_example=task_demos,
+ module=module_code,
+ max_depth=10,
+ ).module_description
except:
if self.verbose: print("Error getting program description. Running without program aware proposer.")
self.program_aware = False
- module_description = self.describe_module(
- program_code=self.program_code_string,
- program_description=program_description,
- program_example=task_demos,
- module=module_code,
- max_depth=10,
- ).module_description
-
# Generate an instruction for our chosen module
if self.verbose: print(f"task_demos {task_demos}")
instruct = self.generate_module_instruction(
diff --git a/dspy/teleprompt/mipro_optimizer_v2.py b/dspy/teleprompt/mipro_optimizer_v2.py
index 9fdbe6a1d..e0c90e88f 100644
--- a/dspy/teleprompt/mipro_optimizer_v2.py
+++ b/dspy/teleprompt/mipro_optimizer_v2.py
@@ -123,7 +123,9 @@ class MIPROv2(Teleprompter):
trainset, valset = self._set_and_validate_datasets(trainset, valset)
# Set hyperparameters based on run mode (if set)
- zeroshot_opt = (self.max_bootstrapped_demos == 0) and (self.max_labeled_demos == 0)
+ zeroshot_opt = (self.max_bootstrapped_demos == 0) and (
+ self.max_labeled_demos == 0
+ )
num_trials, valset, minibatch = self._set_hyperparams_from_run_mode(
student, num_trials, minibatch, zeroshot_opt, valset
)
@@ -162,15 +164,18 @@ class MIPROv2(Teleprompter):
)
# Step 1: Bootstrap few-shot examples
- demo_candidates = self._bootstrap_fewshot_examples(
- program, trainset, seed
- )
+ demo_candidates = self._bootstrap_fewshot_examples(program, trainset, seed)
# Step 2: Propose instruction candidates
instruction_candidates = self._propose_instructions(
- program, trainset, demo_candidates, view_data_batch_size,
- program_aware_proposer, data_aware_proposer, tip_aware_proposer,
- fewshot_aware_proposer
+ program,
+ trainset,
+ demo_candidates,
+ view_data_batch_size,
+ program_aware_proposer,
+ data_aware_proposer,
+ tip_aware_proposer,
+ fewshot_aware_proposer,
)
# If zero-shot, discard demos
@@ -192,9 +197,14 @@ class MIPROv2(Teleprompter):
)
return best_program
-
+
def _set_hyperparams_from_run_mode(
- self, program: Any, num_trials: int, minibatch: bool, zeroshot_opt: bool, valset: List
+ self,
+ program: Any,
+ num_trials: int,
+ minibatch: bool,
+ zeroshot_opt: bool,
+ valset: List,
) -> Tuple[int, List, bool]:
if self.auto is None:
return num_trials, valset, minibatch
@@ -208,9 +218,7 @@ class MIPROv2(Teleprompter):
valset = create_minibatch(valset, batch_size=auto_settings["val_size"])
minibatch = len(valset) > MIN_MINIBATCH_SIZE
self.num_candidates = int(
- np.round(
- np.min([num_trials * num_vars, (1.5 * num_trials) / num_vars])
- )
+ np.round(np.min([num_trials * num_vars, (1.5 * num_trials) / num_vars]))
)
return num_trials, valset, minibatch
@@ -231,7 +239,7 @@ class MIPROv2(Teleprompter):
else:
if len(valset) < 1:
raise ValueError("Validation set must have at least 1 example.")
-
+
return trainset, valset
def _print_auto_run_settings(self, num_trials: int, minibatch: bool, valset: List):
@@ -259,7 +267,9 @@ class MIPROv2(Teleprompter):
estimated_prompt_model_calls = (
10 # Data summarizer calls
+ self.num_candidates * num_predictors # Candidate generation
- + (num_predictors + 1 if program_aware_proposer else 0) # Program-aware proposer
+ + (
+ num_predictors + 1 if program_aware_proposer else 0
+ ) # Program-aware proposer
)
prompt_model_line = (
f"{YELLOW}- Prompt Generation: {BLUE}{BOLD}10{ENDC}{YELLOW} data summarizer calls + "
@@ -303,7 +313,13 @@ class MIPROv2(Teleprompter):
program_aware_proposer: bool,
) -> bool:
prompt_model_line, task_model_line = self._estimate_lm_calls(
- program, num_trials, minibatch, minibatch_size, minibatch_full_eval_steps, valset, program_aware_proposer
+ program,
+ num_trials,
+ minibatch,
+ minibatch_size,
+ minibatch_full_eval_steps,
+ valset,
+ program_aware_proposer,
)
user_message = textwrap.dedent(
@@ -358,14 +374,22 @@ class MIPROv2(Teleprompter):
print(f"Bootstrapping N={self.num_candidates} sets of demonstrations...")
zeroshot = self.max_bootstrapped_demos == 0 and self.max_labeled_demos == 0
-
+
try:
demo_candidates = create_n_fewshot_demo_sets(
student=program,
num_candidate_sets=self.num_candidates,
trainset=trainset,
- max_labeled_demos= LABELED_FEWSHOT_EXAMPLES_IN_CONTEXT if zeroshot else self.max_labeled_demos,
- max_bootstrapped_demos=BOOTSTRAPPED_FEWSHOT_EXAMPLES_IN_CONTEXT if zeroshot else self.max_bootstrapped_demos,
+ max_labeled_demos=(
+ LABELED_FEWSHOT_EXAMPLES_IN_CONTEXT
+ if zeroshot
+ else self.max_labeled_demos
+ ),
+ max_bootstrapped_demos=(
+ BOOTSTRAPPED_FEWSHOT_EXAMPLES_IN_CONTEXT
+ if zeroshot
+ else self.max_bootstrapped_demos
+ ),
metric=self.metric,
max_errors=self.max_errors,
teacher_settings=self.teacher_settings,
@@ -451,9 +475,9 @@ class MIPROv2(Teleprompter):
best_program = program.deepcopy()
trial_logs = {}
total_eval_calls = 0
- if minibatch:
+ if minibatch:
scores = []
- else:
+ else:
scores = [default_score]
full_eval_scores = [default_score]
param_score_dict = defaultdict(list)
@@ -476,7 +500,12 @@ class MIPROv2(Teleprompter):
# Choose instructions and demos, insert them into the program
chosen_params = self._select_and_insert_instructions_and_demos(
- candidate_program, instruction_candidates, demo_candidates, trial, trial_logs, trial_num
+ candidate_program,
+ instruction_candidates,
+ demo_candidates,
+ trial,
+ trial_logs,
+ trial_num,
)
# Log assembled program
@@ -503,10 +532,21 @@ class MIPROv2(Teleprompter):
# Log evaluation results
scores.append(score)
- if minibatch:
- self._log_minibatch_eval(score, best_score, batch_size, chosen_params, scores, full_eval_scores, trial, num_trials)
- else:
- self._log_normal_eval(score, best_score, chosen_params, scores, trial, num_trials)
+ if minibatch:
+ self._log_minibatch_eval(
+ score,
+ best_score,
+ batch_size,
+ chosen_params,
+ scores,
+ full_eval_scores,
+ trial,
+ num_trials,
+ )
+ else:
+ self._log_normal_eval(
+ score, best_score, chosen_params, scores, trial, num_trials
+ )
categorical_key = ",".join(map(str, chosen_params))
param_score_dict[categorical_key].append(
(score, candidate_program),
@@ -520,7 +560,8 @@ class MIPROv2(Teleprompter):
# If minibatch, perform full evaluation at intervals
if minibatch and (
- (trial_num % minibatch_full_eval_steps == 0) or (trial_num == num_trials)
+ (trial_num % minibatch_full_eval_steps == 0)
+ or (trial_num == num_trials)
):
best_score, best_program = self._perform_full_evaluation(
trial_num,
@@ -558,27 +599,35 @@ class MIPROv2(Teleprompter):
print(f"Returning best identified program with score {best_score}!")
return best_program
-
- def _log_minibatch_eval(self, score, best_score, batch_size, chosen_params, scores, full_eval_scores, trial, num_trials):
+
+ def _log_minibatch_eval(
+ self,
+ score,
+ best_score,
+ batch_size,
+ chosen_params,
+ scores,
+ full_eval_scores,
+ trial,
+ num_trials,
+ ):
print(
f"Score: {score} on minibatch of size {batch_size} with parameters {chosen_params}."
)
- print(
- f"Minibatch scores so far: {'['+', '.join([f'{s}' for s in scores])+']'}"
- )
- trajectory = "["+", ".join([f"{s}" for s in full_eval_scores])+"]"
+ print(f"Minibatch scores so far: {'['+', '.join([f'{s}' for s in scores])+']'}")
+ trajectory = "[" + ", ".join([f"{s}" for s in full_eval_scores]) + "]"
print(f"Full eval scores so far: {trajectory}")
print(f"Best full score so far: {best_score}")
- print(f'{"="*len(f"== Minibatch Trial {trial.number+1} / {num_trials} ==")}\n\n')
-
- def _log_normal_eval(self, score, best_score, chosen_params, scores, trial, num_trials):
- print(f"Score: {score} with parameters {chosen_params}.")
print(
- f"Scores so far: {'['+', '.join([f'{s}' for s in scores])+']'}"
- )
- print(
- f"Best score so far: {best_score}"
+ f'{"="*len(f"== Minibatch Trial {trial.number+1} / {num_trials} ==")}\n\n'
)
+
+ def _log_normal_eval(
+ self, score, best_score, chosen_params, scores, trial, num_trials
+ ):
+ print(f"Score: {score} with parameters {chosen_params}.")
+ print(f"Scores so far: {'['+', '.join([f'{s}' for s in scores])+']'}")
+ print(f"Best score so far: {best_score}")
print(f'{"="*len(f"===== Trial {trial.number+1} / {num_trials} =====")}\n\n')
def _select_and_insert_instructions_and_demos(
@@ -598,7 +647,9 @@ class MIPROv2(Teleprompter):
f"{i}_predictor_instruction", range(len(instruction_candidates[i]))
)
selected_instruction = instruction_candidates[i][instruction_idx]
- updated_signature = get_signature(predictor).with_instructions(selected_instruction)
+ updated_signature = get_signature(predictor).with_instructions(
+ selected_instruction
+ )
set_signature(predictor, updated_signature)
trial_logs[trial_num][f"{i}_predictor_instruction"] = instruction_idx
chosen_params.append(f"Predictor {i+1}: Instruction {instruction_idx}")
@@ -630,8 +681,10 @@ class MIPROv2(Teleprompter):
print(f"===== Full Eval {len(fully_evaled_param_combos)+1} =====")
# Identify best program to evaluate fully
- highest_mean_program, mean_score, combo_key = get_program_with_highest_avg_score(
- param_score_dict, fully_evaled_param_combos
+ highest_mean_program, mean_score, combo_key = (
+ get_program_with_highest_avg_score(
+ param_score_dict, fully_evaled_param_combos
+ )
)
print(
f"Doing full eval on next top averaging program (Avg Score: {mean_score}) from minibatch trials..."
@@ -662,10 +715,10 @@ class MIPROv2(Teleprompter):
print(f"{GREEN}New best full eval score!{ENDC} Score: {full_eval_score}")
best_score = full_eval_score
best_program = highest_mean_program.deepcopy()
- trajectory = "["+", ".join([f"{s}" for s in full_eval_scores])+"]"
+ trajectory = "[" + ", ".join([f"{s}" for s in full_eval_scores]) + "]"
print(f"Full eval scores so far: {trajectory}")
print(f"Best full score so far: {best_score}")
- print(len(f"===== Full Eval {len(fully_evaled_param_combos)+1} =====")*'=')
+ print(len(f"===== Full Eval {len(fully_evaled_param_combos)+1} =====") * "=")
print("\n")
return best_score, best_program