Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

adding in / updating documentation for MIPRO

klopsahlong6815334bd679
diff --git a/docs/docs/building-blocks/6-optimizers.md b/docs/docs/building-blocks/6-optimizers.md
index 3f58f2063..e2c0436ed 100644
--- a/docs/docs/building-blocks/6-optimizers.md
+++ b/docs/docs/building-blocks/6-optimizers.md
@@ -42,7 +42,7 @@ In many cases, we found that compiling leads to better prompts than human writin
All of these can be accessed via `from dspy.teleprompt import *`.
-#### Automatic Few-Shot Learning
+### Automatic Few-Shot Learning
These optimizers extend the signature by automatically generating and including **optimized** examples within the prompt sent to the model, implementing few-shot learning.
@@ -50,52 +50,46 @@ These optimizers extend the signature by automatically generating and including
2. **`BootstrapFewShot`**: Uses a `teacher` module (which defaults to your program) to generate complete demonstrations for every stage of your program, along with labeled examples in `trainset`. Parameters include `max_labeled_demos` (the number of demonstrations randomly selected from the `trainset`) and `max_bootstrapped_demos` (the number of additional examples generated by the `teacher`). The bootstrapping process employs the metric to validate demonstrations, including only those that pass the metric in the "compiled" prompt. Advanced: Supports using a `teacher` program that is a *different* DSPy program that has compatible structure, for harder tasks.
-3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program over the optimization. Parameters mirror those of `BootstrapFewShot`, with the addition of `num_candidate_programs`, which specifies the number of random programs evaluated over the optimization, including candidates of the uncompiled program, `LabeledFewShot` optimized program, `BootstrapFewShot` compiled program with unshuffled examples and `num_candidate_programs` of `BootstrapFewShot` compiled programs with randomized example sets.
+3. [**`BootstrapFewShotWithRandomSearch`**](../deep-dive/optimizers/bootstrap-fewshot.md): Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program over the optimization. Parameters mirror those of `BootstrapFewShot`, with the addition of `num_candidate_programs`, which specifies the number of random programs evaluated over the optimization, including candidates of the uncompiled program, `LabeledFewShot` optimized program, `BootstrapFewShot` compiled program with unshuffled examples and `num_candidate_programs` of `BootstrapFewShot` compiled programs with randomized example sets.
-4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` with Optuna optimization across demonstration sets, running trials to maximize evaluation metrics and selecting the best demonstrations.
+4. **`KNNFewShot`**. Uses k-Nearest Neighbors algorithm to find the nearest training example demonstrations for a given input example. These nearest neighbor demonstrations are then used as the trainset for the BootstrapFewShot optimization process. See [this notebook](https://github.com/stanfordnlp/dspy/blob/main/examples/knn.ipynb) for an example.
-5. **`KNNFewShot`**. Uses k-Nearest Neighbors algorithm to find the nearest training example demonstrations for a given input example. These nearest neighbor demonstrations are then used as the trainset for the BootstrapFewShot optimization process. See [this notebook](https://github.com/stanfordnlp/dspy/blob/main/examples/knn.ipynb) for an example.
+### Automatic Instruction Optimization
-#### Automatic Instruction Optimization
+These optimizers produce optimal instructions for the prompt and, in the case of MIPROv2 can also optimize the set of few-shot demonstrations.
-These optimizers produce optimal instructions for the prompt and, in the case of MIPRO also optimize the set of few-shot demonstrations.
+5. [**`COPRO`**](../deep-dive/optimizers/copro.mdx): Generates and refines new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the `trainset`). Parameters include `depth` which is the number of iterations of prompt improvement the optimizer runs over.
-6. **`COPRO`**: Generates and refines new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the `trainset`). Parameters include `depth` which is the number of iterations of prompt improvement the optimizer runs over.
+6. [**`MIPROv2`**](../deep-dive/optimizers/miprov2.md): Generates instructions *and* few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.
-7. **`MIPRO`**: Generates instructions *and* few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.
-
-#### Automatic Finetuning
+### Automatic Finetuning
This optimizer is used to fine-tune the underlying LLM(s).
-6. **`BootstrapFinetune`**: Distills a prompt-based DSPy program into weight updates (for smaller LMs). The output is a DSPy program that has the same steps, but where each step is conducted by a finetuned model instead of a prompted LM.
+7. **`BootstrapFinetune`**: Distills a prompt-based DSPy program into weight updates (for smaller LMs). The output is a DSPy program that has the same steps, but where each step is conducted by a finetuned model instead of a prompted LM.
-#### Program Transformations
+### Program Transformations
8. **`Ensemble`**: Ensembles a set of DSPy programs and either uses the full set or randomly samples a subset into a single program.
## Which optimizer should I use?
-As a rule of thumb, if you don't know where to start, use `BootstrapFewShotWithRandomSearch`.
-
-Here's the general guidance on getting started:
-
-* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`.
-
-* If you have slightly more data, e.g. 50 examples of your task, use `BootstrapFewShotWithRandomSearch`.
-
-* If you have more data than that, e.g. 300 examples or more, use `MIPRO`.
-
-* If you have been able to use one of these with a large LM (e.g., 7B parameters or above) and need a very efficient program, compile that down to a small LM with `BootstrapFinetune`.
+Ultimately, finding the ‘right’ optimizer to use & the best configuration for your task will require experimentation. Success in DSPy is still an iterative process - getting the best performance on your task will require you to explore and iterate.
+That being said, here's the general guidance on getting started:
+* If you have **very few examples** (around 10), start with `BootstrapFewShot`.
+* If you have **more data** (50 examples or more), try `BootstrapFewShotWithRandomSearch`.
+* If you prefer to do **instruction optimization only** (i.e. you want to keep your prompt 0-shot), use `MIPROv2` [configured for 0-shot optimization to optimize](../deep-dive/optimizers/miprov2.md#optimizing-instructions-only-with-miprov2-0-shot).
+* If you’re willing to use more inference calls to perform **longer optimization runs** (e.g. 40 trials or more), and have enough data (e.g. 200 examples or more to prevent overfitting) then try `MIPROv2`.
+* If you have been able to use one of these with a large LM (e.g., 7B parameters or above) and need a very **efficient program**, finetune a small LM for your task with `BootstrapFinetune`.
## How do I use an optimizer?
-They all share this general interface, with some differences in the keyword arguments (hyperparameters).
+They all share this general interface, with some differences in the keyword arguments (hyperparameters). Detailed documentation for key optimizers can be found [here](../deep-dive/optimizers), and a full list can be found [here](../cheatsheet.md#dspy-optimizers).
Let's see this with the most common one, `BootstrapFewShotWithRandomSearch`.
diff --git a/docs/docs/cheatsheet.md b/docs/docs/cheatsheet.md
index c067b6552..40b968834 100644
--- a/docs/docs/cheatsheet.md
+++ b/docs/docs/cheatsheet.md
@@ -304,6 +304,9 @@ loaded_program.load(path=save_path)
### BootstrapFewShotWithRandomSearch
+Detailed documentation on BootstrapFewShotWithRandomSearch can be found [here](deep-dive/optimizers/bootstrap-fewshot.mdx).
+
+
```python
from dspy.teleprompt import BootstrapFewShotWithRandomSearch
@@ -358,6 +361,9 @@ for p in finetune_program.predictors():
### COPRO
+Detailed documentation on COPRO can be found [here](deep-dive/optimizers/copro.mdx).
+
+
```python
from dspy.teleprompt import COPRO
@@ -381,6 +387,78 @@ kwargs = dict(num_threads=NUM_THREADS, display_progress=True, display_table=0)
compiled_program_optimized_bayesian_signature = teleprompter.compile(your_dspy_program, trainset=trainset, num_trials=100, max_bootstrapped_demos=3, max_labeled_demos=5, eval_kwargs=kwargs)
```
+### MIPROv2
+
+Note: detailed documentation can be found [here](deep-dive/optimizers/miprov2.md). `MIPROv2` is the latest extension of `MIPRO` which includes updates such as (1) improvements to instruction proposal and (2) more efficient search with minibatching.
+
+#### Optimizing with MIPROv2
+```python
+# Import the optimizer
+from dspy.teleprompt import MIPROv2
+
+# Initialize optimizer
+teleprompter = MIPROv2(
+ metric=gsm8k_metric,
+ num_candidates=7,
+ init_temperature=0.5,
+ verbose=False,
+ num_threads=4,
+)
+
+# Optimize program
+print(f"Optimizing program with MIPRO...")
+optimized_program = teleprompter.compile(
+ program.deepcopy(),
+ trainset=trainset,
+ max_bootstrapped_demos=3,
+ max_labeled_demos=4,
+ num_trials=15,
+ minibatch_size=25,
+ minibatch_full_eval_steps=10,
+ minibatch=True,
+ requires_permission_to_run=False,
+)
+
+# Save optimize program for future use
+optimized_program.save(f"mipro_optimized")
+
+# Evaluate optimized program
+print(f"Evluate optimized program...")
+evaluate(optimized_program, devset=devset[:])
+```
+
+#### Optimizing instructions only with MIPROv2 (0-Shot)
+```python
+# Import optimizer
+from dspy.teleprompt import MIPROv2
+
+# Initialize optimizer
+teleprompter = MIPROv2(
+ metric=gsm8k_metric,
+ num_candidates=15,
+ init_temperature=0.5,
+ verbose=False,
+ num_threads=4,
+)
+
+# Perform optimization
+print(f"Optimizing program with MIPRO (0-Shot)...")
+zeroshot_optimized_program = teleprompter.compile(
+ program.deepcopy(),
+ trainset=trainset,
+ max_bootstrapped_demos=0, # setting demos to 0 for 0-shot optimization
+ max_labeled_demos=0,
+ num_trials=15,
+ minibatch=False,
+ requires_permission_to_run=False,
+)
+
+
+zeroshot_optimized_program.save(f"mipro_0shot_optimized")
+
+print(f"Evaluate optimized program...")
+evaluate(zeroshot_optimized_program, devset=devset[:])
+```
### Signature Optimizer with Types
```python
diff --git a/docs/docs/deep-dive/teleprompter/_category_.json b/docs/docs/deep-dive/optimizers/_category_.json
similarity index 100%
rename from docs/docs/deep-dive/teleprompter/_category_.json
rename to docs/docs/deep-dive/optimizers/_category_.json
diff --git a/docs/docs/deep-dive/teleprompter/bootstrap-fewshot.mdx b/docs/docs/deep-dive/optimizers/bootstrap-fewshot.mdx
similarity index 100%
rename from docs/docs/deep-dive/teleprompter/bootstrap-fewshot.mdx
rename to docs/docs/deep-dive/optimizers/bootstrap-fewshot.mdx
diff --git a/docs/docs/deep-dive/teleprompter/signature-optimizer.mdx b/docs/docs/deep-dive/optimizers/copro.mdx
similarity index 99%
rename from docs/docs/deep-dive/teleprompter/signature-optimizer.mdx
rename to docs/docs/deep-dive/optimizers/copro.mdx
index 7f60b3bf8..65af4186b 100644
--- a/docs/docs/deep-dive/teleprompter/signature-optimizer.mdx
+++ b/docs/docs/deep-dive/optimizers/copro.mdx
@@ -4,7 +4,7 @@ sidebar_position: 2
import AuthorDetails from '@site/src/components/AuthorDetails';
-# Signature Optimizer
+# `COPRO`
`COPRO` which aims to improve the output prefixes and instruction of the signatures in a module in a zero/few shot setting. This teleprompter is especially beneficial for fine-tuning the prompt for language models and ensure they perform tasks more effectively, all from a vague and unrefined prompt.
diff --git a/docs/docs/deep-dive/teleprompter/img/signature_optimizer.png b/docs/docs/deep-dive/optimizers/img/signature_optimizer.png
similarity index 100%
rename from docs/docs/deep-dive/teleprompter/img/signature_optimizer.png
rename to docs/docs/deep-dive/optimizers/img/signature_optimizer.png
diff --git a/docs/docs/deep-dive/teleprompter/img/signature_optimizer_process.png b/docs/docs/deep-dive/optimizers/img/signature_optimizer_process.png
similarity index 100%
rename from docs/docs/deep-dive/teleprompter/img/signature_optimizer_process.png
rename to docs/docs/deep-dive/optimizers/img/signature_optimizer_process.png
diff --git a/docs/docs/deep-dive/teleprompter/img/signature_optimizer_process_v2.png b/docs/docs/deep-dive/optimizers/img/signature_optimizer_process_v2.png
similarity index 100%
rename from docs/docs/deep-dive/teleprompter/img/signature_optimizer_process_v2.png
rename to docs/docs/deep-dive/optimizers/img/signature_optimizer_process_v2.png
diff --git a/docs/docs/deep-dive/teleprompter/img/signature_optimizer_process_v3.png b/docs/docs/deep-dive/optimizers/img/signature_optimizer_process_v3.png
similarity index 100%
rename from docs/docs/deep-dive/teleprompter/img/signature_optimizer_process_v3.png
rename to docs/docs/deep-dive/optimizers/img/signature_optimizer_process_v3.png
diff --git a/docs/docs/deep-dive/teleprompter/img/signature_optimizer_process_v4.png b/docs/docs/deep-dive/optimizers/img/signature_optimizer_process_v4.png
similarity index 100%
rename from docs/docs/deep-dive/teleprompter/img/signature_optimizer_process_v4.png
rename to docs/docs/deep-dive/optimizers/img/signature_optimizer_process_v4.png
diff --git a/docs/docs/deep-dive/optimizers/miprov2.md b/docs/docs/deep-dive/optimizers/miprov2.md
new file mode 100644
index 000000000..b2e8b03bc
--- /dev/null
+++ b/docs/docs/deep-dive/optimizers/miprov2.md
@@ -0,0 +1,183 @@
+# `MIPROv2` Optimizer
+
+## Table of Contents
+1. [Overview](#overview)
+2. [Example Usage](#example-usage)
+ - [Setting up a Sample Pipeline](#setting-up-a-sample-pipeline)
+ - [Optimizing with MIPROv2](#optimizing-with-miprov2)
+ - [Optimizing instructions only with MIPROv2 (0-Shot)](#optimizing-instructions-only-with-miprov2-0-shot)
+3. [Parameters](#parameters)
+ - [Initialization Parameters](#initialization-parameters)
+ - [Compile Method Specific Parameters](#compile-method-specific-parameters)
+4. [How MIPROv2 Works](#how-miprov2-works)
+
+## Overview
+
+`MIPROv2` (<u>M</u>ultiprompt <u>I</u>nstruction <u>PR</u>oposal <u>O</u>ptimizer Version 2) is an prompt optimizer capable of optimizing both instructions and few-shot examples jointly. It does this by bootstrapping few-shot example candidates, proposing instructions grounded in different dynamics of the task, and finding an optimized combination of these options using Bayesian Optimization. It can be used for optimizing few-shot examples & instructions jointly, or just instructions for 0-shot optimization.
+
+## Example Usage
+
+### Setting up a Sample Pipeline
+
+We'll be making a basic answer generation pipeline over GSM8K dataset that we saw in the [Minimal Example](https://dspy-docs.vercel.app/docs/quick-start/minimal-example), we won't be changing anything in it! So let's start by configuring the LM which will be OpenAI LM client with `gpt-3.5-turbo` as the LLM in use.
+
+```python
+import dspy
+
+turbo = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=250)
+dspy.settings.configure(lm=turbo)
+```
+
+Now that we have the LM client setup it's time to import the train-dev split in `GSM8k` class that DSPy provides us:
+
+```python
+from dspy.datasets.gsm8k import GSM8K, gsm8k_metric
+
+gms8k = GSM8K()
+
+trainset, devset = gms8k.train, gms8k.dev
+```
+
+We'll now define a basic QA inline signature i.e. `question->answer` and pass it to `ChainOfThought` module, that applies necessary addition for CoT style prompting to the Signature.
+
+```python
+class CoT(dspy.Module):
+ def __init__(self):
+ super().__init__()
+ self.prog = dspy.ChainOfThought("question -> answer")
+
+ def forward(self, question):
+ return self.prog(question=question)
+```
+
+Now we need to evaluate this pipeline too! So we'll use the `Evaluate` class that DSPy provides us, as for the metric we'll use the `gsm8k_metric` that we imported above.
+
+```python
+from dspy.evaluate import Evaluate
+
+evaluate = Evaluate(devset=devset[:], metric=gsm8k_metric, num_threads=8, display_progress=True, display_table=False)
+```
+
+To evaluate the `CoT` pipeline we'll need to create an object of it and pass it as an arg to the `evaluator` call.
+
+```python
+program = CoT()
+
+evaluate(program, devset=devset[:])
+```
+
+Now we have the baseline pipeline ready to use, so let's try using the `MIPROv2` optimizer to improve our pipeline's performance!
+
+### Optimizing with `MIPROv2`
+
+```python
+# Import the optimizer
+from dspy.teleprompt import MIPROv2
+
+# Initialize optimizer
+teleprompter = MIPROv2(
+ metric=gsm8k_metric,
+ num_candidates=7,
+ init_temperature=0.5,
+ verbose=False,
+ num_threads=4,
+)
+
+# Optimize program
+print(f"Optimizing program with MIPRO...")
+optimized_program = teleprompter.compile(
+ program.deepcopy(),
+ trainset=trainset,
+ max_bootstrapped_demos=3,
+ max_labeled_demos=4,
+ num_trials=15,
+ minibatch_size=25,
+ minibatch_full_eval_steps=10,
+ minibatch=True,
+ requires_permission_to_run=False,
+)
+
+# Save optimize program for future use
+optimized_program.save(f"mipro_optimized")
+
+# Evaluate optimized program
+print(f"Evluate optimized program...")
+evaluate(optimized_program, devset=devset[:])
+```
+
+### Optimizing instructions only with `MIPROv2` (0-Shot)
+
+```python
+# Import optimizer
+from dspy.teleprompt import MIPROv2
+
+# Initialize optimizer
+teleprompter = MIPROv2(
+ metric=gsm8k_metric,
+ num_candidates=15,
+ init_temperature=0.5,
+ verbose=False,
+ num_threads=4,
+)
+
+# Perform optimization
+print(f"Optimizing program with MIPRO (0-Shot)...")
+zeroshot_optimized_program = teleprompter.compile(
+ program.deepcopy(),
+ trainset=trainset,
+ max_bootstrapped_demos=0, # setting demos to 0 for 0-shot optimization
+ max_labeled_demos=0,
+ num_trials=15,
+ minibatch=False,
+ requires_permission_to_run=False,
+)
+
+
+zeroshot_optimized_program.save(f"mipro_0shot_optimized")
+
+print(f"Evaluate optimized program...")
+evaluate(zeroshot_optimized_program, devset=devset[:])
+```
+
+## Parameters
+
+### Initialization Parameters
+
+- **prompt_model**: Model used for prompt generation. Defaults to the model configured in `dspy.settings`.
+- **task_model**: Model used for task execution. Defaults to the model configured in `dspy.settings`.
+- **metric**: The evaluation metric used to optimize the task model.
+- **num_candidates**: Number of candidate prompts to generate and evaluate. Default is 10.
+- **num_threads**: Threads to use for evaluation.
+- **max_errors**: Maximum errors during an evaluation run that can be made before throwing an Exception.
+- **teacher_settings**: Settings to use for the teacher model that bootstraps few-shot examples.
+- **init_temperature**: The initial temperature for prompt generation, influencing creativity. Default is 1.0.
+- **verbose**: Enables printing intermediate steps and information.
+- **track_stats**: Tracks optimization process statistics if set to True.
+- **log_dir**: Directory for saving logs and output data.
+- **metric_threshold**: A metric threshold is used if we only want to keep bootstrapped few-shot exampels that exceed some threshold of performance. Defaults to None.
+
+### Compile Method Specific Parameters
+
+- **student**: The base program to optimize.
+- **trainset**: Training dataset for model training.
+- **num_batches**: Number of optimization trials to run. Default is 30. Note that if you are using mini-batching, you might want to increase **num_batches**. If not, you might want to lower this.
+- **max_bootstrapped_demos**: Maximum number of bootstrapped demonstrations used for few-shot learning. Default is 4.
+- **max_labeled_demos**: Maximum number of labeled demonstrations used for few-shot learning. Default is 16.
+- **seed**: Seed for random number generator for reproducibility. Default is 9.
+- **minibatch**: Flag to enable minibatching during evaluation. Default is True.
+- **minibatch_size**: Size of minibatches for evaluations. Default is 25.0.
+- **minibatch_full_eval_steps**: Number of minibatch steps before performing a full evaluation.
+- **program_aware_proposer**: Flag to enable summarizing a reflexive view of the code for your LM program . Default is True.
+- **requires_permission_to_run**: Flag to require user confirmation before running optimizations. Default is True.
+
+## How `MIPROv2` works
+
+At a high level, `MIPROv2` works by creating both few-shot examples and new instructions for each predictor in your LM program, and then searching over these using Bayesian Optimization to find the best combination of these variables for your program.
+
+These steps are broken down in more detail below:
+1) **Bootstrap Few-Shot Examples**: The same bootstrapping technique used in `BootstrapFewshotWithRandomSearch` is used to create few-shot examples. This works by randomly sampling examples from your training set, which are then run through your LM program. If the output from the program is correct for this example, it is kept as a valid few-shot example candidate. Otherwise, we try another example until we've curated the specified amount of few-shot example candidates. This step creates `num_candidates` sets of `max_bootstrapped_demos` bootstrapped examples and `max_labeled_demos` basic examples sampled from the training set.
+2) **Propose Instruction Candidates**. Next, we propose instruction candidates for each predictor in the program. This is done using another LM program as a proposer, which bootstraps & summarizes relevant information about the task to generate high quality instructions. Specifically, the instruction proposer includes (1) a generated summary of properties of the training dataset, (2) a generated summary of your LM program's code and the specific predictor that an instruction is being generated for, (3) the previously bootstrapped few-shot examples to show reference inputs / outputs for a given predictor and (4) a randomly sampled tip for generation (i.e. "be creative", "be concise", etc.) to help explore the feature space of potential instructions.
+3. **Find an Optimized Combination of Few-Shot Examples & Instructions**. Finally, now that we've created these few-shot examples and instructions, we use Bayesian Optimization to choose which set of these would work best for each predictor in our program. This works by running a series of `num_batches` trials, where a new set of prompts are evaluated over our training set at each trial. This helps the Bayesian Optimizer learn which combination of prompts work best over time. If `minibatch` is set to `True` (which it is by default), then the new set of prompts are only evaluated on a minibatch of size `minibatch_size` at each trial which generally allows for more efficient exploration / exploitation. The best averaging set of prompts is then evalauted on the full training set every `minibatch_full_eval_steps` get a less noisey performance benchmark. At the end of the optimization process, the LM program with the set of prompts that performed best on the full training set is returned.
+
+
+For those interested in more details, more information on `MIPROv2` along with a study on `MIPROv2` compared with other DSPy optimizers can be found in [this paper](https://arxiv.org/abs/2406.11695).
\ No newline at end of file