DSPy: The framework for programming—not prompting—language models
Python
Update 6-optimizers.md with corrections
arnavsinghvi1106bf5a7639fb
diff --git a/docs/docs/building-blocks/6-optimizers.md b/docs/docs/building-blocks/6-optimizers.mdindex 65c62955b..96dc9b3af 100644--- a/docs/docs/building-blocks/6-optimizers.md+++ b/docs/docs/building-blocks/6-optimizers.md@@ -27,7 +27,7 @@ DSPy programs consist of multiple calls to LMs, stacked together as [DSPy moduleGiven a metric, DSPy can optimize all of these three with multi-stage optimization algorithms. These can combine gradient descent (for LM weights) and discrete LM-driven optimization, i.e. for crafting/updating instructions and for creating/validating demonstrations. DSPy Demonstrations are like few-shot examples, but they're far more powerful. They can be created from scratch, given your program, and their creation and selection can be optimized in many effective ways.-In many cases, we found that compiling leads to better prompts than humans write. Not because DSPy optimizers are more creative than humans, but simply because they can try more things, much more systematically, and tune the metrics directly.+In many cases, we found that compiling leads to better prompts than human writing. Not because DSPy optimizers are more creative than humans, but simply because they can try more things, much more systematically, and tune the metrics directly.## What DSPy Optimizers are currently available?@@ -38,36 +38,32 @@ All of these can be accessed via `from dspy.teleprompt import *`.#### Automatic Few-Shot Learning-These optimizers extend the signature by adding labeled examples to the prompt that will be submitted to the model, automatically implementing few-shot learning.+These optimizers extend the signature by automatically generating and including **optimized** examples within the prompt sent to the model, implementing few-shot learning.-1. **`LabeledFewShot`**: Simply constructs few-shot examples (demos) from provided labeled Q/A pairs. Takes as parameter, `k`, the number of examples to be added to the prompt and chooses `k` examples randomly from the `trainset` parameter.+1. **`LabeledFewShot`**: Simply constructs few-shot examples (demos) from provided labeled input and output data points. Requires `k` (number of examples for the prompt) and `trainset` to randomly select `k` examples from.-2. **`BootstrapFewShot`**: Uses a `teacher` module (which defaults to your program) to generate complete demonstrations for every stage of your program in addition to the labeled examples in `trainset`. This is primarily useful if there are potentially many different ways to phrase the answer to a query, and you do not have a training set that covers the alternative phrasings. In this case a language model can be used to "bootstrap," to generate additional answers to the questions in the training set. Takes as parameters `max_labeled_demos` (the number of demonstrations to choose randomly from the `trainset`), and `max_bootstrapped_demos` (the number of additional examples to be generated by the `teacher`). The bootstrapping process uses the `metric` to determine whether to accept or reject bootstrapped answers. Will simply use the generated demonstrations (if they pass the metric) without any further optimization; i.e., does not check to see if the program's performance is improved by the compilation. Advanced: Supports using a `teacher` program that is a *different* DSPy program that has compatible structure, for harder tasks.+2. **`BootstrapFewShot`**: Uses a `teacher` module (which defaults to your program) to generate complete demonstrations for every stage of your program, along with labeled examples in `trainset`. Parameters include `max_labeled_demos` (the number of demonstrations randomly selected from the `trainset`) and `max_bootstrapped_demos` (the number of additional examples generated by the `teacher`). The bootstrapping process employs the metric to validate demonstrations, including only those that pass the metric in the "compiled" prompt. Advanced: Supports using a `teacher` program that is a *different* DSPy program that has compatible structure, for harder tasks.-3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program. Takes as parameter (in addition to the ones for `BootstrapFewShot`) `num_candidate_programs`: the number of random programs, each generated through `BootstrapFewShot`. Picks a single best based on scores over the `valset` (which defaults to being the same as the `trainset`).+3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program over the optimization. Parameters mirror those of `BootstrapFewShot`, with the addition of `num_candidate_programs`, which specifies the number of random programs evaluated over the optimization, including candidates of the uncompiled program, `LabeledFewShot` optimized program, `BootstrapFewShot` compiled program with unshuffled examples and `num_candidate_programs` of `BootstrapFewShot` compiled programs with randomized example sets.-4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` through Optuna hyperparameter optimization across demonstration sets, running trials to maximize evaluation metrics. As before, chooses the best of a candidate set of demonstration examples, but this time using Optuna to make the choice. <!--- TBQH, I don't understand how Optuna does this. As far as I can tell it simply chooses best based on multiple evaluations, rather than a single one, and mention of "hyperparameters" seems to be a red herring. --->+4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` through Optuna hyperparameter optimization across demonstration sets, running trials to maximize evaluation metrics and selecting the best demonstrations.-5. **`KNNFewShot`**. Selects demonstrations through k-Nearest Neighbors algorithm. Vectorizes the examples, and then clusters them, using cluster centers with `BootstrapFewShot` for bootstrapping/selection process. <!-- I'm not at all sure that this is right. I couldn't follow the KNN code in the repo, so I just assumed that dspy was trying to cover the space of possible examples by picking centers of different clusters. -->+5. **`KNNFewShot`**. Selects demonstrations through k-Nearest Neighbors algorithm. Vectorizes the examples, and then clusters them, using cluster centers with `BootstrapFewShot` for bootstrapping/selection process.#### Automatic Instruction Optimization-Unlike the above optimizers, these attempt to optimize the instructions, rather than the questions.+These optimizers serve to produce optimal instructions for the prompt, in addition to optimized few-shot demonstrations.-6. **`COPRO`**: Uses an LLM to generate and refine new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the `trainset`). The number of iterations of prompt improvement is controlled by the `depth` parameter.--#### Simultaneous optimization--This optimizer combines automatic few-shot learning and instruction optimization in one process.+6. **`COPRO`**: Generates and refines new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the `trainset`). Parameters include `depth` which is the number of iterations of prompt improvement the optimizer runs over.7. **`MIPRO`**: Generates instructions *and* few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.#### Automatic Finetuning-This optimizer is used to fine-tune the underlying LLM(s) instead of the prompting.+This optimizer is used to fine-tune the underlying LLM(s).6. **`BootstrapFinetune`**: Distills a prompt-based DSPy program into weight updates (for smaller LMs). The output is a DSPy program that has the same steps, but where each step is conducted by a finetuned model instead of a prompted LM.@@ -83,7 +79,7 @@ As a rule of thumb, if you don't know where to start, use `BootstrapFewShotWithRHere's the general guidance on getting started:-* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`. <!--- Wouldn't it make sense to simply use LabeledFewShot with k set to use all of the demos? --->+* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`.* If you have slightly more data, e.g. 50 examples of your task, use `BootstrapFewShotWithRandomSearch`.@@ -98,16 +94,12 @@ They all share this general interface, with some differences in the keyword arguLet's see this with the most common one, `BootstrapFewShotWithRandomSearch`.-<!-- The following example says that "we want to "bootstrap" (i.e., self-generate) 8-shot examples of your program's steps." But won't it actually give- 6 demonstrations, 3 taken from the examples (`max_labeled_demos=3`) and 3 self-generated (`max_bootstrapped_demos=3`)? Also, aren't the defaults- 16 labeled + 4 bootstrapped for a total of 20-shot prompting awfully high? -->-```pythonfrom dspy.teleprompt import BootstrapFewShotWithRandomSearch# Set up the optimizer: we want to "bootstrap" (i.e., self-generate) 8-shot examples of your program's steps.# The optimizer will repeat this 10 times (plus some initial attempts) before selecting its best attempt on the devset.-config = dict(max_bootstrapped_demos=3, max_labeled_demos=3, num_candidate_programs=10, num_threads=4)+config = dict(max_bootstrapped_demos=4, max_labeled_demos=4, num_candidate_programs=10, num_threads=4)teleprompter = BootstrapFewShotWithRandomSearch(metric=YOUR_METRIC_HERE, **config)optimized_program = teleprompter.compile(YOUR_PROGRAM_HERE, trainset=YOUR_TRAINSET_HERE)