Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Added more details to the discussion of optimizers/teleprompters.

Robert P. Goldmanf3e77d08a8fc
diff --git a/docs/docs/building-blocks/6-optimizers.md b/docs/docs/building-blocks/6-optimizers.md
index 55698140f..65c62955b 100644
--- a/docs/docs/building-blocks/6-optimizers.md
+++ b/docs/docs/building-blocks/6-optimizers.md
@@ -32,35 +32,48 @@ In many cases, we found that compiling leads to better prompts than humans write
## What DSPy Optimizers are currently available?
+[Subclasses of Teleprompter](figures/teleprompter-classes.png)
+
All of these can be accessed via `from dspy.teleprompt import *`.
#### Automatic Few-Shot Learning
-1. **`LabeledFewShot`**: Simply constructs few-shot examples from provided labeled Q/A pairs.
+These optimizers extend the signature by adding labeled examples to the prompt that will be submitted to the model, automatically implementing few-shot learning.
+
+1. **`LabeledFewShot`**: Simply constructs few-shot examples (demos) from provided labeled Q/A pairs. Takes as parameter, `k`, the number of examples to be added to the prompt and chooses `k` examples randomly from the `trainset` parameter.
+
+2. **`BootstrapFewShot`**: Uses a `teacher` module (which defaults to your program) to generate complete demonstrations for every stage of your program in addition to the labeled examples in `trainset`. This is primarily useful if there are potentially many different ways to phrase the answer to a query, and you do not have a training set that covers the alternative phrasings. In this case a language model can be used to "bootstrap," to generate additional answers to the questions in the training set. Takes as parameters `max_labeled_demos` (the number of demonstrations to choose randomly from the `trainset`), and `max_bootstrapped_demos` (the number of additional examples to be generated by the `teacher`). The bootstrapping process uses the `metric` to determine whether to accept or reject bootstrapped answers. Will simply use the generated demonstrations (if they pass the metric) without any further optimization; i.e., does not check to see if the program's performance is improved by the compilation. Advanced: Supports using a `teacher` program that is a *different* DSPy program that has compatible structure, for harder tasks.
-2. **`BootstrapFewShot`**: Uses your program to self-generate complete demonstrations for every stage of your program. Will simply use the generated demonstrations (if they pass the metric) without any further optimization. Advanced: Supports using a teacher program (a different DSPy program that has compatible structure) and a teacher LM, for harder tasks.
+3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program. Takes as parameter (in addition to the ones for `BootstrapFewShot`) `num_candidate_programs`: the number of random programs, each generated through `BootstrapFewShot`. Picks a single best based on scores over the `valset` (which defaults to being the same as the `trainset`).
-3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program.
+4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` through Optuna hyperparameter optimization across demonstration sets, running trials to maximize evaluation metrics. As before, chooses the best of a candidate set of demonstration examples, but this time using Optuna to make the choice. <!--- TBQH, I don't understand how Optuna does this. As far as I can tell it simply chooses best based on multiple evaluations, rather than a single one, and mention of "hyperparameters" seems to be a red herring. --->
+
+5. **`KNNFewShot`**. Selects demonstrations through k-Nearest Neighbors algorithm. Vectorizes the examples, and then clusters them, using cluster centers with `BootstrapFewShot` for bootstrapping/selection process. <!-- I'm not at all sure that this is right. I couldn't follow the KNN code in the repo, so I just assumed that dspy was trying to cover the space of possible examples by picking centers of different clusters. -->
-4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` through Optuna hyperparameter optimization across demonstration sets, running trials to maximize evaluation metrics.
#### Automatic Instruction Optimization
-4. **`COPRO`**: Generates and refines new instructions for each step, and optimizes them with coordinate ascent.
+Unlike the above optimizers, these attempt to optimize the instructions, rather than the questions.
+
+6. **`COPRO`**: Uses an LLM to generate and refine new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the `trainset`). The number of iterations of prompt improvement is controlled by the `depth` parameter.
+
+#### Simultaneous optimization
-5. **`MIPRO`**: Generates instructions and few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.
+This optimizer combines automatic few-shot learning and instruction optimization in one process.
+
+7. **`MIPRO`**: Generates instructions *and* few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.
#### Automatic Finetuning
+This optimizer is used to fine-tune the underlying LLM(s) instead of the prompting.
+
6. **`BootstrapFinetune`**: Distills a prompt-based DSPy program into weight updates (for smaller LMs). The output is a DSPy program that has the same steps, but where each step is conducted by a finetuned model instead of a prompted LM.
#### Program Transformations
-7. **`KNNFewShot`**. Selects demonstrations through k-Nearest Neighbors algorithm integrating `BootstrapFewShot` for bootstrapping/selection process.
-
8. **`Ensemble`**: Ensembles a set of DSPy programs and either uses the full set or randomly samples a subset into a single program.
@@ -70,7 +83,7 @@ As a rule of thumb, if you don't know where to start, use `BootstrapFewShotWithR
Here's the general guidance on getting started:
-* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`.
+* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`. <!--- Wouldn't it make sense to simply use LabeledFewShot with k set to use all of the demos? --->
* If you have slightly more data, e.g. 50 examples of your task, use `BootstrapFewShotWithRandomSearch`.
@@ -85,6 +98,10 @@ They all share this general interface, with some differences in the keyword argu
Let's see this with the most common one, `BootstrapFewShotWithRandomSearch`.
+<!-- The following example says that "we want to "bootstrap" (i.e., self-generate) 8-shot examples of your program's steps." But won't it actually give
+ 6 demonstrations, 3 taken from the examples (`max_labeled_demos=3`) and 3 self-generated (`max_bootstrapped_demos=3`)? Also, aren't the defaults
+ 16 labeled + 4 bootstrapped for a total of 20-shot prompting awfully high? -->
+
```python
from dspy.teleprompt import BootstrapFewShotWithRandomSearch
@@ -115,4 +132,4 @@ To load a program from a file, you can instantiate an object from that class and
```python
loaded_program = YOUR_PROGRAM_CLASS()
loaded_program.load(path=YOUR_SAVE_PATH)
-```
\ No newline at end of file
+```
diff --git a/docs/docs/building-blocks/figures/teleprompter-classes.dot b/docs/docs/building-blocks/figures/teleprompter-classes.dot
new file mode 100644
index 000000000..05e576f6c
--- /dev/null
+++ b/docs/docs/building-blocks/figures/teleprompter-classes.dot
@@ -0,0 +1,46 @@
+digraph "classes" {
+rankdir=BT
+charset="utf-8"
+"dspy.teleprompt.vanilla.LabeledFewShot" [color="black", fontcolor="black", label=<{LabeledFewShot|k : int<br ALIGN="LEFT"/>student<br ALIGN="LEFT"/>trainset<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+"dspy.teleprompt.bootstrap.BootstrapFewShot" [color="black", fontcolor="black", label=<{BootstrapFewShot|error_count : int<br ALIGN="LEFT"/>error_lock : lock<br ALIGN="LEFT"/>max_bootstrapped_demos : int<br ALIGN="LEFT"/>max_errors : int<br ALIGN="LEFT"/>max_labeled_demos : int<br ALIGN="LEFT"/>max_rounds : int<br ALIGN="LEFT"/>metric : NoneType<br ALIGN="LEFT"/>metric_threshold : NoneType<br ALIGN="LEFT"/>name2predictor : dict<br ALIGN="LEFT"/>name2traces<br ALIGN="LEFT"/>predictor2name : dict<br ALIGN="LEFT"/>student<br ALIGN="LEFT"/>teacher<br ALIGN="LEFT"/>teacher_settings : NoneType, dict<br ALIGN="LEFT"/>trainset<br ALIGN="LEFT"/>validation<br ALIGN="LEFT"/>valset : NoneType<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+"dspy.teleprompt.finetune.BootstrapFinetune" [color="black", fontcolor="black", label=<{BootstrapFinetune|metric : NoneType<br ALIGN="LEFT"/>multitask : bool<br ALIGN="LEFT"/>teacher_settings : dict<br ALIGN="LEFT"/>teleprompter<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+"dspy.teleprompt.teleprompt_optuna.BootstrapFewShotWithOptuna" [color="black", fontcolor="black", label=<{BootstrapFewShotWithOptuna|compiled_teleprompter<br ALIGN="LEFT"/>max_labeled_demos : int<br ALIGN="LEFT"/>max_num_samples : int<br ALIGN="LEFT"/>max_rounds : int<br ALIGN="LEFT"/>metric<br ALIGN="LEFT"/>min_num_samples : int<br ALIGN="LEFT"/>num_candidate_sets : int<br ALIGN="LEFT"/>num_threads : int<br ALIGN="LEFT"/>student<br ALIGN="LEFT"/>teacher<br ALIGN="LEFT"/>teacher_settings : dict<br ALIGN="LEFT"/>trainset<br ALIGN="LEFT"/>valset<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>objective(trial)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+"dspy.teleprompt.random_search.BootstrapFewShotWithRandomSearch" [color="black", fontcolor="black", label=<{BootstrapFewShotWithRandomSearch|max_errors : int<br ALIGN="LEFT"/>max_labeled_demos : int<br ALIGN="LEFT"/>max_num_samples : int<br ALIGN="LEFT"/>max_rounds : int<br ALIGN="LEFT"/>metric<br ALIGN="LEFT"/>metric_threshold : NoneType<br ALIGN="LEFT"/>min_num_samples : int<br ALIGN="LEFT"/>num_candidate_sets : int<br ALIGN="LEFT"/>num_threads : int<br ALIGN="LEFT"/>stop_at_score : NoneType<br ALIGN="LEFT"/>teacher_settings : dict<br ALIGN="LEFT"/>trainset<br ALIGN="LEFT"/>valset<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+"dspy.teleprompt.copro_optimizer.COPRO" [color="black", fontcolor="black", label=<{COPRO|breadth : int<br ALIGN="LEFT"/>depth : int<br ALIGN="LEFT"/>init_temperature : float<br ALIGN="LEFT"/>metric : NoneType<br ALIGN="LEFT"/>prompt_model : NoneType<br ALIGN="LEFT"/>track_stats : bool<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+"dspy.teleprompt.ensemble.Ensemble" [color="black", fontcolor="black", label=<{Ensemble|deterministic : bool<br ALIGN="LEFT"/>reduce_fn : NoneType<br ALIGN="LEFT"/>size : NoneType<br ALIGN="LEFT"/>|compile(programs)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+"dspy.teleprompt.knn_fewshot.KNNFewShot" [color="black", fontcolor="black", label=<{KNNFewShot|KNN<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+"dspy.teleprompt.mipro_optimizer.MIPRO" [color="black", fontcolor="black", label=<{MIPRO|init_temperature : float<br ALIGN="LEFT"/>metric<br ALIGN="LEFT"/>num_candidates : int<br ALIGN="LEFT"/>observations : NoneType<br ALIGN="LEFT"/>prompt_model : NoneType<br ALIGN="LEFT"/>task_model : NoneType<br ALIGN="LEFT"/>teacher_settings : dict<br ALIGN="LEFT"/>track_stats : bool<br ALIGN="LEFT"/>verbose : bool<br ALIGN="LEFT"/>view_data_batch_size : int<br ALIGN="LEFT"/>|compile(student: dspy.Program): dspy.Program<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+
+
+"dspy.teleprompt.vanilla.LabeledFewShot" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.teleprompt.Teleprompter" [color="black", fontcolor="black", label=<{Teleprompter|<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+"dspy.teleprompt.bootstrap.BootstrapFewShot" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.copro_optimizer.COPRO" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.ensemble.Ensemble" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.finetune.BootstrapFinetune" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.knn_fewshot.KNNFewShot" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.mipro_optimizer.MIPRO" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.random_search.BootstrapFewShotWithRandomSearch" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.teleprompt_optuna.BootstrapFewShotWithOptuna" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];
+"dspy.teleprompt.bootstrap.BootstrapFewShot" -> "dspy.teleprompt.finetune.BootstrapFinetune" [arrowhead="diamond", arrowtail="none", fontcolor="green", label="teleprompter", style="solid"];
+
+
+// "dspy.teleprompt.mipro_optimizer.BasicGenerateInstruction" [color="black", fontcolor="black", label=<{BasicGenerateInstruction|basic_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+// "dspy.teleprompt.copro_optimizer.BasicGenerateInstruction" [color="black", fontcolor="black", label=<{BasicGenerateInstruction|basic_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+// "dspy.teleprompt.mipro_optimizer.BasicGenerateInstructionWithDataObservations" [color="black", fontcolor="black", label=<{BasicGenerateInstructionWithDataObservations|basic_instruction : FieldInfo<br ALIGN="LEFT"/>observations : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+// "dspy.teleprompt.mipro_optimizer.BasicGenerateInstructionWithExamples" [color="black", fontcolor="black", label=<{BasicGenerateInstructionWithExamples|basic_instruction : FieldInfo<br ALIGN="LEFT"/>examples : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+// "dspy.teleprompt.mipro_optimizer.BasicGenerateInstructionWithExamplesAndDataObservations" [color="black", fontcolor="black", label=<{BasicGenerateInstructionWithExamplesAndDataObservations|basic_instruction : FieldInfo<br ALIGN="LEFT"/>examples : FieldInfo<br ALIGN="LEFT"/>observations : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+
+
+
+// "dspy.teleprompt.mipro_optimizer.DatasetDescriptor" [color="black", fontcolor="black", label=<{DatasetDescriptor|examples : FieldInfo<br ALIGN="LEFT"/>observations : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+// "dspy.teleprompt.mipro_optimizer.DatasetDescriptorWithPriorObservations" [color="black", fontcolor="black", label=<{DatasetDescriptorWithPriorObservations|examples : FieldInfo<br ALIGN="LEFT"/>observations : FieldInfo<br ALIGN="LEFT"/>prior_observations : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+
+// "dspy.teleprompt.ensemble.Ensemble.compile.EnsembledProgram" [color="black", fontcolor="black", label=<{EnsembledProgram|programs<br ALIGN="LEFT"/>|forward()<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+// "dspy.teleprompt.copro_optimizer.GenerateInstructionGivenAttempts" [color="black", fontcolor="black", label=<{GenerateInstructionGivenAttempts|attempted_instructions : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+// "dspy.teleprompt.signature_opt_typed.make_initial_signature.GenerateInstructionInitial" [color="black", fontcolor="black", label=<{GenerateInstructionInitial|basic_signature : T<br ALIGN="LEFT"/>proposed_signatures : list[T]<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+// "dspy.teleprompt.signature_opt_typed.generate_with_avoidance.GenerateSignature" [color="black", fontcolor="black", label=<{GenerateSignature|analysis : str<br ALIGN="LEFT"/>proposed_signature : T<br ALIGN="LEFT"/>score : float<br ALIGN="LEFT"/>|check_signature_not_attempted(s: T): T<br ALIGN="LEFT"/>}>, shape="record", style="solid"];
+
+// "dspy.teleprompt.mipro_optimizer.ObservationSummarizer" [color="black", fontcolor="black", label=<{ObservationSummarizer|observations : FieldInfo<br ALIGN="LEFT"/>summary : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+// "dspy.teleprompt.signature_opt_typed.OptimizerResult" [color="black", fontcolor="black", label=<{OptimizerResult|program : Module<br ALIGN="LEFT"/>scores : list[float]<br ALIGN="LEFT"/>signatures : list[dict[str, Signature]]<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];
+}
diff --git a/docs/docs/building-blocks/figures/teleprompter-classes.pdf b/docs/docs/building-blocks/figures/teleprompter-classes.pdf
new file mode 100644
index 000000000..4e8ce79ce
Binary files /dev/null and b/docs/docs/building-blocks/figures/teleprompter-classes.pdf differ
diff --git a/docs/docs/building-blocks/figures/teleprompter-classes.png b/docs/docs/building-blocks/figures/teleprompter-classes.png
new file mode 100644
index 000000000..c774c8b5d
Binary files /dev/null and b/docs/docs/building-blocks/figures/teleprompter-classes.png differ