DSPy: The framework for programming—not prompting—language models
Python
Added more details to the discussion of optimizers/teleprompters.
Robert P. Goldmanf3e77d08a8fc
diff --git a/docs/docs/building-blocks/6-optimizers.md b/docs/docs/building-blocks/6-optimizers.mdindex 55698140f..65c62955b 100644--- a/docs/docs/building-blocks/6-optimizers.md+++ b/docs/docs/building-blocks/6-optimizers.md@@ -32,35 +32,48 @@ In many cases, we found that compiling leads to better prompts than humans write## What DSPy Optimizers are currently available?+[Subclasses of Teleprompter](figures/teleprompter-classes.png)+All of these can be accessed via `from dspy.teleprompt import *`.#### Automatic Few-Shot Learning-1. **`LabeledFewShot`**: Simply constructs few-shot examples from provided labeled Q/A pairs.+These optimizers extend the signature by adding labeled examples to the prompt that will be submitted to the model, automatically implementing few-shot learning.++1. **`LabeledFewShot`**: Simply constructs few-shot examples (demos) from provided labeled Q/A pairs. Takes as parameter, `k`, the number of examples to be added to the prompt and chooses `k` examples randomly from the `trainset` parameter.++2. **`BootstrapFewShot`**: Uses a `teacher` module (which defaults to your program) to generate complete demonstrations for every stage of your program in addition to the labeled examples in `trainset`. This is primarily useful if there are potentially many different ways to phrase the answer to a query, and you do not have a training set that covers the alternative phrasings. In this case a language model can be used to "bootstrap," to generate additional answers to the questions in the training set. Takes as parameters `max_labeled_demos` (the number of demonstrations to choose randomly from the `trainset`), and `max_bootstrapped_demos` (the number of additional examples to be generated by the `teacher`). The bootstrapping process uses the `metric` to determine whether to accept or reject bootstrapped answers. Will simply use the generated demonstrations (if they pass the metric) without any further optimization; i.e., does not check to see if the program's performance is improved by the compilation. Advanced: Supports using a `teacher` program that is a *different* DSPy program that has compatible structure, for harder tasks.-2. **`BootstrapFewShot`**: Uses your program to self-generate complete demonstrations for every stage of your program. Will simply use the generated demonstrations (if they pass the metric) without any further optimization. Advanced: Supports using a teacher program (a different DSPy program that has compatible structure) and a teacher LM, for harder tasks.+3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program. Takes as parameter (in addition to the ones for `BootstrapFewShot`) `num_candidate_programs`: the number of random programs, each generated through `BootstrapFewShot`. Picks a single best based on scores over the `valset` (which defaults to being the same as the `trainset`).-3. **`BootstrapFewShotWithRandomSearch`**: Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program.+4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` through Optuna hyperparameter optimization across demonstration sets, running trials to maximize evaluation metrics. As before, chooses the best of a candidate set of demonstration examples, but this time using Optuna to make the choice. <!--- TBQH, I don't understand how Optuna does this. As far as I can tell it simply chooses best based on multiple evaluations, rather than a single one, and mention of "hyperparameters" seems to be a red herring. --->++5. **`KNNFewShot`**. Selects demonstrations through k-Nearest Neighbors algorithm. Vectorizes the examples, and then clusters them, using cluster centers with `BootstrapFewShot` for bootstrapping/selection process. <!-- I'm not at all sure that this is right. I couldn't follow the KNN code in the repo, so I just assumed that dspy was trying to cover the space of possible examples by picking centers of different clusters. -->-4. **`BootstrapFewShotWithOptuna`**: Applies `BootstrapFewShot` through Optuna hyperparameter optimization across demonstration sets, running trials to maximize evaluation metrics.#### Automatic Instruction Optimization-4. **`COPRO`**: Generates and refines new instructions for each step, and optimizes them with coordinate ascent.+Unlike the above optimizers, these attempt to optimize the instructions, rather than the questions.++6. **`COPRO`**: Uses an LLM to generate and refine new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the `trainset`). The number of iterations of prompt improvement is controlled by the `depth` parameter.++#### Simultaneous optimization-5. **`MIPRO`**: Generates instructions and few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.+This optimizer combines automatic few-shot learning and instruction optimization in one process.++7. **`MIPRO`**: Generates instructions *and* few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.#### Automatic Finetuning+This optimizer is used to fine-tune the underlying LLM(s) instead of the prompting.+6. **`BootstrapFinetune`**: Distills a prompt-based DSPy program into weight updates (for smaller LMs). The output is a DSPy program that has the same steps, but where each step is conducted by a finetuned model instead of a prompted LM.#### Program Transformations-7. **`KNNFewShot`**. Selects demonstrations through k-Nearest Neighbors algorithm integrating `BootstrapFewShot` for bootstrapping/selection process.-8. **`Ensemble`**: Ensembles a set of DSPy programs and either uses the full set or randomly samples a subset into a single program.@@ -70,7 +83,7 @@ As a rule of thumb, if you don't know where to start, use `BootstrapFewShotWithRHere's the general guidance on getting started:-* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`.+* If you have very little data, e.g. 10 examples of your task, use `BootstrapFewShot`. <!--- Wouldn't it make sense to simply use LabeledFewShot with k set to use all of the demos? --->* If you have slightly more data, e.g. 50 examples of your task, use `BootstrapFewShotWithRandomSearch`.@@ -85,6 +98,10 @@ They all share this general interface, with some differences in the keyword arguLet's see this with the most common one, `BootstrapFewShotWithRandomSearch`.+<!-- The following example says that "we want to "bootstrap" (i.e., self-generate) 8-shot examples of your program's steps." But won't it actually give+ 6 demonstrations, 3 taken from the examples (`max_labeled_demos=3`) and 3 self-generated (`max_bootstrapped_demos=3`)? Also, aren't the defaults+ 16 labeled + 4 bootstrapped for a total of 20-shot prompting awfully high? -->+```pythonfrom dspy.teleprompt import BootstrapFewShotWithRandomSearch@@ -115,4 +132,4 @@ To load a program from a file, you can instantiate an object from that class and```pythonloaded_program = YOUR_PROGRAM_CLASS()loaded_program.load(path=YOUR_SAVE_PATH)-```\ No newline at end of file+```diff --git a/docs/docs/building-blocks/figures/teleprompter-classes.dot b/docs/docs/building-blocks/figures/teleprompter-classes.dotnew file mode 100644index 000000000..05e576f6c--- /dev/null+++ b/docs/docs/building-blocks/figures/teleprompter-classes.dot@@ -0,0 +1,46 @@+digraph "classes" {+rankdir=BT+charset="utf-8"+"dspy.teleprompt.vanilla.LabeledFewShot" [color="black", fontcolor="black", label=<{LabeledFewShot|k : int<br ALIGN="LEFT"/>student<br ALIGN="LEFT"/>trainset<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+"dspy.teleprompt.bootstrap.BootstrapFewShot" [color="black", fontcolor="black", label=<{BootstrapFewShot|error_count : int<br ALIGN="LEFT"/>error_lock : lock<br ALIGN="LEFT"/>max_bootstrapped_demos : int<br ALIGN="LEFT"/>max_errors : int<br ALIGN="LEFT"/>max_labeled_demos : int<br ALIGN="LEFT"/>max_rounds : int<br ALIGN="LEFT"/>metric : NoneType<br ALIGN="LEFT"/>metric_threshold : NoneType<br ALIGN="LEFT"/>name2predictor : dict<br ALIGN="LEFT"/>name2traces<br ALIGN="LEFT"/>predictor2name : dict<br ALIGN="LEFT"/>student<br ALIGN="LEFT"/>teacher<br ALIGN="LEFT"/>teacher_settings : NoneType, dict<br ALIGN="LEFT"/>trainset<br ALIGN="LEFT"/>validation<br ALIGN="LEFT"/>valset : NoneType<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+"dspy.teleprompt.finetune.BootstrapFinetune" [color="black", fontcolor="black", label=<{BootstrapFinetune|metric : NoneType<br ALIGN="LEFT"/>multitask : bool<br ALIGN="LEFT"/>teacher_settings : dict<br ALIGN="LEFT"/>teleprompter<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+"dspy.teleprompt.teleprompt_optuna.BootstrapFewShotWithOptuna" [color="black", fontcolor="black", label=<{BootstrapFewShotWithOptuna|compiled_teleprompter<br ALIGN="LEFT"/>max_labeled_demos : int<br ALIGN="LEFT"/>max_num_samples : int<br ALIGN="LEFT"/>max_rounds : int<br ALIGN="LEFT"/>metric<br ALIGN="LEFT"/>min_num_samples : int<br ALIGN="LEFT"/>num_candidate_sets : int<br ALIGN="LEFT"/>num_threads : int<br ALIGN="LEFT"/>student<br ALIGN="LEFT"/>teacher<br ALIGN="LEFT"/>teacher_settings : dict<br ALIGN="LEFT"/>trainset<br ALIGN="LEFT"/>valset<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>objective(trial)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+"dspy.teleprompt.random_search.BootstrapFewShotWithRandomSearch" [color="black", fontcolor="black", label=<{BootstrapFewShotWithRandomSearch|max_errors : int<br ALIGN="LEFT"/>max_labeled_demos : int<br ALIGN="LEFT"/>max_num_samples : int<br ALIGN="LEFT"/>max_rounds : int<br ALIGN="LEFT"/>metric<br ALIGN="LEFT"/>metric_threshold : NoneType<br ALIGN="LEFT"/>min_num_samples : int<br ALIGN="LEFT"/>num_candidate_sets : int<br ALIGN="LEFT"/>num_threads : int<br ALIGN="LEFT"/>stop_at_score : NoneType<br ALIGN="LEFT"/>teacher_settings : dict<br ALIGN="LEFT"/>trainset<br ALIGN="LEFT"/>valset<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+"dspy.teleprompt.copro_optimizer.COPRO" [color="black", fontcolor="black", label=<{COPRO|breadth : int<br ALIGN="LEFT"/>depth : int<br ALIGN="LEFT"/>init_temperature : float<br ALIGN="LEFT"/>metric : NoneType<br ALIGN="LEFT"/>prompt_model : NoneType<br ALIGN="LEFT"/>track_stats : bool<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+"dspy.teleprompt.ensemble.Ensemble" [color="black", fontcolor="black", label=<{Ensemble|deterministic : bool<br ALIGN="LEFT"/>reduce_fn : NoneType<br ALIGN="LEFT"/>size : NoneType<br ALIGN="LEFT"/>|compile(programs)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+"dspy.teleprompt.knn_fewshot.KNNFewShot" [color="black", fontcolor="black", label=<{KNNFewShot|KNN<br ALIGN="LEFT"/>|compile(student)<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+"dspy.teleprompt.mipro_optimizer.MIPRO" [color="black", fontcolor="black", label=<{MIPRO|init_temperature : float<br ALIGN="LEFT"/>metric<br ALIGN="LEFT"/>num_candidates : int<br ALIGN="LEFT"/>observations : NoneType<br ALIGN="LEFT"/>prompt_model : NoneType<br ALIGN="LEFT"/>task_model : NoneType<br ALIGN="LEFT"/>teacher_settings : dict<br ALIGN="LEFT"/>track_stats : bool<br ALIGN="LEFT"/>verbose : bool<br ALIGN="LEFT"/>view_data_batch_size : int<br ALIGN="LEFT"/>|compile(student: dspy.Program): dspy.Program<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+++"dspy.teleprompt.vanilla.LabeledFewShot" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.teleprompt.Teleprompter" [color="black", fontcolor="black", label=<{Teleprompter|<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+"dspy.teleprompt.bootstrap.BootstrapFewShot" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.copro_optimizer.COPRO" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.ensemble.Ensemble" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.finetune.BootstrapFinetune" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.knn_fewshot.KNNFewShot" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.mipro_optimizer.MIPRO" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.random_search.BootstrapFewShotWithRandomSearch" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.teleprompt_optuna.BootstrapFewShotWithOptuna" -> "dspy.teleprompt.teleprompt.Teleprompter" [arrowhead="empty", arrowtail="none"];+"dspy.teleprompt.bootstrap.BootstrapFewShot" -> "dspy.teleprompt.finetune.BootstrapFinetune" [arrowhead="diamond", arrowtail="none", fontcolor="green", label="teleprompter", style="solid"];+++// "dspy.teleprompt.mipro_optimizer.BasicGenerateInstruction" [color="black", fontcolor="black", label=<{BasicGenerateInstruction|basic_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+// "dspy.teleprompt.copro_optimizer.BasicGenerateInstruction" [color="black", fontcolor="black", label=<{BasicGenerateInstruction|basic_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+// "dspy.teleprompt.mipro_optimizer.BasicGenerateInstructionWithDataObservations" [color="black", fontcolor="black", label=<{BasicGenerateInstructionWithDataObservations|basic_instruction : FieldInfo<br ALIGN="LEFT"/>observations : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+// "dspy.teleprompt.mipro_optimizer.BasicGenerateInstructionWithExamples" [color="black", fontcolor="black", label=<{BasicGenerateInstructionWithExamples|basic_instruction : FieldInfo<br ALIGN="LEFT"/>examples : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+// "dspy.teleprompt.mipro_optimizer.BasicGenerateInstructionWithExamplesAndDataObservations" [color="black", fontcolor="black", label=<{BasicGenerateInstructionWithExamplesAndDataObservations|basic_instruction : FieldInfo<br ALIGN="LEFT"/>examples : FieldInfo<br ALIGN="LEFT"/>observations : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];++++// "dspy.teleprompt.mipro_optimizer.DatasetDescriptor" [color="black", fontcolor="black", label=<{DatasetDescriptor|examples : FieldInfo<br ALIGN="LEFT"/>observations : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+// "dspy.teleprompt.mipro_optimizer.DatasetDescriptorWithPriorObservations" [color="black", fontcolor="black", label=<{DatasetDescriptorWithPriorObservations|examples : FieldInfo<br ALIGN="LEFT"/>observations : FieldInfo<br ALIGN="LEFT"/>prior_observations : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];++// "dspy.teleprompt.ensemble.Ensemble.compile.EnsembledProgram" [color="black", fontcolor="black", label=<{EnsembledProgram|programs<br ALIGN="LEFT"/>|forward()<br ALIGN="LEFT"/>}>, shape="record", style="solid"];+// "dspy.teleprompt.copro_optimizer.GenerateInstructionGivenAttempts" [color="black", fontcolor="black", label=<{GenerateInstructionGivenAttempts|attempted_instructions : FieldInfo<br ALIGN="LEFT"/>proposed_instruction : FieldInfo<br ALIGN="LEFT"/>proposed_prefix_for_output_field : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+// "dspy.teleprompt.signature_opt_typed.make_initial_signature.GenerateInstructionInitial" [color="black", fontcolor="black", label=<{GenerateInstructionInitial|basic_signature : T<br ALIGN="LEFT"/>proposed_signatures : list[T]<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+// "dspy.teleprompt.signature_opt_typed.generate_with_avoidance.GenerateSignature" [color="black", fontcolor="black", label=<{GenerateSignature|analysis : str<br ALIGN="LEFT"/>proposed_signature : T<br ALIGN="LEFT"/>score : float<br ALIGN="LEFT"/>|check_signature_not_attempted(s: T): T<br ALIGN="LEFT"/>}>, shape="record", style="solid"];++// "dspy.teleprompt.mipro_optimizer.ObservationSummarizer" [color="black", fontcolor="black", label=<{ObservationSummarizer|observations : FieldInfo<br ALIGN="LEFT"/>summary : FieldInfo<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+// "dspy.teleprompt.signature_opt_typed.OptimizerResult" [color="black", fontcolor="black", label=<{OptimizerResult|program : Module<br ALIGN="LEFT"/>scores : list[float]<br ALIGN="LEFT"/>signatures : list[dict[str, Signature]]<br ALIGN="LEFT"/>|}>, shape="record", style="solid"];+}diff --git a/docs/docs/building-blocks/figures/teleprompter-classes.pdf b/docs/docs/building-blocks/figures/teleprompter-classes.pdfnew file mode 100644index 000000000..4e8ce79ceBinary files /dev/null and b/docs/docs/building-blocks/figures/teleprompter-classes.pdf differdiff --git a/docs/docs/building-blocks/figures/teleprompter-classes.png b/docs/docs/building-blocks/figures/teleprompter-classes.pngnew file mode 100644index 000000000..c774c8b5dBinary files /dev/null and b/docs/docs/building-blocks/figures/teleprompter-classes.png differ