DSPy: The framework for programming—not prompting—language models
Python
mwe touchups
Herumb Shandilya1e92b49fba11
diff --git a/docs/docs/quick-start/minimal-example.mdx b/docs/docs/quick-start/minimal-example.mdxindex 46d472dad..e17ad75ce 100644--- a/docs/docs/quick-start/minimal-example.mdx+++ b/docs/docs/quick-start/minimal-example.mdx@@ -125,43 +125,9 @@ The factorization of \( x^{10} + x^5 + 1 \) is \( (x^2 + x + 1)(x^8 - x^7 + x^5In the prompt above there are 3 fields. The `System Message` contains the formatting information of the module, while the `User Message` contains the input question. The `Response` contains the output generated by the model.-## Compile and Evaluate the Model+## Evaluation Metric-With our simple program in place, let's move on to compiling it with the [`MIPROv2`](/docs/deep-dive/optimizers/miprov2.md) teleprompter:--```python-tp = dspy.MIPROv2(- metric=math_metric,- num_candidates=5,- init_temperature=0.7,- verbose=False,- num_threads=4-)-compiled_cot = tp.compile(- CoT(), trainset=math_trainset,- valset=math_valset,- max_bootstrapped_demos=3,- max_labeled_demos=3,- num_trials=15,- requires_permission_to_run=False-)-```--This should take ~10-15 minutes to complete which you can speed but by increasing the `num_threads` parameter but be careful to not overflow the rate limits of API. To check the cost of optimization you can execute the following:--```python-print(f"Optimization Cost: {sum([lm.history[i]['cost'] for i in range(len(lm.history))])}")-```--`lm.history` stores the history of your model's interactions, including the cost of each interaction. We can just sum the cost of these interaction and get the total cost of optimization, which is ~26 cents.--:::note-Make sure that you clear the `lm.history` using `lm.history.clear()` before doing the optimization to get the accurate cost. This is especial-:::--## Evaluate--Now that we have a compiled (optimized) DSPy program, let's move to evaluating its performance and compare it with the unoptimized model on `math_devset`. Before that, we need a metric to score the solutions. For MATH dataset, it's a bit hard to create a quantitative that works on exact answer/term match. Luckily we can create an LLM judge fairly easily with DSPy:+Now that we have a DSPy program, let's move to evaluating its performance and compare it with the optimized model on `math_devset`. Before that, we need a metric to score the solutions. For MATH dataset, it's a bit hard to create a quantitative that works on exact answer/term match. Luckily we can create an LLM judge fairly easily with DSPy:```pythonllm_judge = dspy.ChainOfThought("question, reference_solution, student_solution -> correct: bool")@@ -170,7 +136,7 @@ def math_metric(gold, pred, trace=None):return llm_judge(question=gold.problem, reference_solution=gold.solution, student_solution=pred.answer).correct```-With the metric in place, we can now evaluate the performance of our unoptimized and optimizer models using the `Evaluate` class:+The `:bool` tag in makes sure that the `correct` attribute in output from the module is a boolean. With the metric in place, we can now evaluate the performance of our unoptimized and optimizer models using the `Evaluate` class:```pythonfrom dspy.evaluate import Evaluate@@ -186,21 +152,62 @@ evaluate = Evaluate(# Evaluate our `optimized_cot` program.print(f"Unoptimized Model Score: {evaluate(CoT())}")-print(f"Optimized Model Score: {evaluate(compiled_cot)}")```**Output:**```Unoptimized Model Score: 70.0+```++That's a fairly decent score from a vanilla CoT pipeline!! Let's optimize it to perform better on this metric!++## Compile and Evaluate the Model++With our simple program in place, let's move on to compiling it with the [`MIPROv2`](/docs/deep-dive/optimizers/miprov2.md) teleprompter:++```python+lm.history.clear() // For fair estimation of optimization cost++tp = dspy.MIPROv2(+ metric=math_metric,+ num_candidates=5,+ init_temperature=0.7,+ verbose=False,+ num_threads=4+)+compiled_cot = tp.compile(+ CoT(), trainset=math_trainset,+ valset=math_valset,+ max_bootstrapped_demos=3,+ max_labeled_demos=3,+ num_trials=15,+ requires_permission_to_run=False+)++print(f"Optimized Model Score: {evaluate(compiled_cot)}")+```++**Output:**+```Optimized Model Score: 76.0```-That's a fairly decent boost with a vanilla CoT pipeline and relatively limited training/ testing dataset!!+That's a fairly decent boost from a vanilla CoT pipeline and relatively limited training/ testing dataset!! This should take ~12-15 minutes to complete which you can speed but by increasing the `num_threads` parameter but be careful to not overflow the rate limits of API. To check the cost of optimization you can execute the following::::noteMATH a popular task that the models are directly optimized for resulting in a relatively low gain, however gains are typically larger for harder tasks.:::+```python+print(f"Optimization Cost: {sum([lm.history[i]['cost'] for i in range(len(lm.history))])}")+```++`lm.history` stores the history of your model's interactions, including the cost of each interaction. We can just sum the cost of these interaction and get the total cost of optimization, which is ~26 cents.++:::note+Make sure that you clear the `lm.history` using `lm.history.clear()` before doing the optimization to get the accurate cost. This is especial+:::+## Inspect the Model's HistoryFor understanding the `compiled_cot` module's interactions, we can review the most recent generations through inspecting the model's history: