Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Update longformqa_assertions.ipynb

arnavsinghvi11a166d6022aa3
diff --git a/examples/longformqa/longformqa_assertions.ipynb b/examples/longformqa/longformqa_assertions.ipynb
index f3ae4326a..8d7d2bdb4 100644
--- a/examples/longformqa/longformqa_assertions.ipynb
+++ b/examples/longformqa/longformqa_assertions.ipynb
@@ -569,7 +569,7 @@
"source": [
"Let's evaluate this with assertions. \n",
"\n",
- "**Note** The pipeline here lies in compiling without assertions to give the teleprompter correct bootstrapped examples by the `answer_correctness` metric and then adding **Assertions** post-compilation. This is represented by passing `LongFormQA()` as the teacher and `LongFormQAWithAssertions()` as the student."
+ "**Note** The pipeline here lies in compiling with **Assertions** to give the teleprompter correct bootstrapped examples by the `answer_correctness` metric and then 'teaching' the student with these correct examples. This is represented by passing `LongFormQA()` as the student and `LongFormQAWithAssertions()` as the teacher."
]
},
{
@@ -580,10 +580,29 @@
"source": [
"longformqa = LongFormQA()\n",
"teleprompter = BootstrapFewShotWithRandomSearch(metric = answer_correctness, max_bootstrapped_demos=2, num_candidate_programs=6)\n",
- "cited_longformqa = teleprompter.compile(student=assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), teacher = longformqa, trainset=trainset, valset=devset[:100])\n",
- "evaluate(cited_longformqa)"
- ]
- }
+ "cited_longformqa_teacher = teleprompter.compile(student=longformqa, teacher = assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), trainset=trainset, valset=devset[:100])",
+ "evaluate(cited_longformqa_teacher)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Note** This pipeline on the other hand sets both the teacher and student with `LongFormQAWithAssertions()` to ensure the teacher correctly instructs the student with the right bootstrapped examples and the student has the chance to self-correct with **Assertions** for any examples that are still deemed incorrect."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "longformqa = LongFormQA()\n",
+ "teleprompter = BootstrapFewShotWithRandomSearch(metric = answer_correctness, max_bootstrapped_demos=2, num_candidate_programs=6)\n",
+ "cited_longformqa_student_teacher = teleprompter.compile(student=assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), teacher = assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), trainset=trainset, valset=devset[:100])",
+ "evaluate(cited_longformqa_student_teacher)"
+ ]
+ },
],
"metadata": {
"kernelspec": {