DSPy: The framework for programming—not prompting—language models
Python
updated assertions logic + notebook/caches
Arnav Singhvi9f58eeefd18d
diff --git a/dspy/primitives/assertions.py b/dspy/primitives/assertions.pyindex 19f1aea3a..c9c5c3b46 100644--- a/dspy/primitives/assertions.py+++ b/dspy/primitives/assertions.py@@ -263,6 +263,7 @@ def backtrack_handler(func, bypass_suggest=True, max_backtracks=2):trace_element = dsp.settings.trace[i]mod = trace_element[0]if mod.signature == error_target_module:+ error_state = e.state[i]dspy.settings.backtrack_to = modbreakelse:diff --git a/examples/longformqa/longformqa_assertions.ipynb b/examples/longformqa/longformqa_assertions.ipynbindex cb13909bd..b96e8f56b 100644--- a/examples/longformqa/longformqa_assertions.ipynb+++ b/examples/longformqa/longformqa_assertions.ipynb@@ -41,18 +41,18 @@"metadata": {},"outputs": [],"source": [- "!git clone https://huggingface.co/arnavs11/DSPy_LongFormQA_Cache\n",- "%cd DSPy_LongFormQA_Cache/\n",+ "!git clone https://huggingface.co/arnavs11/DSPy_LongFormQA\n",+ "%cd DSPy_LongFormQA\n","!git checkout master\n","%cd ..\n","import os\n",- "repo_clone_path = '/content/DSPy_LongFormQA_Cache'\n",+ "repo_clone_path = '/content/DSPy_LongFormQA'\n","\n","# Check if '/content' is writable\n","if not os.access('/content', os.W_OK):\n"," # If '/content' is not writable, choose an alternative directory\n"," # Example: using a directory relative to the current working directory\n",- " repo_clone_path = os.path.join(os.getcwd(), 'DSPy_LongFormQA_Cache')\n",+ " repo_clone_path = os.path.join(os.getcwd(), 'DSPy_LongFormQA')\n","\n","# Set up the cache for this notebook\n","os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = repo_clone_path"@@ -131,7 +131,7 @@"source": ["colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n","dspy.settings.configure(rm=colbertv2_wiki17_abstracts)\n",- "turbo = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=500)\n",+ "turbo = dspy.OpenAI(model='gpt-3.5-turbo-0613', max_tokens=500)\n","dspy.settings.configure(lm=turbo, trace=[], temperature=0.7)"]},@@ -419,7 +419,7 @@"metadata": {},"outputs": [],"source": [- "question = devset[6].question\n",+ "question = devset[28].question\n","pred = longformqa(question)\n","citation_faithfulness_score, _ = citation_faithfulness(None, pred, None)\n","\n",@@ -432,7 +432,7 @@"cell_type": "markdown","metadata": {},"source": [- "We can see that the generated paragraph does not properly include citations as intended. In some instances, it follows an incorrect citation format by keeping the referenced source at the beginning of a sentence. In other instances, it does not provide citations for every 1-2 sentences as expected. \n",+ "We can see that the generated paragraph does not properly include citations as intended. In some instances, it follows an incorrect citation format by not keeping the referenced source at before the period at the end of a sentence. In other instances, it does not provide citations for every 1-2 sentences as expected. \n","\n","Additionally, we see that not all included citations are faithful to their preceding text (which can be a result of cited references in an incorrect format)."]@@ -538,7 +538,7 @@"metadata": {},"outputs": [],"source": [- "question = devset[6].question\n",+ "question = devset[28].question\n","pred = longformqa_with_assertions(question)\n","citation_faithfulness_score, _ = citation_faithfulness(None, pred, None)\n","\n",diff --git a/examples/quiz/DSPy_QuizGen_Cache b/examples/quiz/DSPy_QuizGen_Cachedeleted file mode 160000index 27d6d433e..000000000--- a/examples/quiz/DSPy_QuizGen_Cache+++ /dev/null@@ -1 +0,0 @@-Subproject commit 27d6d433e73b91d3cf677ecf1d757813fcbd611ddiff --git a/examples/quiz/quiz_assertions.ipynb b/examples/quiz/quiz_assertions.ipynbindex 917cd6599..4e17c1753 100644--- a/examples/quiz/quiz_assertions.ipynb+++ b/examples/quiz/quiz_assertions.ipynb@@ -96,7 +96,7 @@"source": ["colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n","dspy.settings.configure(rm=colbertv2_wiki17_abstracts)\n",- "turbo = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=500)\n",+ "turbo = dspy.OpenAI(model='gpt-3.5-turbo-0613', max_tokens=500)\n","dspy.settings.configure(lm=turbo, trace=[], temperature=0.7)"]},@@ -290,7 +290,7 @@"metadata": {},"outputs": [],"source": [- "example = devset[38]\n",+ "example = devset[67]\n","quiz_choices = quiz_generator(question=example.question, answer = example.answer)\n","print(f'Generated Quiz Choices: ', quiz_choices.choices)"]@@ -302,7 +302,7 @@"outputs": [],"source": ["for metric in metrics:\n",- " evaluate = Evaluate(metric=metric, devset=devset[38:39], num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate = Evaluate(metric=metric, devset=devset[67:68], num_threads=1, display_progress=True, display_table=5)\n"," evaluate(quiz_generator)"]},@@ -310,7 +310,7 @@"cell_type": "markdown","metadata": {},"source": [- "We see that the generated quiz choices maintain valid JSON formatting and do include the correct answer `\"Exon\"`. However, when looking closely at the other answer choices, we see that the options are not real/common abbreviations of universities, leaving them as uneffective distractor choices. \n",+ "We see that the generated quiz choices do not maintain valid JSON formatting, which violates the valid formatting and correctness check, even though the choices are noted as plausible. We also see that the correct answer is also labeled by \"(Correct Answer)\", which is not the intention of producing good quiz question answer choices. \n","\n","Let's take a look at how we can integrate DSPy Assertions and impose constraints to produce better answer choices."]@@ -386,7 +386,7 @@"metadata": {},"outputs": [],"source": [- "example = devset[38]\n",+ "example = devset[67]\n","quiz_choices = quiz_generator_with_assertions(question=example.question, answer = example.answer)\n","print(f'Generated Quiz Choices: ', quiz_choices.choices)"]@@ -398,7 +398,7 @@"outputs": [],"source": ["for metric in metrics:\n",- " evaluate = Evaluate(metric=metric, devset=devset[38:39], num_threads=1, display_progress=True, display_table=30)\n",+ " evaluate = Evaluate(metric=metric, devset=devset[67:68], num_threads=1, display_progress=True, display_table=30)\n"," evaluate(quiz_generator_with_assertions)"]},@@ -408,7 +408,7 @@"source": ["We see that the quiz choices follow all of our constraints!\n","\n",- "While maintaining the JSON formatting and including the correct answer as before, we now see the answer choices are much more relevant to the question now. Abbreviations like `\"Oxon\", \"Camb\", \"Lond\"` correspond to widely-known universities Oxford, Cambridge and University of London, which is fairly discernible from the abbreviations even if one does not know this information beforehand. This hence makes these choices as much more plausible distractors which forces the test-taker to have had context particularly about the college involved with working on the project. "+ "Not only are the answer choices all plausible, and have removed any indicator of what the correct answer could be, but the answer choices now maintain valid JSON formatting with 4 possible answer choices to the question, which includes the correct answer."]},{diff --git a/examples/tweets/tweets_assertions.ipynb b/examples/tweets/tweets_assertions.ipynbindex 24b7f37d8..f4a42c2a3 100644--- a/examples/tweets/tweets_assertions.ipynb+++ b/examples/tweets/tweets_assertions.ipynb@@ -97,7 +97,7 @@"source": ["colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n","dspy.settings.configure(rm=colbertv2_wiki17_abstracts)\n",- "turbo = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=500)\n",+ "turbo = dspy.OpenAI(model='gpt-3.5-turbo-0613', max_tokens=500)\n","dspy.settings.configure(lm=turbo, trace=[], temperature=0.7)"]},@@ -320,7 +320,7 @@"metadata": {},"outputs": [],"source": [- "example = devset[10]\n",+ "example = devset[118]\n","tweet = tweeter(question=example.question, answer = example.answer)\n","print(f'Generated Tweet: ', tweet.generated_tweet)\n","tweet.context"@@ -333,7 +333,7 @@"outputs": [],"source": ["for metric in metrics:\n",- " evaluate = Evaluate(metric=metric, devset=devset[10:11], num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate = Evaluate(metric=metric, devset=devset[118:119], num_threads=1, display_progress=True, display_table=5)\n"," evaluate(tweeter)"]},@@ -341,9 +341,9 @@"cell_type": "markdown","metadata": {},"source": [- "In this example, we see that the generated tweet is within the length of 280 characters at 120 characters. It does in fact include the correct answer `79 AD`.\n",+ "In this example, we see that the generated tweet is within the length of 280 characters at 151 characters. It does in fact include the correct answer `Hooke`.\n","\n",- "However, it fails to not include hashtags as we see `#MountVesuvius #History` at the end of the tweet. Additionally, the tweet has been determined to not be engaging, which makes sense from an eye-test as it simply states the answer and nothing more. Furthermore, it is determined to not be grounded to its retrieved context.\n",+ "However, it fails to not include hashtags as we see `#knowledge` at the end of the tweet. Additionally, the tweet has been determined to not be engaging, which makes sense from an eye-test as it simply states the answer and nothing more. \n","\n","Let's try to fix this and produce tweets using DSPy Assertions. "]@@ -434,7 +434,7 @@"metadata": {},"outputs": [],"source": [- "example = devset[10]\n",+ "example = devset[118]\n","tweet = tweeter_with_assertions(question=example.question, answer = example.answer)\n","print(f'Generated Tweet: ', tweet.generated_tweet)\n","tweet.context"@@ -447,7 +447,7 @@"outputs": [],"source": ["for metric in metrics:\n",- " evaluate = Evaluate(metric=metric, devset=devset[10:11], num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate = Evaluate(metric=metric, devset=devset[118:119], num_threads=1, display_progress=True, display_table=5)\n"," evaluate(tweeter_with_assertions)"]},