DSPy: The framework for programming—not prompting—language models
Python
documentation and QuizGen+TweetGen notebooks
Arnav Singhvi44329bbf5d42
diff --git a/docs/assertions.md b/docs/assertions.mdnew file mode 100644index 000000000..99da7de07--- /dev/null+++ b/docs/assertions.md@@ -0,0 +1,253 @@+# DSPy Assertions+## Introduction++Language models (LMs) have transformed how we interact with machine learning, offering vast capabilities in natural language understanding and generation. However, ensuring these models adhere to domain-specific constraints remains a challenge. Despite the growth of techniques like fine-tuning or “prompt engineering”, these approaches are extremely tedious and rely on heavy, manual hand-waving to guide the LMs in adhering to specific constraints. Even DSPy's modularity of programming prompting pipelines lacks mechanisms to effectively and automatically enforce these constraints.++To address this, we introduce DSPy Assertions, a feature within the DSPy framework designed to automate the enforcement of computational constraints on LMs. DSPy Assertions empower developers to guide LMs towards desired outcomes with minimal manual intervention, enhancing the reliability, predictability, and correctness of LM outputs.++### dspy.Assert and dspy.Suggest API++We introduce two primary constructs within DSPy Assertions:++- **`dspy.Assert`**:+ - **Parameters**:+ - `constraint (bool)`: Outcome of Python-defined boolean validation check.+ - `msg (Optional[str])`: User-defined error message providing feedback or correction guidance.+ - `backtrack (Optional[module])`: Specifies target module for retry attempts upon constraint failure.+ - **Behavior**: Initiates retry upon failure, dynamically adjusting the pipeline's execution. If failures persist, it halts execution and raises a `dspy.AssertionError`.++- **`dspy.Suggest`**:+ - **Parameters**: Similar to `dspy.Assert`.+ - **Behavior**: Encourages self-refinement through retries without enforcing hard stops. Logs failures after maximum backtracking attempts and continues execution.++- **dspy.Assert vs. Python Assertions**: Unlike conventional Python `assert` statements that terminate the program upon failure, `dspy.Assert` conducts a sophisticated retry mechanism, allowing the pipeline to adjust.++Specifically, when a constraint is not met:++- Backtracking Mechanism: An under-the-hood backtracking is initiated, offering the model a chance to self-refine and proceed, which is done through+- Dynamic Signature Modification: internally modifying your DSPy program’s Signature by adding the following fields:+ - Past Output: your model's past output that did not pass the validation_fn+ - Instruction: your user-defined feedback message on what went wrong and what possibly to fix++If the error continues past the `max_backtracking_attempts`, then `dspy.Assert` will halt the pipeline execution, altering you with an `dspy.AssertionError`. This ensures your program doesn't continue executing with “bad” LM behavior and immediately highlights sample failure outputs for user assessment.++- **dspy.Suggest vs. dspy.Assert**: `dspy.Suggest` on the other hand offers a softer approach. It maintains the same retry backtracking as `dspy.Assert` but instead serves as a gentle nudger. If the model outputs cannot pass the model constraints after the `max_backtracking_attempts`, `dspy.Suggest` will log the persistent failure and continue execution of the program on the rest of the data. This ensures the LM pipeline works in a "best-effort" manner without halting execution.++- **`dspy.Suggestions`** are best utilized as "helpers" during the evaluation phase, offering guidance and potential corrections without halting the pipeline.+- **`dspy.Assertions`** are recommended during the development stage as "checkers" to ensure the LM behaves as expected, providing a robust mechanism for identifying and addressing errors early in the development cycle.+++## Use Case: Including Assertions in DSPy Programs++We start with using an example of a multi-hop QA SimplifiedBaleen pipeline as defined in the intro walkthrough.++```python+class SimplifiedBaleen(dspy.Module):+ def __init__(self, passages_per_hop=2, max_hops=2):+ super().__init__()++ self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]+ self.retrieve = dspy.Retrieve(k=passages_per_hop)+ self.generate_answer = dspy.ChainOfThought(GenerateAnswer)+ self.max_hops = max_hops++ def forward(self, question):+ context = []+ prev_queries = [question]++ for hop in range(self.max_hops):+ query = self.generate_query[hop](context=context, question=question).query+ prev_queries.append(query)+ passages = self.retrieve(query).passages+ context = deduplicate(context + passages)++ pred = self.generate_answer(context=context, question=question)+ pred = dspy.Prediction(context=context, answer=pred.answer)+ return pred++baleen = SimplifiedBaleen()++baleen(question = "Which award did Gary Zukav's first book receive?")+```++To include DSPy Assertions, we simply define our validation functions and declare our assertions following the respective model generation.++For this use case, suppose we want to impose the following constraints:+ 1. Length - each query should be less than 100 characters+ 2. Uniqueness - each generated query should differ from previously-generated queries.++We can define these validation checks as boolean functions:++```python+#simplistic boolean check for query length+len(query) <= 100++#Python function for validating distinct queries+def validate_query_distinction_local(previous_queries, query):+ """check if query is distinct from previous queries"""+ if previous_queries == []:+ return True+ if dspy.evaluate.answer_exact_match_str(query, previous_queries, frac=0.8):+ return False+ return True+```++We can declare these validation checks through `dspy.Suggest` statements (as we want to test the program in a best-effort demonstration). We want to keep these after the query generation `query = self.generate_query[hop](context=context, question=question).query`.++```python+dspy.Suggest(+ len(query) <= 100,+ "Query should be short and less than 100 characters",+)++dspy.Suggest(+ validate_query_distinction_local(prev_queries, query),+ "Query should be distinct from: "+ + "; ".join(f"{i+1}) {q}" for i, q in enumerate(prev_queries)),+)+```++It is recommended to define a program with assertions separately than your original program if you are doing comparative evaluation for the effect of assertions. If not, feel free to set Assertions away!++Let's take a look at how the SimplifiedBaleen program will look with Assertions included:++```python+class SimplifiedBaleenAssertions(dspy.Module):+ def __init__(self, passages_per_hop=2, max_hops=2):+ super().__init__()+ self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]+ self.retrieve = dspy.Retrieve(k=passages_per_hop)+ self.generate_answer = dspy.ChainOfThought(GenerateAnswer)+ self.max_hops = max_hops++ def forward(self, question):+ context = []+ prev_queries = [question]++ for hop in range(self.max_hops):+ query = self.generate_query[hop](context=context, question=question).query++ dspy.Suggest(+ len(query) <= 100,+ "Query should be short and less than 100 characters",+ )++ dspy.Suggest(+ validate_query_distinction_local(prev_queries, query),+ "Query should be distinct from: "+ + "; ".join(f"{i+1}) {q}" for i, q in enumerate(prev_queries)),+ )++ prev_queries.append(query)+ passages = self.retrieve(query).passages+ context = deduplicate(context + passages)++ if all_queries_distinct(prev_queries):+ self.passed_suggestions += 1++ pred = self.generate_answer(context=context, question=question)+ pred = dspy.Prediction(context=context, answer=pred.answer)+ return pred+```++Now calling programs with DSPy Assertions requires one last step, and that is transforming the program to wrap it with internal assertions backtracking and Retry logic.++```python+from dspy.primitives.assertions import assert_transform_module, backtrack_handler++baleen_with_assertions = assert_transform_module(SimplifiedBaleenAssertions().map_named_predictors(Retry), backtrack_handler)+```++Alternatively, you can also directly call `activate_assertions` on the program with `dspy.Assert/Suggest` statements:++```python+baleen_with_assertions = SimplifiedBaleenAssertions().activate_assertions()+```++Now let's take a look at the internal LM backtracking by inspecting the history of the LM query generations. Here we see that when a query fails to pass the validation check of being less than 100 characters, its internal `GenerateSearchQuery` signature is dynamically modified during the backtracking+Retry process to include the past query and the corresponding user-defined instruction: `"Query should be short and less than 100 characters"`.+++```+Write a simple search query that will help answer a complex question.++---++Follow the following format.++Context: may contain relevant facts++Question: ${question}++Reasoning: Let's think step by step in order to ${produce the query}. We ...++Query: ${query}++---++Context:+[1] «Kerry Condon | Kerry Condon (born 4 January 1983) is [...]»+[2] «Corona Riccardo | Corona Riccardo (c. 1878October 15, 1917) was [...]»++Question: Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of "Hamlet." ?++Reasoning: Let's think step by step in order to find the answer to this question. First, we need to identify the actress who played Ophelia in a Royal Shakespeare Company production of "Hamlet." Then, we need to find out if this actress also acted in the short film "The Shore."++Query: "actress who played Ophelia in Royal Shakespeare Company production of Hamlet" + "actress in short film The Shore"++++Write a simple search query that will help answer a complex question.++---++Follow the following format.++Context: may contain relevant facts++Question: ${question}++Past Query: past output with errors++Instructions: Some instructions you must satisfy++Query: ${query}++---++Context:+[1] «Kerry Condon | Kerry Condon (born 4 January 1983) is an Irish television and film actress, best known for her role as Octavia of the Julii in the HBO/BBC series "Rome," as Stacey Ehrmantraut in AMC's "Better Call Saul" and as the voice of F.R.I.D.A.Y. in various films in the Marvel Cinematic Universe. She is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of "Hamlet."»+[2] «Corona Riccardo | Corona Riccardo (c. 1878October 15, 1917) was an Italian born American actress who had a brief Broadway stage career before leaving to become a wife and mother. Born in Naples she came to acting in 1894 playing a Mexican girl in a play at the Empire Theatre. Wilson Barrett engaged her for a role in his play "The Sign of the Cross" which he took on tour of the United States. Riccardo played the role of Ancaria and later played Berenice in the same play. Robert B. Mantell in 1898 who struck by her beauty also cast her in two Shakespeare plays, "Romeo and Juliet" and "Othello". Author Lewis Strang writing in 1899 said Riccardo was the most promising actress in America at the time. Towards the end of 1898 Mantell chose her for another Shakespeare part, Ophelia im Hamlet. Afterwards she was due to join Augustin Daly's Theatre Company but Daly died in 1899. In 1899 she gained her biggest fame by playing Iras in the first stage production of Ben-Hur.»++Question: Who acted in the shot film The Shore and is also the youngest actress ever to play Ophelia in a Royal Shakespeare Company production of "Hamlet." ?++Past Query: "actress who played Ophelia in Royal Shakespeare Company production of Hamlet" + "actress in short film The Shore"++Instructions: Query should be short and less than 100 characters++Query: "actress Ophelia RSC Hamlet" + "actress The Shore"++```+++## Assertion-Driven Optimizations++DSPy Assertions work with optimizations that DSPy offers, particularly with `BootstrapFewShotWithRandomSearch`, including the following settings:++- Compilation with Assertions+ This includes assertion-driven example bootstrapping and counterexample bootstrapping during compilation. The teacher model for bootstrapping few-shot demonstrations can make use of DSPy Assertions to offer robust bootstrapped examples for the student model to learn from during inference.+- Compilation + Inference with Assertions+ -This includes assertion-driven optimizations in both compilation and inference. Now the teacher model offers assertion-driven examples but the student can further optimize with assertions of its own during inference time.+```python+teleprompter = BootstrapFewShotWithRandomSearch(+ metric=validate_context_and_answer_and_hops,+ max_bootstrapped_demos=max_bootstrapped_demos,+ num_candidate_programs=6,+)++#Compilation with Assertions+compiled_with_assertions_baleen = teleprompter.compile(student = baleen, teacher = baleen_with_assertions, trainset = trainset, valset = devset)++#Compilation + Inference with Assertions+compiled_baleen_with_assertions = teleprompter.compile(student=baleen_with_assertions, teacher = baleen_with_assertions, trainset=trainset, valset=devset)++```diff --git a/examples/longformqa/longformqa_assertions.ipynb b/examples/longformqa/longformqa_assertions.ipynbindex 75403383e..4130a9db7 100644--- a/examples/longformqa/longformqa_assertions.ipynb+++ b/examples/longformqa/longformqa_assertions.ipynb@@ -6,7 +6,9 @@"source": ["<img src=\"../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n","\n",- "## **DSPy Assertions**: Asserting Computational Constraints on Foundation Models"+ "## **DSPy Assertions**: Asserting Computational Constraints on Foundation \n",+ "\n",+ "### **LongFormQA**: Generating long-form length responses to answer questions"]},{@@ -592,24 +594,24 @@]},{- "cell_type": "markdown",- "metadata": {},- "source": [- "**Note** This pipeline on the other hand sets both the teacher and student with `LongFormQAWithAssertions()` to ensure the teacher correctly instructs the student with the right bootstrapped examples and the student has the chance to self-correct with **Assertions** for any examples that are still deemed incorrect."- ]- },- {- "cell_type": "code",- "execution_count": null,- "metadata": {},- "outputs": [],- "source": [- "longformqa = LongFormQA()\n",- "teleprompter = BootstrapFewShotWithRandomSearch(metric = answer_correctness, max_bootstrapped_demos=2, num_candidate_programs=6)\n",- "cited_longformqa_student_teacher = teleprompter.compile(student=assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), teacher = assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), trainset=trainset, valset=devset[:100])\n",- "evaluate(cited_longformqa_student_teacher)"- ]- }+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "**Note** This pipeline on the other hand sets both the teacher and student with `LongFormQAWithAssertions()` to ensure the teacher correctly instructs the student with the right bootstrapped examples and the student has the chance to self-correct with **Assertions** for any examples that are still deemed incorrect."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "longformqa = LongFormQA()\n",+ "teleprompter = BootstrapFewShotWithRandomSearch(metric = answer_correctness, max_bootstrapped_demos=2, num_candidate_programs=6)\n",+ "cited_longformqa_student_teacher = teleprompter.compile(student=assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), teacher = assert_transform_module(LongFormQAWithAssertions().map_named_predictors(Retry), backtrack_handler), trainset=trainset, valset=devset[:100])\n",+ "evaluate(cited_longformqa_student_teacher)"+ ]+ }],"metadata": {"kernelspec": {diff --git a/examples/quiz/quiz_assertions.ipynb b/examples/quiz/quiz_assertions.ipynbnew file mode 100644index 000000000..8cdbdbcc9--- /dev/null+++ b/examples/quiz/quiz_assertions.ipynb@@ -0,0 +1,495 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "<img src=\"../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",+ "\n",+ "## **DSPy Assertions**: Asserting Computational Constraints on Foundation Models\n",+ "\n",+ "### **QuizGen**: Generating multiple choice quiz questions"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "[<img align=\"center\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/quiz/quiz_assertions.ipynb)\n",+ "\n",+ "\n",+ "This notebook highlights an example of [**DSPy Assertions**](../../docs/assertions.md), allowing for declaration of computational constraints within DSPy programs. \n",+ "\n",+ "\n",+ "This notebook builds upon the foundational concepts of the **DSPy** framework. Prerequisites of following this notebook is having gone through the [DSPy tutorial](../../intro.ipynb), the [**DSPy Assertions documentation**](../../docs/assertions.md) and the introductory DSPy Assertions [tutorial on LongFormQA](../longformqa/longformqa_assertions.ipynb).\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "!git clone https://huggingface.co/arnavs11/DSPy_QuizGen_Cache\n",+ "%cd DSPy_QuizGen_Cache/\n",+ "!git checkout master\n",+ "%cd ..\n",+ "import os\n",+ "repo_clone_path = '/content/DSPy_QuizGen_Cache'\n",+ "\n",+ "# Set up the cache for this notebook\n",+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = repo_clone_path"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "%load_ext autoreload\n",+ "%autoreload 2\n",+ "\n",+ "import sys\n",+ "import os\n",+ "import regex as re\n",+ "import json\n",+ "\n",+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",+ " import google.colab\n",+ " repo_path = 'dspy'\n",+ " \n",+ " !git -C $repo_path pull origin || git clone https://github.com/stanfordnlp/dspy $repo_path\n",+ "except:\n",+ " repo_path = '.'\n",+ "\n",+ "if repo_path not in sys.path:\n",+ " sys.path.append(repo_path)\n",+ "\n",+ "\n",+ "import pkg_resources # Install the package if it's not installed\n",+ "if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",+ " !pip install -U pip\n",+ " !pip install dspy-ai\n",+ " !pip install openai~=0.28.1\n",+ " !pip install -e $repo_path\n",+ "\n",+ "import dspy\n",+ "from dspy.predict import Retry\n",+ "from dspy.datasets import HotPotQA\n",+ "from dspy.teleprompt import BootstrapFewShotWithRandomSearch\n",+ "from dspy.evaluate.evaluate import Evaluate\n",+ "from dspy.primitives.assertions import assert_transform_module, backtrack_handler"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",+ "dspy.settings.configure(rm=colbertv2_wiki17_abstracts)\n",+ "turbo = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=500)\n",+ "dspy.settings.configure(lm=turbo, trace=[], temperature=0.7)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "dataset = HotPotQA(train_seed=1, train_size=300, eval_seed=2023, dev_size=300, test_size=0, keep_details=True)\n",+ "trainset = [x.with_inputs('question', 'answer') for x in dataset.train]\n",+ "devset = [x.with_inputs('question', 'answer') for x in dataset.dev]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 3] QuizGen\n",+ "\n",+ "Let's introduce a new task: QuizGen. \n",+ "\n",+ "QuizGen takes HotPotQA data points and turns them into multiple choice quiz questions with the corresponding options. Each set of options for the question is produced in a JSON key-value pair format. For this case, we specify the generation of 4 choices."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "With this program, we aim to generate quiz choices that adhere to the following guidelines:\n",+ "1. The generated choices are in a JSON format.\n",+ "2. The generated choices include the correct answer.\n",+ "3. The generated choices include plausible distractor options besides the correct answer."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class GenerateAnswerChoices(dspy.Signature):\n",+ " \"\"\"Generate answer choices in JSON format that include the correct answer and plausible distractors for the specified question.\"\"\"\n",+ " question = dspy.InputField()\n",+ " correct_answer = dspy.InputField()\n",+ " number_of_choices = dspy.InputField()\n",+ " answer_choices = dspy.OutputField(desc='JSON key-value pairs')\n",+ "\n",+ "class QuizAnswerGenerator(dspy.Module):\n",+ " def __init__(self):\n",+ " super().__init__()\n",+ " self.generate_choices = dspy.ChainOfThought(GenerateAnswerChoices)\n",+ "\n",+ " def forward(self, question, answer):\n",+ " choices = self.generate_choices(question=question, correct_answer=answer, number_of_choices=number_of_choices).answer_choices\n",+ " return dspy.Prediction(choices = choices)\n",+ "\n",+ "number_of_choices = '4'\n",+ "quiz_generator = QuizAnswerGenerator()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 4] Evaluation - Intrinsic and Extrinsic\n",+ "\n",+ "#### Intrinsic Metrics: passing internal computational constraints is the goal \n",+ "\n",+ "**Valid Formatting** - The outputted answer choices should be in JSON format which is verified after parsing the key-value pairs.\n",+ "\n",+ "**Correct Answer Inclusion** - This is a general check to ensure the generated quiz choices actually include the correct answer to the question.\n",+ "\n",+ "**Plausible Distractors** - This validation is to check that the generated choices include distractor answer options that are reasonable options as answers to the question. We define and call another **DSPy** program: ``Predict`` on ``AssessQuizChoices``, relying on the same LM to answer the question: `\"Are the distractors in the answer choices plausible and not easily identifiable as incorrect?\"`"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def format_checker(choice_string):\n",+ " try:\n",+ " choices = json.loads(choice_string)\n",+ " if isinstance(choices, dict) and all(isinstance(key, str) and isinstance(value, str) for key, value in choices.items()):\n",+ " return True\n",+ " except json.JSONDecodeError:\n",+ " return False\n",+ "\n",+ " return False\n",+ "\n",+ "def is_correct_answer_included(correct_answer, generated_choices):\n",+ " try:\n",+ " choices_dict = json.loads(generated_choices)\n",+ " return correct_answer in choices_dict.values()\n",+ " except json.JSONDecodeError:\n",+ " return False\n",+ "\n",+ "def is_plausibility_yes(assessment_answer):\n",+ " \"\"\"Check if the first word of the assessment answer is 'yes'.\"\"\"\n",+ " return assessment_answer.split()[0].lower() == 'yes'\n",+ " \n",+ "class AssessQuizChoices(dspy.Signature):\n",+ " \"\"\"Assess the quality of quiz answer choices along specified dimensions.\"\"\"\n",+ " \n",+ " question = dspy.InputField()\n",+ " answer_choices = dspy.InputField()\n",+ " assessment_question = dspy.InputField()\n",+ " assessment_answer = dspy.OutputField(desc=\"Yes or No\")\n",+ " \n",+ "def format_valid_metric(gold, pred, trace=None):\n",+ " generated_choices = pred.choices\n",+ " format_valid = format_checker(generated_choices)\n",+ " score = format_valid\n",+ " return score\n",+ "\n",+ "def is_correct_metric(gold, pred, trace=None):\n",+ " correct_answer, generated_choices = gold.answer, pred.choices\n",+ " correct_included = is_correct_answer_included(correct_answer, generated_choices)\n",+ " score = correct_included\n",+ " return score\n",+ "\n",+ "def plausibility_metric(gold, pred, trace=None):\n",+ " question, generated_choices = gold.question, pred.choices\n",+ " plausibility_question = \"Are the distractors in the answer choices plausible and not easily identifiable as incorrect?\"\n",+ " plausibility_assessment = dspy.Predict(AssessQuizChoices)(question=question, answer_choices=generated_choices, assessment_question=plausibility_question)\n",+ " plausibility_result = plausibility_assessment.assessment_answer.split()[0].lower() == 'yes'\n",+ " score = plausibility_result\n",+ " return score"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "#### Extrinsic Metrics: Assess the overall quality and effectiveness of generated output on downstream task\n",+ "\n",+ "The extrinsic metric is defined as the overall quality of the generated quiz choices and is evaluated over a composite metric, accounting for these constraints.\n",+ "\n",+ "The composite metric maintains the core intrinsic metrics required for producing a valid set of quiz choices in validating valid formatting and correct answere icnlusion, and the overall composite metric returns an averaged score over the 3 intrinsic metrics."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def overall_metric(gold, pred, trace=None):\n",+ " question, correct_answer, generated_choices = gold.question, gold.answer, pred.choices\n",+ " format_valid = format_checker(generated_choices)\n",+ " correct_included = is_correct_answer_included(correct_answer, generated_choices)\n",+ " plausibility_question = \"Are the distractors in the answer choices plausible and not easily identifiable as incorrect?\"\n",+ " plausibility_assessment = dspy.Predict(AssessQuizChoices)(question=question, answer_choices=generated_choices, assessment_question=plausibility_question)\n",+ " plausibility_result = plausibility_assessment.assessment_answer.split()[0].lower() == 'yes'\n",+ " score = (format_valid + correct_included + plausibility_result) / 3.0 if correct_included and format_valid else 0\n",+ " return score"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We hence define the evaluation as follows:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "metrics = [format_valid_metric, is_correct_metric, plausibility_metric, overall_metric]\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(quiz_generator)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's take a look at an example quiz choice generation:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "example = devset[38]\n",+ "quiz_choices = quiz_generator(question=example.question, answer = example.answer)\n",+ "print(f'Generated Quiz Choices: ', quiz_choices.choices)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset[38:39], num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(quiz_generator)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We see that the generated quiz choices maintain valid JSON formatting and do include the correct answer `\"Exon\"`. However, when looking closely at the other answer choices, we see that the options are not real/common abbreviations of universities, leaving them as uneffective distractor choices. \n",+ "\n",+ "Let's take a look at how we can integrate DSPy Assertions and impose constraints to produce better answer choices."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 5] Introducing Assertions: QuizAnswerGeneratorWithAssertions\n",+ "Let's include assertions that simply reiterate our computational constraints within DSPy Assertion semantics. \n",+ "\n",+ "In the first **Assertion**, we check for if the generated quiz choices are in JSON format and if not, assert: **\"The format of the answer choices should be in JSON format. Please revise accordingly.\"**\n",+ "\n",+ "We also check for if the set of quiz choices includes the correct answer and ensure this if violated with the feedback message: **\"The answer choices do not include the correct answer to the question. Please revise accordingly.\"**\n",+ "\n",+ "Lastly, we assess if the plausible distractor choices are indeed good distractor options and if not, assert: **\"The answer choices are not plausible distractors or are too easily identifiable as incorrect. Please revise to provide more challenging and plausible distractors.\"**"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class QuizAnswerGeneratorWithAssertions(dspy.Module):\n",+ " def __init__(self):\n",+ " super().__init__()\n",+ " self.generate_choices = dspy.ChainOfThought(GenerateAnswerChoices)\n",+ "\n",+ " def forward(self, question, answer):\n",+ " choice_string = self.generate_choices(question=question, correct_answer=answer, number_of_choices=number_of_choices).answer_choices\n",+ " dspy.Suggest(format_checker(choice_string), \"The format of the answer choices should be in JSON format. Please revise accordingly.\", target_module=GenerateAnswerChoices)\n",+ " dspy.Suggest(is_correct_answer_included(answer, choice_string), \"The answer choices do not include the correct answer to the question. Please revise accordingly.\", target_module=GenerateAnswerChoices)\n",+ " plausibility_question = \"Are the distractors in the answer choices plausible and not easily identifiable as incorrect?\"\n",+ " plausibility_assessment = dspy.Predict(AssessQuizChoices)(question=question, answer_choices=choice_string, assessment_question=plausibility_question)\n",+ " dspy.Suggest(is_plausibility_yes(plausibility_assessment.assessment_answer), \"The answer choices are not plausible distractors or are too easily identifiable as incorrect. Please revise to provide more challenging and plausible distractors.\", target_module=GenerateAnswerChoices)\n",+ " return dspy.Prediction(choices = choice_string)\n",+ "\n",+ "number_of_choices = '4'\n",+ "quiz_generator_with_assertions = assert_transform_module(QuizAnswerGeneratorWithAssertions().map_named_predictors(Retry), backtrack_handler) "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's evaluate the `QuizAnswerGeneratorWithAssertions` now over the devset."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "metrics = [format_valid_metric, is_correct_metric, plausibility_metric, overall_metric]\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(quiz_generator_with_assertions)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now let's take a look at how our generated set of quiz choices has improved with the addition of assertions."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "example = devset[38]\n",+ "quiz_choices = quiz_generator_with_assertions(question=example.question, answer = example.answer)\n",+ "print(f'Generated Quiz Choices: ', quiz_choices.choices)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset[38:39], num_threads=1, display_progress=True, display_table=30)\n",+ " evaluate(quiz_generator_with_assertions)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We see that the quiz choices follow all of our constraints!\n",+ "\n",+ "While maintaining the JSON formatting and including the correct answer as before, we now see the answer choices are much more relevant to the question now. Abbreviations like `\"Oxon\", \"Camb\", \"Lond\"` correspond to widely-known universities Oxford, Cambridge and University of London, which is fairly discernible from the abbreviations even if one does not know this information beforehand. This hence makes these choices as much more plausible distractors which forces the test-taker to have had context particularly about the college involved with working on the project. "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 6] Compilation With Assertions\n",+ "\n",+ "We can leverage **DSPy**'s`BootstrapFewShotWithRandomSearch` optimizer, to automatically generate few-shot demonstrations and conduct a random search over the candidates to output the best compiled program. We evaluate this over the `final_metric` composite metric. \n",+ "\n",+ "We can first evaluate this on `QuizAnswerGenerator` to see how compilation performs without the inclusion of assertions. "+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "teleprompter = BootstrapFewShotWithRandomSearch(metric = overall_metric, max_bootstrapped_demos=2, num_candidate_programs=6)\n",+ "compiled_quiz_generator = teleprompter.compile(student = quiz_generator, teacher = quiz_generator, trainset=trainset, valset=devset[:100])\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(compiled_quiz_generator)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now we test the compilation on 2 settings with assertions:\n",+ "\n",+ "**Compilation with Assertions**: assertion-driven example bootstrapping and counterexample bootstrapping during compilation. Teacher has assertions while the student does not as the student learns from the teacher's assertion-driven bootstrapped examples. \n",+ "\n",+ "**Compilation + Inference with Assertions**: assertion-driven optimizations for both the teacher and student to offer enhanced assertion-driven outputs during both compilation and inference."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "teleprompter = BootstrapFewShotWithRandomSearch(metric = overall_metric, max_bootstrapped_demos=2, num_candidate_programs=6)\n",+ "compiled_with_assertions_quiz_generator = teleprompter.compile(student=quiz_generator, teacher = quiz_generator_with_assertions, trainset=trainset, valset=devset[:100])\n",+ "\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(compiled_with_assertions_quiz_generator)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "teleprompter = BootstrapFewShotWithRandomSearch(metric = overall_metric, max_bootstrapped_demos=2, num_candidate_programs=6)\n",+ "compiled_quiz_generator_with_assertions = teleprompter.compile(student=quiz_generator_with_assertions, teacher = quiz_generator_with_assertions, trainset=trainset, valset=devset[:100])\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(compiled_quiz_generator_with_assertions)"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "dspy_dev",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.13"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}diff --git a/examples/tweets/tweets_assertions.ipynb b/examples/tweets/tweets_assertions.ipynbnew file mode 100644index 000000000..91dffc47b--- /dev/null+++ b/examples/tweets/tweets_assertions.ipynb@@ -0,0 +1,544 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "<img src=\"../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",+ "\n",+ "## **DSPy Assertions**: Asserting Computational Constraints on Foundation Models\n",+ "\n",+ "### **TweetGen**: Generating tweets to answer questions"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "[<img align=\"center\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/tweets/tweets_assertions.ipynb)\n",+ "\n",+ "\n",+ "This notebook highlights an example of [**DSPy Assertions**](../../docs/assertions.md), allowing for declaration of computational constraints within DSPy programs. \n",+ "\n",+ "\n",+ "This notebook builds upon the foundational concepts of the **DSPy** framework. Prerequisites of following this notebook is having gone through the [DSPy tutorial](../../intro.ipynb), the [**DSPy Assertions documentation**](../../docs/assertions.md) and the introductory DSPy Assertions [tutorial on LongFormQA](../longformqa/longformqa_assertions.ipynb).\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "!git clone https://huggingface.co/arnavs11/DSPy_TweetGen_Cache\n",+ "%cd DSPy_TweetGen_Cache/\n",+ "!git checkout master\n",+ "%cd ..\n",+ "import os\n",+ "repo_clone_path = '/content/DSPy_TweetGen_Cache'\n",+ "\n",+ "# Set up the cache for this notebook\n",+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = repo_clone_path"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "%load_ext autoreload\n",+ "%autoreload 2\n",+ "\n",+ "import sys\n",+ "import os\n",+ "import regex as re\n",+ "import json\n",+ "\n",+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",+ " import google.colab\n",+ " repo_path = 'dspy'\n",+ " \n",+ " !git -C $repo_path pull origin || git clone https://github.com/stanfordnlp/dspy $repo_path\n",+ "except:\n",+ " repo_path = '.'\n",+ "\n",+ "if repo_path not in sys.path:\n",+ " sys.path.append(repo_path)\n",+ "\n",+ "\n",+ "import pkg_resources # Install the package if it's not installed\n",+ "if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",+ " !pip install -U pip\n",+ " !pip install dspy-ai\n",+ " !pip install openai~=0.28.1\n",+ " !pip install -e $repo_path\n",+ "\n",+ "import dspy\n",+ "from dspy.predict import Retry\n",+ "from dspy.datasets import HotPotQA\n",+ "from dspy.teleprompt import BootstrapFewShotWithRandomSearch\n",+ "from dsp.utils import deduplicate\n",+ "from dspy.evaluate.evaluate import Evaluate\n",+ "from dspy.primitives.assertions import assert_transform_module, backtrack_handler"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",+ "dspy.settings.configure(rm=colbertv2_wiki17_abstracts)\n",+ "turbo = dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=500)\n",+ "dspy.settings.configure(lm=turbo, trace=[], temperature=0.7)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "dataset = HotPotQA(train_seed=1, train_size=300, eval_seed=2023, dev_size=300, test_size=0, keep_details=True)\n",+ "trainset = [x.with_inputs('question', 'answer') for x in dataset.train]\n",+ "devset = [x.with_inputs('question', 'answer') for x in dataset.dev]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 3] TweetGen\n",+ "\n",+ "Let's introduce a new task: TweetGen. We extend the `Multi-Hop QA` program, but now aim to present the answer generation in the form of a tweet. \n",+ "\n",+ "The `Tweeter` module captures the iterative multi-hop generation process from `Multi-Hop QA` in query generation, passage retrieval, and context assembly. The `GenerateTweet` layer now utilizes the context alongside the question to generate an tweet that effectively answers the question."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "With this program, we aim to generate tweets that adhere to the following guidelines:\n",+ "1. The tweet has no hashtags. \n",+ "2. The tweet includes the correct answer\n",+ "3. The tweet is within a character limit. \n",+ "4. The tweet is engaging\n",+ "5. The tweet is faithful"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class GenerateSearchQuery(dspy.Signature):\n",+ " \"\"\"Write a simple search query that will help answer a complex question.\"\"\"\n",+ " context = dspy.InputField(desc=\"may contain relevant facts\")\n",+ " question = dspy.InputField()\n",+ " query = dspy.OutputField()\n",+ "\n",+ "class GenerateTweet(dspy.Signature):\n",+ " \"\"\"Generate an engaging tweet that effectively answers a question staying faithful to the context, is less than 280 characters, and has no hashtags.\"\"\"\n",+ " question = dspy.InputField()\n",+ " context = dspy.InputField(desc=\"may contain relevant facts\")\n",+ " tweet = dspy.OutputField()\n",+ "\n",+ "class Tweeter(dspy.Module):\n",+ " def __init__(self):\n",+ " super().__init__()\n",+ " self.generate_tweet = dspy.ChainOfThought(GenerateTweet)\n",+ "\n",+ " def forward(self, question, answer):\n",+ " context = []\n",+ " max_hops=2\n",+ " passages_per_hop=3\n",+ " generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]\n",+ " retrieve = dspy.Retrieve(k=passages_per_hop)\n",+ " for hop in range(max_hops):\n",+ " query = generate_query[hop](context=context, question=question).query\n",+ " passages = retrieve(query).passages\n",+ " context = deduplicate(context + passages)\n",+ " generated_tweet = self.generate_tweet(question=question, context=context).tweet\n",+ " return dspy.Prediction(generated_tweet=generated_tweet, context=context)\n",+ " \n",+ "tweeter = Tweeter()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 4] Evaluation - Intrinsic and Extrinsic\n",+ "\n",+ "#### Intrinsic Metrics: passing internal computational constraints is the goal \n",+ "\n",+ "**No Hashtags** - This is a user-personalized constraint to test how well the model can follow a specific, yet simple guideline of not including any hashtags within the generated tweet.\n",+ "\n",+ "**Correct Answer Inclusion** - This is a general check to ensure the tweet indeed has the correct answer to the question.\n",+ "\n",+ "**Within Length** - This check follows Twitter platform guidelines of 280 character limits per tweet.\n",+ "\n",+ "**Engagement** - To verify the engagement quality of the tweet, we define and call another **DSPy** program: ``Predict`` on ``AssessTweet``, relying on the same LM to answer the question: `\"Does the assessed text make for a self-contained, engaging tweet? Say no if it is not engaging.\"`\n",+ "\n",+ "**Faithfulness** - To verify the faithfulness of the tweet to its referenced context, we similarly use `AssessTweet` as above but prompt it with the question: `\"Is the assessed text grounded in the context? Say no if it includes significant facts not in the context.\"`\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def has_no_hashtags(text):\n",+ " return len(re.findall(r\"#\\w+\", text)) == 0\n",+ "\n",+ "def is_within_length_limit(text, length_limit=280):\n",+ " return len(text) <= length_limit\n",+ "\n",+ "def is_assessment_yes(assessment_answer):\n",+ " \"\"\"Check if the first word of the assessment answer is 'yes'.\"\"\"\n",+ " return assessment_answer.split()[0].lower() == 'yes'\n",+ "\n",+ "def has_correct_answer(text, answer):\n",+ " return answer in text\n",+ "\n",+ "\n",+ "class AssessTweet(dspy.Signature):\n",+ " \"\"\"Assess the quality of a tweet along the specified dimension.\"\"\"\n",+ "\n",+ " context = dspy.InputField(desc='ignore if N/A')\n",+ " assessed_text = dspy.InputField()\n",+ " assessment_question = dspy.InputField()\n",+ " assessment_answer = dspy.OutputField(desc=\"Yes or No\")\n",+ "\n",+ "def no_hashtags_metric(gold, pred, trace=None):\n",+ " tweet = pred.generated_tweet\n",+ " no_hashtags = has_no_hashtags(tweet)\n",+ " score = no_hashtags\n",+ " return score\n",+ "\n",+ "def is_correct_metric(gold, pred, trace=None):\n",+ " answer, tweet = gold.answer, pred.generated_tweet\n",+ " correct = has_correct_answer(tweet, answer)\n",+ " score = correct\n",+ " return score\n",+ "\n",+ "def within_length_metric(gold, pred, trace=None):\n",+ " tweet = pred.generated_tweet\n",+ " within_length_limit = is_within_length_limit(tweet, 280)\n",+ " score = within_length_limit\n",+ " return score\n",+ "\n",+ "def engaging_metric(gold, pred, trace=None):\n",+ " tweet = pred.generated_tweet\n",+ " engaging = \"Does the assessed text make for a self-contained, engaging tweet? Say no if it is not engaging.\"\n",+ " engaging = dspy.Predict(AssessTweet)(context='N/A', assessed_text=tweet, assessment_question=engaging)\n",+ " engaging = engaging.assessment_answer.split()[0].lower() == 'yes'\n",+ " score = engaging\n",+ " return score\n",+ "\n",+ "def faithful_metric(gold, pred, trace=None):\n",+ " context, tweet = pred.context, pred.generated_tweet\n",+ " faithful = \"Is the assessed text grounded in the context? Say no if it includes significant facts not in the context.\" \n",+ " faithful = dspy.Predict(AssessTweet)(context=context, assessed_text=tweet, assessment_question=faithful)\n",+ " faithful = faithful.assessment_answer.split()[0].lower() == 'yes'\n",+ " score = faithful\n",+ " return score"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "#### Extrinsic Metrics: Assess the overall quality and effectiveness of generated output on downstream task\n",+ "\n",+ "The extrinsic metric is defined as the overall quality of the generated tweet in following the mentioned constraints, and this is evaluated over a composite metric.\n",+ "\n",+ "While maintaining the most relevant intrinsic metrics of forming a valid tweet in the correctness and within_length constraints, the overall composite metric returns an averaged score over the 5 intrinsic metrics."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def overall_metric(gold, pred, trace=None):\n",+ " answer, context, tweet = gold.answer, pred.context, pred.generated_tweet\n",+ " no_hashtags = has_no_hashtags(tweet)\n",+ " within_length_limit = is_within_length_limit(tweet, 280)\n",+ " correct = has_correct_answer(tweet, answer)\n",+ " engaging = \"Does the assessed text make for a self-contained, engaging tweet? Say no if it is not engaging.\"\n",+ " faithful = \"Is the assessed text grounded in the context? Say no if it includes significant facts not in the context.\" \n",+ " faithful = dspy.Predict(AssessTweet)(context=context, assessed_text=tweet, assessment_question=faithful)\n",+ " engaging = dspy.Predict(AssessTweet)(context='N/A', assessed_text=tweet, assessment_question=engaging)\n",+ " engaging, faithful = [m.assessment_answer.split()[0].lower() == 'yes' for m in [engaging, faithful]]\n",+ " score = (correct + engaging + faithful + no_hashtags + within_length_limit) if correct and within_length_limit else 0\n",+ " return score / 5.0"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We hence define the evaluation as follows:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "metrics = [no_hashtags_metric, is_correct_metric, within_length_metric, engaging_metric, faithful_metric, overall_metric]\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(tweeter)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's take a look at an example tweet generation:"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "example = devset[10]\n",+ "tweet = tweeter(question=example.question, answer = example.answer)\n",+ "print(f'Generated Tweet: ', tweet.generated_tweet)\n",+ "tweet.context"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset[10:11], num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(tweeter)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "In this example, we see that the generated tweet is within the length of 280 characters at 120 characters. It does in fact include the correct answer `79 AD`.\n",+ "\n",+ "However, it fails to not include hashtags as we see `#MountVesuvius #History` at the end of the tweet. Additionally, the tweet has been determined to not be engaging, which makes sense from an eye-test as it simply states the answer and nothing more. Furthermore, it is determined to not be grounded to its retrieved context.\n",+ "\n",+ "Let's try to fix this and produce tweets using DSPy Assertions. "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 5] Introducing Assertions: TweeterWithAssertions\n",+ "\n",+ "To correct these various errors, let's include assertions that simply reiterate our computational constraints within DSPy Assertion semantics. \n",+ "\n",+ "In the first **Assertion**, we check for if the generated tweet has any hashtags through regex and if violated, assert: **\"Please revise the tweet to remove hashtag phrases following it.\"**\n",+ "\n",+ "Similarly, we check for the tweet length and if it is not within 280 characters, we send the feedback message: **\"Please ensure the tweet is within {280} characters.\"**\n",+ "\n",+ "We check for if the generated tweet has the answer and if not, we assert: **\"The tweet does not include the correct answer to the question. Please revise accordingly.\"**\n",+ "\n",+ "For the engagement and faithfulness checks, we make use of the setup from above, checking for if the respective assessment is determined as `Yes` or `No`.\n"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class TweeterWithAssertions(dspy.Module):\n",+ " def __init__(self):\n",+ " super().__init__()\n",+ " self.generate_tweet = dspy.ChainOfThought(GenerateTweet)\n",+ "\n",+ " def forward(self, question, answer):\n",+ " context = []\n",+ " max_hops=2\n",+ " passages_per_hop=3\n",+ " generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]\n",+ " retrieve = dspy.Retrieve(k=passages_per_hop)\n",+ " for hop in range(max_hops):\n",+ " query = generate_query[hop](context=context, question=question).query\n",+ " passages = retrieve(query).passages\n",+ " context = deduplicate(context + passages)\n",+ " generated_tweet = self.generate_tweet(question=question, context=context).tweet\n",+ " dspy.Suggest(has_no_hashtags(generated_tweet), f\"Please revise the tweet to remove hashtag phrases following it.\", target_module=GenerateTweet)\n",+ " dspy.Suggest(is_within_length_limit(generated_tweet, 280), f\"Please ensure the tweet is within {280} characters.\", target_module=GenerateTweet)\n",+ " dspy.Suggest(has_correct_answer(generated_tweet, answer), \"The tweet does not include the correct answer to the question. Please revise accordingly.\", target_module=GenerateTweet)\n",+ " engaging_question = \"Does the assessed text make for a self-contained, engaging tweet? Say no if it is not engaging.\"\n",+ " engaging_assessment = dspy.Predict(AssessTweet)(context=context, assessed_text=generated_tweet, assessment_question=engaging_question)\n",+ " dspy.Suggest(is_assessment_yes(engaging_assessment.assessment_answer), \"The text is not engaging enough. Please revise to make it more captivating.\", target_module=GenerateTweet)\n",+ " faithful_question = \"Is the assessed text grounded in the context? Say no if it includes significant facts not in the context.\"\n",+ " faithful_assessment = dspy.Predict(AssessTweet)(context='N/A', assessed_text=generated_tweet, assessment_question=faithful_question)\n",+ " dspy.Suggest(is_assessment_yes(faithful_assessment.assessment_answer), \"The text contains unfaithful elements or significant facts not in the context. Please revise for accuracy.\", target_module=GenerateTweet)\n",+ " return dspy.Prediction(generated_tweet=generated_tweet, context=context)\n",+ "\n",+ "tweeter_with_assertions = assert_transform_module(TweeterWithAssertions().map_named_predictors(Retry), backtrack_handler) "+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's evaluate the `TweeterWithAssertions` now over the devset."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "metrics = [no_hashtags_metric, is_correct_metric, within_length_metric, engaging_metric, faithful_metric, overall_metric]\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(tweeter_with_assertions)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now let's take a look at how our generated tweet has improved with the addition of assertions."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "example = devset[10]\n",+ "tweet = tweeter_with_assertions(question=example.question, answer = example.answer)\n",+ "print(f'Generated Tweet: ', tweet.generated_tweet)\n",+ "tweet.context"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset[10:11], num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(tweeter_with_assertions)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We see that the tweet has improved significantly, following all of our set constraints! \n",+ "\n",+ "It no longer has hashtags, and is both engaging and faithful, while maintaining the inclusion of the correct answer within 280 characters. Exciting!"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 6] Compilation With Assertions\n",+ "\n",+ "We can leverage **DSPy**'s`BootstrapFewShotWithRandomSearch` optimizer, to automatically generate few-shot demonstrations and conduct a random search over the candidates to output the best compiled program. We evaluate this over the `overall_metric` composite metric. \n",+ "\n",+ "We can first evaluate this on `Tweeter` to see how compilation performs without the inclusion of assertions. "+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "teleprompter = BootstrapFewShotWithRandomSearch(metric = overall_metric, max_bootstrapped_demos=2, num_candidate_programs=6)\n",+ "compiled_tweeter = teleprompter.compile(student = tweeter, teacher = tweeter, trainset=trainset, valset=devset[:100])\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(compiled_tweeter)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now we test the compilation on 2 settings with assertions:\n",+ "\n",+ "**Compilation with Assertions**: assertion-driven example bootstrapping and counterexample bootstrapping during compilation. Teacher has assertions while the student does not as the student learns from the teacher's assertion-driven bootstrapped examples. \n",+ "\n",+ "**Compilation + Inference with Assertions**: assertion-driven optimizations for both the teacher and student to offer enhanced assertion-driven outputs during both compilation and inference."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "teleprompter = BootstrapFewShotWithRandomSearch(metric = overall_metric, max_bootstrapped_demos=2, num_candidate_programs=6)\n",+ "compiled_with_assertions_tweeter = teleprompter.compile(student=tweeter, teacher = tweeter_with_assertions, trainset=trainset, valset=devset[:100])\n",+ "\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(compiled_with_assertions_tweeter)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "teleprompter = BootstrapFewShotWithRandomSearch(metric = overall_metric, max_bootstrapped_demos=2, num_candidate_programs=6)\n",+ "compiled_tweeter_with_assertions = teleprompter.compile(student=tweeter_with_assertions, teacher = tweeter_with_assertions, trainset=trainset, valset=devset[:100])\n",+ "\n",+ "for metric in metrics:\n",+ " evaluate = Evaluate(metric=metric, devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ " evaluate(compiled_tweeter_with_assertions)"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "dspy_dev",+ "language": "python",+ "name": "python3"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.10.13"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 2+}