DSPy: The framework for programming—not prompting—language models
Python
added notebook for mlc models support
Arnav Singhvi23553271bd74
diff --git a/dsp/modules/hf_client.py b/dsp/modules/hf_client.pyindex 8a674ed6f..549340b5d 100644--- a/dsp/modules/hf_client.py+++ b/dsp/modules/hf_client.py@@ -28,6 +28,7 @@ from dsp.modules.cache_utils import CacheMemory, NotebookCacheMemory, cache_turn# from dsp.modules.adapter import TurboAdapter, DavinciAdapter, LlamaAdapterfrom mlc_chat import ChatModule+from mlc_chat import ChatConfigclass HFClientTGI(HFModel):def __init__(self, model, port, url="http://future-hgx-1", **kwargs):@@ -97,7 +98,7 @@ def send_hftgi_request_v00(arg, **kwargs):class ChatModuleClient(HFModel):def __init__(self, model, model_path):super().__init__(model=model, is_client=True)- self.cm = ChatModule(model=model, lib_path=model_path)+ self.cm = ChatModule(model=model, lib_path=model_path, chat_config=ChatConfig(conv_template="LM"))def _generate(self, prompt, **kwargs):output = self.cm.generate(diff --git a/dspy/__init__.py b/dspy/__init__.pyindex 62924665c..3c46b8cea 100644--- a/dspy/__init__.py+++ b/dspy/__init__.py@@ -1,3 +1,4 @@+from dsp.modules.hf_client import ChatModuleClientfrom .signatures import *from .retrieve import *@@ -16,3 +17,4 @@ settings = dsp.settingsOpenAI = dsp.GPT3ColBERTv2 = dsp.ColBERTv2HFClientTGI = dsp.HFClientTGI+ChatModuleClient = ChatModuleClient\ No newline at end of filediff --git a/intro_with_mlc.ipynb b/intro_with_mlc.ipynbnew file mode 100644index 000000000..5c348756e--- /dev/null+++ b/intro_with_mlc.ipynb@@ -0,0 +1,953 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "<img src=\"docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",+ "\n",+ "## **DSPy**: Programming with Foundation Models\n",+ "\n",+ "[<img align=\"center\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/v2/intro.ipynb)\n",+ "\n",+ "!! TODO: Fix the Colab link from branch v2 to main when we push to main\n",+ "\n",+ "!! TODO: Fix the setting up thing when merged too"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "This notebook introduces the **DSPy** framework for **Programming with Foundation Models**, i.e., language models (LMs) and retrieval models (RMs).\n",+ "\n",+ "We built **DSPy** because we believe there is no ‟single best pipeline” for using LMs across tasks, and we believe you should not need to manually tweak prompts to get your pipelines to work well.\n",+ "\n",+ "To realize this, **DSPy** provides powerful abstractions for you to **build your own pipelines** for your own tasks using clean Pythonic operators over **modules that compose and learn**!"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 0] Setting Up\n",+ "\n",+ "**DSPy** is from the authors of ColBERT and the Demonstrate-Search-Predict framework (i.e., DSP, which is the predecessor of **DSPy**). No background on DSP or ColBERT is needed at all, though!\n",+ "\n",+ "Let's begin by setting things up. The snippet below will also install **DSPy** if it's not there already."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "#FIXME - evaluate functions error out since downloading mlc models triggers python 3.10 which causes errors - ValueError: Only callable can be used as callback\n",+ "\n",+ "\n",+ "# %load_ext autoreload\n",+ "# %autoreload 2\n",+ "\n",+ "import sys\n",+ "import os\n",+ "\n",+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",+ " import google.colab\n",+ " repo_path = 'dspy'\n",+ " !git -C $repo_path pull origin v2 || git clone -b v2 https://github.com/stanfordnlp/dspy $repo_path\n",+ "except:\n",+ " repo_path = '.'\n",+ "\n",+ "if repo_path not in sys.path:\n",+ " sys.path.append(repo_path)\n",+ "\n",+ "# Set up the cache for this notebook\n",+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(repo_path, 'cache')\n",+ "\n",+ "\"\"\"\n",+ "## TODO: Consider this\n",+ "if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",+ " !pip install -U pip\n",+ " !pip install -e $repo_path\n",+ "\"\"\"\n",+ "\n",+ "try:\n",+ " import dspy\n",+ "except Exception:\n",+ " !pip install -U pip\n",+ " !pip install -e $repo_path\n",+ " \n",+ " import dspy"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "#NOTE - run cell above, wait for it to error out saying it can't find mlc and then run cell below, and then rerun the cell above\n",+ "#FIXME - running the MLC prerequisites first causes errors as when dspy is imported, it is not suited for python 3.10 and errors out"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "MLC prerequisites"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "!pip install --pre --force-reinstall mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 -f https://mlc.ai/wheels\n",+ "!pip install transformers\n",+ "!git lfs install\n",+ "!mkdir -p dist/prebuilt\n",+ "!git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git dist/prebuilt/lib\n",+ "!cd dist/prebuilt && git clone https://huggingface.co/mlc-ai/mlc-chat-Llama-2-7b-chat-hf-q4f16_1"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 1] Getting Started\n",+ "\n",+ "We'll start by setting up the language model (LM) and retrieval model (RM). **DSPy** supports multiple API and local models. In this notebook, we'll work with GPT-3.5 (`gpt-3.5-turbo`) and the retriever `ColBERTv2`.\n",+ "\n",+ "To make things easy, we've set up a ColBERTv2 server hosting a Wikipedia 2017 \"abstracts\" search index (i.e., containing first paragraph of each article from this [2017 dump](https://hotpotqa.github.io/wiki-readme.html)), so you don't need to worry about setting one up! It's free.\n",+ "\n",+ "**Note:** _If you want to run this notebook without changing the examples, you don't need an API key. All examples are already cached internally so you can inspect them!_"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "#set the chat module client here with model and model path for llama models\n",+ "turbo = dspy.ChatModuleClient(model='dist/prebuilt/mlc-chat-Llama-2-7b-chat-hf-q4f16_1', model_path='dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-cuda.so')\n",+ "colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",+ "\n",+ "dspy.settings.configure(lm=turbo, rm=colbertv2_wiki17_abstracts)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "In the last line above, we configure **DSPy** to use the turbo LM and the ColBERTv2 retriever (over Wikipedia 2017 abstracts) by default. This will be easy to overwrite for local parts of our programs if needed."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### A word on the workflow"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "You can build your own **DSPy programs** for various tasks, e.g., question answering, information extraction, or text-to-SQL.\n",+ "\n",+ "Whatever the task, the general workflow is:\n",+ "\n",+ "1. **Collect a little bit of data.** Define examples of the inputs and outputs of your program (e.g., questions and their answers). This could just be a handful of quick examples you wrote down. If large datasets exist, the more the merrier!\n",+ "1. **Write your program.** Define the modules (i.e., sub-tasks) of your program and the way they should interact together to solve your task.\n",+ "1. **Define some validation logic.** What makes for a good run of your program? Maybe the answers need to have a certain length or stick to a particular format? Specify the logic that checks that.\n",+ "1. **Compile!** Ask **DSPy** to _compile_ your program using your data. The compiler will use your data and validation logic to optimize your program (e.g., prompts and modules) so it's efficient and effective!\n",+ "1. **Iterate.** Repeat the process by improving your data, program, validation, or by using more advanced features of the **DSPy** compiler.\n",+ "\n",+ "Let's now see this in action."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 2] Task Examples\n",+ "\n",+ "**DSPy** accomodates a wide variety of applications and tasks. **In this intro notebook, we will work on the example task of multi-hop question answering (QA).**\n",+ "\n",+ "Other notebooks and tutorials will present different tasks. Now, let us load a tiny sample from the HotPotQA multi-hop dataset."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from dspy.datasets import HotPotQA\n",+ "\n",+ "# Load the dataset.\n",+ "dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0)\n",+ "\n",+ "# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.\n",+ "trainset = [x.with_inputs('question') for x in dataset.train]\n",+ "devset = [x.with_inputs('question') for x in dataset.dev]\n",+ "\n",+ "len(trainset), len(devset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "We just loaded `trainset` (20 examples) and `devset` (50 examples). Each example in our **training set** contains just a **question** and its (human-annotated) **answer**.\n",+ "\n",+ "**DSPy** typically requires very minimal labeling. Whereas your pipeline may involve six or seven complex steps, you only need labels for the initial question and the final answer. **DSPy** will bootstrap any intermediate labels needed to support your pipeline. If you change your pipeline in any way, the data bootstrapped will change accordingly!\n",+ "\n",+ "Now, let's look at some data examples."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "train_example = trainset[0]\n",+ "print(f\"Question: {train_example.question}\")\n",+ "print(f\"Answer: {train_example.answer}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Examples in the **dev set** contain a third field, namely, **titles** of relevant Wikipedia articles. This is not essential but, for the sake of this intro, it'll help us get a sense of how well our programs are doing."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "dev_example = devset[18]\n",+ "print(f\"Question: {dev_example.question}\")\n",+ "print(f\"Answer: {dev_example.answer}\")\n",+ "print(f\"Relevant Wikipedia Titles: {dev_example.gold_titles}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "After loading the raw data, we'd applied `x.with_inputs('question')` to each example to tell **DSPy** that our input field in each example will be just `question`. Any other fields are labels or metadata that are not given to the system."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "print(f\"For this dataset, training examples have input keys {train_example.inputs().keys()} and label keys {train_example.labels().keys()}\")\n",+ "print(f\"For this dataset, dev examples have input keys {dev_example.inputs().keys()} and label keys {dev_example.labels().keys()}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Note that there's nothing special about the HotPotQA dataset: it's just a list of examples.\n",+ "\n",+ "You can define your own examples as below. A future notebook will guide you through creating your own data in unusual or data-scarce settings, which is a context where **DSPy** excels.\n",+ "\n",+ "```\n",+ "dspy.Example(field1=value, field2=value2, ...)\n",+ "```"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 3] Building Blocks\n",+ "\n",+ "In **DSPy**, we will maintain a clean separation between **defining your modules in a declarative way** and **calling them in a pipeline to solve the task**.\n",+ "\n",+ "This allows you to focus on the information flow of your pipeline. **DSPy** will then take your program and automatically optimize **how to prompt** (or finetune) LMs **for your particular pipeline** so it works well.\n",+ "\n",+ "If you have experience with PyTorch, you can think of DSPy as the PyTorch of the foundation model space. Before we see this in action, let's first understand some key pieces."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### Using the Language Model: **Signatures** & **Predictors**\n",+ "\n",+ "Every call to the LM in a **DSPy** program needs to have a **Signature**.\n",+ "\n",+ "A signature consists of three simple elements:\n",+ "\n",+ "- A minimal description of the sub-task the LM is supposed to solve.\n",+ "- A description of one or more input fields (e.g., input question) that will we will give to the LM.\n",+ "- A description of one or more output fields (e.g., the question's answer) that we will expect from the LM.\n",+ "\n",+ "Let's define a simple signature for basic question answering."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class BasicQA(dspy.Signature):\n",+ " \"\"\"Answer questions with short factoid answers.\"\"\"\n",+ "\n",+ " question = dspy.InputField()\n",+ " answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "In `BasicQA`, the docstring describes the sub-task here (i.e., answering questions). Each `InputField` or `OutputField` can optionally contain a description `desc` too. When it's not given, it's inferred from the field's name (e.g., `question`).\n",+ "\n",+ "Notice that there isn't anything special about this signature in **DSPy**. We can just as easily define a signature that takes a long snippet from a PDF and outputs structured information, for instance.\n",+ "\n",+ "Anyway, now that we have a signature, let's define and use a **Predictor**. A predictor is a module that knows how to use the LM to implement a signature. Importantly, predictors can **learn** to fit their behavior to the task!"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Define the predictor.\n",+ "generate_answer = dspy.Predict(BasicQA)\n",+ "\n",+ "# Call the predictor on a particular input.\n",+ "pred = generate_answer(question=dev_example.question)\n",+ "\n",+ "# Print the input and the prediction.\n",+ "print(f\"Question: {dev_example.question}\")\n",+ "print(f\"Predicted Answer: {pred.answer}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "In the example above, we asked the predictor about the the chef featured in \"Restaurant: Impossible\". The model outputs an answer (\"American\").\n",+ "\n",+ "For visibility, we can inspect how this extremely basic predictor implemented our signature. Let's inspect the history of our LM (**turbo**)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "turbo.inspect_history(n=1)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "It happens that this chef is both British and American, but we have no way of knowing if the model just guessed \"American\" because it's a common answer. In general, adding **retrieval** and **learning** will help the LM be more factual, and we'll explore this in a minute!\n",+ "\n",+ "But before we do that, how about we _just_ change the predictor? It would be nice to allow the model to elicit a chain of thought along with the prediction."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Define the predictor. Notice we're just changing the class. The signature BasicQA is unchanged.\n",+ "generate_answer_with_chain_of_thought = dspy.ChainOfThought(BasicQA)\n",+ "\n",+ "# Call the predictor on the same input.\n",+ "pred = generate_answer_with_chain_of_thought(question=dev_example.question)\n",+ "\n",+ "# Print the input, the chain of thought, and the prediction.\n",+ "print(f\"Question: {dev_example.question}\")\n",+ "print(f\"Thought: {pred.rationale.split('.', 1)[1].strip()}\")\n",+ "print(f\"Predicted Answer: {pred.answer}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "This is indeed a better answer: the model figures out that the chef in question is **Robert Irvine** and correctly identifies that he's British.\n",+ "\n",+ "These predictors (`dspy.Predict` and `dspy.ChainOfThought`) can be applied to _any_ signature. As we'll see below, they can also be optimized to learn from your data and validation logic."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### Using the Retrieval Model\n",+ "\n",+ "Using the retriever is pretty simple. A module `dspy.Retrieve(k)` will search for the top-`k` passages that match a given query.\n",+ "\n",+ "By default, this will use the retriever we configured at the top of this notebook, namely, ColBERTv2 over a Wikipedia index."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "retrieve = dspy.Retrieve(k=3)\n",+ "topK_passages = retrieve(dev_example.question).passages\n",+ "\n",+ "print(f\"Top {retrieve.k} passages for question: {dev_example.question} \\n\", '-' * 30, '\\n')\n",+ "\n",+ "for idx, passage in enumerate(topK_passages):\n",+ " print(f'{idx+1}]', passage, '\\n')"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Feel free to any other queries you like."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "retrieve(\"When was the first FIFA World Cup held?\").passages[0]"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 4] Program 1: Basic Retrieval-Augmented Generation (“RAG”)\n",+ "\n",+ "Let's define our first complete program for this task. We'll build a retrieval-augmented pipeline for answer generation.\n",+ "\n",+ "Given a question, we'll search for the top-3 passages in Wikipedia and then feed them as context for answer generation.\n",+ "\n",+ "Let's start by defining this signature: `context, question --> answer`."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class GenerateAnswer(dspy.Signature):\n",+ " \"\"\"Answer questions with short factoid answers.\"\"\"\n",+ "\n",+ " context = dspy.InputField(desc=\"may contain relevant facts\")\n",+ " question = dspy.InputField()\n",+ " answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Great. Now let's define the actual program. This is a class that inherits from `dspy.Module`.\n",+ "\n",+ "It needs two methods:\n",+ "\n",+ "- The `__init__` method will simply declare the sub-modules it needs: `dspy.Retrieve` and `dspy.ChainOfThought`. The latter is defined to implement our `GenerateAnswer` signature.\n",+ "- The `forward` method will describe the control flow of answering the question using the modules we have."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class RAG(dspy.Module):\n",+ " def __init__(self, num_passages=3):\n",+ " super().__init__()\n",+ "\n",+ " self.retrieve = dspy.Retrieve(k=num_passages)\n",+ " self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n",+ " \n",+ " def forward(self, question):\n",+ " context = self.retrieve(question).passages\n",+ " prediction = self.generate_answer(context=context, question=question)\n",+ " return dspy.Prediction(context=context, answer=prediction.answer)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### Compiling the RAG program\n",+ "\n",+ "Having defined this program, let's now **compile** it. Compiling a program will update the parameters stored in each module. In our setting, this is primarily in the form of collecting and selecting good demonstrations for inclusion in your prompt(s).\n",+ "\n",+ "Compiling depends on three things:\n",+ "\n",+ "1. **A training set.** We'll just use our 20 question–answer examples from `trainset` above.\n",+ "1. **A metric for validation.** We'll define a quick `validate_context_and_answer` that checks that the predicted answer is correct. It'll also check that the retrieved context does actually contain that answer.\n",+ "1. **A specific teleprompter.** The **DSPy** compiler includes a number of **teleprompters** that can optimize your programs."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "**Teleprompters:** Teleprompters are powerful optimizers that can take any program and learn to bootstrap and select effective prompts for its modules. Hence the name, which means \"prompting at a distance\".\n",+ "\n",+ "Different teleprompters offer various tradeoffs in terms of how much they optimize cost versus quality, etc. We will use a simple default `BootstrapFewShot` in this notebook.\n",+ "\n",+ "\n",+ "_If you're into analogies, you could think of this as your training data, your loss function, and your optimizer in a standard DNN supervised learning setup. Whereas SGD is a basic optimizer, there are more sophisticated (and more expensive!) ones like Adam or RMSProp._"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from dspy.teleprompt import BootstrapFewShot\n",+ "\n",+ "# Validation logic: check that the predicted answer is correct.\n",+ "# Also check that the retrieved context does actually contain that answer.\n",+ "def validate_context_and_answer(example, pred, trace=None):\n",+ " answer_EM = dspy.evaluate.answer_exact_match(example, pred)\n",+ " answer_PM = dspy.evaluate.answer_passage_match(example, pred)\n",+ " return answer_EM and answer_PM\n",+ "\n",+ "# Set up a basic teleprompter, which will compile our RAG program.\n",+ "teleprompter = BootstrapFewShot(metric=validate_context_and_answer)\n",+ "\n",+ "# Compile!\n",+ "compiled_rag = teleprompter.compile(RAG(), trainset=trainset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Now that we've compiled our RAG program, let's try it out."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Ask any question you like to this simple RAG program.\n",+ "my_question = \"What castle did David Gregory inherit?\"\n",+ "\n",+ "# Get the prediction. This contains `pred.context` and `pred.answer`.\n",+ "pred = compiled_rag(my_question)\n",+ "\n",+ "# Print the contexts and the answer.\n",+ "print(f\"Question: {my_question}\")\n",+ "print(f\"Predicted Answer: {pred.answer}\")\n",+ "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Excellent. How about we inspect the last prompt for the LM?"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "turbo.inspect_history(n=1)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Even though we haven't written any of this detailed demonstrations, we see that **DSPy** was able to bootstrap this 3,000 token prompt for **3-shot retrieval augmented generation with hard negative passages and chain of thought** from our extremely simple program.\n",+ "\n",+ "This illustrates the power of composition and learning. Of course, this was just generated by a particular teleprompter, which may or may not be perfect in each setting. As you'll see in **DSPy**, there is a large but systematic space of options you have to optimize and validate the quality and cost of your programs.\n",+ "\n",+ "If you're so inclined, you can easily inspect the learned objects themselves."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "for name, parameter in compiled_rag.named_predictors():\n",+ " print(name)\n",+ " print(parameter.demos[0])\n",+ " print()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### Evaluating the Answers\n",+ "\n",+ "We can now evaluate our `compiled_rag` program on the dev set. Of course, this tiny set is _not_ meant to be a reliable benchmark, but it'll be instructive to use it for illustration.\n",+ "\n",+ "For a start, let's evaluate the accuracy (exact match) of the predicted answer."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from dspy.evaluate.evaluate import Evaluate\n",+ "\n",+ "# Set up the `evaluate_on_hotpotqa` function. We'll use this many times below.\n",+ "evaluate_on_hotpotqa = Evaluate(devset=devset, num_threads=1, display_progress=True, display_table=5)\n",+ "\n",+ "# Evaluate the `compiled_rag` program with the `answer_exact_match` metric.\n",+ "metric = dspy.evaluate.answer_exact_match\n",+ "evaluate_on_hotpotqa(compiled_rag, metric=metric)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### Evaluating the Retrieval\n",+ "\n",+ "It may also be instructive to look at the accuracy of retrieval. There are multiple ways to do this. Often, we can just check whether the rertieved passages contain the answer.\n",+ "\n",+ "That said, since our dev set includes the gold titles that should be retrieved, we can just use these here."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def gold_passages_retrieved(example, pred, trace=None):\n",+ " gold_titles = set(map(dspy.evaluate.normalize_text, example['gold_titles']))\n",+ " found_titles = set(map(dspy.evaluate.normalize_text, [c.split(' | ')[0] for c in pred.context]))\n",+ "\n",+ " return gold_titles.issubset(found_titles)\n",+ "\n",+ "#FIXME - errors out over here due to reload extension and python3.10 from mlc incompatability \n",+ "compiled_rag_retrieval_score = evaluate_on_hotpotqa(compiled_rag, metric=gold_passages_retrieved)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Although this simple `compiled_rag` program is able to answer a decent fraction of the questions correctly (on this tiny set, over 40%), the quality of retrieval is much lower.\n",+ "\n",+ "This potentially suggests that the LM is often relying on the knowledge it memorized during training to answer questions. To address this weak retrieval, let's explore a second program that involves more advanced search behavior."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "### 5] Program 2: Multi-Hop Search (“Baleen”)\n",+ "\n",+ "From exploring the harder questions in the training/dev sets, it becomes clear that a single search query is often not enough for this task. For instance, this can be seen when a question ask about, say, the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it wouldn't figure out when he was born.\n",+ "\n",+ "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary. Using **DSPy**, we can easily simulate such systems in a few lines of code.\n",+ "\n",+ "\n",+ "We'll still use the `GenerateAnswer` signature from the RAG implementation above. All we need now is a **signature** for the \"hop\" behavior: taking some partial context and a question, generate a search query to find missing information."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "class GenerateSearchQuery(dspy.Signature):\n",+ " \"\"\"Write a simple search query that will help answer a complex question.\"\"\"\n",+ "\n",+ " context = dspy.InputField(desc=\"may contain relevant facts\")\n",+ " question = dspy.InputField()\n",+ " query = dspy.OutputField()"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Note: We could have written `context = GenerateAnswer.signature.context` to avoid duplicating the description of the `context` field.\n",+ "\n",+ "Now, let's define the program itself `SimplifiedBaleen`. There are many possible ways to implement this, but we'll keep this version down to the key elements for simplicity."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "from dsp.utils import deduplicate\n",+ "\n",+ "class SimplifiedBaleen(dspy.Module):\n",+ " def __init__(self, passages_per_hop=3, max_hops=2):\n",+ " super().__init__()\n",+ "\n",+ " self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]\n",+ " self.retrieve = dspy.Retrieve(k=passages_per_hop)\n",+ " self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n",+ " self.max_hops = max_hops\n",+ " \n",+ " def forward(self, question):\n",+ " context = []\n",+ " \n",+ " for hop in range(self.max_hops):\n",+ " query = self.generate_query[hop](context=context, question=question).query\n",+ " passages = self.retrieve(query).passages\n",+ " context = deduplicate(context + passages)\n",+ "\n",+ " pred = self.generate_answer(context=context, question=question)\n",+ " return dspy.Prediction(context=context, answer=pred.answer)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "As we can see, the `__init__` method defines a few key sub-modules:\n",+ "\n",+ "- **generate_query**: For each hop, we will have one `dspy.ChainOfThought` predictor with the `GenerateSearchQuery` signature.\n",+ "- **retrieve**: This module will do the actual search, using the generated queries.\n",+ "- **generate_answer**: This `dspy.Predict` module will be used after all the search steps. It has a `GenerateAnswer`, to actually produce an answer.\n",+ "\n",+ "The `forward` method uses these sub-modules in simple control flow.\n",+ "\n",+ "1. First, we'll loop up to `self.max_hops` times.\n",+ "1. In each iteration, we'll generate a search query using the predictor at `self.generate_query[hop]`.\n",+ "1. We'll retrieve the top-k passages using that query.\n",+ "1. We'll add the (deduplicated) passages to our accumulator of `context`.\n",+ "1. After the loop, we'll use `self.generate_answer` to produce an answer.\n",+ "1. We'll return a prediction with the retrieved `context` and predicted `answer`."+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### Inspect the zero-shot version of the Baleen program\n",+ "\n",+ "We will also compile this program shortly. But, before that, we can try it out in a \"zero-shot\" setting (i.e., without any compilation).\n",+ "\n",+ "Using a program in zero-shot (uncompiled) setting doesn't mean that quality will be bad. It just means that we're bottlenecked directly by the reliability of the underlying LM to understand our sub-tasks from minimal instructions.\n",+ "\n",+ "This is often just fine when using the most expensive/powerful models (e.g., GPT-4) on the easiest and most standard tasks (e.g., answering simple questions about popular entities).\n",+ "\n",+ "However, a zero-shot approach quickly falls short for more specialized tasks, for novel domains/settings, and for more efficient (or open) models. **DSPy** can help you in all of these settings."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "# Ask any question you like to this simple RAG program.\n",+ "my_question = \"How many storeys are in the castle that David Gregory inherited?\"\n",+ "\n",+ "# Get the prediction. This contains `pred.context` and `pred.answer`.\n",+ "uncompiled_baleen = SimplifiedBaleen() # uncompiled (i.e., zero-shot) program\n",+ "pred = uncompiled_baleen(my_question)\n",+ "\n",+ "# Print the contexts and the answer.\n",+ "print(f\"Question: {my_question}\")\n",+ "print(f\"Predicted Answer: {pred.answer}\")\n",+ "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Let's inspect the last **three** calls to the LM (i.e., generating the first hop's query, generating the second hop's query, and generating the answer)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "turbo.inspect_history(n=3)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### Compiling the Baleen program\n",+ "\n",+ "Now is the time to compile our multi-hop (`SimplifiedBaleen`) program.\n",+ "\n",+ "We will first define our validation logic, which will simply require that:\n",+ "\n",+ "- The predicted answer matches the gold answer.\n",+ "- The retrieved context contains the gold answer.\n",+ "- None of the generated queries is rambling (i.e., none exceeds 100 characters in length).\n",+ "- None of the generated queries is roughly repeated (i.e., none is within 0.8 or higher F1 score of earlier queries)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "def validate_context_and_answer_and_hops(example, pred, trace=None):\n",+ " if not dspy.evaluate.answer_exact_match(example, pred): return False\n",+ " if not dspy.evaluate.answer_passage_match(example, pred): return False\n",+ "\n",+ " hops = [example.question] + [outputs.query for *_, outputs in trace if 'query' in outputs]\n",+ "\n",+ " if max([len(h) for h in hops]) > 100: return False\n",+ " if any(dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8) for idx in range(2, len(hops))): return False\n",+ "\n",+ " return True"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Like we did for RAG, we'll use one of the most basic teleprompters in **DSPy**, namely, `BootstrapFewShot`."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "teleprompter = BootstrapFewShot(metric=validate_context_and_answer_and_hops)\n",+ "compiled_baleen = teleprompter.compile(SimplifiedBaleen(), teacher=SimplifiedBaleen(passages_per_hop=2), trainset=trainset)"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "##### Evaluating the Retrieval\n",+ "\n",+ "Earlier, it appeared like our simple RAG program was not very effective at finding all evidence required for answering each question. Is this resolved by the adding some extra steps in the `forward` function of `SimplifiedBaleen`? What about compiling, does it help for that? \n",+ "\n",+ "The answer for these questions is not always going to be obvious. However, **DSPy** makes it extremely easy to try many diverse approaches with minimal effort.\n",+ "\n",+ "Let's evaluate the quality of retrieval of our compiled and uncompiled Baleen pipelines!"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "uncompiled_baleen_retrieval_score = evaluate_on_hotpotqa(uncompiled_baleen, metric=gold_passages_retrieved, display=False)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "compiled_baleen_retrieval_score = evaluate_on_hotpotqa(compiled_baleen, metric=gold_passages_retrieved)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "print(f\"## Retrieval Score for RAG: {compiled_rag_retrieval_score}\") # note that for RAG, compilation has no effect on the retrieval step\n",+ "print(f\"## Retrieval Score for uncompiled Baleen: {uncompiled_baleen_retrieval_score}\")\n",+ "print(f\"## Retrieval Score for compiled Baleen: {compiled_baleen_retrieval_score}\")"+ ]+ },+ {+ "cell_type": "markdown",+ "metadata": {},+ "source": [+ "Excellent! There might be something to this compiled, multi-hop program then. But this is far from all you can do: **DSPy** gives you a clean space of composable operators to deal with any shortcomings you see.\n",+ "\n",+ "We can inspect a few concrete examples. If we see failure causes, we can:\n",+ "\n",+ "1. Expand our pipeline by using additional sub-modules (e.g., maybe summarize after retrieval?)\n",+ "1. Modify our pipeline by using more complex logic (e.g., maybe we need to break out of the multi-hop loop if we found all information we need?) \n",+ "1. Refine our validation logic (e.g., maybe use a metric that use a second **DSPy** program to do the answer evaluation, instead of relying on strict string matching)\n",+ "1. Use a different teleprompter to optimize your pipeline more aggressively.\n",+ "1. Add more or better training examples!\n",+ "\n",+ "\n",+ "Or, if you really want, we can tweak the descriptions in the Signatures we use in your program to make them more precisely suited for their sub-tasks. This is akin to prompt engineering and should be a final resort, given the other powerful options that **DSPy** gives us!"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "metadata": {},+ "outputs": [],+ "source": [+ "compiled_baleen(\"How many storeys are in the castle that David Gregory inherited?\")\n",+ "turbo.inspect_history(n=3)"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "base",+ "language": "python",+ "name": "base"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.9.12"+ },+ "orig_nbformat": 4+ },+ "nbformat": 4,+ "nbformat_minor": 2+}