Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Prepare README. Apply minor patches.

Omar Khattab1818cc92b2fd
diff --git a/README.md b/README.md
index 718ef1937..eb3b119d4 100644
--- a/README.md
+++ b/README.md
@@ -6,50 +6,376 @@
# DSPy: The Framework for Programming with Foundation Models
-This section will introduce **DSPy** briefly.
-z
-If you're looking for DSP (v1), it's on a branch.
+[<img align="center" src="https://colab.research.google.com/assets/colab-badge.svg" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/v2/intro.ipynb)
-## Installation
+**DSPy** is the framework for _programming_ with language models (LMs) and retrieval models (RMs) to tackle any task. **DSPy** unifies techniques for **prompting** and **fine-tuning** LMs as well as improving them with **reasoning** and **tool/retrieval augmentation**, all expressed through a _minimalistic set of Pythonic operations that compose and learn_.
+
+To make this possible:
+
+- **DSPy** provides **composable and declarative modules** for instructing LMs in a familiar Pythonic syntax. It upgrades "prompting techniques" like chain-of-thought and self-reflection from hand-adapted _string manipulation tricks_ into truly modular _generalized operations that learn to adapt to your task_.
+
+- **DSPy** introduces an **automatic compiler that teaches LMs** how to conduct the declarative steps in your program. Specifically, the **DSPy compiler** will internally _trace_ your program and then **craft high-quality prompts for large LMs (or train automatic finetunes for small LMs)** to teach them the steps of your task.
+
+The **DSPy compiler** _bootstraps_ prompts and finetunes from minimal data **without needing manual labels for the intermediate steps** in your program. Instead of brittle "prompt engineering" with hacky string manipulation, you can explore a systematic space of modular and trainable pieces.
+
+For complex tasks, **DSPy** can routinely teach powerful models like `GPT-3.5` and local models like `T5-base` or `Llama2-13b` to be much more reliable at tasks. **DSPy** will compile the _same program_ into different few-shot prompts and/or finetunes for each LM.
+
+
+If you want to see **DSPy** in action, **[open our intro tutorial notebook](intro.ipynb)**.
+
+
+### a. Table of Contents
+
+
+1. **[Installation](#1-installation)**
+1. **[Framework Syntax](#2-syntax-youre-in-charge-of-the-workflowits-free-form-python-code)**
+1. **[Compiling: Two Powerful Concepts](#3-two-powerful-concepts-signatures--teleprompters)**
+1. **[Tutorials & Documentation](#4-documentation--tutorials)**
+1. **[FAQ: Is DSPy right for me?](#5-faq-is-dspy-right-for-me)**
+
+
+
+### b. Analogy to Neural Networks
+
+If you're looking for an analogy, think of this one. When we build neural networks, we don't write manual _for-loops_ over lists of _hand-tuned_ floats. Instead, you might use a framework like [PyTorch](https://pytorch.org/) to compose declarative layers (e.g., `Convolution` or `Dropout`) and then use optimizers (e.g., SGD or Adam) to learn the parameters of the network.
+
+Ditto! **DSPy** gives you the right minimalistic, general-purpose modules (e.g., `ChainOfThought`, `Retrieve`, etc.) and takes care of optimizing their prompts _for your program_. Whenever you modify your code, your data, or your validation constraints, you can _compile_ your program again and **DSPy** will create new effective prompts that fit your changes.
+
+
+## 1) Installation
+
+All you need is:
```
pip install dspy-ai
```
+Or open our intro notebook in Google Colab: [<img align="center" src="https://colab.research.google.com/assets/colab-badge.svg" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/v2/intro.ipynb)
+
+
+> _Note: If you're looking for Demonstrate-Search-Predict (DSP), which is the previous version of DSPy, you can find it on the [v1](https://github.com/stanfordnlp/dspy/tree/v1) branch of this repo._
+
+
+
+## 2) Syntax: You're in charge of the workflow—it's free-form Python code!
+
+**DSPy** hides tedious prompt engineering, but it exposes the important decisions you need to take: **[1]** what's your system design going to look like? **[2]** what are the important constraints on the behavior of your program?
+
+You express your system as free-form Pythonic modules. **DSPy** will tune the quality of your program _in whatever way_ you use foundation models: you can code with loops, `if` statements, or exceptions, and use **DSPy** modules within any Python control flow you think works for your task.
+
+Suppose you want to build a simple retrieval-augmented generation (RAG) system for question answering. You can define your own `RAG` program like this:
+
+```python
+class RAG(dspy.Module):
+ def __init__(self, num_passages=3):
+ super().__init__()
+ self.retrieve = dspy.Retrieve(k=num_passages)
+ self.generate_answer = dspy.ChainOfThought("context, question -> answer")
+
+ def forward(self, question):
+ context = self.retrieve(question).passages
+ answer = self.generate_answer(context=context, question=question)
+ return answer
+```
+
+A program has two key methods, which you can edit to fit your needs.
+
+**Your `__init__` method** declares the modules you will use. Here, `RAG` will use the built-in `Retrieve` for retrieval and `ChainOfThought` for generating answers. **DSPy** offers general-purpose modules that take the shape of _your own_ sub-tasks — and not pre-built functions for specific applications.
+
+Modules that use the LM, like `ChainOfThought`, require a _signature_. That is a declarative spec that tells the module what it's expected to do. In this example, we use the short-hand signature notation `context, question -> answer` to tell `ChainOfThought` it will be given some `context` and a `question` and must produce an `answer`. We will discuss more advanced **[signatures](#3a-declaring-the-inputoutput-behavior-of-lms-with-dspysignature)** below.
+
+
+**Your `forward` method** expresses any computation you want to do with your modules. In this case, we use the modules `self.retrieve` and `self.generate_answer` to search for some `context` and then use the `context` and `quetion` to generate the `answer`!
+
+You can now either use this `RAG` program in **zero-shot mode**. Or **compile** it to obtain higher quality. Zero-shot usage is simple. Just define an instance of your program and then call it:
+
+```python
+rag = RAG() # zero-shot, uncompiled version of RAG
+rag("what is the capital of France?").answer # -> "Paris"
+```
+
+The next section will discuss how to compile our simple `RAG` program. When we compile it, the **DSPy compiler** will annotate _demonstrations_ of its steps: (1) retrieval, (2) using context, and (3) using _chain-of-thought_ to answer questions. From these demonstrations, the **DSPy compiler** will make sure it produces an effective few-shot prompt that works well with your LM, retrieval model, and data. If you're working with small models, it'll finetune your model (instead of prompting) to do this task.
+
+If you later decide you need another step in your pipeline, just add another module and compile again. Maybe add a module that takes the chat history into account during search?
+
+
+## 3) Two Powerful Concepts: Signatures & Teleprompters
+
+To make it possible to compile any program you write, **DSPy** introduces two simple concepts: Signatures and Teleprompters.
+
+
+#### 3.a) Declaring the input/output behavior of LMs with `dspy.Signature`
+
+When we assign tasks to LMs in **DSPy**, we specify the behavior we need as a **Signature**. A signature is a declarative specification of input/output behavior of a **DSPy module**.
+
+Instead of investing effort into _how_ to get your LM to do a sub-task, signatures enable you to inform **DSPy** _what_ the sub-task is. Later, the **DSPy compiler** will figure out how to build a complex prompt for your large LM (or finetune your small LM) specifically for your signature, on your data, and within your pipeline.
+
+A signature consists of three simple elements:
+
+- A minimal description of the sub-task the LM is supposed to solve.
+- A description of one or more input fields (e.g., input question) that will we will give to the LM.
+- A description of one or more output fields (e.g., the question's answer) that we will expect from the LM.
+
+
+We support two notations for expressing signatures. The **short-hand signature notation** is for quick development. You just provide your module (e.g., `dspy.ChainOfThought`) with a string with `input_field_name_1, ... -> output_field_name_1, ...` with the fields separated by commas.
+
+In the `RAG` class earlier, we saw:
+
+```python
+self.generate_answer = dspy.ChainOfThought("context, question -> answer")
+```
+
+In many cases, this barebones signature is sufficient. However, sometimes you need more control. In these cases, we can use the full notation to express a more fully-fledged signature below.
+
+```python
+class GenerateSearchQuery(dspy.Signature):
+ """Write a simple search query that will help answer a complex question."""
+
+ context = dspy.InputField(desc="may contain relevant facts")
+ question = dspy.InputField()
+ query = dspy.OutputField()
+
+### inside your program's __init__ function
+self.generate_answer = dspy.ChainOfThought(GenerateSearchQuery)
+```
+
+You can optionally provide a `prefix` and/or `desc` key for each input or output field to refine or constraint the behavior of modules using your signature.
+
+
+#### 3.b) Asking **DSPy** to automatically optimize your program with `dspy.teleprompt.*`
+
+After defining the `RAG` program, we can **compile** it. Compiling a program will update the parameters stored in each module. For large LMs, this is primarily in the form of creating and validating good demonstrations for inclusion in your prompt(s).
+
+Compiling depends on three things: a (potentially tiny) training set, a metric for validation, and your choice of teleprompter from **DSPy**. **Teleprompters** are powerful optimizers (included in **DSPy**) that can learn to bootstrap and select effective prompts for the modules of any program. (The "tele-" in the name means "at a distance", i.e., automatic prompting at a distance.)
+
+**DSPy** typically requires very minimal labeling. For example, our `RAG` pipeline may work well with just a handful of examples that contain a **question** and its (human-annotated) **answer**. Your pipeline may involve multiple complex steps: our basic `RAG` example includes a retrieved context, a chain of thought, and the answer. However, you only need labels for the initial question and the final answer. **DSPy** will bootstrap any intermediate labels needed to support your pipeline. If you change your pipeline in any way, the data bootstrapped will change accordingly!
+
+
+```python
+my_rag_trainset = [
+ dspy.Example(
+ question="Which award did Gary Zukav's first book receive?",
+ answer="National Book Award"
+ ),
+ ...
+]
+```
+
+Second, define your validation logic, which will express some constraints on the behavior of your program or individual modules. For `RAG`, we might express a simple check like this:
+
+```python
+def validate_context_and_answer(example, pred, trace=None):
+ # check the gold label and the predicted answer are the same
+ answer_match = example.answer.lower() == pred.answer.lower()
+
+ # check the predicted answer comes from one of the retrieved contexts
+ context_match = any((pred.answer.lower() in c) for c in pred.context)
+
+ return answer_match and context_match
+```
+
+
+Different teleprompters offer various tradeoffs in terms of how much they optimize cost versus quality, etc. For `RAG`, we might use the simple teleprompter called `BootstrapFewShot`. To do so, we instantiate the teleprompter itself with a validation function `my_rag_validation_logic` and then compile against some training set `my_rag_trainset`.
+
+```python
+from dspy.teleprompt import BootstrapFewShot
+
+teleprompter = BootstrapFewShot(metric=my_rag_validation_logic)
+compiled_rag = teleprompter.compile(RAG(), trainset=my_rag_trainset)
+```
+
+If we now use `compiled_rag`, it will invoke our LM with rich prompts with few-shot demonstrations of chain-of-thought retrieval-augmented question answering on our data.
-## Getting Started
-Our [intro notebook](intro.ipynb).
+## 4) Documentation & Tutorials
+While we work on new tutorials and documentation, please check out **[our intro notebook](intro.ipynb)**.
-## Key Concepts
+Or open it directly in free Google Colab: [<img align="center" src="https://colab.research.google.com/assets/colab-badge.svg" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/v2/intro.ipynb)
-in terms of things DSPy can do for you (or allows you to do easily)
-and code examples for a flavor of the abstraction and ergonomics.
+###
+<details>
+ <summary><h3 style="display: inline">Intro Tutorial [coming soon]</h3></summary>
-## 📜 Reading More
+----
-Follow @lateinteraction on Twitter.
+**[Intro-01] Getting Started: High Quality Pipelined Prompts with Minimal Effort**
-OLD stuff:
+**[Intro-02] Using DSPy For Your Own Task: Building Blocks**
-You can get an overview via our Twitter threads:
-* [**Introducing DSP**](https://twitter.com/lateinteraction/status/1617953413576425472) (Jan 24, 2023)
-* [**Releasing the DSP Compiler (v0.1)**](https://twitter.com/lateinteraction/status/1625231662849073160) (Feb 13, 2023)
+**[Intro-03] Adding Complexity: Multi-stage Programs**
-And read more in the academic paper:
-* [**Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP**](https://arxiv.org/abs/2212.14024.pdf)
+**[Intro-04] Adding Complexity for Your Own Task: Design Patterns**
+</details>
-## Contributors
+----
+###
+<details>
+ <summary><h3 style="display: inline">Advanced Demos [coming soon]</h3></summary>
+----
-## ✍️ Reference
+
+**[Advanced-01] Long-Form QA & Programmatic Evaluation.**
+
+**[Advanced-02] Programmatic Evaluation II & Dataset Creation.**
+
+**[Advanced-03] Compiling & Teleprompters.**
+
+**[Advanced-04] Extending DSPy with Modules or Teleprompters.**
+
+**[Advanced-05]: Agents and General Tool Use in DSPy.**
+
+**[Advanced-06]: Reproducibility, Saving Programs, and Advanced Caching.**
+
+
+</details>
+
+----
+
+
+###
+<details>
+ <summary><h3 style="display: inline">Module Reference [coming soon]</h3></summary>
+
+----
+
+
+#### Language Model Clients
+
+- `dspy.OpenAI`
+- `dspy.Cohere`
+- `dspy.TGI`
+- `dspy.VLLM` [porting soon]
+
+#### Retrieval Model Clients
+
+- `dspy.ColBERTv2`
+- `dspy.AzureCognitiveSearch` [porting soon]
+
+
+#### Signatures
+
+- `dspy.Signature`
+- `dspy.InputField`
+- `dspy.OutputField`
+
+#### Modules
+
+- `dspy.Predict`
+- `dspy.Retrieve`
+- `dspy.ChainOfThought`
+- `dspy.SelfConsistency` [coming soon; use functional `dspy.majority` now]
+- `dspy.Reflection` [coming soon]
+- `dspy.MultiChainReasoning` [coming soon]
+
+
+#### Teleprompters
+
+- `dspy.teleprompt.LabeledFewShot`
+- `dspy.teleprompt.BootstrapFewShot`
+- `dspy.teleprompt.BootstrapFewShotWithRandomSearch`
+- `dspy.teleprompt.BootstrapFinetune` [porting soon]
+
+</details>
+
+----
+
+
+
+## 5) FAQ: Is DSPy right for me?
+
+The **DSPy** philosophy and abstraction differ significantly from other libraries and frameworks, so it's usually straightforward to decide when **DSPy** is (or isn't) the right framework for your usecase.
+
+If you're a NLP/AI researcher (or a practitioner exploring new pipelines or new tasks), the answer is generally an invariable **yes**. If you're a practitioner doing other things, please read on.
+
+
+####
+<details>
+ <summary><h4 style="display: inline">[5.a] DSPy vs. thin wrappers around prompts (OpenAI API, MiniChain, basic templating, etc.)</h4></summary>
+
+----
+
+In other words: _Why can't I just write my prompts directly as string templates?_ Well, for extremely simple settings, this _might_ work just fine. (If you're familiar with neural networks, this is like expressing a tiny two-layer NN as a Python for-loop. It kinda works.)
+
+However, when you need higher quality (or manageable cost), then you need to iteratively explore multi-stage decomposition, improved prompting, data bootstrapping, careful finetuning, retrieval augmentation, and/or using smaller (or cheaper, or local) models. The true expressive power of building with foundation models lies in the interactions between these pieces. But every time you change one piece, you likely break (or weaken) multiple other components.
+
+**DSPy** cleanly abstracts away (_and_ powerfully optimizes) the parts of these interactions that are external to your actual system design. It lets you focus on designing the module-level interactions: the _same program_ expressed in 10 or 20 lines of **DSPy** can easily be compiled into multi-stage instructions for `GPT-4`, detailed prompts for `Llama2-13b`, or finetunes for `T5-base`.
+
+Oh, and you wouldn't need to maintain long, brittle, model-specific strings at the core of your project anymore.
+
+</details>
+
+----
+
+####
+<details>
+ <summary><h4 style="display: inline">[5.b] DSPy vs. application development libraries like LangChain, LlamaIndex</h4></summary>
+
+----
+
+> _Note: If you use LangChain as a thin wrapper around your own prompt strings, refer to answer [5.a] instead._
+
+
+LangChain and LlamaIndex are popular libraries that target high-level application development with LMs. They offer many _batteries-included_, pre-built application modules that plug in with your data or configuration. In practice, many usecases genuinely _don't need_ any special components indeed. If you'd be happy to use someone's generic, off-the-shelf prompt for question answering over PDFs or standard text-to-SQL as long as it's easy to set up on your data, then you will probably find a very rich ecosystem in these libraries.
+
+
+Unlike these libraries, **DSPy** doesn't internally contain hand-crafted prompts that target specific applications you can build. Instead, **DSPy** introduces a very small set of much more powerful and general-purpose modules _that can learn to prompt (or finetune) your LM within your pipeline on your data_.
+
+**DSPy** offers a whole different degree of modularity: when you change your data, make tweaks to your program's control flow, or change your target LM, the **DSPy compiler** can map your program into a new set of prompts (or finetunes) that are optimized specifically for this pipeline. Because of this, you may find that **DSPy** obtains the highest quality for your task, with the least effort, provided you're willing to implement (or extend) your own short program.
+
+> If you're familiar with neural networks, this is like the difference between PyTorch (i.e., representing **DSPy**) and HuggingFace Transformers (i.e., representing the higher-level libraries). If you simply want to use off-the-shelf `BERT-base-uncased` or `GPT2-large` or apply minimal finetuning to them, HF Transformers makes it very straightforward. If, however, you're looking to build your own architecture (or extend an existing one significantly), you have to quickly drop down into something much more modular like PyTorch. Luckily, HF Transformers _is_ implemented in backends like PyTorch. We are similarly excited about high-level wrapper around **DSPy** for common applications. If this is implemented using **DSPy**, your high-level application can also adapt significantly to your data in a way that static prompt chains won't. Please [open an issue](https://github.com/stanfordnlp/dspy/issues/new) if this is something you want to help with.
+
+
+</details>
+
+----
+
+
+####
+<details>
+ <summary><h4 style="display: inline">[5.c] DSPy vs. generation control libraries like Guidance, LLMQL, RELM, Outlines</h4></summary>
+
+----
+
+
+Guidance, LLMQL, RELM, and Outlines are all exciting new libraries for controlling the individual completions of LMs, e.g., if you want to enforce JSON output schema or constrain sampling to a particular regular expression.
+
+This is very useful in many settings, but it's generally focused on low-level, structured control of a single LM call. It doesn't help ensure the JSON (or structured output) you get is going to be correct or useful for your task.
+
+In contrast, **DSPy** automatically optimizes the prompts in your programs to align them with various task needs, which may also include producing valid structured ouputs. That said, we are considering allowing **Signatures** in **DSPy** to express regex-like constraints that are implemented by these libraries.
+
+
+
+</details>
+
+
+----
+
+
+
+
+## Contributors & Acknowledgements
+
+**DSPy** is led by **Omar Khattab** at Stanford NLP with **Chris Potts** and **Matei Zaharia**.
+
+Key contributors and team members include **Aranv Singhvi**, **Paridhi Maheshwari**, **Keshav Santhanam**, **Sri Vardhamanan**, **Eric Zhang**, **Hanna Moazam**, and **Thomas Joshi**.
+
+**DSPy** includes important contributions from **Igor Kotenkov** and reflects discussions with **Lisa Li**, **David Hall**, **Ashwin Paranjape**, **Heather Miller**, **Percy Liang**, and many others.
+
+
+
+
+
+## 📜 Citation & Reading More
+
+To stay up to date or learn more, follow [@lateinteraction](https://twitter.com/lateinteraction) on Twitter.
If you use DSPy (or DSPv1) in a research paper, please cite our work as follows:
@@ -61,3 +387,8 @@ If you use DSPy (or DSPv1) in a research paper, please cite our work as follows:
year={2022}
}
```
+
+You can also read more about the old v1 of our framework (Demonstrate–Search–Predict, or DSP):
+* [**Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP**](https://arxiv.org/abs/2212.14024.pdf) (Academic Paper, Dec 2022)
+* [**Introducing DSP**](https://twitter.com/lateinteraction/status/1617953413576425472) (Twitter Thread, Jan 2023)
+* Thread [**Releasing the DSP Compiler (v0.1)**](https://twitter.com/lateinteraction/status/1625231662849073160) (Twitter Thread, Feb 2023)
diff --git a/dspy/predict/chain_of_thought.py b/dspy/predict/chain_of_thought.py
index 6d9c4d44c..d41003c8d 100644
--- a/dspy/predict/chain_of_thought.py
+++ b/dspy/predict/chain_of_thought.py
@@ -12,7 +12,7 @@ class ChainOfThought(Predict):
self.activated = activated
- # signature = signature.signature
+ signature = self.signature
*keys, last_key = signature.kwargs.keys()
DEFAULT_RATIONALE_TYPE = dsp.Type(prefix="Reasoning: Let's think step by step in order to",
diff --git a/dspy/predict/predict.py b/dspy/predict/predict.py
index 20f48049f..5181a60e7 100644
--- a/dspy/predict/predict.py
+++ b/dspy/predict/predict.py
@@ -3,6 +3,8 @@ import random
from dspy.predict.parameter import Parameter
from dspy.primitives.prediction import Prediction
+from dspy.signatures.field import InputField, OutputField
+from dspy.signatures.signature import infer_prefix
class Predict(Parameter):
@@ -11,6 +13,31 @@ class Predict(Parameter):
self.signature = signature #.signature
self.config = config
self.reset()
+
+ # if the signature is a string
+ if isinstance(signature, str):
+ inputs, outputs = signature.split("->")
+ inputs, outputs = inputs.split(","), outputs.split(",")
+ inputs, outputs = [field.strip() for field in inputs], [field.strip() for field in outputs]
+
+ assert all(len(field.split()) == 1 for field in (inputs + outputs))
+
+ inputs_ = ', '.join([f"`{field}`" for field in inputs])
+ outputs_ = ', '.join([f"`{field}`" for field in outputs])
+
+ instructions = f"""Given the fields {inputs_}, produce the fields {outputs_}."""
+
+ inputs = {k: InputField() for k in inputs}
+ outputs = {k: OutputField() for k in outputs}
+
+ for k, v in inputs.items():
+ v.finalize(k, infer_prefix(k))
+
+ for k, v in outputs.items():
+ v.finalize(k, infer_prefix(k))
+
+ self.signature = dsp.Template(instructions, **inputs, **outputs)
+
def reset(self):
self.lm = None
diff --git a/dspy/teleprompt/bootstrap.py b/dspy/teleprompt/bootstrap.py
index e61503470..b6d1e1822 100644
--- a/dspy/teleprompt/bootstrap.py
+++ b/dspy/teleprompt/bootstrap.py
@@ -127,7 +127,12 @@ class BootstrapFewShot(Teleprompter):
if success:
for step in trace:
predictor, inputs, outputs = step
- demo = Example(augmented=True, dspy_uuid=example.dspy_uuid, **inputs, **outputs)
+
+ if 'dspy_uuid' in example:
+ demo = Example(augmented=True, dspy_uuid=example.dspy_uuid, **inputs, **outputs)
+ else:
+ # TODO: FIXME: This is a hack. RandomSearch will complain for now in this edge case.
+ demo = Example(augmented=True, **inputs, **outputs)
try:
predictor_name = self.predictor2name[id(predictor)]
diff --git a/intro.ipynb b/intro.ipynb
index c4dd6f856..0b21fd60f 100644
--- a/intro.ipynb
+++ b/intro.ipynb
@@ -62,20 +62,12 @@
"# Set up the cache for this notebook\n",
"os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(repo_path, 'cache')\n",
"\n",
- "\"\"\"\n",
- "## TODO: Consider this\n",
+ "import pkg_resources # Install the package if it's not installed\n",
"if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",
" !pip install -U pip\n",
" !pip install -e $repo_path\n",
- "\"\"\"\n",
"\n",
- "try:\n",
- " import dspy\n",
- "except Exception:\n",
- " !pip install -U pip\n",
- " !pip install -e $repo_path\n",
- " \n",
- " import dspy"
+ "import dspy"
]
},
{
diff --git a/intro_with_mlc.ipynb b/intro_with_mlc.ipynb
deleted file mode 100644
index 5c348756e..000000000
--- a/intro_with_mlc.ipynb
+++ /dev/null
@@ -1,953 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "<img src=\"docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",
- "\n",
- "## **DSPy**: Programming with Foundation Models\n",
- "\n",
- "[<img align=\"center\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" />](https://colab.research.google.com/github/stanfordnlp/dspy/blob/v2/intro.ipynb)\n",
- "\n",
- "!! TODO: Fix the Colab link from branch v2 to main when we push to main\n",
- "\n",
- "!! TODO: Fix the setting up thing when merged too"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This notebook introduces the **DSPy** framework for **Programming with Foundation Models**, i.e., language models (LMs) and retrieval models (RMs).\n",
- "\n",
- "We built **DSPy** because we believe there is no ‟single best pipeline” for using LMs across tasks, and we believe you should not need to manually tweak prompts to get your pipelines to work well.\n",
- "\n",
- "To realize this, **DSPy** provides powerful abstractions for you to **build your own pipelines** for your own tasks using clean Pythonic operators over **modules that compose and learn**!"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 0] Setting Up\n",
- "\n",
- "**DSPy** is from the authors of ColBERT and the Demonstrate-Search-Predict framework (i.e., DSP, which is the predecessor of **DSPy**). No background on DSP or ColBERT is needed at all, though!\n",
- "\n",
- "Let's begin by setting things up. The snippet below will also install **DSPy** if it's not there already."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "#FIXME - evaluate functions error out since downloading mlc models triggers python 3.10 which causes errors - ValueError: Only callable can be used as callback\n",
- "\n",
- "\n",
- "# %load_ext autoreload\n",
- "# %autoreload 2\n",
- "\n",
- "import sys\n",
- "import os\n",
- "\n",
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab\n",
- " repo_path = 'dspy'\n",
- " !git -C $repo_path pull origin v2 || git clone -b v2 https://github.com/stanfordnlp/dspy $repo_path\n",
- "except:\n",
- " repo_path = '.'\n",
- "\n",
- "if repo_path not in sys.path:\n",
- " sys.path.append(repo_path)\n",
- "\n",
- "# Set up the cache for this notebook\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(repo_path, 'cache')\n",
- "\n",
- "\"\"\"\n",
- "## TODO: Consider this\n",
- "if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",
- " !pip install -U pip\n",
- " !pip install -e $repo_path\n",
- "\"\"\"\n",
- "\n",
- "try:\n",
- " import dspy\n",
- "except Exception:\n",
- " !pip install -U pip\n",
- " !pip install -e $repo_path\n",
- " \n",
- " import dspy"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "#NOTE - run cell above, wait for it to error out saying it can't find mlc and then run cell below, and then rerun the cell above\n",
- "#FIXME - running the MLC prerequisites first causes errors as when dspy is imported, it is not suited for python 3.10 and errors out"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "MLC prerequisites"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!pip install --pre --force-reinstall mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 -f https://mlc.ai/wheels\n",
- "!pip install transformers\n",
- "!git lfs install\n",
- "!mkdir -p dist/prebuilt\n",
- "!git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git dist/prebuilt/lib\n",
- "!cd dist/prebuilt && git clone https://huggingface.co/mlc-ai/mlc-chat-Llama-2-7b-chat-hf-q4f16_1"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 1] Getting Started\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM). **DSPy** supports multiple API and local models. In this notebook, we'll work with GPT-3.5 (`gpt-3.5-turbo`) and the retriever `ColBERTv2`.\n",
- "\n",
- "To make things easy, we've set up a ColBERTv2 server hosting a Wikipedia 2017 \"abstracts\" search index (i.e., containing first paragraph of each article from this [2017 dump](https://hotpotqa.github.io/wiki-readme.html)), so you don't need to worry about setting one up! It's free.\n",
- "\n",
- "**Note:** _If you want to run this notebook without changing the examples, you don't need an API key. All examples are already cached internally so you can inspect them!_"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "#set the chat module client here with model and model path for llama models\n",
- "turbo = dspy.ChatModuleClient(model='dist/prebuilt/mlc-chat-Llama-2-7b-chat-hf-q4f16_1', model_path='dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-cuda.so')\n",
- "colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",
- "\n",
- "dspy.settings.configure(lm=turbo, rm=colbertv2_wiki17_abstracts)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In the last line above, we configure **DSPy** to use the turbo LM and the ColBERTv2 retriever (over Wikipedia 2017 abstracts) by default. This will be easy to overwrite for local parts of our programs if needed."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### A word on the workflow"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "You can build your own **DSPy programs** for various tasks, e.g., question answering, information extraction, or text-to-SQL.\n",
- "\n",
- "Whatever the task, the general workflow is:\n",
- "\n",
- "1. **Collect a little bit of data.** Define examples of the inputs and outputs of your program (e.g., questions and their answers). This could just be a handful of quick examples you wrote down. If large datasets exist, the more the merrier!\n",
- "1. **Write your program.** Define the modules (i.e., sub-tasks) of your program and the way they should interact together to solve your task.\n",
- "1. **Define some validation logic.** What makes for a good run of your program? Maybe the answers need to have a certain length or stick to a particular format? Specify the logic that checks that.\n",
- "1. **Compile!** Ask **DSPy** to _compile_ your program using your data. The compiler will use your data and validation logic to optimize your program (e.g., prompts and modules) so it's efficient and effective!\n",
- "1. **Iterate.** Repeat the process by improving your data, program, validation, or by using more advanced features of the **DSPy** compiler.\n",
- "\n",
- "Let's now see this in action."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 2] Task Examples\n",
- "\n",
- "**DSPy** accomodates a wide variety of applications and tasks. **In this intro notebook, we will work on the example task of multi-hop question answering (QA).**\n",
- "\n",
- "Other notebooks and tutorials will present different tasks. Now, let us load a tiny sample from the HotPotQA multi-hop dataset."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from dspy.datasets import HotPotQA\n",
- "\n",
- "# Load the dataset.\n",
- "dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0)\n",
- "\n",
- "# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.\n",
- "trainset = [x.with_inputs('question') for x in dataset.train]\n",
- "devset = [x.with_inputs('question') for x in dataset.dev]\n",
- "\n",
- "len(trainset), len(devset)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We just loaded `trainset` (20 examples) and `devset` (50 examples). Each example in our **training set** contains just a **question** and its (human-annotated) **answer**.\n",
- "\n",
- "**DSPy** typically requires very minimal labeling. Whereas your pipeline may involve six or seven complex steps, you only need labels for the initial question and the final answer. **DSPy** will bootstrap any intermediate labels needed to support your pipeline. If you change your pipeline in any way, the data bootstrapped will change accordingly!\n",
- "\n",
- "Now, let's look at some data examples."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "train_example = trainset[0]\n",
- "print(f\"Question: {train_example.question}\")\n",
- "print(f\"Answer: {train_example.answer}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Examples in the **dev set** contain a third field, namely, **titles** of relevant Wikipedia articles. This is not essential but, for the sake of this intro, it'll help us get a sense of how well our programs are doing."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "dev_example = devset[18]\n",
- "print(f\"Question: {dev_example.question}\")\n",
- "print(f\"Answer: {dev_example.answer}\")\n",
- "print(f\"Relevant Wikipedia Titles: {dev_example.gold_titles}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "After loading the raw data, we'd applied `x.with_inputs('question')` to each example to tell **DSPy** that our input field in each example will be just `question`. Any other fields are labels or metadata that are not given to the system."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "print(f\"For this dataset, training examples have input keys {train_example.inputs().keys()} and label keys {train_example.labels().keys()}\")\n",
- "print(f\"For this dataset, dev examples have input keys {dev_example.inputs().keys()} and label keys {dev_example.labels().keys()}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note that there's nothing special about the HotPotQA dataset: it's just a list of examples.\n",
- "\n",
- "You can define your own examples as below. A future notebook will guide you through creating your own data in unusual or data-scarce settings, which is a context where **DSPy** excels.\n",
- "\n",
- "```\n",
- "dspy.Example(field1=value, field2=value2, ...)\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 3] Building Blocks\n",
- "\n",
- "In **DSPy**, we will maintain a clean separation between **defining your modules in a declarative way** and **calling them in a pipeline to solve the task**.\n",
- "\n",
- "This allows you to focus on the information flow of your pipeline. **DSPy** will then take your program and automatically optimize **how to prompt** (or finetune) LMs **for your particular pipeline** so it works well.\n",
- "\n",
- "If you have experience with PyTorch, you can think of DSPy as the PyTorch of the foundation model space. Before we see this in action, let's first understand some key pieces."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### Using the Language Model: **Signatures** & **Predictors**\n",
- "\n",
- "Every call to the LM in a **DSPy** program needs to have a **Signature**.\n",
- "\n",
- "A signature consists of three simple elements:\n",
- "\n",
- "- A minimal description of the sub-task the LM is supposed to solve.\n",
- "- A description of one or more input fields (e.g., input question) that will we will give to the LM.\n",
- "- A description of one or more output fields (e.g., the question's answer) that we will expect from the LM.\n",
- "\n",
- "Let's define a simple signature for basic question answering."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "class BasicQA(dspy.Signature):\n",
- " \"\"\"Answer questions with short factoid answers.\"\"\"\n",
- "\n",
- " question = dspy.InputField()\n",
- " answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In `BasicQA`, the docstring describes the sub-task here (i.e., answering questions). Each `InputField` or `OutputField` can optionally contain a description `desc` too. When it's not given, it's inferred from the field's name (e.g., `question`).\n",
- "\n",
- "Notice that there isn't anything special about this signature in **DSPy**. We can just as easily define a signature that takes a long snippet from a PDF and outputs structured information, for instance.\n",
- "\n",
- "Anyway, now that we have a signature, let's define and use a **Predictor**. A predictor is a module that knows how to use the LM to implement a signature. Importantly, predictors can **learn** to fit their behavior to the task!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Define the predictor.\n",
- "generate_answer = dspy.Predict(BasicQA)\n",
- "\n",
- "# Call the predictor on a particular input.\n",
- "pred = generate_answer(question=dev_example.question)\n",
- "\n",
- "# Print the input and the prediction.\n",
- "print(f\"Question: {dev_example.question}\")\n",
- "print(f\"Predicted Answer: {pred.answer}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In the example above, we asked the predictor about the the chef featured in \"Restaurant: Impossible\". The model outputs an answer (\"American\").\n",
- "\n",
- "For visibility, we can inspect how this extremely basic predictor implemented our signature. Let's inspect the history of our LM (**turbo**)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "turbo.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "It happens that this chef is both British and American, but we have no way of knowing if the model just guessed \"American\" because it's a common answer. In general, adding **retrieval** and **learning** will help the LM be more factual, and we'll explore this in a minute!\n",
- "\n",
- "But before we do that, how about we _just_ change the predictor? It would be nice to allow the model to elicit a chain of thought along with the prediction."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Define the predictor. Notice we're just changing the class. The signature BasicQA is unchanged.\n",
- "generate_answer_with_chain_of_thought = dspy.ChainOfThought(BasicQA)\n",
- "\n",
- "# Call the predictor on the same input.\n",
- "pred = generate_answer_with_chain_of_thought(question=dev_example.question)\n",
- "\n",
- "# Print the input, the chain of thought, and the prediction.\n",
- "print(f\"Question: {dev_example.question}\")\n",
- "print(f\"Thought: {pred.rationale.split('.', 1)[1].strip()}\")\n",
- "print(f\"Predicted Answer: {pred.answer}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This is indeed a better answer: the model figures out that the chef in question is **Robert Irvine** and correctly identifies that he's British.\n",
- "\n",
- "These predictors (`dspy.Predict` and `dspy.ChainOfThought`) can be applied to _any_ signature. As we'll see below, they can also be optimized to learn from your data and validation logic."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### Using the Retrieval Model\n",
- "\n",
- "Using the retriever is pretty simple. A module `dspy.Retrieve(k)` will search for the top-`k` passages that match a given query.\n",
- "\n",
- "By default, this will use the retriever we configured at the top of this notebook, namely, ColBERTv2 over a Wikipedia index."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "retrieve = dspy.Retrieve(k=3)\n",
- "topK_passages = retrieve(dev_example.question).passages\n",
- "\n",
- "print(f\"Top {retrieve.k} passages for question: {dev_example.question} \\n\", '-' * 30, '\\n')\n",
- "\n",
- "for idx, passage in enumerate(topK_passages):\n",
- " print(f'{idx+1}]', passage, '\\n')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Feel free to any other queries you like."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "retrieve(\"When was the first FIFA World Cup held?\").passages[0]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 4] Program 1: Basic Retrieval-Augmented Generation (“RAG”)\n",
- "\n",
- "Let's define our first complete program for this task. We'll build a retrieval-augmented pipeline for answer generation.\n",
- "\n",
- "Given a question, we'll search for the top-3 passages in Wikipedia and then feed them as context for answer generation.\n",
- "\n",
- "Let's start by defining this signature: `context, question --> answer`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "class GenerateAnswer(dspy.Signature):\n",
- " \"\"\"Answer questions with short factoid answers.\"\"\"\n",
- "\n",
- " context = dspy.InputField(desc=\"may contain relevant facts\")\n",
- " question = dspy.InputField()\n",
- " answer = dspy.OutputField(desc=\"often between 1 and 5 words\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Great. Now let's define the actual program. This is a class that inherits from `dspy.Module`.\n",
- "\n",
- "It needs two methods:\n",
- "\n",
- "- The `__init__` method will simply declare the sub-modules it needs: `dspy.Retrieve` and `dspy.ChainOfThought`. The latter is defined to implement our `GenerateAnswer` signature.\n",
- "- The `forward` method will describe the control flow of answering the question using the modules we have."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "class RAG(dspy.Module):\n",
- " def __init__(self, num_passages=3):\n",
- " super().__init__()\n",
- "\n",
- " self.retrieve = dspy.Retrieve(k=num_passages)\n",
- " self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n",
- " \n",
- " def forward(self, question):\n",
- " context = self.retrieve(question).passages\n",
- " prediction = self.generate_answer(context=context, question=question)\n",
- " return dspy.Prediction(context=context, answer=prediction.answer)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### Compiling the RAG program\n",
- "\n",
- "Having defined this program, let's now **compile** it. Compiling a program will update the parameters stored in each module. In our setting, this is primarily in the form of collecting and selecting good demonstrations for inclusion in your prompt(s).\n",
- "\n",
- "Compiling depends on three things:\n",
- "\n",
- "1. **A training set.** We'll just use our 20 question–answer examples from `trainset` above.\n",
- "1. **A metric for validation.** We'll define a quick `validate_context_and_answer` that checks that the predicted answer is correct. It'll also check that the retrieved context does actually contain that answer.\n",
- "1. **A specific teleprompter.** The **DSPy** compiler includes a number of **teleprompters** that can optimize your programs."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "**Teleprompters:** Teleprompters are powerful optimizers that can take any program and learn to bootstrap and select effective prompts for its modules. Hence the name, which means \"prompting at a distance\".\n",
- "\n",
- "Different teleprompters offer various tradeoffs in terms of how much they optimize cost versus quality, etc. We will use a simple default `BootstrapFewShot` in this notebook.\n",
- "\n",
- "\n",
- "_If you're into analogies, you could think of this as your training data, your loss function, and your optimizer in a standard DNN supervised learning setup. Whereas SGD is a basic optimizer, there are more sophisticated (and more expensive!) ones like Adam or RMSProp._"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from dspy.teleprompt import BootstrapFewShot\n",
- "\n",
- "# Validation logic: check that the predicted answer is correct.\n",
- "# Also check that the retrieved context does actually contain that answer.\n",
- "def validate_context_and_answer(example, pred, trace=None):\n",
- " answer_EM = dspy.evaluate.answer_exact_match(example, pred)\n",
- " answer_PM = dspy.evaluate.answer_passage_match(example, pred)\n",
- " return answer_EM and answer_PM\n",
- "\n",
- "# Set up a basic teleprompter, which will compile our RAG program.\n",
- "teleprompter = BootstrapFewShot(metric=validate_context_and_answer)\n",
- "\n",
- "# Compile!\n",
- "compiled_rag = teleprompter.compile(RAG(), trainset=trainset)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now that we've compiled our RAG program, let's try it out."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Ask any question you like to this simple RAG program.\n",
- "my_question = \"What castle did David Gregory inherit?\"\n",
- "\n",
- "# Get the prediction. This contains `pred.context` and `pred.answer`.\n",
- "pred = compiled_rag(my_question)\n",
- "\n",
- "# Print the contexts and the answer.\n",
- "print(f\"Question: {my_question}\")\n",
- "print(f\"Predicted Answer: {pred.answer}\")\n",
- "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Excellent. How about we inspect the last prompt for the LM?"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "turbo.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Even though we haven't written any of this detailed demonstrations, we see that **DSPy** was able to bootstrap this 3,000 token prompt for **3-shot retrieval augmented generation with hard negative passages and chain of thought** from our extremely simple program.\n",
- "\n",
- "This illustrates the power of composition and learning. Of course, this was just generated by a particular teleprompter, which may or may not be perfect in each setting. As you'll see in **DSPy**, there is a large but systematic space of options you have to optimize and validate the quality and cost of your programs.\n",
- "\n",
- "If you're so inclined, you can easily inspect the learned objects themselves."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "for name, parameter in compiled_rag.named_predictors():\n",
- " print(name)\n",
- " print(parameter.demos[0])\n",
- " print()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### Evaluating the Answers\n",
- "\n",
- "We can now evaluate our `compiled_rag` program on the dev set. Of course, this tiny set is _not_ meant to be a reliable benchmark, but it'll be instructive to use it for illustration.\n",
- "\n",
- "For a start, let's evaluate the accuracy (exact match) of the predicted answer."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from dspy.evaluate.evaluate import Evaluate\n",
- "\n",
- "# Set up the `evaluate_on_hotpotqa` function. We'll use this many times below.\n",
- "evaluate_on_hotpotqa = Evaluate(devset=devset, num_threads=1, display_progress=True, display_table=5)\n",
- "\n",
- "# Evaluate the `compiled_rag` program with the `answer_exact_match` metric.\n",
- "metric = dspy.evaluate.answer_exact_match\n",
- "evaluate_on_hotpotqa(compiled_rag, metric=metric)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### Evaluating the Retrieval\n",
- "\n",
- "It may also be instructive to look at the accuracy of retrieval. There are multiple ways to do this. Often, we can just check whether the rertieved passages contain the answer.\n",
- "\n",
- "That said, since our dev set includes the gold titles that should be retrieved, we can just use these here."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def gold_passages_retrieved(example, pred, trace=None):\n",
- " gold_titles = set(map(dspy.evaluate.normalize_text, example['gold_titles']))\n",
- " found_titles = set(map(dspy.evaluate.normalize_text, [c.split(' | ')[0] for c in pred.context]))\n",
- "\n",
- " return gold_titles.issubset(found_titles)\n",
- "\n",
- "#FIXME - errors out over here due to reload extension and python3.10 from mlc incompatability \n",
- "compiled_rag_retrieval_score = evaluate_on_hotpotqa(compiled_rag, metric=gold_passages_retrieved)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Although this simple `compiled_rag` program is able to answer a decent fraction of the questions correctly (on this tiny set, over 40%), the quality of retrieval is much lower.\n",
- "\n",
- "This potentially suggests that the LM is often relying on the knowledge it memorized during training to answer questions. To address this weak retrieval, let's explore a second program that involves more advanced search behavior."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### 5] Program 2: Multi-Hop Search (“Baleen”)\n",
- "\n",
- "From exploring the harder questions in the training/dev sets, it becomes clear that a single search query is often not enough for this task. For instance, this can be seen when a question ask about, say, the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it wouldn't figure out when he was born.\n",
- "\n",
- "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary. Using **DSPy**, we can easily simulate such systems in a few lines of code.\n",
- "\n",
- "\n",
- "We'll still use the `GenerateAnswer` signature from the RAG implementation above. All we need now is a **signature** for the \"hop\" behavior: taking some partial context and a question, generate a search query to find missing information."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "class GenerateSearchQuery(dspy.Signature):\n",
- " \"\"\"Write a simple search query that will help answer a complex question.\"\"\"\n",
- "\n",
- " context = dspy.InputField(desc=\"may contain relevant facts\")\n",
- " question = dspy.InputField()\n",
- " query = dspy.OutputField()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note: We could have written `context = GenerateAnswer.signature.context` to avoid duplicating the description of the `context` field.\n",
- "\n",
- "Now, let's define the program itself `SimplifiedBaleen`. There are many possible ways to implement this, but we'll keep this version down to the key elements for simplicity."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from dsp.utils import deduplicate\n",
- "\n",
- "class SimplifiedBaleen(dspy.Module):\n",
- " def __init__(self, passages_per_hop=3, max_hops=2):\n",
- " super().__init__()\n",
- "\n",
- " self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]\n",
- " self.retrieve = dspy.Retrieve(k=passages_per_hop)\n",
- " self.generate_answer = dspy.ChainOfThought(GenerateAnswer)\n",
- " self.max_hops = max_hops\n",
- " \n",
- " def forward(self, question):\n",
- " context = []\n",
- " \n",
- " for hop in range(self.max_hops):\n",
- " query = self.generate_query[hop](context=context, question=question).query\n",
- " passages = self.retrieve(query).passages\n",
- " context = deduplicate(context + passages)\n",
- "\n",
- " pred = self.generate_answer(context=context, question=question)\n",
- " return dspy.Prediction(context=context, answer=pred.answer)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "As we can see, the `__init__` method defines a few key sub-modules:\n",
- "\n",
- "- **generate_query**: For each hop, we will have one `dspy.ChainOfThought` predictor with the `GenerateSearchQuery` signature.\n",
- "- **retrieve**: This module will do the actual search, using the generated queries.\n",
- "- **generate_answer**: This `dspy.Predict` module will be used after all the search steps. It has a `GenerateAnswer`, to actually produce an answer.\n",
- "\n",
- "The `forward` method uses these sub-modules in simple control flow.\n",
- "\n",
- "1. First, we'll loop up to `self.max_hops` times.\n",
- "1. In each iteration, we'll generate a search query using the predictor at `self.generate_query[hop]`.\n",
- "1. We'll retrieve the top-k passages using that query.\n",
- "1. We'll add the (deduplicated) passages to our accumulator of `context`.\n",
- "1. After the loop, we'll use `self.generate_answer` to produce an answer.\n",
- "1. We'll return a prediction with the retrieved `context` and predicted `answer`."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### Inspect the zero-shot version of the Baleen program\n",
- "\n",
- "We will also compile this program shortly. But, before that, we can try it out in a \"zero-shot\" setting (i.e., without any compilation).\n",
- "\n",
- "Using a program in zero-shot (uncompiled) setting doesn't mean that quality will be bad. It just means that we're bottlenecked directly by the reliability of the underlying LM to understand our sub-tasks from minimal instructions.\n",
- "\n",
- "This is often just fine when using the most expensive/powerful models (e.g., GPT-4) on the easiest and most standard tasks (e.g., answering simple questions about popular entities).\n",
- "\n",
- "However, a zero-shot approach quickly falls short for more specialized tasks, for novel domains/settings, and for more efficient (or open) models. **DSPy** can help you in all of these settings."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Ask any question you like to this simple RAG program.\n",
- "my_question = \"How many storeys are in the castle that David Gregory inherited?\"\n",
- "\n",
- "# Get the prediction. This contains `pred.context` and `pred.answer`.\n",
- "uncompiled_baleen = SimplifiedBaleen() # uncompiled (i.e., zero-shot) program\n",
- "pred = uncompiled_baleen(my_question)\n",
- "\n",
- "# Print the contexts and the answer.\n",
- "print(f\"Question: {my_question}\")\n",
- "print(f\"Predicted Answer: {pred.answer}\")\n",
- "print(f\"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's inspect the last **three** calls to the LM (i.e., generating the first hop's query, generating the second hop's query, and generating the answer)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "turbo.inspect_history(n=3)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### Compiling the Baleen program\n",
- "\n",
- "Now is the time to compile our multi-hop (`SimplifiedBaleen`) program.\n",
- "\n",
- "We will first define our validation logic, which will simply require that:\n",
- "\n",
- "- The predicted answer matches the gold answer.\n",
- "- The retrieved context contains the gold answer.\n",
- "- None of the generated queries is rambling (i.e., none exceeds 100 characters in length).\n",
- "- None of the generated queries is roughly repeated (i.e., none is within 0.8 or higher F1 score of earlier queries)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def validate_context_and_answer_and_hops(example, pred, trace=None):\n",
- " if not dspy.evaluate.answer_exact_match(example, pred): return False\n",
- " if not dspy.evaluate.answer_passage_match(example, pred): return False\n",
- "\n",
- " hops = [example.question] + [outputs.query for *_, outputs in trace if 'query' in outputs]\n",
- "\n",
- " if max([len(h) for h in hops]) > 100: return False\n",
- " if any(dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8) for idx in range(2, len(hops))): return False\n",
- "\n",
- " return True"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Like we did for RAG, we'll use one of the most basic teleprompters in **DSPy**, namely, `BootstrapFewShot`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "teleprompter = BootstrapFewShot(metric=validate_context_and_answer_and_hops)\n",
- "compiled_baleen = teleprompter.compile(SimplifiedBaleen(), teacher=SimplifiedBaleen(passages_per_hop=2), trainset=trainset)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "##### Evaluating the Retrieval\n",
- "\n",
- "Earlier, it appeared like our simple RAG program was not very effective at finding all evidence required for answering each question. Is this resolved by the adding some extra steps in the `forward` function of `SimplifiedBaleen`? What about compiling, does it help for that? \n",
- "\n",
- "The answer for these questions is not always going to be obvious. However, **DSPy** makes it extremely easy to try many diverse approaches with minimal effort.\n",
- "\n",
- "Let's evaluate the quality of retrieval of our compiled and uncompiled Baleen pipelines!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "uncompiled_baleen_retrieval_score = evaluate_on_hotpotqa(uncompiled_baleen, metric=gold_passages_retrieved, display=False)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "compiled_baleen_retrieval_score = evaluate_on_hotpotqa(compiled_baleen, metric=gold_passages_retrieved)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "print(f\"## Retrieval Score for RAG: {compiled_rag_retrieval_score}\") # note that for RAG, compilation has no effect on the retrieval step\n",
- "print(f\"## Retrieval Score for uncompiled Baleen: {uncompiled_baleen_retrieval_score}\")\n",
- "print(f\"## Retrieval Score for compiled Baleen: {compiled_baleen_retrieval_score}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Excellent! There might be something to this compiled, multi-hop program then. But this is far from all you can do: **DSPy** gives you a clean space of composable operators to deal with any shortcomings you see.\n",
- "\n",
- "We can inspect a few concrete examples. If we see failure causes, we can:\n",
- "\n",
- "1. Expand our pipeline by using additional sub-modules (e.g., maybe summarize after retrieval?)\n",
- "1. Modify our pipeline by using more complex logic (e.g., maybe we need to break out of the multi-hop loop if we found all information we need?) \n",
- "1. Refine our validation logic (e.g., maybe use a metric that use a second **DSPy** program to do the answer evaluation, instead of relying on strict string matching)\n",
- "1. Use a different teleprompter to optimize your pipeline more aggressively.\n",
- "1. Add more or better training examples!\n",
- "\n",
- "\n",
- "Or, if you really want, we can tweak the descriptions in the Signatures we use in your program to make them more precisely suited for their sub-tasks. This is akin to prompt engineering and should be a final resort, given the other powerful options that **DSPy** gives us!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "compiled_baleen(\"How many storeys are in the castle that David Gregory inherited?\")\n",
- "turbo.inspect_history(n=3)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "base",
- "language": "python",
- "name": "base"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.12"
- },
- "orig_nbformat": 4
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}