DSPy: The framework for programming—not prompting—language models
Python
Add first demo program
Rick Battlee8f679968277
diff --git a/demos/Multi-hop QA Program 1 - Direct Query.ipynb b/demos/Multi-hop QA Program 1 - Direct Query.ipynbnew file mode 100644index 000000000..fd9d058b7--- /dev/null+++ b/demos/Multi-hop QA Program 1 - Direct Query.ipynb@@ -0,0 +1,489 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "id": "551cc8d5-fd87-4d94-ba16-dc2a48698a6f",+ "metadata": {},+ "source": [+ "# Multi-hop QA Program 1: Direct Query\n",+ "\n",+ "This notebook is a stand-alone version of Program 1 from the intro notebook."+ ]+ },+ {+ "cell_type": "markdown",+ "id": "4e59bc24-3aec-459d-a1a5-e98717431907",+ "metadata": {},+ "source": [+ "### Installation\n",+ "\n",+ "If you haven't installed **DSP** already, let's do that.\n",+ "\n",+ "Note: If you're running this from a cloned copy of the repo, then you can skip this block."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "4834f1e4-c3b8-498e-9376-4241c3d2324f",+ "metadata": {},+ "outputs": [],+ "source": [+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",+ " import google.colab \n",+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",+ "except: pass\n",+ "\n",+ "!pip install -U pip dsp-ml"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "7d71eb82-0b21-4982-9b5f-f2ac6a8e5611",+ "metadata": {},+ "source": [+ "### Setting Up\n",+ "\n",+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",+ "\n",+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",+ "\n",+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",+ "\n",+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "id": "2b542158-bc91-48e2-b201-638c9e1ea606",+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Not loading Cohere because it is not installed.\n"+ ]+ }+ ],+ "source": [+ "%load_ext autoreload\n",+ "%autoreload 2\n",+ "\n",+ "try: import google.colab; root_path = 'dsp'\n",+ "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",+ "except: root_path = '../'\n",+ "\n",+ "import os\n",+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",+ "\n",+ "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",+ "import sys\n",+ "sys.path.insert(0, '../')\n",+ "\n",+ "import dsp\n",+ "\n",+ "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",+ "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",+ "\n",+ "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",+ "rm = dsp.ColBERTv2(url=colbert_server)\n",+ "\n",+ "dsp.settings.configure(lm=lm, rm=rm)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "b3289955-e877-4b3e-8b8c-5e9e5a1d2921",+ "metadata": {},+ "source": [+ "### Task Examples\n",+ "\n",+ "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",+ "\n",+ "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",+ "\n",+ "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "id": "ad480ffd-8299-4b36-81d9-493a27d1b28d",+ "metadata": {},+ "outputs": [],+ "source": [+ "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",+ " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",+ " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",+ " ('In what year was the star of To Hell and Back born?', ['1925']),\n",+ " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",+ " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",+ "\n",+ "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "2870fe3d-ffb8-4214-9279-ac040aad1942",+ "metadata": {},+ "source": [+ "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "id": "2ebf9b6d-4dfd-4023-8ded-64dfc11348ae",+ "metadata": {},+ "outputs": [],+ "source": [+ "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",+ " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",+ " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",+ " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",+ " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",+ " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",+ " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",+ " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",+ " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",+ " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",+ " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",+ " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",+ " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",+ "\n",+ "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "d7df81ab-9c30-43ec-abad-25b5a58f0cf1",+ "metadata": {},+ "source": [+ "### Program Definition\n",+ "\n",+ "Direct Query is the simplest program for this task. We'll prompt **GPT-3.5** to answer each question based on its internal parameteric knowledge.\n",+ "\n",+ "We'll start by defining the `Template` that defines how we will communicate with the LM.\n",+ "\n",+ "Specifically, the question–answer template (`qa_template`) will include a question and a short answer for each example."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "id": "72d65b15-443d-4e3e-b327-b19a396ba5ae",+ "metadata": {},+ "outputs": [],+ "source": [+ "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",+ "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",+ "\n",+ "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "4610915b-fad0-4ecf-b994-b9df035e8845",+ "metadata": {},+ "source": [+ "Then, let's define the actual program, `Direct_Query_QA`. It'll accept a string (`question`) and returns another string (its short `answer`)."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "id": "926e9edb-2447-4b32-85e9-4105d5b0c39d",+ "metadata": {},+ "outputs": [],+ "source": [+ "def Direct_Query_QA(question: str) -> str:\n",+ " demos = dsp.sample(train, k=7)\n",+ " example = dsp.Example(question=question, demos=demos)\n",+ "\n",+ " example, completions = dsp.generate(qa_template)(example, stage='qa')\n",+ " return completions.answer"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "e5ea19e3-5a7c-4984-a9b3-eab408244912",+ "metadata": {},+ "source": [+ "Let's invoke the program on a sample question."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "id": "c3c96381-6dab-4214-8faf-1c649de5c5c8",+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",+ "E. L. Doctorow\n"+ ]+ }+ ],+ "source": [+ "print(dev[0].question)\n",+ "print(Direct_Query_QA(dev[0].question))"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "424c4ffb-4e71-4a91-a179-7cacadcbc2f5",+ "metadata": {},+ "source": [+ "Let's inspect the last call to the LM to learn more about the behavior of the program."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "id": "f8fe21c1-e185-44c3-8742-7fb19385e8a9",+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n",+ "\n",+ "\n",+ "\n",+ "Answer questions with short factoid answers.\n",+ "\n",+ "---\n",+ "\n",+ "Follow the following format.\n",+ "\n",+ "Question: ${the question to be answered}\n",+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",+ "\n",+ "---\n",+ "\n",+ "Question: Which award did the first book of Gary Zukav receive?\n",+ "Answer: U.S. National Book Award\n",+ "\n",+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",+ "Answer: Foxcatcher\n",+ "\n",+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",+ "Answer: Kevin Greutert\n",+ "\n",+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",+ "Answer: Butch Vig\n",+ "\n",+ "Question: What city was the victim of Joseph Druces working in?\n",+ "Answer: Boston, Massachusetts\n",+ "\n",+ "Question: In what year was the star of To Hell and Back born?\n",+ "Answer: 1925\n",+ "\n",+ "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",+ "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",+ "\n",+ "\n",+ "\n"+ ]+ }+ ],+ "source": [+ "lm.inspect_history(n=1)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "id": "886ff6e7-0ca9-4dff-a042-5357915924ac",+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 13/13 [00:00<00:00, 121.15it/s]"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Answered 3 / 13 (23.1%) correctly.\n"+ ]+ },+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<style type=\"text/css\">\n",+ "#T_60df5 th {\n",+ " text-align: left;\n",+ "}\n",+ "#T_60df5 td {\n",+ " text-align: left;\n",+ "}\n",+ "</style>\n",+ "<table id=\"T_60df5\">\n",+ " <thead>\n",+ " <tr>\n",+ " <th class=\"blank level0\" > </th>\n",+ " <th id=\"T_60df5_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_60df5_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_60df5_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",+ " <th id=\"T_60df5_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_60df5_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",+ " <td id=\"T_60df5_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",+ " <td id=\"T_60df5_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",+ " <td id=\"T_60df5_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_60df5_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",+ " <td id=\"T_60df5_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",+ " <td id=\"T_60df5_row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer</td>\n",+ " <td id=\"T_60df5_row1_col3\" class=\"data row1 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_60df5_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",+ " <td id=\"T_60df5_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",+ " <td id=\"T_60df5_row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",+ " <td id=\"T_60df5_row2_col3\" class=\"data row2 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_60df5_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",+ " <td id=\"T_60df5_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",+ " <td id=\"T_60df5_row3_col2\" class=\"data row3 col2\" >1966</td>\n",+ " <td id=\"T_60df5_row3_col3\" class=\"data row3 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_60df5_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",+ " <td id=\"T_60df5_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",+ " <td id=\"T_60df5_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",+ " <td id=\"T_60df5_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",+ " <td id=\"T_60df5_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",+ " <td id=\"T_60df5_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",+ " <td id=\"T_60df5_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",+ " <td id=\"T_60df5_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",+ " <td id=\"T_60df5_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",+ " <td id=\"T_60df5_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",+ " <td id=\"T_60df5_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",+ " <td id=\"T_60df5_row6_col3\" class=\"data row6 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",+ " <td id=\"T_60df5_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",+ " <td id=\"T_60df5_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",+ " <td id=\"T_60df5_row7_col2\" class=\"data row7 col2\" >July 30, 1953</td>\n",+ " <td id=\"T_60df5_row7_col3\" class=\"data row7 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",+ " <td id=\"T_60df5_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",+ " <td id=\"T_60df5_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",+ " <td id=\"T_60df5_row8_col2\" class=\"data row8 col2\" >Hudson River</td>\n",+ " <td id=\"T_60df5_row8_col3\" class=\"data row8 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",+ " <td id=\"T_60df5_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",+ " <td id=\"T_60df5_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",+ " <td id=\"T_60df5_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",+ " <td id=\"T_60df5_row9_col3\" class=\"data row9 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",+ " <td id=\"T_60df5_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",+ " <td id=\"T_60df5_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",+ " <td id=\"T_60df5_row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",+ " <td id=\"T_60df5_row10_col3\" class=\"data row10 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",+ " <td id=\"T_60df5_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",+ " <td id=\"T_60df5_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",+ " <td id=\"T_60df5_row11_col2\" class=\"data row11 col2\" >Venus</td>\n",+ " <td id=\"T_60df5_row11_col3\" class=\"data row11 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_60df5_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",+ " <td id=\"T_60df5_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",+ " <td id=\"T_60df5_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",+ " <td id=\"T_60df5_row12_col2\" class=\"data row12 col2\" >1457</td>\n",+ " <td id=\"T_60df5_row12_col3\" class=\"data row12 col3\" >❌</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n"+ ],+ "text/plain": [+ "<pandas.io.formats.style.Styler at 0x7fac858f92e0>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "23.1"+ ]+ },+ "execution_count": 8,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "from dsp.evaluation.utils import evaluate\n",+ "\n",+ "evaluate(Direct_Query_QA, dev)"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "Python (DSP)",+ "language": "python",+ "name": "dsp"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.9.16"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 5+}