Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Delete dsp_v1_demos directory

Omar Khattabebd5870e479a
diff --git a/dsp_v1_demos/Multi-hop QA Program 1 - Direct Query.ipynb b/dsp_v1_demos/Multi-hop QA Program 1 - Direct Query.ipynb
deleted file mode 100644
index fd9d058b7..000000000
--- a/dsp_v1_demos/Multi-hop QA Program 1 - Direct Query.ipynb
+++ /dev/null
@@ -1,489 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "551cc8d5-fd87-4d94-ba16-dc2a48698a6f",
- "metadata": {},
- "source": [
- "# Multi-hop QA Program 1: Direct Query\n",
- "\n",
- "This notebook is a stand-alone version of Program 1 from the intro notebook."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4e59bc24-3aec-459d-a1a5-e98717431907",
- "metadata": {},
- "source": [
- "### Installation\n",
- "\n",
- "If you haven't installed **DSP** already, let's do that.\n",
- "\n",
- "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4834f1e4-c3b8-498e-9376-4241c3d2324f",
- "metadata": {},
- "outputs": [],
- "source": [
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab \n",
- " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
- "except: pass\n",
- "\n",
- "!pip install -U pip dsp-ml"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7d71eb82-0b21-4982-9b5f-f2ac6a8e5611",
- "metadata": {},
- "source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
- "\n",
- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
- "\n",
- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
- "\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "2b542158-bc91-48e2-b201-638c9e1ea606",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Not loading Cohere because it is not installed.\n"
- ]
- }
- ],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2\n",
- "\n",
- "try: import google.colab; root_path = 'dsp'\n",
- "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
- "except: root_path = '../'\n",
- "\n",
- "import os\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
- "\n",
- "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
- "import sys\n",
- "sys.path.insert(0, '../')\n",
- "\n",
- "import dsp\n",
- "\n",
- "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
- "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
- "\n",
- "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
- "rm = dsp.ColBERTv2(url=colbert_server)\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b3289955-e877-4b3e-8b8c-5e9e5a1d2921",
- "metadata": {},
- "source": [
- "### Task Examples\n",
- "\n",
- "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
- "\n",
- "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
- "\n",
- "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "ad480ffd-8299-4b36-81d9-493a27d1b28d",
- "metadata": {},
- "outputs": [],
- "source": [
- "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2870fe3d-ffb8-4214-9279-ac040aad1942",
- "metadata": {},
- "source": [
- "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "2ebf9b6d-4dfd-4023-8ded-64dfc11348ae",
- "metadata": {},
- "outputs": [],
- "source": [
- "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
- " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
- " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
- " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
- " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
- " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
- " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
- " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
- " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
- " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
- " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
- " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
- " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d7df81ab-9c30-43ec-abad-25b5a58f0cf1",
- "metadata": {},
- "source": [
- "### Program Definition\n",
- "\n",
- "Direct Query is the simplest program for this task. We'll prompt **GPT-3.5** to answer each question based on its internal parameteric knowledge.\n",
- "\n",
- "We'll start by defining the `Template` that defines how we will communicate with the LM.\n",
- "\n",
- "Specifically, the question–answer template (`qa_template`) will include a question and a short answer for each example."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "72d65b15-443d-4e3e-b327-b19a396ba5ae",
- "metadata": {},
- "outputs": [],
- "source": [
- "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
- "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
- "\n",
- "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4610915b-fad0-4ecf-b994-b9df035e8845",
- "metadata": {},
- "source": [
- "Then, let's define the actual program, `Direct_Query_QA`. It'll accept a string (`question`) and returns another string (its short `answer`)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "926e9edb-2447-4b32-85e9-4105d5b0c39d",
- "metadata": {},
- "outputs": [],
- "source": [
- "def Direct_Query_QA(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " example = dsp.Example(question=question, demos=demos)\n",
- "\n",
- " example, completions = dsp.generate(qa_template)(example, stage='qa')\n",
- " return completions.answer"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e5ea19e3-5a7c-4984-a9b3-eab408244912",
- "metadata": {},
- "source": [
- "Let's invoke the program on a sample question."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "c3c96381-6dab-4214-8faf-1c649de5c5c8",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "E. L. Doctorow\n"
- ]
- }
- ],
- "source": [
- "print(dev[0].question)\n",
- "print(Direct_Query_QA(dev[0].question))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "424c4ffb-4e71-4a91-a179-7cacadcbc2f5",
- "metadata": {},
- "source": [
- "Let's inspect the last call to the LM to learn more about the behavior of the program."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "f8fe21c1-e185-44c3-8742-7fb19385e8a9",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- }
- ],
- "source": [
- "lm.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "886ff6e7-0ca9-4dff-a042-5357915924ac",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 121.15it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 3 / 13 (23.1%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\">\n",
- "#T_60df5 th {\n",
- " text-align: left;\n",
- "}\n",
- "#T_60df5 td {\n",
- " text-align: left;\n",
- "}\n",
- "</style>\n",
- "<table id=\"T_60df5\">\n",
- " <thead>\n",
- " <tr>\n",
- " <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th id=\"T_60df5_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
- " <th id=\"T_60df5_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
- " <th id=\"T_60df5_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th id=\"T_60df5_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
- " </tr>\n",
- " </thead>\n",
- " <tbody>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_60df5_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_60df5_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_60df5_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_60df5_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_60df5_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_60df5_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_60df5_row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer</td>\n",
- " <td id=\"T_60df5_row1_col3\" class=\"data row1 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_60df5_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_60df5_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_60df5_row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
- " <td id=\"T_60df5_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_60df5_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_60df5_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_60df5_row3_col2\" class=\"data row3 col2\" >1966</td>\n",
- " <td id=\"T_60df5_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_60df5_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_60df5_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_60df5_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_60df5_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_60df5_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_60df5_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_60df5_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_60df5_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_60df5_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_60df5_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_60df5_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_60df5_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_60df5_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_60df5_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_60df5_row7_col2\" class=\"data row7 col2\" >July 30, 1953</td>\n",
- " <td id=\"T_60df5_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_60df5_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_60df5_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_60df5_row8_col2\" class=\"data row8 col2\" >Hudson River</td>\n",
- " <td id=\"T_60df5_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_60df5_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_60df5_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_60df5_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
- " <td id=\"T_60df5_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_60df5_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_60df5_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_60df5_row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
- " <td id=\"T_60df5_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_60df5_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_60df5_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_60df5_row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
- " <td id=\"T_60df5_row11_col3\" class=\"data row11 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_60df5_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_60df5_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_60df5_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_60df5_row12_col2\" class=\"data row12 col2\" >1457</td>\n",
- " <td id=\"T_60df5_row12_col3\" class=\"data row12 col3\" >❌</td>\n",
- " </tr>\n",
- " </tbody>\n",
- "</table>\n"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7fac858f92e0>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "23.1"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from dsp.evaluation.utils import evaluate\n",
- "\n",
- "evaluate(Direct_Query_QA, dev)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python (DSP)",
- "language": "python",
- "name": "dsp"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/dsp_v1_demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb b/dsp_v1_demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb
deleted file mode 100644
index 22b8632e9..000000000
--- a/dsp_v1_demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb
+++ /dev/null
@@ -1,502 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "6a0e89c2-8e49-4b0d-9b82-7b85919c9597",
- "metadata": {},
- "source": [
- "# Multi-hop QA Program 2: Retrieve then Read\n",
- "\n",
- "This notebook is a stand-alone version of Program 2 from the intro notebook."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9943c9eb-a4ab-4ea1-85ef-f6f263abaa56",
- "metadata": {},
- "source": [
- "### Installation\n",
- "\n",
- "If you haven't installed **DSP** already, let's do that.\n",
- "\n",
- "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "45373bac-1863-4ec5-a4cc-15f046de273f",
- "metadata": {},
- "outputs": [],
- "source": [
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab \n",
- " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
- "except: pass\n",
- "\n",
- "!pip install -U pip dsp-ml"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f535fe40-34a6-41e4-ac1b-b53dd1ac3246",
- "metadata": {},
- "source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
- "\n",
- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
- "\n",
- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
- "\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "1a15edff-40a8-4237-9b5f-6be8de7cb1df",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Not loading Cohere because it is not installed.\n"
- ]
- }
- ],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2\n",
- "\n",
- "try: import google.colab; root_path = 'dsp'\n",
- "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
- "except: root_path = '../'\n",
- "\n",
- "import os\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
- "\n",
- "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
- "import sys\n",
- "sys.path.insert(0, '../')\n",
- "\n",
- "import dsp\n",
- "\n",
- "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
- "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
- "\n",
- "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
- "rm = dsp.ColBERTv2(url=colbert_server)\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ff9b842b-4cb9-4756-b375-c4204c78413b",
- "metadata": {},
- "source": [
- "### Task Examples\n",
- "\n",
- "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
- "\n",
- "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
- "\n",
- "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "e245f705-4602-485d-a322-0dba969868b0",
- "metadata": {},
- "outputs": [],
- "source": [
- "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a280ea89-476e-43d6-a485-2cb064dda4bf",
- "metadata": {},
- "source": [
- "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "23e5011c-1d5d-4355-8d9e-5ca83c4d332c",
- "metadata": {},
- "outputs": [],
- "source": [
- "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
- " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
- " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
- " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
- " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
- " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
- " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
- " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
- " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
- " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
- " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
- " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
- " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "687f0c48-5dcf-4679-a498-90498ed65f6c",
- "metadata": {},
- "source": [
- "### Program Definition\n",
- "\n",
- "Program 1 relies entirely on the factual knowledge memorized by GPT-3. As we have seen, it's sometimes correct but it's far from reliable.\n",
- "\n",
- "Let's try to address this. Our first improvement to the vanilla **GPT-3.5** strategy is to include search results in the LM prompt, often called a *retrieve-then-read* approach.\n",
- "\n",
- "In particular, we'll retrieve the most relevant passage to the question (according to the retriever) from the corpus and concatenate it into the prompt. This may help GPT-3.5 answer more factually.\n",
- "\n",
- "To do this, we'll need to modify the `Template`. In particular, we'll define a `qa_template_with_passages` template, which will include a `context` field in addition to a question and an answer."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "c07db044-fe6a-47ed-b754-2d2ba23600b2",
- "metadata": {},
- "outputs": [],
- "source": [
- "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
- "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
- "\n",
- "Context = dsp.Type(\n",
- " prefix=\"Context:\\n\",\n",
- " desc=\"${sources that may contain relevant content}\",\n",
- " format=dsp.passages2text\n",
- ")\n",
- "\n",
- "qa_template_with_passages = dsp.Template(\n",
- " instructions=\"Answer questions with short factoid answers.\",\n",
- " context=Context(), question=Question(), answer=Answer()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1a55b5d3-b69c-475f-b83a-b6e066fd8a4b",
- "metadata": {},
- "source": [
- "Now let's modify the program. We'll call `dsp.retrieve`, which we hooked earlier in this notebook to a **ColBERTv2** retriever serving a Wikipedia (Dec 2018) index."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "0fa13294-7730-448d-992a-a4a6d1ac61d5",
- "metadata": {},
- "outputs": [],
- "source": [
- "def Retrieve_then_Read_QA(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " passages = dsp.retrieve(question, k=1)\n",
- " \n",
- " example = dsp.Example(question=question, context=passages, demos=demos)\n",
- " example, completions = dsp.generate(qa_template_with_passages)(example, stage='qa')\n",
- "\n",
- " return completions.answer"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "52a6bf3b-44ca-47d6-9d47-a3ac5ea0c600",
- "metadata": {},
- "source": [
- "Let's invoke this on a sample question and let's inspect the last call to the LM to learn more about the behavior of the program."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "1fbb8316-9ada-4047-a5c0-754040b08f5d",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "«Julia Peterkin | Julia Peterkin Julia Peterkin (October 31, 1880 – August 10, 1961) was an American author from South Carolina. In 1929 she won the Pulitzer Prize for Novel/Literature, for her novel \"Scarlet Sister Mary.\" She wrote several novels about the plantation South, especially the Gullah people of the Low Country. She was one of the few white authors who wrote about the African-American experience. Julia Mood was born in Laurens County, South Carolina. Her father was a physician, and she was the third of his four children. Her mother died soon after her birth, and her father later married Janie Brogdon.»\n",
- "\n",
- "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "\n",
- "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('E. L. Doctorow', None)"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "Retrieve_then_Read_QA(dev[0].question), lm.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ad0d7128-9ce6-42c4-af33-2acd0cb2620c",
- "metadata": {},
- "source": [
- "Now, let's evaluate this on the dev examples."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "157c24a3-8daa-4ec7-9261-5e03f33fddb3",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 130.12it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 5 / 13 (38.5%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\">\n",
- "#T_dbaed th {\n",
- " text-align: left;\n",
- "}\n",
- "#T_dbaed td {\n",
- " text-align: left;\n",
- "}\n",
- "</style>\n",
- "<table id=\"T_dbaed\">\n",
- " <thead>\n",
- " <tr>\n",
- " <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th id=\"T_dbaed_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
- " <th id=\"T_dbaed_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
- " <th id=\"T_dbaed_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th id=\"T_dbaed_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
- " </tr>\n",
- " </thead>\n",
- " <tbody>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_dbaed_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_dbaed_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_dbaed_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_dbaed_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_dbaed_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_dbaed_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_dbaed_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_dbaed_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_dbaed_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_dbaed_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_dbaed_row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida.</td>\n",
- " <td id=\"T_dbaed_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_dbaed_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_dbaed_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_dbaed_row3_col2\" class=\"data row3 col2\" >1854</td>\n",
- " <td id=\"T_dbaed_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_dbaed_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_dbaed_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_dbaed_row4_col2\" class=\"data row4 col2\" >John Braine is English.</td>\n",
- " <td id=\"T_dbaed_row4_col3\" class=\"data row4 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_dbaed_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_dbaed_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_dbaed_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_dbaed_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_dbaed_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_dbaed_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_dbaed_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_dbaed_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_dbaed_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_dbaed_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_dbaed_row7_col2\" class=\"data row7 col2\" >August 5, 1962</td>\n",
- " <td id=\"T_dbaed_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_dbaed_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_dbaed_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_dbaed_row8_col2\" class=\"data row8 col2\" >Dundalk, Ireland</td>\n",
- " <td id=\"T_dbaed_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_dbaed_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_dbaed_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_dbaed_row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
- " <td id=\"T_dbaed_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_dbaed_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_dbaed_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_dbaed_row10_col2\" class=\"data row10 col2\" >Enid, Oklahoma</td>\n",
- " <td id=\"T_dbaed_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_dbaed_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_dbaed_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_dbaed_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_dbaed_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbaed_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_dbaed_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_dbaed_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_dbaed_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_dbaed_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody>\n",
- "</table>\n"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f2ddcdf3580>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "38.5"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from dsp.evaluation.utils import evaluate\n",
- "\n",
- "evaluate(Retrieve_then_Read_QA, dev)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python (DSP)",
- "language": "python",
- "name": "dsp"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/dsp_v1_demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb b/dsp_v1_demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb
deleted file mode 100644
index 6ee648e0f..000000000
--- a/dsp_v1_demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb
+++ /dev/null
@@ -1,521 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "252556ab-bf1d-4574-9763-d66fc2fd872b",
- "metadata": {},
- "source": [
- "# Multi-hop QA Program 3: Retrieve then Read w/ Self-Consistency\n",
- "\n",
- "This notebook is a stand-alone version of Program 3 from the intro notebook."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6aa1b28c-3c3d-4436-a54d-e141945bc69f",
- "metadata": {},
- "source": [
- "### Installation\n",
- "\n",
- "If you haven't installed **DSP** already, let's do that.\n",
- "\n",
- "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "7378685e-2100-4e05-8694-ef9b6ef8c0b1",
- "metadata": {},
- "outputs": [],
- "source": [
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab \n",
- " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
- "except: pass\n",
- "\n",
- "!pip install -U pip dsp-ml"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8ee794e0-ef54-4c6c-86ac-3a8f7b207440",
- "metadata": {},
- "source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
- "\n",
- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
- "\n",
- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
- "\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "26cca18f-aefc-4233-bdbf-93ae1fbf7212",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Not loading Cohere because it is not installed.\n"
- ]
- }
- ],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2\n",
- "\n",
- "try: import google.colab; root_path = 'dsp'\n",
- "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
- "except: root_path = '../'\n",
- "\n",
- "import os\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
- "\n",
- "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
- "import sys\n",
- "sys.path.insert(0, '../')\n",
- "\n",
- "import dsp\n",
- "\n",
- "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
- "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
- "\n",
- "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
- "rm = dsp.ColBERTv2(url=colbert_server)\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0e03e272-c76d-4cc4-948d-1fdb6842d3be",
- "metadata": {},
- "source": [
- "### Task Examples\n",
- "\n",
- "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
- "\n",
- "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
- "\n",
- "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "b57307c5-2bfa-4c43-81e0-d9246a65f11a",
- "metadata": {},
- "outputs": [],
- "source": [
- "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e9a6364c-12d8-4298-b087-4b84976b7f92",
- "metadata": {},
- "source": [
- "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "75b5b666-afe0-4d28-b398-1dcc3823175c",
- "metadata": {},
- "outputs": [],
- "source": [
- "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
- " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
- " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
- " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
- " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
- " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
- " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
- " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
- " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
- " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
- " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
- " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
- " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b95dc2be-2513-4238-a60a-d9b24184eef7",
- "metadata": {},
- "source": [
- "### Program Definition\n",
- "\n",
- "It's clear that retrieval has the capacity to help **GPT-3.5** answer a larger number of questions more factually. However, these questions are too complicated for a single retrieved passage to suffice.\n",
- "\n",
- "In this program, we try to make some improvements to the pipeline from **Program 2**.\n",
- "\n",
- "In particular:\n",
- "\n",
- "- We will include 5 passages (instead of a single passage) into the prompt.\n",
- "- We will ask GPT-3.5 to generate a Chain-of-Thought (CoT) rationale to more effectively extract the answer from the passages.\n",
- "- We will use Self-Consistency (SC) to marginalize this prediction across many chains of thought."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "bcdae9d5-4d60-4f95-8561-37a8073422ea",
- "metadata": {},
- "outputs": [],
- "source": [
- "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
- "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
- "\n",
- "Context = dsp.Type(\n",
- " prefix=\"Context:\\n\",\n",
- " desc=\"${sources that may contain relevant content}\",\n",
- " format=dsp.passages2text\n",
- ")\n",
- "\n",
- "Rationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step.\",\n",
- " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
- ")\n",
- "\n",
- "qa_template_with_CoT = dsp.Template(\n",
- " instructions=\"Answer questions with short factoid answers.\",\n",
- " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5595b3eb-6e8d-47fd-b5b0-6c2d3fa9422e",
- "metadata": {},
- "source": [
- "The snippet above updates our template to include a `rationale` field.\n",
- "\n",
- "Below, we define the new program `Retrieve_then_Read_QA_v2`. In it, we define a new DSP _transformation_ `QA_predict`, which we will re-use in multiple programs.\n",
- "\n",
- "The transformation `QA_predict` takes an `Example` (which has a question, context, and demonstrations) and generates an answer. It can be configured to use self-consistency (`sc=True`) or not."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "688eabdb-5878-46db-8141-ebd8c67acb4d",
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsp.transformation\n",
- "def QA_predict(example: dsp.Example, sc=True):\n",
- " if sc:\n",
- " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
- " completions = dsp.majority(completions)\n",
- " else:\n",
- " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
- " \n",
- " return example.copy(answer=completions.answer)\n",
- "\n",
- "def Retrieve_then_Read_QA_v2(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " passages = dsp.retrieve(question, k=5)\n",
- " example = dsp.Example(question=question, context=passages, demos=demos)\n",
- " \n",
- " return QA_predict(example).answer"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fc6b16c7-f7d9-4b74-9f90-45840b4f33a2",
- "metadata": {},
- "source": [
- "Let's inspect an example."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "2d5889d6-88a1-4a60-9069-5f22a0bcafd9",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "[2] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[3] «Right Back at It Again | band played \"Right Back at It Again\" on both occasions. AbsolutePunk writer Thomas Nassiff said the \"mosh call\" lyric \"\"Brace for impact\"\" had \"pretty much ruin[ed] that track\". Tamsyn Wilce for Alter the Press! noted that the way \"Right Back at it Again\" follows on from opening track \"City of Ocala\" \"continues the catchy hardcore beats\" that is present in all of the band's albums. \"Rock Sound\"'s Andy Ritchie called the song \"unmistakably the 'All I Want' of 'Common Courtesy'\". A music video for the song was released on 19 December 2013. The music video was nominated for Best Video»\n",
- "[4] «Born Again in Dixieland | Born Again in Dixieland \"Born Again in Dixieland\" is a song co-written and recorded by Canadian country music artist Jason McCoy. Released in June 1997, it was the lead single from his album \"Playin' for Keeps\". The song reached #3 on the \"RPM\" Country Tracks chart in September 1997. The song won a SOCAN Song of the Year award. McCoy said that he started writing the song with Sheridan and he was playing songs for MCA Canada label and they asked if he had any other songs. He played a verse and the chorus from the song thinking it wasn't»\n",
- "[5] «Right Back Where We Started From | 1974. In a 3 May 2008 interview with Michael Shelley of WFMU, Edwards recalled that after hearing Maxine Nightingale sing on the session for Al Matthews' \"Fool\" that track's producer Pierre Tubbs had come up with \"Right Back Where We Started From\" as a good title for a song for Nightingale herself to record and had invited Edwards to co-write the song. Utilizing a tune which Edwards had written \"a couple of years before\", Tubbs and Edwards wrote \"Right Back Where We Started From\" in about seven minutes while driving to Charing Cross Hospital where Tubbs' wife Gabrielle (née Zimmerman)»\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The song is by A Day to Remember, and the lyrics were co-written by the singer, Jeremy McKinnon. We just need to find out where McKinnon was born.\n",
- "\n",
- "Answer: Orlando, Florida\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('Ocala, Florida', None)"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "Retrieve_then_Read_QA_v2(dev[2].question), lm.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "4691922b-7725-48cb-812a-9051cdb2b3d4",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 31.30it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 7 / 13 (53.8%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\">\n",
- "#T_fe9ce th {\n",
- " text-align: left;\n",
- "}\n",
- "#T_fe9ce td {\n",
- " text-align: left;\n",
- "}\n",
- "</style>\n",
- "<table id=\"T_fe9ce\">\n",
- " <thead>\n",
- " <tr>\n",
- " <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th id=\"T_fe9ce_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
- " <th id=\"T_fe9ce_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
- " <th id=\"T_fe9ce_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th id=\"T_fe9ce_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
- " </tr>\n",
- " </thead>\n",
- " <tbody>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_fe9ce_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_fe9ce_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_fe9ce_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_fe9ce_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_fe9ce_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_fe9ce_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_fe9ce_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_fe9ce_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_fe9ce_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_fe9ce_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_fe9ce_row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
- " <td id=\"T_fe9ce_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_fe9ce_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_fe9ce_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_fe9ce_row3_col2\" class=\"data row3 col2\" >1824</td>\n",
- " <td id=\"T_fe9ce_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_fe9ce_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_fe9ce_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_fe9ce_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_fe9ce_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_fe9ce_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_fe9ce_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_fe9ce_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_fe9ce_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_fe9ce_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_fe9ce_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_fe9ce_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
- " <td id=\"T_fe9ce_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_fe9ce_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_fe9ce_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_fe9ce_row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
- " <td id=\"T_fe9ce_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_fe9ce_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_fe9ce_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_fe9ce_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_fe9ce_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_fe9ce_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_fe9ce_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_fe9ce_row9_col2\" class=\"data row9 col2\" >Aeronautics and Astronautics</td>\n",
- " <td id=\"T_fe9ce_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_fe9ce_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_fe9ce_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_fe9ce_row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
- " <td id=\"T_fe9ce_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_fe9ce_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_fe9ce_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_fe9ce_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_fe9ce_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_fe9ce_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_fe9ce_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_fe9ce_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_fe9ce_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_fe9ce_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody>\n",
- "</table>\n"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f8a980bd5b0>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "53.8"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from dsp.evaluation.utils import evaluate\n",
- "\n",
- "evaluate(Retrieve_then_Read_QA_v2, dev)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python (DSP)",
- "language": "python",
- "name": "dsp"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/dsp_v1_demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb b/dsp_v1_demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb
deleted file mode 100644
index 95229e448..000000000
--- a/dsp_v1_demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb
+++ /dev/null
@@ -1,634 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "e1877b7d-f0b3-4430-a81a-69954d9fc40b",
- "metadata": {},
- "source": [
- "# Multi-hop QA Program 4: Multi-hop Retrieval\n",
- "\n",
- "This notebook is a stand-alone version of Program 4 from the intro notebook."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "60078f63-0739-4217-8cf0-607e2238b9e9",
- "metadata": {},
- "source": [
- "### Installation\n",
- "\n",
- "If you haven't installed **DSP** already, let's do that.\n",
- "\n",
- "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "bc0733b3-d712-4539-b38a-ed8204b2e117",
- "metadata": {},
- "outputs": [],
- "source": [
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab \n",
- " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
- "except: pass\n",
- "\n",
- "!pip install -U pip dsp-ml"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b6215c64-6afa-4f78-8026-5ce373863a8a",
- "metadata": {},
- "source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
- "\n",
- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
- "\n",
- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
- "\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "083dc4db-e9bf-4e00-8b20-555931c44bb0",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Not loading Cohere because it is not installed.\n"
- ]
- }
- ],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2\n",
- "\n",
- "try: import google.colab; root_path = 'dsp'\n",
- "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
- "except: root_path = '../'\n",
- "\n",
- "import os\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
- "\n",
- "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
- "import sys\n",
- "sys.path.insert(0, '../')\n",
- "\n",
- "import dsp\n",
- "\n",
- "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
- "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
- "\n",
- "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
- "rm = dsp.ColBERTv2(url=colbert_server)\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8fbd3484-6447-4086-9bd5-4897a45c7c96",
- "metadata": {},
- "source": [
- "### Task Examples\n",
- "\n",
- "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
- "\n",
- "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
- "\n",
- "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "a8a383c8-5f7f-44c1-bb73-1f7218c2ccdf",
- "metadata": {},
- "outputs": [],
- "source": [
- "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e69681a4-6fe0-4d84-a6ab-f7c2bc61ddce",
- "metadata": {},
- "source": [
- "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "dc4c710b-c850-41ad-973e-661d352db565",
- "metadata": {},
- "outputs": [],
- "source": [
- "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
- " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
- " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
- " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
- " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
- " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
- " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
- " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
- " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
- " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
- " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
- " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
- " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fbd4f6a5-fff6-4832-8fdf-e0f9bd505a65",
- "metadata": {},
- "source": [
- "### Program Definition\n",
- "\n",
- "From Programs 1 to 3, it becomes clear that a single search query is not enough for this task. This can be seen when trying to find the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it doesn't mention when he was born.\n",
- "\n",
- "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019), IRRR (Qi et al., 2021) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "d8647458-84f6-4fde-84c0-ff8181d935f2",
- "metadata": {},
- "outputs": [],
- "source": [
- "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
- "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
- "\n",
- "Context = dsp.Type(\n",
- " prefix=\"Context:\\n\",\n",
- " desc=\"${sources that may contain relevant content}\",\n",
- " format=dsp.passages2text\n",
- ")\n",
- "\n",
- "Rationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step.\",\n",
- " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
- ")\n",
- "\n",
- "qa_template_with_CoT = dsp.Template(\n",
- " instructions=\"Answer questions with short factoid answers.\",\n",
- " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6bd8492c-df35-4d18-a516-64083933a081",
- "metadata": {},
- "source": [
- "With **DSP**, we can easily simulate such systems in a few lines of code. Let's begin by defining two new templates.\n",
- "\n",
- "First, in `rewrite_template`, we will re-write the input question into a simple search query. This will serve as our \"first hop\" query."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "0b84b21f-459a-4df7-8c3e-0fa392a0e46d",
- "metadata": {},
- "outputs": [],
- "source": [
- "SearchRationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",
- " desc=\"${the missing information}\"\n",
- ")\n",
- "\n",
- "SearchQuery = dsp.Type(\n",
- " prefix=\"Search Query:\",\n",
- " desc=\"${a simple question for seeking the missing information}\"\n",
- ")\n",
- "\n",
- "rewrite_template = dsp.Template(\n",
- " instructions=\"Write a search query that will help answer a complex question.\",\n",
- " question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "ec31457c-a3cb-46b1-b08f-545023856737",
- "metadata": {},
- "outputs": [],
- "source": [
- "CondenseRationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\",\n",
- " desc=\"${information from the context that provides useful clues}\"\n",
- ")\n",
- "\n",
- "hop_template = dsp.Template(\n",
- " instructions=rewrite_template.instructions,\n",
- " context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3d18d0c5-1994-4c63-8fc6-17570b5fde9a",
- "metadata": {},
- "source": [
- "Let's build our `multihop_QA_v1` program. The hallmark of this program is the search transformation called `multihop_search_v1`.\n",
- "\n",
- "This will largely simulate the IRRR system. In each hop, we will retrieve `k` (e.g., `k=2`) passages and concatenate them to the context retrieved from earlier hops to form a `context` chain.\n",
- "\n",
- "For simplicity, we will repeat this for a total of `max_hops` hops (e.g., `max_hops=2`). It's easy to add a stopping criteria in which the LM is asked whether more hops are required."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "e53dae43-7f15-4160-82aa-a09fc262ed0c",
- "metadata": {},
- "outputs": [],
- "source": [
- "from dsp.utils import deduplicate\n",
- "\n",
- "@dsp.transformation\n",
- "def QA_predict(example: dsp.Example, sc=True):\n",
- " if sc:\n",
- " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
- " completions = dsp.majority(completions)\n",
- " else:\n",
- " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
- " \n",
- " return example.copy(answer=completions.answer)\n",
- "\n",
- "def Retrieve_then_Read_QA_v2(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " passages = dsp.retrieve(question, k=5)\n",
- " example = dsp.Example(question=question, context=passages, demos=demos)\n",
- " \n",
- " return QA_predict(example).answer\n",
- "\n",
- "@dsp.transformation\n",
- "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",
- " example.context = []\n",
- " \n",
- " for hop in range(max_hops):\n",
- " # Generate a query based\n",
- " template = rewrite_template if hop == 0 else hop_template\n",
- " example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
- "\n",
- " # Retrieve k results based on the query generated\n",
- " passages = dsp.retrieve(completions.query, k=k)\n",
- "\n",
- " # Update the context by concatenating old and new passages\n",
- " example.context = deduplicate(example.context + passages)\n",
- "\n",
- " return example\n",
- "\n",
- "\n",
- "def multihop_QA_v1(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " x = dsp.Example(question=question, demos=demos)\n",
- " \n",
- " x = multihop_search_v1(x)\n",
- " x = QA_predict(x, sc=False)\n",
- "\n",
- " return x.answer"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "f47ff2cd-3502-4df3-81d7-310eb0db5ac2",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the singer\n",
- "Search Query: Who sings Right Back At It Again\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
- "\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The song is called \"Right Back at It Again\" and it is by the band A Day to Remember. The singer, Jeremy McKinnon, wrote the lyrics.\n",
- "\n",
- "Search Query: Where was Jeremy McKinnon born?\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "[3] «Jeremy McKinnon | Jeremy McKinnon Jeremy Wade McKinnon (born December 17, 1985) is an American musician and record producer, best known as a founding member and the lead vocalist of A Day to Remember. He has produced full-length albums for The Devil Wears Prada, The Ghost Inside, Neck Deep, and Wage War. Jeremy McKinnon was born in Gainesville, Florida and grew up with two sisters. His parents are originally from Brooklyn, New York and is of Irish and Italian descent. McKinnon first got a job at Boston Market and later went on to do construction work. McKinnon's interest in music came as a»\n",
- "[4] «Martin McKinnon | Martin McKinnon Martin \"Marty\" McKinnon (born 5 July 1975 in Adelaide) is a former Australian rules footballer who played with Adelaide, Geelong and the Brisbane Lions in the Australian Football League (AFL). McKinnon was recruited by Adelaide in the 1992 AFL Draft, with their first ever national draft pick. He was the youngest player on Adelaide's list at the time and played for Central District in the SANFL when not appearing with Adelaide. He made his AFL debut late in the 1994 AFL season and was a regular in the team the following year, playing mostly as a wingman. He»\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The song \"Right Back at It Again\" is co-written by the singer Jeremy McKinnon. A quick search of \"Jeremy McKinnon\" reveals that he was born in Gainesville, Florida.\n",
- "\n",
- "Answer: Gainesville, Florida\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('Gainesville, Florida', None)"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "multihop_QA_v1(dev[2].question), lm.inspect_history(n=3)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "51e7739c-7c5e-4417-a66e-030f615c4858",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 48.52it/s]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 7 / 13 (53.8%) correctly.\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\">\n",
- "#T_dbce4 th {\n",
- " text-align: left;\n",
- "}\n",
- "#T_dbce4 td {\n",
- " text-align: left;\n",
- "}\n",
- "</style>\n",
- "<table id=\"T_dbce4\">\n",
- " <thead>\n",
- " <tr>\n",
- " <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th id=\"T_dbce4_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
- " <th id=\"T_dbce4_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
- " <th id=\"T_dbce4_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th id=\"T_dbce4_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
- " </tr>\n",
- " </thead>\n",
- " <tbody>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_dbce4_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_dbce4_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_dbce4_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_dbce4_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_dbce4_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_dbce4_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_dbce4_row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer: A Mother's Hunt for Justice</td>\n",
- " <td id=\"T_dbce4_row1_col3\" class=\"data row1 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_dbce4_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_dbce4_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_dbce4_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
- " <td id=\"T_dbce4_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_dbce4_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_dbce4_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_dbce4_row3_col2\" class=\"data row3 col2\" >1947</td>\n",
- " <td id=\"T_dbce4_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_dbce4_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_dbce4_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_dbce4_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_dbce4_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_dbce4_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_dbce4_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_dbce4_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_dbce4_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_dbce4_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_dbce4_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_dbce4_row6_col2\" class=\"data row6 col2\" >UK</td>\n",
- " <td id=\"T_dbce4_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_dbce4_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_dbce4_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_dbce4_row7_col2\" class=\"data row7 col2\" >September 28, 1969</td>\n",
- " <td id=\"T_dbce4_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_dbce4_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_dbce4_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_dbce4_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_dbce4_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_dbce4_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_dbce4_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_dbce4_row9_col2\" class=\"data row9 col2\" >space technology</td>\n",
- " <td id=\"T_dbce4_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_dbce4_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_dbce4_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_dbce4_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
- " <td id=\"T_dbce4_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_dbce4_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_dbce4_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_dbce4_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_dbce4_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_dbce4_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_dbce4_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_dbce4_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_dbce4_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_dbce4_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody>\n",
- "</table>\n"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f51f31e6970>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "53.8"
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from dsp.evaluation.utils import evaluate\n",
- "\n",
- "evaluate(multihop_QA_v1, dev)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python (DSP)",
- "language": "python",
- "name": "dsp"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/dsp_v1_demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb b/dsp_v1_demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb
deleted file mode 100644
index b5fde8052..000000000
--- a/dsp_v1_demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb
+++ /dev/null
@@ -1,755 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "cfc088a0-2448-46a3-ba0b-3024bddd0a4b",
- "metadata": {},
- "source": [
- "# Multi-hop QA Program 5: Multi-hop Condensed Retrieval w/ Automatic Demos and Query Fusion\n",
- "\n",
- "This notebook is a stand-alone version of Program 5 from the intro notebook."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0adb6404-7357-43f9-88dc-fdc1d72ad0c1",
- "metadata": {},
- "source": [
- "### Installation\n",
- "\n",
- "If you haven't installed **DSP** already, let's do that.\n",
- "\n",
- "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "0dab61b4-dad5-4d80-bd41-2ec229f7bf83",
- "metadata": {},
- "outputs": [],
- "source": [
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab \n",
- " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
- "except: pass\n",
- "\n",
- "!pip install -U pip dsp-ml"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "14abe1f4-2f0c-4b9d-8ef3-a1800350fb3f",
- "metadata": {},
- "source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
- "\n",
- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
- "\n",
- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
- "\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "aa84afba-2e23-4091-9103-e4f5ee077db2",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Not loading Cohere because it is not installed.\n"
- ]
- }
- ],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2\n",
- "\n",
- "try: import google.colab; root_path = 'dsp'\n",
- "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
- "except: root_path = '../'\n",
- "\n",
- "import os\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
- "\n",
- "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
- "import sys\n",
- "sys.path.insert(0, '../')\n",
- "\n",
- "import dsp\n",
- "\n",
- "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
- "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
- "\n",
- "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
- "rm = dsp.ColBERTv2(url=colbert_server)\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "238c8ab9-b950-4b9d-b20b-9b4ab4ef029f",
- "metadata": {},
- "source": [
- "### Task Examples\n",
- "\n",
- "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
- "\n",
- "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
- "\n",
- "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "178c1031-b350-4189-a209-4e916383a791",
- "metadata": {},
- "outputs": [],
- "source": [
- "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "170501b3-b7e9-46a5-ba8c-e3b0b586f8ab",
- "metadata": {},
- "source": [
- "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "ed645900-9e75-44f3-8469-0d9eb127d987",
- "metadata": {},
- "outputs": [],
- "source": [
- "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
- " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
- " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
- " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
- " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
- " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
- " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
- " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
- " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
- " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
- " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
- " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
- " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2ac85dda-9526-4e73-b4d8-932c25fab9fb",
- "metadata": {},
- "source": [
- "### Program Definition\n",
- "\n",
- "Through Program 4, we've begun to explore some of the power of the DSP abstraction. However, if you look closely, you will see a few downsides of the previous approach:\n",
- "\n",
- "1. The search transformations invoke the LM without any demonstrations in the prompt. That is because we only have training data for the question–answer pairs and not for the intermediate labels (e.g., search queries).\n",
- "2. The QA prompt uses passages (`context`) and a Chain-of-Thought (`rationale`) for the question to be answered. However, the training demonstrations include neither context nor CoT because they aren't available in our labels.\n",
- "3. The search transformations commit to a single query per hop, which may single out an unproductive chain of passages and hence fail to uncover relevant information."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "1fb2dd8e-b45f-4a62-8351-8274eb71f269",
- "metadata": {},
- "outputs": [],
- "source": [
- "from dsp.utils import deduplicate\n",
- "\n",
- "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
- "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
- "\n",
- "Context = dsp.Type(\n",
- " prefix=\"Context:\\n\",\n",
- " desc=\"${sources that may contain relevant content}\",\n",
- " format=dsp.passages2text\n",
- ")\n",
- "\n",
- "Rationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step.\",\n",
- " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
- ")\n",
- "\n",
- "qa_template_with_CoT = dsp.Template(\n",
- " instructions=\"Answer questions with short factoid answers.\",\n",
- " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
- ")\n",
- "\n",
- "SearchRationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",
- " desc=\"${the missing information}\"\n",
- ")\n",
- "\n",
- "SearchQuery = dsp.Type(\n",
- " prefix=\"Search Query:\",\n",
- " desc=\"${a simple question for seeking the missing information}\"\n",
- ")\n",
- "\n",
- "CondenseRationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\",\n",
- " desc=\"${information from the context that provides useful clues}\"\n",
- ")\n",
- "\n",
- "rewrite_template = dsp.Template(\n",
- " instructions=\"Write a search query that will help answer a complex question.\",\n",
- " question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",
- ")\n",
- "\n",
- "hop_template = dsp.Template(\n",
- " instructions=rewrite_template.instructions,\n",
- " context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",
- ")\n",
- "\n",
- "@dsp.transformation\n",
- "def QA_predict(example: dsp.Example, sc=True):\n",
- " if sc:\n",
- " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
- " completions = dsp.majority(completions)\n",
- " else:\n",
- " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
- " \n",
- " return example.copy(answer=completions.answer)\n",
- "\n",
- "@dsp.transformation\n",
- "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",
- " example.context = []\n",
- " \n",
- " for hop in range(max_hops):\n",
- " # Generate a query based\n",
- " template = rewrite_template if hop == 0 else hop_template\n",
- " example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
- "\n",
- " # Retrieve k results based on the query generated\n",
- " passages = dsp.retrieve(completions.query, k=k)\n",
- "\n",
- " # Update the context by concatenating old and new passages\n",
- " example.context = deduplicate(example.context + passages)\n",
- "\n",
- " return example"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "6275c005-4188-4fe1-8e08-cfd0d4870d5f",
- "metadata": {},
- "source": [
- "We address these problems automatically in the program below.\n",
- "\n",
- "In it, we begin by defining `multihop_demonstrate` (which uses `multihop_attempt`) to automatically **annotate** demonstrations for the complex multi-hop pipeline. These demonstrations will be provided to the LM when it's invoked for each transformation."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "e2901fb7-06fb-41d5-a158-bb86b99be5b6",
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsp.transformation\n",
- "def multihop_attempt(d: dsp.Example) -> dsp.Example:\n",
- " # Prepare unaugmented demonstrations for the example.\n",
- " x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))\n",
- " \n",
- " # Search. And skip examples where search fails.\n",
- " # Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.\n",
- " x = multihop_search_v1(x)\n",
- " if not dsp.passage_match(x.context, d.answer): return None\n",
- " \n",
- " # Predict. And skip examples where predict fails.\n",
- " x = QA_predict(x, sc=False)\n",
- " if not dsp.answer_match(x.answer, d.answer): return None\n",
- " \n",
- " return d.copy(**x)\n",
- "\n",
- "@dsp.transformation\n",
- "def multihop_demonstrate(x: dsp.Example) -> dsp.Example:\n",
- " demos = dsp.sample(train, k=7)\n",
- " x.demos = dsp.annotate(multihop_attempt)(demos, k=3, return_all=True)\n",
- " return x"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d2b5d5ef-88f8-40de-8a55-0303ebe6699d",
- "metadata": {},
- "source": [
- "We now implement `multihop_search_v2` as part of `multihop_QA_v2`.\n",
- "\n",
- "In addition to the changes mentioned earlier, this program simulates the Baleen system (Khattab et al., 2021) in a few lines of code.\n",
- "\n",
- "In each retrieval hop (after the very first hop), the summary of the previous hop(s) is included in the prompt. This allows us to efficiently read a larger number of passages from the current hop."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "52efb3e1-2544-4a16-acbb-3be181bfc559",
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsp.transformation\n",
- "def multihop_search_v2(example: dsp.Example, max_hops=2, k=5) -> dsp.Example:\n",
- " example.context = []\n",
- "\n",
- " for hop in range(max_hops):\n",
- " # Generate queries\n",
- " template = rewrite_template if hop == 0 else hop_template\n",
- " example, completions = dsp.generate(template, n=10, temperature=0.7)(example, stage=f'h{hop}')\n",
- " \n",
- " # Collect the queries and search with result fusion\n",
- " queries = [c.query for c in completions] + [example.question]\n",
- " example.context = dsp.retrieveEnsemble(queries, k=k)\n",
- "\n",
- " # Arrange the passages for the next hop\n",
- " if hop > 0:\n",
- " example.context = [completions[0].rationale] + example.context\n",
- " \n",
- " return example\n",
- "\n",
- "def multihop_QA_v2(question: str) -> str:\n",
- " x = dsp.Example(question=question)\n",
- " x = multihop_demonstrate(x)\n",
- " x = multihop_search_v2(x)\n",
- " x = QA_predict(x)\n",
- " return x.answer"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "4b8603e4-335c-4334-aa44-8b6ce5490326",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
- "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
- "Search Query: \"Heir to the Du Pont family fortune\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
- "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the party of the winner of the 1971 San Francisco mayoral election.\n",
- "Search Query: \"Party of the winner of the 1971 San Francisco mayoral election\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
- "\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
- "\n",
- "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
- "\n",
- "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
- "\n",
- "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «1971 San Francisco mayoral election | 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another as advocates for law and order. Alioto challenged Dobbs to commit to keeping Police Chief Donald Scott and accused Feinstein of wanting to create a civilian review board to investigate police conduct.»\n",
- "[2] «1971 San Francisco mayoral election | the November 2 election was the re-election of Mayor Joseph Alioto by a 3–2 margin over his nearest challenger. Alioto quoted Jackie Gleason—\"How sweet it is!\"—at a victory celebration and referred to his federal indictment as an attempt by the \"national administration\" to \"knock out a mayor\". Among other races on the ballot was Richard Hongisto's surprise victory for sheriff over incumbent Matthew Carberry. On the ballot was a referendum limiting new-building heights to six stories, which was defeated. A referendum to make the school board an elective one, it being the only appointive board in the state, passed. The»\n",
- "[3] «1971 San Francisco mayoral election | school-board measure was regarded as a reaction to busing initiated by the school board under court order. Another measure approved was to keep cable car service at no less than current levels. 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another»\n",
- "[4] «1971 San Francisco mayoral election | achieve ethnic balance in elementary schools. Former newspaper editor Scott Newhall proposed staging a world ecology fair and replacing California's water plan by importing icebergs, and imputed little difference between the top three contenders, whom he lumped together as \"Joe Dobbstein\". Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official. As of November, Dobbs, the only Republican in the race, had passed Feinstein in polls and was gaining on Alioto. The result of»\n",
- "[5] «2003 San Francisco mayoral election | against leftist/progressive Gonzalez, a member of the Green Party. It was the first mayoral election in San Francisco that a Green Party candidate took a noticeable amount of the vote. The election was close, with Gonzalez leading in the polls and winning the popular vote among ballots cast on election day, while Newsom had a larger lead on absentee ballots. 2003 San Francisco mayoral election The 2003 San Francisco mayoral election occurred on November 4, 2003. The incumbent, Willie Brown, was termed out of office and could not seek a third term. The general election included three top candidates including»\n",
- "\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.\n",
- "\n",
- "Search Query: What year was the Democratic Party founded?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
- "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
- "\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
- "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "\n",
- "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
- "\n",
- "Answer: Foxcatcher team\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
- "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
- "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "\n",
- "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
- "\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.»\n",
- "[2] «Democratic Party (UK, 1998) | Democratic Party (UK, 1998) The Democratic Party was a political party in the United Kingdom, founded in November 1998 by Malvern businessman Geoff Southall, who also funded the party. It was described as \"right wing\" or \"hard right\" in news reports, but claimed to occupy the centre-ground of British politics. It aimed to reduce Britain's involvement with the European Union, opposed the adoption of the Euro, called for direct democracy, and argued for limits on immigration. The slogan was \"The will of the people NOT the party\". In 1999 it had a few hundred members, including previous supporters of James»\n",
- "[3] «Democratic Party (Brazil, 1925-1934) | Democratic Party (Brazil, 1925-1934) The Democratic Party was a Brazilian political party, but state character, founded by dissidents of the Paulista Republican Party (PRP) on February 24, 1925, during the Old Republic and was its manifesto signed by 599 signatories. It was a representative organization of the traditional middle class, linked to coffee sectors, but especially the urban. In May 1926, the Youth Party (Youth Wing of dissenters, who had organized this caption for the federal elections of 1925), also dissident perrepismo, joined the Democratic Party. His most important names were Waldemar Ferreira, José Adriano Marrey Junior, Francisco Morato, Paulo»\n",
- "[4] «Democratic Party (Turkey, 1970) | Democratic Party (Turkey, 1970) Democratic Party (Turkish: \"Demokratik Parti\", DP for short) was a former political party in Turkey founded on 18 December 1970 . It should not be confused with the historical Democrat Party or the current Democrat Party in Turkey. The historical Democrat Party which was the ruling party between 1950-1960 was closed by the military rule. In 1961 and 1965 elections most of ex Democratic Party adherents supported Justice Party, a party founded in 1961. After 1965 elections Justice Party came to power. But following 1969 elections in which Justice Party was again successful, tension between the»\n",
- "[5] «Democratic Party (Japan, 1947) | Democratic Party (Japan, 1947) Democratic Party (民主党, Minshutō) was a centrist political party in Japan. It was founded in spring 1947 by merging the Progressive Party (Shinpo-tō) of Inukai Takeru with a faction of Liberal Party led by Hitoshi Ashida and obtained 124 seats in 1947 elections. The party had held seven seats in Tetsu Katayama's government in 1947-1948. For some months in 1948, party's leader Ashida was Prime minister. In March 1948, part of DP members led by Kijūrō Shidehara joined the Liberal Party to form the Democratic Liberal Party. In 1949 elections, the DP got 69 seats. The»\n",
- "[6] «Democratic Party (United States) | Democratic Party (United States) The Democratic Party is one of the two major contemporary political parties in the United States, along with the Republican Party. Tracing its heritage back to Thomas Jefferson and James Madison's Democratic-Republican Party, the modern-day Democratic Party was founded around 1828 by supporters of Andrew Jackson, making it the world's oldest active political party. The Democrats' dominant worldview was once social conservatism and economic liberalism while populism was its leading characteristic in the rural South. In 1912, Theodore Roosevelt ran as a third-party candidate in the Progressive (\"Bull Moose\") Party, beginning a switch of political platforms»\n",
- "\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The question is asking for the year the party of the mayor of San Francisco in 1971 was founded. In source 1, we see that the election was in 1971 and the winner was Joseph Alioto. In source 2, we see that the party was founded in 1998, which is not correct. In source 3, we see that the party was founded in 1925, which is not correct. In source 4, we see that the party was founded in 1970, which is not correct. In source 5, we see that the party was founded in 1947, which is not correct. In source 6, we see that the party was founded around 1828. This must be the correct answer, so we can check source 1 to confirm. In source 1\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('1828', None)"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "multihop_QA_v2(dev[3].question), lm.inspect_history(n=3)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "7c945f07-27b5-4f1c-a347-9c935a83fd28",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:02<00:00, 6.48it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 11 / 13 (84.6%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\">\n",
- "#T_6a173 th {\n",
- " text-align: left;\n",
- "}\n",
- "#T_6a173 td {\n",
- " text-align: left;\n",
- "}\n",
- "</style>\n",
- "<table id=\"T_6a173\">\n",
- " <thead>\n",
- " <tr>\n",
- " <th class=\"blank level0\" >&nbsp;</th>\n",
- " <th id=\"T_6a173_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
- " <th id=\"T_6a173_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
- " <th id=\"T_6a173_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
- " <th id=\"T_6a173_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
- " </tr>\n",
- " </thead>\n",
- " <tbody>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_6a173_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_6a173_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_6a173_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_6a173_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_6a173_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_6a173_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_6a173_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_6a173_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_6a173_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_6a173_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_6a173_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
- " <td id=\"T_6a173_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_6a173_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_6a173_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_6a173_row3_col2\" class=\"data row3 col2\" >1828</td>\n",
- " <td id=\"T_6a173_row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_6a173_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_6a173_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_6a173_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_6a173_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_6a173_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_6a173_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_6a173_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_6a173_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_6a173_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_6a173_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_6a173_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
- " <td id=\"T_6a173_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_6a173_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_6a173_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_6a173_row7_col2\" class=\"data row7 col2\" >February 13, 1980</td>\n",
- " <td id=\"T_6a173_row7_col3\" class=\"data row7 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_6a173_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_6a173_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_6a173_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_6a173_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_6a173_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_6a173_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_6a173_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
- " <td id=\"T_6a173_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_6a173_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_6a173_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_6a173_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
- " <td id=\"T_6a173_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_6a173_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_6a173_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_6a173_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_6a173_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_6a173_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_6a173_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_6a173_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_6a173_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_6a173_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody>\n",
- "</table>\n"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f8adaac2cd0>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "84.6"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from dsp.evaluation.utils import evaluate\n",
- "\n",
- "evaluate(multihop_QA_v2, dev)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python (DSP)",
- "language": "python",
- "name": "dsp"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/dsp_v1_demos/compiler.ipynb b/dsp_v1_demos/compiler.ipynb
deleted file mode 100644
index 3ef43dd17..000000000
--- a/dsp_v1_demos/compiler.ipynb
+++ /dev/null
@@ -1,1558 +0,0 @@
-{
- "cells": [
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 🎓 𝗗𝗦𝗣 Compiler\n",
- "\n",
- "If this is your first time working with the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework, we recommend starting with the [introduction notebook](https://github.com/stanfordnlp/dsp/blob/main/intro.ipynb).\n",
- "\n",
- "Foundation models are expected to serve as the core infrastructure for essentially all AI services for the coming decade or more. However, the best of these models are extraordinarily expensive to use, and as **DSP** shows, securing high quality on difficult tasks requires using these already-expensive models in elaborate pipelines. Whereas the **DSP** abstraction makes it easy to build such pipelines, DSP program compilation aims to dramatically reduce costs while maintaining the quality of these programs.\n",
- "\n",
- "This notebook introduces **v0.1 of the compiler** for **DSP** programs. The compiler takes a DSP program (alongside a number of _unlabeled_ example inputs) and returns a (much) more computationally-efficient version of the program.\n",
- "\n",
- "The DSP compiler has the following interface.\n",
- "\n",
- "```\n",
- "cheap_dsp_program = dsp.compile(original_dsp_program, unlabeled_input_examples)\n",
- "```"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We illustrate the usage of the compiler on the multi-hop question answering (QA) task. To do so, we build on a variant of the fifth program introduced in the intro notebook. For a more accessible summary of what the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework is, refer to that notebook instead.\n",
- "\n",
- "Using the compiler, we'll obtain a (much) cheaper version of the Search stage of the program. In particular, the compiled program will only need to use the smallest version of GPT-3 (i.e., `ada` at 350M parameter) instead of the largest version of GPT3.5 (i.e., `text-davinci-*`, which is typically estimated to have 175B parameters) to conduct multi-hop search. Our compiled program will cost much less to conduct search, and yet it will roughly preserve the quality of the expensive variant."
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Installation\n",
- "\n",
- "If you haven't installed **DSP** already, let's do that."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab \n",
- " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
- "except: pass\n",
- "\n",
- "!pip install -U pip dsp-ml"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM). Refer to the intro notebook for more details.\n",
- "\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2\n",
- "\n",
- "try: import google.colab; root_path = 'dsp'\n",
- "except: root_path = '.'\n",
- "\n",
- "import os\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
- "\n",
- "import dsp\n",
- "import ujson\n",
- "\n",
- "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
- "davinci = dsp.GPT3(model='text-davinci-002')\n",
- "colbert = dsp.ColBERTv2(url=colbert_server)\n",
- "\n",
- "dsp.settings.configure(lm=davinci, rm=colbert)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Task Examples\n",
- "\n",
- "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
- "\n",
- "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "train = [\n",
- "('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- "('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- "('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- "('In what year was the star of To Hell and Back born?', ['1925']),\n",
- "('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- "('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),\n",
- "]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark! That said, it'll be instructive to use it for illustration."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "dev = [\n",
- "('How many seasons did the man who runs KnowledgeWare play in the NFL ?', ['18', '18 seasons']),\n",
- "('Who co-wrote a memoir with the retired airline captain celebrated for the water landing of Flight 1549?', ['Jeffrey Zaslow']),\n",
- "('Who was born earlier, Habash al-Marwazi or Mostafa El-Sayed?', [\"Habash al-Marwazi\"]),\n",
- "('What is the oldest section from where Egyptologist Robert Hay was born?', ['Norman Keep','Pele Tower', 'Norman Keep or Pele Tower']),\n",
- "('Which American physician was a co-founder alongside Charles Mayo?', ['Augustus Stinchfield', 'Augustus W. Stinchfield']),\n",
- "('Simon Lindberg has been the leader of the group that was disbanded for inactivity in what nation in 2016?', ['Denmark']),\n",
- "('Which actor from Trainspotting also played Obi-Wan Kenobi?', ['Ewan McGregor']),\n",
- "('What is the publisher of the female superhero created by J. H. Williams III?', ['DC Comics']),\n",
- "('In what year did the team that built the Lotus E21 begin to compete under the Lotus name?', ['2012']),\n",
- "('What ranking did the man who Jonathan Brookins defeated on December 4, 2010 have in The Ultimate Fighter?', ['runner-up']),\n",
- "('Which of the lower keys is served by the bridge that crosses over the Pigeon Key Historic District?', ['Little Duck Key']),\n",
- "('The city that hosted the 1996 Summer Paralympics is the seat of what county?', ['Fulton County']),\n",
- "('What actor who was in Sixteen candles is also in the romantic comedy Results?', ['Anthony Michael Hall']),\n",
- "('Did Scott Draper or Kevin Curren play more sports?', ['Scott Draper']),\n",
- "('What role did Denis Lawson play in the 1983 British comedy-drama film written and directed by Bill Forsyth?', ['Gordon Urquhart']),\n",
- "('Which band is from a country closer to Canada, Local H or Bodyjar?', ['Local H']),\n",
- "('What animated show on Cartoon Network did Wil Wheaton have a role in?', ['Ben 10', 'Teen Titans']),\n",
- "]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next, let's load 800 questions (without answers). These will be used by the compiler."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "['Which year was the 49th NHL Entry Draft where Emil Molin selected by the Dallas Stars in the 4th round?',\n",
- " 'When did the man overthrown in the Libyan Crisis become the leader of Libya?',\n",
- " 'Are Cynara and Piptanthus both flowering plants?']"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "unlabeled_questions_path = os.path.join(root_path, 'data', 'multihop_800_unlabeled_questions.json')\n",
- "\n",
- "with open(unlabeled_questions_path) as f:\n",
- " multihop_800_unlabeled_questions = ujson.loads(f.readline())\n",
- "\n",
- "multihop_800_unlabeled_questions = [example['question'] for example in multihop_800_unlabeled_questions]\n",
- "\n",
- "multihop_800_unlabeled_questions[:3]"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Predict Transformation\n",
- "\n",
- "Let's define the question answering (QA) transformation used in our programs. This will represent the **Predict** stage of **DSP**. Refer to the intro notebook for more details on these."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
- "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
- "Context = dsp.Type(prefix=\"Context:\\n\", desc=\"${sources that may contain relevant content}\", format=dsp.passages2text)\n",
- "Rationale = dsp.Type(prefix=\"Rationale: Let's think step by step.\", desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\")\n",
- "\n",
- "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())\n",
- "qa_template_with_CoT = dsp.Template(instructions=qa_template.instructions, context=Context(), question=Question(), rationale=Rationale(), answer=Answer())"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsp.transformation\n",
- "def QA_predict(example: dsp.Example, n=20):\n",
- " if n == 0:\n",
- " return example.copy(answer='')\n",
- "\n",
- " if n > 1:\n",
- " example, completions = dsp.generate(qa_template_with_CoT, n=n, temperature=0.7)(example, stage='qa')\n",
- " completions = dsp.majority(completions)\n",
- " else:\n",
- " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
- " \n",
- " return example.copy(answer=completions.answer)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Search Transformation\n",
- "\n",
- "Similarly, we define the **Search** stage for our program below. As you can see below, we use the decorator `@dsp.compiled` for the `multihop_search` transformation. This tells the compiler that we'd like it to optimize this function for us."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [],
- "source": [
- "SearchQuery = dsp.Type(prefix=\"Search Query:\", desc=\"${a simple question for seeking the missing information}\")\n",
- "SearchRationale = dsp.Type(prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\", desc=\"${the missing information}\")\n",
- "CondenseRationale = dsp.Type(prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\", desc=\"${information from the context that provides useful clues}\")\n",
- "\n",
- "rewrite_template = dsp.Template(instructions=\"Write a search query that will help answer a complex question.\", question=Question(), rationale=SearchRationale(), query=SearchQuery())\n",
- "hop_template = dsp.Template(instructions=rewrite_template.instructions, context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery())"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [],
- "source": [
- "from dsp.utils import deduplicate\n",
- "\n",
- "@dsp.compiled\n",
- "@dsp.transformation\n",
- "def multihop_search(example: dsp.Example, max_hops=2, num_queries=10, k=7) -> dsp.Example:\n",
- " example.background = []\n",
- " example.context = []\n",
- "\n",
- " for hop in range(max_hops):\n",
- " # Generate queries\n",
- " template = rewrite_template if hop == 0 else hop_template\n",
- "\n",
- " if num_queries == 1:\n",
- " example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
- " passages = dsp.retrieve(completions.query, k=k)\n",
- "\n",
- " else:\n",
- " num_queries = int(num_queries)\n",
- " example, completions = dsp.generate(template, n=num_queries, temperature=0.7)(example, stage=f'h{hop}')\n",
- " queries = [c.query for c in completions] + [example.question]\n",
- " passages = dsp.retrieveEnsemble(queries, k=k)\n",
- "\n",
- " # Arrange the passages for the next hop\n",
- " if hop == 0:\n",
- " example.context = passages\n",
- " else:\n",
- " example.context = [completions[0].rationale] + example.background + passages\n",
- "\n",
- " example.context = deduplicate(example.context)[:k]\n",
- " example.background = deduplicate(example.background + passages)[:hop+1]\n",
- " \n",
- " return example"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Demonstrate Transformation\n",
- "\n",
- "... and the **Demonstrate** stage also.\n",
- "\n",
- "In here, we annotate a single example automatically (i.e., calling `dsp.annotate` with `k=1`) which will be used in the prompts. Thus, the program will bootstrap itself with one-shot prompts, creating annotations for all intermediate transformations (e.g., search queries). This can easily be increased (e.g., `k=3`) potentially leading to a more reliable program, albeit at a higher cost."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsp.transformation\n",
- "def multihop_attempt(d: dsp.Example) -> dsp.Example:\n",
- " x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))\n",
- " x = multihop_search(x, num_queries=1, k=4)\n",
- " \n",
- " if not dsp.passage_match(x.context, d.answer): return None\n",
- " x = QA_predict(x, n=1)\n",
- "\n",
- " if not dsp.answer_match(x.answer, d.answer): return None\n",
- " return d.copy(**x)\n",
- "\n",
- "@dsp.transformation\n",
- "def multihop_demonstrate(x: dsp.Example) -> dsp.Example:\n",
- " demos = dsp.sample(train, k=16)\n",
- " x.demos = dsp.annotate(multihop_attempt)(demos, k=1, return_all=True)\n",
- " return x"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Multi-Hop QA Program\n",
- "\n",
- "We'll make the program parameteric:\n",
- "\n",
- "- `num_queries` is the number of queries it generates in each hop (defaults to one)\n",
- "- `num_preds` is the number of answers it generates before selecting the majority vote (defaults to zero, which means skip the Predict stage)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [],
- "source": [
- "def multihop_QA(question: str, num_queries=1, num_preds=0):\n",
- " x = dsp.Example(question=question)\n",
- " x = multihop_demonstrate(x)\n",
- " x = multihop_search(x, num_queries=num_queries)\n",
- " x = QA_predict(x, n=num_preds)\n",
- " return x"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "So far, this is a very typical DSP program. Let's try to run it on the first example in our development set."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "How many seasons did the man who runs KnowledgeWare play in the NFL ?\n",
- "18\n"
- ]
- }
- ],
- "source": [
- "question = dev[0].question\n",
- "answer = multihop_QA(dev[0].question, num_queries=10, num_preds=1).answer\n",
- "\n",
- "print(question)\n",
- "print(answer)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "As usual, we can inspect the inputs and outputs of the language model at each of the stages of the program. We can see the bootstrapped example in each of the prompts."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
- "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: How many seasons did the man who runs KnowledgeWare play in the NFL ?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the man who runs KnowledgeWare.\n",
- "Search Query: \"Who runs KnowledgeWare?\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
- "\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "[3] «Gary Zukav | this book enjoyable and informative.\" David Bohm, renowned quantum physicist, wrote a personal endorsement provided to the book's publisher Harper Collins: \"Recommended highly for those who want to understand the essential significance of modern physics, and for those who are concerned with its implications for possible transformation of human consciousness.\" Zukav's next book, \"The Seat of the Soul\", published in 1989, was a No. 1 New York Times Best Seller for 31 weeks and remained on the list for three years. In an interview by Jeffrey Mishlove, for the popular Public Television series \"Thinking Allowed\", Zukav summarized the concepts presented»\n",
- "[4] «Gary Zukav | of Budapest, World Business Academy, Native American Earth Ambassadors, CoCreate with the Earth Foundation, EarthSave, and the Intuition Network, Earth Day 1990, the chair of the Government and Politics Strategy Group for the Campaign for the Earth. Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979),»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book is \"The Dancing Wu Li Masters\".\n",
- "\n",
- "Search Query: What award did \"The Dancing Wu Li Masters\" receive?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «KnowledgeWare | KnowledgeWare KnowledgeWare was a software company headquartered in Atlanta, Georgia co-founded by James Martin and run by Fran Tarkenton. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench). KnowledgeWare was sold to Sterling Software in 1994, which was in its turn acquired by Computer Associates. Tarkenton is credited with having coined, \"A fool with a tool is a faster fool\" while offering classes at their Peach Tree headquarters. Tarkenton, Don Addington and other executives were eventually involved in legal actions brought by the SEC for engaging in a fraudulent scheme to inflate KnowledgeWare's financial results»\n",
- "[2] «KnowledgeWare | to meet sales and earnings projections. KnowledgeWare KnowledgeWare was a software company headquartered in Atlanta, Georgia co-founded by James Martin and run by Fran Tarkenton. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench). KnowledgeWare was sold to Sterling Software in 1994, which was in its turn acquired by Computer Associates. Tarkenton is credited with having coined, \"A fool with a tool is a faster fool\" while offering classes at their Peach Tree headquarters. Tarkenton, Don Addington and other executives were eventually involved in legal actions brought by the SEC for engaging in a fraudulent»\n",
- "[3] «David Neft | new look at professional football before 1933. Because the National Football League hadn't kept official statistics in its first thirteen seasons, knowledge of the period was extremely limited. Neft led a team of researchers that meticulously reconstructed the statistical record using box scores, play-by-play accounts, and game stories from local newspapers. The result was a remarkable new look at the teams and players from the NFL's earliest seasons. Neft returned to Harris, then owned by Gannett, in 1977, serving as executive vice president. In 1985, he became Gannett's director of research, a position he held until his retirement in 2002.»\n",
- "[4] «Ken Iman | Ken Iman Kenneth Charles Iman (February 8, 1939 – November 13, 2010) was an American football center who played 15 seasons in the National Football League (NFL) with the Green Bay Packers and the Los Angeles Rams, from 1961 to 1975. Iman played in three consecutive NFL championship games with the Packers from 1960 to 1962, winning two, as well as one with the Rams (1974), a loss. He started 140 straight games with the Rams from 1965 to '74 and was voted team MVP in 1972. After losing divisional round games in 1969 and 1973, the Rams won one»\n",
- "[5] «Ken Mendenhall | Ken Mendenhall Ken E. Mendenhall (born August 11, 1948) is a former American football center who played ten seasons in the National Football League (NFL) for the Baltimore Colts. Born in Stillwater, Mendenhall grew up in Pawhuska in Osage County, Oklahoma. He later attended Enid High School and went on to play for the University of Oklahoma, where he was \"described as a devastating one-on-one blocker.\" As a 1969 All-American center, he was valued by his college teammates. \"He's always got the path cleared for you,\" said tailback Steve Owens. \"He's fantastic coming off the ball on a man right»\n",
- "[6] «Peter Boulware | Peter Boulware Peter Nicholas Boulware (born December 18, 1974) is a former American college and professional football player who was a linebacker in the National Football League (NFL) for nine seasons. He played college football for Florida State University, and was recognized as an All-American. A first-round pick of the Baltimore Ravens in the 1997 NFL Draft, he played his entire pro career for the Ravens. Boulware was a Republican candidate for the Florida House of Representatives in the 2008 general election. Boulware was born in Columbia, South Carolina. He played at Heathwood Hall Episcopal School, and then graduated first»\n",
- "[7] «Ken Strong | Red Grange and Chris Cagle rolled into one human form and there was nothing Carnegie Tech had that could stop his march. Carnegie Tech coach Walter Steffen said of Strong's performance: \"This is the first time in my career that one man was good enough to run over and completely wreck an exceptionally good team. I can tell you he is better than Heston or Thorpe.\" Strong played 16 seasons of professional football from 1929 to 1940 and 1944 to 1947. He earned a reputation as a triple-threat man and a versatile athlete who played on offense and defense and»\n",
- "\n",
- "Question: How many seasons did the man who runs KnowledgeWare play in the NFL ?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m Fran Tarkenton is a man who ran KnowledgeWare and he is credited with having coined, \"A fool with a tool is a faster fool\".\n",
- "\n",
- "Search Query: How many seasons did Fran Tarkenton play in the NFL?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book is \"The Dancing Wu Li Masters\".»\n",
- "[2] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
- "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\". It was awarded a 1980 U.S. National Book Award in category of Science.\n",
- "\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Fran Tarkenton is a man who ran KnowledgeWare and he is credited with having coined, \"A fool with a tool is a faster fool\".»\n",
- "[2] «KnowledgeWare | KnowledgeWare KnowledgeWare was a software company headquartered in Atlanta, Georgia co-founded by James Martin and run by Fran Tarkenton. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench). KnowledgeWare was sold to Sterling Software in 1994, which was in its turn acquired by Computer Associates. Tarkenton is credited with having coined, \"A fool with a tool is a faster fool\" while offering classes at their Peach Tree headquarters. Tarkenton, Don Addington and other executives were eventually involved in legal actions brought by the SEC for engaging in a fraudulent scheme to inflate KnowledgeWare's financial results»\n",
- "[3] «Fran Tarkenton | Fran Tarkenton Francis Asbury Tarkenton (born February 3, 1940) is a former National Football League (NFL) quarterback, television personality, and computer software executive. He played in the NFL for 18 seasons and spent the majority of his career with the Minnesota Vikings. Tarkenton's tenure with the Vikings spanned thirteen non-consecutive seasons, playing with the team for six seasons from 1961 to 1966, then for seven seasons from 1972 to 1978. In between his years in Minnesota, Tarkenton was a member of the New York Giants for five seasons. At the time of his retirement, Tarkenton owned every major quarterback record.»\n",
- "[4] «Fran Tarkenton | During the 2016 Republican National Convention, Tarkenton gave a speech endorsing Republican presidential nominee Donald Trump. Fran Tarkenton Francis Asbury Tarkenton (born February 3, 1940) is a former National Football League (NFL) quarterback, television personality, and computer software executive. He played in the NFL for 18 seasons and spent the majority of his career with the Minnesota Vikings. Tarkenton's tenure with the Vikings spanned thirteen non-consecutive seasons, playing with the team for six seasons from 1961 to 1966, then for seven seasons from 1972 to 1978. In between his years in Minnesota, Tarkenton was a member of the New York»\n",
- "[5] «Gary Cuozzo | After the 1971 season, the Vikings reacquired Fran Tarkenton from the New York Giants, with Snead and the Vikings' leading receiver from 1971, Bob Grim, going to the Big Apple. Tarkenton played his first six seasons (1961-66) in Minnesota, coinciding with the Vikings' first six seasons in the NFL. Cuozzo was traded to the St. Louis Cardinals in the deal which sent fleet wide receiver John Gilliam to the Vikings. Cardinals coach Bob Hollway was familiar with Cuozzo, having served as Minnesota's defensive coordinator under Bud Grant prior to leaving for St. Louis in 1971. Cuozzo was part of a»\n",
- "[6] «Fran Tarkenton | NFL records at the time of his retirement. Tarkenton's 47,003 career passing yards rank him 8th all time, while his 342 career passing touchdowns is 6th all time in NFL history. He also is 6th on the all-time list of wins by a starting quarterback with 124 regular season victories. He also used his impressive scrambling ability to rack up 3,674 rushing yards and 32 touchdowns on 675 carries. During his career, Tarkenton ran for a touchdown in 15 different seasons, an NFL record among quarterbacks. He ranks fifth in career rushing yards among quarterbacks, behind Randall Cunningham, Steve Young,»\n",
- "[7] «Fran Tarkenton | Vikings to three National Football Conference championships, but in each instance the Vikings lost the ensuing Super Bowl. In Tarkenton's first Super Bowl appearance his team lost to the Miami Dolphins 24–7 in Houston. It lost the second to the Pittsburgh Steelers 16–6 in New Orleans, and in the last Super Bowl Tarkenton played (and Minnesota's last Super Bowl to date), the Vikings lost to the Oakland Raiders 32–14 at the Rose Bowl in Pasadena, California. In his 18 NFL seasons, Tarkenton completed 3,686 of 6,467 passes for 47,003 yards and 342 touchdowns, with 266 interceptions, all of which were»\n",
- "\n",
- "Question: How many seasons did the man who runs KnowledgeWare play in the NFL ?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The man who runs KnowledgeWare is Fran Tarkenton. He played in the NFL for 18 seasons.\n",
- "\n",
- "Answer: 18\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- }
- ],
- "source": [
- "davinci.inspect_history(n=3)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Compile\n",
- "\n",
- "Now, let's compile the program to its more efficient version. When doing this, the original DSP program will be traced by the DSP runtime. Calls (and outputs) to the LM will be intercepted and an appropriately simplified version of them will become training data for fine-tuning a much smaller LM. The fine-tuned smaller LM will be automatically invoked instead of the larger LM for the transformations decorated with `@dsp.compiled` (i.e., the **Search** stage).\n",
- "\n",
- "To make it easy for anyone to use this notebook, we can set `force_reuse_cached_compilation=True` below. This will skip attempting to check that you can access the compiled version of the LM. (If you'd like to try examples NOT in this notebook, you can easily turn that off and re-compile. Compilation may take an hour or two in practice, depending on the OpenAI server load.)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [],
- "source": [
- "dsp.settings.configure(force_reuse_cached_compilation=True)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's run compilation! We'll get back the more efficient version of the program, which we call `compiled_QA`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 798/798 [00:04<00:00, 169.83it/s]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "52ed1b7eeba61e06\n",
- "ada:ft-stanfordpraglab-2023-02-06-21-06-50\n"
- ]
- }
- ],
- "source": [
- "compiled_QA = dsp.compile(program=multihop_QA, examples=multihop_800_unlabeled_questions)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can try this out on the example question."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "How many seasons did the man who runs KnowledgeWare play in the NFL ?\n",
- "18\n"
- ]
- }
- ],
- "source": [
- "question = dev[0].question\n",
- "output = compiled_QA(dev[0].question, num_queries=10, num_preds=1)\n",
- "answer = output.answer\n",
- "\n",
- "print(question)\n",
- "print(answer)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And we can inspect the small LM's outputs for the **Search** stage as well as the large LM's outputs for the **Predict** stage."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Question: How many seasons did the man who runs KnowledgeWare play in the NFL ?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the man who runs KnowledgeWare.\n",
- "\n",
- "Search Query: \"Who runs KnowledgeWare?\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «KnowledgeWare | KnowledgeWare KnowledgeWare was a software company headquartered in Atlanta, Georgia co-founded by James Martin and run by Fran Tarkenton. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench). KnowledgeWare was sold to Sterling Software in 1994, which was in its turn acquired by Computer Associates. Tarkenton is credited with having coined, \"A fool with a tool is a faster fool\" while offering classes at their Peach Tree headquarters. Tarkenton, Don Addington and other executives were eventually involved in legal actions brought by the SEC for engaging in a fraudulent scheme to inflate KnowledgeWare's financial results»\n",
- "[2] «KnowledgeWare | to meet sales and earnings projections. KnowledgeWare KnowledgeWare was a software company headquartered in Atlanta, Georgia co-founded by James Martin and run by Fran Tarkenton. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench). KnowledgeWare was sold to Sterling Software in 1994, which was in its turn acquired by Computer Associates. Tarkenton is credited with having coined, \"A fool with a tool is a faster fool\" while offering classes at their Peach Tree headquarters. Tarkenton, Don Addington and other executives were eventually involved in legal actions brought by the SEC for engaging in a fraudulent»\n",
- "[3] «David Neft | new look at professional football before 1933. Because the National Football League hadn't kept official statistics in its first thirteen seasons, knowledge of the period was extremely limited. Neft led a team of researchers that meticulously reconstructed the statistical record using box scores, play-by-play accounts, and game stories from local newspapers. The result was a remarkable new look at the teams and players from the NFL's earliest seasons. Neft returned to Harris, then owned by Gannett, in 1977, serving as executive vice president. In 1985, he became Gannett's director of research, a position he held until his retirement in 2002.»\n",
- "[4] «Ken Iman | Ken Iman Kenneth Charles Iman (February 8, 1939 – November 13, 2010) was an American football center who played 15 seasons in the National Football League (NFL) with the Green Bay Packers and the Los Angeles Rams, from 1961 to 1975. Iman played in three consecutive NFL championship games with the Packers from 1960 to 1962, winning two, as well as one with the Rams (1974), a loss. He started 140 straight games with the Rams from 1965 to '74 and was voted team MVP in 1972. After losing divisional round games in 1969 and 1973, the Rams won one»\n",
- "[5] «Ken Mendenhall | Ken Mendenhall Ken E. Mendenhall (born August 11, 1948) is a former American football center who played ten seasons in the National Football League (NFL) for the Baltimore Colts. Born in Stillwater, Mendenhall grew up in Pawhuska in Osage County, Oklahoma. He later attended Enid High School and went on to play for the University of Oklahoma, where he was \"described as a devastating one-on-one blocker.\" As a 1969 All-American center, he was valued by his college teammates. \"He's always got the path cleared for you,\" said tailback Steve Owens. \"He's fantastic coming off the ball on a man right»\n",
- "[6] «Peter Boulware | Peter Boulware Peter Nicholas Boulware (born December 18, 1974) is a former American college and professional football player who was a linebacker in the National Football League (NFL) for nine seasons. He played college football for Florida State University, and was recognized as an All-American. A first-round pick of the Baltimore Ravens in the 1997 NFL Draft, he played his entire pro career for the Ravens. Boulware was a Republican candidate for the Florida House of Representatives in the 2008 general election. Boulware was born in Columbia, South Carolina. He played at Heathwood Hall Episcopal School, and then graduated first»\n",
- "[7] «Ken Strong | Red Grange and Chris Cagle rolled into one human form and there was nothing Carnegie Tech had that could stop his march. Carnegie Tech coach Walter Steffen said of Strong's performance: \"This is the first time in my career that one man was good enough to run over and completely wreck an exceptionally good team. I can tell you he is better than Heston or Thorpe.\" Strong played 16 seasons of professional football from 1929 to 1940 and 1944 to 1947. He earned a reputation as a triple-threat man and a versatile athlete who played on offense and defense and»\n",
- "\n",
- "Question: How many seasons did the man who runs KnowledgeWare play in the NFL ?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m KnowledgeWare was a software company headquartered in Atlanta, Georgia. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench).\n",
- "\n",
- "Search Query: How many seasons did KnowledgeWare's James Martin run?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. He has appeared more than 30 times on \"The Oprah Winfrey Show\".»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "[3] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[4] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\". It was awarded a 1980 U.S. National Book Award.\n",
- "\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «KnowledgeWare was a software company headquartered in Atlanta, Georgia. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench).»\n",
- "[2] «KnowledgeWare | KnowledgeWare KnowledgeWare was a software company headquartered in Atlanta, Georgia co-founded by James Martin and run by Fran Tarkenton. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench). KnowledgeWare was sold to Sterling Software in 1994, which was in its turn acquired by Computer Associates. Tarkenton is credited with having coined, \"A fool with a tool is a faster fool\" while offering classes at their Peach Tree headquarters. Tarkenton, Don Addington and other executives were eventually involved in legal actions brought by the SEC for engaging in a fraudulent scheme to inflate KnowledgeWare's financial results»\n",
- "[3] «KnowledgeWare | to meet sales and earnings projections. KnowledgeWare KnowledgeWare was a software company headquartered in Atlanta, Georgia co-founded by James Martin and run by Fran Tarkenton. It produced a Computer Aided Software Engineering (CASE) tool called IEW (Information Engineering Workbench). KnowledgeWare was sold to Sterling Software in 1994, which was in its turn acquired by Computer Associates. Tarkenton is credited with having coined, \"A fool with a tool is a faster fool\" while offering classes at their Peach Tree headquarters. Tarkenton, Don Addington and other executives were eventually involved in legal actions brought by the SEC for engaging in a fraudulent»\n",
- "[4] «Fran Tarkenton | Fran Tarkenton Francis Asbury Tarkenton (born February 3, 1940) is a former National Football League (NFL) quarterback, television personality, and computer software executive. He played in the NFL for 18 seasons and spent the majority of his career with the Minnesota Vikings. Tarkenton's tenure with the Vikings spanned thirteen non-consecutive seasons, playing with the team for six seasons from 1961 to 1966, then for seven seasons from 1972 to 1978. In between his years in Minnesota, Tarkenton was a member of the New York Giants for five seasons. At the time of his retirement, Tarkenton owned every major quarterback record.»\n",
- "[5] «Jim Martin (American football) | Jim Martin (American football) James Richard \"Jungle Jim\" Martin (April 8, 1924 – October 9, 2002) was an American football guard, linebacker and placekicker who played fourteen seasons in the National Football League (NFL) in the 1950s and 1960s, mainly for the Detroit Lions. He was selected to the Pro Bowl, the NFL's all-star game, after the 1961 season, and went on to be an assistant coach after his playing career. He was an All-American at the University of Notre Dame and was inducted into the College Football Hall of Fame in 1995. Martin was a star athlete at his»\n",
- "[6] «James Martin (author) | later renamed James Martin Associates (JMA), which was (partly) bought by Texas Instruments Software in 1991. He later co-founded Database Design Inc. (DDI), in Ann Arbor, Michigan, to promulgate his database design techniques and to develop tools to help implement them. After becoming the market leader in information technology engineering software, DDI was renamed KnowledgeWare and eventually purchased by Fran Tarkenton, who took it public. Martin was awarded an honorary fellowship by Keble College, Oxford and an honorary Doctor of Science degree by the University of Warwick in July 2009. He gave the Turing Lecture in 2008. According to \"Computerworld\"’s»\n",
- "[7] «Fran Tarkenton | During the 2016 Republican National Convention, Tarkenton gave a speech endorsing Republican presidential nominee Donald Trump. Fran Tarkenton Francis Asbury Tarkenton (born February 3, 1940) is a former National Football League (NFL) quarterback, television personality, and computer software executive. He played in the NFL for 18 seasons and spent the majority of his career with the Minnesota Vikings. Tarkenton's tenure with the Vikings spanned thirteen non-consecutive seasons, playing with the team for six seasons from 1961 to 1966, then for seven seasons from 1972 to 1978. In between his years in Minnesota, Tarkenton was a member of the New York»\n",
- "\n",
- "Question: How many seasons did the man who runs KnowledgeWare play in the NFL ?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The man who runs KnowledgeWare is Fran Tarkenton. He played in the NFL for 18 seasons.\n",
- "\n",
- "Answer: 18\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- }
- ],
- "source": [
- "compiled_QA.lm.inspect_history(n=2)\n",
- "davinci.inspect_history(n=1)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "As the inspection above shows, `ada` generates 10 queries and ensembles retrieval across them. This makes it more likely that it'll find relevant content.\n",
- "\n",
- "Let's inspect all unique queries generated in each of the hops."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "{'\"Who runs knowledge-ware?\"', '\"Who runs KnowledgeWare?\"'}\n",
- "{'How many seasons did IEW produce?', 'How many seasons did James Martin play in the NFL?', 'How many seasons did James Martin run KnowledgeWare?', 'How many seasons did Tarkenton play in the NFL?', 'How many seasons did Tarkenton run KnowledgeWare?', \"How many seasons did KnowledgeWare's man run the software company?\", \"How many seasons did KnowledgeWare's James Martin run?\", 'How many seasons did KnowledgeWare run?'}\n"
- ]
- }
- ],
- "source": [
- "hop0_queries = [c.query for c in output.h0.completions]\n",
- "hop1_queries = [c.query for c in output.h1.completions]\n",
- "\n",
- "print(set(hop0_queries))\n",
- "print(set(hop1_queries))"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For the first hop, only two unique queries were generated by the model. Coming up with these was relatively easy. Indeed, both queries appropriately ask about who runs KnowledgeWare.\n",
- "\n",
- "For the second hop, quite a few distinct queries are generated. This required reading seven short snippets, which include many potentially distracting information, making it quite a bit more challenging. In this case, the relevant and most useful one is \"How many seasons did Tarkenton play in the NFL?\""
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Evaluating Retrieval Quality\n",
- "\n",
- "Below, we run our programs and evaluate the quality of retrieval. That is, we ask whether the top-7 contexts (collected over the hops) contain the answer string.\n",
- "\n",
- "We will evaluate:\n",
- "\n",
- "- The original program `multihop_QA` (which uses long prompts for `text-davinci-002`), and\n",
- "- The efficient program `compiled_QA` (which uses short prompts for the much smaller model `ada` for **Search**)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 17/17 [00:00<00:00, 17.22it/s]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 17 / 17 (100.0%) correctly.\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_f1d71ca0_af5d_11ed_b417_8d30c4037e3c th {\n",
- " text-align: left;\n",
- " } #T_f1d71ca0_af5d_11ed_b417_8d30c4037e3c td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3c\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow0_col0\" class=\"data row0 col0\" >How many seasons did the man who runs KnowledgeWare play in the NFL ?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow0_col1\" class=\"data row0 col1\" >['18', '18 seasons']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow0_col2\" class=\"data row0 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow1_col0\" class=\"data row1 col0\" >Who co-wrote a memoir with the retired airline captain celebrated for the water landing of Flight 1549?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow1_col1\" class=\"data row1 col1\" >['Jeffrey Zaslow']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow1_col2\" class=\"data row1 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow2_col0\" class=\"data row2 col0\" >Who was born earlier, Habash al-Marwazi or Mostafa El-Sayed?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow2_col1\" class=\"data row2 col1\" >['Habash al-Marwazi']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow2_col2\" class=\"data row2 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow3_col0\" class=\"data row3 col0\" >What is the oldest section from where Egyptologist Robert Hay was born?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow3_col1\" class=\"data row3 col1\" >['Norman Keep', 'Pele Tower', 'Norman Keep or Pele Tower']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow3_col2\" class=\"data row3 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow4_col0\" class=\"data row4 col0\" >Which American physician was a co-founder alongside Charles Mayo?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow4_col1\" class=\"data row4 col1\" >['Augustus Stinchfield', 'Augustus W. Stinchfield']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow4_col2\" class=\"data row4 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow5_col0\" class=\"data row5 col0\" >Simon Lindberg has been the leader of the group that was disbanded for inactivity in what nation in 2016?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow5_col1\" class=\"data row5 col1\" >['Denmark']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow5_col2\" class=\"data row5 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow6_col0\" class=\"data row6 col0\" >Which actor from Trainspotting also played Obi-Wan Kenobi?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow6_col1\" class=\"data row6 col1\" >['Ewan McGregor']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow6_col2\" class=\"data row6 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow7_col0\" class=\"data row7 col0\" >What is the publisher of the female superhero created by J. H. Williams III?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow7_col1\" class=\"data row7 col1\" >['DC Comics']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow7_col2\" class=\"data row7 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow8_col0\" class=\"data row8 col0\" >In what year did the team that built the Lotus E21 begin to compete under the Lotus name?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow8_col1\" class=\"data row8 col1\" >['2012']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow8_col2\" class=\"data row8 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow9_col0\" class=\"data row9 col0\" >What ranking did the man who Jonathan Brookins defeated on December 4, 2010 have in The Ultimate Fighter?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow9_col1\" class=\"data row9 col1\" >['runner-up']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow9_col2\" class=\"data row9 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow10_col0\" class=\"data row10 col0\" >Which of the lower keys is served by the bridge that crosses over the Pigeon Key Historic District?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow10_col1\" class=\"data row10 col1\" >['Little Duck Key']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow10_col2\" class=\"data row10 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow11_col0\" class=\"data row11 col0\" >The city that hosted the 1996 Summer Paralympics is the seat of what county?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow11_col1\" class=\"data row11 col1\" >['Fulton County']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow11_col2\" class=\"data row11 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow12_col0\" class=\"data row12 col0\" >What actor who was in Sixteen candles is also in the romantic comedy Results?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow12_col1\" class=\"data row12 col1\" >['Anthony Michael Hall']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow12_col2\" class=\"data row12 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow13_col0\" class=\"data row13 col0\" >Did Scott Draper or Kevin Curren play more sports?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow13_col1\" class=\"data row13 col1\" >['Scott Draper']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow13_col2\" class=\"data row13 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow14_col0\" class=\"data row14 col0\" >What role did Denis Lawson play in the 1983 British comedy-drama film written and directed by Bill Forsyth?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow14_col1\" class=\"data row14 col1\" >['Gordon Urquhart']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow14_col2\" class=\"data row14 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow15_col0\" class=\"data row15 col0\" >Which band is from a country closer to Canada, Local H or Bodyjar?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow15_col1\" class=\"data row15 col1\" >['Local H']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow15_col2\" class=\"data row15 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3clevel0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow16_col0\" class=\"data row16 col0\" >What animated show on Cartoon Network did Wil Wheaton have a role in?</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow16_col1\" class=\"data row16 col1\" >['Ben 10', 'Teen Titans']</td>\n",
- " <td id=\"T_f1d71ca0_af5d_11ed_b417_8d30c4037e3crow16_col2\" class=\"data row16 col2\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f0a4d0570a0>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "from dsp.evaluation.utils import evaluateRetrieval, evaluateAnswer\n",
- "\n",
- "evaluateRetrieval(lambda question: multihop_QA(question, num_queries=10, num_preds=1), dev)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 17/17 [00:01<00:00, 14.36it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 16 / 17 (94.1%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_f2abf93e_af5d_11ed_b417_8d30c4037e3c th {\n",
- " text-align: left;\n",
- " } #T_f2abf93e_af5d_11ed_b417_8d30c4037e3c td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3c\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow0_col0\" class=\"data row0 col0\" >How many seasons did the man who runs KnowledgeWare play in the NFL ?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow0_col1\" class=\"data row0 col1\" >['18', '18 seasons']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow0_col2\" class=\"data row0 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow1_col0\" class=\"data row1 col0\" >Who co-wrote a memoir with the retired airline captain celebrated for the water landing of Flight 1549?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow1_col1\" class=\"data row1 col1\" >['Jeffrey Zaslow']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow1_col2\" class=\"data row1 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow2_col0\" class=\"data row2 col0\" >Who was born earlier, Habash al-Marwazi or Mostafa El-Sayed?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow2_col1\" class=\"data row2 col1\" >['Habash al-Marwazi']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow2_col2\" class=\"data row2 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow3_col0\" class=\"data row3 col0\" >What is the oldest section from where Egyptologist Robert Hay was born?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow3_col1\" class=\"data row3 col1\" >['Norman Keep', 'Pele Tower', 'Norman Keep or Pele Tower']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow3_col2\" class=\"data row3 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow4_col0\" class=\"data row4 col0\" >Which American physician was a co-founder alongside Charles Mayo?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow4_col1\" class=\"data row4 col1\" >['Augustus Stinchfield', 'Augustus W. Stinchfield']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow4_col2\" class=\"data row4 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow5_col0\" class=\"data row5 col0\" >Simon Lindberg has been the leader of the group that was disbanded for inactivity in what nation in 2016?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow5_col1\" class=\"data row5 col1\" >['Denmark']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow5_col2\" class=\"data row5 col2\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow6_col0\" class=\"data row6 col0\" >Which actor from Trainspotting also played Obi-Wan Kenobi?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow6_col1\" class=\"data row6 col1\" >['Ewan McGregor']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow6_col2\" class=\"data row6 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow7_col0\" class=\"data row7 col0\" >What is the publisher of the female superhero created by J. H. Williams III?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow7_col1\" class=\"data row7 col1\" >['DC Comics']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow7_col2\" class=\"data row7 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow8_col0\" class=\"data row8 col0\" >In what year did the team that built the Lotus E21 begin to compete under the Lotus name?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow8_col1\" class=\"data row8 col1\" >['2012']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow8_col2\" class=\"data row8 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow9_col0\" class=\"data row9 col0\" >What ranking did the man who Jonathan Brookins defeated on December 4, 2010 have in The Ultimate Fighter?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow9_col1\" class=\"data row9 col1\" >['runner-up']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow9_col2\" class=\"data row9 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow10_col0\" class=\"data row10 col0\" >Which of the lower keys is served by the bridge that crosses over the Pigeon Key Historic District?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow10_col1\" class=\"data row10 col1\" >['Little Duck Key']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow10_col2\" class=\"data row10 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow11_col0\" class=\"data row11 col0\" >The city that hosted the 1996 Summer Paralympics is the seat of what county?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow11_col1\" class=\"data row11 col1\" >['Fulton County']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow11_col2\" class=\"data row11 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow12_col0\" class=\"data row12 col0\" >What actor who was in Sixteen candles is also in the romantic comedy Results?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow12_col1\" class=\"data row12 col1\" >['Anthony Michael Hall']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow12_col2\" class=\"data row12 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow13_col0\" class=\"data row13 col0\" >Did Scott Draper or Kevin Curren play more sports?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow13_col1\" class=\"data row13 col1\" >['Scott Draper']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow13_col2\" class=\"data row13 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow14_col0\" class=\"data row14 col0\" >What role did Denis Lawson play in the 1983 British comedy-drama film written and directed by Bill Forsyth?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow14_col1\" class=\"data row14 col1\" >['Gordon Urquhart']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow14_col2\" class=\"data row14 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow15_col0\" class=\"data row15 col0\" >Which band is from a country closer to Canada, Local H or Bodyjar?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow15_col1\" class=\"data row15 col1\" >['Local H']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow15_col2\" class=\"data row15 col2\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3clevel0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow16_col0\" class=\"data row16 col0\" >What animated show on Cartoon Network did Wil Wheaton have a role in?</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow16_col1\" class=\"data row16 col1\" >['Ben 10', 'Teen Titans']</td>\n",
- " <td id=\"T_f2abf93e_af5d_11ed_b417_8d30c4037e3crow16_col2\" class=\"data row16 col2\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f0a56956ee0>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "evaluateRetrieval(lambda question: compiled_QA(question, num_queries=10, num_preds=1), dev)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Evaluating Answer Quality\n",
- "\n",
- "Next, let's evaluate the output (i.e., whether the final prediction of the program matches the ground-truth label)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 17/17 [00:00<00:00, 180.88it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 15 / 17 (88.2%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_f2c86510_af5d_11ed_b417_8d30c4037e3c th {\n",
- " text-align: left;\n",
- " } #T_f2c86510_af5d_11ed_b417_8d30c4037e3c td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3c\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow0_col0\" class=\"data row0 col0\" >How many seasons did the man who runs KnowledgeWare play in the NFL ?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow0_col1\" class=\"data row0 col1\" >['18', '18 seasons']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow0_col2\" class=\"data row0 col2\" >18</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow1_col0\" class=\"data row1 col0\" >Who co-wrote a memoir with the retired airline captain celebrated for the water landing of Flight 1549?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow1_col1\" class=\"data row1 col1\" >['Jeffrey Zaslow']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow1_col2\" class=\"data row1 col2\" >Jeffrey Zaslow</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow2_col0\" class=\"data row2 col0\" >Who was born earlier, Habash al-Marwazi or Mostafa El-Sayed?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow2_col1\" class=\"data row2 col1\" >['Habash al-Marwazi']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow2_col2\" class=\"data row2 col2\" >Habash al-Marwazi</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow2_col3\" class=\"data row2 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow3_col0\" class=\"data row3 col0\" >What is the oldest section from where Egyptologist Robert Hay was born?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow3_col1\" class=\"data row3 col1\" >['Norman Keep', 'Pele Tower', 'Norman Keep or Pele Tower']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow3_col2\" class=\"data row3 col2\" >1320</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow4_col0\" class=\"data row4 col0\" >Which American physician was a co-founder alongside Charles Mayo?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow4_col1\" class=\"data row4 col1\" >['Augustus Stinchfield', 'Augustus W. Stinchfield']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow4_col2\" class=\"data row4 col2\" >Augustus Stinchfield</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow5_col0\" class=\"data row5 col0\" >Simon Lindberg has been the leader of the group that was disbanded for inactivity in what nation in 2016?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow5_col1\" class=\"data row5 col1\" >['Denmark']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow5_col2\" class=\"data row5 col2\" >Sweden</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow5_col3\" class=\"data row5 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow6_col0\" class=\"data row6 col0\" >Which actor from Trainspotting also played Obi-Wan Kenobi?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow6_col1\" class=\"data row6 col1\" >['Ewan McGregor']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow6_col2\" class=\"data row6 col2\" >Ewan McGregor</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow6_col3\" class=\"data row6 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow7_col0\" class=\"data row7 col0\" >What is the publisher of the female superhero created by J. H. Williams III?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow7_col1\" class=\"data row7 col1\" >['DC Comics']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow7_col2\" class=\"data row7 col2\" >DC Comics</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow7_col3\" class=\"data row7 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow8_col0\" class=\"data row8 col0\" >In what year did the team that built the Lotus E21 begin to compete under the Lotus name?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow8_col1\" class=\"data row8 col1\" >['2012']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow8_col2\" class=\"data row8 col2\" >2012</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow8_col3\" class=\"data row8 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow9_col0\" class=\"data row9 col0\" >What ranking did the man who Jonathan Brookins defeated on December 4, 2010 have in The Ultimate Fighter?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow9_col1\" class=\"data row9 col1\" >['runner-up']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow9_col2\" class=\"data row9 col2\" >runner-up</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow9_col3\" class=\"data row9 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow10_col0\" class=\"data row10 col0\" >Which of the lower keys is served by the bridge that crosses over the Pigeon Key Historic District?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow10_col1\" class=\"data row10 col1\" >['Little Duck Key']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow10_col2\" class=\"data row10 col2\" >Little Duck Key</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow10_col3\" class=\"data row10 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow11_col0\" class=\"data row11 col0\" >The city that hosted the 1996 Summer Paralympics is the seat of what county?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow11_col1\" class=\"data row11 col1\" >['Fulton County']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow11_col2\" class=\"data row11 col2\" >Fulton County</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow12_col0\" class=\"data row12 col0\" >What actor who was in Sixteen candles is also in the romantic comedy Results?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow12_col1\" class=\"data row12 col1\" >['Anthony Michael Hall']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow12_col2\" class=\"data row12 col2\" >Anthony Michael Hall</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow13_col0\" class=\"data row13 col0\" >Did Scott Draper or Kevin Curren play more sports?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow13_col1\" class=\"data row13 col1\" >['Scott Draper']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow13_col2\" class=\"data row13 col2\" >Scott Draper</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow13_col3\" class=\"data row13 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow14_col0\" class=\"data row14 col0\" >What role did Denis Lawson play in the 1983 British comedy-drama film written and directed by Bill Forsyth?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow14_col1\" class=\"data row14 col1\" >['Gordon Urquhart']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow14_col2\" class=\"data row14 col2\" >Gordon Urquhart</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow14_col3\" class=\"data row14 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow15_col0\" class=\"data row15 col0\" >Which band is from a country closer to Canada, Local H or Bodyjar?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow15_col1\" class=\"data row15 col1\" >['Local H']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow15_col2\" class=\"data row15 col2\" >Local H</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow15_col3\" class=\"data row15 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3clevel0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow16_col0\" class=\"data row16 col0\" >What animated show on Cartoon Network did Wil Wheaton have a role in?</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow16_col1\" class=\"data row16 col1\" >['Ben 10', 'Teen Titans']</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow16_col2\" class=\"data row16 col2\" >Teen Titans</td>\n",
- " <td id=\"T_f2c86510_af5d_11ed_b417_8d30c4037e3crow16_col3\" class=\"data row16 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f0a47726610>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "from dsp.utils import EM, F1\n",
- "\n",
- "evaluateAnswer(lambda question: multihop_QA(question, num_queries=10, num_preds=1), dev, metric=EM)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 17/17 [00:00<00:00, 174.52it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 14 / 17 (82.4%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_f2f29a60_af5d_11ed_b417_8d30c4037e3c th {\n",
- " text-align: left;\n",
- " } #T_f2f29a60_af5d_11ed_b417_8d30c4037e3c td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3c\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow0_col0\" class=\"data row0 col0\" >How many seasons did the man who runs KnowledgeWare play in the NFL ?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow0_col1\" class=\"data row0 col1\" >['18', '18 seasons']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow0_col2\" class=\"data row0 col2\" >18</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow1_col0\" class=\"data row1 col0\" >Who co-wrote a memoir with the retired airline captain celebrated for the water landing of Flight 1549?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow1_col1\" class=\"data row1 col1\" >['Jeffrey Zaslow']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow1_col2\" class=\"data row1 col2\" >Jeffrey Zaslow</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow2_col0\" class=\"data row2 col0\" >Who was born earlier, Habash al-Marwazi or Mostafa El-Sayed?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow2_col1\" class=\"data row2 col1\" >['Habash al-Marwazi']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow2_col2\" class=\"data row2 col2\" >Habash al-Marwazi</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow2_col3\" class=\"data row2 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow3_col0\" class=\"data row3 col0\" >What is the oldest section from where Egyptologist Robert Hay was born?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow3_col1\" class=\"data row3 col1\" >['Norman Keep', 'Pele Tower', 'Norman Keep or Pele Tower']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow3_col2\" class=\"data row3 col2\" >Norman Keep or Pele Tower</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow3_col3\" class=\"data row3 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow4_col0\" class=\"data row4 col0\" >Which American physician was a co-founder alongside Charles Mayo?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow4_col1\" class=\"data row4 col1\" >['Augustus Stinchfield', 'Augustus W. Stinchfield']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow4_col2\" class=\"data row4 col2\" >Augustus Stinchfield</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow5_col0\" class=\"data row5 col0\" >Simon Lindberg has been the leader of the group that was disbanded for inactivity in what nation in 2016?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow5_col1\" class=\"data row5 col1\" >['Denmark']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow5_col2\" class=\"data row5 col2\" >Sweden</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow5_col3\" class=\"data row5 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow6_col0\" class=\"data row6 col0\" >Which actor from Trainspotting also played Obi-Wan Kenobi?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow6_col1\" class=\"data row6 col1\" >['Ewan McGregor']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow6_col2\" class=\"data row6 col2\" >Ewan McGregor</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow6_col3\" class=\"data row6 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow7_col0\" class=\"data row7 col0\" >What is the publisher of the female superhero created by J. H. Williams III?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow7_col1\" class=\"data row7 col1\" >['DC Comics']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow7_col2\" class=\"data row7 col2\" >DC Comics</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow7_col3\" class=\"data row7 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow8_col0\" class=\"data row8 col0\" >In what year did the team that built the Lotus E21 begin to compete under the Lotus name?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow8_col1\" class=\"data row8 col1\" >['2012']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow8_col2\" class=\"data row8 col2\" >2012</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow8_col3\" class=\"data row8 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow9_col0\" class=\"data row9 col0\" >What ranking did the man who Jonathan Brookins defeated on December 4, 2010 have in The Ultimate Fighter?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow9_col1\" class=\"data row9 col1\" >['runner-up']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow9_col2\" class=\"data row9 col2\" >runner-up</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow9_col3\" class=\"data row9 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow10_col0\" class=\"data row10 col0\" >Which of the lower keys is served by the bridge that crosses over the Pigeon Key Historic District?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow10_col1\" class=\"data row10 col1\" >['Little Duck Key']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow10_col2\" class=\"data row10 col2\" >Little Duck Key</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow10_col3\" class=\"data row10 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow11_col0\" class=\"data row11 col0\" >The city that hosted the 1996 Summer Paralympics is the seat of what county?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow11_col1\" class=\"data row11 col1\" >['Fulton County']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow11_col2\" class=\"data row11 col2\" >Fulton County</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow12_col0\" class=\"data row12 col0\" >What actor who was in Sixteen candles is also in the romantic comedy Results?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow12_col1\" class=\"data row12 col1\" >['Anthony Michael Hall']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow12_col2\" class=\"data row12 col2\" >Michael Schoeffling</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow12_col3\" class=\"data row12 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow13_col0\" class=\"data row13 col0\" >Did Scott Draper or Kevin Curren play more sports?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow13_col1\" class=\"data row13 col1\" >['Scott Draper']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow13_col2\" class=\"data row13 col2\" >Scott Draper</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow13_col3\" class=\"data row13 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow14_col0\" class=\"data row14 col0\" >What role did Denis Lawson play in the 1983 British comedy-drama film written and directed by Bill Forsyth?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow14_col1\" class=\"data row14 col1\" >['Gordon Urquhart']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow14_col2\" class=\"data row14 col2\" >Gordon Urquhart</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow14_col3\" class=\"data row14 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow15_col0\" class=\"data row15 col0\" >Which band is from a country closer to Canada, Local H or Bodyjar?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow15_col1\" class=\"data row15 col1\" >['Local H']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow15_col2\" class=\"data row15 col2\" >Bodyjar</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow15_col3\" class=\"data row15 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3clevel0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow16_col0\" class=\"data row16 col0\" >What animated show on Cartoon Network did Wil Wheaton have a role in?</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow16_col1\" class=\"data row16 col1\" >['Ben 10', 'Teen Titans']</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow16_col2\" class=\"data row16 col2\" >Teen Titans</td>\n",
- " <td id=\"T_f2f29a60_af5d_11ed_b417_8d30c4037e3crow16_col3\" class=\"data row16 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f0a568c6f40>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "evaluateAnswer(lambda question: compiled_QA(question, num_queries=10, num_preds=1), dev, metric=EM) # watch cost increase from $13 to ?? for 100 examples"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Citation\n",
- "\n",
- "If you use DSP, please cite:\n",
- "```\n",
- "@article{khattab2022demonstrate,\n",
- " title={Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive {NLP}},\n",
- " author={Khattab, Omar and Santhanam, Keshav and Li, Xiang Lisa and Hall, David and Liang, Percy and Potts, Christopher and Zaharia, Matei},\n",
- " journal={arXiv preprint arXiv:2212.14024},\n",
- " year={2022}\n",
- "}\n",
- "```\n",
- "\n",
- "\n",
- "If you found this notebook helpful, please cite:\n",
- "```\n",
- "@misc{khattab2023dspcompiler,\n",
- "\tauthor = {},\n",
- "\ttitle = {DSP compiler v0.1},\n",
- "\thowpublished = {\\url{https://github.com/stanfordnlp/dsp/blob/main/compiler.ipynb}},\n",
- "\tyear = {},\n",
- "\tnote = {[Accessed 13-Feb-2023]},\n",
- "}\n",
- "\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "base",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.5"
- },
- "orig_nbformat": 4,
- "vscode": {
- "interpreter": {
- "hash": "de72ef8826336f005e74d4a3f2b2dfb9239c3f097fd15aff69f3f6a4fc3bd8b8"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/dsp_v1_demos/intro.ipynb b/dsp_v1_demos/intro.ipynb
deleted file mode 100644
index 8fd0704d1..000000000
--- a/dsp_v1_demos/intro.ipynb
+++ /dev/null
@@ -1,2289 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 🎓 𝗗𝗦𝗣: 𝗗emonstrate–𝗦earch–𝗣redict\n",
- "\n",
- "This notebook illustrates the Demonstrate–Search–Predict (𝗗𝗦𝗣) framework on the multi-hop question answering (QA) task.\n",
- "\n",
- "Below, we'll build **five** small 𝗗𝗦𝗣 programs that implement strategies for this task. The programs range from extremely simple ones (programs 1 and 2) to moderate ones (programs 3 and 4). Program 5 considers a fairly advanced pipeline."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Installation\n",
- "\n",
- "If you haven't installed **DSP** already, let's do that."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
- " import google.colab \n",
- " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
- "except: pass\n",
- "\n",
- "!pip install -U pip dsp-ml"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Setting Up\n",
- "\n",
- "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
- "\n",
- "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
- "\n",
- "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
- "\n",
- "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "%load_ext autoreload\n",
- "%autoreload 2\n",
- "\n",
- "try: import google.colab; root_path = 'dsp'\n",
- "except: root_path = '.'\n",
- "\n",
- "import os\n",
- "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
- "\n",
- "import dsp\n",
- "\n",
- "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
- "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
- "\n",
- "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
- "rm = dsp.ColBERTv2(url=colbert_server)\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Task Examples\n",
- "\n",
- "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
- "\n",
- "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
- "\n",
- "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
- " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
- " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
- " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
- " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
- " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
- " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
- " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
- " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
- " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
- " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
- " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
- " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Program 1: Vanilla GPT-3.5\n",
- "\n",
- "Let's begin by implementing the simplest program for this task. We'll prompt **GPT-3.5** to answer each question based on its internal parameteric knowledge.\n",
- "\n",
- "We'll start by defining the `Template` that defines how we will communicate with the LM.\n",
- "\n",
- "Specifically, the question–answer template (`qa_template`) will include a question and a short answer for each example."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
- "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
- "\n",
- "qa_template = dsp.Template(instructions=\"Answer questions with short factoid answers.\", question=Question(), answer=Answer())"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Then, let's define the actual program, `vanilla_LM_QA`. It'll accept a string (`question`) and returns another string (its short `answer`)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "def vanilla_LM_QA(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " example = dsp.Example(question=question, demos=demos)\n",
- "\n",
- " example, completions = dsp.generate(qa_template)(example, stage='qa')\n",
- " return completions.answer"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's invoke the program on a sample question."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "E. L. Doctorow\n"
- ]
- }
- ],
- "source": [
- "print(dev[0].question)\n",
- "print(vanilla_LM_QA(dev[0].question))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's inspect the last call to the LM to learn more about the behavior of the program."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- }
- ],
- "source": [
- "lm.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can see that the program has organized the examples in a simple prompt and generated an answer for the `dev[0]` question we asked.\n",
- "\n",
- "Let's try on the remaining development examples."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 456.27it/s]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 3 / 13 (23.1%) correctly.\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Melbourne, Australia</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1966</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >July 30, 1953</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Hudson River</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Hong Kong</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Venus</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1457</td>\n",
- " <td id=\"T_a2eab340_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >❌</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9bd91ce1c0>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "23.1"
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from dsp.evaluation.utils import evaluate\n",
- "\n",
- "evaluate(vanilla_LM_QA, dev)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Program 2: Retrieve-then-Read w/ GPT-3.5\n",
- "\n",
- "The program above relies entirely on the factual knowledge memorized by GPT-3. As we have seen, it's sometimes correct but it's far from reliable.\n",
- "\n",
- "Let's try to address this. Our first improvement to the vanilla **GPT-3.5** strategy is to include search results in the LM prompt, often called a *retrieve-then-read* approach.\n",
- "\n",
- "In particular, we'll retrieve the most relevant passage to the question (according to the retriever) from the corpus and concatenate it into the prompt. This may help GPT-3.5 answer more factually.\n",
- "\n",
- "To do this, we'll need to modify the `Template`. In particular, we'll define a `qa_template_with_passages` template, which will include a `context` field in addition to a question and an answer."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [],
- "source": [
- "Context = dsp.Type(\n",
- " prefix=\"Context:\\n\",\n",
- " desc=\"${sources that may contain relevant content}\",\n",
- " format=dsp.passages2text\n",
- ")\n",
- "\n",
- "qa_template_with_passages = dsp.Template(\n",
- " instructions=qa_template.instructions,\n",
- " context=Context(), question=Question(), answer=Answer()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now let's modify the program. We'll call `dsp.retrieve`, which we hooked earlier in this notebook to a **ColBERTv2** retriever serving a Wikipedia (Dec 2018) index."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [],
- "source": [
- "def retrieve_then_read_QA(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " passages = dsp.retrieve(question, k=1)\n",
- " \n",
- " example = dsp.Example(question=question, context=passages, demos=demos)\n",
- " example, completions = dsp.generate(qa_template_with_passages)(example, stage='qa')\n",
- "\n",
- " return completions.answer"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's invoke this on a sample question and let's inspect the last call to the LM to learn more about the behavior of the program."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "«Julia Peterkin | Julia Peterkin Julia Peterkin (October 31, 1880 – August 10, 1961) was an American author from South Carolina. In 1929 she won the Pulitzer Prize for Novel/Literature, for her novel \"Scarlet Sister Mary.\" She wrote several novels about the plantation South, especially the Gullah people of the Low Country. She was one of the few white authors who wrote about the African-American experience. Julia Mood was born in Laurens County, South Carolina. Her father was a physician, and she was the third of his four children. Her mother died soon after her birth, and her father later married Janie Brogdon.»\n",
- "\n",
- "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
- "\n",
- "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('E. L. Doctorow', None)"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "retrieve_then_read_QA(dev[0].question), lm.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now, let's evaluate this on the dev examples."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 376.92it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 5 / 13 (38.5%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida.</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1854</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine is English.</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >August 5, 1962</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk, Ireland</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Enid, Oklahoma</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_a36fcc88_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9bb40bea30>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "38.5"
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "evaluate(retrieve_then_read_QA, dev)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Program 3: Retrieve-then-Read w/ Self-Consistency\n",
- "\n",
- "It's clear that retrieval has the capacity to help **GPT-3.5** answer a larger number of questions more factually. However, these questions are too complicated for a single retrieved passage to suffice.\n",
- "\n",
- "In this program, we try to make some improvements to the pipeline from **Program 2**.\n",
- "\n",
- "In particular:\n",
- "\n",
- "- We will include 5 passages (instead of a single passage) into the prompt.\n",
- "- We will ask GPT-3.5 to generate a Chain-of-Thought (CoT) rationale to more effectively extract the answer from the passages.\n",
- "- We will use Self-Consistency (SC) to marginalize this prediction across many chains of thought."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [],
- "source": [
- "Rationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step.\",\n",
- " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
- ")\n",
- "\n",
- "qa_template_with_CoT = dsp.Template(\n",
- " instructions=qa_template.instructions,\n",
- " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The snippet above updates our template to include a `rationale` field.\n",
- "\n",
- "Below, we define the new program `retrieve_then_read_QA_v2`. In it, we define a new DSP _transformation_ `QA_predict`, which we will re-use in multiple programs.\n",
- "\n",
- "The transformation `QA_predict` takes an `Example` (which has a question, context, and demonstrations) and generates an answer. It can be configured to use self-consistency (`sc=True`) or not."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsp.transformation\n",
- "def QA_predict(example: dsp.Example, sc=True):\n",
- " if sc:\n",
- " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
- " completions = dsp.majority(completions)\n",
- " else:\n",
- " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
- " \n",
- " return example.copy(answer=completions.answer)\n",
- "\n",
- "def retrieve_then_read_QA_v2(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " passages = dsp.retrieve(question, k=5)\n",
- " example = dsp.Example(question=question, context=passages, demos=demos)\n",
- " \n",
- " return QA_predict(example).answer"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's inspect an example."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "[2] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[3] «Right Back at It Again | band played \"Right Back at It Again\" on both occasions. AbsolutePunk writer Thomas Nassiff said the \"mosh call\" lyric \"\"Brace for impact\"\" had \"pretty much ruin[ed] that track\". Tamsyn Wilce for Alter the Press! noted that the way \"Right Back at it Again\" follows on from opening track \"City of Ocala\" \"continues the catchy hardcore beats\" that is present in all of the band's albums. \"Rock Sound\"'s Andy Ritchie called the song \"unmistakably the 'All I Want' of 'Common Courtesy'\". A music video for the song was released on 19 December 2013. The music video was nominated for Best Video»\n",
- "[4] «Born Again in Dixieland | Born Again in Dixieland \"Born Again in Dixieland\" is a song co-written and recorded by Canadian country music artist Jason McCoy. Released in June 1997, it was the lead single from his album \"Playin' for Keeps\". The song reached #3 on the \"RPM\" Country Tracks chart in September 1997. The song won a SOCAN Song of the Year award. McCoy said that he started writing the song with Sheridan and he was playing songs for MCA Canada label and they asked if he had any other songs. He played a verse and the chorus from the song thinking it wasn't»\n",
- "[5] «Right Back Where We Started From | 1974. In a 3 May 2008 interview with Michael Shelley of WFMU, Edwards recalled that after hearing Maxine Nightingale sing on the session for Al Matthews' \"Fool\" that track's producer Pierre Tubbs had come up with \"Right Back Where We Started From\" as a good title for a song for Nightingale herself to record and had invited Edwards to co-write the song. Utilizing a tune which Edwards had written \"a couple of years before\", Tubbs and Edwards wrote \"Right Back Where We Started From\" in about seven minutes while driving to Charing Cross Hospital where Tubbs' wife Gabrielle (née Zimmerman)»\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The song is by A Day to Remember, and the lyrics were co-written by the singer, Jeremy McKinnon. We just need to find out where McKinnon was born.\n",
- "\n",
- "Answer: Orlando, Florida\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('Ocala, Florida', None)"
- ]
- },
- "execution_count": 16,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "retrieve_then_read_QA_v2(dev[2].question), lm.inspect_history(n=1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And let's run this on the other dev examples."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:01<00:00, 11.23it/s]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 7 / 13 (53.8%) correctly.\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1824</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aeronautics and Astronautics</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_a4df802c_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9baab4a850>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "53.8"
- ]
- },
- "execution_count": 17,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "evaluate(retrieve_then_read_QA_v2, dev)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Program 4: Multi-Hop Retrieval\n",
- "\n",
- "From these simple programs, it becomes clear that a single search query is not enough for this task. This can be seen when trying to find the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it doesn't mention when he was born.\n",
- "\n",
- "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019), IRRR (Qi et al., 2021) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "With **DSP**, we can easily simulate such systems in a few lines of code. Let's begin by defining two new templates.\n",
- "\n",
- "First, in `rewrite_template`, we will re-write the input question into a simple search query. This will serve as our \"first hop\" query."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {},
- "outputs": [],
- "source": [
- "SearchRationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",
- " desc=\"${the missing information}\"\n",
- ")\n",
- "\n",
- "SearchQuery = dsp.Type(\n",
- " prefix=\"Search Query:\",\n",
- " desc=\"${a simple question for seeking the missing information}\"\n",
- ")\n",
- "\n",
- "rewrite_template = dsp.Template(\n",
- " instructions=\"Write a search query that will help answer a complex question.\",\n",
- " question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next, in `hop_template`, we will use the retrieved information from earlier hops to generate additional search queries for missing information."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [],
- "source": [
- "CondenseRationale = dsp.Type(\n",
- " prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\",\n",
- " desc=\"${information from the context that provides useful clues}\"\n",
- ")\n",
- "\n",
- "hop_template = dsp.Template(\n",
- " instructions=rewrite_template.instructions,\n",
- " context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's build our `multihop_QA_v1` program. The hallmark of this program is the search transformation called `multihop_search_v1`.\n",
- "\n",
- "This will largely simulate the IRRR system. In each hop, we will retrieve `k` (e.g., `k=2`) passages and concatenate them to the context retrieved from earlier hops to form a `context` chain.\n",
- "\n",
- "For simplicity, we will repeat this for a total of `max_hops` hops (e.g., `max_hops=2`). It's easy to add a stopping criteria in which the LM is asked whether more hops are required."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {},
- "outputs": [],
- "source": [
- "from dsp.utils import deduplicate\n",
- "\n",
- "@dsp.transformation\n",
- "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",
- " example.context = []\n",
- " \n",
- " for hop in range(max_hops):\n",
- " # Generate a query based\n",
- " template = rewrite_template if hop == 0 else hop_template\n",
- " example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
- "\n",
- " # Retrieve k results based on the query generated\n",
- " passages = dsp.retrieve(completions.query, k=k)\n",
- "\n",
- " # Update the context by concatenating old and new passages\n",
- " example.context = deduplicate(example.context + passages)\n",
- "\n",
- " return example\n",
- "\n",
- "\n",
- "def multihop_QA_v1(question: str) -> str:\n",
- " demos = dsp.sample(train, k=7)\n",
- " x = dsp.Example(question=question, demos=demos)\n",
- " \n",
- " x = multihop_search_v1(x)\n",
- " x = QA_predict(x, sc=False)\n",
- "\n",
- " return x.answer"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's run this on one example and inspect the last 3 calls to the LM (i.e., search query 1, search query 2, and QA)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the singer\n",
- "Search Query: Who sings Right Back At It Again\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
- "\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The song is called \"Right Back at It Again\" and it is by the band A Day to Remember. The singer, Jeremy McKinnon, wrote the lyrics.\n",
- "\n",
- "Search Query: Where was Jeremy McKinnon born?\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Answer: Foxcatcher\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
- "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
- "[3] «Jeremy McKinnon | Jeremy McKinnon Jeremy Wade McKinnon (born December 17, 1985) is an American musician and record producer, best known as a founding member and the lead vocalist of A Day to Remember. He has produced full-length albums for The Devil Wears Prada, The Ghost Inside, Neck Deep, and Wage War. Jeremy McKinnon was born in Gainesville, Florida and grew up with two sisters. His parents are originally from Brooklyn, New York and is of Irish and Italian descent. McKinnon first got a job at Boston Market and later went on to do construction work. McKinnon's interest in music came as a»\n",
- "[4] «Martin McKinnon | Martin McKinnon Martin \"Marty\" McKinnon (born 5 July 1975 in Adelaide) is a former Australian rules footballer who played with Adelaide, Geelong and the Brisbane Lions in the Australian Football League (AFL). McKinnon was recruited by Adelaide in the 1992 AFL Draft, with their first ever national draft pick. He was the youngest player on Adelaide's list at the time and played for Central District in the SANFL when not appearing with Adelaide. He made his AFL debut late in the 1994 AFL season and was a regular in the team the following year, playing mostly as a wingman. He»\n",
- "\n",
- "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The song \"Right Back at It Again\" is co-written by the singer Jeremy McKinnon. A quick search of \"Jeremy McKinnon\" reveals that he was born in Gainesville, Florida.\n",
- "\n",
- "Answer: Gainesville, Florida\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('Gainesville, Florida', None)"
- ]
- },
- "execution_count": 21,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "multihop_QA_v1(dev[2].question), lm.inspect_history(n=3)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now, let's run this on the remaining dev examples."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:00<00:00, 167.43it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 7 / 13 (53.8%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer: A Mother's Hunt for Justice</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1947</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >UK</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >September 28, 1969</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >space technology</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_a5d5e426_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9bb041dc10>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "53.8"
- ]
- },
- "execution_count": 22,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "evaluate(multihop_QA_v1, dev)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Program 5: Multi-Hop Condensed Retrieval w/ Automatic Demos and Query Fusion\n",
- "\n",
- "Through Program 4, we've begun to explore some of the power of the DSP abstraction. However, if you look closely, you will see a few downsides of the previous approach:\n",
- "\n",
- "1. The search transformations invoke the LM without any demonstrations in the prompt. That is because we only have training data for the question–answer pairs and not for the intermediate labels (e.g., search queries).\n",
- "2. The QA prompt uses passages (`context`) and a Chain-of-Thought (`rationale`) for the question to be answered. However, the training demonstrations include neither context nor CoT because they aren't available in our labels.\n",
- "3. The search transformations commit to a single query per hop, which may single out an unproductive chain of passages and hence fail to uncover relevant information."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We address these problems automatically in the program below.\n",
- "\n",
- "In it, we begin by defining `multihop_demonstrate` (which uses `multihop_attempt`) to automatically **annotate** demonstrations for the complex multi-hop pipeline. These demonstrations will be provided to the LM when it's invoked for each transformation."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsp.transformation\n",
- "def multihop_attempt(d: dsp.Example) -> dsp.Example:\n",
- " # Prepare unaugmented demonstrations for the example.\n",
- " x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))\n",
- " \n",
- " # Search. And skip examples where search fails.\n",
- " # Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.\n",
- " x = multihop_search_v1(x)\n",
- " if not dsp.passage_match(x.context, d.answer): return None\n",
- " \n",
- " # Predict. And skip examples where predict fails.\n",
- " x = QA_predict(x, sc=False)\n",
- " if not dsp.answer_match(x.answer, d.answer): return None\n",
- " \n",
- " return d.copy(**x)\n",
- "\n",
- "@dsp.transformation\n",
- "def multihop_demonstrate(x: dsp.Example) -> dsp.Example:\n",
- " demos = dsp.sample(train, k=7)\n",
- " x.demos = dsp.annotate(multihop_attempt)(demos, k=3, return_all=True)\n",
- " return x"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We now implement `multihop_search_v2` as part of `multihop_QA_v2`.\n",
- "\n",
- "In addition to the changes mentioned earlier, this program simulates the Baleen system (Khattab et al., 2021) in a few lines of code.\n",
- "\n",
- "In each retrieval hop (after the very first hop), the summary of the previous hop(s) is included in the prompt. This allows us to efficiently read a larger number of passages from the current hop."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "metadata": {},
- "outputs": [],
- "source": [
- "@dsp.transformation\n",
- "def multihop_search_v2(example: dsp.Example, max_hops=2, k=5) -> dsp.Example:\n",
- " example.context = []\n",
- "\n",
- " for hop in range(max_hops):\n",
- " # Generate queries\n",
- " template = rewrite_template if hop == 0 else hop_template\n",
- " example, completions = dsp.generate(template, n=10, temperature=0.7)(example, stage=f'h{hop}')\n",
- " \n",
- " # Collect the queries and search with result fusion\n",
- " queries = [c.query for c in completions] + [example.question]\n",
- " example.context = dsp.retrieveEnsemble(queries, k=k)\n",
- "\n",
- " # Arrange the passages for the next hop\n",
- " if hop > 0:\n",
- " example.context = [completions[0].rationale] + example.context\n",
- " \n",
- " return example\n",
- "\n",
- "def multihop_QA_v2(question: str) -> str:\n",
- " x = dsp.Example(question=question)\n",
- " x = multihop_demonstrate(x)\n",
- " x = multihop_search_v2(x)\n",
- " x = QA_predict(x)\n",
- " return x.answer"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 25,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
- "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
- "Search Query: \"Heir to the Du Pont family fortune\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
- "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the party of the winner of the 1971 San Francisco mayoral election.\n",
- "Search Query: \"Party of the winner of the 1971 San Francisco mayoral election\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
- "\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
- "\n",
- "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
- "\n",
- "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
- "\n",
- "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «1971 San Francisco mayoral election | 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another as advocates for law and order. Alioto challenged Dobbs to commit to keeping Police Chief Donald Scott and accused Feinstein of wanting to create a civilian review board to investigate police conduct.»\n",
- "[2] «1971 San Francisco mayoral election | the November 2 election was the re-election of Mayor Joseph Alioto by a 3–2 margin over his nearest challenger. Alioto quoted Jackie Gleason—\"How sweet it is!\"—at a victory celebration and referred to his federal indictment as an attempt by the \"national administration\" to \"knock out a mayor\". Among other races on the ballot was Richard Hongisto's surprise victory for sheriff over incumbent Matthew Carberry. On the ballot was a referendum limiting new-building heights to six stories, which was defeated. A referendum to make the school board an elective one, it being the only appointive board in the state, passed. The»\n",
- "[3] «1971 San Francisco mayoral election | school-board measure was regarded as a reaction to busing initiated by the school board under court order. Another measure approved was to keep cable car service at no less than current levels. 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another»\n",
- "[4] «1971 San Francisco mayoral election | achieve ethnic balance in elementary schools. Former newspaper editor Scott Newhall proposed staging a world ecology fair and replacing California's water plan by importing icebergs, and imputed little difference between the top three contenders, whom he lumped together as \"Joe Dobbstein\". Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official. As of November, Dobbs, the only Republican in the race, had passed Feinstein in polls and was gaining on Alioto. The result of»\n",
- "[5] «2003 San Francisco mayoral election | against leftist/progressive Gonzalez, a member of the Green Party. It was the first mayoral election in San Francisco that a Green Party candidate took a noticeable amount of the vote. The election was close, with Gonzalez leading in the polls and winning the popular vote among ballots cast on election day, while Newsom had a larger lead on absentee ballots. 2003 San Francisco mayoral election The 2003 San Francisco mayoral election occurred on November 4, 2003. The incumbent, Willie Brown, was termed out of office and could not seek a third term. The general election included three top candidates including»\n",
- "\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.\n",
- "\n",
- "Search Query: What year was the Democratic Party founded?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
- "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
- "\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
- "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "\n",
- "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
- "\n",
- "Answer: Foxcatcher team\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
- "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
- "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "\n",
- "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
- "\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.»\n",
- "[2] «Democratic Party (UK, 1998) | Democratic Party (UK, 1998) The Democratic Party was a political party in the United Kingdom, founded in November 1998 by Malvern businessman Geoff Southall, who also funded the party. It was described as \"right wing\" or \"hard right\" in news reports, but claimed to occupy the centre-ground of British politics. It aimed to reduce Britain's involvement with the European Union, opposed the adoption of the Euro, called for direct democracy, and argued for limits on immigration. The slogan was \"The will of the people NOT the party\". In 1999 it had a few hundred members, including previous supporters of James»\n",
- "[3] «Democratic Party (Brazil, 1925-1934) | Democratic Party (Brazil, 1925-1934) The Democratic Party was a Brazilian political party, but state character, founded by dissidents of the Paulista Republican Party (PRP) on February 24, 1925, during the Old Republic and was its manifesto signed by 599 signatories. It was a representative organization of the traditional middle class, linked to coffee sectors, but especially the urban. In May 1926, the Youth Party (Youth Wing of dissenters, who had organized this caption for the federal elections of 1925), also dissident perrepismo, joined the Democratic Party. His most important names were Waldemar Ferreira, José Adriano Marrey Junior, Francisco Morato, Paulo»\n",
- "[4] «Democratic Party (Turkey, 1970) | Democratic Party (Turkey, 1970) Democratic Party (Turkish: \"Demokratik Parti\", DP for short) was a former political party in Turkey founded on 18 December 1970 . It should not be confused with the historical Democrat Party or the current Democrat Party in Turkey. The historical Democrat Party which was the ruling party between 1950-1960 was closed by the military rule. In 1961 and 1965 elections most of ex Democratic Party adherents supported Justice Party, a party founded in 1961. After 1965 elections Justice Party came to power. But following 1969 elections in which Justice Party was again successful, tension between the»\n",
- "[5] «Democratic Party (Japan, 1947) | Democratic Party (Japan, 1947) Democratic Party (民主党, Minshutō) was a centrist political party in Japan. It was founded in spring 1947 by merging the Progressive Party (Shinpo-tō) of Inukai Takeru with a faction of Liberal Party led by Hitoshi Ashida and obtained 124 seats in 1947 elections. The party had held seven seats in Tetsu Katayama's government in 1947-1948. For some months in 1948, party's leader Ashida was Prime minister. In March 1948, part of DP members led by Kijūrō Shidehara joined the Liberal Party to form the Democratic Liberal Party. In 1949 elections, the DP got 69 seats. The»\n",
- "[6] «Democratic Party (United States) | Democratic Party (United States) The Democratic Party is one of the two major contemporary political parties in the United States, along with the Republican Party. Tracing its heritage back to Thomas Jefferson and James Madison's Democratic-Republican Party, the modern-day Democratic Party was founded around 1828 by supporters of Andrew Jackson, making it the world's oldest active political party. The Democrats' dominant worldview was once social conservatism and economic liberalism while populism was its leading characteristic in the rural South. In 1912, Theodore Roosevelt ran as a third-party candidate in the Progressive (\"Bull Moose\") Party, beginning a switch of political platforms»\n",
- "\n",
- "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The question is asking for the year the party of the mayor of San Francisco in 1971 was founded. In source 1, we see that the election was in 1971 and the winner was Joseph Alioto. In source 2, we see that the party was founded in 1998, which is not correct. In source 3, we see that the party was founded in 1925, which is not correct. In source 4, we see that the party was founded in 1970, which is not correct. In source 5, we see that the party was founded in 1947, which is not correct. In source 6, we see that the party was founded around 1828. This must be the correct answer, so we can check source 1 to confirm. In source 1\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('1828', None)"
- ]
- },
- "execution_count": 25,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "multihop_QA_v2(dev[3].question), lm.inspect_history(n=3)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 26,
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "100%|██████████| 13/13 [00:02<00:00, 5.38it/s]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Answered 11 / 13 (84.6%) correctly.\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "<style type=\"text/css\" >\n",
- " #T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2 th {\n",
- " text-align: left;\n",
- " } #T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2 td {\n",
- " text-align: left;\n",
- " }</style><table id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2\" ><thead> <tr> <th class=\"blank level0\" ></th> <th class=\"col_heading level0 col0\" >question</th> <th class=\"col_heading level0 col1\" >answer</th> <th class=\"col_heading level0 col2\" >prediction</th> <th class=\"col_heading level0 col3\" >correct</th> </tr></thead><tbody>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col2\" class=\"data row3 col2\" >1828</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col2\" class=\"data row7 col2\" >February 13, 1980</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row7_col3\" class=\"data row7 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row8_col3\" class=\"data row8 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row9_col3\" class=\"data row9 col3\" >❌</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
- " </tr>\n",
- " <tr>\n",
- " <th id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col2\" class=\"data row12 col2\" >1442</td>\n",
- " <td id=\"T_a7f48afa_af5d_11ed_93d3_e9fb445dd3f2row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
- " </tr>\n",
- " </tbody></table>"
- ],
- "text/plain": [
- "<pandas.io.formats.style.Styler at 0x7f9ba36cfb50>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "text/plain": [
- "84.6"
- ]
- },
- "execution_count": 26,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "evaluate(multihop_QA_v2, dev)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Ask your own questions\n",
- "\n",
- "You could also ask your own questions. This can be a productive vehicle to find problems in the programs above and implement patches for them! (Keep in mind that the search index we're using currently is from Dec 2018.)\n",
- "\n",
- "To do this, make sure you set your API key. This can be done by defining the `OPENAI_API_KEY` environemt variable or setting it at the beginning of the program (Cell #2)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
- "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
- "Search Query: \"Heir to the Du Pont family fortune\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
- "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
- "\n",
- "---\n",
- "\n",
- "Question: When was the creator of Hadoop given an award?\n",
- "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the creator of Hadoop.\n",
- "Search Query: \"Name of the creator of Hadoop\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Write a search query that will help answer a complex question.\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
- "\n",
- "Search Query: ${a simple question for seeking the missing information}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
- "\n",
- "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
- "\n",
- "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
- "\n",
- "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Apache Hadoop | the Java programming language, with some native code in C and command line utilities written as shell scripts. Though MapReduce Java code is common, any programming language can be used with Hadoop Streaming to implement the map and reduce parts of the user's program. Other projects in the Hadoop ecosystem expose richer user interfaces. According to its co-founders, Doug Cutting and Mike Cafarella, the genesis of Hadoop was the Google File System paper that was published in October 2003. This paper spawned another one from Google \"MapReduce: Simplified Data Processing on Large Clusters\". Development started on the Apache Nutch project,»\n",
- "[2] «Doug Cutting | Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked at Excite, where he was one of the chief designers of the search engine, and Apple Inc.,»\n",
- "[3] «Doug Cutting | September 2010, he was elected its chairman. In 2015, O'Reilly awarded an open source award to Cutting. Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked»\n",
- "[4] «Apache Hadoop | but was moved to the new Hadoop subproject in January 2006. Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant. The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce. The first committer to add to the Hadoop project was Owen O'Malley (in March 2006); Hadoop 0.1.0 was released in April 2006. It continues to evolve through contributions that are being made to the project. Hadoop consists of the \"Hadoop Common\" package, which provides file system»\n",
- "[5] «Doug Cutting | on the MapReduce algorithm, which allows very large scale computations to be trivially parallelized across large clusters of servers. Cutting and Cafarella, realizing the importance of this paper to extending Lucene into the realm of extremely large search problems, created the open-source Hadoop framework that allows applications based on the MapReduce paradigm to be run on large clusters of commodity hardware. Cutting was an employee of Yahoo!, where he led the Hadoop project full-time. He later went on to work for Cloudera. In July 2009, Cutting was elected to the board of directors of the Apache Software Foundation, and in»\n",
- "\n",
- "Question: When was the creator of Hadoop given an award?\n",
- "\n",
- "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m Apache Hadoop is an open-source software framework for distributed storage and distributed processing of big data sets on computer clusters. The creator of Hadoop is Doug Cutting. In 2015, O'Reilly awarded an open source award to Cutting.\n",
- "\n",
- "Search Query: When was the creator of Hadoop given an award?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "Answer questions with short factoid answers.\n",
- "\n",
- "---\n",
- "\n",
- "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Answer: Butch Vig\n",
- "\n",
- "Question: What city was the victim of Joseph Druces working in?\n",
- "Answer: Boston, Massachusetts\n",
- "\n",
- "Question: In what year was the star of To Hell and Back born?\n",
- "Answer: 1925\n",
- "\n",
- "---\n",
- "\n",
- "Follow the following format.\n",
- "\n",
- "Context:\n",
- "${sources that may contain relevant content}\n",
- "\n",
- "Question: ${the question to be answered}\n",
- "\n",
- "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
- "\n",
- "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
- "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
- "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
- "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
- "\n",
- "Question: Which award did the first book of Gary Zukav receive?\n",
- "\n",
- "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
- "\n",
- "Answer: U.S. National Book Award\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
- "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
- "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
- "\n",
- "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
- "\n",
- "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
- "\n",
- "Answer: Foxcatcher team\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
- "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
- "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
- "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
- "\n",
- "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
- "\n",
- "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
- "\n",
- "Answer: Kevin Greutert\n",
- "\n",
- "---\n",
- "\n",
- "Context:\n",
- "[1] «Apache Hadoop is an open-source software framework for distributed storage and distributed processing of big data sets on computer clusters. The creator of Hadoop is Doug Cutting. In 2015, O'Reilly awarded an open source award to Cutting.»\n",
- "[2] «Doug Cutting | September 2010, he was elected its chairman. In 2015, O'Reilly awarded an open source award to Cutting. Doug Cutting Douglass Read Cutting is a software designer and advocate and creator of open-source search technology. He founded Lucene and, with Mike Cafarella, Nutch, both open-source search technology projects which are now managed through the Apache Software Foundation. Cutting and Cafarella are also the co-founders of Apache Hadoop. Cutting holds a bachelor's degree from Stanford University. Prior to developing Lucene, Cutting held search technology positions at Xerox PARC where he worked on the Scatter/Gather algorithm and on computational stylistics. He also worked»\n",
- "[3] «Anoop Swarup | 1984. He initiated the UNDP-NYKS Poverty Alleviation Awards in 2000, was instrumental in launching a ‘National Think Tank’ in the Ministry of Finance in 2004 and campaign on ‘Indian Civilization through the Millennium’ (ICTM) in 2008 now an ongoing yearly event at Hastinapur. He founded the ‘Sambhav’ Trust in the Indian State of Rajasthan for de addiction in 2008. Anoop Swarup is associated with various organizations that include Board of Directors of OISCA Organization for Industrial, Spiritual and Cultural Advancement International, Tokyo Japan, United Nations Inter Agency working group on Population and Development as part of United Nations Development Assistance»\n",
- "[4] «Apache Hadoop | but was moved to the new Hadoop subproject in January 2006. Doug Cutting, who was working at Yahoo! at the time, named it after his son's toy elephant. The initial code that was factored out of Nutch consisted of about 5,000 lines of code for HDFS and about 6,000 lines of code for MapReduce. The first committer to add to the Hadoop project was Owen O'Malley (in March 2006); Hadoop 0.1.0 was released in April 2006. It continues to evolve through contributions that are being made to the project. Hadoop consists of the \"Hadoop Common\" package, which provides file system»\n",
- "[5] «Apache SystemML | November 2, 2015. On May 17, 2017, the Apache Software Foundation Board approved the graduation of Apache SystemML as an Apache Top Level Project. Apache SystemML Apache SystemML is a flexible machine learning system that automatically scales to Spark and Hadoop clusters. SystemML's distinguishing characteristics are: SystemML was created in 2010 by researchers at the IBM Almaden Research Center led by IBM Fellow Shivakumar Vaithyanathan. It was observed that data scientists would write machine learning algorithms in languages such as R and Python for small data. When it came time to scale to big data, a systems programmer would be»\n",
- "[6] «Matei Zaharia | the Apache Mesos project. He also designed and implemented the core scheduling algorithms used in Apache Hadoop. He received two Best Paper awards at NSDI 2012 and SIGCOMM 2012, Honorable Mention for Community Award at NSDI 2012, and a Best Demo Award at SIGMOD 2012. Jointly with Reynold Xin, Parviz Deyhim, Xiangrui Meng, and Ali Ghodsi, he holds the 2014 world record in Daytona GraySort using Apache Spark. In 2015 he received the ACM Doctoral Dissertation Award. Matei Zaharia Matei Zaharia is a Romanian-Canadian computer scientist specializing in big data, distributed systems, and cloud computing. He is a co-founder and»\n",
- "\n",
- "Question: When was the creator of Hadoop given an award?\n",
- "\n",
- "Rationale: Let's think step by step.\u001b[32m The question is asking for the year when the creator of Hadoop was given an award. In source 1, we see that the creator of Hadoop is Doug Cutting and that in 2015, O'Reilly awarded an open source award to Cutting. Therefore, we can infer that the answer is 2015. We can check sources 2 and 3 to see if there is any mention of an award given to Doug Cutting in 2015, but since there is no mention, we can assume that source 1 is correct and the answer is 2015.\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
- "\n",
- "\n",
- "\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "('2015', None)"
- ]
- },
- "execution_count": 27,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "multihop_QA_v2(\"When was the creator of Hadoop given an award?\"), lm.inspect_history(n=3)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Additional Programs...\n",
- "\n",
- "While Program 5 is undeniably better than, say, Program 1, it's far from perfect.\n",
- "\n",
- "Many different ideas can easily be implemented to make it more general or more powerful. Refer to the DSP paper for inspiration!\n",
- "\n",
- "One of your first improvements to Programs 4 and 5 would be to allow it to do more than two hops and to automatically stop when additional hops are not required.\n",
- "\n",
- "**Hint:** This is pretty easy to do. You can start with Program 4 and modify `hop_template` so its `SearchQuery` produces the string `N/A` when additional hops are not required.\n",
- "\n",
- "In the `multihop_search_v1` function, you can then break from the loop — i.e., `if completions.query == 'N/A': break` whenever this happens. How will you apply this to Program 5?\n",
- "\n",
- "```\n",
- "SearchQueryWithStopping = dsp.Type(\n",
- " prefix=\"Search Query:\",\n",
- " desc=\"${a simple question for seeking the missing or required information --- say N/A if the context above contains all of the required information}\"\n",
- ")\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Other incremental improvements will likely help program 5, including keeping the top passage from each hop in the context and/or the summary of each hop. To decide which changes to make, you can inspect the trace of the program when it makes mistakes!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Program 4 with max_hops=4 and automatic stopping."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Additional Capabilities"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## more LLMs for use\n",
- "Eventhough the most quality tested model is openai text-davinci, we also offer support for `openai chatgpt, cohere, and hf models`.\n",
- "\n",
- "`Note:` The quality and stability might be different based on the provider and the model. Please report any bugs that you come across."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# initializing chatgpt\n",
- "\n",
- "lm = dsp.GPT3(model='gpt-3.5-turbo', api_key=openai_key, model_type='chat')\n",
- "\n",
- "# initializing cohere\n",
- "\n",
- "lm = dsp.Cohere(model='command-xlarge-nightly')\n",
- "\n",
- "# initializing hf models\n",
- "\n",
- "lm = dsp.HFModel(model=\"google/flan-t5-base\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Re-Ranker\n",
- "\n",
- "You could also construct powerful retrieval programs that uses both `retriever and re-ranker model`.\n",
- "\n",
- "Currently, we support `Sentence-Transformers cross-encoder` for reranking, and an example initialisation and usage is provided below"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "sbert_reranker = dsp.SentenceTransformersCrossEncoder(\"cross-encoder/ms-marco-MiniLM-L-12-v2\")\n",
- "\n",
- "dsp.settings.configure(lm=lm, rm=rm, reranker=sbert_reranker)\n",
- "\n",
- "# dsp.retrieve, and retrieveEnsemble already takes care of retrieving and re-ranking based on initiated rm, and reranker.\n",
- "\n",
- "reranked_passages = dsp.retrieve(\"When was the creator of Hadoop given an award?\", k=3)"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.4"
- },
- "vscode": {
- "interpreter": {
- "hash": "de72ef8826336f005e74d4a3f2b2dfb9239c3f097fd15aff69f3f6a4fc3bd8b8"
- }
- },
- "widgets": {
- "application/vnd.jupyter.widget-state+json": {
- "state": {},
- "version_major": 2,
- "version_minor": 0
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/dsp_v1_demos/knn_demonstrations_test.ipynb b/dsp_v1_demos/knn_demonstrations_test.ipynb
deleted file mode 100644
index 88fa7c7c8..000000000
--- a/dsp_v1_demos/knn_demonstrations_test.ipynb
+++ /dev/null
@@ -1,372 +0,0 @@
-{
- "cells": [
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## 🎓 DSP + KNN: use similar Questions to add better in-context examples\n",
- "\n",
- "This notebook illustrates the KNN-based prompt picking for Demonstrate stage of DSP framework.\n",
- "\n",
- "There are two types of supported embedders:\n",
- "1. with the help of models from `sentence-transformers` library;\n",
- "2. using OpenAI's paid API. \n",
- "\n",
- "For the first option, you need to install `sentence-transformers` using pip (`pip install sentence-transformers`). Please check this [page](https://www.sbert.net/) to check the installation instructions. This library allows you to use a set of pre-trained feature extractors so that it will convert sentences to embeddings under the hood.\n",
- "\n",
- "For both options, you need to install the FAISS library to perform KNN/ANN (Approximate Nearest Neighbor) search.\n",
- "\n",
- "You can install the CPU version of the library with `pip install faiss-cpu`. This is enough for most of the use-cases, especially if your dataset of Questions and Answers is small (<1M samples).\n",
- "\n",
- "For the official instructions (on FAISS with GPU support) and guidelines, please check [this doc page](https://github.com/facebookresearch/faiss/blob/main/INSTALL.md). "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "import dsp\n",
- "import os\n",
- "import openai\n",
- "# api_key = '<replace with your API key>'\n",
- "# openai.api_key = api_key\n",
- "\n",
- "os.environ['KMP_DUPLICATE_LIB_OK'] = 'TRUE'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Case 1: precomputed embeddings\n",
- "\n",
- "If you have precomputed embeddings, you can manually assign a vector for each of the Examples. \n",
- "\n",
- "For the demonstration reason, we will shift some embeddings so that we will know they aren't the closest ones for some queries."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "\n",
- "EMB_DIM = 768\n",
- "np.random.seed(42)\n",
- "\n",
- "\n",
- "three_closest_embeddings = np.random.randn(3, EMB_DIM)\n",
- "# shift the last three so they won't be the closest ones\n",
- "three_furthest_embeddings = three_closest_embeddings.copy() + 10\n",
- "train_embeddings = np.concatenate([three_closest_embeddings, three_furthest_embeddings], axis=0)\n",
- "\n",
- "for idx, example in enumerate(train):\n",
- " example.vectorized = train_embeddings[idx]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow'])]\n",
- "\n",
- "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]\n",
- "dev[0].vectorized = train_embeddings[1] # the same as in train, 2nd will be closest 100%"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "dsp.settings.configure(vectorizer=dsp.NaiveGetFieldVectorizer())"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [],
- "source": [
- "knn_func = dsp.knn(train)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[1, 0, 2]"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "knn_res_dev = knn_func(dev[0], 3)\n",
- "[train.index(i) for i in knn_res_dev]\n",
- "# first 3 train examples are the closest, the first one with index 1 is the first in the list"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[4, 3, 5]"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "knn_res_train = knn_func(train[-2], 3)\n",
- "[train.index(i) for i in knn_res_train]\n",
- "# as expected, Eamples 0-1-2 can't be among 3 closest, because they were shifted"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Case 2: compute embeddings using pre-trained Sentence-Transformers\n",
- "\n",
- "In this case, the model will be downloaded & used to extract features from the question text."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [],
- "source": [
- "# we need to add some relevant and irrelevant questions to double-check the logic\n",
- "\n",
- "train = [\n",
- " # base questions\n",
- " ('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
- " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
- " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
- " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
- " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
- " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),\n",
- " # extra, with the same entities (persons) mentioned\n",
- " ('Who founded Smart Studios in Madison, while still performing drums in Spooner at night and driving a taxi cab during the day?', ['Butch Vig']),\n",
- " ('There were two founders of recording studio called \"Smart Studios\". The first one was Butch Vig. Who was the second founder?', ['Steve Marker']),\n",
- "\n",
- " ('What is the first book of Gary Zukav?', ['\"The Dancing Wu Li Masters\"', \"'The Dancing Wu Li Masters'\", \"The Dancing Wu Li Masters\"]),\n",
- " ('When did Zukav returned to the United States?', ['1970'])\n",
- "]\n",
- "\n",
- "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Who produced the album that included a re-recording of \"Lithium\"?\n",
- "--------------------------------------------------------------------------------\n",
- "Who produced the album that included a re-recording of \"Lithium\"?\n",
- "Who founded Smart Studios in Madison, while still performing drums in Spooner at night and driving a taxi cab during the day?\n",
- "There were two founders of recording studio called \"Smart Studios\". The first one was Butch Vig. Who was the second founder?\n"
- ]
- }
- ],
- "source": [
- "QUERY_IDX = 0\n",
- "\n",
- "# context manager is working properly\n",
- "with dsp.settings.context(vectorizer=dsp.SentenceTransformersVectorizer()):\n",
- " knn_func = dsp.knn(train)\n",
- "\n",
- " knn_res_train_vec = knn_func(train[QUERY_IDX], 3)\n",
- " print(train[QUERY_IDX].question) # the query itself\n",
- " # similar questions\n",
- " print('-'*80)\n",
- " for i in [train[train.index(i)].question for i in knn_res_train_vec]:\n",
- " print(i)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "<dsp.modules.sentence_vectorizer.NaiveGetFieldVectorizer at 0x7faae3d71fd0>"
- ]
- },
- "execution_count": 11,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "# still Naive, not SentenceTransformers - because context has ended its scope\n",
- "dsp.settings.vectorizer"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [],
- "source": [
- "# set new vectorizer for all calls\n",
- "dsp.settings.configure(vectorizer=dsp.SentenceTransformersVectorizer())"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Which award did the first book of Gary Zukav receive?\n",
- "--------------------------------------------------------------------------------\n",
- "Which award did the first book of Gary Zukav receive?\n",
- "What is the first book of Gary Zukav?\n",
- "When did Zukav returned to the United States?\n"
- ]
- }
- ],
- "source": [
- "QUERY_IDX = 4\n",
- "\n",
- "knn_res_train_vec = knn_func(train[QUERY_IDX], 3)\n",
- "print(train[QUERY_IDX].question) # the query itself\n",
- "# similar questions\n",
- "print('-'*80)\n",
- "for i in [train[train.index(i)].question for i in knn_res_train_vec]:\n",
- " print(i)"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Case 3: compute embeddings using paid API from OpenAI\n",
- "\n",
- "In this case, we will send Questions to the API endpoint and get embeddings to search among."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "<dsp.modules.sentence_vectorizer.OpenAIVectorizer object at 0x7faac15c6190>\n",
- "Who produced the album that included a re-recording of \"Lithium\"?\n",
- "--------------------------------------------------------------------------------\n",
- "Who produced the album that included a re-recording of \"Lithium\"?\n",
- "There were two founders of recording studio called \"Smart Studios\". The first one was Butch Vig. Who was the second founder?\n",
- "Who founded Smart Studios in Madison, while still performing drums in Spooner at night and driving a taxi cab during the day?\n"
- ]
- }
- ],
- "source": [
- "QUERY_IDX = 0\n",
- "\n",
- "# let's try OpenAI embeddings\n",
- "with dsp.settings.context(vectorizer=dsp.OpenAIVectorizer()):\n",
- " print(dsp.settings.vectorizer)\n",
- " knn_func = dsp.knn(train)\n",
- "\n",
- " knn_res_train_vec = knn_func(train[QUERY_IDX], 3)\n",
- " print(train[QUERY_IDX].question) # the query itself\n",
- " # similar questions\n",
- " print('-'*80)\n",
- " for i in [train[train.index(i)].question for i in knn_res_train_vec]:\n",
- " print(i)\n",
- "\n",
- "# the order is different from embeddings from SentenceTransformers"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "base",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.5"
- },
- "orig_nbformat": 4,
- "vscode": {
- "interpreter": {
- "hash": "3b53cdb0bf9b815b12de7512a3c2ebfaf76b70a6c1108d7bdac63e2ca9602dd2"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}