DSPy: The framework for programming—not prompting—language models
Python
Add Program 4
Rick Battle441dca822f61
diff --git a/demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb b/demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynbnew file mode 100644index 000000000..95229e448--- /dev/null+++ b/demos/Multi-hop QA Program 4 - Multi-hop Retrieval.ipynb@@ -0,0 +1,634 @@+{+ "cells": [+ {+ "cell_type": "markdown",+ "id": "e1877b7d-f0b3-4430-a81a-69954d9fc40b",+ "metadata": {},+ "source": [+ "# Multi-hop QA Program 4: Multi-hop Retrieval\n",+ "\n",+ "This notebook is a stand-alone version of Program 4 from the intro notebook."+ ]+ },+ {+ "cell_type": "markdown",+ "id": "60078f63-0739-4217-8cf0-607e2238b9e9",+ "metadata": {},+ "source": [+ "### Installation\n",+ "\n",+ "If you haven't installed **DSP** already, let's do that.\n",+ "\n",+ "Note: If you're running this from a cloned copy of the repo, then you can skip this block."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": null,+ "id": "bc0733b3-d712-4539-b38a-ed8204b2e117",+ "metadata": {},+ "outputs": [],+ "source": [+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",+ " import google.colab \n",+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",+ "except: pass\n",+ "\n",+ "!pip install -U pip dsp-ml"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "b6215c64-6afa-4f78-8026-5ce373863a8a",+ "metadata": {},+ "source": [+ "### Setting Up\n",+ "\n",+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",+ "\n",+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",+ "\n",+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",+ "\n",+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 1,+ "id": "083dc4db-e9bf-4e00-8b20-555931c44bb0",+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Not loading Cohere because it is not installed.\n"+ ]+ }+ ],+ "source": [+ "%load_ext autoreload\n",+ "%autoreload 2\n",+ "\n",+ "try: import google.colab; root_path = 'dsp'\n",+ "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",+ "except: root_path = '../'\n",+ "\n",+ "import os\n",+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",+ "\n",+ "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",+ "import sys\n",+ "sys.path.insert(0, '../')\n",+ "\n",+ "import dsp\n",+ "\n",+ "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",+ "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",+ "\n",+ "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",+ "rm = dsp.ColBERTv2(url=colbert_server)\n",+ "\n",+ "dsp.settings.configure(lm=lm, rm=rm)"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "8fbd3484-6447-4086-9bd5-4897a45c7c96",+ "metadata": {},+ "source": [+ "### Task Examples\n",+ "\n",+ "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",+ "\n",+ "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",+ "\n",+ "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 2,+ "id": "a8a383c8-5f7f-44c1-bb73-1f7218c2ccdf",+ "metadata": {},+ "outputs": [],+ "source": [+ "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",+ " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",+ " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",+ " ('In what year was the star of To Hell and Back born?', ['1925']),\n",+ " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",+ " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",+ "\n",+ "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "e69681a4-6fe0-4d84-a6ab-f7c2bc61ddce",+ "metadata": {},+ "source": [+ "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 3,+ "id": "dc4c710b-c850-41ad-973e-661d352db565",+ "metadata": {},+ "outputs": [],+ "source": [+ "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",+ " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",+ " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",+ " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",+ " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",+ " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",+ " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",+ " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",+ " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",+ " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",+ " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",+ " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",+ " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",+ "\n",+ "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "fbd4f6a5-fff6-4832-8fdf-e0f9bd505a65",+ "metadata": {},+ "source": [+ "### Program Definition\n",+ "\n",+ "From Programs 1 to 3, it becomes clear that a single search query is not enough for this task. This can be seen when trying to find the birth city of the writer of \"Right Back At It Again\". A search query identifies the author correctly as \"Jeremy McKinnon\", but it doesn't mention when he was born.\n",+ "\n",+ "The standard approach for this challenge in the retrieval-augmented NLP literature is to build multi-hop search systems, like GoldEn (Qi et al., 2019), IRRR (Qi et al., 2021) and Baleen (Khattab et al., 2021). These systems read the retrieved results and then generate additional queries to gather additional information if necessary."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 4,+ "id": "d8647458-84f6-4fde-84c0-ff8181d935f2",+ "metadata": {},+ "outputs": [],+ "source": [+ "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",+ "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",+ "\n",+ "Context = dsp.Type(\n",+ " prefix=\"Context:\\n\",\n",+ " desc=\"${sources that may contain relevant content}\",\n",+ " format=dsp.passages2text\n",+ ")\n",+ "\n",+ "Rationale = dsp.Type(\n",+ " prefix=\"Rationale: Let's think step by step.\",\n",+ " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",+ ")\n",+ "\n",+ "qa_template_with_CoT = dsp.Template(\n",+ " instructions=\"Answer questions with short factoid answers.\",\n",+ " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "6bd8492c-df35-4d18-a516-64083933a081",+ "metadata": {},+ "source": [+ "With **DSP**, we can easily simulate such systems in a few lines of code. Let's begin by defining two new templates.\n",+ "\n",+ "First, in `rewrite_template`, we will re-write the input question into a simple search query. This will serve as our \"first hop\" query."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 5,+ "id": "0b84b21f-459a-4df7-8c3e-0fa392a0e46d",+ "metadata": {},+ "outputs": [],+ "source": [+ "SearchRationale = dsp.Type(\n",+ " prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",+ " desc=\"${the missing information}\"\n",+ ")\n",+ "\n",+ "SearchQuery = dsp.Type(\n",+ " prefix=\"Search Query:\",\n",+ " desc=\"${a simple question for seeking the missing information}\"\n",+ ")\n",+ "\n",+ "rewrite_template = dsp.Template(\n",+ " instructions=\"Write a search query that will help answer a complex question.\",\n",+ " question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",+ ")"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 6,+ "id": "ec31457c-a3cb-46b1-b08f-545023856737",+ "metadata": {},+ "outputs": [],+ "source": [+ "CondenseRationale = dsp.Type(\n",+ " prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\",\n",+ " desc=\"${information from the context that provides useful clues}\"\n",+ ")\n",+ "\n",+ "hop_template = dsp.Template(\n",+ " instructions=rewrite_template.instructions,\n",+ " context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",+ ")"+ ]+ },+ {+ "cell_type": "markdown",+ "id": "3d18d0c5-1994-4c63-8fc6-17570b5fde9a",+ "metadata": {},+ "source": [+ "Let's build our `multihop_QA_v1` program. The hallmark of this program is the search transformation called `multihop_search_v1`.\n",+ "\n",+ "This will largely simulate the IRRR system. In each hop, we will retrieve `k` (e.g., `k=2`) passages and concatenate them to the context retrieved from earlier hops to form a `context` chain.\n",+ "\n",+ "For simplicity, we will repeat this for a total of `max_hops` hops (e.g., `max_hops=2`). It's easy to add a stopping criteria in which the LM is asked whether more hops are required."+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 7,+ "id": "e53dae43-7f15-4160-82aa-a09fc262ed0c",+ "metadata": {},+ "outputs": [],+ "source": [+ "from dsp.utils import deduplicate\n",+ "\n",+ "@dsp.transformation\n",+ "def QA_predict(example: dsp.Example, sc=True):\n",+ " if sc:\n",+ " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",+ " completions = dsp.majority(completions)\n",+ " else:\n",+ " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",+ " \n",+ " return example.copy(answer=completions.answer)\n",+ "\n",+ "def Retrieve_then_Read_QA_v2(question: str) -> str:\n",+ " demos = dsp.sample(train, k=7)\n",+ " passages = dsp.retrieve(question, k=5)\n",+ " example = dsp.Example(question=question, context=passages, demos=demos)\n",+ " \n",+ " return QA_predict(example).answer\n",+ "\n",+ "@dsp.transformation\n",+ "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",+ " example.context = []\n",+ " \n",+ " for hop in range(max_hops):\n",+ " # Generate a query based\n",+ " template = rewrite_template if hop == 0 else hop_template\n",+ " example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",+ "\n",+ " # Retrieve k results based on the query generated\n",+ " passages = dsp.retrieve(completions.query, k=k)\n",+ "\n",+ " # Update the context by concatenating old and new passages\n",+ " example.context = deduplicate(example.context + passages)\n",+ "\n",+ " return example\n",+ "\n",+ "\n",+ "def multihop_QA_v1(question: str) -> str:\n",+ " demos = dsp.sample(train, k=7)\n",+ " x = dsp.Example(question=question, demos=demos)\n",+ " \n",+ " x = multihop_search_v1(x)\n",+ " x = QA_predict(x, sc=False)\n",+ "\n",+ " return x.answer"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 8,+ "id": "f47ff2cd-3502-4df3-81d7-310eb0db5ac2",+ "metadata": {},+ "outputs": [+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "\n",+ "\n",+ "\n",+ "\n",+ "Write a search query that will help answer a complex question.\n",+ "\n",+ "---\n",+ "\n",+ "Follow the following format.\n",+ "\n",+ "Question: ${the question to be answered}\n",+ "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",+ "Search Query: ${a simple question for seeking the missing information}\n",+ "\n",+ "---\n",+ "\n",+ "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",+ "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the singer\n",+ "Search Query: Who sings Right Back At It Again\u001b[0m\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "Write a search query that will help answer a complex question.\n",+ "\n",+ "---\n",+ "\n",+ "Follow the following format.\n",+ "\n",+ "Context:\n",+ "${sources that may contain relevant content}\n",+ "\n",+ "Question: ${the question to be answered}\n",+ "\n",+ "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",+ "\n",+ "Search Query: ${a simple question for seeking the missing information}\n",+ "\n",+ "---\n",+ "\n",+ "Context:\n",+ "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",+ "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",+ "\n",+ "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",+ "\n",+ "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The song is called \"Right Back at It Again\" and it is by the band A Day to Remember. The singer, Jeremy McKinnon, wrote the lyrics.\n",+ "\n",+ "Search Query: Where was Jeremy McKinnon born?\u001b[0m\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "\n",+ "Answer questions with short factoid answers.\n",+ "\n",+ "---\n",+ "\n",+ "Question: Which award did the first book of Gary Zukav receive?\n",+ "Answer: U.S. National Book Award\n",+ "\n",+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",+ "Answer: Foxcatcher\n",+ "\n",+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",+ "Answer: Kevin Greutert\n",+ "\n",+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",+ "Answer: Butch Vig\n",+ "\n",+ "Question: What city was the victim of Joseph Druces working in?\n",+ "Answer: Boston, Massachusetts\n",+ "\n",+ "Question: In what year was the star of To Hell and Back born?\n",+ "Answer: 1925\n",+ "\n",+ "---\n",+ "\n",+ "Follow the following format.\n",+ "\n",+ "Context:\n",+ "${sources that may contain relevant content}\n",+ "\n",+ "Question: ${the question to be answered}\n",+ "\n",+ "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",+ "\n",+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",+ "\n",+ "---\n",+ "\n",+ "Context:\n",+ "[1] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",+ "[2] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",+ "[3] «Jeremy McKinnon | Jeremy McKinnon Jeremy Wade McKinnon (born December 17, 1985) is an American musician and record producer, best known as a founding member and the lead vocalist of A Day to Remember. He has produced full-length albums for The Devil Wears Prada, The Ghost Inside, Neck Deep, and Wage War. Jeremy McKinnon was born in Gainesville, Florida and grew up with two sisters. His parents are originally from Brooklyn, New York and is of Irish and Italian descent. McKinnon first got a job at Boston Market and later went on to do construction work. McKinnon's interest in music came as a»\n",+ "[4] «Martin McKinnon | Martin McKinnon Martin \"Marty\" McKinnon (born 5 July 1975 in Adelaide) is a former Australian rules footballer who played with Adelaide, Geelong and the Brisbane Lions in the Australian Football League (AFL). McKinnon was recruited by Adelaide in the 1992 AFL Draft, with their first ever national draft pick. He was the youngest player on Adelaide's list at the time and played for Central District in the SANFL when not appearing with Adelaide. He made his AFL debut late in the 1994 AFL season and was a regular in the team the following year, playing mostly as a wingman. He»\n",+ "\n",+ "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",+ "\n",+ "Rationale: Let's think step by step.\u001b[32m The song \"Right Back at It Again\" is co-written by the singer Jeremy McKinnon. A quick search of \"Jeremy McKinnon\" reveals that he was born in Gainesville, Florida.\n",+ "\n",+ "Answer: Gainesville, Florida\u001b[0m\n",+ "\n",+ "\n",+ "\n"+ ]+ },+ {+ "data": {+ "text/plain": [+ "('Gainesville, Florida', None)"+ ]+ },+ "execution_count": 8,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "multihop_QA_v1(dev[2].question), lm.inspect_history(n=3)"+ ]+ },+ {+ "cell_type": "code",+ "execution_count": 9,+ "id": "51e7739c-7c5e-4417-a66e-030f615c4858",+ "metadata": {},+ "outputs": [+ {+ "name": "stderr",+ "output_type": "stream",+ "text": [+ "100%|██████████| 13/13 [00:00<00:00, 48.52it/s]\n"+ ]+ },+ {+ "name": "stdout",+ "output_type": "stream",+ "text": [+ "Answered 7 / 13 (53.8%) correctly.\n"+ ]+ },+ {+ "data": {+ "text/html": [+ "<style type=\"text/css\">\n",+ "#T_dbce4 th {\n",+ " text-align: left;\n",+ "}\n",+ "#T_dbce4 td {\n",+ " text-align: left;\n",+ "}\n",+ "</style>\n",+ "<table id=\"T_dbce4\">\n",+ " <thead>\n",+ " <tr>\n",+ " <th class=\"blank level0\" > </th>\n",+ " <th id=\"T_dbce4_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",+ " <th id=\"T_dbce4_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",+ " <th id=\"T_dbce4_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",+ " <th id=\"T_dbce4_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",+ " </tr>\n",+ " </thead>\n",+ " <tbody>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",+ " <td id=\"T_dbce4_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",+ " <td id=\"T_dbce4_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",+ " <td id=\"T_dbce4_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",+ " <td id=\"T_dbce4_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",+ " <td id=\"T_dbce4_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",+ " <td id=\"T_dbce4_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",+ " <td id=\"T_dbce4_row1_col2\" class=\"data row1 col2\" >The Long Island Serial Killer: A Mother's Hunt for Justice</td>\n",+ " <td id=\"T_dbce4_row1_col3\" class=\"data row1 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",+ " <td id=\"T_dbce4_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",+ " <td id=\"T_dbce4_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",+ " <td id=\"T_dbce4_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",+ " <td id=\"T_dbce4_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",+ " <td id=\"T_dbce4_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",+ " <td id=\"T_dbce4_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",+ " <td id=\"T_dbce4_row3_col2\" class=\"data row3 col2\" >1947</td>\n",+ " <td id=\"T_dbce4_row3_col3\" class=\"data row3 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",+ " <td id=\"T_dbce4_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",+ " <td id=\"T_dbce4_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",+ " <td id=\"T_dbce4_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",+ " <td id=\"T_dbce4_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",+ " <td id=\"T_dbce4_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",+ " <td id=\"T_dbce4_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",+ " <td id=\"T_dbce4_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",+ " <td id=\"T_dbce4_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",+ " <td id=\"T_dbce4_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",+ " <td id=\"T_dbce4_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",+ " <td id=\"T_dbce4_row6_col2\" class=\"data row6 col2\" >UK</td>\n",+ " <td id=\"T_dbce4_row6_col3\" class=\"data row6 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",+ " <td id=\"T_dbce4_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",+ " <td id=\"T_dbce4_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",+ " <td id=\"T_dbce4_row7_col2\" class=\"data row7 col2\" >September 28, 1969</td>\n",+ " <td id=\"T_dbce4_row7_col3\" class=\"data row7 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",+ " <td id=\"T_dbce4_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",+ " <td id=\"T_dbce4_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",+ " <td id=\"T_dbce4_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",+ " <td id=\"T_dbce4_row8_col3\" class=\"data row8 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",+ " <td id=\"T_dbce4_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",+ " <td id=\"T_dbce4_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",+ " <td id=\"T_dbce4_row9_col2\" class=\"data row9 col2\" >space technology</td>\n",+ " <td id=\"T_dbce4_row9_col3\" class=\"data row9 col3\" >❌</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",+ " <td id=\"T_dbce4_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",+ " <td id=\"T_dbce4_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",+ " <td id=\"T_dbce4_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",+ " <td id=\"T_dbce4_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",+ " <td id=\"T_dbce4_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",+ " <td id=\"T_dbce4_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",+ " <td id=\"T_dbce4_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",+ " <td id=\"T_dbce4_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",+ " </tr>\n",+ " <tr>\n",+ " <th id=\"T_dbce4_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",+ " <td id=\"T_dbce4_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",+ " <td id=\"T_dbce4_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",+ " <td id=\"T_dbce4_row12_col2\" class=\"data row12 col2\" >1442</td>\n",+ " <td id=\"T_dbce4_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",+ " </tr>\n",+ " </tbody>\n",+ "</table>\n"+ ],+ "text/plain": [+ "<pandas.io.formats.style.Styler at 0x7f51f31e6970>"+ ]+ },+ "metadata": {},+ "output_type": "display_data"+ },+ {+ "data": {+ "text/plain": [+ "53.8"+ ]+ },+ "execution_count": 9,+ "metadata": {},+ "output_type": "execute_result"+ }+ ],+ "source": [+ "from dsp.evaluation.utils import evaluate\n",+ "\n",+ "evaluate(multihop_QA_v1, dev)"+ ]+ }+ ],+ "metadata": {+ "kernelspec": {+ "display_name": "Python (DSP)",+ "language": "python",+ "name": "dsp"+ },+ "language_info": {+ "codemirror_mode": {+ "name": "ipython",+ "version": 3+ },+ "file_extension": ".py",+ "mimetype": "text/x-python",+ "name": "python",+ "nbconvert_exporter": "python",+ "pygments_lexer": "ipython3",+ "version": "3.9.16"+ }+ },+ "nbformat": 4,+ "nbformat_minor": 5+}