Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Add Program 5

Rick Battle3619f5fa8c0d
diff --git a/demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb b/demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb
new file mode 100644
index 000000000..b5fde8052
--- /dev/null
+++ b/demos/Multi-hop QA Program 5 - Multi-hop Condensed Retrieval with Automatic Demos and Query Fusion.ipynb
@@ -0,0 +1,755 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "cfc088a0-2448-46a3-ba0b-3024bddd0a4b",
+ "metadata": {},
+ "source": [
+ "# Multi-hop QA Program 5: Multi-hop Condensed Retrieval w/ Automatic Demos and Query Fusion\n",
+ "\n",
+ "This notebook is a stand-alone version of Program 5 from the intro notebook."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0adb6404-7357-43f9-88dc-fdc1d72ad0c1",
+ "metadata": {},
+ "source": [
+ "### Installation\n",
+ "\n",
+ "If you haven't installed **DSP** already, let's do that.\n",
+ "\n",
+ "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0dab61b4-dad5-4d80-bd41-2ec229f7bf83",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
+ " import google.colab \n",
+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
+ "except: pass\n",
+ "\n",
+ "!pip install -U pip dsp-ml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "14abe1f4-2f0c-4b9d-8ef3-a1800350fb3f",
+ "metadata": {},
+ "source": [
+ "### Setting Up\n",
+ "\n",
+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
+ "\n",
+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
+ "\n",
+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
+ "\n",
+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "aa84afba-2e23-4091-9103-e4f5ee077db2",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Not loading Cohere because it is not installed.\n"
+ ]
+ }
+ ],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2\n",
+ "\n",
+ "try: import google.colab; root_path = 'dsp'\n",
+ "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
+ "except: root_path = '../'\n",
+ "\n",
+ "import os\n",
+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
+ "\n",
+ "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
+ "import sys\n",
+ "sys.path.insert(0, '../')\n",
+ "\n",
+ "import dsp\n",
+ "\n",
+ "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
+ "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
+ "\n",
+ "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
+ "rm = dsp.ColBERTv2(url=colbert_server)\n",
+ "\n",
+ "dsp.settings.configure(lm=lm, rm=rm)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "238c8ab9-b950-4b9d-b20b-9b4ab4ef029f",
+ "metadata": {},
+ "source": [
+ "### Task Examples\n",
+ "\n",
+ "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
+ "\n",
+ "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
+ "\n",
+ "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "178c1031-b350-4189-a209-4e916383a791",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
+ " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
+ " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
+ " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
+ " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
+ " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
+ "\n",
+ "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "170501b3-b7e9-46a5-ba8c-e3b0b586f8ab",
+ "metadata": {},
+ "source": [
+ "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "ed645900-9e75-44f3-8469-0d9eb127d987",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
+ " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
+ " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
+ " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
+ " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
+ " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
+ " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
+ " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
+ " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
+ " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
+ " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
+ " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
+ " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
+ "\n",
+ "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2ac85dda-9526-4e73-b4d8-932c25fab9fb",
+ "metadata": {},
+ "source": [
+ "### Program Definition\n",
+ "\n",
+ "Through Program 4, we've begun to explore some of the power of the DSP abstraction. However, if you look closely, you will see a few downsides of the previous approach:\n",
+ "\n",
+ "1. The search transformations invoke the LM without any demonstrations in the prompt. That is because we only have training data for the question–answer pairs and not for the intermediate labels (e.g., search queries).\n",
+ "2. The QA prompt uses passages (`context`) and a Chain-of-Thought (`rationale`) for the question to be answered. However, the training demonstrations include neither context nor CoT because they aren't available in our labels.\n",
+ "3. The search transformations commit to a single query per hop, which may single out an unproductive chain of passages and hence fail to uncover relevant information."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "1fb2dd8e-b45f-4a62-8351-8274eb71f269",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from dsp.utils import deduplicate\n",
+ "\n",
+ "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
+ "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
+ "\n",
+ "Context = dsp.Type(\n",
+ " prefix=\"Context:\\n\",\n",
+ " desc=\"${sources that may contain relevant content}\",\n",
+ " format=dsp.passages2text\n",
+ ")\n",
+ "\n",
+ "Rationale = dsp.Type(\n",
+ " prefix=\"Rationale: Let's think step by step.\",\n",
+ " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
+ ")\n",
+ "\n",
+ "qa_template_with_CoT = dsp.Template(\n",
+ " instructions=\"Answer questions with short factoid answers.\",\n",
+ " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
+ ")\n",
+ "\n",
+ "SearchRationale = dsp.Type(\n",
+ " prefix=\"Rationale: Let's think step by step. To answer this question, we first need to find out\",\n",
+ " desc=\"${the missing information}\"\n",
+ ")\n",
+ "\n",
+ "SearchQuery = dsp.Type(\n",
+ " prefix=\"Search Query:\",\n",
+ " desc=\"${a simple question for seeking the missing information}\"\n",
+ ")\n",
+ "\n",
+ "CondenseRationale = dsp.Type(\n",
+ " prefix=\"Rationale: Let's think step by step. Based on the context, we have learned the following.\",\n",
+ " desc=\"${information from the context that provides useful clues}\"\n",
+ ")\n",
+ "\n",
+ "rewrite_template = dsp.Template(\n",
+ " instructions=\"Write a search query that will help answer a complex question.\",\n",
+ " question=Question(), rationale=SearchRationale(), query=SearchQuery()\n",
+ ")\n",
+ "\n",
+ "hop_template = dsp.Template(\n",
+ " instructions=rewrite_template.instructions,\n",
+ " context=Context(), question=Question(), rationale=CondenseRationale(), query=SearchQuery()\n",
+ ")\n",
+ "\n",
+ "@dsp.transformation\n",
+ "def QA_predict(example: dsp.Example, sc=True):\n",
+ " if sc:\n",
+ " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
+ " completions = dsp.majority(completions)\n",
+ " else:\n",
+ " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
+ " \n",
+ " return example.copy(answer=completions.answer)\n",
+ "\n",
+ "@dsp.transformation\n",
+ "def multihop_search_v1(example: dsp.Example, max_hops=2, k=2) -> dsp.Example:\n",
+ " example.context = []\n",
+ " \n",
+ " for hop in range(max_hops):\n",
+ " # Generate a query based\n",
+ " template = rewrite_template if hop == 0 else hop_template\n",
+ " example, completions = dsp.generate(template)(example, stage=f'h{hop}')\n",
+ "\n",
+ " # Retrieve k results based on the query generated\n",
+ " passages = dsp.retrieve(completions.query, k=k)\n",
+ "\n",
+ " # Update the context by concatenating old and new passages\n",
+ " example.context = deduplicate(example.context + passages)\n",
+ "\n",
+ " return example"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6275c005-4188-4fe1-8e08-cfd0d4870d5f",
+ "metadata": {},
+ "source": [
+ "We address these problems automatically in the program below.\n",
+ "\n",
+ "In it, we begin by defining `multihop_demonstrate` (which uses `multihop_attempt`) to automatically **annotate** demonstrations for the complex multi-hop pipeline. These demonstrations will be provided to the LM when it's invoked for each transformation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "e2901fb7-06fb-41d5-a158-bb86b99be5b6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "@dsp.transformation\n",
+ "def multihop_attempt(d: dsp.Example) -> dsp.Example:\n",
+ " # Prepare unaugmented demonstrations for the example.\n",
+ " x = dsp.Example(question=d.question, demos=dsp.all_but(train, d))\n",
+ " \n",
+ " # Search. And skip examples where search fails.\n",
+ " # Annotate demonstrations for multihop_search_v2 with the simpler multihop_search_v1 pipeline.\n",
+ " x = multihop_search_v1(x)\n",
+ " if not dsp.passage_match(x.context, d.answer): return None\n",
+ " \n",
+ " # Predict. And skip examples where predict fails.\n",
+ " x = QA_predict(x, sc=False)\n",
+ " if not dsp.answer_match(x.answer, d.answer): return None\n",
+ " \n",
+ " return d.copy(**x)\n",
+ "\n",
+ "@dsp.transformation\n",
+ "def multihop_demonstrate(x: dsp.Example) -> dsp.Example:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " x.demos = dsp.annotate(multihop_attempt)(demos, k=3, return_all=True)\n",
+ " return x"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d2b5d5ef-88f8-40de-8a55-0303ebe6699d",
+ "metadata": {},
+ "source": [
+ "We now implement `multihop_search_v2` as part of `multihop_QA_v2`.\n",
+ "\n",
+ "In addition to the changes mentioned earlier, this program simulates the Baleen system (Khattab et al., 2021) in a few lines of code.\n",
+ "\n",
+ "In each retrieval hop (after the very first hop), the summary of the previous hop(s) is included in the prompt. This allows us to efficiently read a larger number of passages from the current hop."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "52efb3e1-2544-4a16-acbb-3be181bfc559",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "@dsp.transformation\n",
+ "def multihop_search_v2(example: dsp.Example, max_hops=2, k=5) -> dsp.Example:\n",
+ " example.context = []\n",
+ "\n",
+ " for hop in range(max_hops):\n",
+ " # Generate queries\n",
+ " template = rewrite_template if hop == 0 else hop_template\n",
+ " example, completions = dsp.generate(template, n=10, temperature=0.7)(example, stage=f'h{hop}')\n",
+ " \n",
+ " # Collect the queries and search with result fusion\n",
+ " queries = [c.query for c in completions] + [example.question]\n",
+ " example.context = dsp.retrieveEnsemble(queries, k=k)\n",
+ "\n",
+ " # Arrange the passages for the next hop\n",
+ " if hop > 0:\n",
+ " example.context = [completions[0].rationale] + example.context\n",
+ " \n",
+ " return example\n",
+ "\n",
+ "def multihop_QA_v2(question: str) -> str:\n",
+ " x = dsp.Example(question=question)\n",
+ " x = multihop_demonstrate(x)\n",
+ " x = multihop_search_v2(x)\n",
+ " x = QA_predict(x)\n",
+ " return x.answer"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "4b8603e4-335c-4334-aa44-8b6ce5490326",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Write a search query that will help answer a complex question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out ${the missing information}\n",
+ "Search Query: ${a simple question for seeking the missing information}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the first book of Gary Zukav.\n",
+ "Search Query: \"What is the name of Gary Zukav's first book?\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out who the heir to the Du Pont family fortune is.\n",
+ "Search Query: \"Heir to the Du Pont family fortune\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out the name of the 2009 movie featuring Peter Outerbridge as William Easton.\n",
+ "Search Query: \"2009 movie featuring Peter Outerbridge as William Easton\"\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
+ "Rationale: Let's think step by step. To answer this question, we first need to find out\u001b[32m the name of the party of the winner of the 1971 San Francisco mayoral election.\n",
+ "Search Query: \"Party of the winner of the 1971 San Francisco mayoral election\"\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Write a search query that will help answer a complex question.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. ${information from the context that provides useful clues}\n",
+ "\n",
+ "Search Query: ${a simple question for seeking the missing information}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
+ "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. Gary Zukav is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. His first book, \"The Dancing Wu Li Masters\" was written with extensive help from Jack Sarfatti and other physicists.\n",
+ "\n",
+ "Search Query: Which award did \"The Dancing Wu Li Masters\" receive?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
+ "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. The Du Pont family is a wealthy American family with a long history of business and philanthropic ventures. In the 1980s, one of the heirs, John Eleuthère du Pont, became interested in wrestling and established a wrestling facility at his farm.\n",
+ "\n",
+ "Search Query: Which wrestling team did John Eleuthère du Pont sponsor?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
+ "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following. Outerbridge was born and raised in Toronto, he is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film.\n",
+ "\n",
+ "Search Query: Who was the director of the 2009 movie \"Saw VI\"?\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «1971 San Francisco mayoral election | 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another as advocates for law and order. Alioto challenged Dobbs to commit to keeping Police Chief Donald Scott and accused Feinstein of wanting to create a civilian review board to investigate police conduct.»\n",
+ "[2] «1971 San Francisco mayoral election | the November 2 election was the re-election of Mayor Joseph Alioto by a 3–2 margin over his nearest challenger. Alioto quoted Jackie Gleason—\"How sweet it is!\"—at a victory celebration and referred to his federal indictment as an attempt by the \"national administration\" to \"knock out a mayor\". Among other races on the ballot was Richard Hongisto's surprise victory for sheriff over incumbent Matthew Carberry. On the ballot was a referendum limiting new-building heights to six stories, which was defeated. A referendum to make the school board an elective one, it being the only appointive board in the state, passed. The»\n",
+ "[3] «1971 San Francisco mayoral election | school-board measure was regarded as a reaction to busing initiated by the school board under court order. Another measure approved was to keep cable car service at no less than current levels. 1971 San Francisco mayoral election The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote, from among 11 candidates, there being no provision for a runoff. The three major contenders—Joe Alioto; Harold Dobbs, former member of the Board of Supervisors; and Dianne Feinstein, president of the Board of Supervisors—tried to outdo one another»\n",
+ "[4] «1971 San Francisco mayoral election | achieve ethnic balance in elementary schools. Former newspaper editor Scott Newhall proposed staging a world ecology fair and replacing California's water plan by importing icebergs, and imputed little difference between the top three contenders, whom he lumped together as \"Joe Dobbstein\". Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official. As of November, Dobbs, the only Republican in the race, had passed Feinstein in polls and was gaining on Alioto. The result of»\n",
+ "[5] «2003 San Francisco mayoral election | against leftist/progressive Gonzalez, a member of the Green Party. It was the first mayoral election in San Francisco that a Green Party candidate took a noticeable amount of the vote. The election was close, with Gonzalez leading in the polls and winning the popular vote among ballots cast on election day, while Newsom had a larger lead on absentee ballots. 2003 San Francisco mayoral election The 2003 San Francisco mayoral election occurred on November 4, 2003. The incumbent, Willie Brown, was termed out of office and could not seek a third term. The general election included three top candidates including»\n",
+ "\n",
+ "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
+ "\n",
+ "Rationale: Let's think step by step. Based on the context, we have learned the following.\u001b[32m The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.\n",
+ "\n",
+ "Search Query: What year was the Democratic Party founded?\u001b[0m\u001b[31m \t (and 9 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
+ "\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Gary Zukav | Gary Zukav Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award. Gary Zukav was born in Port Arthur, Texas, and spent his early childhood in San Antonio and Houston. His family moved to Pittsburg, Kansas, while he was in fourth grade. In»\n",
+ "[2] «Gary Zukav | experience that led to his first book, \"Dancing Wu Li Masters\", written with extensive help from Jack Sarfatti and other physicists he met through Sarfatti as described in David Kaiser's book \"How the Hippies Saved Physics\". He later described this book as his \"first gift to Life\". In 1987 he moved to Mount Shasta, California, where he lived in a cabin as a self-described \"secular monk\" and spent extensive time in the surrounding wilderness. In 1993 he met Linda Francis. They co-founded the Seat of the Soul Institute in 1998 and moved to Ashland, Oregon, in 2000. Christopher Lehmann-Haupt reviewed»\n",
+ "[3] «The Dancing Wu Li Masters | The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the»\n",
+ "[4] «The Dancing Wu Li Masters | a 1976 physics conference of eastern and western scientists at Esalen Institute, California; and he used the occasion as material for his book. At the conference, it was said that the Chinese term for physics is 'Wu Li', or \"patterns of organic energy.\" Zukav, among others, conceptualized 'physics' as the dance of the Wu Li Masters – teachers of physical essence. Zukav explains the concept further: The Dancing Wu Li Masters The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980»\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "\n",
+ "Rationale: Let's think step by step. The first book is \"The Dancing Wu Li Masters\" and it was awarded a U.S. National Book Award.\n",
+ "\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «John du Pont | John du Pont John Eleuthère du Pont (November 22, 1938 – December 9, 2010) was an American philanthropist, heir to the Du Pont family fortune, and a convicted murderer. He had been a published ornithologist, philatelist, conchologist, sports enthusiast, and self-styled wrestling coach. He died in prison while serving a sentence of 30 years for the murder of Dave Schultz. In 1972, du Pont founded and directed the Delaware Museum of Natural History and contributed to Villanova University and other institutions. In the 1980s, he established a wrestling facility at his Foxcatcher Farm after becoming interested in the sport and»\n",
+ "[2] «Du Pont family | with political and business ventures, as well as philanthropic efforts on behalf of various organizations such as The Jockey Club, the Philadelphia Orchestra, the Delaware Art Museum, and the Grand Opera House. Two family members were the subjects of well-publicized criminal cases. Heir John Eleuthère du Pont was convicted of murdering wrestling coach Dave Schultz in 1996 and heir Robert H. Richards IV was convicted of raping his 3-year-old daughter in 2009. The family is depicted in the 2014 biographical film \"Foxcatcher\" with Steve Carell playing John Eleuthère du Pont and Vanessa Redgrave playing Jean du Pont, the wife of»\n",
+ "[3] «John du Pont | Foxcatcher team. Du Pont became a sponsor in wrestling, swimming, track, and the modern pentathlon. He was also involved in promoting a subset of the modern pentathlon (run, swim, shoot) as a separate event. He took up athletics and became a competitive wrestler in his 50s. His only prior wrestling experience was as a freshman in high school. He began competing again at the age of 55 in the 1992 Veteran's World Championships in Cali, Colombia; following that in 1993 in Toronto, Ontario; in 1994 in Rome, Italy; and in 1995 in Sofia, Bulgaria. In August 1988, a problem-plagued wrestling»\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "\n",
+ "Rationale: Let's think step by step. The question is asking for a wrestling team that the heir to the Du Pont family sponsored. In the context, it is mentioned that the heir, John du Pont, was interested in wrestling and even competed in wrestling competitions in his 50s. Furthermore, it is stated that he sponsored a wrestling team. Therefore, the answer must be a wrestling team.\n",
+ "\n",
+ "Answer: Foxcatcher team\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Peter Outerbridge | Peter Outerbridge Peter Outerbridge (born June 30, 1966) is a Canadian actor, best known for his role as Ari Tasarov in the CW action series \"Nikita\", Dr. David Sandström in the TMN series \"ReGenesis\", Henrik \"Hank\" Johanssen in \"Orphan Black\", Bob Corbett in \"Bomb Girls\", William Easton in \"Saw VI\" and George Brown in the television film \"\". He also played the lead role of Detective William Murdoch in a three-episode mini-series, \"The Murdoch Mysteries\", in its initial run on Canadian television, with two episodes shown in 2004 and a third in 2005. Outerbridge was born and raised in Toronto,»\n",
+ "[2] «Except the Dying | 2008. On March 3, 2015, Acorn Media announced a re-release for all three movies, set for May 26, 2015. Except the Dying Except the Dying is a 2004 made-for-TV film starring Peter Outerbridge, Colm Meaney, Keeley Hawes, William B. Davis and Flora Montgomery. It was adapted by Janet MacLean from the novel of the same name by Maureen Jennings. In this film, William Murdoch is introduced, as a man of strong principles, who uses his unique abilities to solve crimes, sometimes using advanced science for his time. On the street of Toronto, in the 1890s, the naked body of a»\n",
+ "[3] «Saw VI | Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus on the posthumous effects of the Jigsaw Killer and the progression of his successor, Mark Hoffman. In this film, Hoffman sets a series of traps for an insurance executive, William»\n",
+ "[4] «Saw VI | is a crafty move on the screenwriters' part, but once you get past that pseudo-ironic touch, this Saw is no more or less disturbing than any other in the series\". Saw VI Saw VI is a 2009 American horror film directed by Kevin Greutert from a screenplay written by Patrick Melton and Marcus Dunstan. It is the sixth installment in the \"Saw\" franchise and stars Tobin Bell, Costas Mandylor, Betsy Russell, Mark Rolston, Peter Outerbridge, and Shawnee Smith. It was produced by Mark Burg and Oren Koules of Twisted Pictures and distributed by Lionsgate. \"Saw VI\" continues the franchise's focus»\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "\n",
+ "Rationale: Let's think step by step. The question is asking for the director of the 2009 movie that features Peter Outerbridge as William Easton. We can use process of elimination to figure this out. In source 1, there is no mention of a 2009 movie. In source 2, the only mention of Peter Outerbridge is in the cast list for a 2004 made-for-TV film. In source 3, we see that Outerbridge was in a 2009 movie, Saw VI, and that it was directed by Kevin Greutert. This must be the correct answer, so we can check source 4 to confirm. In source 4, we see that Kevin Greutert is in fact the director of Saw VI.\n",
+ "\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «The San Francisco mayoral election of 1971 was held on November 2, 1971, with incumbent Joseph Alioto being re-elected with 38.6 percent of the vote. Alioto was a slight favorite to win in spite of being scheduled to go on trial the following January on charges of conspiracy, mail fraud, and using the mail to promote bribery of a public official.»\n",
+ "[2] «Democratic Party (UK, 1998) | Democratic Party (UK, 1998) The Democratic Party was a political party in the United Kingdom, founded in November 1998 by Malvern businessman Geoff Southall, who also funded the party. It was described as \"right wing\" or \"hard right\" in news reports, but claimed to occupy the centre-ground of British politics. It aimed to reduce Britain's involvement with the European Union, opposed the adoption of the Euro, called for direct democracy, and argued for limits on immigration. The slogan was \"The will of the people NOT the party\". In 1999 it had a few hundred members, including previous supporters of James»\n",
+ "[3] «Democratic Party (Brazil, 1925-1934) | Democratic Party (Brazil, 1925-1934) The Democratic Party was a Brazilian political party, but state character, founded by dissidents of the Paulista Republican Party (PRP) on February 24, 1925, during the Old Republic and was its manifesto signed by 599 signatories. It was a representative organization of the traditional middle class, linked to coffee sectors, but especially the urban. In May 1926, the Youth Party (Youth Wing of dissenters, who had organized this caption for the federal elections of 1925), also dissident perrepismo, joined the Democratic Party. His most important names were Waldemar Ferreira, José Adriano Marrey Junior, Francisco Morato, Paulo»\n",
+ "[4] «Democratic Party (Turkey, 1970) | Democratic Party (Turkey, 1970) Democratic Party (Turkish: \"Demokratik Parti\", DP for short) was a former political party in Turkey founded on 18 December 1970 . It should not be confused with the historical Democrat Party or the current Democrat Party in Turkey. The historical Democrat Party which was the ruling party between 1950-1960 was closed by the military rule. In 1961 and 1965 elections most of ex Democratic Party adherents supported Justice Party, a party founded in 1961. After 1965 elections Justice Party came to power. But following 1969 elections in which Justice Party was again successful, tension between the»\n",
+ "[5] «Democratic Party (Japan, 1947) | Democratic Party (Japan, 1947) Democratic Party (民主党, Minshutō) was a centrist political party in Japan. It was founded in spring 1947 by merging the Progressive Party (Shinpo-tō) of Inukai Takeru with a faction of Liberal Party led by Hitoshi Ashida and obtained 124 seats in 1947 elections. The party had held seven seats in Tetsu Katayama's government in 1947-1948. For some months in 1948, party's leader Ashida was Prime minister. In March 1948, part of DP members led by Kijūrō Shidehara joined the Liberal Party to form the Democratic Liberal Party. In 1949 elections, the DP got 69 seats. The»\n",
+ "[6] «Democratic Party (United States) | Democratic Party (United States) The Democratic Party is one of the two major contemporary political parties in the United States, along with the Republican Party. Tracing its heritage back to Thomas Jefferson and James Madison's Democratic-Republican Party, the modern-day Democratic Party was founded around 1828 by supporters of Andrew Jackson, making it the world's oldest active political party. The Democrats' dominant worldview was once social conservatism and economic liberalism while populism was its leading characteristic in the rural South. In 1912, Theodore Roosevelt ran as a third-party candidate in the Progressive (\"Bull Moose\") Party, beginning a switch of political platforms»\n",
+ "\n",
+ "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
+ "\n",
+ "Rationale: Let's think step by step.\u001b[32m The question is asking for the year the party of the mayor of San Francisco in 1971 was founded. In source 1, we see that the election was in 1971 and the winner was Joseph Alioto. In source 2, we see that the party was founded in 1998, which is not correct. In source 3, we see that the party was founded in 1925, which is not correct. In source 4, we see that the party was founded in 1970, which is not correct. In source 5, we see that the party was founded in 1947, which is not correct. In source 6, we see that the party was founded around 1828. This must be the correct answer, so we can check source 1 to confirm. In source 1\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "('1828', None)"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "multihop_QA_v2(dev[3].question), lm.inspect_history(n=3)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "7c945f07-27b5-4f1c-a347-9c935a83fd28",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:02<00:00, 6.48it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 11 / 13 (84.6%) correctly.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\">\n",
+ "#T_6a173 th {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_6a173 td {\n",
+ " text-align: left;\n",
+ "}\n",
+ "</style>\n",
+ "<table id=\"T_6a173\">\n",
+ " <thead>\n",
+ " <tr>\n",
+ " <th class=\"blank level0\" >&nbsp;</th>\n",
+ " <th id=\"T_6a173_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_6a173_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_6a173_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_6a173_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_6a173_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_6a173_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_6a173_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_6a173_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_6a173_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_6a173_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_6a173_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_6a173_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_6a173_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_6a173_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_6a173_row2_col2\" class=\"data row2 col2\" >Gainesville, Florida</td>\n",
+ " <td id=\"T_6a173_row2_col3\" class=\"data row2 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_6a173_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_6a173_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_6a173_row3_col2\" class=\"data row3 col2\" >1828</td>\n",
+ " <td id=\"T_6a173_row3_col3\" class=\"data row3 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_6a173_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_6a173_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_6a173_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_6a173_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_6a173_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_6a173_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_6a173_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_6a173_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_6a173_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_6a173_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_6a173_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
+ " <td id=\"T_6a173_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_6a173_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_6a173_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_6a173_row7_col2\" class=\"data row7 col2\" >February 13, 1980</td>\n",
+ " <td id=\"T_6a173_row7_col3\" class=\"data row7 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_6a173_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_6a173_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_6a173_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
+ " <td id=\"T_6a173_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_6a173_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_6a173_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_6a173_row9_col2\" class=\"data row9 col2\" >Aerospace engineering</td>\n",
+ " <td id=\"T_6a173_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_6a173_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_6a173_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_6a173_row10_col2\" class=\"data row10 col2\" >Guildford, Surrey</td>\n",
+ " <td id=\"T_6a173_row10_col3\" class=\"data row10 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_6a173_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_6a173_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_6a173_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_6a173_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_6a173_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_6a173_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_6a173_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_6a173_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_6a173_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7f8adaac2cd0>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "84.6"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from dsp.evaluation.utils import evaluate\n",
+ "\n",
+ "evaluate(multihop_QA_v2, dev)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python (DSP)",
+ "language": "python",
+ "name": "dsp"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.16"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}