Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Add Program 3

Rick Battleda41e9e8ccd8
diff --git a/demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb b/demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb
new file mode 100644
index 000000000..6ee648e0f
--- /dev/null
+++ b/demos/Multi-hop QA Program 3 - Retrieve then Read with Self-Consistency.ipynb
@@ -0,0 +1,521 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "252556ab-bf1d-4574-9763-d66fc2fd872b",
+ "metadata": {},
+ "source": [
+ "# Multi-hop QA Program 3: Retrieve then Read w/ Self-Consistency\n",
+ "\n",
+ "This notebook is a stand-alone version of Program 3 from the intro notebook."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6aa1b28c-3c3d-4436-a54d-e141945bc69f",
+ "metadata": {},
+ "source": [
+ "### Installation\n",
+ "\n",
+ "If you haven't installed **DSP** already, let's do that.\n",
+ "\n",
+ "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7378685e-2100-4e05-8694-ef9b6ef8c0b1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
+ " import google.colab \n",
+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
+ "except: pass\n",
+ "\n",
+ "!pip install -U pip dsp-ml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8ee794e0-ef54-4c6c-86ac-3a8f7b207440",
+ "metadata": {},
+ "source": [
+ "### Setting Up\n",
+ "\n",
+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
+ "\n",
+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
+ "\n",
+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
+ "\n",
+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "26cca18f-aefc-4233-bdbf-93ae1fbf7212",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Not loading Cohere because it is not installed.\n"
+ ]
+ }
+ ],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2\n",
+ "\n",
+ "try: import google.colab; root_path = 'dsp'\n",
+ "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
+ "except: root_path = '../'\n",
+ "\n",
+ "import os\n",
+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
+ "\n",
+ "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
+ "import sys\n",
+ "sys.path.insert(0, '../')\n",
+ "\n",
+ "import dsp\n",
+ "\n",
+ "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
+ "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
+ "\n",
+ "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
+ "rm = dsp.ColBERTv2(url=colbert_server)\n",
+ "\n",
+ "dsp.settings.configure(lm=lm, rm=rm)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e03e272-c76d-4cc4-948d-1fdb6842d3be",
+ "metadata": {},
+ "source": [
+ "### Task Examples\n",
+ "\n",
+ "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
+ "\n",
+ "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
+ "\n",
+ "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "b57307c5-2bfa-4c43-81e0-d9246a65f11a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
+ " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
+ " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
+ " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
+ " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
+ " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
+ "\n",
+ "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e9a6364c-12d8-4298-b087-4b84976b7f92",
+ "metadata": {},
+ "source": [
+ "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "75b5b666-afe0-4d28-b398-1dcc3823175c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
+ " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
+ " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
+ " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
+ " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
+ " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
+ " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
+ " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
+ " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
+ " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
+ " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
+ " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
+ " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
+ "\n",
+ "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b95dc2be-2513-4238-a60a-d9b24184eef7",
+ "metadata": {},
+ "source": [
+ "### Program Definition\n",
+ "\n",
+ "It's clear that retrieval has the capacity to help **GPT-3.5** answer a larger number of questions more factually. However, these questions are too complicated for a single retrieved passage to suffice.\n",
+ "\n",
+ "In this program, we try to make some improvements to the pipeline from **Program 2**.\n",
+ "\n",
+ "In particular:\n",
+ "\n",
+ "- We will include 5 passages (instead of a single passage) into the prompt.\n",
+ "- We will ask GPT-3.5 to generate a Chain-of-Thought (CoT) rationale to more effectively extract the answer from the passages.\n",
+ "- We will use Self-Consistency (SC) to marginalize this prediction across many chains of thought."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "bcdae9d5-4d60-4f95-8561-37a8073422ea",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
+ "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
+ "\n",
+ "Context = dsp.Type(\n",
+ " prefix=\"Context:\\n\",\n",
+ " desc=\"${sources that may contain relevant content}\",\n",
+ " format=dsp.passages2text\n",
+ ")\n",
+ "\n",
+ "Rationale = dsp.Type(\n",
+ " prefix=\"Rationale: Let's think step by step.\",\n",
+ " desc=\"${a step-by-step deduction that identifies the correct response, which will be provided below}\"\n",
+ ")\n",
+ "\n",
+ "qa_template_with_CoT = dsp.Template(\n",
+ " instructions=\"Answer questions with short factoid answers.\",\n",
+ " context=Context(), question=Question(), rationale=Rationale(), answer=Answer()\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5595b3eb-6e8d-47fd-b5b0-6c2d3fa9422e",
+ "metadata": {},
+ "source": [
+ "The snippet above updates our template to include a `rationale` field.\n",
+ "\n",
+ "Below, we define the new program `Retrieve_then_Read_QA_v2`. In it, we define a new DSP _transformation_ `QA_predict`, which we will re-use in multiple programs.\n",
+ "\n",
+ "The transformation `QA_predict` takes an `Example` (which has a question, context, and demonstrations) and generates an answer. It can be configured to use self-consistency (`sc=True`) or not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "688eabdb-5878-46db-8141-ebd8c67acb4d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "@dsp.transformation\n",
+ "def QA_predict(example: dsp.Example, sc=True):\n",
+ " if sc:\n",
+ " example, completions = dsp.generate(qa_template_with_CoT, n=20, temperature=0.7)(example, stage='qa')\n",
+ " completions = dsp.majority(completions)\n",
+ " else:\n",
+ " example, completions = dsp.generate(qa_template_with_CoT)(example, stage='qa')\n",
+ " \n",
+ " return example.copy(answer=completions.answer)\n",
+ "\n",
+ "def Retrieve_then_Read_QA_v2(question: str) -> str:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " passages = dsp.retrieve(question, k=5)\n",
+ " example = dsp.Example(question=question, context=passages, demos=demos)\n",
+ " \n",
+ " return QA_predict(example).answer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fc6b16c7-f7d9-4b74-9f90-45840b4f33a2",
+ "metadata": {},
+ "source": [
+ "Let's inspect an example."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "2d5889d6-88a1-4a60-9069-5f22a0bcafd9",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Answer: Foxcatcher\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Rationale: Let's think step by step. ${a step-by-step deduction that identifies the correct response, which will be provided below}\n",
+ "\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "[1] «Right Back at It Again | Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it sounded great, so on it went.\" \"Right Back»\n",
+ "[2] «Right Back at It Again | at the Kerrang! Awards. Personnel per digital booklet. Right Back at It Again \"Right Back at It Again\" is the second track and the first single from A Day to Remember's fifth album, \"Common Courtesy\" (2013). In October 20, 2015, the song was featured in Activision rhythm-music game, \"\". Vocalist, Jeremy McKinnon wrote the lyrics, while the music was written by McKinnon, former guitarist Tom Denney, guitarist Neil Westfall and producer Andrew Wade. \"Right Back at It Again\" almost wasn't included on the album as it was one of the excess songs the band had recorded, \"we realised that it»\n",
+ "[3] «Right Back at It Again | band played \"Right Back at It Again\" on both occasions. AbsolutePunk writer Thomas Nassiff said the \"mosh call\" lyric \"\"Brace for impact\"\" had \"pretty much ruin[ed] that track\". Tamsyn Wilce for Alter the Press! noted that the way \"Right Back at it Again\" follows on from opening track \"City of Ocala\" \"continues the catchy hardcore beats\" that is present in all of the band's albums. \"Rock Sound\"'s Andy Ritchie called the song \"unmistakably the 'All I Want' of 'Common Courtesy'\". A music video for the song was released on 19 December 2013. The music video was nominated for Best Video»\n",
+ "[4] «Born Again in Dixieland | Born Again in Dixieland \"Born Again in Dixieland\" is a song co-written and recorded by Canadian country music artist Jason McCoy. Released in June 1997, it was the lead single from his album \"Playin' for Keeps\". The song reached #3 on the \"RPM\" Country Tracks chart in September 1997. The song won a SOCAN Song of the Year award. McCoy said that he started writing the song with Sheridan and he was playing songs for MCA Canada label and they asked if he had any other songs. He played a verse and the chorus from the song thinking it wasn't»\n",
+ "[5] «Right Back Where We Started From | 1974. In a 3 May 2008 interview with Michael Shelley of WFMU, Edwards recalled that after hearing Maxine Nightingale sing on the session for Al Matthews' \"Fool\" that track's producer Pierre Tubbs had come up with \"Right Back Where We Started From\" as a good title for a song for Nightingale herself to record and had invited Edwards to co-write the song. Utilizing a tune which Edwards had written \"a couple of years before\", Tubbs and Edwards wrote \"Right Back Where We Started From\" in about seven minutes while driving to Charing Cross Hospital where Tubbs' wife Gabrielle (née Zimmerman)»\n",
+ "\n",
+ "Question: Right Back At It Again contains lyrics co-written by the singer born in what city?\n",
+ "\n",
+ "Rationale: Let's think step by step.\u001b[32m The song is by A Day to Remember, and the lyrics were co-written by the singer, Jeremy McKinnon. We just need to find out where McKinnon was born.\n",
+ "\n",
+ "Answer: Orlando, Florida\u001b[0m\u001b[31m \t (and 19 other completions)\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "('Ocala, Florida', None)"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "Retrieve_then_Read_QA_v2(dev[2].question), lm.inspect_history(n=1)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "4691922b-7725-48cb-812a-9051cdb2b3d4",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:00<00:00, 31.30it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 7 / 13 (53.8%) correctly.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\">\n",
+ "#T_fe9ce th {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_fe9ce td {\n",
+ " text-align: left;\n",
+ "}\n",
+ "</style>\n",
+ "<table id=\"T_fe9ce\">\n",
+ " <thead>\n",
+ " <tr>\n",
+ " <th class=\"blank level0\" >&nbsp;</th>\n",
+ " <th id=\"T_fe9ce_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_fe9ce_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_fe9ce_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_fe9ce_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_fe9ce_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_fe9ce_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_fe9ce_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_fe9ce_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_fe9ce_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_fe9ce_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_fe9ce_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_fe9ce_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_fe9ce_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_fe9ce_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_fe9ce_row2_col2\" class=\"data row2 col2\" >Ocala, Florida</td>\n",
+ " <td id=\"T_fe9ce_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_fe9ce_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_fe9ce_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_fe9ce_row3_col2\" class=\"data row3 col2\" >1824</td>\n",
+ " <td id=\"T_fe9ce_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_fe9ce_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_fe9ce_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_fe9ce_row4_col2\" class=\"data row4 col2\" >John Braine</td>\n",
+ " <td id=\"T_fe9ce_row4_col3\" class=\"data row4 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_fe9ce_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_fe9ce_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_fe9ce_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_fe9ce_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_fe9ce_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_fe9ce_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_fe9ce_row6_col2\" class=\"data row6 col2\" >Cheshire, UK</td>\n",
+ " <td id=\"T_fe9ce_row6_col3\" class=\"data row6 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_fe9ce_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_fe9ce_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_fe9ce_row7_col2\" class=\"data row7 col2\" >September 27, 1974</td>\n",
+ " <td id=\"T_fe9ce_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_fe9ce_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_fe9ce_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_fe9ce_row8_col2\" class=\"data row8 col2\" >Dundalk</td>\n",
+ " <td id=\"T_fe9ce_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_fe9ce_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_fe9ce_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_fe9ce_row9_col2\" class=\"data row9 col2\" >Aeronautics and Astronautics</td>\n",
+ " <td id=\"T_fe9ce_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_fe9ce_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_fe9ce_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_fe9ce_row10_col2\" class=\"data row10 col2\" >Italy</td>\n",
+ " <td id=\"T_fe9ce_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_fe9ce_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_fe9ce_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_fe9ce_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_fe9ce_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_fe9ce_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_fe9ce_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_fe9ce_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_fe9ce_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_fe9ce_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7f8a980bd5b0>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "53.8"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from dsp.evaluation.utils import evaluate\n",
+ "\n",
+ "evaluate(Retrieve_then_Read_QA_v2, dev)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python (DSP)",
+ "language": "python",
+ "name": "dsp"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.16"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}