Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Add Program 2

Rick Battle3d67301b6f0e
diff --git a/demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb b/demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb
new file mode 100644
index 000000000..6b1db8d80
--- /dev/null
+++ b/demos/Multi-hop QA Program 2 - Retrieve then Read.ipynb
@@ -0,0 +1,502 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "6a0e89c2-8e49-4b0d-9b82-7b85919c9597",
+ "metadata": {},
+ "source": [
+ "# Multi-hop QA Program 2: Retrieve then Read\n",
+ "\n",
+ "This notebook is a stand-alone version of Program 1 from the intro notebook."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9943c9eb-a4ab-4ea1-85ef-f6f263abaa56",
+ "metadata": {},
+ "source": [
+ "### Installation\n",
+ "\n",
+ "If you haven't installed **DSP** already, let's do that.\n",
+ "\n",
+ "Note: If you're running this from a cloned copy of the repo, then you can skip this block."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "45373bac-1863-4ec5-a4cc-15f046de273f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
+ " import google.colab \n",
+ " !git -C dsp/ pull || git clone https://github.com/stanfordnlp/dsp\n",
+ "except: pass\n",
+ "\n",
+ "!pip install -U pip dsp-ml"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f535fe40-34a6-41e4-ac1b-b53dd1ac3246",
+ "metadata": {},
+ "source": [
+ "### Setting Up\n",
+ "\n",
+ "We'll start by setting up the language model (LM) and retrieval model (RM).\n",
+ "\n",
+ "We will work with the **GPT-3.5** LM (`text-davinci-002`) and the **ColBERTv2** RM.\n",
+ "\n",
+ "To use GPT-3, you'll need an OpenAI key. For ColBERTv2, we've set up a server hosting a Wikipedia (Dec 2018) search index, so you don't need to worry about setting one up!\n",
+ "\n",
+ "To make things easy, we've set up a cache in this repository. _If you want to run this notebook without changing the code or examples, you don't need an API key. All examples are cached._"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "1a15edff-40a8-4237-9b5f-6be8de7cb1df",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Not loading Cohere because it is not installed.\n"
+ ]
+ }
+ ],
+ "source": [
+ "%load_ext autoreload\n",
+ "%autoreload 2\n",
+ "\n",
+ "try: import google.colab; root_path = 'dsp'\n",
+ "# The root path is ../ if you're running this from the demo folder of the cloned repository\n",
+ "except: root_path = '../'\n",
+ "\n",
+ "import os\n",
+ "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(root_path, 'cache')\n",
+ "\n",
+ "# Add ../ to the path to import dsp if you're running this directly from the cloned copy of the repo (without pip installing dsp)\n",
+ "import sys\n",
+ "sys.path.insert(0, '../')\n",
+ "\n",
+ "import dsp\n",
+ "\n",
+ "openai_key = os.getenv('OPENAI_API_KEY') # or replace with your API key (optional)\n",
+ "colbert_server = 'http://ec2-44-228-128-229.us-west-2.compute.amazonaws.com:8893/api/search'\n",
+ "\n",
+ "lm = dsp.GPT3(model='text-davinci-002', api_key=openai_key)\n",
+ "rm = dsp.ColBERTv2(url=colbert_server)\n",
+ "\n",
+ "dsp.settings.configure(lm=lm, rm=rm)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ff9b842b-4cb9-4756-b375-c4204c78413b",
+ "metadata": {},
+ "source": [
+ "### Task Examples\n",
+ "\n",
+ "Next, let's look at a few examples of the task. Each example consists of a question and one or more gold answers.\n",
+ "\n",
+ "We have six training examples (`train`), which we'll feed into the programs. These will help define the task.\n",
+ "\n",
+ "Notice that our examples only have input (`question`) and output (`answer`) fields. When our advanced programs build sophisticated pipelines, training \"demonstrations\" for other fields will be constructed automatically."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "e245f705-4602-485d-a322-0dba969868b0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "train = [('Who produced the album that included a re-recording of \"Lithium\"?', ['Butch Vig']),\n",
+ " ('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', ['Kevin Greutert']),\n",
+ " ('The heir to the Du Pont family fortune sponsored what wrestling team?', ['Foxcatcher', 'Team Foxcatcher', 'Foxcatcher Team']),\n",
+ " ('In what year was the star of To Hell and Back born?', ['1925']),\n",
+ " ('Which award did the first book of Gary Zukav receive?', ['U.S. National Book Award', 'National Book Award']),\n",
+ " ('What city was the victim of Joseph Druces working in?', ['Boston, Massachusetts', 'Boston']),]\n",
+ "\n",
+ "train = [dsp.Example(question=question, answer=answer) for question, answer in train]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a280ea89-476e-43d6-a485-2cb064dda4bf",
+ "metadata": {},
+ "source": [
+ "The development examples (`dev`) will be used to assess the behavior of each program we build. Of course, this tiny set is not meant to be a reliable benchmark, but it'll be instructive to use it for illustration."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "23e5011c-1d5d-4355-8d9e-5ca83c4d332c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', ['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']),\n",
+ " ('What documentary about the Gilgo Beach Killer debuted on A&E?', ['The Killing Season']),\n",
+ " ('Right Back At It Again contains lyrics co-written by the singer born in what city?', ['Gainesville, Florida', 'Gainesville']),\n",
+ " ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', ['1828']),\n",
+ " ('Which author is English: John Braine or Studs Terkel?', ['John Braine']),\n",
+ " ('Anthony Dirrell is the brother of which super middleweight title holder?', ['Andre Dirrell']),\n",
+ " ('In which city is the sports nutrition business established by Oliver Cookson based ?', ['Cheshire', 'Cheshire, UK']),\n",
+ " ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', ['February 13, 1980']),\n",
+ " ('Kyle Moran was born in the town on what river?', ['Castletown', 'Castletown River']),\n",
+ " (\"What is the name of one branch of Robert D. Braun's speciality?\", ['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']),\n",
+ " (\"Where was the actress who played the niece in the Priest film born?\", ['Surrey', 'Guildford, Surrey']),\n",
+ " ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', ['Portrait of a Marriage']),\n",
+ " ('What year was the father of the Princes in the Tower born?', ['1442'])]\n",
+ "\n",
+ "dev = [dsp.Example(question=question, answer=answer) for question, answer in dev]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "687f0c48-5dcf-4679-a498-90498ed65f6c",
+ "metadata": {},
+ "source": [
+ "### Program Definition\n",
+ "\n",
+ "Program 1 relies entirely on the factual knowledge memorized by GPT-3. As we have seen, it's sometimes correct but it's far from reliable.\n",
+ "\n",
+ "Let's try to address this. Our first improvement to the vanilla **GPT-3.5** strategy is to include search results in the LM prompt, often called a *retrieve-then-read* approach.\n",
+ "\n",
+ "In particular, we'll retrieve the most relevant passage to the question (according to the retriever) from the corpus and concatenate it into the prompt. This may help GPT-3.5 answer more factually.\n",
+ "\n",
+ "To do this, we'll need to modify the `Template`. In particular, we'll define a `qa_template_with_passages` template, which will include a `context` field in addition to a question and an answer."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "c07db044-fe6a-47ed-b754-2d2ba23600b2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "Question = dsp.Type(prefix=\"Question:\", desc=\"${the question to be answered}\")\n",
+ "Answer = dsp.Type(prefix=\"Answer:\", desc=\"${a short factoid answer, often between 1 and 5 words}\", format=dsp.format_answers)\n",
+ "\n",
+ "Context = dsp.Type(\n",
+ " prefix=\"Context:\\n\",\n",
+ " desc=\"${sources that may contain relevant content}\",\n",
+ " format=dsp.passages2text\n",
+ ")\n",
+ "\n",
+ "qa_template_with_passages = dsp.Template(\n",
+ " instructions=\"Answer questions with short factoid answers.\",\n",
+ " context=Context(), question=Question(), answer=Answer()\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1a55b5d3-b69c-475f-b83a-b6e066fd8a4b",
+ "metadata": {},
+ "source": [
+ "Now let's modify the program. We'll call `dsp.retrieve`, which we hooked earlier in this notebook to a **ColBERTv2** retriever serving a Wikipedia (Dec 2018) index."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "0fa13294-7730-448d-992a-a4a6d1ac61d5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def Retrieve_then_Read_QA(question: str) -> str:\n",
+ " demos = dsp.sample(train, k=7)\n",
+ " passages = dsp.retrieve(question, k=1)\n",
+ " \n",
+ " example = dsp.Example(question=question, context=passages, demos=demos)\n",
+ " example, completions = dsp.generate(qa_template_with_passages)(example, stage='qa')\n",
+ "\n",
+ " return completions.answer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "52a6bf3b-44ca-47d6-9d47-a3ac5ea0c600",
+ "metadata": {},
+ "source": [
+ "Let's invoke this on a sample question and let's inspect the last call to the LM to learn more about the behavior of the program."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "1fbb8316-9ada-4047-a5c0-754040b08f5d",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "Answer questions with short factoid answers.\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Question: Which award did the first book of Gary Zukav receive?\n",
+ "Answer: U.S. National Book Award\n",
+ "\n",
+ "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
+ "Answer: Foxcatcher\n",
+ "\n",
+ "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
+ "Answer: Kevin Greutert\n",
+ "\n",
+ "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
+ "Answer: Butch Vig\n",
+ "\n",
+ "Question: What city was the victim of Joseph Druces working in?\n",
+ "Answer: Boston, Massachusetts\n",
+ "\n",
+ "Question: In what year was the star of To Hell and Back born?\n",
+ "Answer: 1925\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Follow the following format.\n",
+ "\n",
+ "Context:\n",
+ "${sources that may contain relevant content}\n",
+ "\n",
+ "Question: ${the question to be answered}\n",
+ "\n",
+ "Answer: ${a short factoid answer, often between 1 and 5 words}\n",
+ "\n",
+ "---\n",
+ "\n",
+ "Context:\n",
+ "«Julia Peterkin | Julia Peterkin Julia Peterkin (October 31, 1880 – August 10, 1961) was an American author from South Carolina. In 1929 she won the Pulitzer Prize for Novel/Literature, for her novel \"Scarlet Sister Mary.\" She wrote several novels about the plantation South, especially the Gullah people of the Low Country. She was one of the few white authors who wrote about the African-American experience. Julia Mood was born in Laurens County, South Carolina. Her father was a physician, and she was the third of his four children. Her mother died soon after her birth, and her father later married Janie Brogdon.»\n",
+ "\n",
+ "Question: Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?\n",
+ "\n",
+ "Answer:\u001b[32m E. L. Doctorow\u001b[0m\n",
+ "\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "('E. L. Doctorow', None)"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "Retrieve_then_Read_QA(dev[0].question), lm.inspect_history(n=1)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ad0d7128-9ce6-42c4-af33-2acd0cb2620c",
+ "metadata": {},
+ "source": [
+ "Now, let's evaluate this on the dev examples."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "157c24a3-8daa-4ec7-9261-5e03f33fddb3",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13/13 [00:00<00:00, 130.12it/s]"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Answered 5 / 13 (38.5%) correctly.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "<style type=\"text/css\">\n",
+ "#T_dbaed th {\n",
+ " text-align: left;\n",
+ "}\n",
+ "#T_dbaed td {\n",
+ " text-align: left;\n",
+ "}\n",
+ "</style>\n",
+ "<table id=\"T_dbaed\">\n",
+ " <thead>\n",
+ " <tr>\n",
+ " <th class=\"blank level0\" >&nbsp;</th>\n",
+ " <th id=\"T_dbaed_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
+ " <th id=\"T_dbaed_level0_col1\" class=\"col_heading level0 col1\" >answer</th>\n",
+ " <th id=\"T_dbaed_level0_col2\" class=\"col_heading level0 col2\" >prediction</th>\n",
+ " <th id=\"T_dbaed_level0_col3\" class=\"col_heading level0 col3\" >correct</th>\n",
+ " </tr>\n",
+ " </thead>\n",
+ " <tbody>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
+ " <td id=\"T_dbaed_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
+ " <td id=\"T_dbaed_row0_col1\" class=\"data row0 col1\" >['E. L. Doctorow', 'E.L. Doctorow', 'Doctorow']</td>\n",
+ " <td id=\"T_dbaed_row0_col2\" class=\"data row0 col2\" >E. L. Doctorow</td>\n",
+ " <td id=\"T_dbaed_row0_col3\" class=\"data row0 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
+ " <td id=\"T_dbaed_row1_col0\" class=\"data row1 col0\" >What documentary about the Gilgo Beach Killer debuted on A&E?</td>\n",
+ " <td id=\"T_dbaed_row1_col1\" class=\"data row1 col1\" >['The Killing Season']</td>\n",
+ " <td id=\"T_dbaed_row1_col2\" class=\"data row1 col2\" >The Killing Season</td>\n",
+ " <td id=\"T_dbaed_row1_col3\" class=\"data row1 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
+ " <td id=\"T_dbaed_row2_col0\" class=\"data row2 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
+ " <td id=\"T_dbaed_row2_col1\" class=\"data row2 col1\" >['Gainesville, Florida', 'Gainesville']</td>\n",
+ " <td id=\"T_dbaed_row2_col2\" class=\"data row2 col2\" >Daytona Beach, Florida.</td>\n",
+ " <td id=\"T_dbaed_row2_col3\" class=\"data row2 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
+ " <td id=\"T_dbaed_row3_col0\" class=\"data row3 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
+ " <td id=\"T_dbaed_row3_col1\" class=\"data row3 col1\" >['1828']</td>\n",
+ " <td id=\"T_dbaed_row3_col2\" class=\"data row3 col2\" >1854</td>\n",
+ " <td id=\"T_dbaed_row3_col3\" class=\"data row3 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
+ " <td id=\"T_dbaed_row4_col0\" class=\"data row4 col0\" >Which author is English: John Braine or Studs Terkel?</td>\n",
+ " <td id=\"T_dbaed_row4_col1\" class=\"data row4 col1\" >['John Braine']</td>\n",
+ " <td id=\"T_dbaed_row4_col2\" class=\"data row4 col2\" >John Braine is English.</td>\n",
+ " <td id=\"T_dbaed_row4_col3\" class=\"data row4 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
+ " <td id=\"T_dbaed_row5_col0\" class=\"data row5 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
+ " <td id=\"T_dbaed_row5_col1\" class=\"data row5 col1\" >['Andre Dirrell']</td>\n",
+ " <td id=\"T_dbaed_row5_col2\" class=\"data row5 col2\" >Andre Dirrell</td>\n",
+ " <td id=\"T_dbaed_row5_col3\" class=\"data row5 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
+ " <td id=\"T_dbaed_row6_col0\" class=\"data row6 col0\" >In which city is the sports nutrition business established by Oliver Cookson based ?</td>\n",
+ " <td id=\"T_dbaed_row6_col1\" class=\"data row6 col1\" >['Cheshire', 'Cheshire, UK']</td>\n",
+ " <td id=\"T_dbaed_row6_col2\" class=\"data row6 col2\" >Manchester, England</td>\n",
+ " <td id=\"T_dbaed_row6_col3\" class=\"data row6 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
+ " <td id=\"T_dbaed_row7_col0\" class=\"data row7 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
+ " <td id=\"T_dbaed_row7_col1\" class=\"data row7 col1\" >['February 13, 1980']</td>\n",
+ " <td id=\"T_dbaed_row7_col2\" class=\"data row7 col2\" >August 5, 1962</td>\n",
+ " <td id=\"T_dbaed_row7_col3\" class=\"data row7 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
+ " <td id=\"T_dbaed_row8_col0\" class=\"data row8 col0\" >Kyle Moran was born in the town on what river?</td>\n",
+ " <td id=\"T_dbaed_row8_col1\" class=\"data row8 col1\" >['Castletown', 'Castletown River']</td>\n",
+ " <td id=\"T_dbaed_row8_col2\" class=\"data row8 col2\" >Dundalk, Ireland</td>\n",
+ " <td id=\"T_dbaed_row8_col3\" class=\"data row8 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
+ " <td id=\"T_dbaed_row9_col0\" class=\"data row9 col0\" >What is the name of one branch of Robert D. Braun's speciality?</td>\n",
+ " <td id=\"T_dbaed_row9_col1\" class=\"data row9 col1\" >['aeronautical engineering', 'astronautical engineering', 'aeronautics', 'astronautics']</td>\n",
+ " <td id=\"T_dbaed_row9_col2\" class=\"data row9 col2\" >Jet Propulsion</td>\n",
+ " <td id=\"T_dbaed_row9_col3\" class=\"data row9 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
+ " <td id=\"T_dbaed_row10_col0\" class=\"data row10 col0\" >Where was the actress who played the niece in the Priest film born?</td>\n",
+ " <td id=\"T_dbaed_row10_col1\" class=\"data row10 col1\" >['Surrey', 'Guildford, Surrey']</td>\n",
+ " <td id=\"T_dbaed_row10_col2\" class=\"data row10 col2\" >Enid, Oklahoma</td>\n",
+ " <td id=\"T_dbaed_row10_col3\" class=\"data row10 col3\" >❌</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
+ " <td id=\"T_dbaed_row11_col0\" class=\"data row11 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
+ " <td id=\"T_dbaed_row11_col1\" class=\"data row11 col1\" >['Portrait of a Marriage']</td>\n",
+ " <td id=\"T_dbaed_row11_col2\" class=\"data row11 col2\" >Portrait of a Marriage</td>\n",
+ " <td id=\"T_dbaed_row11_col3\" class=\"data row11 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " <tr>\n",
+ " <th id=\"T_dbaed_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
+ " <td id=\"T_dbaed_row12_col0\" class=\"data row12 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
+ " <td id=\"T_dbaed_row12_col1\" class=\"data row12 col1\" >['1442']</td>\n",
+ " <td id=\"T_dbaed_row12_col2\" class=\"data row12 col2\" >1442</td>\n",
+ " <td id=\"T_dbaed_row12_col3\" class=\"data row12 col3\" >✔️</td>\n",
+ " </tr>\n",
+ " </tbody>\n",
+ "</table>\n"
+ ],
+ "text/plain": [
+ "<pandas.io.formats.style.Styler at 0x7f2ddcdf3580>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "38.5"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from dsp.evaluation.utils import evaluate\n",
+ "\n",
+ "evaluate(Retrieve_then_Read_QA, dev)"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python (DSP)",
+ "language": "python",
+ "name": "dsp"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.16"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}